Table 1 Discussion topics for each paper

Internal validity: Are our studies bias-free and replicable?

Würbel (2017) [1] (other informative works include those by Leonard Freedman and Glenn Begley)

If only 50% biomedical experiments are replicable, what might the equivalent metric be for welfare research?

If a welfare study proves non-replicable, does this matter most for the animals who were the subjects, those who are the “real-world” applied targets, policy makers, or future scientists?

Würbel lists nine factors that can reduce replicability, often acting in different ways. How can underpowered studies do this, for instance? How can non-randomisation? And which might particularly impact the replicability of animal welfare research?

Kilkenny et al. (2009) [2] (see also https://arriveguidelines.org/arrive-guidelines. Other informative articles include ones by Dorothy Bishop, John Ioannidis, Malcolm Macleod, and Emily Sena)

Should the authors have considered blinding in all studies, not just those using subjective scoring?

Do the findings surprise you? Which worries you most and why?

If similar surveys were conducted of animal welfare research, what might they find?

Tuyttens et al. (2014) [3] (see also a 2016 follow-up study and interesting work by Nicole Nelson)

This work used veterinary students. Would biases be even stronger in people concerned about publication?

In your research, are those who handle animals blind to hypothesis and/or treatment groups? Are those collecting the data? Are those analysing the data? What are the merits (e.g. in terms of practicality) of these different types of blinding?

Lazic (2010) [4]

If 12–48% of neuroscience papers may have pseudoreplicated, what might a similar survey of welfare research find?

How does a legitimate repeated measures model differ from pseudoreplication? And if cage/pen is not the experimental unit to which a treatment is applied, do you still include it in your models?

External validity: Are our studies relevant to real-world situations?

Dawkins 2012 [5]

Dawkins urges for more studies in commercial agricultural facilities. Do the benefits outweigh the costs? If you do not do this, what are the barriers? And are you worried that your data then lack generalisability to the populations needing them?

Construct validity: Do our measures mean what we think they mean?

Rosso et al. (2022) [6]

This meta-analysis shows the dangers of indicators that can be interpreted in diverse ways, and HARK-ing’s seductive pull. Are the indicators you use, and the meanings of increased/decreased values, always clear before starting an experiment? Have you ever “spun” an effect to make it fit expectations? When is hypothesis-generation good but HARK-ing dangerous?

Mason & Veasey (2010) [7]

Do the three construct validation methods make sense, and are there are additional ones?

Are the authors pessimistic or realistic, when they state “no one single welfare index is perfect”? Should a welfare indicator’s imperfections be factored into its use? For example, could it be useful to consider a priori the ‘false-negative’ and ‘false-positive’ results that an indicator is prone to, before use in welfare assessment?

Browning (2023) [8]

What do you think of conceptualising welfare as a “hidden target”? Do you like the way causes and effects of poor welfare are parsed out? What do you think of Browning’s recommended tests for robustness? And how often do you feel welfare research follows the logical pathways laid out here?

Sandem et al. (2002) [9] + three follow-up experiments using additional manipulations including a pharmacological treatment

Are you impressed at the range of situations used in this validatory research?

Should eye white have been scored blind? Should future work check that arousal is not a confound? What other species might this eye white metric be useful in?

To end

Muñoz-Tamayo et al. (2022) [10]

What do you think of this guide to open research? Are you comfortable sharing data (or organised enough to do so!)? Are you tempted by pre-registered reports? Should the journals used by welfare scientists change practices at all?