Original Article

Quality of Reporting in Clinical Trials of Preharvest FoodSafety Interventions and Associations with Treatment Effect

Jan M. Sargeant,1,2 Jacqueline Saint-Onge,1 James Valcour,1,2 Adam Thompson,1

Robyn Elgie,1 Kate Snedeker,1,2 and Pasha Marcynuk1,2

Abstract

Randomized controlled trials (RCTs) are the gold standard for evaluating treatment efficacy. Therefore, it isimportant that RCTs are conducted with methodological rigor to prevent biased results and report results in amanner that allows the reader to evaluate internal and external validity. Most human health journals nowrequire manuscripts to meet the Consolidated Standards of Reporting Trials (CONSORT) criteria for reporting ofRCTs. Our objective was to evaluate preharvest food safety trials using a modification of the CONSORT criteriato assess methodological quality and completeness of reporting, and to investigate associations between re-porting and treatment effects. One hundred randomly selected trials were evaluated using a modifiedCONSORT statement. The majority of the selected trials (84%) used a deliberate disease challenge, with theremainder representing natural pathogen exposure. There were widespread deficiencies in the reporting ofmany trial features. Randomization, double blinding, and the number of subjects lost to follow-up were reportedin only 46%, 0%, and 43% of trials, respectively. The inclusion criteria for study subjects were only described in16% of trials, and the number of animals housed together was only stated in 52% of the trials. Although 91 trialshad more than one outcome, no trials specified the primary outcome of interest. There were significant bivariableassociations between the proportion of positive treatment effects and failure to report the number of subjects lostto follow-up, the number of animals housed together in a group, the level of treatment allocation, and possiblestudy limitations. The results suggest that there are substantive deficiencies in reporting of preharvest foodsafety trials, and that these deficiencies may be associated with biased treatment effects. The creation andadoption of standards for reporting in preharvest food safety trials will help to ensure the inclusion of importanttrial details in all publications.

Introduction

Randomized controlled trials (RCTs) are consideredthe gold standard for evaluating the efficacy of inter-

ventions. Because they provide the highest level of evidencefor efficacy, it is important that RCTs are free from bias andreported in sufficient detail to allow the reader to assess bothinternal and external validity. In the human health literature,standardized guidelines for the reporting of clinical trials havebeen developed. The Consolidated Standards of ReportingTrials (CONSORT) statement was originally published in1996 (Begg et al., 1996), with a revised version simultaneouslypublished by four leading medical journals in 2001 (Moheret al., 2001a, 2001b, 2001c, 2001d). The CONSORT statementconsists of a 22-item checklist and a flow diagram forreporting an RCT (Moher et al., 2001a). The items were se-

lected for inclusion because there was empirical evidence inthe literature indicating the potential for biased estimates oftreatment effects when these items were not reported or be-cause the information was deemed essential to evaluate thereliability or relevance of the findings. An explanation andelaboration document was published concurrently to defineand discuss the importance of each item, in addition to pro-viding examples of how this information could be provided ina publication (Altman et al., 2001). Subsequent evaluationssuggest that the CONSORT statement has improved thequality of reporting of RCTs in the human health literature(Plint et al., 2006; Kane et al., 2007).

Recent systematic reviews have highlighted concerns withthe reporting of methodological features in published trials ofpreharvest food safety interventions. A systematic review ofvaccination to reduce the prevalence of Salmonella in swine

1Centre for Public Health and Zoonoses and 2Department of Population Medicine, Ontario Veterinary College, University of Guelph,Guelph, Ontario, Canada.

FOODBORNE PATHOGENS AND DISEASEVolume 6, Number 8, 2009ª Mary Ann Liebert, Inc.DOI: 10.1089=fpd.2009.0321

989

identified 28 separate trials (5 field trials and 23 trials that usedan artificial disease challenge) (Denagamage et al., 2007). Theauthors rated 23 of the 28 trials as low quality because the useof randomization was not explicitly stated or a few combi-nations of the vaccination protocol, challenge protocol, out-come assessment, or statistical analyses were not fullydescribed. A systematic review focusing on feed managementand prevalence of Salmonella in swine identified failure toreport randomization to treatment group and failure todescribe methods to control for confounding variables insome of the publications assessed (O’Connor et al., 2008). Asystematic review of on-farm interventions to reduce fecalshedding of Escherichia coli O157 in postweaned domesticruminants also reported substantive issues with methodo-logical quality (Sargeant et al., 2007). In the E. coli O157 review,which included observational studies, challenge trials, andnatural disease exposure RCTs, 16 of 44 publications did notmeet key methodological criteria, with failure to report ran-domization to treatment group as the most common omis-sion. While these reviews raise issues related to reporting ofkey methodological features, they represent a relatively smallcross section of the published RCTs related to preharvest foodsafety interventions. The objective of the current study was toevaluate a randomly selected sample of preharvest foodsafety intervention trials using a modified checklist based onthe CONSORT statement. A secondary objective was to in-vestigate whether reporting of key methodological qualityand completeness of reporting items was associated with trialoutcomes.

Materials and Methods

Selection of trials

A systematic search for recently published clinical trials inpreharvest food safety was conducted in May 2008 usingthree electronic databases: PubMed, Medline, and Centre forAgricultural Bioscience International Direct. The search wasrestricted to papers published after 1997. Search terms relatedto animal species, clinical trials, and food safety were enteredinto each of the selected electronic databases (Table 1). Bool-ean operators were used to link search terms: ‘‘AND’’ wasused to separate terms between animal species, clinical trials,and food safety categories, and ‘‘OR’’ was used to separateterms within categories. The Boolean operator ‘‘NOT’’ wasused to exclude studies pertaining to fish, trout, salmon, pork,carcass, ground beef, yogurt, and cheese.

Citations identified by the search were imported into areference management program (RefWorks; www.refworks.com), and duplicate citations were identified and removed.The title and abstract of each citation were evaluated for rel-evance. A publication was deemed to be relevant if it reportedthe results of original research related to food animal species,involved evaluation of an intervention in live animals (pre-ventive or therapeutic), and was related to preharvest foodsafety. Publications reporting the results of descriptive stud-ies, observational studies, and in vitro experiments were ex-cluded. Evaluation of an intervention included clinical trialsconducted in commercial herds or in research settings andchallenge trials (deliberate exposure to an infectious diseaseoutcome of interest) in the species in which the interventionwas intended for use. Relevance screening was conducted bya single reviewer ( J.S.O.).

Each publication was assigned a unique identifier. Using arandom number generator, 150 trials were randomly selectedfor a secondary level of screening. Publications were excludedduring this phase if the publication was not written in Englishor if it was not possible to obtain a full copy of the article orreport. After the second phase of screening, 100 publicationswere randomly selected from the relevant English-languagepublications for inclusion in the study. Full papers were ob-tained for the 100 selected publications. For publicationsdescribing the results of more than one trial, a single trial wasrandomly selected by coin-toss or number draw.

Evaluation of selected trials

Each of the 100 trials was evaluated for reporting ofmethodological quality and completeness of reporting using astructured checklist based on the CONSORT statement(Moher et al., 2001a) (Table 2). Additional questions were in-cluded to address issues specific to trials in livestock popu-lations. These included information on the disease challenge ifexposure to the outcome was not natural. The challenge trial,while common in veterinary medicine, is not a trial designthat is used in human populations. Although not directly re-lated to methodological quality or completeness of reporting,two additional questions were included to evaluate thenumber of trials that included one or more outcomes relatedto production or performance and the number of trials thatincluded evaluation of a foodborne pathogen other than theprimary pathogen of interest. Initially, all of the authorsevaluated 10 papers, and the interrater reliability of each

Table 1. List of Search Terms Entered into Three Electronic Databases to Identify Clinical

Trials in Preharvest Food Safety Published Between 1998 and 2008

Animal termsClinical trial

termsPreharvest food

safety terms

BovineSwine

PoultryOvine

Cattle, cow*, dairy, beef, bovine, heifer*, steer*, bull*, calves, calfPig*, piglet*, swine, gilt*, boar*, sow*, finisher, weaner, barrow*,

grower*, hog, hogs, porcineChick*, poult*, turkey*, broiler*, layer*, hen, hens, rooster*, pullet*Lamb*, ovine, ram, rams, ewe*

TreatmentInterventionMitigationVacc*Prophylactic

FoodborneFood borneFood-borneO157Salmonell*Campy*

*Denotes when a term was shortened to account for multiple versions of a root word (e.g., vacc* identifies vaccine, vaccines, andvaccination).

990 SARGEANT ET AL.

(continued)

Table 2. Number of Trial Publications Addressing Methodological Quality Criteria and Completeness

of Reporting Items from 100 Preharvest Food Safety Trials Published Between 1998 and 2008

Number oftrials

Number of trialsincluded in denominator

TITLE AND ABSTRACTTerm random (randomized) used in title or abstract 7 100

METHODSObjectivesStudy objectives stated 91 100If yes, more than one objective included 49 91If more than one objective, primary objective identified 0 49

Study subjectsInclusion=exclusion criteria described 16 100Geographic location of study described 30 100Housing conditions described 77 100Number of animals housed together in group stated 52 100

InterventionsConcurrent control (comparison) group used 98 100Intervention described in sufficient detail for replication 88 100Level of treatment allocation (individual, pen, and herd) described 90 100

Outcome(s)More than one outcome used 91 100If yes, primary outcome identified 0 91Measurement of all outcomes described 85 100Methods used to enhance quality of measurement (e.g., multiple

observations or observers, explicit training)30 100

Infectious disease outcome 100 100If yes, disease challenge used 84 98If yes, challenge protocol described in sufficient detail for replication 72 84

Sample sizeSample size stated 90 100If yes, sample size justified 1 90If no significant results, statistical power discussed 0 58Trial stopped early 0 100If yes, description of a priori stopping rules NA NA

RandomizationRandom allocation to treatment reported 45 98 (Trials with

comparison group)If yes, method to generate random allocation sequence described 2 45Random allocation sequence concealed until interventions assigned 0 45Description of who generated the allocation sequence relative to who

enrolled and assigned participants0 45

Additional restrictions (e.g., blocking=stratification) described 15 97

Blinding(1) Client owned animals 6 98 (Trials with

comparison group)Blinding possible 6 6Client blinded to treatment group 0 6Person administering treatment blinded (single blinding) 0 6Outcome evaluator blinded 0 6Both person administering and person evaluating outcome blinded (double blinding) 0 6Mechanism of blinding described NA NA(2) Research animals 91 98 (Trials with

comparison group)Blinding possible 91 91Person administering treatment blinded (single) 0 91Outcome evaluator blinded 5 91Both person administering and person evaluating outcome blinded (double) 0 91Mechanism of blinding described 2 5 (Trials with single

or double blinding)

Statistical methodsFormal statistical analyses performed 90 98 (Trials with

comparison group)

QUALITY OF REPORTING OF PREHARVEST FOOD SAFETY TRIALS 991

Table 2. (Continued)

Number oftrials

Number of trialsincluded in denominator

If yes, statistical methods described for all outcomes 81 90Multiple observations used for any outcome 65 90If yes, repeated measures controlled in analyses 30 65

RESULTSParticipant flow(1) Client-owned animals 6 100Reported number of animals (herds) evaluated for participation 2 6Reported number of animals (herds) not meeting inclusion criteria 1 6Reported number of eligible animals (herds) who declined to participate 1 6Reported number of animals (herds) assigned to treatment 5 6Reported number of animals (herds) receiving treatment as allocated by

study group2 6

Reported number of animals (herds) completing treatment as allocated bystudy group

3 6

Reported number of animals (herds) completing study (i.e., lost to follow-up) 3 6Reported number of animals (herds) included in main analysis by study group 4 6(2) Research animals 64 yes,

30 unsure100

Reported number of animals (herds) not completing study (i.e., lost to follow-up) 40 94Reported number of animals (herds) included in main analysis by study group 62 94Client and research animal trialsDeviations from study protocol, or lack thereof, described 5 100If yes, reasons provided 5 5

RecruitmentTrial start=end dates provided (at least month or season) 7 100Length of follow-up period described 97 100Description of where participants were recruited from 46 100

Baseline dataAnimal signalment described (sex, age or weight, and breed) 37 100Significant tests of baseline differences reported (any variable) 10 100

Numbers analyzedReported number of subjects in each group included in analysis 72 83 (Trials with statistical

analyses performed andlevel of treatment

allocation described)Analysis stated as ‘‘intention-to-treat’’ 1 6 (Trials of client-owned

animals, statisticalanalysis performed, and

level of treatmentallocation described)

Data analyzed at same level as treatment 46 yes,21 unsure

83

Outcome and estimationGraphical display of results ONLY 12 100Study results as a summary of the outcome in each group 98 100Treatment group contrast(s) reported (e.g., relative risk and odds ratio) 5 90 (Trials with a

comparison groupand formal statisticalanalysis performed)

Confidence intervals or measures of variability reported 44 90Exact p-value given or nominal p-value for significance described 79 90

Ancillary analysesReported analyses not described in objectives 23 100If yes, described as exploratory 2 23Reported associations with one or more measures of production=performance 28 100Measured foodborne pathogens other than primary pathogen of interest 5 100

Adverse eventsInformation provided on adverse effects or lack thereof 26 100

DISCUSSIONPossible study limitations discussed 25 100

NA, not applicable.

992 SARGEANT ET AL.

question on the checklist was assessed using the kappa sta-tistic. The agreement for all questions was evaluated anddiscussed among reviewers, and questions were re-worded orremoved if agreement was below 0.5. Thereafter, two indi-viduals independently evaluated each publication using thechecklist. Differences in responses among reviewers wereresolved by consensus.

A second template was created to extract data from eachtrial on the number of outcomes and the association betweeneach outcome and the treatment of interest (i.e., the treatmenteffect). The outcomes were categorized into mortality, mor-bidity, pathogen frequency or concentration, performance orproduction, immunological, physiological, or other. For eachcategory, the number of outcomes that were (a) positivelyassociated with the treatment, (b) negatively associated withthe treatment, (c) not statistically associated with the treat-ment, (d) described numerically with no statistical compari-son reported, or (e) described in the methods section with noresults subsequently described was recorded. When there wasmore than one level of the treatment variable reported, sta-tistical significance was based on the comparison between thehighest dose and any other treatment dosage. If there wasmore than one treatment reported in the publication, and theprimary treatment of interest was not clearly identified, thetreatment of interest was defined as the first treatmentdescribed in the methods section, with the determination ofstatistically significant results based on comparison of thistreatment to the control group. The outcome template wascompleted independently by two reviewers with disagree-ments resolved by consensus. The reviewers completing theoutcome template for a specific trial were not the same re-viewers who completed the modified CONSORT checklist forthat trial.

Data analysis

The number of publications reporting each of the meth-odological quality and completeness of reporting items wassummarized for all 100 trials. The outcome data were sum-marized using descriptive statistics. To evaluate associationsbetween trial features and treatment effects, nine key criteriarelated to methodological quality and seven key items re-lated to completeness of reporting were selected for furtheranalyses. The methodological quality criteria were (a) use ofa concurrent comparison group, (b) random allocationto treatment group, (c) a description of the method used togenerate the random allocation sequence, (d) double blind-ing (defined as blinding of the person administering thetreatment and the person assessing the outcome), (e) a de-scription of the mechanism of blinding, (f) a description ofthe number of study subjects lost to follow-up, (g) a de-scription of the statistical methods used to evaluate all out-comes, (h) a description of the number of animals housedtogether, and (i) a description of the level at which treatmentwas allocated (i.e., individual, pen, and herd). The key cri-teria for completeness of reporting were (a) a description ofthe inclusion=exclusion criteria for study subjects, (b) a de-scription of the intervention protocol in sufficient detail toallow replication, (c) a description of animal signalment,including age or weight, sex, and breed, (d) a comparison ofbaseline differences among treatment groups for at least onevariable, (e) a description of the measurement of all out-

comes, (f) a stated sample size, and (g) a discussion of po-tential study limitations.

Simple (bivariable) associations between each of the keymethodological quality criteria and completeness of reportingitems and the proportion of positive treatment effects withintrials were evaluated using generalized linear models. Theoutcome was the number of statistically significant positivetreatment effects divided by the number of outcomes thatwere statistically evaluated (i.e., those with a positive, nega-tive, or no statistically significant association, but excludingoutcomes for which no statistical analysis was reported). Thecriterion for the use of a concurrent comparison group wasexcluded from this analysis because associations with treat-ment could not be evaluated in trials where no comparisongroup was used. Associations were considered to be statisti-cally significant at a p-value �0.05.

Least squares linear regression models were used to in-vestigate associations with the number of key features ade-quately addressed in each trial. Two outcomes wereinvestigated separately: the number of key methodologicalcriteria adequately addressed (0 to 9) and the number ofcompleteness of reporting items adequately addressed (0 to 7).The independent variables evaluated were the total numberof outcomes in each trial, the proportion of the total number ofoutcomes that were statistically evaluated, the proportion ofthe statistically evaluated outcomes that were positively as-sociated with treatment, and a binary variable representingthat the outcome resulted from a deliberate disease challenge(1) or was due to natural disease exposure (0). Each variablewas initially evaluated independently as a bivariable asso-ciation. All variables were then entered into a model and re-moved via backward elimination based on p-values. Allvariables significant at p� 0.05 were retained in the finalmodel.

Results

The selected trials were published between January 1998and May 2008, with the number of trials per year ranging from3 in 2008 to 30 in 2007. The trials were from 30 journals, withthe number of trials per journal ranging from 1 to 14(mean¼ 3.3). The species investigated included poultry (57trials in meat birds and 4 trials in eggs), bovine (19 trials),swine (16 trials), and ovine (4 trials). A variety of inventiontypes were assessed: 18 trials evaluated therapeutic treat-ments, 38 evaluated preventive treatments, 15 evaluated die-tary treatments as therapy, 14 evaluated dietary preventivetreatments, and 15 evaluated management manipulations. Allof the trials investigated interventions for one or more food-borne pathogens, although six trials reported only antibodyresponse with no direct pathogen outcome reported. Themajority of the trials (84) used a deliberate disease challenge ofall study subjects to the disease outcome of interest.

Methodological quality and completeness of reporting

The number of publications meeting each of the method-ological quality criteria and completeness of reporting items isshown in Table 2. There were substantive issues in reportingof many items. The majority of the trials included an objec-tive statement, with 53% of the trials including more than oneobjective. However, none of the trials with more than one

QUALITY OF REPORTING OF PREHARVEST FOOD SAFETY TRIALS 993

objective identified the primary objective. The description ofstudy participants was poorly reported, with only 16 trialsproviding inclusion=exclusion criteria and only 52 trials in-dicating the number of animals housed together in eachtreatment group. Two trials did not have an explicit com-parison group; these studies were included because the au-thors stated in the abstract that they were clinical trials orbecause the authors made statements related to treatmentefficacy in the abstract. The level at which treatment was al-located was not described in 10 trials, and 12 trials did notdescribe the intervention protocol in sufficient detail to allowreplication. The majority of studies included more than oneoutcome, and yet none of the studies identified the primaryoutcome, which is necessary for calculating sample size. Itwas not possible to determine the exact sample size used in 10of the trials, and only 1 trial provided a sample size justifi-cation. Post hoc calculations of statistical power were notprovided for any of the 58 trials that did not find any statis-tically significant results.

Random allocation to treatment group was reported in lessthan half of the trials that included a formal comparisongroup, with almost none of the trials providing a descriptionof the method of random allocation (Table 2). Similarly, de-tails on whether blinding was used, as well as the method ofblinding, were poorly reported. Statistical analysis was notperformed in 10 trials, and less than half of the trials with anoutcome measured at multiple time points controlled for therepeated measures in the analysis. Control of clustering(nonindependence among study subjects) was not evaluatedin this study.

For many trials, it was not possible to follow the number ofanimals included in each stage of the trial, and an analysis ofbaseline differences among treatment groups was only re-ported in 10 trials. Almost all of the trials reported the studyresults as a summary of the outcome in each group (98%);however, treatment group contrasts (relative risk or oddsratios), statistical variability among groups, and p-valueswere not consistently reported. Adverse effects of the treat-ment, or lack thereof, were described in less than one-third ofthe trials, and only 25 trials included a discussion of possiblestudy limitations. One or more outcomes associated witheither production or performance were included in 28 trials,although only 5 trials included an outcome related to afoodborne pathogen other than the pathogen of primary in-terest.

Outcome reporting

The number of outcomes per trial ranged from 1 to 51(mean¼ 8.5; frequency distribution in Fig. 1). The averagenumber of outcomes by category was 1.4 for mortality out-comes (in nine trials reporting outcomes in this category), 2.2for morbidity outcomes (in 17 trials), 5.1 for outcomes de-scribing bacterial, viral, or parasitic prevalence, incidence, orconcentration (in 93 trials), 4.4 for immunological outcomes(in 26 trials), 6.2 for physiological outcomes (in 17 trials), 2.2for performance=production-related outcomes (in 28 trials),and 5.8 for other outcomes (in 8 trials), including histologicaloutcomes, outcomes related to genetics or gene expression,nutrition outcomes, and environmental measurements. Of the854 outcomes included in the 100 selected trials, 212 (24.8%)only provided a description of the outcome overall or by

treatment group, with no formal statistical comparison de-scribed. A further 46 outcomes (5.4%) were described in thematerials and methods section of the trial, but no results forthat outcome were provided in the manuscript. Of the 596outcomes for which a formal statistical test was reported, 275(46.1%) of the treatment comparisons had a statistically posi-tive association with the outcome, 73 (12.3%) were negativelyassociated with the outcome, and 248 (41.6%) were not sig-nificantly associated with the outcome.

Associations between reporting and treatment effects

The number of the nine key methodological quality criteriaaddressed within each trial ranged from 3 to 8, with a mean of5.2 (Fig. 2). Simple associations between key methodologicalquality criteria and the proportion of evaluated outcomes thatwere positively associated with the treatment are shown inTable 3. There were 86 trials included in these analyses, whichrepresented trials with a comparison group for which at leastone outcome was evaluated for association with the treatmentusing a formal statistical comparison. None of the studiesreported the use of double blinding; therefore, this variablecould not be assessed. There were statistically significantnegative associations with reporting the number of studysubjects lost to follow-up, describing the number of animalshoused together, and reporting the level of treatment alloca-tion. Thus, for each of these criteria, trials that reported thecriterion had a significantly lower proportion of positive as-

FIG. 1. Frequency distribution of the number of outcomesincluded in each of 100 preharvest food safety trials pub-lished between 1998 and 2008.

FIG. 2. Frequency distribution of the number of methodo-logical quality criteria adequately addressed within each of100 preharvest food safety trials published between 1998 and2008 (maximum¼nine).

994 SARGEANT ET AL.

sociations with treatment. There were no statistically signifi-cant bivariable associations between the total number of keymethodological quality criteria addressed and the number ofoutcomes, the proportion of outcomes statistically evaluated,the proportion of positive treatment effects, or the use of achallenge exposure to the disease of interest (Table 4).Therefore, multivariable analysis of this outcome was notperformed.

The number of the seven key completeness of reportingitems adequately addressed in each trial ranged from 0 to 6,with a mean of 3.5 (Fig. 3). The majority of the key itemsrelated to completeness of reporting were not significantlyassociated with the proportion of positive treatment associa-tions, although trials that described participant signalmenthad a significantly higher proportion of positive treatmenteffects and trials that discussed study limitations had a sig-nificantly lower proportion of positive treatment effects

(Table 3). The total number of completeness of reporting itemsthat were adequately addressed was positively associatedwith the number of outcomes and negatively associated withthe use of an artificial disease challenge in the multivariablemodel (Table 5).

Discussion

The results of this study suggest that there are substantivedeficiencies in the reporting of important features of prehar-vest food safety trials in the published literature, includingboth criteria related to methodological quality and itemsnecessary to interpreting the external validity (representa-tiveness) of a trial. Under-reporting of key features is consis-tent with previous observations from published systematicreviews (Denagamage et al., 2007; Sargeant et al., 2007;O’Connor et al., 2008). The checklist used in this evaluationwas based on items included in the CONSORT statement,which is a standard for reporting of clinical trials for humanhealth interventions that has been endorsed by over a hun-dred journals (www.consort-statement.org). Additional itemswere added to the checklist to address more basic issues thanthose included in the CONSORT statement, such as the use ofa concurrent comparison group and a clear statement ofsample size. Additional questions related to trial components

Table 3. Simple Associations Between the Proportion

of Evaluated Outcomes with a Positive Association

to Treatment and Selected Trial Components in

100 Preharvest Food Safety Intervention Trials

Published Between 1998 and 2008 Estimated

Using a Generalized Linear Model

Independent variableb-Coefficient

(standard error) p-Value

Methodological quality criteriaUse of a comparison group NA NARandom allocation to

treatment reported0.03 (0.17) 0.86

Method to generate randomallocation sequence describeda

0.56 (0.65) 0.39

Double blinding reported NA NAMechanism of blinding

describedb�0.13 (0.56) 0.98

Reported number of animalsnot completing study(i.e., lost to follow-up)

�0.36 (0.17) 0.03

Statistical methods describedfor all outcomes

�0.45 (0.32) 0.16

Number of animals housedtogether in group stated

�0.46 (0.17) 0.007

Level of treatment allocation(individual, pen, and herd)described

�1.12 (0.33) <0.001

Completeness of reporting itemsInclusion=exclusion criteria

described�0.18 (0.19) 0.35

Intervention described insufficient detail for replication

0.33 (0.26) 0.21

Animal signalment described(sex, age or weight, and breed)

0.34 (0.17) 0.04

Significant tests of baselinedifferences reported(any variable)

0.32 (0.25) 0.20

Sample size stated �0.64 (0.38) 0.09Measurement of all outcomes

described�0.04 (0.25) 0.88

Possible study limitationsdiscussed

�0.59 (0.19) 0.002

aFor trials where random allocation to treatment group was stated.bFor trials where at least single blinding was reported.NA, not applicable.

Table 4. Linear Regression of Associations with the

Number of Key Methodological Quality Criteria

Adequately Addressed from a Total of Nine Criteria

in 100 Preharvest Food Safety Intervention Trials

Published Between 1998 and 2008

Variableb-Coefficient

(standard error) p-Value

Bivariable associationsNumber of outcomes 0.05 (0.01) 0.27Proportion of outcomes

statistically evaluated0.38 (0.30) 0.21

Proportion of statisticallyevaluated outcomespositively associatedwith treatment

�0.23 (0.33) 0.49

Deliberate exposure todisease challenge

�0.34 (0.32) 0.28

FIG. 3. Frequency distribution of the number of items re-lated to completeness of reporting addressed within each of100 preharvest food safety trials published between 1998 and2008 (maximum¼ seven).

QUALITY OF REPORTING OF PREHARVEST FOOD SAFETY TRIALS 995

specific to livestock species were also added, includinghousing and grouping of animals, a clear statement of thelevel of treatment allocation, and the use of challenge trials.We included these items to reflect concerns identified in pre-vious systematic reviews and to capture some of the uniquefeatures of clinical trials in farm settings.

The use of multiple outcomes was common, and 28 trialsincluded one or more outcomes specifically related to pro-duction or performance. Determining whether an interven-tion is associated with production parameters is necessaryinformation when considering the economic benefits or costsof implementing an intervention, and is thereby important tothe decision-making process. Only five trials included themeasurement of foodborne pathogens in addition to the pri-mary pathogen of interest. Although there is a paucity ofliterature to describe coinfection with multiple foodbornepathogens in naturally infected livestock, there is some evi-dence that animals may shed more than one foodbornepathogen at a given time, and that interventions might dif-ferentially impact fecal shedding. Callaway et al. (2006) re-ported concurrent fecal shedding of Salmonella, E. coli O157,and Listeria within feedlots, although different bacterial spe-cies were rarely isolated from within the same pen. A study ofdairy cattle from a single farm reported a prevalence of 30%to 35% for E. coli O157:H7 and 30% to 33% for Salmonella,although concurrent shedding of both pathogens by the sameanimals was not evaluated (Fitzgerald et al., 2003). A fieldtrial investigating the effect of Ractopamine HCl supple-mentation in feedlot cattle reported a decrease in fecal shed-ding of E. coli O157 and an increase in Salmonella shedding infeedlot cattle (Edrington et al., 2006). Further work is neededto understand coshedding of foodborne pathogens, but thepotential effects of interventions for one foodborne pathogenon the occurrence of other foodborne pathogens may requireconsideration.

The use of a deliberate disease challenge for study unitsallocated to treatment groups (challenge trial) was a commondesign, and the challenge trials reported significantly lesscompleteness of reporting items (Table 5). This study designprovides efficiencies in sample size, because all of the trial

animals develop the disease before treatment allocation (fortherapeutic interventions) or are exposed to the disease agent(for preventive interventions). However, challenge trials oftenuse animals housed either individually or in small groups,and may occur under more controlled conditions than istypical in field settings. Therefore, it is important to discussexternal validity when drawing conclusions from challengetrials. To this end, there is a need to improve reporting of itemsessential for the reader to evaluate external validity.

Key features related to methodological quality and com-pleteness of reporting were identified to investigate potentialbiases related to nonreporting of these items. The methodo-logical quality items were selected based on a modification ofthe Jadad scale ( Jadad et al., 1996). The Jadad scale is com-monly used as a quality appraisal tool in the evaluation ofhuman health trials (Kjaergard et al., 2001) and includes fivecriteria: randomization, describing the method of randomi-zation, blinding, describing the method of blinding, and re-porting the number lost to follow-up. For the current study, weadded the use of a concurrent comparison group, a descriptionof statistical methods for all outcomes, and a description of thegrouping of animals and level of treatment allocation.

A number of studies published in the human health litera-ture have investigated the potential for biased treatment effectsin RCTs not reporting specific trial components (Chalmers et al.,1983; Schulz et al., 1995; Kunz and Oxman, 1998; Moher et al.,1998; Juni et al., 2001; Kjaergard et al., 2001). Although com-bining multiple trial components to create overall quality scalesfor investigating treatment effects may depend on the scaleused ( Juni et al., 1999), studies in the human health literaturehave reported that trials with a low score on the Jadad scalewere associated with larger treatment effects (Moher et al.,1998; Juni et al., 1999; Kjaergard et al., 2001).

In the current study, the majority of trials had more thanone outcome, and none of the trials stated which was con-sidered the primary outcome. Therefore, we were unable toevaluate associations between reporting of trial features andthe primary treatment effect. Instead, we evaluated the asso-ciation between trial features and the proportion of positivetreatment effects within trials. For this reason, our resultsare not directly comparable to the cited human health trials.We did not find associations between the proportion of pos-itive treatment effects and reporting of random allocation orthe method used to generate the random allocation sequence.In addition, we did not evaluate associations with conceal-ment of the random allocation sequence, as none of the trialsreported this feature. Although the results from the humanhealth studies varied, exaggerated treatment effects have beenobserved in trials that do not report random allocation(Chalmers et al., 1983; Kunz and Oxman, 1998), the method ofgenerating the random allocation sequence ( Juni et al., 2001;Kjaergard et al., 2001), or inadequate allocation concealment(Schulz et al., 1995; Kunz and Oxman, 1998; Moher et al., 1998;Juni et al., 2001; Kjaergard et al., 2001). None of the trials in thecurrent study reported double blinding (defined as blindingof the person administering the treatment and the outcomeassessor), and only five trials reported blinding of the personassessing the outcome. Therefore, we were unable to evaluateassociations between blinding and treatment effects. How-ever, in the human health literature, studies have reportedthat trials not reporting blinding have larger reported treat-ment effects (Schulz et al., 1995; Juni et al., 2001; Kjaergard et al.,

Table 5. Linear Regression of Associations

with Number of Completeness of Reporting Items

Adequately Addressed (from Zero to Seven)

in 100 Preharvest Food Safety Intervention Trials

Published Between 1998 and 2008

Variableb-Coefficient

(standard error) p-Value

Bivariable associationsNumber of outcomes 0.03 (0.01) 0.02Proportion of outcomes

statistically evaluated0.64 (0.30) 0.04

Proportion of evaluatedoutcomes positivelyassociated with treatment

0.28 (0.34) 0.41

Deliberate exposure todisease challenge

�0.88 (0.32) 0.006

Multivariable associationsNumber of outcomes 0.03 (0.01) 0.02Deliberate exposure to

disease challenge�0.90 (0.31) 0.004

996 SARGEANT ET AL.

2001). We also found significant negative associations bet-ween failure to report numbers lost to follow-up, numbershoused together, level of treatment effect, and possible studylimitations and the proportion of positive treatment effects.We evaluated a number of different trial components and didnot control for the multiplicity of tests. Therefore, it is possiblethat one or more of the associations found to be significantwere the results of type I errors. Nonetheless, these resultssuggest that trials not reporting certain trial features may havebiased outcomes.

Of concern was our finding that many outcomes were notstatistically evaluated, while others were identified in thematerials and methods section but not reported in the re-sults. In addition, eight trials with a control group did notprovide formal statistical analyses for any of the outcomes,and a further nine did not describe the statistical methods forall of the outcomes that were evaluated. This makes it dif-ficult for the reader of a trial to interpret the results andconclusions. We did not find significant associations be-tween the proportion of outcomes statistically evaluated andthe number of key methodological quality criteria addressedor the number of completeness of reporting items addressed.However, Chan et al. (2004) investigated the completeness ofreporting of treatment effects and the potential for bias.Their definition of complete reporting of treatment effectsvaried somewhat based on the outcome data type, but in-cluded (a) a description of the number of participants ana-lyzed per group, (b) a description of the treatment effect size,and (c) a measure of precision or precise p-value. Overall,statistically significant treatment effects were 2.4 times morelikely to be completely reported compared to nonsignificantresults.

Of the 90 trials that reported a formal statistical analysis, 65used multiple observations for at least one outcome. How-ever, only 30 of these reported whether repeated measureswere controlled in the analyses. Failure to control for repeatedmeasures of an outcome generally results in inappropriatelysmall standard errors of the treatment effect and spuriouslysignificant results (Petrie and Watson, 2006). Another com-mon statistical issue in trials in livestock populations is clus-tering of observations due to the housing of animals ingroups, a common practice in clinical trials of livestock as itreflects actual housing conditions in commercial settings. Biascan occur when the treatment is allocated at the group levelwith the outcome measured at the individual level andwithout controlling for the incomplete statistical indepen-dence among animals within a group (McDermott andSchukken, 1994). Due to lack of clarity related to the numberof animals housed together (48 of the 100 trials) and the levelof treatment allocation (10 of 100 trials), we did not evaluatestatistical issues related to clustering of animals in groups.These findings are consistent with Elbers and Schukken(1995), who were unable to determine whether clustering waspresent in 38 of 147 veterinary field trials because the housingconditions of the animals were not described. However,McDermott and Schukken (1994) reported that 46% of 67veterinary studies with potential clustering did not describeany control for clustering, suggesting that lack of consider-ation of clustering may be an area of potential concern. Anextension of the CONSORT statement for clustered trials hasbeen developed and provides recommendations for reportingthis type of trial (Campbell et al., 2004).

Our study had several limitations. First, we based ourevaluation on reporting of trial features. It is possible thatthese components were adequately addressed in the designand conduct of a trial, but just not explicitly reported in thesubsequent publication. To evaluate this possibility, investi-gators in the human health field contacted trial authors toclarify whether specific trial features that were not reportedwere, in reality, a part of the trial conduct (Hill et al., 2002;Devereaux et al., 2004). These studies reported that non-reporting of trial features, such as the method of random al-location sequence generation, allocation concealment, andblinding, frequently were adequately performed in RCTs thatdid not explicitly report them. Failure to report trial detailscould be related to lack of understanding by the authors as towhich features should be reported or the need to address re-viewer or editor concerns on other issues within journal pagelimits. Nonetheless, the reader of a clinical trial has only thepublished information with which to judge the internal andexternal validity of the trial. Standard reporting guidelines,such as the CONSORT statement, can provide valuableguidelines as to essential trial features to report.

A second potential limitation of the current study was thesample size. We selected 100 trials based on feasibility and toprovide a descriptive summary of current reporting. Thus,our sample size was not based on the number of trials neededto test hypotheses related to potential biases in the treatmenteffect. Some of the criteria that were evaluated for associationshad little variability, because they were reported in most ofthe trials or very few of the trials. Therefore, our sample sizemay have resulted in insufficient power to detect significantassociations for many criteria. Conversely, we did not controlfor the multiplicity of associations that were evaluated.Therefore, it is possible that some of the statistically significantassociations that we observed were the result of type I errors.

The reviewers in this study were not blinded with regard toauthor names or affiliations, which could have led to inad-vertent bias. The use of two reviewers independently asses-sing the trials should have reduced bias in the data collection.However, blinding reviewers to author information should beconsidered in future studies of this type.

Finally, it is possible that the trials selected for inclusion inthis study do not represent the preharvest food safety litera-ture in general. We consider this to be unlikely, in that thetrials were selected randomly, albeit from selected electronicdatabases. However, the electronic databases represent com-prehensive literature listings. We also sampled only from therecent literature to ensure that historical trial reporting wouldnot be a consideration.

Overall, this study identified substantive deficiencies in thereporting of methodological quality criteria and items relatedto completeness of reporting for determination of externalrelevance in preharvest food safety trials. Although not all ofthese features were associated with treatment effects, therewas some evidence that lack of reporting may be associatedwith a higher proportion of positive treatment effects. The useof a standard set of guidelines could be a valuable resource,not only for trial authors but also for reviewers, editors, andthose reading trial reports. The CONSORT statement provi-des a valuable guide, but may not address all of the issuesrelevant to trials in preharvest food safety. There is a need toestablish guidelines suitable for use in preharvest food safetytrials to improve reporting of these trials.

QUALITY OF REPORTING OF PREHARVEST FOOD SAFETY TRIALS 997

Acknowledgments

The authors thank Jim Brett for library assistance, AnnetteWilkins for helpful comments on a draft of this manuscript,and Aodhan Wall for technical assistance in the preparationof this manuscript. Partial funding for this project was ob-tained from the Laboratory for Foodborne Zoonoses, PublicHealth Agency of Canada, and from the Canadian Institutesof Health Research (CIHR) Institute of Population and PublicHealth=Public Health Agency of Canada Applied PublicHealth Chair.

Disclosure Statement

No competing financial interests exist.

References

Altman DG, Schulz KF, Moher D, Egger M, Davidoff F,Elbourne D, Gøtzsche PC, Lang T, and CONSORT Group(Consolidated Standards of Reporting Trials). The revisedCONSORT statement for reporting randomized trials: ex-planation and elaboration. Ann Intern Med 2001;134:663–694.

Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, PitkinR, Rennie D, Schulz KF, Simel D, and Stroup DF. Improvingthe quality of reporting of randomized controlled trials. TheCONSORT statement. JAMA 1996;276:637–639.

Callaway TR, Edrington TS, Brabban AD, Keen JE, AndersonRC, Rossman ML, Engler MJ, Genovese KJ, Gwartney BL,Reagan JO, Poole TL, Harvey RB, Kutter EM, and Nisbet DJ.Fecal prevalence of Escherichia coli O157, Salmonella, Listeria,and Bacteriophage infecting E. coli O157:H7 in feedlot cattle inthe Southern Plains region of the United States. FoodbornePathog Dis 2006;3:234–244.

Campbell MK, Elbourne DR, Altman DG, and CONSORTgroup. CONSORT statement: extension to cluster randomisedtrials. BMJ 2004;328:702–708.

Chalmers TC, Celano P, Sacks HS, and Smith H Jr. Bias intreatment assignment in controlled clinical trials. N Eng J Med1983;309:1358–1361.

Chan AW, Hrobjartsson A, Haahr MT, Gøtzsche PC, andAltman DG. Empirical evidence for selective reporting ofoutcomes in randomized trials: comparison of protocols topublished articles. JAMA 2004;291:2457–2465.

Denagamage TN, O’Connor AM, Sargeant JM, Rajic A, andMcKean JD. Efficacy of vaccination to reduce Salmonellaprevalence in live and slaughtered swine: a systematic reviewof literature from 1979 to 2007. Foodborne Pathog Dis2007;4:539–549.

Devereaux PJ, Choi PT, El-Dika S, Bhandari M, Montori VM,Schunemann HJ, Garg AX, Busse JW, Heels-Ansdell D, GhaliWA, Manns BJ, and Guyatt GH. An observational studyfound that authors of randomized controlled trials frequentlyuse concealment of randomization and blinding, despite thefailure to report these methods. J Clin Epidemiol 2004;57:1232–1236.

Edrington TS, Callaway TR, Ives SE, Engler MJ, Welsh TH,Hallford DM, Genovese KJ, Anderson RC, and Nisbet DJ.Effect of ractopamine HCl supplementation on fecal sheddingof Escherichia coli O157:H7 and Salmonella in feedlot cattle.Curr Microbiol 2006;53:340–345.

Elbers AR and Schukken YH. Critical features of veterinary fieldtrials. Vet Rec 1995;136:187–192.

Fitzgerald AC, Edrington TS, Looper ML, Callaway TR, Geno-vese KJ, Bischoff KM, McReynolds JL, Thomas JD, AndersonRC, and Nisbet DJ. Antimicrobial susceptibility and factorsaffecting the shedding of E. coli O157:H7 and Salmonella indairy cattle. Lett Appl Microbiol 2003;37:392–398.

Hill CL, LaValley MP, and Felson DT. Discrepancy betweenpublished report and actual conduct of randomized clinicaltrials. J Clin Epidemiol 2002;55:783–786.

Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ,Gavaghan DJ, and McQuay HJ. Assessing the quality of re-ports of randomized clinical trials: is blinding necessary?Control Clin Trials 1996;17:1–12.

Juni P, Altman DG, and Egger M. Systematic reviews inhealthcare: assessing the quality of controlled clinical trials.BMJ 2001;323:42–46.

Juni P, Witschi A, Bloch R, and Egger M. The hazards of scoringthe quality of clinical trials for meta-analysis. JAMA 1999;282:1054–1060.

Kane RL, Wang J, and Garrard J. Reporting in randomizedclinical trials improved after adoption of the CONSORTstatement. J Clin Epidemiol 2007;60:241–249.

Kjaergard LL, Villumsen J, and Gluud C. Reported methodolo-gic quality and discrepancies between large and small ran-domized trials in meta-analyses. Ann Intern Med 2001;135:982–989.

Kunz R and Oxman AD. The unpredictability paradox: reviewof empirical comparisons of randomised and non-randomisedclinical trials. BMJ 1998;317:1185–1190.

McDermott JJ and Schukken YH. A review of methods usedto adjust for cluster effects in explanatory epidemiologicalstudies of animal populations. Prev Vet Med 1994;18:155–173.

Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M,Tugwell P, and Klassen TP. Does quality of reports of ran-domised trials affect estimates of intervention efficacy re-ported in meta-analyses? Lancet 1998;352:609–613.

Moher D, Schulz KF, and Altman DG. The CONSORT state-ment: revised recommendations for improving the quality ofreports of parallel-group randomised trials. Lancet 2001a;357:1191–1194.

Moher D, Schulz KF, Altman DG, and CONSORT Group(Consolidated Standards of Reporting Trials). The CONSORTstatement: revised recommendations for improving the qua-lity of reports of parallel-group randomized trials. Ann InternMed 2001b;134:657–662.

Moher D, Schulz KF, Altman D, and Consort Group (Con-solidated Standards of Reporting Trials). The CONSORTstatement: revised recommendations for improving the qua-lity of reports of parallel-group randomized trials. JAMA2001c;285:1987–1991.

Moher D, Schulz KF, Altman DG, and CONSORT Group. TheCONSORT statement: revised recommendations for improv-ing the quality of reports of parallel group randomized trials.2001d. Available at http:==www.biomedcentral.com=1471-2288=1=2, accessed November 27, 2008. (Online.)

O’Connor AM, Denagamage T, Sargeant JM, Rajic A, andMcKean J. Feeding management practices and feed charac-teristics associated with Salmonella prevalence in live andslaughtered market-weight finisher swine: a systematic re-view and summation of evidence from 1950 to 2005. Prev VetMed 2008;87:213–228.

Petrie A and Watson P. Further regression analyses: regressionmethods for clustered data. In: Statistics for Veterinary andAnimal Science, 2nd edition. Oxford, UK: Blackwell Publishing,2006, p. 154.

998 SARGEANT ET AL.

Plint AC, Moher D, Morrison A, Schulz K, Altman DG, Hill C,and Gaboury I. Does the CONSORT checklist improve thequality of reports of randomised controlled trials? A system-atic review. Med J Aust 2006;185:263–267.

Sargeant JM, Amezcua MR, Rajic A, and Waddell L. Pre-harvestinterventions to reduce the shedding of E. coli O157 in thefaeces of weaned domestic ruminants: a systematic review.Zoonoses Public Health 2007;54:260–277.

Schulz KF, Chalmers I, Hayes RJ, and Altman DG. Empiricalevidence of bias. Dimensions of methodological quality asso-ciated with estimates of treatment effects in controlled trials.JAMA 1995;273:408–412.

Address correspondence to:Jan M. Sargeant, D.V.M., M.Sc., Ph.D.

Centre for Public Health and ZoonosesOntario Veterinary College

103 MacNabb HouseUniversity of Guelph

Guelph N1G 2W1OntarioCanada

E-mail: sargeanj@uoguelph.ca

QUALITY OF REPORTING OF PREHARVEST FOOD SAFETY TRIALS 999