OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen esUntra table statisti al assumptionsandContent robustnessHerman J. AdèrSo iaal Wetens happelijke Se tie (VVS), 10 november 2010Herman J. Adèr Untra table statisti al assumptions 1OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es

OutlineIntrodu tionRandomisationPropensity s oresHow do I tell my lient?Herman J. Adèr Untra table statisti al assumptions 2

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Introdu tion: literatureExamples: solvableExamples: di� ult or untra tableExamples: Random sele tion & random assignmentContent robustnessINTRODUCTION

Herman J. Adèr Untra table statisti al assumptions 3OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Introdu tion: literatureExamples: solvableExamples: di� ult or untra tableExamples: Random sele tion & random assignmentContent robustnessIntrodu tion: literatureKNAW 2007 Colloquium pro eedings (Adèr & Mellenbergh,2008)Advising on resear h methods: A onsultant's ompanion(Adèr, Mellenbergh, & Hand, 2008)Bekend maar onbemind (deel I & II) (Mellenbergh, 1977)Strategies to Approximate Random Sampling and Assignment(Dattalo, 2010) Referen esHerman J. Adèr Untra table statisti al assumptions 4

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Introdu tion: literatureExamples: solvableExamples: di� ult or untra tableExamples: Random sele tion & random assignmentContent robustness(Partial) solution availableProblem Statisti alTool Impli ationsof violations Usualapproa h OtherpossibilitiesResidual normal-ity in analysis of( o)varian e Regressiondiagnosti s unpredi table ignore Multilevel analysisDistribution of ba-si variables in Fa toranalysis/SEM Basi statisti s(kurtosis) In PC analy-sis: limited;SEM: unpre-di table leave them out Mokken s aleTriangle unequality ofasso iation/distan ematrix Analysis ispossible Probablydevastatingin lusteranalysis unaware:ignore Che k it; (multiply)impute; sensitivityanalysisHerman J. Adèr Untra table statisti al assumptions 5OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Introdu tion: literatureExamples: solvableExamples: di� ult or untra tableExamples: Random sele tion & random assignmentContent robustness

Solution di� ult or unavailableProblem Statisti alTool Impli ationsof violations Usualapproa h OtherpossibilitiesModel misspe i� ation R2, Goodness-of-�t (CFI) devastating ata substantivelevel ignore try alternative models;bootstrapping, ross-validationMultivariate normality Wald's statisti unpredi table ignore bootstrapping, simula-tionMissingness (M(C)AR) Missing valueanalysis unpredi table ignore Sensitivity analysis(with and without(multiple) imputedvalues) and rossvali-dationHerman J. Adèr Untra table statisti al assumptions 6

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Introdu tion: literatureExamples: solvableExamples: di� ult or untra tableExamples: Random sele tion & random assignmentContent robustnessSolution di� ult or unavailableProblem Statisti alTool Impli ationsof violations Usualapproa h OtherpossibilitiesSu ess of random sele -tion Comparison withpopulation data generalisationjeopardized ignore Compare basi statis-ti s with other samplesfrom the same popula-tionSu ess of randomassignment (randomisa-tion) Comparing base-line statisti s devastating ignore propensity s ore or-re tion and rossvali-dationEquipoise Systemati review ethi al; over-powering; so- ietal ( ost) foiled by thefunding sys-tem To be sorted out

Herman J. Adèr Untra table statisti al assumptions 7OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Introdu tion: literatureExamples: solvableExamples: di� ult or untra tableExamples: Random sele tion & random assignmentContent robustnessDe�nition: Content robustnessA statisti al analysis pro edure is alled ontent robust if we are on�dent that violations of the assumptions will have no disruptinge�e t on the on lusions we will draw to answer the resear hquestion.

Herman J. Adèr Untra table statisti al assumptions 8

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessRANDOMIZATION Herman J. Adèr Untra table statisti al assumptions 9OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustness

RandomizationFisher (1935) is onsidered to be the �rst one to formulate thestri t design of the randomized experiment and its advantagesBradford Hill (1977) was the �rst to ondu t a randomized lini alexperiment in the UK (Po o k, 1999).Herman J. Adèr Untra table statisti al assumptions 10

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessSir Austin Bradford Hill (Bradford Hill, 1990):`At the outset, I think I pleaded that trials should be made usingalternate ases. I suspe t if (and its a very large IF) if that, in fa t,were done stri tly they would be random.'`I deliberately left out the words "randomization" and "randomsampling numbers" at that time, be ause I was trying to persuadethe do tors to ome into ontrolled trials in the very simplest formand I might have s ared them o�. I think the on epts of"randomization" and "random sampling numbers" are slightly oddto the layman, or, for that matter, to the lay do tor, when it omesto statisti s.' Herman J. Adèr Untra table statisti al assumptions 11OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessSir Austin Bradford Hill (Bradford Hill, 1990):`I thought it would be better to get do tors to walk �rst, before Itried to get them to run.'

Herman J. Adèr Untra table statisti al assumptions 12

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessRandomization: aimsHidden bias. Countera t hidden bias to be able to eliminatealternative explanations.Prin iple of equal opportunities. Give parti ipants equalopportunities when they agree to parti ipate in a lini al trial

Herman J. Adèr Untra table statisti al assumptions 13OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessPrin iple of equal opportunitiesAlthough the damage in�i ted by violation of this prin iple dependsheavily on the nature of the intervention, equal probability ofassignment to ea h of the treatment groups is preferable, bothfrom an ethi al as from a statisti al point of view.

Herman J. Adèr Untra table statisti al assumptions 14

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessAspe tsAssumption of equal distribution of hidden bias over treatmentgroupsPrin iple of Equipoise: genuine doubt about whether one ourse of a tion is better than another (Stolberg, Norman, &Trop, 2004).Equipoise in a medi al setting: if there is genuine un ertaintywithin the expert medi al ommunity � not ne essarily on thepart of the individual investigator � about the preferredtreatment (Freedman, 1987).Informed onsentCompare the Betting model (Hofstee, 1984)Herman J. Adèr Untra table statisti al assumptions 15OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessEqual opportunities: Problems1. Unbalan edness: Blo k randomisation2. Number and/or size of ontrol groups unequal to thenumber/size of treatment groups3. Funding foils Equipoise. Consequen e: Overpowering.Equal opportunities: Solutions1. Blo k randomisation with allowan e for small unbalan e2. Adjust number and size of ontrol groups by splitting or ombining3. Sequential Sampling and interim analysisHerman J. Adèr Untra table statisti al assumptions 16

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessHidden bias: Problems1. How an we he k if randomization su eeded?2. In luster randomized trials, bias is unavoidableHidden bias: Solutions (?)1. We hardly an, be ause:a. Che king on baseline distribution gives only information on overtbias.b. Call on ontent robustness? But: It is impossible to get animpression of the in�uen e of hidden bias on our on lusions . Do a pilot to get an impression of possible biasing elements in thesampling pro edure;d. In lude more ovariates next time (but even then the problemremains)2. Use propensity s ores Herman J. Adèr Untra table statisti al assumptions 17OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es How to randomize ?Randomization: aimsPrin iple of equal opportunitiesPrin iple of equal opportunities: aspe tsEqual opportunities: Problems and solutionsHidden bias: Problems and solutionsContent robustnessRandomization: ontent robustnessCon�den e in the on lusions based on the analysis results an beenhan ed by:Preparatory systemati review of the literatureCal ulation of propensity s ores, even in randomisedexperiments.Corre tion for relevant onfounders and propensity s oreProperly ondu ted interim analysis (Pros han, Lan, & Wittes,2006)Crossvalidation ombined with bootstrapping or otherresampling methodsHerman J. Adèr Untra table statisti al assumptions 18

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es De�nitionDiagram of the pro edure (Lu & van Duijn, 2008)PROPENSITY

Herman J. Adèr Untra table statisti al assumptions 19OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es De�nitionDiagram of the pro edure (Lu & van Duijn, 2008)Propensity s ore (de�nition)A propensity s ore of a patient/respondent in a lini al trial or anobservational study is an estimate λ of his or her han e to belongto the treatment groupThis estimate may be based on any of the observed variables apartfrom the out omes

Herman J. Adèr Untra table statisti al assumptions 20

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es De�nitionDiagram of the pro edure (Lu & van Duijn, 2008)Propensity score models: M = {M : T = X |T = Ti(i = 1, 2)}

Treatment: T = {T1, T2}; Covariates: X = {Xj|j = 1, . . .m}

Higher order terms: Xp; Interaction terms: XX

Prototype model

��

��

logistic

regression analysisT, X

X

X, Xp, XX

��

��

logistic

regression analysis

-

s

s

��

��Check balance�

���Check homogeneity

'

&

$

%Propensity score model

1

Propensity score model

2

�

Distribution intreatment groups

��

��t-test

?

Herman J. Adèr Untra table statisti al assumptions 21OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Example: Collaborative are for Major depressionCollaborative Care: Consultation issuesHOW DO I TELL MY CLIENT?

Herman J. Adèr Untra table statisti al assumptions 22

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Example: Collaborative are for Major depressionCollaborative Care: Consultation issuesAbstra tCollaborative are for Major depression in primary areObje tive. To evaluate the short term e�e tiveness of a ollaborative are (CC) model for Major DepressiveDisorder (MDD) in primary are in the Netherlands,in luding a preferen e armDesign. Randomised ontrolled trial in primary are withrandomisation at the level of 18 (sub)urban primary are entres: 9 entres re eived training andsupervision to provide CC to the patients, and 9 entres did not re eive this training and provided areas usual (CAU).Parti ipants. In the CC group, 93 patiens were identi�ed bys reening and 57 by the GP (the preferen e group).Herman J. Adèr Untra table statisti al assumptions 23OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen es Example: Collaborative are for Major depressionCollaborative Care: Consultation issuesConsultation issuesCluster randomisationPropensity s oresPreferen e arm

Herman J. Adèr Untra table statisti al assumptions 24

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen esSummary

Herman J. Adèr Untra table statisti al assumptions 25OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen esSummary IA ase an be made for ready-made statisti al models that more loselyrepresent the resear h design and allow for unequivo al interpretation ofthe analysis results.`Bekend maar onbemind'. In empiri al resear h it is ommon pra ti e notto he k statisti al analysis assumptions.But if they are he ked, and violations are obvious, there is usually noremedy.In many ases, violations are onsidered harmless and ontent robustnessis assumed.There are several data analyti tools to in rease ontent robustness.Systemati review, ross validation and sensitivity analysis are examples.Herman J. Adèr Untra table statisti al assumptions 26

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen esSummary IIEquipoise seems an essential ingredient of any resear h plan. Not onlyethi al and statisti al aims are served, but the notion an be extended toresear h relevan e as well.Randomization is aimed at:(a) Countera ting hidden bias;(b) Guaranteeing equal opportunities for parti ipants.Sin e it is impossible to get an impression of hidden bias, it is impossibleto get absolute ertainty about the su ess of the randomisation.There are several resear h designs that, although randomized,unavoidably produ e biased data. Blo k randomization and Clusterrandomization are examples.Estimating propensity s ores provide a possibility to orre t for overt bias.Herman J. Adèr Untra table statisti al assumptions 27OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen esBa k to Introdu tionAdèr, H. J., & Mellenbergh, G. J. (2008). Advising on Resear h Methods:Pro eedings of the 2007 KNAW Colloquium. Huizen, The Netherlands: VanKessel. Available from www.jvank.nl/ARMPro sAdèr, H. J., Mellenbergh, G. J., & Hand, D. J. (2008). Advising on resear h methods:A onsultant's ompanion. Huizen: Johannes van Kessel. Available fromwww.jvank.nl/ARMHomeBradford Hill, A. (1977). Prin iples of Medi al Statisti s (10th ed.). London: TheLan et.Bradford Hill, A. (1990). Memories of the British streptomy in trial in tuber ulosis:the �rst randomized lini al trial. Controlled Clini al Trials, 11, 77�79.Dattalo, P. (2010). Strategies to approximate random sampling and assignment.Oxford: Oxford University Press.Fisher, R. A. (1935). The design of experiments. Edinburgh: Oliver & Boyd.Freedman, B. (1987). Equipoise and the ethi s of lini al resear h. The New EnglandJournal of Medi ine, 317(3), 141�145.Hofstee, W. K. B. (1984). Methodologi al de ision rules as resear h poli ies: Abetting re onstru tion of empiri al resear h. A ta Psy hologi a, 56, 93�109.Herman J. Adèr Untra table statisti al assumptions 28

OutlineIntrodu tionRandomizationPropensity s oresHow do I tell my lient?SummaryReferen esLu, B., & van Duijn, M. (2008). Redu ing sele tion bias using propensity s oremat hing. In H. J. Adèr & G. J. Mellenbergh (Eds.), Advising on resear hmethods: Pro eedings of the 2007 KNAW olloquium. Huizen, TheNetherlands: van Kessel. Available from www.jvank.nl/Pro sMellenbergh, G. J. (1977). Bekend, maar onbemind (Vol. II; Syllabus). Amsterdam:University of Amsterdam, Subfa ulty Psy hology.Po o k, S. J. (1999). Clini al Trials: A Statisti ian's Perspe tive. In H. J. Adèr &G. J. Mellenbergh (Eds.), Resear h Methodology in the So ial, Behaviouoraland Life S ien es. London Thousand Oaks New Delhi: Sage Publi ations.Pros han, M. A., Lan, K. K. G., & Wittes, J. T. (2006). Statisti al monitoring of lini al trials: A uni�ed approa h. New York: Springer.Stolberg, H. O., Norman, G., & Trop, I. (2004). Fundamentals of lini al resear h forradiologists. Ameri an Journal of Roentgenology, 183(5), 1539�1544.Herman J. Adèr Untra table statisti al assumptions 29