DEPARTMENT OF PEDIATRICSResearch Seminar

STUDY DESIGN10/22/09

David H. Rubin, MDDavid H. Rubin, MD

Chairman, Department of Pediatrics, St. Chairman, Department of Pediatrics, St. Barnabas HospitalBarnabas Hospital

Professor of Clinical PediatricsProfessor of Clinical Pediatrics

Albert Einstein College of MedicineAlbert Einstein College of Medicine

STUDY STUDY DESIGNDESIGN

FEATUREFEATURE EXAMPLEEXAMPLE

Descriptive Descriptive ReportsReports

Recognize Recognize new/atypical new/atypical characteristic of characteristic of diseasedisease

Case report – first Case report – first case(s) of pediatric case(s) of pediatric lyme disease lyme disease

CohortCohort 1 group followed 1 group followed over timeover time

Infants followed for Infants followed for effects of smoke effects of smoke exposure for 2 exposure for 2 yearsyears

Cross-Cross-SectionalSectional

A group examined A group examined at 1 point in timeat 1 point in time

Psychometric Psychometric testing in homeless testing in homeless vs. nonhomeless vs. nonhomeless childrenchildren

Case-ControlCase-Control Two groups, based Two groups, based on outcomeon outcome

Aspirin and Reyes Aspirin and Reyes SyndromeSyndrome

Randomized Randomized TrialTrial

Two groups, Two groups, randomly created, randomly created, blinded blinded interventionintervention

Effect of Effect of educational educational intervention on intervention on asthma morbidityasthma morbidity

RESEARCH DESIGN(Jekel, 2007)

STUDY DESIGNSTUDY DESIGN ADVANTAGEADVANTAGE DISADVANTAGEDISADVANTAGE

CohortCohort Useful for measure of Useful for measure of risk and many risk and many disease outcomes; disease outcomes; restrosp or prospectrestrosp or prospect

Time consuming and Time consuming and costly; can only study costly; can only study risk factors identified at risk factors identified at start of studystart of study

Cross-SectionalCross-Sectional Fairly easy to Fairly easy to perform; useful for perform; useful for hypothesis hypothesis generationgeneration

No evidence of temporal No evidence of temporal relationship between risk relationship between risk and diseaseand disease

Case-ControlCase-Control Easy to perform, can Easy to perform, can study several risk study several risk factors/rare diseases factors/rare diseases without waiting for without waiting for diseasedisease

Can obtain only relative Can obtain only relative measure of risk; recall measure of risk; recall bias and selection of bias and selection of case and control biascase and control bias

Randomized Randomized TrialTrial

““Gold standard” for Gold standard” for treatment treatment interventionsinterventions

Time consuming/costly; Time consuming/costly; problems with dropoutsproblems with dropouts

ADDITIONAL STUDY DESIGNS

•Medical record reviewMedical record review•Survey studySurvey study

CASE CONTROL STUDIES

POPULATION AT RISK

YES

NO

YES

NO

YES

NO

RESEARCHTIME

CASE CONTROL STUDY

CONDITION OR PROBLEM

EXPOSURE?

(Fletcher, 1996)

CASE CONTROL STUDIES/RECENT

LITERATURE• Pubmed search of “case control studies”Pubmed search of “case control studies”

• 356,299 studies identified356,299 studies identified• Case control studies and emergency Case control studies and emergency

medicine journalsmedicine journals• Ann Emerg Med: 602 studies (1980-2007)Ann Emerg Med: 602 studies (1980-2007)• Jour Emerg Med: 197 studies (1984-2007)Jour Emerg Med: 197 studies (1984-2007)• Acad Emerg Med: 357 studies (1994-Acad Emerg Med: 357 studies (1994-

2007)2007)• Ped Emerg Care: 288 studies (1985-2007)Ped Emerg Care: 288 studies (1985-2007)

NESTED CASE CONTROL STUDY (Gordis, 2000)

• Case control study “nested” in cohort Case control study “nested” in cohort studystudy

• Population identified and followed Population identified and followed over timeover time

• Disease develops in some members Disease develops in some members of the populationof the population• Case control study of Case control study of

• Cases (disease develops) and Cases (disease develops) and • Controls (disease does not Controls (disease does not develop)develop)

ADVANTAGES OF CASE CONTROL STUDIES

(Fletcher et al, 1996, Newman et al, 2001)

• Relatively easy to performRelatively easy to perform• Can study several risk factors and rare Can study several risk factors and rare

diseases without waiting for disease to occurdiseases without waiting for disease to occur• Beneficial for diseases with long latencyBeneficial for diseases with long latency

• High yield of information with relatively few High yield of information with relatively few subjectssubjects

• Ability to examine a large number of predictor Ability to examine a large number of predictor variables makes case control studies useful for variables makes case control studies useful for generating hypothesesgenerating hypotheses

DISADVANTAGES OF CASE CONTROL STUDIES

(Fletcher et al, 1996, Newman et al, 2001)

• Information available may be limitedInformation available may be limited• No direct method to estimate incidence No direct method to estimate incidence

or prevalence of a diseaseor prevalence of a disease• Only 1 main outcome can be studiedOnly 1 main outcome can be studied

• In cohort and cross sectional studies, In cohort and cross sectional studies, several outcomes can be examinedseveral outcomes can be examined

• Biggest weakness is bias Biggest weakness is bias

BIAS IN CASE CONTROL STUDIES(Fletcher 1996)

• Investigators create the comparison groups Investigators create the comparison groups – there is no waiting to see who becomes a – there is no waiting to see who becomes a case and who becomes a control case and who becomes a control

• Cases and controls are comparable if: Cases and controls are comparable if: • Controls would have been defined as Controls would have been defined as

cases cases if they developed the condition if they developed the condition under studyunder study

• Cases and controls need to be members Cases and controls need to be members of the same “base population”of the same “base population”

COHORT STUDIES

COHORT STUDYCOHORT STUDY

T0 T1

•Population followed forward over time

•Baseline: acute pharyngitis

•Outcome: Prevention of rheumatic fever or glomerulonephritis

•Admission Criteria?: Evidence of ß-hemolytic streptococcus vs pharyngeal inflammation

NO

YES

NO

TIME

COHORT STUDYCONDITION OR PROBLEM

EXPOSURE? YES

POPULATION

SAMPLE

YES

NO

(Fletcher, 1996)

CROSS SECTIONAL STUDIES

CROSS SECTIONAL CROSS SECTIONAL STUDYSTUDY

•Collect data on 2 groups at 1 point in time

•Compare group differences

•Cholesterol levels in athletes vs. non athletes at a midwest university

T0 T1

CROSS SECTIONAL CROSS SECTIONAL DESIGNDESIGN

• Face to face interview Face to face interview • Mailed questionnaireMailed questionnaire• Emailed questionnaireEmailed questionnaire• Telephone interviewTelephone interview• Mailed interview with telephone Mailed interview with telephone

F/UF/U• Interview and observationInterview and observation

RANDOMIZED CLINICAL TRIALS

RANDOMIZED CONTROL RANDOMIZED CONTROL TRIALTRIAL

CONTROL

EXPERIMENTALRANDOMIZATION

TIME 0; BASELINE T1; FOLLOWUP

ENROLL SUBJECTS

SURVEY STUDIES

SURVEY STUDIESSURVEY STUDIES

•Collect information about Collect information about people to describe, people to describe, compare, or explain compare, or explain •KnowledgeKnowledge•AttitudeAttitude•BehaviorBehavior

SURVEY STUDIESSURVEY STUDIES

• Features of good survey studiesFeatures of good survey studies• Specific measurable objectivesSpecific measurable objectives• Solid research designSolid research design• Good choice of population or Good choice of population or

samplesample• Reliable and valid instrumentsReliable and valid instruments• Comprehensive analysisComprehensive analysis• Accurate reporting of results Accurate reporting of results

MEASURABLE MEASURABLE OBJECTIVESOBJECTIVES

• Define aim of the studyDefine aim of the study• Define hypothesisDefine hypothesis• Define outcomesDefine outcomes• Define independent, Define independent,

dependant, and confounding dependant, and confounding variablesvariables

SAMPLING OF SAMPLING OF POPULATIONPOPULATION

• Many optionsMany options• Sample - subset of the population chosen for Sample - subset of the population chosen for

study (characteristics similar to larger group)study (characteristics similar to larger group)• Representative sample – use an unbiased Representative sample – use an unbiased

method to choose survey participantsmethod to choose survey participants

• All members of the pediatric clinic at All members of the pediatric clinic at SBH who are between 2 and 3 years of SBH who are between 2 and 3 years of ageage

• Children seen at the Pediatric Endocrine Children seen at the Pediatric Endocrine Clinic for any illness related to diabetes Clinic for any illness related to diabetes mellitusmellitus

DESCRIPTIVE REPORTS

DESCRIPTIVE DESCRIPTIVE REPORTSREPORTS

• Description of a new aspect or Description of a new aspect or new diseasenew disease

• No comparison group neededNo comparison group needed• Description is usually a basic Description is usually a basic

statistic summary or profile of statistic summary or profile of the group of casesthe group of cases• Mean, SD, range, confidence Mean, SD, range, confidence

intervals, correlation between intervals, correlation between variablesvariables

MEDICAL RECORD REVIEW

MEDICAL RECORD MEDICAL RECORD REVIEWREVIEW

• Uses pre-recorded patient Uses pre-recorded patient focused data as the primary focused data as the primary source of information in a source of information in a research studyresearch study• Physician, nurses notesPhysician, nurses notes• Ambulance call reportsAmbulance call reports• Diagnostic testsDiagnostic tests• Clinic, administrative, government Clinic, administrative, government

recordsrecords• Computerized databasesComputerized databases

WHY SELECT THIS WHY SELECT THIS DESIGN?DESIGN?

• Addresses issues that cannot be Addresses issues that cannot be addressed with prospective studiesaddressed with prospective studies• Effect of harmful exposures (no randomization Effect of harmful exposures (no randomization

possible)possible)• Effect of potentially beneficial exposuresEffect of potentially beneficial exposures• Occurrence of rare eventsOccurrence of rare events• Studies of patterns of disease or behaviorStudies of patterns of disease or behavior• Quality assurance studiesQuality assurance studies• Studies where cases may be shared (trauma Studies where cases may be shared (trauma

database)database)• Pilot studies for prospective studiesPilot studies for prospective studies

DATA QUALITYDATA QUALITY

• ““Free form” quality of medical Free form” quality of medical records may increase missing records may increase missing and/or erroneous dataand/or erroneous data

• Handwriting may be illegible or Handwriting may be illegible or uninterruptibleuninterruptible

• May miss examining potential May miss examining potential casescases

• Computer vs paper recordsComputer vs paper records• Data abstraction techniques Data abstraction techniques

require standardizationrequire standardization

SAMPLE SIZESAMPLE SIZE

• Usually determined based on Usually determined based on the summary measure and the summary measure and the size/width of the the size/width of the confidence interval desired confidence interval desired • An interval with a greater CI (eg 99% An interval with a greater CI (eg 99%

CI v 95% CI) is wider and more likely CI v 95% CI) is wider and more likely includes the true population valueincludes the true population value

• The width of the CI depends on The width of the CI depends on sample size sample size

SAMPLINGSAMPLING

• Select all cases within a given time frameSelect all cases within a given time frame• For nonconsecutive sampling it is best to For nonconsecutive sampling it is best to

choose probability samplingchoose probability sampling• Provides equal opportunity for each eligible Provides equal opportunity for each eligible

case to be selectedcase to be selected• Use random number generatorUse random number generator

• Triage levelTriage level

• Incidental sampling – choosing most Incidental sampling – choosing most easily accessible caseseasily accessible cases

• Systematic sampling – choosing every xSystematic sampling – choosing every xthth casecase

RELIABILITYRELIABILITY

• Very important Very important • Any differences in data extraction by 2 Any differences in data extraction by 2

different people?different people?• KappaKappa

• Value ranges from -1 (perfect Value ranges from -1 (perfect disagreement) to 1 (perfect agreement)disagreement) to 1 (perfect agreement)

• K = [observed agreement (%) – expected K = [observed agreement (%) – expected agreement (%) / [100% - expected agreement (%) / [100% - expected agreement (%)]agreement (%)]

• Try to achieve kappa of 0.6 or better (60% Try to achieve kappa of 0.6 or better (60% agreement)agreement)

MINIMUM REQUIREMENTS FOR MINIMUM REQUIREMENTS FOR MEDICAL RECORD REVIEWSMEDICAL RECORD REVIEWS

(Lowenstein, 2005)(Lowenstein, 2005)

1)1) Explicit protocols for case Explicit protocols for case selection/exclusionselection/exclusion

2)2) Abstractor trainingAbstractor training3)3) Precise definitions of key variablesPrecise definitions of key variables4)4) Use of standardized abstraction and Use of standardized abstraction and

coding formscoding forms5)5) Monitoring of abstractor performanceMonitoring of abstractor performance6)6) Blinding of abstractors to study Blinding of abstractors to study

hypothesis and patient groupshypothesis and patient groups7)7) Testing of interrater reliabilityTesting of interrater reliability

QUALITY OF MEDICAL RECORD QUALITY OF MEDICAL RECORD REVIEWSREVIEWS(Badcock, 2005)(Badcock, 2005)

• Observational study of medical record Observational study of medical record reviews published in several reviews published in several emergency medicine journalsemergency medicine journals

• 107 articles analyzed107 articles analyzed• Clear aim reported in 93%Clear aim reported in 93%• Standard abstraction forms: 51%Standard abstraction forms: 51%• Interrater reliability: 25%Interrater reliability: 25%• Ethics approval: 68%Ethics approval: 68%• Sample size/power: 10%Sample size/power: 10%

METHODOLOGY FOR METHODOLOGY FOR RETROSPECTIVE REVIEWS IN RETROSPECTIVE REVIEWS IN

CHILD PSYCHIATRYCHILD PSYCHIATRY

• Conceive questionConceive question• Literature reviewLiterature review• Proposal methodsProposal methods• Create data abstraction Create data abstraction

instrument and manualinstrument and manual• Sample sizeSample size• Obtain IRB approvalObtain IRB approval• Pilot studyPilot study

SAMPLE SIZESAMPLE SIZE(Gearing 2006)(Gearing 2006)

• Estimate 10 charts per variable Estimate 10 charts per variable (Sackett, 1991)(Sackett, 1991)

• Others estimate 5-7 Others estimate 5-7 charts/variablecharts/variable

• Convenience sampling – select Convenience sampling – select cases over specific time periodcases over specific time period

• Quota sampling – predetermined Quota sampling – predetermined number samplednumber sampled

• Systematic sampling – every “nth” Systematic sampling – every “nth” case chosencase chosen

PRACTICAL ISSUESPRACTICAL ISSUES

• Check all possible CPT codes for Check all possible CPT codes for diagnosis or procedure codediagnosis or procedure code• Febrile seizure may have been coded as Febrile seizure may have been coded as

seizureseizure• Gastroenteritis may have been coded Gastroenteritis may have been coded

as viral syndromeas viral syndrome• Pilot your Data Abstraction FormPilot your Data Abstraction Form• Create detailed “Codebook” for Create detailed “Codebook” for

your studyyour study• Especially critical if > 1 researcher on Especially critical if > 1 researcher on

studystudy

REFERENCESREFERENCES

• Fink A. How to design survey studies. Fink A. How to design survey studies. Sage Publications, Thousand Oaks, CA. Sage Publications, Thousand Oaks, CA. 2003.2003.

• Kline TJB. Psychological testing –a Kline TJB. Psychological testing –a practical approach to design and practical approach to design and evaluation. 2005. Thousand Oaks. Sage.evaluation. 2005. Thousand Oaks. Sage.

• Friedman JN. Development of a clinical Friedman JN. Development of a clinical dehydration scale for use in children dehydration scale for use in children between 1 and 36 months of age. J between 1 and 36 months of age. J Pediatr 2004:145:201-207.Pediatr 2004:145:201-207.

REFERENCES• Cumming RG, Le Couteur D. Benzodiazepines Cumming RG, Le Couteur D. Benzodiazepines

and risk of hip fractures in older people; a and risk of hip fractures in older people; a review of the evidence. CNS Drugs. review of the evidence. CNS Drugs. 2003;17(11):825-837.2003;17(11):825-837.

• Ding R, McCarthy M, Li G. Patients who leave Ding R, McCarthy M, Li G. Patients who leave without being seen: their characteristics and without being seen: their characteristics and history of emergency department use. Ann history of emergency department use. Ann Emerg Med 2006;48:686-693.Emerg Med 2006;48:686-693.

• Feinstein AR. Clinical Epidemiology. Feinstein AR. Clinical Epidemiology. Philadelphia. WB Saunders. 1985Philadelphia. WB Saunders. 1985

• Fletcher RH, Fletcher SW, Wagner EH. Fletcher RH, Fletcher SW, Wagner EH. Clinical Clinical Epidemiology The EssentialsEpidemiology The Essentials. Philadelphia: . Philadelphia: Lippincott Williams and Wilkins, 1996.Lippincott Williams and Wilkins, 1996.

• Gordis L. Epidemiology 2Gordis L. Epidemiology 2ndnd edition. Philadelphia: edition. Philadelphia: WB Saunders, 2000.WB Saunders, 2000.

REFERENCES• Worster A, Haines T. Advanced statistics: Worster A, Haines T. Advanced statistics:

understanding medical record review (MRR) understanding medical record review (MRR) studies. Acad Emerg Med 2004;11:187-192.studies. Acad Emerg Med 2004;11:187-192.

• Lowenstein SR. Medical record reviews in Lowenstein SR. Medical record reviews in emergency medicine: the blessing and the cure. emergency medicine: the blessing and the cure. Annals Emerg Med April 2005;45(4):452-455.Annals Emerg Med April 2005;45(4):452-455.

• Babcock D et al. The quality of medical record Babcock D et al. The quality of medical record review studies in the international emergency review studies in the international emergency medicine literature. 2005;45(4):444-447.medicine literature. 2005;45(4):444-447.

• Worster A. et al. Reassessing the methods of Worster A. et al. Reassessing the methods of medical record review studies 2005;45:448-451.medical record review studies 2005;45:448-451.

• Gearing et al. Methodology for Retrospective Gearing et al. Methodology for Retrospective chart review in child adolescent psychiatry. J Can chart review in child adolescent psychiatry. J Can Acad Child Adoles Psychiatry 15:3:2006Acad Child Adoles Psychiatry 15:3:2006

REFERENCES• Hellems MA, Kramer MS, Hayden G. Case control Hellems MA, Kramer MS, Hayden G. Case control

confusion. Ambulatory Pediatrics 2006;6:96-99. confusion. Ambulatory Pediatrics 2006;6:96-99. Altzema C, Ann Emerg Med 2004;44:169-174Altzema C, Ann Emerg Med 2004;44:169-174

• Horwitz RI, Feinstein AR. Methodologic standards Horwitz RI, Feinstein AR. Methodologic standards and contradictory results in case control studies. and contradictory results in case control studies. Amer Jour of Med 1979;66:556-564.Amer Jour of Med 1979;66:556-564.

• Hurwitz ES, Barrett MJ, Bregman D et al. Public Hurwitz ES, Barrett MJ, Bregman D et al. Public Health Service study of Reye’s syndrome and Health Service study of Reye’s syndrome and medications. Report of the main study. JAMA medications. Report of the main study. JAMA 1987;257:1905-11.1987;257:1905-11.

• Jekel JF, Katz DL, Elmore JG, Wild DMG. Jekel JF, Katz DL, Elmore JG, Wild DMG. Epidemiology, Biostatistics and Preventive Epidemiology, Biostatistics and Preventive Medicine 3Medicine 3rdrd edition. Philadelphia: Saunders, edition. Philadelphia: Saunders, 2007.2007.

REFERENCES• Katz AR. Selection of cases and controls. Katz AR. Selection of cases and controls.

Additional information is needed. Int J Additional information is needed. Int J Cardiology 2007;doi:10.1016/j.ijcard206.12.047Cardiology 2007;doi:10.1016/j.ijcard206.12.047

• Levy JA, Bachur RG. Intravenous dextrose Levy JA, Bachur RG. Intravenous dextrose during outpatient rehydration in pediatric during outpatient rehydration in pediatric gastroenteritis. Acad Emerg Med 2007;14:324-gastroenteritis. Acad Emerg Med 2007;14:324-331. 331.

• Newman TB, Browner W, Cummings SR, Hulley Newman TB, Browner W, Cummings SR, Hulley SB in Hulley SB et al. Designing Clinical SB in Hulley SB et al. Designing Clinical Research. 2Research. 2ndnd edition. Philadelphia: Lippincott, edition. Philadelphia: Lippincott, 2001.2001.

• Neuman MI, Kelley M, Harper MB et al. Factors Neuman MI, Kelley M, Harper MB et al. Factors associated with antimicrobial resistance and associated with antimicrobial resistance and mortality in pneumococcal bacteremia. Jour mortality in pneumococcal bacteremia. Jour Emerg Med 2006;32(4):349-357.Emerg Med 2006;32(4):349-357.

REFERENCES• O’Brien KL, Selanikio JD, Hecdivert C et al. O’Brien KL, Selanikio JD, Hecdivert C et al.

Epidemic of pediatric deaths from acute renal Epidemic of pediatric deaths from acute renal failure caused by diethylene glycol poisoning. failure caused by diethylene glycol poisoning. Acute Renal Failure Investigating Team. JAMA Acute Renal Failure Investigating Team. JAMA 1998;279:1175-1180. 1998;279:1175-1180.

• Panagiotakos DB, Rallidis LS, Pitsavos C et al. Panagiotakos DB, Rallidis LS, Pitsavos C et al. Cigarette smoking and myocardial infarction in Cigarette smoking and myocardial infarction in young men and women: A case-control studyyoung men and women: A case-control studyInternational Journal of Cardiology, Volume 116, International Journal of Cardiology, Volume 116, Issue 3, 4 April 2007: 371-375.Issue 3, 4 April 2007: 371-375.

• Pierfitte C, Macouillard G, Thicoipe M et al. Pierfitte C, Macouillard G, Thicoipe M et al. Benzodiazepines and hip fractures in elderly Benzodiazepines and hip fractures in elderly people. BMJ 2001;322:704-708.people. BMJ 2001;322:704-708.

• Schultz CH, Koenig KL, Lewis RJ. Decisionmaking Schultz CH, Koenig KL, Lewis RJ. Decisionmaking in hospital earthquake evacuation: does distance in hospital earthquake evacuation: does distance from the epicenter matter? Ann Emerg Med from the epicenter matter? Ann Emerg Med 2007;50:320-326.2007;50:320-326.