RELIABILITY AND VALIDITY LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON

STRUCTURE OF THE CHAPTERDefining validityValidity in quantitative researchValidity in qualitative researchTypes of validityTriangulationValidity in mixed methods researchEnsuring validityReliabilityReliability in quantitative researchReliability in qualitative research

STRUCTURE OF THE CHAPTERValidity and reliability in interviewsValidity and reliability in experimentsValidity and reliability in questionnairesValidity and reliability in observationsValidity and reliability in testsValidity and reliability in life histories

BASES OF VALIDITY IN

QUANTITATIVE RESEARCHBASES OF VALIDITY IN

QUALITATIVE RESEARCHControllabilityNaturalIsolation, control, manipulation of

VariablesThick descriptionReplicabilityUniquenessPredictabilityEmergence, unpredictabilityGeneralizabilityUniquenessContext-freedomContext-boundednessFragmentation and atomizationHolismRandomization of samplesPurposive sample/no samplingNeutralityValue-ladenness of observationsObjectivityConfirmabilityObservabilityObservable and non-observable

meanings/ intentionsInferenceDescription, inference, explanationEtic researchEmic researchObservationsMeanings

BASES OF RELIABILITY IN

QUANTITATIVE RESEARCHBASES OF RELIABILITY IN

QUALITATIVE RESEARCHReliabilityDependabilityDemonstrabilityTrustworthinessStability and replicabilityStability and replicabilityParallel formsParallel formsContext-freedomContext-specificityObjectivityAuthenticity and confirmabilityCoverage of domain Comprehensiveness of situationVerification of data and analysisHonesty and candourAnswering research questionsDepth of responseMeaningfulness to the researchMeaningfulness to respondentsParsimonyRichnessInternal consistencyCredibilityGeneralizabilityTransferabilityInter-rater reliability & triangulationInter-rater reliability and triangulationAccuracy and precisionAccuracy and comprehensivenessNeutralityMultiple interests representedConsistencyConsistencyAlternative forms (equivalence)Split-half and inter-item correlation

VALIDITY IN QUANTITATIVE AND QUALITATIVE RESEARCHValidity in quantitative research often concerns: objectivity, generalizability, replicability, predictability, controllability, nomothetic statements.Validity in qualitative research often concerns: honesty, richness, authenticity, depth, scope, subjectivity, strength of feeling, catching uniqueness, idiographic statements.

CatalyticConcurrentConsequentialConstructContentCriterion-relatedConvergent & discriminantCross-culturalCultural validityDescriptive

EcologicalEvaluativeExternalFaceInternalInterpretiveJuryPredictiveSystemicTheoreticalTYPES OF VALIDITY

VALIDITY IN QUANTITATIVE RESEARCHConcurrentConstructContentCriterion-relatedConvergent & discriminantCross-cultural

EvaluativeExternalFaceInternalJuryPredictiveTheoretical

VALIDITY IN MIXED METHODS RESEARCHRepresentationLegitimation Sample integration Inside-outside Weakness minimization Sequential Conversion Paradigmatic mixing Commensurability Multiple validities Political Integration (of methods)

ESTABLISHING VALIDITY IN QUALITATIVE RESEARCHProlonged engagement in the fieldPersistent observation Triangulation Leaving an audit trail Respondent validationWeighting the evidence (giving priority)Checking for representativeness Checking for researcher effectsMaking contrast/comparisons Theoretical sampling Checking the meaning of outliers Using extreme cases

ESTABLISHING VALIDITY IN QUALITATIVE RESEARCHRuling out spurious relations Replicating a finding Referential adequacy Following up surprises Structural relationships Peer debriefing Rich and thick description Looking for possible sources of invalidityAssessing rival explanations Negative case analysisConfirmatory data analysis Effect sizes

THREATS TO VALIDITY IN QUANTITATIVE RESEARCHHistory Maturation Statistical regression Testing Instrumentation Selection Bias Experimental mortality Instrument reactivity Selection-maturation interaction Type I and Type II errors

VALIDITY PROBLEMS IN CROSS-CULTURAL RESEARCHFailure to operationalize elements of culturesWhose construction of culture to adopt: emic/etic False attribution of causality to cultural factors rather than non-cultural factors Directions of causalityEcological fallacySampling and instrumentationConvergent and discriminant validity Response bias and preparation of participants Language problemsProblems of equivalence (conceptual, psychological, meaning, instrument, understanding, significance, relevance, measurement, linguistic)

THREATS TO EXTERNAL VALIDITY IN QUANTITATIVE RESEARCHFailure to describe independent variables explicitly Lack of representativeness of available and target populations Hawthorne effect Inadequate operationalizing of dependent variables Sensitization/reactivity to experimental/research conditions Interaction effects of extraneous factors and experimental/ research treatments Invalidity or unreliability of instruments Ecological validityMultiple treatment validity

THE HAWTHORNE EFFECTBetween 1927 and 1932 researchers carried out experiments at the Western Electric Companys Hawthorne plant.Purposes: To examine the effects of changes of working conditions on output of workersSample: Six women, chosen as average workersMethod: Women worked in a test room. Output measured under different conditions (e.g. no change change to method of payment introduce two rest periods introduce six rest periods changes in lighting conditions, early clocking-off, five-day working week return to initial conditionsDuration: 15 weeks

THE HAWTHORNE EFFECTResults: Output rose steadily during test period and after the test period. Conclusion: Output did not seem to depend on test conditions. Increased output seemed to be due to the fact that the people had been involved in the experiment itself, i.e. the act of research had affected the results. The results were a research of the research itself.Implications: The act of being involved in research itself affects the results.

THREATS TO EXTERNAL VALIDITY IN QUALITATIVE RESEARCHSelection effectsSetting effects History effects Construct effects

ENSURING VALIDITY AT THE DESIGN STAGEChoose an appropriate time scale;Ensure adequate resources for the researchSelect appropriate methodology Select appropriate instruments Use an appropriate sample Ensure reliability Select appropriate fociAvoid having biased researcher(s)

ENSURING VALIDITY AT THE DATA COLLECTION STAGEReduce the Hawthorne effect Minimize reactivity Avoid drop-out rates amongst respondentsTake steps to avoid non-return of questionnairesAvoid too long or too short an interval between pre-tests and post-testsEnsure inter-rater reliabilityMatch control and experimental groupsEnsure standardized procedures for gathering dataBuild on the motivations of respondentsTailor instruments to situational factors Address researcher characteristics

ENSURING VALIDITY AT THE DATA ANALYSIS STAGEUse respondent validation;Avoid subjective interpretation of data Reduce the halo effectUse appropriate statistical treatmentsRecognize extraneous factors which may affect data Avoid poor coding of qualitative dataAvoid making inferences/generalizations beyond the dataAvoid equating correlations and causesAvoid selective use of dataAvoid unfair aggregation of dataAvoid degrading the data;Avoid Type I and/or Type II errors

ENSURING VALIDITY AT THE DATA REPORTING STAGEAvoid using data selectively and unrepresentatively Indicate the context and parameters of the research Present the data without misrepresenting the messageMake claims which are sustainable by the dataAvoid inaccurate or wrong reporting of data Ensure that the research questions are answeredRelease research results neither too soon nor too late

RELIABILITY IN QUANTITATIVE AND QUALITATIVE RESEARCHReliability in quantitative research:consistency (stability), accuracy, predictability, equivalence, replicability, concurrence, descriptive and causal potential.Reliability in qualitative research:accuracy, fairness, dependability, comprehensiveness, respondent validation, checkability, empathy, uniqueness, explanatory and descriptive potential, confirmability.

Reliability as stability: Consistency over time and samples;Reliability as equivalence: Equivalent forms of same instrument;Inter-rater reliability;Reliability as internal consistency:Split half reliability (e.g. for test items)TYPES OF RELIABILITY IN QUANTITATIVE RESEARCH

TRIANGULATIONMethodologiesInstrumentsResearchersTimeLocationTheoriesSamplesParticipantsData

SPLIT-HALF RELIABILITY(Spearman-Brown)Reliability =

r = the actual correlation between the two halves of the instrument (e.g. 0.85);

Reliability = = = 0.919

RELIABILITY IN QUALITATIVE RESEARCHCredibilityNeutralityConfirmabilityDependabilityConsistencyApplicabilityTrustworthinessTransferability

RELIABILITY AND REPLICATION IN QUALITATIVE RESEARCHRepeat:The status position of the researcherThe choice of informants/respondentsThe social situations and conditionsThe analytic constructs usedThe methods of data collection and analysisAddress:Stability of observationsParallel formsInter-rater reliabilityRespondent validation

IMPROVING RELIABILITYMinimise external sources of variation;Standardise conditions under which measurement occurs;Improve researcher consistency;Broaden the sample of measurement questions by: adding similar questions to the instrument;increasing the number of researchers (triangulation);increasing the number of occasions in an observational study.Exclude extreme responses (outliers).

RELIABILITY AND VALIDITY AT ALL STAGESDesign and methodologySamplingInstrumentationTimingData collectionData analysisData reporting