HANDBOOK OF PSYCHOLOGY

HANDBOOK OF PSYCHOLOGYVOLUME 2: RESEARCH METHODS IN PSYCHOLOGY

Second Edition

Volume Editors

JOHN A. SCHINKA AND WAYNE F. VELICER

Editor-in-Chief

IRVING B. WEINER

John Wiley & Sons, Inc.

This book is printed on acid-free paper.

Copyright © 2013 by John Wiley & Sons, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical,photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, withouteither the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright ClearanceCenter, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600, or on the web at www.copyright.com. Requests tothe Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030,(201) 748-6011, fax (201) 748-6008.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make norepresentations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any impliedwarranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written salesmaterials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional whereappropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited tospecial, incidental, consequential, or other damages.

This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold with theunderstanding that the publisher is not engaged in rendering professional services. If legal, accounting, medical, psychological or any other expertassistance is required, the services of a competent professional person should be sought.

The contents of this work are intended to further general scientific research, understanding, and discussion only and are not intended and should notbe relied upon as recommending or promoting a specific method, diagnosis, or treatment by physicians for any particular patient. The publisher andthe author make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaimall warranties, including without limitation any implied warranties of fitness for a particular purpose. In view of ongoing research, equipmentmodifications, changes in governmental regulations, and the constant flow of information relating to the use of medicines, equipment, and devices,the reader is urged to review and evaluate the information provided in the package insert or instructions for each medicine, equipment, or devicefor, among other things, any changes in the instructions or indication of usage and for added warnings and precautions. Readers should consult witha specialist where appropriate. The fact that an organization or Website is referred to in this work as a citation and/or a potential source of furtherinformation does not mean that the author or the publisher endorses the information the organization or Website may provide or recommendations itmay make. Further, readers should be aware that Internet Websites listed in this work may have changed or disappeared between when this workwas written and when it is read. No warranty may be created or extended by any promotional statements for this work. Neither the publisher northe author shall be liable for any damages arising herefrom.

Designations used by companies to distinguish their products are often claimed as trademarks. In all instances where John Wiley & Sons, Inc. isaware of a claim, the product names appear in initial capital or all capital letters. Readers, however, should contact the appropriate companies formore complete information regarding trademarks and registration.

For general information on our other products and services please contact our Customer Care Department within the United States at (800)762-2974, outside the United States at (317) 572-3993 or fax (317) 572-4002.

Wiley publishes in a variety of print and electronic formats and by print-on-demand. Some material included with standard print versions of thisbook may not be included in e-books or in print-on-demand. If this book refers to media such as a CD or DVD that is not included in the versionyou purchased, you may download this material at http://booksupport.wiley.com. For more information about Wiley products, visit www.wiley.com.

Library of Congress Cataloging-in-Publication Data:Handbook of psychology / Irving B. Weiner, editor-in-chief. — 2nd ed.

v. cm.Includes bibliographical references and index.ISBN 978-0-470-61904-9 (set) – ISBN 978-0-470-89064-6 (cloth : v.2); ISBN 978-1-118-28203-8 (ebk.); ISBN 978-1-118-28254-0 (ebk.);

ISBN 978-1-118-28628-9 (ebk.)1. Psychology. I. Weiner, Irving B.BF121.H213 2013150—dc23

2012005833

Printed in the United States of America10 9 8 7 6 5 4 3 2 1

My efforts in this work are dedicated to Couper Harowski and Henry Harowski.J.A.S.

This volume is dedicated to Scott and Clayton, two young men who make a father very proud.W.F.V.

Editorial Board

Volume 1History of PsychologyDonald K. Freedheim, PhDCase Western Reserve UniversityCleveland, Ohio

Volume 2Research Methods in PsychologyJohn A. Schinka, PhDUniversity of South FloridaTampa, Florida

Wayne F. Velicer, PhDUniversity of Rhode IslandKingston, Rhode Island

Volume 3Behavioral NeuroscienceRandy J. Nelson, PhDOhio State UniversityColumbus, Ohio

Sheri J. Y. Mizumori, PhDUniversity of WashingtonSeattle, Washington

Volume 4Experimental PsychologyAlice F. Healy, PhDUniversity of ColoradoBoulder, Colorado

Robert W. Proctor, PhDPurdue UniversityWest Lafayette, Indiana

Volume 5Personality and Social PsychologyHoward Tennen, PhDUniversity of Connecticut Health CenterFarmington, Connecticut

Jerry Suls, PhDUniversity of IowaIowa City, Iowa

Volume 6Developmental PsychologyRichard M. Lerner, PhDM. Ann Easterbrooks, PhDJayanthi Mistry, PhDTufts UniversityMedford, Massachusetts

Volume 7Educational PsychologyWilliam M. Reynolds, PhDHumboldt State UniversityArcata, California

Gloria E. Miller, PhDUniversity of DenverDenver, Colorado

Volume 8Clinical PsychologyGeorge Stricker, PhDArgosy University DCArlington, Virginia

Thomas A. Widiger, PhDUniversity of KentuckyLexington, Kentucky

vii

viii Editorial Board

Volume 9Health PsychologyArthur M. Nezu, PhDChristine Maguth Nezu, PhDPamela A. Geller, PhDDrexel UniversityPhiladelphia, Pennsylvania

Volume 10Assessment PsychologyJohn R. Graham, PhDKent State UniversityKent, Ohio

Jack A. Naglieri, PhDUniversity of VirginiaCharlottesville, Virginia

Volume 11Forensic PsychologyRandy K. Otto, PhDUniversity of South FloridaTampa, Florida

Volume 12Industrial and Organizational

PsychologyNeal W. Schmitt, PhDMichigan State UniversityEast Lansing, Michigan

Scott Highhouse, PhDBowling Green State UniversityBowling Green, Ohio

Contents

Handbook of Psychology Preface xiiiIrving B. Weiner

Volume Preface xvJohn A. Schinka and Wayne F. Velicer

Contributors xxi

I FOUNDATIONS OF RESEARCH ISSUES 1

1 EXPERIMENTAL DESIGN 3Roger E. Kirk

2 EXPLORATORY DATA ANALYSIS 34John T. Behrens, Kristen E. DiCerbo, Nedim Yel, and Roy Levy

3 STATISTICAL POWER ANALYSIS 71Joseph S. Rossi

4 METHODS FOR HANDLING MISSING DATA 109John W. Graham, Patricio E. Cumsille, and Allison E. Shevock

5 EFFECT SIZE ESTIMATION AND CONFIDENCE INTERVALS 142Fiona Fidler and Geoff Cumming

6 EXPLORATORY FACTOR ANALYSIS: BASICS AND BEYOND 164James B. Hoelzle and Gregory J. Meyer

7 CLUSTERING AND CLASSIFICATION METHODS 189Glenn W. Milligan and Stephen C. Hirtle

ix

x Contents

II RESEARCH METHODS IN SPECIFIC CONTENT AREAS 211

8 CLINICAL FORENSIC PSYCHOLOGY 213Kevin S. Douglas, Randy K. Otto, Sarah L. Desmarais, and Randy Borum

9 HEALTH PSYCHOLOGY 245Linda J. Luecken and Rika Tanaka

10 RESEARCH METHODS IN NEUROPSYCHOLOGY 274Russell M. Bauer and Callie Beck Dunn

11 RESEARCH DESIGNS FOR PROGRAM EVALUATION 316Vivian C. Wong, Coady Wing, Peter M. Steiner, Manyee Wong, and Thomas D. Cook

12 BEHAVIOR GENETICS 342Christina N. Lessov-Schlaggar, Arpana Agrawal, and Gary E. Swan

III MEASUREMENT ISSUES 367

13 MOOD MEASUREMENT: CURRENT STATUSAND FUTURE DIRECTIONS 369David Watson and Jatin G. Vaidya

14 MEASURING PERSONALITY AND PSYCHOPATHOLOGY 395Leslie C. Morey

15 UTILIZING MIXED METHODS IN PSYCHOLOGICAL RESEARCH 428Abbas Tashakkori, Charles Teddlie, and Marylyn C. Sines

16 ITEM RESPONSE THEORY AND MEASURING ABILITIES 451Karen M. Schmidt and Susan E. Embretson

17 AN OVERVIEW OF LATENT CURVE AND LATENT CHANGESCORE ANALYSES 474John J. McArdle and John R. Nesselroade

IV DATA ANALYSIS ISSUES 509

18 MULTIPLE LINEAR REGRESSION 511Leona S. Aiken, Stephen G. West, Steven C. Pitts, Amanda N. Baraldi, and Ingrid C. Wurpts

19 LOGISTIC REGRESSION: BASIC FOUNDATIONSAND NEW DIRECTIONS 543Alfred De Maris

Contents xi

20 META-ANALYSIS 571Frank L. Schmidt

21 SURVIVAL ANALYSIS 595John B. Willett, Judith D. Singer, and Suzanne E. Graham

22 TIME SERIES ANALYSIS FOR PSYCHOLOGICAL RESEARCH 628Wayne F. Velicer and Peter C. Molenaar

23 STRUCTURAL EQUATION MODELING 661Jodie B. Ullman and Peter M. Bentler

24 AN INTRODUCTION TO LATENT CLASS AND LATENTTRANSITION ANALYSIS 691Stephanie T. Lanza, Bethany C. Bray, and Linda M. Collins

25 STATISTICAL MEDIATION ANALYSIS 717David P. MacKinnon and Davood Tofighi

Author Index 737

Subject Index 765

Handbook of Psychology Preface

The first edition of the 12-volume Handbook of Psychol-ogy was published in 2003 to provide a comprehensiveoverview of the current status and anticipated future direc-tions of basic and applied psychology and to serve asa reference source and textbook for the ensuing decade.With 10 years having elapsed, and psychological knowl-edge and applications continuing to expand, the time hascome for this second edition to appear. In addition to well-referenced updating of the first edition content, this secondedition of the Handbook reflects the fresh perspectives ofsome new volume editors, chapter authors, and subjectareas. However, the conceptualization and organizationof the Handbook , as stated next, remain the same.

Psychologists commonly regard their discipline as thescience of behavior, and the pursuits of behavioral scien-tists range from the natural sciences to the social sciencesand embrace a wide variety of objects of investigation.Some psychologists have more in common with biologiststhan with most other psychologists, and some have morein common with sociologists than with most of their psy-chological colleagues. Some psychologists are interestedprimarily in the behavior of animals, some in the behav-ior of people, and others in the behavior of organizations.These and other dimensions of difference among psycho-logical scientists are matched by equal if not greater het-erogeneity among psychological practitioners, who apply avast array of methods in many different settings to achievehighly varied purposes. This 12-volume Handbook of Psy-chology captures the breadth and diversity of psychologyand encompasses interests and concerns shared by psy-chologists in all branches of the field. To this end, lead-ing national and international scholars and practitionershave collaborated to produce 301 authoritative and detailedchapters covering all fundamental facets of the discipline.

Two unifying threads run through the science of behav-ior. The first is a common history rooted in conceptualand empirical approaches to understanding the nature ofbehavior. The specific histories of all specialty areas inpsychology trace their origins to the formulations of theclassical philosophers and the early experimentalists, andappreciation for the historical evolution of psychology inall of its variations transcends identifying oneself as a par-ticular kind of psychologist. Accordingly, Volume 1 in theHandbook , again edited by Donald Freedheim, is devotedto the History of Psychology as it emerged in many areasof scientific study and applied technology.

A second unifying thread in psychology is a commit-ment to the development and utilization of research meth-ods suitable for collecting and analyzing behavioral data.With attention both to specific procedures and to theirapplication in particular settings, Volume 2, again editedby John Schinka and Wayne Velicer, addresses ResearchMethods in Psychology .

Volumes 3 through 7 of the Handbook present thesubstantive content of psychological knowledge in fiveareas of study. Volume 3, which addressed Biological Psy-chology in the first edition, has in light of developments inthe field been retitled in the second edition to cover Behav-ioral Neuroscience. Randy Nelson continues as editor ofthis volume and is joined by Sheri Mizumori as a new co-editor. Volume 4 concerns Experimental Psychology andis again edited by Alice Healy and Robert Proctor. Volume5 on Personality and Social Psychology has been reorga-nized by two new co-editors, Howard Tennen and JerrySuls. Volume 6 on Developmental Psychology is againedited by Richard Lerner, Ann Easterbrooks, and Jayan-thi Mistry. William Reynolds and Gloria Miller continueas co-editors of Volume 7 on Educational Psychology .

xiii

xiv Handbook of Psychology Preface

Volumes 8 through 12 address the application of psy-chological knowledge in five broad areas of professionalpractice. Thomas Widiger and George Stricker continue asco-editors of Volume 8 on Clinical Psychology . Volume 9on Health Psychology is again co-edited by Arthur Nezu,Christine Nezu, and Pamela Geller. Continuing to co-editVolume 10 on Assessment Psychology are John Grahamand Jack Naglieri. Randy Otto joins the Editorial Boardas the new editor of Volume 11 on Forensic Psychology .Also joining the Editorial Board are two new co-editors,Neal Schmitt and Scott Highhouse, who have reorganizedVolume 12 on Industrial and Organizational Psychology .

The Handbook of Psychology was prepared to educateand inform readers about the present state of psychologicalknowledge and about anticipated advances in behavioralscience research and practice. To this end, the Handbookvolumes address the needs and interests of three groups.First, for graduate students in behavioral science, the vol-umes provide advanced instruction in the basic conceptsand methods that define the fields they cover, togetherwith a review of current knowledge, core literature, andlikely future directions. Second, in addition to serving asgraduate textbooks, the volumes offer professional psy-chologists an opportunity to read and contemplate theviews of distinguished colleagues concerning the cen-tral thrusts of research and the leading edges of practice

in their respective fields. Third, for psychologists seek-ing to become conversant with fields outside their ownspecialty and for persons outside of psychology seekinginformation about psychological matters, the Handbookvolumes serve as a reference source for expanding theirknowledge and directing them to additional sources inthe literature.

The preparation of this Handbook was made possibleby the diligence and scholarly sophistication of 24 vol-ume editors and co-editors who constituted the EditorialBoard. As Editor-in-Chief, I want to thank each of thesecolleagues for the pleasure of their collaboration in thisproject. I compliment them for having recruited an out-standing cast of contributors to their volumes and thenworking closely with these authors to achieve chaptersthat will stand each in their own right as valuable con-tributions to the literature. Finally, I would like to thankBrittany White for her exemplary work as my adminis-trator for our manuscript management system, and theeditorial staff of John Wiley & Sons for encouraging andhelping bring to fruition this second edition of the Hand-book , particularly Patricia Rossi, Executive Editor, andKara Borbely, Editorial Program Coordinator.

Irving B. WeinerTampa, Florida

Volume Preface

We were pleased to be invited to work on this secondedition of our volume for the Handbook of Psychology . Wehad felt that the first edition was a very solid effort, but didnot anticipate the degree of success in terms of acceptanceby the profession, reviews, and citations. Accepting theoffer to work on the second edition was an easy and quickdecision. In preparing the second edition, we carefullyreviewed the offerings of the first edition. We felt thatmany of the chapters continued to be important and wouldbe especially valuable with substantial updates. We alsodecided to replace several chapters with those coveringnew topics that we felt were increasingly important. Fortopics continued in the second edition, there were somechanges in authors, primarily due to lack of availability.We placed as much emphasis on recruiting authors as wedid in the first edition. In all cases, we were able to recruitindividuals who are well-recognized as experts.

We were very impressed by how dynamic the area ofresearch methodology is currently. As a rough estimate,more than 35% of the material in this edition is new sincethe first edition. The area continues to rapidly evolve withnew methods being developed and older methods beingrefined. This is driven partially by the advances in infor-mation systems technology and partially by conceptualadvances. We continue to believe, as we did in the firstedition, that the evolution in research methodology in psy-chology is characterized by six features that guided ourselection of topics. We repeat them from the preface ofthe first edition.

First , there has been a focus on the development of pro-cedures that employ statistical control rather than experi-mental control. Because most of the recent growth involvesresearch in areas that preclude direct control of indepen-dent variables, multivariate statistics and the developmentof methods such as path analysis and structural equation

modeling have been critical developments. Second , therehas been an increasing focus on construct-driven, or latentvariable, research. A construct is defined by multipleobserved variables. Constructs can be viewed as morereliable and more generalizable than a single observed vari-able. Constructs are also theoretically based. This theory-based approach serves to guide study design, the choiceof variables, the data analysis, and the data interpreta-tion. Third , there has been an increasing emphasis on thedevelopment of new measures and new measurement mod-els. With the movement of the field out of the laboratoryand with advances in technology, the repertoire of mea-sures, the quality of the measures, and the sophistication ofthe measurement models have all increased dramatically.Fourth , there is increasing recognition of the importance ofthe temporal dimension in understanding a broad range ofpsychological phenomena. Psychology has become moreof an intervention-oriented science, recognizing not onlythe complexity of treatment effects but also the impor-tance of the change in patterns of the effects over time.Fifth , new methods of analysis have been developed thatno longer require the assumption of a continuous, equal-interval, normally distributed variable. New methods havebeen developed for categorical, ordinal, or simply nonnor-mal variables that can perform an equally sophisticatedanalysis. Sixth , the importance of individual differencesis increasingly emphasized in intervention studies. Thisbecomes increasingly important as we recognize that inter-ventions do not affect everyone in exactly the same ways,and interventions become more and more tailored to theindividual.

In addition, a seventh area that became clear in prepar-ing this edition is the continuing evolution of meth-ods based on effect size estimation, starting with poweranalysis and continuing through meta-analysis and, more

xv

xvi Volume Preface

recently, the development of alternatives to null hypothe-sis testing based on effect size estimation and confidenceintervals.

As in our first edition, the text is organized into fourparts. The first part addresses issues that are fundamentalto all behavioral science research and is labeled Foun-dations of Research Issues . The focus of this part is onstudy design, data management, data reduction, and datasynthesis. The first chapter, “Experimental Design” byRoger E. Kirk, discusses principles of good experimen-tal design—random assignment, replication, and localcontrol—that were initially championed by Ronald A.Fisher almost one hundred years ago. Despite the increas-ing complexity of psychological research methodologies,Kirk demonstrates how complex experimental designs canbe understood in terms of three simple building blockdesigns: the completely randomized design, randomizedblock design, and Latin square design.

The second chapter, “Exploratory Data Analysis” byJohn T. Behrens, Kristen E. DiCerbo, Nedim Yel, andRoy Levy, reminds us of the fundamental importance oflooking at data in the most basic ways as a first step inany data analysis. In some ways this represents a “back tothe future” chapter. Advances in computer-based graph-ical methods have brought a great deal of sophisticationto this very basic first step. The third chapter, “StatisticalPower Analysis” by Joseph S. Rossi, reflects the criticalchange in focus for psychological research. Originally,the central focus of a test of significance was on control-ling Type I error rates. The late Jacob Cohen emphasizedthat researchers should be equally concerned with Type IIerrors. This resulted in an emphasis on the careful plan-ning of a study and a concern with effect size and selectingthe appropriate sample size. Rossi updates and extendsthese concepts. Chapter 4, “Methods for Handling Miss-ing Data” by John W. Graham, Patricio E. Cumsille, andAllison E. Shevock, describes the impressive statisticaladvances in addressing the common practical problem ofmissing observations. Previously, researchers had reliedon a series of ad hoc procedures, often resulting in veryinaccurate estimates. The new statistical procedures allowthe researcher to articulate the assumptions about the rea-son the data are missing and make very sophisticatedestimates of the missing value based on all the availableinformation. This topic has taken on even more impor-tance with the increasing emphasis on longitudinal studiesand the inevitable problem of attrition.

The fifth chapter, “Effect Size Estimation and Confi-dence Intervals” by Fiona Fidler and Geoff Cummings,describes an alternative approach to traditional null

hypothesis testing. There has been a growing consensusthat null hypothesis testing is a flawed basis for performingstatistical decision making. The chapter’s focus is on effectsize estimation and confidence intervals and representsan alternative approach that has the potential to achieveeverything that traditional null hypothesis approacheshave achieved but with a focus on quantitative estimationrather than just binary decision making. Informationabout the strength of an intervention rather than just theexistence of an intervention effect has the potential toguide the comparison of alternative interventions.

The last two chapters in this part, “Exploratory Fac-tor Analysis: Basics and Beyond” by James B. Hoelzleand Gregory J. Meyer and “Clustering and ClassificationMethods” by Glenn W. Milligan and Stephen C. Hirtle,describe two widely employed parsimony methods. Fac-tor analysis operates in the variable domain and attemptsto reduce a set of p observed variables to a smaller setof m factors. These factors, or latent variables, are moreeasily interpreted and facilitate interpretation. Hoelzle andMeyer present a discussion of the issues that are criticalin conducting a factor analysis, including determinationof sample size, the impact of test variables on findings,empirically supported methods to guide factor retention,and the impact of various factor rotation decisions oninterpretation. They then illustrate these issues by present-ing a principal components analysis (PCA; a commonlyused method of factor analysis) based on data from awidely used personality inventory. In the last section ofthe chapter they introduce material to illustrate extensionsof PCA that allow for investigation of structural congru-ency across samples and estimation for how a derivedPCA model may fare when evaluated by more restrictiveanalyses (i.e., confirmatory factor analysis). Cluster analy-sis operates in the person domain and attempts to reduce aset of N individuals to a set of k clusters. Cluster analysisserves to explore the relationships among individuals andto organize the set of individuals into a limited numberof subtypes that share essential features. These methodsare basic to the development of construct-driven methodsand the focus on individual differences.

The second part, Research Methods in Specific Con-tent Areas , addresses research methods and issues asthey apply to specific content areas. Content areas werechosen in part to parallel the other texts of the Hand-book . More importantly, however, we attempted to samplecontent areas from a broad spectrum of specialization,with the hope that these chapters would provide insightsinto methodological concerns and solutions that wouldgeneralize to other areas. The eighth chapter, “Clinical

Volume Preface xvii

Forensic Psychology” by Kevin S. Douglas, Randy K.Otto, Sarah L. Desmarais, and Randy Borum, addressesresearch methods and issues that occur in assessment andtreatment contexts. This chapter is substantially updatedfrom the previous edition, reflecting the rapidly increas-ing body of research published in the past 10 years. Foreach task that is unique to clinical forensic psychologyresearch, the authors provide examples of the clinicalchallenges confronting the psychologist, identify prob-lems faced when researching the issues or constructs, anddescribe research strategies that have been employed, theirstrengths, and their limitations.

The emerging field of health psychology promotes abiopsychosocial framework that reflects the complex inter-play of biological, psychological, and social factors thataffect, and are affected by, one’s health. In the biopsy-chosocial framework, health and illness emerge frominteracting influences at the cellular, organismic, interper-sonal, and environmental levels. Insight into the method-ological issues and solutions for research in each of thesetopical areas is provided by Linda J. Luecken and RikaTanaka in the ninth chapter, “Health Psychology.” Theseauthors begin by briefly discussing major thematic areasof research in health psychology (e.g., health behaviorsand risk reduction), describe the multiple levels of analysisthat characterize health psychology research, address com-mon issues in study design and methods commonly usedto understand health, and conclude by reviewing emerg-ing methods that offer promise for advancing researchon mind–body connections in health and illness. In the10th chapter, “Research Methods in Neuropsychology,”Russell M. Bauer and Callie Beck Dunn provide a dis-cussion of neuropsychological inference; an overview ofmajor approaches to neuropsychological research; and anupdated review of emergent techniques of structural andfunctional neuroimaging (e.g., PET, SPECT, functionalMRI, magnetoencephalography [MEG], voxel-based mor-phometry [VBM], diffusion tensor imaging [DTI] trac-tography, and magnetic resonance spectroscopy) and theutilization of new tracing compounds, electrophysiol-ogy, and reversible lesion and stimulation methods (e.g.,transcranial magnetic stimulation, transcranial direct cur-rent stimulation, and intracarotid sodium amobarbitaltechnique). Particularly informative is new material onunderstanding normal and disordered neuropsychologi-cal function by means of cross-platform and transla-tional approaches that combine the strengths of thesevarious techniques within the same experimental inves-tigation and that speed basic discoveries to the bedsideor clinic.

Whatever their specialty area, when psychologists eval-uate a program or policy, the question of impact is often atcenter stage. The 11th chapter, “Research Designs for Pro-gram Evaluation” by Vivian C. Wong, Coady Wing, PeterSteiner, Manyee Wong, and Thomas D. Cook, addressesthe advances in improving methodological approaches forestablishing causal inference in program evaluation. Thechapter presents four research designs for assessing pro-gram effects. For each, basic features of the approach,potential outcomes, and common research issues are con-sidered. Examples illustrate how these designs have beenused to assess program effects. Chapter 12, the finalchapter in this part, is “Behavior Genetics,” by ChristinaN. Lessov-Schlaggar, Arpana Agrawal, and Gary E. Swan.This topic was added to the volume to reflect the explosionof behavior genetics research in the past decade, reflect-ing new technologies for genetic and genomic analysisand advances in study designs. The authors introduce thefundamental concepts about transmission of genetic mate-rial from parents to offspring and then provide descrip-tions of measures, study designs, and analytical meth-ods that allow for estimation of genetic, environmental,and gene–environment interaction components of humanbehavior.

The third part addresses Measurement Issues . Advancesin measurement typically combine innovation in technol-ogy and progress in theory. As our measures become moresophisticated, the areas of application also increase. Moodemerged as a seminal concept within psychology duringthe 1980s, and its prominence has continued unabated eversince. In the 13th chapter, “Mood Measurement: CurrentStatus and Future Directions,” David Watson and JatinG. Vaidya examine recent research regarding the underly-ing structure of mood, describe and evaluate many of themost important mood measures, and discuss several issuesrelated to the reliability and construct validity of moodmeasurement. Their review of the existing evidence pointsto specific approaches that can be applied to improve thereliability and construct validity of trait affect measures.In Chapter 14, “Measuring Personality and Psychopathol-ogy,” Leslie C. Morey uses objective self-report methodsof measurement to illustrate contemporary procedures forscale development and validation, addressing issues criti-cal to all measurement. In this updated chapter he addressesnew studies and approaches to issues of reliability andvalidity with the goal of providing a state-of-the-art per-spective on self-report methods.

As the content of our previous volume reflected, psy-chological research has focused almost exclusively onquantitative experimental and quasi-experimental designs.

xviii Volume Preface

In the past decade, however, increasing attention has beendirected to qualitative research—the examination, analy-sis, and interpretation of observations for the purpose ofdiscovering underlying meanings and patterns of relation-ships that do not involve mathematical models. This efforthas produced a hybrid approach that attempts to combinethe most productive features of both quantitative and qual-itative perspectives. In the 15th chapter, “Utilizing MixedMethods in Psychological Research,” Abbas Tashakkori,Charles Teddlie, and Marylyn C. Sines describe com-monly used designs and procedures in the hybrid “mixedmethod” approach with the goal of highlighting the ben-efits of the approach over solo quantitative or qualitativeapproaches.

The 16th chapter, “Item Response Theory and Mea-suring Abilities” by Karen M. Schmidt and Susan E.Embretson, describes the formal models that have beendeveloped to guide measure development. For many years,most tests of ability and achievement have relied on Clas-sical Test Theory as a framework to guide both measuredevelopment and measure evaluation. Item Response The-ory updates this model in many important ways, permit-ting the development of a new generation of measuresof abilities and achievement that are particularly appro-priate for a more interactive model of assessment. The17th chapter, “An Overview of Latent Curve and LatentChange Score Analyses” by John J. McArdle and John R.Nesselroade, describes new quantitative methods for thestudy of change. The focus shifts to the pattern of changeover time. The methods permit the researcher to model awide variety of different patterns of developmental changeover time. This represents a shift from cross-sectional tolongitudinal thinking.

The final part, Data Analysis Issues , addresses statis-tical procedures that are increasingly valued but still notwidely employed by many researchers. These methodsare especially useful in addressing more complex researchquestions. The 18th chapter, “Multiple Linear Regression”by Leona S. Aiken, Stephen G. West, Steven C. Pitts,Amanda N. Baraldi, and Ingrid C. Wurpts, describes theadvances in multiple linear regression that permit applica-tions of this very basic method to the analysis of complexdata sets and the incorporation of conceptual models toguide the analysis. The testing of theoretical predictionsand the identification of implementation problems are thetwo major foci of this chapter. Multiple regression alsorepresents a method that serves as a basis for most othermultivariate methods.

Chapter 19, “Logistic Regression” by Alfred DeMaris,describes a parallel method to multiple regression analy-sis for categorical variables. The procedure has primarilybeen outside psychology, but is now being used muchmore frequently to address psychological questions andis generally preferable to discriminant function analysis.This updated chapter now includes discussion of topicssuch as propensity-score matching and fixed-effects logis-tic regression models. Chapter 20, “Meta-Analysis” byFrank L. Schmidt, describes procedures that have beendeveloped for the quantitative integration of research find-ings across multiple studies. Previously, research findingswere integrated in narrative form, subject to the biases ofthe reviewer. The method also focuses attention on theimportance of effect size estimation.

Chapter 21, “Survival Analysis” by John B. Willett,Judith D. Singer, and Suzanne E. Graham, describes arecently developed method for analyzing longitudinal data.One approach is to code whether an event has occurred ata given occasion. By switching the focus on the time to theoccurrence of the event, a much more powerful and sophis-ticated analysis can be performed. Again, the developmentof this procedure has occurred largely outside psychologybut is being employed much more frequently. Chapter 22,“Time Series Analysis for Psychological Research” byWayne F. Velicer and Peter C. M. Molenaar, describes amethod for studying the change in a single individual overtime. Instead of a single observation on many subjects, thismethod relies on many observations on a single subject.This method is representative of a broad class of meth-ods called idiographic methods that is distinctly differentfrom the nomothetic methods that are widely employedin the behavioral sciences. The focus is on the patternof change over time, and the method permits researchersto address unique research questions. Chapter 23, “Struc-tural Equation Modeling” by Jodie B. Ullman and PeterM. Bentler, describes a very general method that com-bines three key themes: the focus on constructs or latentvariables, the focus on statistical control, and the focus ontheory to guide data analysis. First employed as an analyticmethod little more than 20 years ago, the method is nowwidely disseminated in the behavioral sciences.

Chapter 24, “An Introduction to Latent Class and LatentTransition Analysis” by Stephanie T. Lanza, Bethany C.Bray, and Linda M. Collins, describes a new method foranalyzing change over time. It is particularly appropriatewhen the change process can be conceptualized as aseries of discrete states. The approach provides another

Volume Preface xix

method of studying the pattern of change over time.Chapter 25, “Statistical Mediation Analysis” by DavidP. MacKinnon and Davood Tofighi, describes a methodthat focuses on determining how and why two thingsare related. The variables that can explain an observedrelationship are called mediating variables. The mediatingvariable is intermediate in a causal sequence relating anindependent variable to a dependent variable. Questionsabout mediating processes are central to basic and appliedresearch in many fields.

In completing this edition, we were again fortunateto have Irving Weiner as our executive editor. His sup-port and counsel were invaluable in all stages of thework. We were also fortunate to have contributing authorswho brought the highest level of professionalism to theeffort. Consistent with their reputations, these individu-als delivered chapters of exceptional quality. Because ofthe length of the project, we shared many contributors’experiences—marriages, births, illnesses, family crises. Adefinite plus for us has been the formation of new friend-ships and professional liaisons.

Our editorial tasks were also assisted greatly by the gen-erous assistance of our reviewers, most of whom will bequickly recognized by our readers for their own expertise

in research methodology. We thank Steven Babbin,Heather Belanger, Nayena Blankson, William Chaplin,Glenn Curtiss, Kevin Delucchi, Ron Gironda, LisaHarlow, Donald Hedecker, Beth Jenkins, Lisa Keller, RexKline, Todd Little, Scott Maxwell, Roderick McDonald,Heather McGee, Kevin Murphy, Daniel Ozer, RichardPalumbo, Kristopher J. Preacher, Stephen Reise, JohnRock, Joseph Rossi, Brent Small, Daniel Stahl, JudithStein, John Tisak, Paul Velleman, and several anonymousreviewers for their thorough work and good advice.

We finish with a caveat. This text’s audience willinevitably discover several contradictions or disagree-ments across the chapter offerings. Inevitably, researchersin different areas solve similar methodological problemsin different ways. These differences are reflected in theofferings of this text, and we have not attempted to medi-ate these differing viewpoints. Rather, we believe theserious researcher will welcome the opportunity to reviewsolutions suggested or supported by differing approaches.For flaws in the text, however, the usual rule applies: Weassume all responsibility.

John A. Schinka

Wayne F. Velicer

Contributors

Arpana Agrawal, PhDDepartment of PsychiatryWashington UniversitySt. Louis, Missouri

Leona S. Aiken, PhDDepartment of PsychologyArizona State UniversityTempe, Arizona

Amanda N. Baraldi, MADepartment of PsychologyArizona State UniversityTempe, Arizona

Russell M. Bauer, PhDDepartment of Clinical and Health PsychologyUniversity of FloridaGainesville, Florida

John T. Behrens, PhDNetworking Academy & Corporate AffairsCiscoMishawaka, Indiana

Peter M. Bentler, PhDDepartment of PsychologyUniversity of CaliforniaLos Angeles, California

Bethany C. Bray, PhDDepartment of PsychologyVirginia Tech UniversityBlacksburg, Virginia

Randy Borum, PsyDDepartment of Mental Health Law & PolicyFlorida Mental Health InstituteUniversity of South FloridaTampa, Florida

Linda M. Collins, PhDThe Methodology CenterThe Pennsylvania State UniversityUniversity Park, Pennsylvania

Thomas D. Cook, PhDInstitute for Policy ResearchEvanston, Illinois

Geoff Cumming, DPhilSchool of Psychological ScienceLa Trobe UniversityMelbourne CampusVictoria, Australia

Patricio E. Cumsille, PhDEscuela de PsicologıaPontificia Universidad Catolica de Chile

Vicuna Mackenna 4860Macul, Santiago, Chile

Alfred DeMaris, PhDDepartment of SociologyBowling Green State UniversityBowling Green, Ohio

Sarah L. Desmarais, PhDDepartment of Mental Health

Law & PolicyFlorida Mental Health InstituteUniversity of South FloridaTampa, Florida

Kristen E. DiCerbo, PhDAvondale, Arizona

Kevin S. Douglas, PhD, LLBDepartment of PsychologySimon Fraser UniversityBurnaby, BC Canada

xxi

xxii Contributors

Callie Beck Dunn, MSDepartment of Clinical and Health PsychologyUniversity of FloridaGainesville, Florida

Elvira Elek-Fisk, PhDThe Methodology CenterPennsylvania State UniversityUniversity Park, Pennsylvania

Susan E. Embretson, PhDSchool of PsychologyGeorgia Institute of TechnologyAtlanta, Georgia

Fiona Fidler, PhDAustralian Centre of Excellence

for Risk Analysis (ACERA)The University of MelbourneVictoria, Australia

Suzanne E. Graham, EdDDepartment of EducationUniversity of New HampshireDurham, New Hampshire

John W. Graham, PhDDepartment of Biobehavioral HealthThe Pennsylvania State UniversityUniversity Park, Pennsylvania

Stephen C. Hirtle, PhDSchool of Information SciencesUniversity of PittsburghPittsburgh, Pennsylvania

James B. Hoelzle, PhDDepartment of PsychologyMarquette UniversityMilwaukee, Wisconsin

Roger E. Kirk, PhDDepartment of Psychology and NeuroscienceBaylor UniversityWaco, Texas

Stephanie T. Lanza, PhDThe Methodology CenterThe Pennsylvania State UniversityState College, Pennsylvania

Christina N. Lessov-Schlaggar, PhDDepartment of PsychiatryWashington UniversitySt. Louis, Missouri

Roy Levy, PhDSchool of Social and Family DynamicsArizona State UniversityTempe, Arizona

Linda J. Luecken, PhDDepartment of PsychologyArizona State UniversityTempe, Arizona

David P. MacKinnonDepartment of PsychologyArizona State UniversityTempe, Arizona

John J. McArdle, PhDDepartment of PsychologyThe University of Southern CaliforniaLos Angeles, California

Gregory J. Meyer, PhDDepartment of PsychologyUniversity of ToledoToledo, Ohio

Glenn W. Milligan, PhDFisher College of BusinessThe Ohio State UniversityColumbus, Ohio

Peter Molenaar, PhDThe Pennsylvania State UniversityUniversity Park, Pennsylvania

Leslie C. Morey, PhDDepartment of PsychologyTexas A&M UniversityCollege Station, Texas

John R. NesselroadeDepartment of PsychologyThe University of VirginiaCharlottesville, Virginia

Contributors xxiii

Randy K. Otto, PhDDepartment of Mental Health Law & PolicyFlorida Mental Health InstituteUniversity of South FloridaTampa, Florida

Steven C. Pitts, PhDDepartment of PsychologyUniversity of Maryland Baltimore CountyBaltimore, Maryland

Joseph S. Rossi, PhDCancer Prevention Research CenterUniversity of Rhode IslandKingston, Rhode Island

Karen M. Schmidt, PhDDepartment of PsychologyUniversity of VirginiaCharlottesville, Virginia

Frank Schmidt, PhDTippie College of BusinessUniversity of IowaIowa City, Iowa

Allison E. ShevockDover, Delaware

Marilyn C. Sines, PhDNaples, Florida

Judith D. Singer, PhDOffice of the ProvostHolyoke CenterHarvard UniversityCambridge, Massachusetts

Peter M. Steiner, PhDDepartment of Educational PsychologyUniversity of WisconsinMadison, Wisconsin

Gary E. Swan, PhDCenter for Health SciencesSRI InternationalMenlo Park, California

Rika TanakaDepartment of PsychologyArizona State UniversityTempe, Arizona

Abbas Tashakkori, PhDDepartment of Educational PsychologyUniversity of North TexasDenton, Texas

Charles Teddlie, PhDDepartment of Educational Theory,

Policy, and PracticeLouisiana State UniversityBaton Rouge, Louisiana

Davood Tofighi, PhDSchool of PsychologyGeorgia Institute of TechnologyAtlanta, Georgia

Jodie B. Ullman, PhDDepartment of PsychologyCalifornia State UniversitySan Bernardino, California

Jatin Vaidya, PhDDepartment of PsychiatryUniversity of IowaIowa City, Iowa

Wayne F. Velicer, PhDCancer Prevention Research CenterUniversity of Rhode IslandKingston, Rhode Island

David Watson, PhDDepartment of PsychologyUniversity of Notre DameIowa City, Iowa

Stephen G. West, PhDDepartment of PsychologyArizona State UniversityTempe, Arizona

xxiv Contributors

John B. Willett, PhDGraduate School of EducationHarvard UniversityCambridge, Massachusetts

Coady Wing, PhDSchool of Public HealthUniversity of Illinois at ChicagoChicago, Illinois

Vivian C. Wong, PhDInstitute for Policy ResearchNorthwestern UniversityEvanston, Illinois

Manyee Wong, PhDAmerican Institutes for ResearchChicago, Illinois

Ingrid C. WurptsDepartment of PsychologyArizona State UniversityTempe, Arizona

Nedim YelDivision of Educational Leadership

and InnovationArizona State UniversityTempe, Arizona

PART I

Foundations of Research Issues

CHAPTER 1

Experimental Design

ROGER E. KIRK

SOME BASIC EXPERIMENTAL DESIGNCONCEPTS 3

THREE BUILDING BLOCK DESIGNS 4CLASSIFICATION OF EXPERIMENTAL DESIGNS 11FACTORIAL DESIGNS 12

RANDOMIZED BLOCK FACTORIAL DESIGN 21FACTORIAL DESIGNS WITH CONFOUNDING 22HIERARCHICAL DESIGNS 28EXPERIMENTAL DESIGNS WITH A COVARIATE 30REFERENCES 32

SOME BASIC EXPERIMENTAL DESIGNCONCEPTS

Experimental design is concerned with the skillful inter-rogation of nature. Unfortunately, nature is reluctant toreveal her secrets. Joan Fisher Box (1978) observed inher biography of her famous father, Ronald A. Fisher,“Far from behaving consistently, however, Nature appearsvacillating, coy, and ambiguous in her answers” (p. 140).Nature’s most effective tool for confusing researchers isvariability—in particular, variability among subjects orexperimental units. Although nature can be duplicitous,Ronald A. Fisher showed that by comparing the vari-ability among subjects treated differently to the variabil-ity among subjects treated alike, researchers can makeinformed choices between competing hypotheses in sci-ence and technology.

We must never underestimate nature—she is a formi-dable foe. Carefully designed and executed experimentsare required to learn her secrets. An experimental designis a plan for assigning subjects to experimental conditionsand the statistical analysis associated with the plan (Kirk,2012, p. 1). The design of an experiment involves fiveinterrelated activities:

1. Formulation of statistical hypotheses that are germaneto the scientific hypothesis. A statistical hypothesis isa statement about (a) one or more parameters of apopulation or (b) the functional form of a population.Statistical hypotheses are rarely identical to scientifichypotheses—they are testable formulations of scien-tific hypotheses.

2. Determination of the experimental conditions (inde-pendent variable) to be manipulated, the measurement(dependent variable) to be recorded, and the extraneousconditions (nuisance variables) that must be controlled.

3. Specification of the number of subjects required andthe population from which they will be sampled.

4. Specification of the procedure for assigning the sub-jects to the experimental conditions.

5. Determination of the statistical analysis that will beperformed.

In short, an experimental design identifies the indepen-dent, dependent, and nuisance variables and indicates theway in which the randomization and statistical aspects ofan experiment are to be carried out.

Analysis of Variance

Analysis of variance (ANOVA) is a procedure for decom-posing the total variation displayed by a set of obser-vations into two or more identifiable sources of variation.The procedure enables researchers to interpret the variabil-ity in designed experiments. The seminal ideas for bothANOVA and experimental design can be traced to RonaldA. Fisher, a statistician, eugenicist, evolutionary biologist,and geneticist who worked at the Rothamsted Experi-mental Station that is 25 miles northwest of London.According to Box (1978, p. 100), Fisher developed thebasic ideas of ANOVA between 1919 and 1925. The firsthint of what was to come appeared in a 1918 paper inwhich Fisher partitioned the total variance of a human

3

Handbook of Psychology, Second Edition, edited by Irving B. Weiner. Copyright © 2013 John Wiley & Sons, Inc.

4 Foundations of Research Issues

attribute into portions attributed to heredity, environment,and other factors. The analysis of variance table for a two-treatment factorial design first appeared in a 1923 paperpublished with M. A. Mackenzie (Fisher & Mackenzie,1923). Fisher referred to the table as a convenient wayof arranging the arithmetic. In 1924, Fisher (1925) intro-duced the Latin square design in connection with a forestnursery experiment. The publication in 1925 of his classictextbook Statistical Methods for Research Workers and ashort paper the following year (Fisher, 1926) presentedall the essential ideas of analysis of variance. The text-book (Fisher, 1925, pp. 244–249) included a table of thecritical values of the ANOVA test statistic in terms of afunction called z , where z = 1/2(lnσ2

Treatment − ln σ2Error).

The statistics σ2Treatment and σ2

Error denote, respectively,treatment and error variance. A more convenient formof Fisher’s z table that did not require looking up logvalues was developed by George Snedecor (1934). Hiscritical values are expressed in terms of the function F =σ2

Treatment/σ2Error that is obtained directly from the ANOVA

calculations. He named it F in honor of Fisher. Fisher’sfield of experimentation—agriculture—was a fortunatechoice because results had immediate application withassessable economic value, because simplifying assump-tions such as normality and independence of errors wereusually tenable, and because the cost of conducting exper-iments was modest.

Three Principles of Good Experimental Design

The publication of Fisher’s Statistical Methods for Re-search Workers and his 1935 The Design of Experimentsgradually led to the acceptance of what today is consideredto be the cornerstone of good experimental design: ran-domization. It is hard to imagine the hostility that greetedthe suggestion that subjects or experimental units shouldbe randomly assigned to treatment levels. Before Fisher’swork, most researchers used systematic schemes, not sub-ject to the laws of chance, to assign subjects. According toFisher, random assignment has several purposes. It helps todistribute the idiosyncratic characteristics of subjects overthe treatment levels so that they do not selectively biasthe outcome of the experiment. Also, random assignmentpermits the computation of an unbiased estimate of erroreffects—those effects not attributable to the manipulationof the independent variable—and it helps to ensure thatthe error effects are statistically independent.

Fisher popularized two other principles of good experi-mentation: replication and local control or blocking. Repli-cation is the observation of two or more subjects under

identical experimental conditions. Fisher observed thatreplication enables a researcher to estimate error effectsand to obtain a more precise estimate of treatment effects.Blocking , on the other hand, is an experimental procedurefor isolating variation attributable to a nuisance variable.As the name suggests, nuisance variables are undesiredsources of variation that can affect the dependent variable.There are many sources of nuisance variation. Differ-ences among subjects comprise one source. Other sourcesinclude variation in the presentation of instructions to sub-jects, changes in environmental conditions, and the effectsof maturation, fatigue, and learning when subjects areobserved several times. Three experimental approachesare used to deal with nuisance variables:

1. Holding the variable constant.2. Assigning subjects randomly to the treatment levels so

that known and unsuspected sources of variation amongthe subjects are distributed over the entire experimentand do not affect the subjects in just one or a limitednumber of treatment levels.

3. Including the nuisance variable as one of the factors inthe experiment.

The third experimental approach uses local control orblocking to isolate variation attributable to the nuisancevariable so that it does not appear in estimates of treatmentand error effects. A statistical approach also can be usedto deal with nuisance variables. The approach is calledanalysis of covariance and is described in the last sectionof this chapter. The three principles that Fisher vigor-ously championed—randomization, replication, and localcontrol—remain the cornerstones of good experimentaldesign.

THREE BUILDING BLOCK DESIGNS

In this section I describe three simple analysis of variancedesigns that can be combined to form more complexdesigns. They are the completely randomized design, therandomized block design, and the Latin square design.I call these designs building block designs .

Completely Randomized Design

One of the simplest experimental designs is the random-ization and analysis plan that is used with a t statistic forindependent samples. Consider an experiment to comparethe effectiveness of two diets for obese teenagers. The