3 9 1NEL

CHapteR

14Mixed Methods ReseaRch

Perhaps the biggest surprise for those of us whose interests focus on research methods and their development is how much the research enterprise has changed in a relatively short period of time. Although there are many different areas in which those changes have occurred, a couple stand out for the two of us.

One is in the area of research infrastructure, where digital technologies in particular have had an overwhelmingly pervasive impact on virtually every aspect of how we do research. A mere 20 years ago, when the first edition of Research Decisions was published (see Palys 1992), for example, personal computers were still relatively new, the Internet did not yet exist in any publicly available form, and hence the search capabilities that we now take for granted—let alone all the digital content we now search for and the social network sites we are members of—also did not yet exist. While many researchers still seem stuck in 20th century technol-ogies and hence have been slow to consider how the digital revolution has opened up new possibilities for every aspect of the research process (see Palys & Atchison 2009, in press), in this edition of Research Decisions almost every chapter includes discussion of the ways the digital revolution is opening doors and providing new opportunities for the ways we can search for relevant literature, contact and recruit participants, and gather, manage, and ana-lyze information.

A second area of change has been in the bour-geoning embrace of greater methodological diversity in many quarters of the social and health sciences. In that regard, it is worth noting that the first edi-tion of Research Decisions was the first methods text we know of to actually consider qualitative and

quantitative approaches under the same cover and to extol the ways in which the two approaches could complement one another as contributing compon-ents of a larger multi-method strategy. Although there are many researchers who remain convinced their preferred approach is the one and only royal road to truth, most researchers (in our circles at least) seem to have put these “paradigm wars” behind them. We are encouraged to see more and more researchers seeking to broaden the range of methodological tools at their disposal and/or who seek to collaborate with others who bring different strategies and perspectives to the table in order to more effectively and comprehensively understand whatever phenomenon happens to be their focus. Although we have mentioned this propensity and the advantages it brings at various points in this book, we bring this edition of Research Decisions to a close by discussing mixed methods research in slightly greater detail.

A THIRD WAY?

By now you should be familiar with the differ-ences and similarities among and between various methodological approaches and have a fairly good understanding of their respective strengths and weaknesses. This is crucial for social and health researchers who wish to utilize two or more methods within a single project, a practice commonly referred to as mixed methods approach.

While social and health researchers have been mixing multiple methods of data collection within a single study for well over a century (Maxwell & Loomis 2003), Donald T. Campbell, with various collaborators, is widely regarded as one of the first

NEL-PALYS-12-0801-014.indd 391 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H3 9 2

NEL

to formally encourage their use in the more con-temporary context. Campbell and Fiske (1959), for example, argued that by using multiple methods (e.g., a paper-and-pencil test, direct observation, a performance measure) to measure multiple ‘traits’ (i.e., attributes or concepts), researchers could dra-matically improve the validity of their observations since they could establish that the measure and not the method was responsible for producing the observed results. They referred to this procedure as a multi-trait–multi-method matrix.

Unobtrusive Measures: Non-Reactive Research in the Social Sciences (Webb, Campbell, Schwartz, & Sechrest 1966) continued this argument by pointing out the benefits to be gained by encouraging and imple-menting methodological pluralism. In another classic article, Campbell (1969c) encouraged researchers of different disciplines to collaborate and enjoy the bene-fits that would accrue from the diversity of approaches they would bring, and his notion of the “experi-menting society” and quasi-experimentation was based in large part on the analytical power that could be mustered by strategically combining different types and sources of data (e.g., Campbell 1969b; Cook & Campbell 1979).

Denzin (1970, 1978) built upon these ideas when he coined the term triangulation, which is a research strategy that permits us to validate our observations by drawing upon multiple sources or perspectives within the same investigation. He sug-gested there were four distinct ways this triangula-tion could occur:

u Theoretical triangulation involves employing multiple theories throughout the design, col-lection, and analysis process. Proceeding in this manner would involve a researcher or group of researchers developing research questions from different theoretical vantage points and thereby studying a phenomenon through multiple lenses.

u Investigator triangulation refers to the practice of several different researchers contributing in the study to collect, analyze, and interpret data and observations. This practice is thought to

improve both the credibility of the observations and the resulting interpretation of the research. One place you see investigator triangulation is in the progressively more common practice of multi-, trans-, and inter-disciplinary research collaboration that brings together teams of researchers from different disciplines in order to research a problem of common interest (e.g., see Campbell 1969c; Leavy 2011).

u Methodological triangulation involves employing multiple methods to study a par-ticular phenomenon in order to overcome the deficiencies and biases that may result from employing a single method approach. Certainly Research Decisions, which extols the virtue of combining qualitative and quantitative approaches, exemplifies this approach.

u Data analysis triangulation refers to the prac-tice of employing several different methods of analyzing and interpreting data in order to improve the validity of the conclusions by ensuring the robustness of one’s results.*

These early writings by Campbell and his col-leagues in the more quantitative realm and Denzin in the qualitative set the stage for the emergence of what some (e.g., Johnson & Onwuegbuzie 2004; Johnson, Onwuegbuzie, & Turner 2007) have labelled a “third paradigm” of research—mixed methods—which Johnson et al. (2007) define as “the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of quali-tative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad pur-poses of breadth and depth of understanding and corroboration” (123).

While use of the term “paradigm” may paint a fairly simple and logical idea—the idea that two heads can be better than one—with an overly

* Denzin, N.K. (1970). The research act: A theoretical intro-duction to sociological methods. Chicago: Aldine. Denzin, N. K. (1978). The logic of naturalistic inquiry. In N. K. Denzin (Ed.), Sociological methods: A sourcebook. New York: McGraw-Hill

NEL-PALYS-12-0801-014.indd 392 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H 3 9 3

NEL

grandiose brush (see Kuhn 1970 for the origins of the term), at their base mixed methods approaches are premised on the belief that qualitative and quantitative methods occupy a shared terrain when it comes to objectives, scope, and nature of inquiry, and reflect the desire of many researchers to bridge the ‘divide’ that had existed between qualitative and quantitative approaches. Researchers adopting this perspective encourage an eclectic approach to the research process that draws upon the comple-mentary strengths of qualitative and quantita-tive techniques. Proponents believe that the best answers to any research problem come when we consider multiple viewpoints, perspectives, posi-tions, and standpoints instead of one.

One of the most exciting aspects of social and health research is the continued development of mixed methods approaches, which we view as an excellent addition to the research tool belt. However, since mixed methods research designs are more complex than single method projects there are a variety of practical, theoretical, and pro-cedural considerations that should be considered when deciding whether and how to employ them in your particular project. In the remainder of this chapter we will: (1) briefly discuss the philosoph-ical underpinnings of this approach to research; (2) examine the types of questions that mixed methods approaches are particularly well-suited to answer; and (3) assess the unique methodological consider-ations that researchers thinking of employing this approach need to keep in mind when designing, sampling, collecting, and analyzing the data from mixed methods investigations.

PHILOSOPHICAL UNDERPINNINGS

In Chapter 1 we introduced you to the epistemo-logical traditions of positivism and phenomenolo-gism and noted how social and health researchers working within these traditions have developed dif-ferent perspectives on what is knowable and how best to study social and health-related phenomena. We illustrated the different perspectives by highlighting what many see as the polar opposite positions of

direct, naive, or “classic” realism and constructivism. Recall that this extreme version of realism maintains there is a single reality that exists independent of the researcher that can be understood and discovered through the identification of the “right” theoretical concepts and the testing of these theoretical concepts using the appropriate—most often quantitative—empirical methods. At the other extreme are simi-larly dogmatic constructivists who believe that knowledge, truth, and reality are socially constructed and thus always in a state of change and never really “knowable” in a finite sense. They argue that the best we can hope for is to arrive at rich qualitative and human-centred description that allows us to under-stand the processes by which constructions arise and the ways they can be changed.

The distinct differences between these visions of how to define and study truth, knowledge, and reality led many social and health researchers working within each tradition to become increas-ingly rigid about their preference for, and defence of, either quantitative or qualitative methods. Over time, perspective and method became conflated so that realism became inseparable from quantitative approaches and constructivism became inseparable from qualitative approaches. During the 1970s and 1980s an increasing number of people working within each paradigm began to publish books and articles championing the virtues of their per-spective/method while simultaneously calling into question the validity of research conducted within the ‘opposing’ perspective/method. The realist/quantitative–constructivist/qualitative divide even-tually became so pronounced that some observers began to refer to them as the “paradigm wars” or “science wars” (e.g., Ross 1996).

While the “paradigm wars” appear for the most part to have come to an end, there are still some who cling to the belief that fundamental differences in the philosophical underpinnings of qualitative and quantitative research paradigms make the mixing of the two approaches impossible, a position known as the incompatibility thesis (Howe 1988). Researchers who are open to mixed methods approaches reject this claim. They feel

NEL-PALYS-12-0801-014.indd 393 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H3 9 4

NEL

that social and health-related phenomena are better studied using both qualitative and quantitative methods and have sought to ground their approach in an epistemological position that is capable of seeing a middle ground between direct realism and constructivism.

Enter pragmatism, which is a philosophical tradition that has its roots in the late 19th cen-tury through the works of Charles Saders Pierce, John Dewy, and William James and played a major role in the emergence of symbolic interactionism (e.g., Mead and Cooley). Pragmatism is not com-mitted to any single system of philosophy or view of reality. The central position advanced within pragmatism is the rejection of traditional dualisms of realism versus constructivism, free-will versus determinism, subjectivism versus objectivism, and induction versus deduction in favour of taking the position that works best in a particular situation (Johnson & Onwuegbuzie 2004).

Pragmatism is founded on a method of inquiry that is based on an iterative relationship between the processes of discovery and action as opposed to the search for a single truth or correct answer. Pragmatists favour eclecticism and pluralism as opposed to dogmatism when it comes to theor-etical, methodological, and analytical approaches to understanding the social world. Pragmatists are results- or outcome-oriented and less concerned with prior knowledges, laws, or rules governing what is to be considered valid knowledge (Maxcy 2003). They are concerned with finding the best or most complete answers to research ques-tions through the best method or combination of methods and they have a strong commitment to praxis (i.e., theory informing practice).

MIXED METHODS RESEARCH QUESTIONS AND OBJECTIVES

Many mixed methods investigations are often simultaneously inductive and deductive, so the types of questions that researchers working within this perspective ask tend to be more layered, nuanced and comprehensive than those that inspire

single-method studies. The questions motivating many mixed methods investigations are often con-cerned with both variance and process. As Maxwell and Loomis (2003) explain, variance questions frequently involve describing the frequency with which phenomena of interest occur, and often use aggregate statistics to explain and understand the relationships or predictive capacity of one or more variables or theories related to these phe-nomena. In contrast, process questions are directed at exploring, describing, understanding, and/or explaining the origins, meanings, and relationships connected to particular events, phenomena, or processes—how they occur and what they mean to those involved.

For example, in his research with sex buyers, Chris sought to describe and understand how men understood and defined the risks associated with their decisions to purchase sex and to explain how the attitudes and beliefs that these men held about the meanings of risk was related to their sex buying behaviour and the relationships they formed with sex workers. His research questions were both inductive and deductive as he sought a deeper understanding of how risk is defined and under-stood by men who buy sex, and to explain how variations in the way different sex buyers defined and understood risk resulted in different behav-ioural outcomes for them and the sex workers. It was clear to Chris that a single-method research design would not be suitable for finding answers to his questions.

Similarly, when Ted and his colleagues sought to understand how police officers in a large urban police department used a state-of-the-art mobile data access system and the impacts it had, their research crossed all the dualisms we have discussed. They wanted quantitative measures of frequency of system use, the circumstances in which it was used, and any outcomes that directly arose from these searches, but at the same time sought to understand officers’ perceptions of the system and the meaning it had for the officers and their views of themselves and their job. The project would incorporate both quantitative and qualitative data,

NEL-PALYS-12-0801-014.indd 394 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H 3 9 5

NEL

include both behavioural and perceptual measures, deductively address their own theoretical interests while looking inductively at the way officers and the police administration understood the system, and look at how attitudes were distributed among patrol officers and relevant others while also trying to understand the basis of those attitudes and their implications for policing. This called for a mixed methods approach to engender a more comprehen-sive and balanced understanding of the system.

DESIGN ISSUES IN MIXED METHODS RESEARCH

Researchers employing a mixed methods paradigm do so because they believe that bringing together different methodological approaches will give them the best chance of finding comprehensive answers to their specific and multifaceted research questions. Accordingly, while research questions form the foundation upon which mixed methods investigations are built, different projects with their differing objectives and priorities will influence the way the various methods are brought together within the project, although in our experience the way this happens will in most cases involve a mix-ture of strategic choice and happenstance.

As we have seen throughout the book, a solid research design requires that you pay attention to the relationships among and between various com-ponents of your research, and this is particularly so for mixed methods designs. Ideally all of the components will be compatible and “speak” to and complement each other in order to gain the com-prehensive understanding being sought. One aspect that may vary is the order in which the different methods are engaged.

Order

Authors in the field suggest there are only two options to choose from—concurrent or sequential ordering (e.g., Morse 1991). However, in our experi-ence the distinction between the two is often more easily made in textbooks than it is in the field, where

pragmatic considerations and project-specific cir-cumstances invariably intervene. We also acknow-ledge that a priori distinctions between concurrent or sequential strategies are more common in more structured projects that can be proposed completely ahead of time, while more intentionally inductive and emergent strategies often are less concerned with establishing order ahead of time and prefer to enjoy a more ad hoc approach where different sources are sought and embraced as one’s under-standing of a phenomenon develops and different sources of information are identified. Although the description below maintains this distinction—with our addition of an “ad hoc” category—the main principle you should keep in mind is that the job of the researcher is to bring together all the relevant data one can muster, and not to feel that some source of information is somehow illegiti mate or off limits because it was not anticipated at the time you were writing the proposal. The best researchers are data hungry—always looking for new informa-tion and new sources of information that will offer a different perspective and more comprehensive understanding of whatever phenomenon or process one is trying to understand.

ConCurrent Mixed Method designs

Concurrent mixed method designs are said to occur when both qualitative and quantitative data collection occurs at the same time. This approach most frequently occurs with more deductive approaches where researchers bring their theories and/or interests to bear on a site of their choosing, such that one begins with research questions and data sources that are identifiable from the outset. Data-gathering efficiency in such situations may see different members of a coordinated research team working in parallel, each with their separate area of responsibility, and adding their respective contributions to the whole. For example, Chris’s research with sex buyers was actually one part of a broader research project that involved other health researchers examining how notions of risk played out in various locations within the sex industry. Thus, while Chris was seeking out both qualitative

NEL-PALYS-12-0801-014.indd 395 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H3 9 6

NEL

and quantitative information from sex buyers regarding their perceptions of the different types of risk their sex buying entailed, others in the team were working with sex workers to better understand these interactions and their consequences from their perspective; a second group was gathering informa-tion from the romantic partners of sex workers; a third was focusing on those who “manage” sex workers in venues such as massage parlours, escort agencies, and exotic dance establishments; and a fourth was focusing on the various laws, policies, and legislation that municipal police use to police the sex industry, and the impact of these practices on sex worker health and safety.1

Concurrent data acquisition can occur within a specific study as well. For example, in Ted’s evalua-tion of a police department’s mobile data system, one part of the study involved doing ridealongs with patrol officers during actual work shifts in order to see firsthand how they used the system. This immediately raised the question of what effect observers’ presence in the patrol cars would have on the way officers used the system. Would they use it more than usual in order to impress the researchers with how useful it was? Would they use it less because of some concern about how their use of it would be viewed? Or might they simply use it differently—using some aspects of it and not others—depending on how the researchers or their employers might view their use?

The use of mixed methods allowed Ted to address that issue head on. The mobile data system incorporated eight different “forms” that involved different ways of using the system to access dif-ferent data sources, and one of the pieces of infor-mation captured during the observational sessions was how frequently each one of the “forms” was used. However, the system itself also maintained an ongoing archive of every query that was made in all patrol cars at all times, such that access to this archive would allow the research team to see how frequently each one of the eight forms was used when observers were not present, thereby enabling a comparison with the data that were gathered when they were present. By examining these two

concurrent sets of data, the researchers were able to establish that, with one exception, officers’ overall use of the system when observers were in their patrol cars mirrored their use of the system when no observers were present. The one exception was a “comment” form—rather like an early version of email that allowed officers to send typed mes-sages between patrol cars—that was used far less frequently when observers were present than when they were not. Although comforting to know, one limitation to that particular comparison was that, while the researchers could see whether and what differences in overall use did or did not appear, mere examination of the distributions would give no indication of why any differences appeared. Nonetheless, by identifying the issue, another research question arose that, not coincidentally, could be addressed by yet another method.2

sequential Mixed Method designs

Sequential mixed method designs break the data collection process into stages that follow one another in some logically considered order. This approach is most commonly employed by having a primary data collection strategy followed by col-lection of supplementary information during a sec-ondary stage that allows you to clarify aspects of the data during analysis. It also happens to be the one that Chris employed in his study of sex buyers. As he explained in his original proposal to the funding agency,

A mixed method strategy will unfold sequentially in two phases. … The first phase will involve a structured self-administered questionnaire administered via the Internet and the project web site. The second phase will involve in-depth conversational interviews either in-person, on the telephone or online. The combined methods will facilitate a phenomenological investigation of individual attitudes, beliefs and experiences, with a broader investigation of the aggregate patterns of experience and behaviour. They will also provide greater depth and breadth of infor-mation than would be available through a single

NEL-PALYS-12-0801-014.indd 396 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H 3 97

NEL

method approach, and they will enhance our ability to posit a wider range of answers to our research questions.*

Starting with the structured survey was a rea-sonable thing to do in this case because the main issues he wanted to address were identified ahead of time, and the focus of that phase of the research was largely to generate enough information through a systematic and structured method that would allow him to aggregate the information to create a “big picture” view of a diverse sample of sex buyers and begin to examine statistically the relations among different variables of interest. As he outlined in his proposal,

We will use a revised version of Atchison, Lowman and Fraser’s (1998) instrument to col-lect data. The instrument consists of a series of open and close-ended questions that measure sex buyers’ attitudes, beliefs and experiences. The questionnaire is organized into a series of the-matic categories including: general and specific purchasing experience and behaviours; attitudes and understanding of the regulation and struc-turing of the sex industry; attitudes toward and understandings of sex work; and, personal back-ground and history.†

Notwithstanding the inclusion of some open-ended questions that gave respondents the oppor-tunity to explain and comment at greater length, for the most part the items in the survey were highly structured categorical items that were useful for sta-tistical analysis. Their weakness is that categorical, structured items tend not to be all that informa-tive as to the reasoning or logic that lay behind the choices those categories signify. For that reason, Chris also encouraged those respondents who were

willing to engage in a follow-up interview that would allow him to explore in much greater detail buyers’ perceptions of their interactions with sex workers. As he explained in his research proposal,

We use Plummer’s framework of sexual story-telling and Herdt and Stoller’s concept of intimate communications to frame the inter-view. This orientation will lead to a more open discussion and ensure that participants will have the opportunity to define and deliberate on their experiences, concerns, and insights. Respondents will provide personal narratives relating to their positive and negative experiences of paying for sex and with violence, victimiza-tion and safety within the context of commer-cial sex relationships. By using this more open format to the conversation, we will avoid being overly constrained by previous research, and avoid preconceived—perhaps even stereotypical—images of sex buyers.‡

The study that Ted did with the urban police department and their use of their mobile data system also employed a sequential strategy whose ordering went in the opposite direction. In Ted’s study, because no one had ever done a systematic evaluation of that sort of system before, the major emphasis initially was on simply watching patrol officers use the system on actual working shifts; gathering archival information from policing “trade” magazines that talked about systems of this type and the hopes and concerns that police officers expressed about them; and using targeted samples of patrol officers, managerial level officers, and radio dispatch personnel to try to ascertain the range of attitudes, and the reasons underlying those attitudes, which patrol officers and others in the police department held about the system. But while these interviews were rich with detail and originated from targeted samples of individuals who collectively represented a useful cross-section of the police department

* Atchison, C. (1998). Men who buy sex: A Preliminary de-scription based on the results from a survey of the Internet-using population. M.A. thesis,. Simon Fraser University

† Atchison, C. (1998). Men who buy sex: A Preliminary de-scription based on the results from a survey of the Internet-using population. M.A. thesis,. Simon Fraser University

‡ Atchison, C. (1998). Men who buy sex: A Preliminary de-scription based on the results from a survey of the Internet-using population. M.A. thesis,. Simon Fraser University

NEL-PALYS-12-0801-014.indd 397 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H3 9 8

NEL

being studied, their main limitation was that the researchers were left with no idea how commonly the different views about the system were held, or how opinions or practices in one sphere were related to any other. That was exactly the strength that using a structured survey—eventually administered to a 20 percent sample of all patrol officers in the department—would afford, with all the preliminary exploratory work helping ensure that the survey that was created would address issues that were estab-lished to be important, and that would reflect and address officers’ beliefs and behaviour in relation to the system.

As these two examples suggest, there is no inherently “right” way to proceed through a multi-method strategy. Which sequence is “best” will depend on many factors such as the amount of prior research in the area, the extent to which the researcher is following a more inductive, deductive, or combined strategy; and/or simply the pragmatics of the situation.

integrative Mixed Method designs

Although our discussion of concurrent and sequen-tial designs depicts an orderly and tidy process of a considered and formally executed design, you also should see hints in the descriptions that suggest it does not always happen that way. In particular, two deviations from those elegant and planned procedures should be noted, which we refer to as systematic mixed method integration and ad hoc mixed method integration.

SyStematic mixed method integration Do not come away from the above feeling that you must make a decision between pursuing either a concurrent or a sequential strategy. There is no such requirement. Many multi-method studies incorporate elements of both, and the two pro-jects we describe above are examples of exactly that. In the sex work research project that Chris was involved with, a concurrent strategy was employed insofar as different researchers were gathering data simultaneously from dif-ferent sources using a variety of methods, while

individual researchers, such as Chris, were also employing sequential strategies within their areas of responsibility.

The same mixture of course also can occur within any given study, as was evident in Ted’s study of the police department’s mobile data system. In that research one phase saw several different methods being used concurrently—the in-car observation, archival analysis of the fre-quency with which different forms were being used, and in-person interviews with a targeted sample of respondents—which were then followed sequentially by a survey that addressed all the rele-vant issues that had been identified in the earlier phase of that research.

ad hoc mixed method integration The second way that multi-method integration can occur arises from the facts that life involves surprises and does not always proceed as we expect, coupled with our admonition that the main principle in any multi-method study or program of research should be to bring together all the relevant data sources you can. Ted’s experience with the mobile data system study is an example of that, as he did not know when that research began that an archive existed of every transaction made over the system. It was only during a preliminary phase of the research that the existence of the archive was noted by one of the computer programmers during casual conversa-tion, with Ted immediately recognizing its signifi-cance and the role it could play in helping to deal with one particular rival plausible explanation that would arise in the study.

And finally, no discussion of integrative multi-method strategies would be complete without also paying homage to the ad hoc discovery of data sources that is almost inevitable in any field-based research, and is particularly characteristic of more ethnographic or inductively guided qualitative research. Many of the examples we have cited in this text offer examples of that. Recall, for example, the way that Howard Becker’s (1993) study of medical students bounced back and forth between interviewing and observation and followed what proved an interesting diversion when one student

NEL-PALYS-12-0801-014.indd 398 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H 3 9 9

NEL

unexpectedly referred to one of the patients he saw on rounds as a “crock.” Similarly, many inter-view, oral history, and ethnographic studies end up incorporating methods and sources that were not originally planned or foreseen, such as when an interviewee makes note of a diary that was kept during a particular period in their life that we are asking about, or makes mention of the many pic-tures that were taken during a holiday that we are asking about, or starts pulling out newspaper clip-pings that reveal how media portrayed a particular episode at the time it was unfolding (see Margolis 1994 for an excellent example of this sort).

SAMPLING ISSUES IN MIXED METHODS RESEARCH

Sampling in mixed methods designs involves con-sidered and often complex mixtures of quantitative and qualitative methodologies relying upon some combination of probabilistic and purposive sam-pling strategies. As with single-method designs, the type of sampling procedure that is most appropriate is largely determined by your research objectives. If you want to sample from a well-defined popula-tion in a manner that will allow you to generalize your results back to that wider population, as is often the case in large-scale descriptive, relational, and explanatory studies, a probabilistic sampling technique is beneficial. If your goal is exploring, describing, or understanding underlying processes in order to identify key aspects of a phenomenon in order to generate new ideas and theories, or you are simply engaging in research where particular identifiable individuals are the best sources for the information that you need—because of their unique experience, or position in an organization, or whatever—then a purposive and targeted sam-pling technique will be more appropriate.

Multifaceted research questions frequently require that you use multiple and diverse (prob-ability and purposive) sampling techniques to select the people, places, and/or things that are best suited for providing the information necessary to answer your questions. No matter what sampling

technique(s) you use, you should keep in mind that overlapping sampling strategies in mixed methods designs normally will need to be congruent. While there is generally no problem with mixing multiple purposive sampling techniques, you often need to be careful when combining probability and pur-posive techniques within the same investigation.

For example, in most situations it wouldn’t make much sense to draw a large convenience sample of participants to complete a self-administered ques-tionnaire and then follow this up by randomly sam-pling the people who completed the questionnaire to ask them to participate in an in-depth interview. The opposite, however, may well make a lot of sense, e.g., when you begin with a random sample of individuals from some well-defined population in order to see how certain attitudes are distributed in the group as a whole, and then follow this with targeted samples of individuals who represent dif-ferent constituencies or perspectives within that whole.

For example, during Chris’s research with sex buyers he combined network, purposive and quota sampling strategies in order to acquire a sample of Canadian sex buyers to participate in either an in-depth interview or questionnaire. Close to 50 people contacted him after hearing about his research from people in their social network (e.g., family, friends, co-workers, sex workers, other sex buyers). He then used a pre-interview screening process to collect information about each individ-ual’s sexual background and experience at paying for sex. Individuals who had a lengthy history of sex buying and who represented particular demo-graphics of buyers were selected and asked to par-ticipate in an in-depth interview. Those that did not fit the specifications were directed to the research website and asked to complete the browser-based questionnaire.

Sequential design sampling strategies have the potential to dramatically improve sample selection and solicitation as the technique used in one stage of the research can be used to assist you in identi-fying and/or contacting the sample for the second stage of the project. This can mean including

NEL-PALYS-12-0801-014.indd 399 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H4 0 0

NEL

the participant in both stages of the research or redirecting them to stages that might be more appropriate for them. For example, if you admin-ister a questionnaire via Web browser in a net-work environment you can use scripts to conduct rudimentary data analysis on your participant’s responses while they are completing the survey and when specific types of participants (e.g., extreme cases, deviant cases, theoretically important cases) are identified on the basis of their responses to particular items or sets of items in the question-naire, they could be presented with a dialogue box, pop-up window, or follow-up email asking them to participate in an interview or focus group.

MIXING METHODS OF DATA COLLECTION AND OBSERVATION

As we have seen throughout the book, there are a wide variety of qualitative and quantitative observation and data collection techniques avail-able to social and health researchers. Some of the quantitative methods we have discussed include questionnaires, structured interviews, structured observation, archival techniques, experiments, and program and policy evaluation. Some of the qualita-tive strategies we have reviewed include interviews, life and oral histories, case studies, focus groups, field groups, observational studies, ethnographies, archival or document discovery, and program or policy evaluations. We have encouraged you to see these different data collection and observation strategies as tools in your research tool belt with some tools being better suited for certain jobs than others. Remember that in order to determine which tool is best suited for providing the best answers to your particular research question(s), you need to be familiar with the strengths and weaknesses of each. Having a solid understanding of the relative strengths and weaknesses of the various methods is a vital component of mixed methods research.

Tashakkori and Teddlie (1998) assert that the fundamental principle of mixed methods research is that methods should be mixed in order to emphasize the complementary strengths

and non-overlapping weaknesses of the methods. Ideally, a mixed methods design should combine methods in a way that allows you to overcome the weaknesses or limitations of one method by drawing upon the strengths of another. The weak-nesses of a particular data collection approach may be in the type, depth, or breadth of data that it can provide. For example, while questionnaires are an excellent method for quickly and efficiently col-lecting large amounts of data about the relatively static aspects of social and health phenomena, they are a notoriously weak method for acquiring infor-mation about context. Similarly, while observa-tional strategies are excellent ways to develop a rich description of context within which attitudes and beliefs are formed and behaviours are displayed, researchers employing these strategies often have no idea where to focus their observations. A sequential mixed methods design that integrates question-naire and observational strategies would allow you to capture both breadth and depth of description and understanding by capitalizing on the unique strengths of each method.

In addition to helping to neutralize the effects of the inherent weaknesses of any single method, you can use mixed methods observation and data col-lection strategies to help you develop more reliable and valid instruments. For example, Chris recently became involved in another large interdisciplinary mixed methods study looking at the intersection of gender, conflict, and health within Canada’s sex industry. While concepts such as gender, conflict, and health have been defined and measured in a variety of ways, few studies have been done to deter-mine if there are particular definitions and mea-sures that are more valid and reliable than others. In order to determine if there are indeed “better” ways of measuring these concepts, Chris and his research partners have decided to conduct a multi-stage sequential mixed methods investigation. The first stage involves conducting a series of individual and focus group interviews with various stakeholders (e.g., buyers and sellers of sex, healthcare providers, police, social support service providers) in order to identify what each concept means to them and

NEL-PALYS-12-0801-014.indd 400 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H 4 0 1

NEL

how it impacts their experiences in and around the sex industry. The analysis of the interview data will allow the team to develop a preliminary pool of qualitative and quantitative measures they will then evaluate through a series of tests during the second stage of the study. Once the measures have been thoroughly tested and evaluated in various contexts the team will be in the position to identify which measures will be used in the third and final stage of the research.

Discussions about the integration of methods during data collection and observation often focus on whether a researcher is going to con-duct the research sequentially or concurrently. In both cases, the assumption is often that the researcher is relying upon two independent data collection strategies. With some of the advances in computer-assisted observation and data collection techniques we envision a possibility where ele-ments of two different methods can be seamlessly integrated or nested within a single data collection protocol to produce a truly mixed method data collection strategy. For example, using advanced scripting techniques it is well within the realm of possibility to create a data collection strategy that is a hybrid of the browser-based self- administered questionnaire and the conversational inter-view. As another example, virtual worlds and online gaming environments are ideal spaces in which to integrate quasi-experimental and micro- ethnographic observation seamlessly. The possibil ities that network technology offer in this respect are limitless.

While mixing two or more data collection strategies within a single investigation can offer unique advantages, it is important to acknow-ledge that doing this can also sometimes result in poorer quality data and observations. There are two main situations where the quality of the data could be compromised. The first occurs when a sort of “shotgun” logic prevails that seems governed by the idea that, with so many pellets heading off in the same direction, surely one will hit the target. In methodological terms, this is often manifest in the researcher applying one or

all of the methods less rigorously than would have been the case had s/he opted for a well-aimed, single-method study. Simply throwing all your darts at the board in the hope one will hit the bull’s-eye is not a recommended strategy; more often than not they all will miss.

The second situation is when you develop the secondary method to “fit” the format and content of the primary method. In this case the secondary method is not used in a way that the research bene-fits from its unique strengths and the resulting data will likely be severely compromised. For example, it is not unusual for social and health researchers con-ducting mixed methods research involving ques-tionnaire and interview approaches to simply turn the interview into a researcher-administered version of the questionnaire. In this situation instead of producing rich and nuanced information about the processes underlying the phenomenon under inves-tigation the interviews produce rigid and structured responses that are incapable of revealing the infor-mation the researcher would need to answer their qualitative research questions.

ANALYTICAL ISSUES IN MIXED METHODS RESEARCH

All mixed methods investigations require that you bring together the data and observations for your analysis and interpretation. One of the greatest benefits of mixed methods designs comes at the data analysis stage when the different types of data can be brought together and used to shed light on each other. Data from mixed methods investiga-tions are capable of providing you with a much greater diversity of divergent views (Teddlie & Tashakkori 2003), and when divergent views are found it gives you the opportunity to falsify aspects of your theory or to question the assumptions upon which your understanding of a phenomenon are based. Mixed methods analysis allows you to have greater confidence in your research since ideally you will analyze the data in a way that allows you to interrogate more thoroughly the plausibility of your conclusions.

NEL-PALYS-12-0801-014.indd 401 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H4 0 2

NEL

The major issue confronting most researchers is whether to transform one type of data so that it is directly comparable and compatible with the other or to maintain the integrity of each form of data and develop analytical and interpretive tech-niques that effectively integrate both forms. Since quantitative data are already structured and cannot easily be transformed to become less structured, one of the toughest decisions you have to make during analysis is how you are going to handle your qualitative data. Transforming qualitative data and observations into a more quantitative form by assigning elements of talk, text, or image into static and structured categories can defeat the purpose of collecting the rich qualitative data in the first place. Alternatively, ensuring different types of data “speak” to one another can be very challenging.

At this point there are no analytical techniques we are aware of that have been developed specif-ically for the simultaneous analysis of qualitative and quantitative data, and there is no single data analysis program that is capable of analyzing both numeric and non-numeric data. While both SPSS and NVivo have the capacity to handle mixed forms of data, neither product offers a complete range of options for analyzing mixed data. Having said this, SPSS can be used to conduct rudimentary analysis of text or “string” data, and datasets can be imported into NVivo for simple analysis and coded and linked to qualitative data. Perhaps with an increasing emphasis on ‘convergence’ in the technological world we will see this type of integra-tion at some point down the road and SPSS and NVivo will be merged into a single program. For now, sophisticated analyses of mixed data require the researcher to use at least two different programs and then integrate their analysis during the inter-pretive phase.

SUMMING UP AND LOOKING AHEAD

Johnson and Onwuegbuzie (2004) aptly point out that the epistemological position that a researcher holds does not dictate the methodological and analytical approaches that they must use in order

to justify their knowledge claims. Yet there are still many who believe that quantitative and qualita-tive methodological approaches are fundamen-tally incompatible—or that only one of the two embodies the qualities of “real” science while the other is hopelessly esoteric and misguided—and that researchers who attempt to combine them in a single project are destined for failure (e.g., compare Gendreau & Bonta 1991 with Smith & Heshusius 1986). We feel that this view of research is an incredibly limiting way of looking at the methodological possibilities for social and health research.

Throughout the book we have urged you to place your research question(s) at the centre of all of your design, sampling and solicitation, data col-lection, and observation and analysis decisions. We have argued that all methodological and analytic decisions follow from your research questions and that your research decisions should be mindful, not mindless, dogmatic or rote. This is also a central tenet of the pragmatic philosophy that underlies mixed methods research.

The decision to undertake a mixed methods investigation in any particular project is not one that you should make without due considera-tion. While there are obvious advantages to mixed methods research designs, researchers contem-plating these types of designs should assess critic-ally whether they are the most appropriate for achieving their specific research objectives. Mixed methods designs are not always your best option; they can be difficult to manage, time consuming, costly, and require that the researcher or research team be thoroughly familiar with the strengths and weaknesses of both qualitative and quantitative methods. Researchers always need to ask themselves if their specific questions could be better answered through a design that relies solely on qualitative or quantitative methods.

In any lengthier program of research, however, we would consider anyone who avoids multi-method inquiry (whether by themselves or in com-bination with others) as something of a one-trick pony that does an injustice to the phenomena that

NEL-PALYS-12-0801-014.indd 402 13/10/12 2:47 PM

C H a p t e R 1 4 – M i x e d M e t H o d s R e s e a R C H 4 0 3

NEL

interest them. For example, we do not understand researchers who engage in statistical analysis of sex offender data in study after study, and make recommendations for policy, without ever having actually spoken to a sex offender. We are equally perplexed when we meet those who engage only in ethnographic or interview-based research and ignore the results of more quantitative studies that would help place their more focused experience in some broader context. What we particularly appreciate about multi-method research is the extent to which it is consistent with a theme we have returned to throughout this book, i.e., that it is one’s research questions that should guide inquiry—not some epistemological orthodoxy that happens to be the one thing one learned in graduate school—and that, in the process of addressing those questions, one should leave no stone unturned.

STUDY QUESTIONS

1. What four modes of triangulation were artic-ulated by Denzin (1970)?

2. What do the “paradigm wars” refer to?

3. What is the “incompatibility thesis” and how is it viewed by those who engage in mixed method research?

4. In what sense do mixed methods approaches reflect the philosophical tradition known as pragmatism?

5. Explain how “variance questions” and “pro-cess questions” can be viewed as complemen-tary. Which is more allied with qualitative approaches and which is more aligned with quantitative?

6. Explain the difference between concurrent, sequential, and integrative multi-method designs.

7. In what sense might ad hoc integration be considered synonymous with qualitative approaches?

8. Do mixed methods strategies favour proba-bilistic or purposive sampling strategies or “all of the above”? Explain.

9. “When engaged in mixed method research involving surveys and interview methods, the more qualitative interview techniques should always come first.” Would you agree with that statement? Explain.

10. In general, the chapter extols the virtues of mixed method approaches because of the enhanced information it produces. But are there circumstances in which it might actu-ally act to diminish data quality? Explain.

11. How do mixed methods data assist you at the analytical/interpretive stage of your project? What obstacles currently exist to a more com-plete integration at that stage of your research?

NOTES

1. The project was actually even more exten-sive than this. We identify only a few of the concurrent projects here just to give a feel for some of the diversity of sources and methods that were involved.

2. During casual conversation that in-car observers had with officers and in some of the interviews that were conducted, several officers revealed that they had been told by senior officers to minimize their “frivolous” use of the system during the evaluation. The officers told us that they often used the “com-ment” form to trade off-colour jokes or make other comments that they would not want others to hear, and were avoiding doing so when the observers were present.

NEL-PALYS-12-0801-014.indd 403 13/10/12 2:47 PM

NEL-PALYS-12-0801-014.indd 404 13/10/12 2:47 PM