A qualitative and comparative study on eLearning in Germany and ...
Introduction to Qualitative Comparative Analysis · S 28-03-2017 | 1 28-03-2017 | 1 Introduction to...
Transcript of Introduction to Qualitative Comparative Analysis · S 28-03-2017 | 1 28-03-2017 | 1 Introduction to...
1 | 28-03-2017
S
1 | 28-03-2017
Introduction to Qualitative Comparative Analysis (QCA)
Invited Presentation for Regioplan | Tuesday, March 28th 2017 | Location: Amsterdam School of Real Estate
2 | 28-03-2017
3 | 28-03-2017
The classical idea of evaluation
beleidseffect
effectniveau na beleid
effectniveau
voor beleid
effect verandering
effectvariable
t
effect met programma
effect zonder programma
VOOR
beleidsinitiatief
TIJDENS
beleidsinitiatief
NA
beleidsinitiatief
Source: Pattyn & Verweij (2014)
4 | 28-03-2017
What is the problem?
› Causality is more complex than “X Y”
Context is not stable but changes occur
Policies interact with other factors (e.g., other policies)
› Okay, in the classical approach we may be measuring the effect of a policy…
…but what are the mechanisms that are activated?
…and under which conditions does the policy work or not?
Source: Verweij & Gerrits (2013)
5 | 28-03-2017
How about case studies then?
Context Interactions between factors Mechanisms Conditions
6 | 28-03-2017
The problem with case studies
› When you study a case, you can show and take into account the complexity, but:
Everything seems to interact with everything… What are the necessary elements for the policy?
What are the relevant context factors for the policy? Which conditions really matter?
› How do we learn from a single case for another case? If every case is unique (cf. case study), how can lessons learned be transferred?
7 | 28-03-2017
Source: Verweij et al. (2013)
[C*I*M] + [~C*I*M] O [C and I and M] or [~C and I and M] O
The truth table is central in QCA
8 | 28-03-2017
Today’s objectives & program
› We will not aim to understand all the techniques and nuances of QCA
› We will focus on:
1) What is QCA, and when and why is using QCA a good idea?
2) How do we get to the truth table?
3) How do we analyze it?
4) How do we interpret the results?
9 | 28-03-2017
QCA: what, when and why?
› “QCA is both a research approach and a data analysis technique”…
› Qualitative comparative analysis as an approach is…
Case-based/oriented
Comparative
Set-theoretic
Source: Schneider & Wagemann (2012)
How do you conduct a QCA?
How do you design a QCA?
10 | 28-03-2017
QCA: what, when and why?
› Qualitative comparative analysis as an approach is…
Case-based/oriented
Comparative
Set-theoretic It strives to “gather in-
depth insight in the different cases and
capturing the complexity of the cases”
Source: Rihoux & Lobe (2009)
11 | 28-03-2017
QCA: what, when and why?
› Qualitative comparative analysis as an approach is…
Case-based/oriented
Comparative
Set-theoretic It strives to “gather in-
depth insight in the different cases and
capturing the complexity of the cases”
… and to “produce some level of generalization”
Source: Rihoux & Lobe (2009)
12 | 28-03-2017
QCA: what, when and why?
› Qualitative comparative analysis as an approach is…
Case-based/oriented
Comparative
Set-theoretic
Source: Ragin (1994)
Many
Qualitative
Research
Comparative
Aspects of Cases Research
Quantitative
Research
Few
Few Number of Cases Many
In QCA, aspects of cases are
called “conditions”
and understood as “sets”
13 | 28-03-2017
QCA: what, when and why?
Quantitative
Typical method:
regression analysis
QCA
Qualitative
Typical method: single case study
Linear causality Complex causality Complex causality
or non-causal (‘interpretive’)
Variable-based Case-based and
comparative Case-based
Large-N of cases Medium-N of cases Small-N of cases
Pattern recognition,
generalization (small number of variables)
Between generality and complexity
(medium number of conditions)
High level of detail, in-depth case insights
(large number of ‘variables’)
Adapted from: Verweij & Gerrits (2013)
14 | 28-03-2017
› Qualitative comparative analysis as an approach is…
Case-based/oriented
Comparative
Set-theoretic
QCA: what, when and why?
Source: Rihoux et al. (2013)
Advise: study between 7-50 cases,
and between 3-5 conditions
15 | 28-03-2017
QCA: what, when and why?
Source: Marx & Duşa (2011)
16 | 28-03-2017
QCA: what, when and why?
Cases can be micro-level, meso-level, or
macro-level
17 | 28-03-2017
QCA: what, when and why?
› A huge advantage of QCA is that it formalizes and systematizes case comparison; this is important!
› “The problem is that, when it comes to comparing more than, say, two or three cases, in many instances the comparison of the case study material is rather loose or not formalized – hence the scientificity of case studies is often questioned (…).”
Source: Rihoux & Lobe (2009)
18 | 28-03-2017
QCA: what, when and why?
› Some authors are even more trenchant. Jonathan Aus stated:
› “There can be no doubt that ‘thick descriptions’, as for instance employed in anthropology, may contribute to a better understanding of human behavior in specific social contexts. Yet the interpretation of data gathered in a theoretical vacuum remains largely intuitive (…). Nevertheless, most case studies (…) could maliciously be qualified as a-theoretical ‘data dumps’.”
Source: Aus (2009)
19 | 28-03-2017
QCA: what, when and why?
› There is another important reason why applying QCA is valuable: it is set-theoretic!
› Qualitative comparative analysis as an approach is…
Case-based/oriented
Comparative
Set-theoretic
You can make causal statements
You can systematically unravel complex causality
20 | 28-03-2017
QCA: what, when and why?
› QCA is a set-theoretic research approach, which makes it fundamentally different from regression-analytical methods
› Causal inference in regression, e.g.:
The more of X, the more of Y
The less of X, the less of Y
The less of X, the more of Y
› Causal inference in QCA, inter alia:
Only if X{1}, then Y{1} X{1} Y{1}
If X{1}, then Y{1} X{1} Y{1}
See for the full argument: Thiem, Baumgartner & Bol (2016)
Covariation vs.
implication
21 | 28-03-2017
QCA: what, when and why?
› Causal inference in QCA, inter alia:
Only if X{1}, then Y{1} X{1} Y{1}
If X{1}, then Y{1} X{1} Y{1}
› The first statement indicates a relationship of necessity: condition X has to be present for outcome Y to occur
In other words: Y implies X
Or: outcome Y is a subset of X
22 | 28-03-2017
23 | 28-03-2017
QCA: what, when and why?
› Causal inference in QCA, inter alia:
Only if X{1}, then Y{1} X{1} Y{1}
If X{1}, then Y{1} X{1} Y{1}
› The second statement indicates a relationship of sufficiency: condition X can produce the outcome Y by itself
In other words: X implies Y
Or: condition X is a subset of Y
24 | 28-03-2017
25 | 28-03-2017
QCA: what, when and why?
› However, causality is often more complex than just simple sufficiency or necessity! Conjunctural causation: combinations of conditions (i.e.,
configurations) produce the outcome [Logical-AND]
Equifinality: multiple configurations can produce the outcome [Logical-OR]
Asymmetry: the presence of X for Y does not imply the absence of X (i.e. ~X) for the absence of Y (i.e. ~Y)
Source: Verweij et al. (2013)
26 | 28-03-2017
The truth table: how to get there?
Gaining theoretical and case knowledge
Case construction
Raw data matrix
Truth table
Patterns
Interpretation
Return to the cases/theory
where the QCA
techniques come in
Adapted from: Verweij (2015)
Case study research
QCA is an iterative process
27 | 28-03-2017
The truth table: how to get there?
› Important to always keep in mind that, conceptually, in QCA, a case = a configuration of conditions plus the outcome
Constructing your cases:
1. Research question
2. Case selection
3. Defining the outcome
4. Selection of conditions
5. Gaining case knowledge
See: Rihoux & Lobe (2009) and Byrne & Ragin (2009)
Y
28 | 28-03-2017
The truth table: how to get there?
› Important to always keep in mind that, conceptually, in QCA, a case = a configuration of conditions plus the outcome
1. Research question
Should fit the set-theoretic nature of QCA
Cf. Berg-Schlosser et al. (2009) in Rihoux & Ragin (2009)
29 | 28-03-2017
The truth table: how to get there?
› A few examples of research questions from my own research
What combinations of network complexity,
stakeholder involvement, and
network management are necessary and/or sufficient for realizing
stakeholder satisfaction in governance networks?
What are the necessary and/or sufficient (combinations of)
conditions that explain the variation in
environmental justice policy adoption among
states in the US?
Is there a relation between lower bids by
contractors and the size of contract changes?
And what are the sizes of and reasons for contract changes in
transportation infrastructure projects?
30 | 28-03-2017
The truth table: how to get there?
› Important to always keep in mind that, conceptually, in QCA, a case = a configuration of conditions plus the outcome
1. Research question
Should fit the set-theoretic nature of QCA
QCA can be used for multiple purposes, inter alia:
Testing hypotheses or theories
Exploring patterns in data
Summarizing data
Building new theories
Cf. Berg-Schlosser et al. (2009) in Rihoux & Ragin (2009)
31 | 28-03-2017
› Important to always keep in mind that, conceptually, in QCA, a case = a configuration of conditions plus the outcome
2. Case selection
Cases can be micro, meso, macro
Cases should be comparable (i.e., defined by the same ‘scope conditions’), but have variation on the conditions
You are allowed to be flexible: it is OK to drop/add cases (but justify)
The truth table: how to get there?
32 | 28-03-2017
The truth table: how to get there?
› Important to always keep in mind that, conceptually, in QCA, a case = a configuration of conditions plus the outcome
3. Defining the outcome
You must have a clear definition of the outcome you want to explain across cases, prior to starting the analysis
You can only study one outcome per analysis
Include cases with Y and ~Y
Cf. Berg-Schlosser et al. (2009) in Rihoux & Ragin (2009)
33 | 28-03-2017
The truth table: how to get there?
› Important to always keep in mind that, conceptually, in QCA, a case = a configuration of conditions plus the outcome
4. Selection of conditions
Study the same conditions for all the cases but, ideally, cases show variation in terms of their scores (0 or 1) on the conditions
If theory is available, formulate expectations about the effect of conditions on Y or ~Y
Justify your selection of conditions
The n of conditions is kept low
Cf. Berg-Schlosser et al. (2009) in Rihoux & Ragin (2009)
34 | 28-03-2017
› 2 conditions (sets)
22 = 4 possible combinations of conditions (i.e., configurations/truth table rows)
› 3 conditions (sets)
23 = 8 possible combinations of conditions (i.e., configurations/truth table rows)
› 24 = 16
› 32, 64, 128…
QCA: what, when and why?
Try to have a good balance between n of
cases and n of conditions
Rule of Thumb: 2k
0 0
0 1
1 0
1 1
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1
35 | 28-03-2017
The truth table: how to get there?
› Important to always keep in mind that, conceptually, in QCA, a case = a configuration of conditions plus the outcome
5. Gaining case knowledge
You can use a variety of data sources, both qualitative and quantitative
Cf. Berg-Schlosser et al. (2009) in Rihoux & Ragin (2009)
36 | 28-03-2017
The truth table: how to get there?
› Important to always keep in mind that, conceptually, in QCA, a case = a configuration of conditions plus the outcome
5. Gaining case knowledge
You can use a variety of data sources, both qualitative and quantitative
Trade-off between number of cases and the in-depth knowledge you can have of the cases
Cf. Berg-Schlosser et al. (2009) in Rihoux & Ragin (2009)
37 | 28-03-2017
The truth table: how to get there?
Gaining theoretical and case knowledge
Case construction
Raw data matrix
Truth table
Patterns
Interpretation
Return to the cases/theory
where the QCA
techniques come in
Adapted from: Verweij (2015)
Case study research
QCA is an iterative process
38 | 28-03-2017
The truth table: how to get there?
› Once cases are constructed and selected
Data collection
Qualitative coding
Calibrating the coded data
Constructing the data-matrix
› Calibration is “the process of using empirical information on cases for assigning set-membership to them (…).”
Source: Schneider & Wagemann (2012)
39 | 28-03-2017
The truth table: how to get there?
› Cases have membership in sets (recall: conditions and outcomes = sets)
› There are three main important ‘anchor points’ in calibration:
0.0 full non-membership, out of the set
0.5 ambiguity, cross-over point
1.0 full membership, fully in the set
› Basically, the result of calibration is the grouping of the similar cases and the separating of different cases, per condition
40 | 28-03-2017
The truth table: how to get there?
› Calibration is “the process of using empirical information on cases for assigning set-membership to them (…).”
An example of
calibration:
Democratic control
in USA state
legislatures
Source: Kim & Verweij (2016)
41 | 28-03-2017
The truth table: how to get there?
Table: Ragin (2008)
Differences in degree
Differences in degree
Difference in kind
42 | 28-03-2017
The truth table: how to get there?
Another example of calibration
Source: Verweij (2015)
43 | 28-03-2017
The truth table: how to get there?
Another example of calibration: contracts, project scope, and project size
Source: Verweij (2015)
Crisp-set calibration (direct) D&C 0.00 DBFM 1.00
Fuzzy set four-value calibration (direct) Renewal 0.00 New build 0.33 Complex 0.67 Integral 1.00
Fuzzy set four-value calibration (cluster analysis)
44 | 28-03-2017
The truth table: how to get there?
Another example of calibration: management and cooperation
Source: Verweij (2015)
45 | 28-03-2017
The truth table: how to get there?
A final example of calibration: satisfaction
Source: Verweij (2015)
46 | 28-03-2017
47 | 28-03-2017
The truth table: how to get there?
› The result of your calibration is a data matrix
› In the case of crisp-set QCA, it only features case memberships of 0 or 1, as shown here
› Each row is a case
48 | 28-03-2017
The truth table: how to get there?
› In the data matrix, each row is a case
› In the truth table, you group similar cases as combinations of conditions (sets) each row
is now a configuration (combination of sets)
49 | 28-03-2017
Analyzing the truth table
› Size of the truth table is determined by the number of conditions: 2k, where k is number of conditions
› Each truth table row is a statement of sufficiency
Adapted from: Verweij et al. (2013)
Recall: complex causality! C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
Asymmetry
Equifinality
Configurational
50 | 28-03-2017
Analyzing the truth table
› First of all: examine the truth table and deal with problems occurring in it
Limited diversity
Contradictory rows
Adapted from: Verweij et al. (2013)
Logical Contradictions
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
Limited Diversity
51 | 28-03-2017
Analyzing the truth table
› Why are limited diversity and contradictory rows a problem?
› Because: the rule for truth table minimization is:
Pairwise compare configurations that agree on the outcome and differ in but one condition
Adapted from: Verweij et al. (2013)
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
52 | 28-03-2017
Analyzing the truth table
› Dealing with limited diversity
Add cases
Drop conditions
Counterfactual analysis
Recalibration
Adapted from: Verweij et al. (2013)
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
Limited Diversity
53 | 28-03-2017
Analyzing the truth table
› Dealing with contradictory rows
Drop cases
Drop configurations
Add conditions
Recalibration
Adapted from: Verweij et al. (2013)
Logical Contradictions
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
54 | 28-03-2017
Analyzing the truth table
› Notice the interplay between dealing with limited diversity and contradictions
Add cases Drop cases
Drop conditions Add conditions
Counterfactual analysis Drop configurations
Recalibration
Adapted from: Verweij et al. (2013)
Logical Contradictions
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
Limited Diversity
55 | 28-03-2017
Analyzing the truth table
› After the problems have been dealt with, the truth table can really be minimized
› The rule for truth table minimization is:
Pairwise compare configurations that agree on the outcome and differ in but one condition
Adapted from: Verweij et al. (2013)
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
56 | 28-03-2017
Analyzing the truth table
› The rule for truth table minimization is:
Pairwise compare cases that agree on the outcome and differ in but one condition
› This example: analysis for the presence of Y
Note that truth table
analysis is normally
done with software
(see compasss.org)
Adapted from: Verweij et al. (2013)
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
57 | 28-03-2017
Analyzing the truth table
› First, rewrite the rows in a statement of sufficiency; second, pairwise compare
› C*I*M + C*~I*M + ~C*I*M + ~C*~I*~M Y
C*M + I*M + ~C*~I*~M Y
Adapted from: Verweij et al. (2013)
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
Pairwise compare configurations that
agree on the outcome and differ in but one
condition
58 | 28-03-2017
Interpreting the results
› Our solution is: C*M + I*M + ~C*~I*~M Y
› Complex causality
Sufficient conditions: none Necessary conditions: none Sufficient configurations: three Necessary configurations: none INUS: all conditions
INUS = “Insufficient but Necessary parts of a
configuration which is itself Unnecessary but Sufficient”
Consistency of results: high, because no contradictory rows
are represented by the solution formula Empirical coverage of results is perfectly acceptable, as the
solution formula covers 9 out of 14 cases
59 | 28-03-2017
Interpreting the results
› How about lesson-drawing and policy transfer?
› For instance, looking at this table, what can be learned by comparing PERKT/DELFT with WIER?
C I M Outcome Y Cases
1 1 1 1 ZUID, NOORD, IJSS, SIJT, WEST
1 1 0 C LENT, SCHEL
1 0 1 1 PERK, DELFT
1 0 0 0 WIER
0 1 1 1 BROEK
0 1 0 C WAAL, GOUW
0 0 1 ???
0 0 0 1 DIEF
60 | 28-03-2017
Small exercise
› Construct a truth table from this data matrix and subsequently minimize it
Source: Li et al. (2016)
61 | 28-03-2017
Small exercise (solution)
› Construct a truth table from this data matrix and subsequently minimize it
62 | 28-03-2017
Concluding remarks
› The results need to be interpreted also by going back to the cases and theory
› In that sense is the solution formula often the starting point for the selection and study of typical or deviant (a-typical) cases (see Schneider & Rohlfing, 2013)
› You don’t have to do the whole QCA-process, you can also use elements (e.g., summarizing data or exploring patterns by drafting a truth table)
› More questions? Feel free to contact me! › Publications available via www.stefanverweij.eu or
ResearchGate
63 | 28-03-2017
References
› Aus, J.P. (2009). Conjunctural causation in comparative case-oriented research. Quality & Quantity, 43(2), 173-183.
› Berg-Schlosser, D., De Meur, G., Rihoux, B. & Ragin, C.C. (2009). Qualitative comparative analysis (QCA) as an approach. In B. Rihoux & C.C. Ragin (Eds.), Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques (pp. 1-18). London: Sage.
› Byrne, D.S. & Ragin, C.C. (2009). The Sage handbook of case-based methods. London: Sage.
› Kim, Y. & Verweij, S. (2016). Two effective causal paths that explain the adoption of US state environmental justice policy. Policy Sciences, 49(4), 119-139,
› Li, Y., Koppenjan, J.F.M. & Verweij, S. (2016). Governing environmental conflicts in China: Under what conditions do local governments compromise? Public Administration, 94(3), 806-822,
› Marx, A. & Duşa, A. (2011). Crisp-set qualitative comparative analysis (csQCA), contradictions and consistency benchmarks for model specification. Methodological Innovations Online, 6(2), 103-148.
› Pattyn, V. & Verweij, S. (2014). Beleidsevaluaties tussen methode en praktijk: Naar een meer realistische evaluatiebenadering. Burger, Bestuur & Beleid, 8(4), 260-267.
› Ragin, C.C. (1994). Constructing social research: The unity and diversity of method. Sage: New York.
› Ragin, C.C. (2008). Redesigning social inquiry: Fuzzy sets and beyond. Chicago: University of Chicago Press.
› Rihoux, B., Álamos-Concha, P., Bol, D., Marx, A. & Rezsöhazy, I. (2013). From niche to mainstream method? A comprehensive mapping of QCA applications in journal articles from 1984 to 2011. Political Research Quarterly, 66(1), 175-184.
› Rihoux, B. & Lobe, B. (2009). The case for qualitative comparative analysis (QCA): Adding leverage for thick cross-case comparison. In D.S. Byrne & C.C. Ragin (Eds.), The Sage handbook of case-based methods (pp. 222-242). London: Sage.
› Rihoux, B. & Ragin, C.C. (Eds.). (2009). Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques. London: Sage.
› Schneider, C.Q. & Rohlfing, I. (2013). Combining QCA and process tracing in set-theoretic multi-method research. Sociological Methods & Research, 42(4), 559-597.
› Schneider, C.Q. & Wagemann, C. (2012). Set-theoretic methods for the social sciences: A guide to qualitative comparative analysis. Cambridge: Cambridge University Press.
› Thiem, A., Baumgartner, M. & Bol, D. (2016). Still lost in translation! A correction of three misunderstandings between configurational comparativists and regressional analysts. Comparative Political Studies, 49(6), 742-774.
› Verweij, S. (2015). Once the shovel hits the ground: Evaluating the management of complex implementation processes of public-private partnership infrastructure projects with qualitative comparative analysis. Rotterdam: Erasmus University Rotterdam.
› Verweij, S. & Gerrits, L.M. (2013). Understanding and researching complexity with qualitative comparative analysis: Evaluating transportation infrastructure projects. Evaluation, 19(1), 40-55.
› Verweij, S., Klijn, E.H., Edelenbos, J. & Van Buuren, M.W. (2013). What makes governance networks work? A fuzzy set qualitative comparative analysis of 14 Dutch spatial planning projects. Public Administration, 91(4), 1035-1055.