Think Tank Methods: Quantitative Survey AnalysisOn Think Tanks #TTMethods Session 3

Courtney TolmieOctober 21, 2015

2 | R4D.org

Session Plan Introduction – what is Quantitative Analysis?

Quantitative Surveys and Data

Methods of Analysis

Deeper dive – Linking Context study

3 | R4D.org

Introduction – what is Quantitative Analysis Quantitative analysis – use of statistical/econometric

techniques to measure relationships between independent and dependent factors

Types of Quantitative Analysis:

Summary Stats

Cross tabulations

OLS regression

Structural equation models

Complexity, logistical needs, and sophistication

4 | R4D.org

Introduction – Advantages and Disadvantages

Advantages

• Allows assessment of trends across larger samples

• Allows for “simpler” comparisons of unlike units

• Allows for corrections of endogeneity/unobserved heterogeneity

• Allows researcher to assess causality, not just correlation

Disadvantages

• Methods – can gloss over nuanced differences • Methods – harder to have revealed

conclusions/findings• Logistics – the challenge of data

5 | R4D.org

Quantitative Surveys and Data - Options

Use Existing Data Collect New Data Do Both

6 | R4D.org

Quantitative Surveys and Data – Considerations for Collecting Data Developing the model to inform variables – dependent and

independent

Unit of analysis – organization, other?

Sampling – size and bias

Survey – cross-section, repeated cross-section, panel

Method of data collection – in person surveys, phone, electronic

Is there existing data that you can merge?

7 | R4D.org

Analysis - Overview

What do you WANT to test? Change over time Changes compared to the counterfactual Correlations Causation

What CAN you test with the data you have?

Summary Stats

Cross tabulations

OLS regression

Structural equation models

8 | R4D.org

Analysis – Summary Statistics and Cross Tabs

Summary Statistics Averages for variables across the dataset Comparing averages across subgroups of dataset

Cross tabulations Basic correlations between variables in the dataset Significance tests possible for cross tabulations or pairwise

tabulations

Summary Stats

Cross tabulations

OLS regression

Structural equation models

9 | R4D.org

Analysis – Regression analysis

More sophisticated analysis of relationships between variables

What regressions can (but will not necessarily) do: Assess impact of changes in think tank characteristics or other

factors over time (panel data) Correct for unobserved heterogeneity Assess causation

Summary Stats

Cross tabulations

OLS regression

Structural equation models

10 | R4D.org

Analysis – an example

Qualitative research Summary stats

Averages Cross tabulations

Significance of relationship between these Regression analysis

Direction of relationship Other important factor (number of PhD researchers on staff, for

example)

Think tank funding Placement of research in top-tier

journals

11 | R4D.org

Case – Linking Think Tank Performance, Decisions, and Context Research by Results for Development Institute and University

of Washington, with funding from the Think Tank Initiative

Objective – explore the relationship between political, economic, and social contexts and think tanks’ strategic behavior and performance

Mixed methods – literature mapping, comparative case studies, interviews/FGDs, and quantitative analysis

Study completed in 2014

12 | R4D.org

Linking Context – the model

Unit of analysis – individual organization Context variables– country level Third term – how exogenous context interacts with

endogenous characteristics Challenges:

How do we define outcomes? How do we assess think tank characteristics?

13 | R4D.org

Linking Context – the hypotheses

14 | R4D.org

Linking Context – the data

Think tank data Electronic survey Convenience sample Three languages – English, Spanish,

French 94 respondents in 48 countries Combined with existing datasets

Challenges: Response rate – 94 of over 300 sent Completion – 61% Development of survey itself

15 | R4D.org

Linking Context – other data sources

16 | R4D.org

Linking Context analysis – summary statistics

17 | R4D.org

Linking Context analysis – cross tabulations

18 | R4D.org

Lessons

Significant challenges to fielding quantitative survey in the field – should be weighed out

With proper data, quantitative analysis has potential to assess causation and correct for endogeneity

Smart mixed methods is best

Thank you!

19 | R4D.org