Survey Routers: A PrimerNancy Brigham, Ph.D., VP Global Operations, Ipsos

Michael Fallig, Ph.D., Managing Director, Cogenti Applied Strategies

April 20, 2012 © 2012 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may not bedisclosed or reproduced without the prior written consent of Ipsos.

Agenda

Online research today – why the move to routers?

Major benefits & uses of routers

Key terminology / concepts

History / timeline of routers & common misconceptions

How routing works: big picture

Risks with routers

Ensuring good sample quality

Research-on-research

Q&A

3

State of Online Research Today

We put people on panels and target them directly to a specific study

InefficientLots of non-qualifiers

Changing Market / Internet Environment

Declining response rates

Decline in panel “joiners”

Changing Internet landscape and behaviors

Rapid growth of online studies

Traditional Online Research

4

Moving from Today to Tomorrow

Routers and Non-Panel Online Samples

Key challenge facing the online industry today: Sample Capacity

Consequences Inefficient use of sample – capacity not optimized Respondent engagement suffers Less reflectivity of the general population

Where are we going?

5

History / Timeline of Routers

Routers aren’t new – they’ve been around since 1996!

1996 DMS creates the first router

1998 Early routers are created (e.g., OTX, Greenfield Online)

Routers gain in popularity with clients and suppliers

Quality and sampling considerations become more important

Late 2000s Major suppliers start to create their own routers or more frequently buy routed sample

2010 Ipsos buys OTX – Ipsos panel is integrated into router

2011 ARF launches industry router quality initiative as part of FOQ2

Online survey

research takes off

6

Common Misconceptions About Routers

Routers mean sample blending Routers mean river sampling Routers are a “black box”

Routers ≠ river River sample needs routers, but

routers don’t need river.

Routers ≠ sample blending Routers are sample agnostic – they can

use just one, or many sources.

Routers are simply a technology that allows us to route respondents to a survey

Sample sourcing is a separate issue

Routers ≠ black boxWe can follow each respondent in the router and see exactly what happened

to them

7

Routers: In Depth….

Routers are the technology that allow us to manage multiple surveys and potentially multiple sources within a closed, controlled system. Routers allow us to group surveys and manage them using a certain,

consistent approach. Allocates sample across a set of studies, not just a single study Rules are used to assign respondents to one or more of a group of active

surveys, based on research and sample capacity needs.

Enable the efficient use of multiple sample sources (although multiple sources are not a requisite) Blending: the use of multiple sample sources within a single study Routers aren’t necessary for sample blending , but they make the process

much more efficient and introduce better control over the entire survey system

8

Some Key Terminology/Concepts….

Router

•Technology that lets us send respondents from one study to another

Reallocati

on

•Respondents who don’t qualify for a survey can attempt to qualify for other surveys that are open at that time.

Blended sample

•A sample that contains multiple sources (multiple panels, multiple non-panels, or mixture of both).

Non-pane

l sources

•Sample obtained directly from the Internet

•Social media, communities, reward/loyalty programs, ad networks, pop-ups, etc.

9

Some Key Terminology/Concepts….

Parallel &

serial

router

•Serial: Respondents attempt to qualify for studies one at a time.

•Parallel: Respondents answer screening questions that will allow them to pre-qualify for multiple studies.

Router

selection algorithm

•How studies are selected by the router for a respondent

•Random, priority, hybrid

Mini-

routers

•Smaller routers that are set up for some specific purpose

•Usually done to manage risk & get greater efficiencies

Screenin

g process

•Potentially get screening questions from multiple studies

10

How Routing Works: Big Picture

SOURCE 1

SOURCE 2

SOURCE 6

SOURCE 5

SOURCE 4

SOURCE 3

SOURCE 7

Multiple Sample Sources – Panel & Non-Panel

B

A

Screening for Survey…

Survey D

Survey E

Sent to Survey Qualified For

D

C

E

Router – Survey Qualification

Survey C

Survey B

Survey A

The router is the engine that lets us manage all the sample sources for a study • could be a single source, or blended sources

11

Major Benefits of Routers

Increased Sample Capacity &

Population Reach

Easier Survey Adaptation to New

Online Realities

• Allows studies to be tailored to people accessing them using different devices (e.g., mobile)

• More efficient use of sample in general, and especially lower-response demographics

• Access to parts of the population that don’t traditionally join panels

• Increased respondent engagement and satisfaction

• Even better control over operational quality aspects (e.g., automated)

Increased Sample Capacity &

Population Reach

Higher Quality (Data & Respondents)

12

How Businesses are Using Routers

How sample providers are using routers Maximize efficiency

─ Increase productivity of each sample respondent ─ Efficiently manage the sample coming from a variety of sources─ Increase the consistency of processes across studies─ More efficient method for monitoring multiple “live” projects

Access and provide respondents Accommodate the needs of many different clients

How research organizations are using routers Increase feasibility of conducting studies Provide higher quality of data & respondent satisfaction Maximize efficiency Manage category exclusions, quotas, lockouts, etc., more efficiently Increase productivity of each sample respondent Used primarily for own clients Targeted mini-routers (e.g., ethnic, client-based, study type) Increase consistency across studies

13

Managing Routers

There are a variety of ways of managing routers The buyer needs to be observant of this

Major aspects to consider

Sampling • Degree of randomization in survey selection• How screeners are presented to respondents• How priority is managed• How multiple sources are accessed and managed

Environment• Number & diversity of studies in the router• Rules to ensure one study doesn’t bias another• Monitoring of the environment

14

Risks with Routers

• Selection bias• Incidence – ability to get and report incidence

correctlySampling/Routing

• Who has responsibility for router decisions• Common screening environment• Router “rules” and best practices• Business pressures (such as very low cost)Environment

• Use proper sources/recruiting• Inconsistent use of sources when consistency is

needed• Types of studies Sources

15

Ensuring Good Sample Quality with Routers

With the right rules and controls in place, sample quality can be even higher with routers

Much easier to monitor one environment with 300 studies than 300 environments with one study each

Routers automate many of the controls and monitoring

Benchmarking and parallel testing to measure sample consistency

Keeping track of panelist experience in routerRouters can record every single thing that happens to a respondent

while in the router

Also do quality control at “back-end” (through quotas at survey level)

16

Research-on-Research with Routers

Route 66 The Long Road to Efficient and Effective Router Use (Miller, 2010) Presented at 15th Annual CASRO Technology Conference, June 3,2010 Covers priority/randomization, taking multiple surveys in one sitting, effect of screener position

Factorial Design on Survey Router Effects (Johnson & Fawson, 2010) Presented at the 2010 CASRO Panel Conference in Las Vegas Covers taking multiple surveys in one sitting, panelist fatigue

Measuring Selection Bias Introduced by Routing (Porter, Scott, de Guademar & Kimura, 2010) Presented at the 2010 CASRO Panel Conference in Las Vegas Covers selection bias due to reallocation

Sampling with Routers: Comparing Survey Results of “Reallocated” and Traditionally Sampled Respondents (Brigham, Porter, Markowitz & Fuller, 2011) Presented at the 2011 CASRO Technology Conference Covers priority/random assignment, selection bias due to reallocation, percent of sample

reallocated in a single study, bias effect due associated with studies of low priority

Survey Router Management: An Experimental Examination of the Impact on Survey Results (Brigham & Fuller, 2012) Presented at the 2012 CASRO Online Conference in Las Vegas Covers number of studies on router, correlation of studies on router, time spent in router being

screened

FoQ 2 Router Initiative

Mission & Goals

Team MissionExpand industry knowledge about routers by understanding the effects of practices on research outputs. Provide a level playing field in terms of transparency between clients and router providers.

Team Goals» Document current router practices, key variables and terminology» Examine router optimization practices, evaluate router impact on

sample survey results and overall quality; begin mapping out appropriate practices

To best accomplish its mission and goals, the Router team has partitioned itself into three groups with distinct responsibilities…

18

Router Subgroups

Subgroup Objectives» Group 1:

• Identify current practices including key variables that can impact sample, respondent and overall data quality.

• Design topic guides for Focus groups and compile participant lists to examine current practices

• Compile existing R-on-R and summarize

• Identify the universe of router variables around which practices may differ, including “upstream” and “downstream” variables that are not within the team’s present scope to examine.

• Work with independent consultant to develop a report of findings regarding the primary research effort

19

Router Subgroups

Subgroup Objectives (cont’d)» Group 2:

• Design and conduct primary research to assess router optimization practices, configurations and sample re-allocation practices.

• Study design includes a simulation component and a database that can be used for additional explorations.

• Independent consultant analyzes results

» Group 3:

• Develop a set of generic router schematics to aid end users of the technology and those who buy routed sample – to aid the understanding of the router process.

• Establish a set of appropriate router use practices

• Develop a set of questions that buyers can ask when licensing routers or purchasing routed sample

• Compile a dictionary of router terms

20

Team Members, Group Affiliation& Oversight Leadership

» Michael Fallig, Cogenti Applied Strategies(co-chair) (Group 2)

» Chuck Miller, dm2corp (co-chair) (Group 3)

» John Bremer, Toluna (Group 2)» Nancy Brigham, Ipsos (Group 2 leader)» Pete Cape, SSI (Group 3)» Mike Conklin, MarketTools (Group 2)» Steve Gittelman, MKTG Inc. Group 1)» Paul Johnson, Opinionology/SSI (Group 1)» Peter Milla, CASRO (Group 3)» Robin Murphy, Ipsos (Group 2)» Efrain Ribeiro, Kantar Lightspeed (Group 1)

Oversight Leadership

» Bill Cook, ARF

» Don Gloeckler, ARF

» Gian Fulgoni, comScore

21

Potential Key Variables to Assess

Independent variables (examined via manipulation or simulation)Priority vs. Random AssignmentNumber of studies in routerIncidence rates of studies on routerCorrelation among screening questionsManipulate number of surveys someone can take in one sitting?Number of times someone could be reallocated? (Limit number of touches)

Dependent variablesResponse rate, completion rate, over quota’s, incidence calculationsEfficiency metrics – capacity lift due to reallocation, etc.Sub-categories of questions – e.g., demos, attitudinal, behavioral, population benchmarksRouter engagement - measures of respondent straight lining, speeding (Group 1 should say what these are)Respondent Satisfaction

Group 2, Core Activity: Quantitative Research Study

27

FoQ2 Research Priority 7: (cont’d)

28

Thank you and Q&A

Nancy L. Brigham, Ph.D.VP Global Operations, Respondent Access & Engagement

160 Bloor St. East, Suite 300Toronto, ON, Canada M4X 1A2

Phone: 1-647-259-9578email: nancy.brigham@ipsos.com

Michael A. Fallig, Ph.D.Managing Director/CTO, Cogenti Applied Strategies, Inc.

1350 Avenue of the AmericasNew York, NY 10019

Phone: 646-218-9859email: michael.fallig@cogenti.com