© 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing...
-
Upload
rafael-grimmer -
Category
Documents
-
view
213 -
download
0
Transcript of © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing...
![Page 1: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/1.jpg)
© 2009 Knowledge Networks, Inc.
Mario Callegaro
Charles DiSogra
Knowledge Networks
Computing response metrics foronline panels
DC AAPOR Workshop on Web Survey Methods, September 9 th 2009
![Page 2: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/2.jpg)
What metrics for what panel
Pre-recruited probability-based online panels Response rates can be calculated because the frame is known
(AAPOR, 2006)
Volunteer opt-in panels “Response rates” cannot be computed (AAPOR, 2007) However, other metrics can be calculated, e.g. completion rate
2
![Page 3: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/3.jpg)
3
Current status• Volunteer, non-probability (opt-in) panels, widely used in
market research, outnumber probability-based Web panels
• More and more probability-based online panels being built 2007-2009 American National Election Studies (ANES) Panel Face-to-Face Recruited Internet Survey Platform (FFRISP, 2008) Dutch Long-term Internet Study for the Social Science (LISS) panel (2007)
• Still no officially agreed standard on how to compute response rates for online panels
![Page 4: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/4.jpg)
Review of current standards
Many efforts and proposals by different national and international organizations:
European Society for Opinion and Marketing Research – ESOMAR European Federation of Associations of Market Research Orgs. –EFAMRO Interactive Marketing Research Organization – IMRO Advertising Research Association – ARF quality initiative Bob Lederer proposal endorsed by the American Marketing Association
(AMA) Latest effort by ISO (standard #26362) touches on subject
4
![Page 5: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/5.jpg)
Some journals are giving guidelines on how response rates should be computed specifically for online surveys (not necessarily online panels)
Journals enforcing AAPOR standards: (e.g. POQ, IJPOR…) Journal of Medical Internet Research
Journal of Medical Internet Research (Eysenbach, 2004):
Journal recommendations
5
In online surveys, there is no single response rate.
Rather, there are multiple potential methods for calculating a response rate, depending on what are chosen as the numerator and denominator.
As there is no standard methodology, we suggest avoiding the term “response rate” and have defined how, at least in this journal, response metrics such as, what we call, the view rate, participation rate and completion rate should be calculated.
![Page 6: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/6.jpg)
ESOMAR and IMRO examples
• ESOMAR (2005) metrics: “Response based on the total amount of invites (% of full numbers)
per sample drawn (country, questionnaire) % questionnaire opened % questionnaire completed (including screen-out) % in target group (based on quotas) % validated (the balance is cleaned out, if applicable)” (p. 20).
• IMRO (2006) metrics: Response rate is “based on the people who have accepted the
invitation to the survey and started to complete the survey. Even if they are disqualified during screening, the attempt qualifies as a response” (p. 13).
Completion rate “is calculated as the proportion of those who have started, qualified, and then completed the survey” (p. 13).
6
![Page 7: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/7.jpg)
AMA platform for data quality progress:
7
removal)for Request or Errors ks,(Bouncebac - tss/intercepinvitation Total
responses attempted of #Tot Rate Response
criteria screening passing # Total
CompletesRate Completion
criteria screening passing # Total
resuming without pausingor Quitting #Raten Terminatio-Mid
responses attempted # Total
criteria screeening passing # TotalRateion Qualificat
Platform for Data Quality Progress, Bob Lederer(under AMA umbrella, Nov 2008)
![Page 8: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/8.jpg)
ISO 26362:2009
• Participation rate: ‘number of panel members who have provided a usable response divided by the total number of initial invitations requesting members to participate (p. 3)
• Usable response is one where the respondent has provided answers to all the questions required by the survey design
• The term “response rate’ cannot be used to describe respondent cooperation for access panels
8
![Page 9: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/9.jpg)
Necessary information to compute response metrics
• In order to compute response metrics for online panels we need to understand how panel members are recruited and what stages are used to build a panel
• Volunteer-opt-in design
• Probability-based design
9
![Page 10: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/10.jpg)
Generalized volunteer opt-in panel design
• Stage 1: Encounter, discover, or seek out to join
• Stage 2: Provide profile information
• Stage 3: Get and do surveys
10
![Page 11: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/11.jpg)
Volunteer opt-in panels: Stages
11
Respondent decidesto opt-in
Opt-in panel portalEnter some basic
information
Email confirmation(double opt-in)
Profile survey
Double opt-in
Single opt-in
Active panel
Postoaca, 2007
![Page 12: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/12.jpg)
Stages for probability-based online panels
• Stage 1: Recruitment from frame
• Stage 2: Welcome and get profiled
• Stage 3: Active membership, ready for surveys, actual study
12
![Page 13: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/13.jpg)
Common steps in building a probability-based panel
1. Recruitment Rate (RECR): the recruitment of potential panel members
Recruitment rate calculation will depend on the recruitment mode: face -to-face, telephone, mail
2. Profile Rate (PROR): empanelling recruited persons This stage counts panel members that answered their profile survey,
generally a questionnaire collecting background information and welcoming respondents to the panel
The computation of the profile rate (a.k.a., connection rate) will depend on the data collection mode
Profiled members are considered to be “active members” in the pool from which study samples can be drawn
13
![Page 14: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/14.jpg)
Probability-based design features Implications for computing response rates
1. Single recruitment cohort (one-time effort) vs. multiple recruitment cohorts (on-going recruitment)
2. Within-household selection to recruit one person vs. whole household recruitment of all eligible persons
3. The data collection mode used for non-internet households (no access to online surveys at time of recruitment)
14
![Page 15: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/15.jpg)
Methods of dealing with non-Internet households
15
Probability-basedsample
Internethousehold
Non- Internethousehold
Member(s) use theirdevice and Internet
connection tocomplete surveys
Member(s) aregiven a device andInternet connectionto complete surveys
Member(s)complete surveys in
another mode
All members are given adevice to complete
surveys no matter theirInternet status
Mail Phone IVR
Assessment ofInternet status
![Page 16: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/16.jpg)
Probability- based web panels: Recruitment
16
Probability Sample (PS)
Knowneligibility
Unknowneligibility(UH UO)
EligibleNot
eligible
Initial consent(IC)
Stage 1 Recruitment
Refusals andbreak offs
(R)
Non -contacts
(NC)
Others(O)
![Page 17: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/17.jpg)
Probability- based web panels: Profile
17
"Profile"survey
Returnedquestionnaire
(I or P)Connection stage
Eligible - noninterview
(NC)
Refusals andbreak offs
(R)
Non -contacts
(NC)
Others(O)
![Page 18: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/18.jpg)
Probability- based web panels: Actual studySame design for volunteer-opt-in panels
18
Active Panel (AP)
Sample frame forspecific study (SF)
Complete(I)
Partial(P)
Break-off(R)
Refusal(R)
Non contact(NC)
Other noninterview
(O)Eligible?
Not eligibleUnknowneligibility(UH UO)
NoN
o
Yes
Yes
Prescreening
on database
![Page 19: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/19.jpg)
Active panel dynamics
19
Active Panel
Temporaryinactive
Notavailable
forsampling
Continuousrecruitment
Involuntaryattrition
Voluntaryattrition
Mortality
Re-recruitment
Re-recruitment
![Page 20: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/20.jpg)
Stage 1 of probability-based web panels
IC = Initial consent
R = Refusal
UH and UO = Unknown if household or unknown “other”
NC = Non-Contact
O = Other non-interview
e = Estimated proportion of unknown eligibility cases
R Refusal (REFR) = Rate IC + (R + NC + O) + e(UH + UO)
IC Recruitment (RECR) = Rate IC + (R + NC + O) + e(UH + UO)
20
Example: P_RECR = .4 x 100% = 40%
![Page 21: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/21.jpg)
I = Profile survey complete
P = Profile survey partial but acceptable
Stage 2 (more likely for probability-based panel)
* Opt-in panels may not know the denominator components.
(I + P) Profile Rate (PROR) =
(I + P) + (R + NC + O)*
RRefusal to Profile (REFP) =
(I + P) + (R + NC + O)*
21
Example: PROR = .6 x 100% = 60%
![Page 22: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/22.jpg)
Stage 3 Specific Study Rates
BF = Break-offs -- when the number of answers is below the definition of partial interview, it can be considered a break-off. R = Other than for the break-off rate, R includes break-offs as refusals
(I + P) Completion Rate (COMR) =
(I + P) + (R + NC + O)
BF Break-off Rate (BFR) =
(I + P) + BF
Study R Refusal (SREF) = Rate (I + P) + (R + NC + O)
22
Example: COMR = .7 x 100% = 70%
![Page 23: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/23.jpg)
Cumulative Response RateOnly for pre-recruited probability-based online panels
P_RECR = Person recruitment rate
PROR = Profile rate
COMR = Completion rate for the single study
RETR = Retention rate
A multiplicative function
Cumulative RR (CURR) = P_RECR x PROR x COMR
23
Cumulative RR2 (CURR) = P_RECR x PROR x RETR x COMR
Example CURR= .4 x .6 x .7 = .168 x 100% = 16.8%
Example CURR2= .4 x .6 x .8 x .7 = .134 x 100% = 13.4%
![Page 24: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/24.jpg)
Recruitment
CohortRecruitedmembers
RecruitmentRate Profiled
members
ProfileRate
Activemembers
RetentionRate
Sample
Respondents
Completionrate
The computation of a CUMRR isstraightforward when the panelis built with a single recruitment cohort
Computing CUMRR with 1 cohort
24
Study Respondents
RECR PROR RETR
COMR
![Page 25: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/25.jpg)
Recruitment
Cohort1
Recruitedmembers
RecruitmentRate Profiled
members
ProfileRate
Activemembers
RetentionRate
Sample
Recruitment
Cohort2
Recruitedmembers
RecruitmentRate Profiled
members
ProfileRate
Activemembers
RetentionRate
Sample
Recruitment
Cohort3
Recruitedmembers
RecruitmentRate Profiled
members
ProfileRate
Activemembers
RetentionRate
Sample
Unequal cohort contributions to a
study sample selected from
among all active members
25
Computing CUMRR with 3 cohorts
![Page 26: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/26.jpg)
Formulas dealing with multiple cohorts (1.)
RECR, PROR, RETR are calculated as the weighted average of the size contribution of each cohort
Example to calculate RECRtotal
26
cnccc
cncncccccctotal WWWW
RECRWRECRWRECRWRECRWRECR
...
...
321
332211
Where Wcn = the number of cases contributed to the sample from cohort n
![Page 27: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/27.jpg)
Example of RECR with 3 cohorts
Cohort 1 Cohort 2 Cohort 3
Size in the final sample
200 100 50
Recruitment rate (RECR)
.35 .27 .15
27
50100200
15.5027.10035.200totalRECR
2985.350
5.104
350
5.72770
totalRECR
![Page 28: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/28.jpg)
Formulas dealing with multiple cohorts (2.)
28
cnccc
cncncccccctotal WWWW
RECRWRECRWRECRWRECRWRECR
...
...
321
332211
cnccc
cncncccccctotal WWWW
PRORWPRORWPRORWPRORWPROR
...
...
321
332211
cnccc
cncncccccctotal WWWW
RETRWRETRWRETRWRETRWRETR
...
...
321
332211
![Page 29: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/29.jpg)
Full example with 3 cohorts
Cohort 1 Cohort 2 Cohort 3 ___Rtotal
Size 200 100 50
RECR .35 .27 .15 .299
PROR .57 .65 .70 .611
RETR .50 .67 .85 .599
29
%0.13%100130.713.611.299.CUMRR1total %8.7%100078.713.599.611.299.CUMRR2 total
Assume a survey completion rate (COMR) of .713
![Page 30: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/30.jpg)
Non- Internethousehold
Member(s) aregiven a device andInternet connectionto complete surveys
Member(s)complete surveys in
another mode
Mail Phone IVR
Computing completion rate (COMR) when multiple data collection modes are used
Completion rates need to be computed separately for each mode
Web survey Mail, phone or IVR
These rates should also be combined as a weighted average
30
![Page 31: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/31.jpg)
Technical condition in order to compute response metrics
• In order to compute response metrics each panel organization must keep an historical database with rates for each member
• More specifically for probability-based online panels it is necessary that: Each panel member ever recruited must have a record of his/her:
– Recruitment rate cohort value– Profile rate cohort value– Retention rate cohort value
31
![Page 32: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/32.jpg)
Which formula for which panel?
Metric Probability-based
Volunteer opt-in
Recruitment Yes N/A
Refusal to be recruited Yes N/A
Profile Yes Maybe
Refusal to profile Yes Maybe
Screening Yes Yes
Eligibility Yes Yes
Completion Yes Yes
Break-off Yes Yes
Refusal Yes Yes
Cumulative Response Yes N/A32
![Page 33: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/33.jpg)
Which formula for which panel? II
Metric Pre-recruited Volunteer
Attrition cross sectional Yes Yes
Attrition longitudinal Yes Yes
Reinterview Yes Yes
33
![Page 34: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/34.jpg)
Dutch study (Vonk, van Ossenbruggen, & Willems, Esomar 2006)
Panel Management or Manipulation?
34
![Page 35: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/35.jpg)
Some factors affecting each rateRecruitment rate
Recruitment methods Incentives
Profile rate Incentives Panel management efforts
Retention rate Time elapsed since recruitment Incentives Panel management efforts
Survey completion rate Field time Incentives Reminders
35
![Page 36: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/36.jpg)
References
• Callegaro, M. and DiSogra, C. (2008). Computing response metrics for online panels. Public Opinion Quarterly, 72, pp. 1008-1032.
• DiSogra, C. and Callegaro, M. (forthcoming). Computing response rates for probability based web panels. In American Statistical Association (Ed.). Proceedings of the joint statistical meetings: section on survey research methods [Cd-Rom]. Alexandria, VA: American Statistical Association.
36
![Page 37: © 2009 Knowledge Networks, Inc. Mario Callegaro Charles DiSogra Knowledge Networks Computing response metrics for online panels DC AAPOR Workshop on Web.](https://reader036.fdocuments.us/reader036/viewer/2022070307/551a442e550346545e8b5127/html5/thumbnails/37.jpg)
Future work
• Recruitment level computed at a household or at a person level (when recruiting multiple members per household)
• Attrition rates for cross sectional design
• Attrition rates for longitudinal designs
• Response rates for longitudinal designs
37