Focus docs - crowdsourcing your evaluation
-
Upload
steve-powell -
Category
Science
-
view
125 -
download
1
Transcript of Focus docs - crowdsourcing your evaluation
![Page 1: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/1.jpg)
Focus docs: Crowdsourcing your evaluation
Steve Powell, UNEP brown-bag, Nairobi March [email protected]
![Page 2: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/2.jpg)
What’s the problem?
• Greedy ToRs• … mean evaluation is the act of balancing
narratives• Evaluation matrix: way to go around the
houses• Recommendations out of thin air?
![Page 3: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/3.jpg)
Evaluation matrixTeam Web
surveyDocument analysis
Youth Youth in villages
…
QA: relevance
QA1: stakeholder relevanceQA1x: stakeholder relevance of component x
x x
…
QM: what shall we do about ….
x x
…
x x
![Page 4: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/4.jpg)
Around the houses
Eval
uatio
n qu
estio
ns
Evaluation
matrix
Ask q’s
Find other
evidence
Synthesise Compare, contrast opinions
Dream up
recommen
dationsGenerate
new
evidence
Evaluation
arithmetic1-5
judgments
Write report!
![Page 5: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/5.jpg)
Focus Docs: the steps
• Preparation• Clarify Privacy• Face to face• Inititialise questions• Invite• Moderate• Summarise• Write
![Page 6: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/6.jpg)
Evalcrowd discussion The evaluator
Results: strategy 1
Stakeholders
Results: strategy 2
Conclusions: strategy 1
Conclusions: strategy 2
“These are the mandatory questions for the report…”
post questions give answersask evaluator for new evidenceadd evidenceadd comments &”likes”
“.. But you can add your own”
posts some questions,moderates,challenges,adds new evidence, synthesises into final report
Using Focus Docs to crowdsource your evaluation
![Page 7: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/7.jpg)
Some other advantages of the Focus Doc procedure
• Break down answers by type of respondent e.g. by gender• discussion between all respondents almost simultaneously• Respondents quote and/or link to additional evidence • allows respondents to flag up where evidence seems to be
missing, i.e. where the evaluator can help• frees up some evaluator time to carry out some more in-depth
investigations where really needed• allows the evaluator to focus on issues which are critical and/or
controversial for contributors and/or minority opinions• makes it easy to bring in subject experts • allows social network analysis of the responses
![Page 8: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/8.jpg)
Disadvantages of Focus Docs?
• Low familiarity with web tools Security/Privacy
• Potential data loss – need for backups, etc ….• Fear of “on the record” contributions
![Page 9: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/9.jpg)
Evaluator role (1): Focus on addressing hidden topics
• What themes are we not talking about?• … would we rather not talk about?• some themes, some groups?• Do some groups get left out? Why?• Rewards / sanctions for saying …
• (Focus Docs disadvantage = advantage!)
![Page 10: Focus docs - crowdsourcing your evaluation](https://reader037.fdocuments.us/reader037/viewer/2022100120/58ee179b1a28ab76078b46cd/html5/thumbnails/10.jpg)
Evaluator role
2. The spirit and the letter3. Irreplaceable!4. Still face-to-face, but less5. Address bias6. Moderating the process and interaction7. Ensuring participation & equal contribution