How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple...
Transcript of How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple...
![Page 1: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/1.jpg)
How to Develop Outcome Measures &
Design Effective Surveys
April 11, 2018
![Page 2: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/2.jpg)
Today’s Speakers
Joyce ChapmanAssessment Analyst & Consultant
Assessment & User Experience Dept.
Duke University Libraries
Emily PlagmanProject Manager
Public Library Association
![Page 3: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/3.jpg)
Agenda
▪ Project Outcome’s Outcome Measurement
Guidelines
▪ Developing Outcome Measures
▪ Writing Effective, Actionable, and Unbiased Survey
Questions
▪ Writing Outcome Measures
▪ Announcements / Q&A
![Page 4: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/4.jpg)
Project Outcome’s
Outcome Measurement Guidelines
Emily PlagmanProject Manager
Public Library Association
![Page 5: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/5.jpg)
What is Project Outcome?It’s FREE!
www.ProjectOutcome.org
![Page 6: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/6.jpg)
Project Outcome Toolkit
Access to:
▪ Quick & simple surveys
▪ Easy-to-use survey tool
▪ Custom data reports
▪ Interactive data dashboards
▪ Resources & training
www.ProjectOutcome.org
It’s FREE!
![Page 7: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/7.jpg)
Training Resources
▪ Getting Started
▪ Surveys
▪ Data Collection
▪ Data Analysis
▪ Taking Action
▪ From the Field
Tutorial Videos
Getting Started with Project Outcome Online Tools
www.ProjectOutcome.org
![Page 8: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/8.jpg)
Training Resources
www.ProjectOutcome.org
![Page 9: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/9.jpg)
Outcome Measurement Guidelines
login ->
resources ->
from the field:
outcome
measurement
guidelines
www.ProjectOutcome.org
![Page 10: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/10.jpg)
Outcome Measurement Guidelines
Designed to help guide & support libraries with:
▪ Developing their own outcome measures
▪ Implementing data collection methods
▪ Measuring outcome data over time
▪ Developing strategies for working with partners on
outcome measurement projects
www.ProjectOutcome.org
![Page 11: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/11.jpg)
Outcome Measurement Guidelines
Designed to help guide & support libraries with:
▪ Developing their own outcome measures
▪ Implementing data collection methods
▪ Measuring outcome data over time
▪ Developing strategies for working with partners on
outcome measurement projects
www.ProjectOutcome.org
Other Guidelines topics to be covered throughout 2018
![Page 12: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/12.jpg)
Outcome Measurement Guidelines
GUIDELINESADDITIONAL
RESOURCESCASE STUDIES
www.ProjectOutcome.org
![Page 13: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/13.jpg)
Worksheet: Developing Outcome Measures
![Page 14: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/14.jpg)
Peer Discussion: Program Planning
Community.projectoutcome.org/t/program-planning/411
![Page 15: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/15.jpg)
Developing Outcome Measures
Emily PlagmanProject Manager
Public Library Association
![Page 16: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/16.jpg)
Why develop your own outcome measures?
Share in the chat
▪ Project Outcome questions don’t fit your program?
▪ Partnership requirements?
▪ Grant/funding requirements?
▪ Measuring specific strategic planning goals?
![Page 17: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/17.jpg)
What is an Outcome?
Needs Assessment Patron Satisfaction
Outputs Outcomes
![Page 18: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/18.jpg)
What is an Outcome?
Specific benefit from a library program/service
Can be quantitative or qualitative
Expressed as changes that individuals perceive
![Page 19: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/19.jpg)
Answering “So What?”
Establishing a causal relationship (A – B)
▪ If A is introduced, B is more likely to occur
Remember that…
▪ B doesn’t always occur with A
▪ Sometimes B occurs even without A
▪ Passage of time means many other factors
impacting patrons
▪ Program/service can never take full credit
![Page 20: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/20.jpg)
Answering “So What?”
In/On:
✓ Knowledge
✓ Confidence
✓ Skills
✓ Attitudes
✓ Behaviors
✓ Status
Establishing a causal relationship (A – B)
Outcome questions ask:
➢ Changes
➢ Effects
➢ Impacts
![Page 21: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/21.jpg)
Answering “So What?”
Example: Build-a-Budget…
✓ Changed their attitudes about the benefits of building
a budget
✓ Increased their knowledge about budgeting
✓ Changed their behavior and created a budget to meet
their financial goals
![Page 22: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/22.jpg)
Develop Outcome Measures
Program Goals
Intended result of the program
• Education
• Skill Building
• Engagement
• Creativity
• Entertainment
Desired Outcomes
Benefits to the patron based on program goals
• Knowledge
• Confidence
• Skills
• Attitudes
• Behaviors
• Status
Outcome Measures
Did the program have a change or effect on X for the patron or community?
• Outcome #1
• Outcome #2
• Outcome #3
![Page 23: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/23.jpg)
Step 1: Identify Program Goals
Program Goals
Intended result of the program
• Education
• Skill Building
• Engagement
• Creativity
• Entertainment
Desired Outcomes
Benefits to the patron based on program goals
• Knowledge
• Confidence
• Skills
• Attitudes
• Behaviors
• Status
Outcome Measures
Did the program have a change or effect on X for the patron or community?
• Outcome #1
• Outcome #2
• Outcome #3
![Page 24: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/24.jpg)
Step 1: Identify Program Goals
▪ What are the intended results of the
program/service?
▪ What should patrons walk away knowing, doing, or
feeling?
![Page 25: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/25.jpg)
Example: Build-a-Budget
Program Goals
Intended result of
Build-a-Budget
• Education
• Skill Building
• Engagement
• Creativity
• Entertainment
Help patrons understand the value of creating and
following a budget
Deliver instruction that will help patrons create their
own budget
Provide enough resources and
instruction so that patrons use that budget when they return home
![Page 26: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/26.jpg)
Step 2: Identify Desired Outcomes
Program Goals
Intended result of the program
• Education
• Skill Building
• Engagement
• Creativity
• Entertainment
Desired Outcomes
Benefits to the patron based on program goals
• Knowledge
• Confidence
• Skills
• Attitudes
• Behaviors
• Status
Outcome Measures
Did the program have a change or effect on X for the patron or community?
• Outcome #1
• Outcome #2
• Outcome #3
![Page 27: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/27.jpg)
Step 2: Identify Desired Outcomes
Is one of the program goals to effect a specific change in
the patron?
➢ If not, the program may not be the best fit for
outcome measurement
Identify desired outcomes based on the program goals.
➢ Helps you be realistic about outcomes you want to
achieve & understand whether you achieved them
➢ Limit to outcomes most relevant to your program
goals
![Page 28: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/28.jpg)
Desired Outcomes
Benefits to the patron based on education & skill building
• Knowledge
• Skills
• Attitudes
• Status
• Confidence
• Behaviors
Example: Build-a-Budget
Patrons have a better understanding of how a budget will help them
Patrons are able to use a budget template in Excel
Patrons will apply what they learned at home
![Page 29: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/29.jpg)
Step 3: Develop Outcome Measures
Program Goals
Intended result of the program
• Education
• Skill Building
• Engagement
• Creativity
• Entertainment
Desired Outcomes
Benefits to the patron based on program goals
• Knowledge
• Confidence
• Skills
• Attitudes
• Behaviors
• Status
Outcome Measures
Did the program have a change or effect on X for the patron or community?
• Outcome #1
• Outcome #2
• Outcome #3
![Page 30: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/30.jpg)
Step 3: Develop Outcome Measures
Goal of measuring outcomes is to answer:
Did the program have a change or effect on X for the
patron or community?
X = DESIRED OUTCOME
Outcomes can be measured using surveys,
interviews, focus groups or other one-on-one
approaches.
![Page 31: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/31.jpg)
Example: Build-a-Budget
Did the Build-a-Budget program increase or improve
budgeting knowledge/skills/behaviors for the patron?
![Page 32: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/32.jpg)
I understand the value of having a personal budget.
Yes | No | Please explain
I have created a budget using the Excel template.
Yes | No | Please explain
Example: Build-a-Budget
I have used my budget to: (check all that apply)
• Categorize my spending
• Track my spending for at least one month
• Identify at least one way to save money
• None of the above
• Other (please specify)
Outcome Measures
Did the program
increase or improve
budgeting
knowledge/skills/beh
aviors for the patron?
• Outcome #1
• Outcome #2
• Outcome #3
![Page 33: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/33.jpg)
Example: Build-a-Budget
Program Goals
Intended result of Build-a-Budget
• Education
• Skill Building
Desired Outcomes
Benefits to the patron based on education & skill building
• Knowledge
• Skills
• Behaviors
Outcome Measures
Did the program increase or improve budgeting knowledge/skills/behaviors for the patron?
1.I understand the value of having a personal budget.
2.I have created a budget using the Excel template.
3.I have used my budget to: (check all that apply)
• Categorize my spending
• Track my spending for at least one month
• Identify at least one way to save money
• None of the above
• Other (please specify)
![Page 34: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/34.jpg)
From the Field: Skokie Public Library
Staff worksheets: A) What Are You Hoping to Capture?
![Page 35: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/35.jpg)
From the Field: Skokie Public Library
Staff worksheets: B) Outcome Goals and Indicators
![Page 36: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/36.jpg)
What are the outcomes you want to measure?
Share in the chat
In/On:
✓ Knowledge
✓ Confidence
✓ Skills
✓ Attitudes
✓ Behaviors
✓ Status
Outcome questions ask:
➢ Changes
➢ Effects
➢ Impacts
![Page 37: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/37.jpg)
Writing Effective, Actionable, and
Unbiased Survey Questions
Joyce ChapmanAssessment Analyst & Consultant
Assessment & User Experience Dept.
Duke University Libraries
![Page 38: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/38.jpg)
WRITING UNBIASED, ACTIONABLE, EFFECTIVE QUESTIONS
![Page 39: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/39.jpg)
Avoid jargon
▪ Avoid library terms, colloquialisms, and acronyms.
▪ Think of your audience.
▪ Wording should be simple.
▪ Avoid technical words.
X "How many times last week did you use the library's Internet-enabled public access computers?"
![Page 40: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/40.jpg)
Avoid jargon and colloquialisms
▪ Think of your audience.
▪ Wording should be simple.
▪ Avoid technical words.
X "How many times last week did you use the library's Internet-enabled public access computers?"
O "How many times last week did you use the library's computers [to access the Internet]?"
![Page 41: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/41.jpg)
Abbreviations and acronyms
Assume your audience does not know any of these.
X “CCPL is interested in replacing its OPAC."
X “How frequently do you use our ILL services?”
![Page 42: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/42.jpg)
Abbreviations and acronyms
Assume your audience does not know any of these.
X “CCPL is interested in replacing its OPAC."
O “Cleveland County Public Library is interested in replacing its online catalog.”
X “How frequently do you use our ILL services?”
O “How frequently do you use our Interlibrary Loan services? [This service allows us to request material from another library
for you if we do not have it here].”
![Page 43: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/43.jpg)
Avoid ambiguity
Look at the survey from every angle: are there ways that someone could interpret a question to have two meanings?
X "What is your income?"
X "How many people are there in your household?"
![Page 44: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/44.jpg)
Avoid ambiguity
Look at the survey from every angle: are there ways that someone could interpret a question to have two meanings?
X "What is your income?"
O "What is your income before taxes? Include salary as well as other sources of income."
X "How many people are there in your household?"
O "Including yourself, how many people are there in your household?"
![Page 45: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/45.jpg)
Confusing phrasing
The respondent should not have to spend time re-reading/interpreting the question.
X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?"
It seems likely __ It seems unlikely __ I’m not sure __
![Page 46: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/46.jpg)
Confusing phrasing
The respondent should not have to spend time re-reading/interpreting the question.
X "Does it seem likely or does it seem unlikely to you that you would use a Maker Space if the library had one?"
It seems likely __ It seems unlikely __ I’m not sure __
O "If the library had a Maker Space, would you use it?"
Yes__ No__ Unsure__
![Page 47: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/47.jpg)
Avoid double barreled questions
A question that introduces two or more issues with the expectation of a single response
X "Is our staff friendly and professional?"
![Page 48: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/48.jpg)
Avoid double barreled questions
A question that introduces two or more issues with the expectation of a single response
X "Is our staff friendly and professional?"
O Question1: "Is our staff friendly?"
O Question 2: "Is our staff professional?"
![Page 49: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/49.jpg)
Manipulative information
Certain questions require some background. Be careful that explanatory statements do not unduly influence responses.
X The county government spends approximately $10 per resident annually on landscaping public areas. Do you believe that the county
government is adequately allocating funds for our library by designating only $1.15 per resident?"
![Page 50: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/50.jpg)
Manipulative information
Certain questions require some background. Be careful that explanatory statements do not unduly influence responses.
X The county government spends approximately $10 per resident annually on landscaping public areas. Do you believe that the county
government is adequately allocating funds for our library by designating only $1.15 per resident?"
O "Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per resident annually?"
![Page 51: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/51.jpg)
Manipulative informationWe often are interested to know how knowledge of the difference in spending might affect responses, first ask straightforward, then with additional info.
O "Do you believe that the county government is adequately allocating funds for our library by designating $1.15 per
resident annually?" O "If you were to learn that the county government spends
approximately $10 per resident annually on landscaping public areas, would that change your opinion about the
adequacy of allocating $1.15 per resident annually to the library?"
![Page 52: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/52.jpg)
Order of response options
Often there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer, or have software randomize them.
X Group study rooms
Digital media lab
Laptop lending
E-books
Printing/copying
![Page 53: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/53.jpg)
Order of response options
Often there is a logical, inherent order. If order is irrelevant, list choices alphabetically so respondents don't assume answers at the top are more important to the interviewer, or have software randomize them.
X Group study rooms O Digital media lab
Digital media lab E-books
Laptop lending Group study rooms
E-books Laptop lending
Printing/copying Printing/copying
![Page 54: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/54.jpg)
Interval categories
Do not allow to overlap. Provide an unbounded final category if appropriate.
X Age 0-10
Age 10-15
Age 15-20
Age 20-50
Age 50-75
![Page 55: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/55.jpg)
Interval categories
Do not allow to overlap. Provide an unbounded final category if appropriate.
X Age 0-10 O Age 0-9
Age 10-15 Age 10-19
Age 15-20 Age 20-29
Age 20-50 Age 30-39
Age 50-75 Age 40-49
Age 50+
![Page 56: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/56.jpg)
Multiple response clarification
Sometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear when/if you are allowing multiple!
X For which of the following reasons do you use the library?
X__ Y__ Z__
![Page 57: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/57.jpg)
Multiple response clarification
Sometimes we allow respondents to choose only one option and sometimes we let them choose multiple. Be very clear when/if you are allowing multiple!
X For which of the following reasons do you use the library?
X__ Y__ Z__
O For which of the following reasons do you use the library? Choose
all that apply.
X__ Y__ Z__
![Page 58: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/58.jpg)
Appropriate response choices
– Surveys can be frustrating when the questions are fixed response without appropriate answer choices.
– Provide answer choices such as “Don’t know,” “N/A,” “Unsure,” and “Other” where appropriate.
X Does the laptop lending program meet your needs?
Yes__ No__
![Page 59: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/59.jpg)
Appropriate response choices
– Surveys can be frustrating when the questions are fixed response without appropriate answer choices.
– Provide answer choices such as “Don’t know,” “N/A,” “Unsure,” and “Other” where appropriate.
X Does the laptop lending program meet your needs?
Yes__ No__
O Does the laptop lending program meet your needs?
Completely__ Partially __ Not at all__ N/A__ [or “I’ve never used this program__”]
![Page 60: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/60.jpg)
Images: www.clustershot.com/garysimmons/photo649731 and Image: https://flic.kr/p/coyARS
![Page 61: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/61.jpg)
Closed responses
Advantages
• Uniform response set facilitates comparison, choices clarify meaning of question, reminder of alternatives, pre-establishment for sensitive questions, increased response rate and speed
![Page 62: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/62.jpg)
Closed responses
Advantages
• Uniform response set facilitates comparison, choices clarify meaning of question, reminder of alternatives, pre-establishment for sensitive questions, increased response rate and speed
Disadvantages
• Random selection, “closest representation” issue, loss of distinction
![Page 63: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/63.jpg)
Open responses
Advantages
• Allow for deep explanations, only way to get responses you wouldn’t otherwise know
![Page 64: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/64.jpg)
Open responses
Advantages
• Allow for deep explanations, only way to get responses you wouldn’t otherwise know
Disadvantages
• Requires communication skills from respondents, more time consuming for respondent and analysis
![Page 65: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/65.jpg)
Compromise
Would you recommend this search results screen to a friend or colleague?
Yes
Maybe
No
Why, or why not?
![Page 67: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/67.jpg)
Writing Outcome Measures
Emily PlagmanProject Manager
Public Library Association
![Page 68: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/68.jpg)
Example: Build-a-Budget
Program Goals
Intended result of Building a Budget
• Education
• Skill Building
Desired Outcomes
Benefits to the patron based on education & skill building
• Knowledge
• Skills
• Behaviors
Outcome Measures
Did the program increase or improve budgeting knowledge/skills/behaviors for the patron?
1.I understand the value of having a personal budget.
2.I have created a budget using the Excel template.
3.I have used my budget to: (check all that apply)
• Categorize my spending
• Track my spending for at least one month
• Identify at least one way to save money
• None of the above
• Other (please specify)
![Page 69: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/69.jpg)
Writing Your Own Outcome Survey
Putting it together:
• Title your survey
• Include program
name in title or
description
• Offer introductory
statement describing
the “why”
• Add in outcome
statements
• Measure!
![Page 70: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/70.jpg)
Guidelines: Additional Resources
![Page 71: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/71.jpg)
Guidelines: Additional Resources
www.ProjectOutcome.org
login ->
resources ->
from the field->
outcome
measurement
guidelines ->
developing
outcome
measures
![Page 72: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/72.jpg)
Peer Discussion: Program Planning
Community.projectoutcome.org/t/program-planning/411
![Page 73: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/73.jpg)
Announcements
Q&A
Emily PlagmanProject Manager
Public Library Association
![Page 74: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/74.jpg)
Visit Outcome Measurement Guidelines
login ->
resources ->
from the field ->
outcome
measurement
guidelines
www.ProjectOutcome.org
![Page 75: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/75.jpg)
2018 Webinars
Outcome Measurement Guidelines
▪ Developing their own outcome measures
▪ Implementing data collection methods
▪ Measuring outcome data over time
▪ Developing strategies for working with partners on
outcome measurement projects
www.ProjectOutcome.org
![Page 76: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/76.jpg)
What’s Next: Ready, Set, Measure!
▪ Register for FREE at www.projectoutcome.org
▪ Tutorial videos to help you get started
▪ Case studies to learn more about libraries taking action
![Page 77: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/77.jpg)
Book a Regional Training!
Includes:
▪ Full-day, in-person
workshop (60 ppl max)
▪ Expert trainer
▪ Workshop materials
▪ Quarterly online forums
exclusively for regional
training participants
▪ Interested? Email:
![Page 78: How to Develop Outcome Measures & Design Effective Surveys · 2018. 4. 16. · Quick & simple surveys Easy-to-use survey tool Custom data reports Interactive data dashboards ... Build-a-Budget](https://reader035.fdocuments.us/reader035/viewer/2022063007/5fbb08149adb20458e006adc/html5/thumbnails/78.jpg)
Questions?
Joyce ChapmanAssessment Analyst & Consultant
Assessment & User Experience Dept.
Duke University Libraries
Emily PlagmanManager, Impact & Advocacy
Public Library Association