Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 1
Data Collection and Aggregation
Presented byAmeriCorps Program Staff
and JBS International
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 2
Learning Outcomes
As a result of this session, participants will:• Understand core concepts associated with data
collection and aggregation;• Better understand how to assess the quality of your
current data collection tools and systems; and• Identify upgrades you need to make to insure
rigorous data collection and meaningful reporting.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 3
Agenda
• What are You Trying to Measure?• Are You Measuring it the Best Way?• Measuring Outcomes
– Data Collection Method Considerations – Instrument Considerations
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 4
Food for Thought
• Why are you doing the intervention?• What change do you want to create?• Can you measure the change/outcome?• If you can’t measure the outcome, are you sure you
are doing the right thing?
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 5
Now What?
You have identified the desired change (outcome) through your program design and theory of change….
Are you clear about WHAT to measure?AND
Are you measuring it the “BEST” way?
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 6
What are Your Desired Outcomes?
• What question are you trying to answer? What change are you expecting?– Did you select one of the CNCS priority measures?– Did you create your own program-specific outcome?
Either way…the data collection and aggregation basics are the same.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 7
Driving School Example
• Intervention: ABS driving school gives a 10 week course that meets twice a week for 60 minutes that include classroom-based and on road lessons on driving skills
• Desired Outcome: Students have basic driving proficiency.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 8
Real Life Example: Driving Test
Question: Do you like driving?Answer: I LOVE driving!!
“Do you like driving?” gets information about an attitude – not skill level.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 9
Real Life Example: Driving Test
Question: Do you think you are a skilled driver?
Answer: I think I am a GREAT driver!
“Are you a skilled driver” gets information about self perception, a thought – not actual skill level.Self-ratings are subjective NOT objective.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 10
Real Life Example: Driving Test
Question: Do you know the state driving laws ?
Answer: I got 100% correct on my written driver’s test!
Knowing state driving laws reflects knowledge – not actual skill level even though it is objective.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 11
Real Life Example: Driving Test
Question: Did you pass your road test?Answer: YES!
An on-road driving test DOES measure skill level or proficiency and is objective.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 12
What kind of information do you want?
• Subjective - includes an element of opinion or personal feeling.– Example: how someone feels about driving, confidence
• Objective - is not dependent upon opinions or personal feelings. It is based on facts that are observable and measurable. – Example: knowledge of driving laws and skill driving
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 13
How Do You Choose Your Method?
• Choice depends on: – what you want to measure; and.– the situation (i.e. resources for data collection/aggregation,
site/partner agreements/restrictions, etc.)• Each method is more appropriate in some situations than
others (e.g., age, language, content sensitivity, etc.)• Will it get you high quality data?
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 14
Does Your Method Measure Your Outcome?
• Commonly used data collection methods– Surveys– Pre/post tests– Observations– Standardized tests– Interviews– Focus Groups– Diaries, Journals, Self-reported Checklists – Available secondary data
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 15
How Do You Choose Your Instrument?• Whichever method you select, what instrument will
you use?• “Borrow” vs. develop• Does it ask the “right” questions to get at your desired
outcome?• Does it have all the necessary components?• What information will each question yield? • How will you use information, if not related to outcome?
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 16
What is “Qualitative” Data?
• Describes or characterizes through words• Focuses on meaning, experience or attitudes • Collected through focus groups, interviews, opened
ended questionnaire items, and other less structured situations.
• Not the same as anecdotal Information
Qualitative → Quality
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 17
What is “Quantitative” data?
• Focus on numbers and frequencies.• Data which can be measured• Length, height, area, volume, weight, speed, time,
temperature, humidity, sound levels, cost, ages, scores, etc.
Quantitative → Quantity
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 18
Example: How Do You Feel?
Qualitative Data Quantitative DataHave you felt sad or depressed at all lately, or have you generally been in good spirits?
Thinking about the past week, how depressed would you say you have been on a scale from 0 to 10, where 0 means "not at
all" and 10 means "the most possible
I’m not at all depressed. I feel great! I love my new job. I’ve lost 20 pounds and feel healthier than I have in years.
0
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 19
Are You Asking the Question You Mean to Ask?
Does your instrument or data collection method help you measure your desired outcome?•Example:
– If desired outcome = improved academic performance• DON’T measure attendance or attitude toward school• DO measure improved proficiency in a subject
VALIDITY
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 20
Validity Example• Desired Outcome (O11): Economically disadvantaged
individuals transitioned into safe, healthy, affordable housing
• Data to collect (from NOFO):– An inspection report and certificate of occupancy, – proof of residence such as lease or mortgage, or– other verification from an external agency that the work
was completed and is being occupied might be used.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 21
Validity Example (cont.)• Which questions are most likely to yield the desired
information?– Do you feel safe in your new home?– Can you afford this house?– Do you like this house?– Do you have a lease or mortgage for this house? Who
holds the lease/mortgage? Please share a copy of the signed paperwork.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 22
What is a Standardized Test?• A test that:
– is administered and scored in a consistent or “standard” manner.
– has been validated externally on a randomly-selected population
• Need not be “high-stakes” tests, time-limited tests, or multiple choice tests– e.g., state administered proficiency tests generally should
NOT be used
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 23
Can It Be Repeated?
1. Does your instrument measure the same thing, the same way every time it is used?
2. Does every person collecting data use the instrument the same way? Have they been trained?
3. Are your instrument instructions clear so respondents have a similar frame to answer?
RELIABILITY
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 24
Reliability Example
• Intervention: Preschool children attend early childhood education programs providing school readiness activities in three areas (social/emotional development, literacy and numeracy skills) 4 days a week for 4 hours/day
• Desired Outcome (ED23): Children demonstrate gains in social and/or emotional development.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 25
Reliability Example (cont.)
• Observations:– Open-ended Question: Does the child seem well adjusted
and ready to attend kindergarten?
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 26
Instrument Pilot Testing
• Test before using - with small number of people similar to those who will respond
• Pilot test analysis should look at: – Were the questions clear enough? Did people understand
what you were asking?– Do the answers seem appropriate given what you were
asking?
• Make revisions based on results of pilot
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 27
Is Something Systematically Off?Bias:• Problems with WHO you ask
– sampling bias– response rates, etc.
• Problems with HOW you ask– method inappropriate – construction of your instrument
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 28
What is Bias?• Measurement bias occurs when information
collected for use is inaccurate.• Bias may be introduced by poor measurement design
or poor data collection.• Bias cannot be “controlled for” at the analysis stage.• Bias risks readers drawing conclusions that are
systematically different from the truth.• Bias can lead to an over or underestimation of an
effect.
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 29
Revisit: What to Look For?
• Are you measuring what counts/matters? • Is your measurement approach credible?• Are your instruments valid?• Are your instruments reliable?• Are your measurements precise?
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 30
Instrument Mapping• Look at each question on your data collection tool and ask:
– Does this help us measure the desired outcome?• Is there one question? More than one?
– What kind of data will we get? • Subjective? Objective?• Quantitative? Qualitative?
– If it doesn’t measure the outcome, do we really need to ask it? How will we use the answer?
• Nice to know but won’t use it? Internal use? – How will we analyze this?– What is our target? How much change is “enough”?
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 31
Target Example: Pre/Post test
“Mentored children will enhance developmental assets in the areas of social competence and positive identity.”
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 32
Do You Have a Summary Sheet w/Target?
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 33
After You Get the Data, then What?
• Do you have a plan for who and how the data collected will be aggregated and summarized?– Given the types of questions on your tool(s), is this
realistic?• Which data DIRECTLY relate to the desired outcome?
NOTE: Just because you asked it, doesn’t mean it helps you report on your outcome…don’t confuse people!
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 34
Data Collection and Aggregation
04/10/23 Data Collection and Aggregation 35
Top Related