Measurement Issues General steps –Determine concept –Decide best way to measure –What...
-
Upload
christal-quinn -
Category
Documents
-
view
218 -
download
0
Transcript of Measurement Issues General steps –Determine concept –Decide best way to measure –What...
![Page 1: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/1.jpg)
Measurement Issues
• General steps– Determine concept– Decide best way to measure–What indicators are available– Select intermediate, alternate or indirect
measures
![Page 2: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/2.jpg)
Measurement Issues
• General steps– Consider limitations of measures
selected– Collect or secure info/data– Summarize findings in writing
![Page 3: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/3.jpg)
• What is the relation between concepts, variables, instruments & measures?
![Page 4: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/4.jpg)
Concepts
• Program is based on conceptual basis of why people behave the way they do
![Page 5: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/5.jpg)
• Why do you think people behave the way they do?
• Think of food and nutrition issues
![Page 6: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/6.jpg)
Variables
• A theory has variables
• Variables define concepts
• Theory states how the variables interact or are related
![Page 7: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/7.jpg)
Variables
• Variables of the theory are what you measure
• Variables are the verbal or written abstractions of the ideas that exist in the mind
![Page 8: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/8.jpg)
• Why should an intervention be based on a theory?
![Page 9: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/9.jpg)
Why use theory?
• Know what you are to address in the intervention
• Makes evaluation easier
• Know what to measure to evaluate
![Page 10: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/10.jpg)
• Figure 6.1 A simple social learning theory model for reducing salt in the diet
• Comes next
![Page 11: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/11.jpg)
Fig. 6.1 Social learning theory
![Page 12: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/12.jpg)
• Need measurements and instruments to assess changes in the variables of interest
![Page 13: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/13.jpg)
Instruments
• Something that produces a measure of an object
• Series of questions to measure the variable, concept
• Includes instructions
![Page 14: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/14.jpg)
Measures
• The numbers that come from the person answering questions on the instrument
![Page 15: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/15.jpg)
• Figure 6.2 Relation among models, variables, measures, and an instrument
• Comes next
![Page 16: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/16.jpg)
Fig. 6.2
![Page 17: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/17.jpg)
• Based on why you think people behave the way the do, list possible variables to consider to measure this variable.
• What might be variables of the social learning theory?
![Page 18: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/18.jpg)
• What about variables that would verify if a change has or has not taken place?
![Page 19: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/19.jpg)
• Figure 6.1 A simple social learning theory model for reducing salt in the diet
• Comes next
• See how the program links with the theory & what measure
![Page 20: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/20.jpg)
Fig. 6.1 Social learning theory
![Page 21: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/21.jpg)
Reliability
• The extent to which an instrument will produce the same result (measure or score) if applied two different or more times.
![Page 22: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/22.jpg)
Reliability
• X = T + E
• X is measure
• T is true value
• E is random error
![Page 23: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/23.jpg)
Reliability
• Measurement error reduces the ability to have reliable and valid results.
![Page 24: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/24.jpg)
Reliability
• Random error is all chance factors that confound the measurement.
• Always present
• Effects reliability but doesn’t bias results
![Page 25: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/25.jpg)
Reliability
• Figure 6.5 Distribution of scores of multiple applications of a test with random error
• A is true score
• a is measure
![Page 26: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/26.jpg)
Fig. 6.5 Distribution of scores of multiple applications of a test and random error
![Page 27: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/27.jpg)
Distribution
• Can have the same mean with two different distributions
• Figure 6.6 next
![Page 28: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/28.jpg)
Fig. 6.6 Two distributions of scores aroundthe true mean
![Page 29: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/29.jpg)
• Which distribution has less variability?
• Which distribution has less random error?
![Page 30: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/30.jpg)
Sources of Random Error
• Day-to-day variability
• Confusing instructions
• Unclear instrument
• Sloppy data collector
![Page 31: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/31.jpg)
Sources of Random Error
• Distracting environment
• Respondents
• Data-management error
![Page 32: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/32.jpg)
• What can you do to reduce random error and increase reliability?
![Page 33: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/33.jpg)
Variability & the Subject
• What you want to measure will vary from day to day and within the person
![Page 34: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/34.jpg)
Variability & the Subject
• Intraindividual variability– variability among the true scores within
a person over time
![Page 35: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/35.jpg)
• Figure 6.7 True activity scores (A, B, C) for 3 days with three measures (a, b, c) per day
• Comes next
![Page 36: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/36.jpg)
Fig. 6.7 True activity (A, B, C) for 3 days with three measures (a, b, c) per day
![Page 37: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/37.jpg)
Variability & the Subject
• Interindividual variability– variability between each person in the
sample
![Page 38: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/38.jpg)
• Figure 6.8 Interindividual (A, X) and intraindividual (A1, A2, A3) variability for two people (A, X) in level of physical activity
• Comes next
![Page 39: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/39.jpg)
Fig. 6.8 Interindividual (A, X) and intraindividual (A1, A2, A3) variability for two people (A, X) in levelof physical activity
![Page 40: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/40.jpg)
Assessing Reliability
• Need to know the reliability of your instruments
• Reliability coefficient of 1 is highest, no error
• Reliability coefficient of 0 is lowest, all error
![Page 41: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/41.jpg)
Factors of Reliability
• Type of instrument– observer– self-report
• Times instrument applied– same time– different time
![Page 42: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/42.jpg)
• Figure 6.9 Types of reliability
• Comes next
![Page 43: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/43.jpg)
Fig. 6.9 Types of reliability
![Page 44: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/44.jpg)
Assessing Reliability
• Interobserver reliability– have 2 different observers rate same
action at same time– reproducibility
![Page 45: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/45.jpg)
Assessing Reliability
• Intraobserver reliability– 1 observer assesses same person at
two different times– video tape the action & practice
![Page 46: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/46.jpg)
Assessing Reliability
• Repeat method– self-report or survey– repeat the same item/question at 2
points in survey
![Page 47: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/47.jpg)
Assessing Reliability
• Internal consistency– average inter-item correlation among
items in an instrument that are cognitively related
![Page 48: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/48.jpg)
Assessing Reliability
• Internal consistency– Cronbach’s alpha– 0.70 & above a good score
![Page 49: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/49.jpg)
Assessing Reliability
• Test-retest reliability (internal consistency method)– same survey/test at 2 different times to
same person
![Page 50: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/50.jpg)
Validity
• Degree to which an instrument measures what the evaluator wants it to measure
![Page 51: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/51.jpg)
Bias
• Systematic error that produces a systematic difference between an obtained score and the true score
• Bias threatens validity
![Page 52: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/52.jpg)
Bias
• Figure 6.10 Distribution of scores of multiple applications of a test with systematic error
• Comes next
![Page 53: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/53.jpg)
Fig. 6.10 Distribution of scores of multiple applications of a test with systematic error
![Page 54: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/54.jpg)
• What will basis do to your ability to make conclusions about your subjects?
![Page 55: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/55.jpg)
• Figure 6.11 Effect of bias on conclusions
• Comes next
![Page 56: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/56.jpg)
Fig. 6.11 Effect of bias on conclusions
![Page 57: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/57.jpg)
Types of Validity
• Face
• Content
• Criterion
![Page 58: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/58.jpg)
Face Validity
• Describes the extent to which an instrument appears to measure what it is suppose to measure
• How many veg did you eat yesterday?
![Page 59: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/59.jpg)
Content Validity
• Extent to which an instrument is expected to cover several domains of the content
• Consult a group of experts
![Page 60: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/60.jpg)
Criterion Validity
• How accurate is a less costly way to measure the variable compared to the valid and more expensive instrument
![Page 61: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/61.jpg)
What can lower validity?
• Guinea pig effect– awareness of being tested
• Role selection– awareness of being measured may
make people feel they have to play a role
![Page 62: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/62.jpg)
What can lower validity?
• Measurement as a change agent– act of measurement could change
future behavior
![Page 63: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/63.jpg)
What can lower validity?
• Response sets– respond in a predictable way that has
nothing to do with the questions
![Page 64: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/64.jpg)
What can lower validity?
• Interviewer effects– characteristics of the interviewer affects
the receptivity and answers of the respondent
![Page 65: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/65.jpg)
What can lower validity?
• Population restrictions– if people can’t use the method of data
collection, can’t generalize to others
![Page 66: Measurement Issues General steps –Determine concept –Decide best way to measure –What indicators are available –Select intermediate, alternate or indirect.](https://reader036.fdocuments.us/reader036/viewer/2022081603/5697bf851a28abf838c87755/html5/thumbnails/66.jpg)
• End of reliability and validity
• Questions
• Look at CNEP Survey