Measurement Issues

• General steps– Determine concept– Decide best way to measure–What indicators are available– Select intermediate, alternate or indirect

measures

Measurement Issues

• General steps– Consider limitations of measures

selected– Collect or secure info/data– Summarize findings in writing

• What is the relation between concepts, variables, instruments & measures?

Concepts

• Program is based on conceptual basis of why people behave the way they do

• Why do you think people behave the way they do?

• Think of food and nutrition issues

Variables

• A theory has variables

• Variables define concepts

• Theory states how the variables interact or are related

Variables

• Variables of the theory are what you measure

• Variables are the verbal or written abstractions of the ideas that exist in the mind

• Why should an intervention be based on a theory?

Why use theory?

• Know what you are to address in the intervention

• Makes evaluation easier

• Know what to measure to evaluate

• Figure 6.1 A simple social learning theory model for reducing salt in the diet

• Comes next

Fig. 6.1 Social learning theory

• Need measurements and instruments to assess changes in the variables of interest

Instruments

• Something that produces a measure of an object

• Series of questions to measure the variable, concept

• Includes instructions

Measures

• The numbers that come from the person answering questions on the instrument

• Figure 6.2 Relation among models, variables, measures, and an instrument

• Comes next

Fig. 6.2

• Based on why you think people behave the way the do, list possible variables to consider to measure this variable.

• What might be variables of the social learning theory?

• What about variables that would verify if a change has or has not taken place?

• Figure 6.1 A simple social learning theory model for reducing salt in the diet

• Comes next

• See how the program links with the theory & what measure

Fig. 6.1 Social learning theory

Reliability

• The extent to which an instrument will produce the same result (measure or score) if applied two different or more times.

Reliability

• X = T + E

• X is measure

• T is true value

• E is random error

Reliability

• Measurement error reduces the ability to have reliable and valid results.

Reliability

• Random error is all chance factors that confound the measurement.

• Always present

• Effects reliability but doesn’t bias results

Reliability

• Figure 6.5 Distribution of scores of multiple applications of a test with random error

• A is true score

• a is measure

Fig. 6.5 Distribution of scores of multiple applications of a test and random error

Distribution

• Can have the same mean with two different distributions

• Figure 6.6 next

Fig. 6.6 Two distributions of scores aroundthe true mean

• Which distribution has less variability?

• Which distribution has less random error?

Sources of Random Error

• Day-to-day variability

• Confusing instructions

• Unclear instrument

• Sloppy data collector

Sources of Random Error

• Distracting environment

• Respondents

• Data-management error

• What can you do to reduce random error and increase reliability?

Variability & the Subject

• What you want to measure will vary from day to day and within the person

Variability & the Subject

• Intraindividual variability– variability among the true scores within

a person over time

• Figure 6.7 True activity scores (A, B, C) for 3 days with three measures (a, b, c) per day

• Comes next

Fig. 6.7 True activity (A, B, C) for 3 days with three measures (a, b, c) per day

Variability & the Subject

• Interindividual variability– variability between each person in the

sample

• Figure 6.8 Interindividual (A, X) and intraindividual (A1, A2, A3) variability for two people (A, X) in level of physical activity

• Comes next

Fig. 6.8 Interindividual (A, X) and intraindividual (A1, A2, A3) variability for two people (A, X) in levelof physical activity

Assessing Reliability

• Need to know the reliability of your instruments

• Reliability coefficient of 1 is highest, no error

• Reliability coefficient of 0 is lowest, all error

Factors of Reliability

• Type of instrument– observer– self-report

• Times instrument applied– same time– different time

• Figure 6.9 Types of reliability

• Comes next

Fig. 6.9 Types of reliability

Assessing Reliability

• Interobserver reliability– have 2 different observers rate same

action at same time– reproducibility

Assessing Reliability

• Intraobserver reliability– 1 observer assesses same person at

two different times– video tape the action & practice

Assessing Reliability

• Repeat method– self-report or survey– repeat the same item/question at 2

points in survey

Assessing Reliability

• Internal consistency– average inter-item correlation among

items in an instrument that are cognitively related

Assessing Reliability

• Internal consistency– Cronbach’s alpha– 0.70 & above a good score

Assessing Reliability

• Test-retest reliability (internal consistency method)– same survey/test at 2 different times to

same person

Validity

• Degree to which an instrument measures what the evaluator wants it to measure

Bias

• Systematic error that produces a systematic difference between an obtained score and the true score

• Bias threatens validity

Bias

• Figure 6.10 Distribution of scores of multiple applications of a test with systematic error

• Comes next

Fig. 6.10 Distribution of scores of multiple applications of a test with systematic error

• What will basis do to your ability to make conclusions about your subjects?

• Figure 6.11 Effect of bias on conclusions

• Comes next

Fig. 6.11 Effect of bias on conclusions

Types of Validity

• Face

• Content

• Criterion

Face Validity

• Describes the extent to which an instrument appears to measure what it is suppose to measure

• How many veg did you eat yesterday?

Content Validity

• Extent to which an instrument is expected to cover several domains of the content

• Consult a group of experts

Criterion Validity

• How accurate is a less costly way to measure the variable compared to the valid and more expensive instrument

What can lower validity?

• Guinea pig effect– awareness of being tested

• Role selection– awareness of being measured may

make people feel they have to play a role

What can lower validity?

• Measurement as a change agent– act of measurement could change

future behavior

What can lower validity?

• Response sets– respond in a predictable way that has

nothing to do with the questions

What can lower validity?

• Interviewer effects– characteristics of the interviewer affects

the receptivity and answers of the respondent

What can lower validity?

• Population restrictions– if people can’t use the method of data

collection, can’t generalize to others

• End of reliability and validity

• Questions

• Look at CNEP Survey