Assessment Fellows Training

Brian Clark, Stan DuraAssessment Fellows Meeting

11/21/14

Brian Clark, Stan Dura 1/17/14

Training Agenda

• Review of Foundational Concepts• Prediction, Control and Understanding– Increasingly complex models

• Univariate = 1 variable– Descriptive Statistics

• Bivariate = 2 variables– Correlations

• Multivariate = 3+ variables– Regresssion– Factor Analysis

• Discussion and Q & A

Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler

11.85

11.9

11.95

12

12.05

Chart Title

Series1

Measurement is imprecise

What “construct” are you measuring?

Survey and question design

and content

Responses

Is it really measuring it?Does it measure it the same each time?

Any errors in the questions or scales?Any bias in the questions asked or not asked?

Response set– -Are they representative?-How complete are they?-Is there non-response bias?

Over 80 cognitive and

memory biases

Surveys especially vulnerable

Cognitive Stages Definition Vulnerabilities

Stage 1 Comprehension Understanding the question as author intended

Unknown or misused words, ambiguity, complexity, length

Stage 2 Retrieval Search of memory for relevant information

Memory bias, recall error, fatigue

Stage 3 JudgmentConsiders information

retrieved, makes “guestimations” and decides

Social or political bias, “fuzzy logic,” error in “guesstimation”,

personal sensitivity

Stage 4 Response Provides the information Human error, incomplete response, wrong format

Respondents Prone to Error

Research, Assessment, & Program Evaluation

Research• Systematic collection of data• Generalize to larger populations• Based on Research question• Rigorous methodology• Used to test hypotheses• Contributes to the development

of theory and models

Assessment• Systematic collection of data• Generalize to larger population• Based on Assessment question• Borrows rigor when practical• Used to evaluate effectiveness• Contributes to judgments of the

quality of programs or activities

Foundational concepts:• Entity – the things we examine and assess• Property – the characteristics that describe them• Variable – Properties that vary between people• Prediction and Control – Predict variable behavior• Relationship – Understand variable behavior• Statistical techniques – Analyze variable behavior

Foundational Concepts“Things and Stuff”

Objects and properties– Humans

• Height, satisfaction, health, perceptions, emotions, beliefs, etc.

– Students (whether human or not )

• Age, GPA, knowledge, skills, engagement, class standing, etc.

– Physical Objects• Weight, chemical composition, density, etc.

– Forces• Magnitude, direction, etc.

Foundational Concepts“Things and Stuff”

Some variables are straightforward-

–Height is often measured in Inches–Weight is often measured in Kilograms–Age is often measured in years

Foundational Concepts“Things and Stuff”

Some variables are not-

– Satisfaction is often measured in… ???–Anger is often measured in… ???– Engagement is often measured in… ???

Foundational Concepts“Things and Stuff”

Foundational ConceptsPrediction and Control

Performance

Haste Practice

Foundational ConceptsPrediction and Control

Performance

Practice

If we did not Control for Haste, we would think all of this was

the result of Practice

Foundational ConceptsPrediction and Control

Performance

Practice

When really, this was the impact.

Haste

So…• Recognizing the relationships between

variables, and • controlling for them • is critical to understanding the impact

of any one of them.

Foundational ConceptsPrediction and Control

Foundational ConceptsModeling

When we conduct quantitative or qualitative analyses, we are constructing models

• That model can be very, very simple• Mean – this is how the average person did…• Mode – The most frequent score was X

• Or complex… • For every hour engaged in co-curricular activities, when controlling for HS

GPA, Class standing, and Major, retention likelihood increases by .45%

• Descriptive stats = the weakest and least informative models

Foundational ConceptsModeling

So when we assess, we need to Think of our Model

• What are the entities and properties we’re examining?• What are the variables involved? Operationalize them.• What is their relationship; how do they interact?• How simple or complex/accurate a model do I need?• How can we predict and control for these variables?

One variable models = Univariate

This is when we look at 1 property of an entity.• Age of person• Height of a building• Responses to a single question

Prediction, Control and UnderstandingOne Variable Models

One variable models = Univariate

These are generally very easy models to work with

Prediction, Control and UnderstandingOne Variable Models

Mean

The average of all measures of a variable…

“The average score of all students was a 68”

Prediction, Control and UnderstandingOne Variable Models

What does this tell us of other scores?

Mean

The average of all measures or scores

Prediction, Control and UnderstandingOne Variable Models

0 40 80 100

What does this tell us of other scores?

Median

The measure in the exact middle of all scores

Prediction, Control and UnderstandingOne Variable Models

1, 3, 4, 4, 6, 6, 7, 7, 7, 7, 8

Mean = 5.5 Median = 6

Mode

The most frequent measure observed

Prediction, Control and UnderstandingOne Variable Models

Mean = 5.5 Median = 6 Mode = 7

1, 3, 4, 4, 6, 6, 7, 7, 7, 7, 8

Measures of central tendency

Mean, median and mode are all measures of central tendency because…

They attempt to describe the entire data set by identifying a central position.

How well do they help us predict & understand?

Prediction, Control and UnderstandingOne Variable Models

Variance and Dispersion

When central tendency doesn’t help us understand, we can look at the dispersion or distribution of scores

Dispersion or Variance is the degree to which scores differ in relationship to each other

Prediction, Control and UnderstandingOne Variable Models

Variance and Dispersion

• Low Dispersion – scores are grouped tightly

Prediction, Control and UnderstandingOne Variable Models

LOW

Variance and Dispersion

• High Dispersion – scores are widely dispersed

Prediction, Control and UnderstandingOne Variable Models

HIGH

Variance and Dispersion

• Frequency distribution

Prediction, Control and UnderstandingOne Variable Models

Variance and Dispersion

• Probability distribution

• Grouped (frequency or probability)

Prediction, Control and UnderstandingOne Variable Models

Normal Distribution

Layperson’s definition: Mean is exactly in the middle, 50% of scores on each side.

Perfectly Normal Distribution is RARE

Prediction, Control and UnderstandingOne Variable Models

Normal Distribution

A critical assumption involved in most statistics is the concept that the scores are evenly distributed on each side of the mean

Thus, the more abnormal a distribution is, the more inappropriate it is to use inferential stats.

Prediction, Control and UnderstandingOne Variable Models

Normal Distribution

Prediction, Control and UnderstandingOne Variable Models

Distributions that are not normal

Prediction, Control and UnderstandingOne Variable Models

Distributions

The shape of a distribution tells us more than one score…

It can show us the pattern of scores.

How well does it help us predict & understand?

Prediction, Control and UnderstandingOne Variable Models

Standard Deviation

A measure that describes the dispersion of scores in relationship to their distance from the mean.

1 SD = 68.2% of all scores fall within 1 SD2 SD = 95.4% of all scores fall within 2 SD3 SD = 99.7% of all scores fall within 3 SD

Prediction, Control and UnderstandingOne Variable Models

Standard Deviation

Prediction, Control and UnderstandingOne Variable Models

Standard Deviation

Prediction, Control and UnderstandingOne Variable Models

Standard Deviation

Prediction, Control and UnderstandingOne Variable Models

Standard Deviation

Prediction, Control and UnderstandingOne Variable Models

How well does it help us predict & understand?

When relying on descriptive stats

• Avoid making any significant inferences • If you must make inferences, evaluate the

measure in relationship to the standard deviation.

Prediction, Control and UnderstandingTwo Variable Models

Performance

Practice

This also represents a correlational relationship

Prediction, Control and UnderstandingTwo Variable Models

Correlation

Prediction, Control and UnderstandingTwo Variable Models

0.5 1 1.5 2 2.5 3 3.5 4 4.50

0.51

1.52

2.53

3.54

4.5

HS and College GPA

College GPA

HS

GPA

This is called a linear relationship

Notice tight grouping – indicates a stronger correlation

Correlation

Prediction, Control and UnderstandingTwo Variable Models

How well does it help us predict & understand?

0.5 1 1.5 2 2.5 3 3.5 4 4.50123456789

10

GPA and Drinking Frequency

GPA

Drin

king

Fre

quen

cy

Notice looser grouping – indicates a weaker correlation

Three or more Variable Models = Multivariate

Become increasingly complex as the number of variables increases, but generally provide much better data and understanding

Regression – A measure of the strength of a variable’s relationship with many other variables.

Factor Analysis – Groups inter-correlated variables into factors where the variables are most correlated together while being less correlated to the other groupings- reduces the # of variables and tend to be more stable

Prediction, Control and UnderstandingThree or more Variable Models

Regression

Prediction, Control and UnderstandingThree or more Variable Models

Retention

Belonging

Academic Skills

Negative Social

Positive Social

Health

+.21

+.20

-.35

+.11

+.15

Retention

Factor Analysis

Prediction, Control and UnderstandingThree or more Variable Models

Strengths and Weaknesses

• Mean, Median, and Mode• Standard Deviation• Correlation• Regression• Factor Analysis

Discussion

What inferences can you draw from:

• Mean, Median, and Mode• Standard Deviation• Correlation• Regression• Factor Analysis

Discussion

Questions?Concerns?

Snide Comments?

(Practical Application Next)