1 Florida Board of Governors Student Affairs Committee September 24, 2003 Tallahassee, Florida.
Student Affairs Assessment Committee Training Part 2
Transcript of Student Affairs Assessment Committee Training Part 2
Assessment Fellows Training
Brian Clark, Stan DuraAssessment Fellows Meeting
11/21/14
Brian Clark, Stan Dura 1/17/14
Training Agenda
• Review of Foundational Concepts• Prediction, Control and Understanding– Increasingly complex models
• Univariate = 1 variable– Descriptive Statistics
• Bivariate = 2 variables– Correlations
• Multivariate = 3+ variables– Regresssion– Factor Analysis
• Discussion and Q & A
Ruler 1 Ruler 2 Ruler 3 Ruler 4 Ruler 5 Ruler 6 Ruler 7 Ruler 8 Ruler 9 Ruler 10 Laser Ruler
11.85
11.9
11.95
12
12.05
Chart Title
Series1
Measurement is imprecise
What “construct” are you measuring?
Survey and question design
and content
Responses
Is it really measuring it?Does it measure it the same each time?
Any errors in the questions or scales?Any bias in the questions asked or not asked?
Response set– -Are they representative?-How complete are they?-Is there non-response bias?
Over 80 cognitive and
memory biases
Surveys especially vulnerable
Cognitive Stages Definition Vulnerabilities
Stage 1 Comprehension Understanding the question as author intended
Unknown or misused words, ambiguity, complexity, length
Stage 2 Retrieval Search of memory for relevant information
Memory bias, recall error, fatigue
Stage 3 JudgmentConsiders information
retrieved, makes “guestimations” and decides
Social or political bias, “fuzzy logic,” error in “guesstimation”,
personal sensitivity
Stage 4 Response Provides the information Human error, incomplete response, wrong format
Respondents Prone to Error
Research, Assessment, & Program Evaluation
Research• Systematic collection of data• Generalize to larger populations• Based on Research question• Rigorous methodology• Used to test hypotheses• Contributes to the development
of theory and models
Assessment• Systematic collection of data• Generalize to larger population• Based on Assessment question• Borrows rigor when practical• Used to evaluate effectiveness• Contributes to judgments of the
quality of programs or activities
Foundational concepts:• Entity – the things we examine and assess• Property – the characteristics that describe them• Variable – Properties that vary between people• Prediction and Control – Predict variable behavior• Relationship – Understand variable behavior• Statistical techniques – Analyze variable behavior
Foundational Concepts“Things and Stuff”
Objects and properties– Humans
• Height, satisfaction, health, perceptions, emotions, beliefs, etc.
– Students (whether human or not )
• Age, GPA, knowledge, skills, engagement, class standing, etc.
– Physical Objects• Weight, chemical composition, density, etc.
– Forces• Magnitude, direction, etc.
Foundational Concepts“Things and Stuff”
Some variables are straightforward-
–Height is often measured in Inches–Weight is often measured in Kilograms–Age is often measured in years
Foundational Concepts“Things and Stuff”
Some variables are not-
– Satisfaction is often measured in… ???–Anger is often measured in… ???– Engagement is often measured in… ???
Foundational Concepts“Things and Stuff”
Foundational ConceptsPrediction and Control
Performance
Haste Practice
Foundational ConceptsPrediction and Control
Performance
Practice
If we did not Control for Haste, we would think all of this was
the result of Practice
Foundational ConceptsPrediction and Control
Performance
Practice
When really, this was the impact.
Haste
So…• Recognizing the relationships between
variables, and • controlling for them • is critical to understanding the impact
of any one of them.
Foundational ConceptsPrediction and Control
Foundational ConceptsModeling
When we conduct quantitative or qualitative analyses, we are constructing models
• That model can be very, very simple• Mean – this is how the average person did…• Mode – The most frequent score was X
• Or complex… • For every hour engaged in co-curricular activities, when controlling for HS
GPA, Class standing, and Major, retention likelihood increases by .45%
• Descriptive stats = the weakest and least informative models
Foundational ConceptsModeling
So when we assess, we need to Think of our Model
• What are the entities and properties we’re examining?• What are the variables involved? Operationalize them.• What is their relationship; how do they interact?• How simple or complex/accurate a model do I need?• How can we predict and control for these variables?
One variable models = Univariate
This is when we look at 1 property of an entity.• Age of person• Height of a building• Responses to a single question
Prediction, Control and UnderstandingOne Variable Models
One variable models = Univariate
These are generally very easy models to work with
Prediction, Control and UnderstandingOne Variable Models
Mean
The average of all measures of a variable…
“The average score of all students was a 68”
Prediction, Control and UnderstandingOne Variable Models
What does this tell us of other scores?
Mean
The average of all measures or scores
Prediction, Control and UnderstandingOne Variable Models
0 40 80 100
What does this tell us of other scores?
Median
The measure in the exact middle of all scores
Prediction, Control and UnderstandingOne Variable Models
1, 3, 4, 4, 6, 6, 7, 7, 7, 7, 8
Mean = 5.5 Median = 6
Mode
The most frequent measure observed
Prediction, Control and UnderstandingOne Variable Models
Mean = 5.5 Median = 6 Mode = 7
1, 3, 4, 4, 6, 6, 7, 7, 7, 7, 8
Measures of central tendency
Mean, median and mode are all measures of central tendency because…
They attempt to describe the entire data set by identifying a central position.
How well do they help us predict & understand?
Prediction, Control and UnderstandingOne Variable Models
Variance and Dispersion
When central tendency doesn’t help us understand, we can look at the dispersion or distribution of scores
Dispersion or Variance is the degree to which scores differ in relationship to each other
Prediction, Control and UnderstandingOne Variable Models
Variance and Dispersion
• Low Dispersion – scores are grouped tightly
Prediction, Control and UnderstandingOne Variable Models
LOW
Variance and Dispersion
• High Dispersion – scores are widely dispersed
Prediction, Control and UnderstandingOne Variable Models
HIGH
Variance and Dispersion
• Frequency distribution
Prediction, Control and UnderstandingOne Variable Models
Variance and Dispersion
• Probability distribution
• Grouped (frequency or probability)
Prediction, Control and UnderstandingOne Variable Models
Normal Distribution
Layperson’s definition: Mean is exactly in the middle, 50% of scores on each side.
Perfectly Normal Distribution is RARE
Prediction, Control and UnderstandingOne Variable Models
Normal Distribution
A critical assumption involved in most statistics is the concept that the scores are evenly distributed on each side of the mean
Thus, the more abnormal a distribution is, the more inappropriate it is to use inferential stats.
Prediction, Control and UnderstandingOne Variable Models
Normal Distribution
Prediction, Control and UnderstandingOne Variable Models
Distributions that are not normal
Prediction, Control and UnderstandingOne Variable Models
Distributions
The shape of a distribution tells us more than one score…
It can show us the pattern of scores.
How well does it help us predict & understand?
Prediction, Control and UnderstandingOne Variable Models
Standard Deviation
A measure that describes the dispersion of scores in relationship to their distance from the mean.
1 SD = 68.2% of all scores fall within 1 SD2 SD = 95.4% of all scores fall within 2 SD3 SD = 99.7% of all scores fall within 3 SD
Prediction, Control and UnderstandingOne Variable Models
Standard Deviation
Prediction, Control and UnderstandingOne Variable Models
Standard Deviation
Prediction, Control and UnderstandingOne Variable Models
Standard Deviation
Prediction, Control and UnderstandingOne Variable Models
Standard Deviation
Prediction, Control and UnderstandingOne Variable Models
How well does it help us predict & understand?
When relying on descriptive stats
• Avoid making any significant inferences • If you must make inferences, evaluate the
measure in relationship to the standard deviation.
Prediction, Control and UnderstandingTwo Variable Models
Performance
Practice
This also represents a correlational relationship
Prediction, Control and UnderstandingTwo Variable Models
Correlation
Prediction, Control and UnderstandingTwo Variable Models
0.5 1 1.5 2 2.5 3 3.5 4 4.50
0.51
1.52
2.53
3.54
4.5
HS and College GPA
College GPA
HS
GPA
This is called a linear relationship
Notice tight grouping – indicates a stronger correlation
Correlation
Prediction, Control and UnderstandingTwo Variable Models
How well does it help us predict & understand?
0.5 1 1.5 2 2.5 3 3.5 4 4.50123456789
10
GPA and Drinking Frequency
GPA
Drin
king
Fre
quen
cy
Notice looser grouping – indicates a weaker correlation
Three or more Variable Models = Multivariate
Become increasingly complex as the number of variables increases, but generally provide much better data and understanding
Regression – A measure of the strength of a variable’s relationship with many other variables.
Factor Analysis – Groups inter-correlated variables into factors where the variables are most correlated together while being less correlated to the other groupings- reduces the # of variables and tend to be more stable
Prediction, Control and UnderstandingThree or more Variable Models
Regression
Prediction, Control and UnderstandingThree or more Variable Models
Retention
Belonging
Academic Skills
Negative Social
Positive Social
Health
+.21
+.20
-.35
+.11
+.15
Retention
Factor Analysis
Prediction, Control and UnderstandingThree or more Variable Models
Strengths and Weaknesses
• Mean, Median, and Mode• Standard Deviation• Correlation• Regression• Factor Analysis
Discussion
What inferences can you draw from:
• Mean, Median, and Mode• Standard Deviation• Correlation• Regression• Factor Analysis
Discussion
Questions?Concerns?
Snide Comments?
(Practical Application Next)