Measuring Grit

Do Non-Cognitive Attributes Impact Academic Success, Engagement, Satisfaction and Retention?

Dr. Mac Adkins, President SmarterServices

Provided by

Measuring Grits

• We recommend a glass beaker for measuring grits. We are from Alabama, we know about these things.

What is Grit?

• Why does one student who had straight A’s in high school drop out of college after one year, and another one excel?

• Why does one single mom with three children graduate Summa Cum Laude and another one drop out?

What is Grit?

• Grit is that elusive quality that prompts one student to stick with it while others quit.

• For over ten years we have measured levels of grit in over 2,300,000 students at over 500 colleges and universities.

• Today I want to share with you the results of research related to the impact that grit has on student success.

Types of Data Used To Predict Learner Success

APTITUDE ATTITUDE SITUATION

Importance of Retention

Some Students Seem To Have More Grit Than Others

Three Approaches to Measuring Grit

• Stick your head in the sand.• Use a brief, non-prescriptive survey.• Use SmarterMeasure Learning Readiness

Indicator.

Is Distance Learning For Me?

SmarterMeasure Learning Readiness Indicator

• A 124-item online skills test and attributes inventory that measures a student’s level of readiness for studying online

• Used by over 500 Colleges and Universities• Since 2002 taken by over 2,300,000 students

What Are The Ingredients In Our Grits?

What Does The Assessment Measure?INTERNAL

INDIVIDUAL ATTRIBUTES

MotivationProcrastination

Time ManagementHelp Seeking

Locus of Control

LEARNING STYLES

VisualVerbalSocial

SolitaryPhysical

AuralLogical

EXTERNAL

LIFE FACTORS

Availability of TimeDedicated Place

ReasonSupport from Family

SKILLS

TECHNICAL

Technology UsageLife Application

Tech VocabularyComputing Access

TYPING

RateAccuracy

ON-SCREEN READING

RateRecall

WCET Survey, May 2013

How Does The Beaker Work?

How Do You Spot a Bad Grit?

Adjusting Readiness Ranges

Adjusting the cut points can make the reporting a more accurate predictor of success.

How Do Schools Use It?

• Orientation Course• Enrollment Process• Information Webinar• Public Website• Class Participation• Facebook• 68% of client schools administer the

assessment to all students, not just eLearning students

Thermometer Analogy

• More important than taking your child’s temperature is taking appropriate action based on their temperature.

• More important than measuring student readiness is taking appropriate action based on the scores.

Measuring the Grits

Predictive

Correlation

Comparison

Descriptive

Student Service

Progression of SmarterMeasure Data Utilization

Research Ideas on the Research Page of the Website

Internally

Conducted

Company

Assisted

Professionall

y Assiste

d

Approaches to Research Projects

Middlesex Community College

• 6% to 13% more students failed online courses than on-ground courses.

• Intervention Plan- Administer SmarterMeasure- Identify which constructs best predicted success- Provide “Success Tips” as identified

Distributed by website, email, orientation course, records office, library, posters, and mail

Research Findings

• Analyzed 3228 cases over two years• Significant positive correlation between

individual attributes and grades

GradesImpactsMotivatio

n

Results of Middlesex Research

Before SmarterMeasure™ was implemented, 6% to 13% more students failed online courses than students taking on-ground courses. After theimplementation, the gaps were narrowed: 1.3% to 5.8% more online students failed than on-ground students.

Results of Middlesex ResearchFailure rates reduced by as much as 10%

Action Plan

• Empower eLearning staff, faculty advisors, and academic counselors with student data

MotivationSelf

Discipline

Time Managemen

t

Three areas of

focus

Project Summary

“In summary, the implementation of SmarterMeasure has helped students to achieve better academic success by identifying their strengths and weaknesses in online learning.”

In essence, with various strategies implemented to promote SmarterMeasure™, a “culture” was created during advising and registration for students, faculty, and support staff to know that there is a way for students to see if they are a good fit for learning online.

CEC - The Need

• We need to know which students to advise to take online, hybrid or on-campus courses.

• We need to know which students to direct to which student services to help them succeed.

• We need to know how to best design our courses so that new students are not overwhelmed.

The Analysis

• What is the relationship between measures of student readiness and variables of:– Academic Success - GPA– Engagement – Survey (N=587)– Satisfaction – Survey (Representative Sample

based on GPA and number of courses taken per term)

– Retention – Re-enrollment data

The Analysis

• Phase One – Summer 2011– Included data from all three delivery systems – online, hybrid

and on-campus– Analyzed data at the scale level

• Phase Two – Fall 2011– Focused the research on online learners only– Analyzed data at the sub-scale level

• A neutral, third-part research firm (Applied Measurement Associates) used the following statistical analyses in the project:– ANOVA, Independent Samples t-tests, Discriminant Analysis,

Structural Equation Modeling, Multiple Regression, Correlation.

The Findings

• Academic Achievement– The scales of Individual Attributes, Technical

Knowledge, and Life Factors had statistically significant mean differences with the measures of GPA.

The Findings

• Retention– The measure of Learning Styles produced a

statistically significant mean difference between students who were retained and those who left. • A 73% classification accuracy of this retention

measure was achieved.

– The scales of Individual Attributes and Technical Knowledge were statistically significant predictors of retention as measured by the number of courses taken per term.

The Findings

• Engagement– The scales of Individual Attributes and Technical

Competency had statistically significant relationships with the four survey items related to Engagement.

– The scales of Life Factors, Individual Attributes, Technical Competency, Technical Knowledge, and Learning Styles were used to correctly classify responses to the survey questions related to engagement and satisfaction with up to 93% classification accuracy.

The Findings

• Satisfaction– Structural equation modeling was used to create a

hypothesized theoretical model to determine if SmarterMeasure scores would predict satisfaction as measured by the survey.

– Results indicated that prior to taking online courses, student responses to the readiness variables were statistically significant indicators of later student satisfaction.

– Therefore, the multiple SmarterMeasure assessment scores are a predictor of the Career Education survey responses.

The Findings

• Statistically Significant RelationshipsAcademic Achievement

Engagement Retention

Individual Attributes

X X X

Technical Knowledge

X X X

Learning Styles

X X

Life Factors X X

Technical Competency

X

The Findings

• Student Categorizations– Enrollment Status

• Positive – active/graduated (34.3%)• Negative – withdrew/dismissed/transfer (65.7%)

– Academic Success Status• Passing – A, B or C (48.9%)• Failing – D, F or Other (21.1%)

– Transfer Credit – (21.8%)– Not reported – (8.2%)

The Findings - Correlates

Readiness Domain Readiness Domain Subscales

  Positive vs. Negative Pass vs. Fail

Life Factor Place, Reason, and Skills Place

Learning Styles

Socialand

LogicalN/A

Personal Attributes

Academic, Help Seeking, Procrastination, Time Management, and Locus of Control

Time Management

Technical Competency

Internet CompetencyInternet Competency

andComputer Competency

Technical Knowledge

Technology Usageand

Technical VocabularyTechnical Vocabulary

The Findings - Predictors

Readiness Domains GPA F p

 Life Factor Place and Skills 12.35 .0001

 Learning Styles Verbal a and Logical 3.95 .02

 Personal Attributes  

Help Seeking, Time Management, and Locus of

Control

 21.11

 .0001

 Technical Competency  

Computer and Internet Competency

 22.75

 .0001

 Technical Knowledge  

Technology Vocabulary 

 38.76

 .0001

The Findings - Predictors

Readiness Domains Credit Hours Earned F p

 Life Factor Place 12.37 .0001

 Learning Styles Visual 6.81 .01

 Personal Attributes

  Academic Attributes, Help

Seeking, and Locus of Control

 13.40

 .0001

 Technical Competency  

Computer Competencyand Internet Competency

 12.23

 .0001

 Technical Knowledge  

Technology Usage and Technology Vocabulary

 

 26.97

 .0001

The Recommendations

• We need to know which students to advise to take online, hybrid or on-campus courses.– A profile of a strong online student is one who:

• Has a dedicated place to study online• Possesses strong time management skills• Demonstrates strong technical skills• Exhibits a strong vocabulary of technology terms

The Recommendations

• We need to know which students to direct to which student services to help them succeed.– An online student who should be directed toward

remedial/support resources is one who:• Has a weak reason for returning to school• Has weak prior academic skills• Is not likely to seek help on their own• Is prone to procrastinate• Has low, internal locus of control• Has weak technology skills

The Recommendations

• We need to know how to best design our courses so that new students are not overwhelmed.– Limit advanced technology in courses offered early in

a curriculum– Foster frequent teacher to student interaction early in

the course– Require milestones in assignments to prevent

procrastination– Clearly provide links to people/resources for

assistance

Argosy University

• Required in Freshman Experience course• Students reflect on scores and identify

areas for improvement in their Personal Development Plan

• Group reflection with others with similar levels of readiness

Argosy University - COMPARE

• Compared the traits, attributes, and skills of the online and hybrid students.

• Substantial differences between the two groups existed. • Changes were made to the instructional design process

for each delivery system.

Online

Hybrid

Argosy University - EXPLORE

• Correlational analysis between SmarterMeasure scores and student satisfaction, retention, and academic success

SatisfactionRetentionSuccess

Technical

Motivation TimeStatistically Significant

Factors:

Technical Competency Motivation

Availability of Time.

Argosy University - TREND

• Aggregate analysis of SmarterMeasure data to identify mean scores for students.

• Comparison made to the national mean scores from the Student Readiness Report.

National Scores

Argosy Scores

Argosy University - APPLY

• Findings were shared with the instructional design and student services groups and improvements in processes were made.

For example, since technical competency scores increase as the students take more online courses, the instructional designers purposefully allowed only basic forms of technology to be infused into the first courses that students take.

J. Sargeant Reynolds Community College

• Required as admissions assessment

• Integral part of their QEP• Computed correlations

with grades and SmarterMeasure sub-scales of over 4000 students.

• P

Grades

Findings• Statistically significant correlations:

Scores

- Dedicated place, support from employers and family, access to study resources, and academic skills (Life Factors)

- Tech vocabulary (Technical Knowledge)

- Procrastination (Individual Attributes)

Academic Success Rates

Skills Resources Time0

10

20

30

40

50

60

70

High Score

Low Score

Less than 10% of students with low scores experienced academic success.

Five Schools

What is the relationship between measures of online student readiness and measures

of online student satisfaction?

Methodology

Data from 1,611 students who completed both the SmarterMeasure Learning Readiness Indicator and the Priority Survey for Online Learners were analyzed.

Incoming vs Outgoing

Findings• There were statistically significant

relationships between factors of readiness and satisfaction.

Comparison to Compass Scores

North Central Michigan College - Petoskey, MI

National Data

• 2013 Student Readiness Report• Data from 639,324 students from 275

colleges and universities

Online Learner Demographics

• 69% were female• 54% were Caucasian/White• 54% had never taken an online course before• 40% were traditional aged college students • 53% were students at an associate’s level

institution

Online Learner Demographics

• Dominant Social learning style• Highly motivated• Moderate reading skills• Pressed for time• Increasing technical skills

Profile of a Successful Online Student

• Four demographic variables have had a statistically significant higher mean for five years in a row.

Females higher in Individual Attributes, Academic Attributes, and Time Management.

Males higher in Technical Knowledge.

Profile of a Successful Online Student

• Caucasians have had the highest means for five years in Technical Knowledge.

• Students who have taken five or more online courses have had the highest means for five years in Individual Attributes, and Technical Knowledge.

Conclusion

• Statistically significant relationships exist between measures of online student readiness and measures of academic success, engagement, satisfaction and retention.

Readiness Impacts Satisfaction

Conclusion

• Students individually benefit and schools collectively benefit from measuring learner readiness and appropriately responding.

Schools and Students Benefit

SmarterMeasure.com

Middlesex, Argosy, J. Sargeant Reynolds: http://smartermeasure.com/research/research-results/

Noel Levitz Study: https://www.noellevitz.com/papers-research-higher-education/2011/2011-adult-and-online-learner-satisfaction-priorities-reports

Student Readiness Report: http://smartermeasure.com/smartermeasure/assets/File/Online-Student-Readiness-Report.pdf

SmarterMeasure References

“Live as if you were to die tomorrow. Learn as if you were to live forever.”

Mahatma Gandhi

For More Info

SmarterServices.com(877) 499-SMARTER

info@SmarterServices.com