Using Grades over 26 Years to Evaluate a Career Course: Two Studies Robert C. Reardon, Ph.D. Stephen...
-
Upload
mildred-stevens -
Category
Documents
-
view
214 -
download
0
Transcript of Using Grades over 26 Years to Evaluate a Career Course: Two Studies Robert C. Reardon, Ph.D. Stephen...
Using Grades over 26 Years to Using Grades over 26 Years to Evaluate a Career Course: Evaluate a Career Course:
Two Studies Two Studies
Robert C. Reardon, Ph.D.Robert C. Reardon, Ph.D.Stephen J. Leierer, Ph.D.Stephen J. Leierer, Ph.D.Donghyuck Lee, M.Ed.Donghyuck Lee, M.Ed.
Florida State UniversityFlorida State University
Course History
• 1973: The career course developed
• 1980: The team-teaching of 3 or 4 instructors
• 1981: From the quarter to semester system
• 1984: Plus/minus grading system
• 1984 – 1988: Catalyst materials introduced.
Course History
• 1994: Cognitive Information Processing (CIP) theory introduced.
• 1996 – 2000: Once per week sections offered.
• 1998: Friday section offered.
• 1999: CIP text and student manual introduced.
• 1999: Internet used in students’ research
Course Objectives
1. Perceive the individual as purposefully responsible and active in the life/career planning process and to develop skills for increasing such behavior in others and oneself.
2. Understand how personal characteristics influence career development.
3. Become oriented to the socioeconomic world of work as it impacts individual and family career systems.
4. Identify appropriate academic major and/or occupational alternatives in relation to personal characteristics.
Course Objectives
5. Learn about and use a variety of information resources to explore academic major or occupational options.
6. Understand career development theories and use decision-making skills for life/career planning and management.
7. Learn about and use job-hunting skills needed for employment.
8. Formulate action plans and strategies for moving oneself or other persons toward implementation of life/career goals.
Course ActivitiesActivities & Assignments Total Points Related Course Objective(s)
Participation 30 1
Attendance 58 1
Chapter 1 Quiz 10 6
Autobiography 60 2
SDS Interpretive Report 10 2,4,6
2 Occupational Interviews 40 1,3,5
Career Field Analysis 200 3,4,5
Skills Assessment Activity 10 2,4
SIGI PLUS or Discover Computer Feedback Form 10 2,4
Choices Computer Feedback Form 10 2,4
Career Thought Inventory Profile 10 1,6
Individual Action Plan 10 1,5,8
Academic/Career Plan Project 100 1,8
Resume 15 7
Cover Letter 5 7
Performance Tests 75 2,3,5,6,7
Course Structure
• Unit I: “Career Concepts and Applications”
– Focuses on self-knowledge, knowledge about options, and decision making
– Assignments: Writing an autobiography, completing the Self-Directed Search, and a skills assessment activity.
Course Structure
• Unit II: “Social Conditions Affecting Career Development”– Focuses on current social, economic, family, and
organizational changes affecting the career planning process and the need for students to develop more complex cognitive schema to solve career problems.
– Assignments: Complete autobiography, Career Field Analysis (CFA) paper, and two information interview reports
Course Structure
• Unit III: “Implementing a Strategic Career Plan”
– Focuses on employability skills and strategies for implementing academic/career plans.
– Assignments: Two information interview reports, the completion of a resume and cover letter, and a strategic/academic career plan paper
Instructional Methods
• A mixture of lecture, panel presentations, and small and large group activities.
• Each instructor works with a small group of students in breakout sessions and evaluates their work.
• Instructors meet individually with the students to discuss their assessments and progress.
Course Grading Procedures
• Grades are based on the successful execution of a performance contract (PC) by the student.
• The PC includes 16 different graded activities spread across the three units of the course.
• 28 different activities are graded in the 3-credit version of the course.
Population
• 6,176 undergraduate students who completed the course, “Introduction to Career Development.”
• 15% to 25% of the class composed of students with officially undeclared majors.
• 60% unsure, dissatisfied, or undecided with current career situation.
• 75 academic periods (semester/quarter) studied.
Ethnic Distribution
4%12%
3% 7%
74%
4%
AmericanIndian
AfricanAmerican
Asian HispanicAmerican
Caucasian Other
Data Collection Procedures
• Study #1: Archived course grade data 1978-2004 by academic term obtained from the university registrar.
• Study #2: Archived course grade data and student evaluation of teaching (SET) data 1999-2004 by course section obtained from registrar and instructors.
Working with Archival Data
• Retrieving electronic grades by historical term– Aggregate group records by quarter/semester– Precluded analysis of student characteristics
• Issues in retrieving SET from faculty– Confidentiality and security– Course section data
• Large dataset: 6,176 students over 75 terms
Research Questions Study #1
1. Did grades provide evidence of students meeting the course learning objectives?
2. How did changes in course structures and procedures affect student learning?
3. Did grades differ by semester?
4. Did the mean grade point average earned in the course change over time?
Results
• Evidence of Students Meeting the Course Learning Objectives– 74% of the students earned a B+ or better– Mean GPA was 3.44 (SD = .84)– Negative correlation between mean semester GPA
and the semester identification number (r = -.38, p = .002)
Results
44.4%
19.2%
10.0%
13.4%
3.7%1.8% 2.5%
0.9% 0.3% 0.6% 0.3%2.8%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
A A- B+ B B- C+ C C- D+ D D- F
Results
• Eight course objectives are connected to the 16 graded course learning objectives.
• 3% receiving ‘F’ grade provides evidence of the course demands.
Results
• Course Structures and Procedures
M SD F (1, 6174)
Quarter system
Semester system
3.47
3.44
.88
.84.28
5-point grading system
11-point grading system
3.48
3.44
.92
.841.09
Results
• Course Structures and Procedures
M SD F (1, 6174)
Semester without Catalyst materials
Semester with Catalyst materials
3.41
3.53
.88
.7425.30*
Before CIP integrated into the course
After CIP integrated into the course
3.56
3.33
.75
.92115.06*
Before textbook used
After textbook used
3.51
3.24
.80
.90127.75*
Results
• The changes in semester system and grading system did not produce a difference in course grades.
• The intensive infusion of the work-family life balance materials associated with higher grades.
• The Infusion of CIP theory and textbook made the course more challenging and lowered grades.– Using internet-based sources in researching occupations
resulted in lower grades on the career field analysis (CFA) research paper.
Results
• Grade Distribution by Semester– Significant difference in the aggregated GPA by semester
(F = 6280.86, p < .0005)
3.48
3.58
3.39
3.25
3.3
3.35
3.4
3.45
3.5
3.55
3.6
Spring Summer Fall
Aggre
gat
ed G
PA
Results
• Course grades varied by semester.• Grades in the summer were significantly higher than
in other terms.– The summer term provides a more intensive course
experience (intensive class schedule; students’ study load).
• Grades in the fall were significantly lower than in other semesters.– Four-month intermission after registration (perhaps lowed
motivation).
Results
• Grade Distribution over Time– Significant difference in the aggregated GPA by time
(F = 23.69, p < .0005)
3.633.5
3.543.47 3.36
3.23
33.13.23.33.43.53.63.73.83.9
4
1 2 3 4 5 6
Time Period
Agg
rega
ted
GPA
1: Fall 1978 - Spring 1981
2: Fall 1981 - Summer 1985
3: Fall 1985 - Summer 1990
4: Fall 1990 - Summer 1995
5: Fall 1995 - Summer 2006
6: Fall 2000 - Summer 2004
Results
• Students in the latest time period (fall 2000 through summer 2004) had significantly lower grades than those in any other time period.
• Grade inflation was not the case with this career course.
Conclusions
• The career course investigated in this study appears to be an effective intervention as evident in student grades.
• Grade inflation was not the case for this course.
• However, grades were affected by historical events, temporal conditions, and course modifications.
Limitations
• Using aggregate grades across academic terms rather than individual student grades.
• This precluded an examination of ethnicity, gender, or other learner characteristics in this research.
Research Questions Study #2
1. What was the nature of students’ evaluation of teaching (SET) in this career course?
2. Were earned or expected grades in class sections different across semesters or class meeting times?
3. Were earned or expected grades related to SET ratings?
Sample for Study #2
• Fall 1999 – Summer 2004
• 62 course sections led by 12 different instructors who taught from 1 to 10 times
• 74% of sections reported expected grades (we found no bias pattern in missing data)
• 92% of sections reported SET ratings
Data Analysis
1 Day 2 Days 3 Days 4 Days
Fall
Spring
Summer
• The ideal design to examine the research questions about SET, Earned Grade Point Average, and Expected Grade Point Average would be a Split-Plot design.
• The data were archival, there are no observations for some Semester X Days combinations. There are two possible solutions to this problem
– Split-plot design using ANOVA for unbalanced data (paper)– Create a 7-level variable of Semester-Days (NCDA presentation)
Student Evaluation of Teaching (SET)Summary
Fall-Spring 1, 3 Fall-Spring 2 Summer 4
Student Evaluation of Teaching
Less than
the Mean
-
Greater than
the Mean
+
Less than
the Mean
-Earned Grade
Less than
the Mean
-
Less than
the Mean
-
Greater than the Mean
+
Expected Grade
Less than
the Mean
-
Greater than
the Mean
+
Greater than the Mean
+
Student Evaluation of Teaching (SET)
• The aggregated earned grade in the Summer semester was significantly higher than the aggregated earned grade during the Fall or Spring semesters.
• The aggregated expected grade was significantly different across semesters. Summer > Fall-Spring-2 > Fall-Spring 1, 3
• The aggregated expected grade was significantly higher than the aggregated earned grade.
• The difference between earned and expected grades was influenced by the Semester-Days variable. Summer > Fall-Spring-2 > Fall-Spring 1, 3
Summary
Student Evaluation of Teaching (SET)
• SET ratings on the difference of aggregated earned and expected grade varies according to semester-class times
Summer > Fall-Spring-2 > Fall-Spring 1, 3
• Difference in EGPA and XGPA influenced by SET
Summary
Student Evaluation of Teaching (SET)
• When controlling for SET, there was no significant difference between earned GPA and expected GPA.
• Aggregated EGPA and XGPA were influenced by variables related to the process of teaching, e.g., number of class meetings per week (1, 2, 3) and length of the semester (6 vs. 17 wks.).
• A significant amount of variability that exists between earned and expected grades can be traced to Student Evaluation of Teaching (SET).
Conclusions
Student Evaluation of Teaching (SET)
• Archival data analysis can become complex.• Most of the variables in these studies are aggregate measures. • Using aggregated grades as individual performance
indicators can be misleading.• SET is voluntarily collected, making for poor coverage and
probably poor utility for an ongoing archival project. • It is often unclear whether SET came as a result of the
outcome (high or low expected and earned grades) or it led to the outcome.
• These data are based on archived records collected by the University in multiple classes for the purpose of grading students in a class.
Limitations
Student Evaluation of Teaching (SET)
Presentation available at:
www.career.fsu.edu/techcenter
Robert Reardon, PhD
Steve Leierer, PhD
Donghyuck Lee, MEd
Contact Information