Post on 09-Mar-2018
Assessing Learning of Student
Employees
Douglas S. Franklin, Ph.D.,
Assistant Dean of Students
Kevin A. Smith, Assistant Director for Leadership &
Community Service
Learning Objectives
• Participants will:
1. gain and apply knowledge of theories relevant to
student employment;
2. learn how to develop learning outcomes relevant
to a student employee work;
3. learn how to construct a rubric for use in
employee performance management;
4. learn how to develop a survey to assess student
employee perceptions of growth.
Project Goal
• Define, establish learning objectives,
and assess the achievement of learning
through student employment as it
relates to the institutional core values of
commitment, community, citizenship,
civility, and character.
Why Student Employment?
• 6 Years of Student
Employment Experience at
University of Arkansas
• 20 Years as U.S. Naval
Reserve Officer
Our responsibilities!
• Empower students through intellectual and practical skills
• Inform students about conditions that affect their lives.
• Recognize student learning occurs in the active part of student‟s lives.
“The conception of mind as a purely isolated possession of
the self is at the very antipodes of the truth…the self is not
a separate mind building up knowledge anew on its own
account. (Dewey, 1958 1929, p. 317)
Common Perceptions of Student Employees
• “Many students in work-study programs seek library
positions because they think they will be able to study while
on the job. …"We're not looking for people just to pay
students to study” (Sharon Welsh, director of student employment at
Rutgers University)
• “I cleaned out grease traps for minimum wage. I received no
guidance, took home my paycheck and bought beer” (Student
employee, Ohio University
• “Students can‟t even answer the telephone correctly” (Anonymous School Director, Ohio University)
Reality
• About 41 % of full-time and 76 percent of
part-time college students ages 16–24 were
employed (NCES, 2009)
“postsecondary education has become our
(nation‟s) core workforce-development
system” Carnevale, A.P (2008)
Percentage of 16- to 24-year-old college students who were
employed, by attendance status and hours worked per week:
October 1970 through October 2009
Source: NCES, 2011
Academic Impact: Mixed Bag• Little impact on GPA between working
and nonworking groups but student
grades declined with increasing hours of
work (Gleason, „93; Hay & Lindsay,‟69)
• Employment had little impact on cognitive
growth, reading comprehension,
mathematics, and critical thinking,
(Pascarella et al., „94)
• Increasing hours of student employment
associated with decreased likelihood of
regular class attendance and less regular
review of class materials (Lammers et al.
,2001)
• Student employment increased amount of
time required to graduate (Canabal,
1998; Gleason, 1993).
• No overall negative relationship between
student employment and educational
performance, although the author
suggested a negative link for African
American and Hispanic students. Canabal
(„98)
• Students working less than 15 hours
weekly did not demonstrate a statistically
significant negative impact on GPA (Henry,
„67; Kaiser & Bergen, „68;Merritt, „70).
• Students working 1–15 hours weekly had
significantly higher GPA than students
working 16 or more hours. Students
working 1–15 hours weekly had a higher
GPA than nonworking students, NCES
(„94)
Career Development: Mostly Positive
• Working students were as persistent, and had as high of a professional
commitment, as nonworking students” (Fjortoft, „95)
• Work or internship experiences during college …have a positive net
influence on the development of career-related skills and the likelihood of
being employed immediately after college” (Pascarella & Terenzini, „05)
• Graduates with work or internship experience produced higher quality
work, accepted supervision and direction more willingly, demonstrated
better time management skills, and were better able to interact with
coworkers on team projects (Casella & Brougham, „95)
• Student employees demonstrated a lack of Moral and Alienative
Commitment which is indicative of a lack of commitment to the
organization (Pereles, „07)
Learning Through Work
• Billett (2001) examined student employment concluded
learning through workplace experiences include:
– access to authentic work activities (i.e. authentic activities,
novel and routine);
– observation and listening (cues and clues – indirect guidance);
– access to more experienced co-workers (direct guidance –
development of heuristics);
– practice (opportunities to reinforce, refine and hone).
“Learning that is personally transformative turns
out to be the learning that involves membership in a
community of practice” Wenger, 1999, p. 32
Developing educational worth from work
(practiced based) experiences (Adapted from Billett, S. 2009)
• Identify experiences to develop, sustain and utilize
students‟ personal epistemologies, including their
critical engagement and reflection.
• Identify and acknowledge the pedagogic potential of
practice experiences,
– How can experiences be integrated into the curricula to
maximize students‟ learning?
• Prepare, position, sequence and identify the most
appropriate duration of experiences
All Employment Not the Same
• Astin (1993)
– observed off-campus work (both full-time and part-time) is
“uniformly negative” but noted on-campus employment has a
completely different impact on students.
– Effects of on-campus employment were characterized as positive in
nature and had the same positive outcomes as other student
involvement.
• Anderson (1981)
– compared with nonworking students who lived on campus, the
author concluded that students who worked on campus (work
study) and lived at home were more likely to persist, while students
who lived and worked off campus were less likely to remain in
school.
Theoretical Framework
• Astin‟s Theory of Involvement
– “students learn more the more they are
involved in both the academic and social
aspects of the collegiate experience.”
“The world‟s problems are not going to be solved by
math and science and technology; they are human
problems, problems of beliefs and values and feelings.
It would behoove higher education to begin to attend
more to these aspects of student development” (Astin,
2003)
Theoretical Framework
• Bridges Transition Theory1. Ending, Losing, Letting Go – “Letting go of old ways and old
identities”
2. The Neutral Zone – “Going through an in-between time when
the old is gone but the new isn‟t fully operational”
3. The New Beginning – “Coming out of the transition and
making a new beginning”
“It isn‟t the changes that do you in,
it‟s the transitions” Bridges, 2003
Theoretical Framework
• Bloom's Taxonomy of Learning (Domains)
– Cognitive: mental skills (Knowledge)
– Affective: growth in feelings or emotional areas
(Attitude)
– Psychomotor: manual or physical skills (Skills)
“There is more than one type of learning”
Bloom, 1956
Student Employment at OHIO
Future Research? What is the effect of campus location (urban verses rural)
and type (commuter verse residential) on student employment?
Assessing the 5C‟s
• Creating definitions
– Used current higher education literature
– Sample learning outcomes developed from
learning outcome domains and dimensions
from the Council for the Advancement of
Standards (CAS).
5C‟s- Character
• Definition- The moral and ethical qualities of persons as well as the demonstration of those qualities in their emotional responses, reasoning, and behavior.
– Associated with such qualities as integrity, responsibility, trustworthiness.
• CAS Domain- Interpersonal
• SLO- Establishes mutually beneficial relationships with others
5C‟s-Civility
• Definition: Implies respect for others, a
willingness to hear and seek others views,
and the exercise of restraint in criticizing.
– Creates an environment where the views and
actions of others are tolerated.
• CAS: Domain: Cognitive Complexity
• CAS: Dimension: Critical Thinking
• SLO: Assesses assumptions and considers
alternative perspectives and solutions
5C‟s: Community
• Definition: Supportive environment where
members with shared values work
collaboratively toward a common purpose or
engage in a common experience.
• CAS Domain: Practical Competence
• CAS Dimension: Social Responsibility
• SLO: Appropriately challenges the unfair,
unjust, or uncivil behavior of other individuals
or groups
5C‟s-Citizenship
• Definition: Process whereby an individual and the collaborative group become responsibly connected to the community and society; to be a good citizen is to work for positive change on the behalf of others and the community
• CAS Domain: Knowledge acquisition, constraint, interjection, or application
• CAS Dimension: Constructing Knowledge
• SLO: Makes meaning from text, instruction, and/or experience
5C‟s: Commitment
• Definition: the level of one‟s personal
investment
– Associated with high levels of intention,
persistence and resolve.
• CAS: Domain: Intrapersonal Development
• CAS: Dimension: Identity Development
• SLO: Identifies and commits to important
aspects of self.
Assessing the 5C‟s
• Measuring Commitment
• Self Assessment (Indirect measure)
– Pre and post testing (Rate the impact
employment had on your ability to:
• Work to overcome obstacles that hamper goal
achievement (Very Low to Very High)
Assessing the 5C‟s
• Testing (Direct measure)
– How many times was student late or no-
show? (not implemented)
• Rubric (Example)SLO Approaches Meets Exceeds
Establishes a
work ethic
Is tardy less than once
a month and misses
work less than once
per quarter
Not tardy and
does not miss
work
Identifies scheduling
problems and seeks to fill
operational weakness
Limitations and Delimitations• Scope is limited by:
– Lack of cohesive support (ad hoc)
– Lack of experience
– Inconsistent distribution of survey
– Use of indirect measures of proxy pre-post
instruments
• Scope is limited to:
– Ohio University Division of Student Affairs
Common Ground: Developing
Learning Outcomes
• As a result of (describe work experience) the student
will (action verb) (knowledge, skill or attitude).
• As a result of working as a supervisor at the Ping
Center, the student will demonstrate effective verbal
communication skills with patrons.
“Objectives indicate what we want our students to learn;
they are explicit formulations of the ways in which
students are expected to be changed by the educative
process” (A taxonomy for learning, teaching, and
assessing . Editors Anderson & Krathwohl, „01)
Bloom‟s Taxonomy
What‟s a Rubric?
Dimension Scale Level 1 Scale Level 2 Scale Level 3
1
2
3
4
“…a rubric is a scoring tool that lays out the specific
expectations for an assignment” Stevens (2005),
Introduction to Rubrics, p. 3
Learning from the Samurai?Learning outcome
from participation /
employment
Supportive
developme
ntal or
educationa
l theory
Departmental
Vision and
philosophy
Activity
Description
Outcome Measures
Approaching Meets Exceeds
“…a life of service, discipline and compassion…what could be more necessary?” (The Last Samurai, 2003)
Leadership
Development:
Supervisor
demonstrates
a thorough
understanding
of complex
facility and
program
operations
Social
Change
Model:
See
FALDO,
pg. 94
“Learning-
centered
environment
that
promotes
the growth
of all
students”
Facility or
Program
Supervisor
Seeks to
broaden
understan
ding and
acquires
knowledge
Analyzes
and
applies
knowledge
Seamlessly
integrates
actions
into daily
life.
Learning Outcomes Rubric
• Response
– 10 Res Housing
– 70 Campus Recreation
• Data not analyzed yet
.
Performance Management Rubric
• Rubric for Campus Involvement Center
Proxy Pre-Post Test
• Distributed Electronically (Survey Monkey)
• Adapted from the Measuring Outcomes from Recsports
Experiences (M.O.R.E.) (Haines & Fortman, 2008)
– Demonstrated construct validity through parallel model design (Carini, Kuh, & Klein, 2006; Cheln, Li, Gliklich, Chu, Liang, & Wang, 2002;
Elliot, Smith, Smith, & Chambers, 2000; Gonyea, Kish, Kuh, Muthia & Thomas,
2003, Gordon, Ludlum, & Hoey, 2006)
Descriptive Statistics
Area Worked Distribution Response % Response Response
Rate
Campus Involvement Center 21 5 5.5% 23.8%
Campus Recreation 426 50 55.5% 11.7%
Event Services 291 18 20% 6.1%
Career Services NA 4 4.4%
Residential Housing 14 14 15.5% 100%
University Judiciaries (DOS) 6 3 3.3% 50%
Responded 106 13.9%
Answered All Questions 90 12.3%
Skipped Questions 16
Descriptive Statistics
Current Job Count % of Response
Resident , Facility or Program Assistant 28 29.8%
Supervisor 20 21.2%
Manager or Student Director 17 18%
Specialists (Sports Official, Life Guard, Instructor, Personal Trainer 11 11.7%
Support (Technical Assistants and Receptionists) 16 17%
Graduate Assistants 4 4.2%
Answered Question 94
Skipped Question 12
Descriptive Statistics
Ethnicity Count % of
Response
Institutional
Data 2009
African American 4 4.25% 5%
Native American 1 1% .4%
Asian American 0 0.0% 1.3%
Hispanic 1 1% 1.9%
Caucasian 84 89.3% 84.6%
Multiple 1 1%
Other or unknown 1 1%
Prefer not to respond 4 4.25%
Answered Question 94
Skipped Question 12
Descriptive Statistics
Relationship to major Response Count % of Response
Directly 21 23.1%
Not Directly 72 79.1%
Answered Question 91
Skipped Question 15
Gender Response
Count
% of
Response
Institutional
Data 2009
Male 36 40% 48.3%
Female 58 60% 51.7%
Answered Question 94
Skipped Question 12
Major Areas (Domains)
• Knowledge of Policies and Procedures
• Decision Making
• Customer Service
• Teamwork (Community, Citizenship, Civility)
• Pursuing Goals (Commitment)
• Effective Communication (Civility)
• Managing Personal Affairs (Commitment)
• Commitment to Ethics and Integrity (Character)
Knowledge of Policies and ProceduresLearning Outcome Pre Ave Post Ave Significance
Ability to use work related policies 105 4.38 90 5.36 ***
Ability to find information 105 4.68 90 5.42 ***
Adherence to policies 105 4.89 90 5.44 ***
Ability to offer helpful information w/o provocation 104 4.42 89 5.46 ***
Decision Making
Learning Outcome Pre Ave. Post Ave Significance
Making decisions alone 105 4.50 90 5.36 ***
Understanding the „big picture‟ 105 4.70 90 5.47 ***
Demonstrating good judgment 104 4.98 90 5.57 ***
Asking superiors for help when necessary 105 4.88 88 5.47 ***
Learning Outcome Pre Ave. Post Ave Significance
Demonstrating respect when interacting with others 105 5.30 90 5.60 ***
Answering questions without hesitation 105 4.55 90 5.39 ***
Understanding the reasoning behind rules 104 4.92 90 5.46 ***
Customer Service
Teamwork (Community, Citizenship
and Civility)
Learning Outcome Pre Ave. Post Ave Significance
Giving constructive feedback 105 4.37 90 5.23 ***
Receiving constructive feedback 105 4.53 90 5.23 ***
Ability to handle problems w/o arguments 105 4.90 90 5.47 ***
Supporting others decisions 105 4.91 90 5.47 ***
Raising concerns when necessary 105 4.60 89 5.40 ***
Pursuing Goals (Commitment)
Learning Outcome Pre Ave. Post Ave Significance
Pursuing individual goals 105 4.91 90 5.46 ***
Articulates rationale for personal or
educational goals
105 4.73 90 5.38 ***
Makes plans to achieve goals 105 4.80 90 5.42 ***
Works to overcome obstacles that hamper
goal achievement
105 4.83 89 5.43 ***
Effective Communication (Civility)
Learning Outcome Pre Ave. Post Ave Significance
Conveys meaning to others through writing 105 4.54 90 5.20 ***
Articulates meaning to others 104 4.59 90 5.32 ***
Responds after reflection 105 4.60 90 5.24 ***
Demonstrates the ability to persuade 105 4.50 89 5.17 ***
Demonstrates the ability to explain abstract ideas 105 4.50 90 5.21 ***
Uses appropriate grammar 105 4.89 90 5.46 ***
Appears comfortable when presenting to others 103 4.46 89 5.25 ***
Managing Personal Affairs
(Commitment)
Learning Outcome Pre Ave. Post Ave Significance
Demonstrates self-reliance 105 4.88 89 5.51 ***
Manages time effectively 105 4.61 90 5.46***
Meets routine obligations w/o assistance 104 4.96 90 5.62***
Avoids conflict between work and personal
obligations
105 4.93 90 5.43
***
Is productive in personal affairs 105 4.90 89 5.48***
Commitment to Ethics and Integrity
(Character)
Learning Outcome Pre Ave. Post Ave Significance
Incorporating ethical reasoning into actions 104 4.88 90 5.48 ***
Considering personal values in decision -making 103 4.95 90 5.50 ***
Aligning personal values with actions 104 4.85 90 5.51 ***
Demonstrates dependability 104 5.06 90 5.61 ***
Demonstrates honesty 103 5.15 90 5.66 ***
Accepting responsibility for actions 104 5.10 90 5.62 ***
Top 9 Learning OutcomesLearning Outcome Ave Significance
Demonstrates honesty (Character) 5.66***
Meets routine obligations w/o assistance (Commitment) 5.62 *** Accepting responsibility for actions (Character) 5.62 *** Demonstrates dependability (Commitment) 5.61 *** Demonstrating respect when interacting with others (Civility and
Community)
5.60 ***
Demonstrating good judgment (Character) 5.57 ***
Demonstrates self-reliance (Commitment) 5.51 ***
Aligning personal values with actions (Character) 5.51 *** Considering personal values in decision -making (Character) 5.50
***
Job Satisfaction: Ping Exit Survey n=20
• Goal Setting, Time
management
• Adjustment to change
• Mentor training
• Appropriate training &
supervision
• Responsiveness (Pro staff, GA,
Sup, Mgr
• Fit with whole experience
• Scheduling
• Fairness of evaluation
• Appropriateness of orientation
and staff meetings
• Equipment condition
• Respect (Pro staff, GA, Sup,
Mgr.)
• Co-worker interactions
• Ability to express opinions
• Value of input
Learning Outcomes Ping Exit Survey CAS Learning Domain (2007) n=15/14 4 pt scale
Communication 3.53
Collaboration 3.36
Interpersonal Relationships 3.20
Social Responsibility 3.20
Satisfying and Productive Lifestyles 3.14
Leadership Development 3.13
Independence 3.11
Self-esteem 3.07
Healthy Behaviors 3.07
Intellectual growth 2.86
Career Choices 2.80
Self appraisal 2.73
Personal and Educational Goals 2.67
Appreciation for Diversity 2.64
Clarification of Values 2.57
Spiritual Awareness 2.00
Student Voices
• “Time management, problem solving, „handy‟
knowledge, communication”
• “Leadership skills, organizational skills,
professional skills”
• “Communication, adapting to changing situations,
first hand first aid experience”
Students employees from the Ping Center when answering
the question “List 3 transferable skills learned during
employment in Campus Recreation”
Student Voices: Informal Learning
“I …appreciated the growing responsibilities I acquired
throughout the four years that I‟ve worked here… I‟ve been
entrusted with information …and I‟m relied upon by the
accountants to help complete deposits. It‟s certainly helped to
make me feel more adult and responsible… something that helps
the transition from Athens to the real world easier…On the whole,
I can say I‟ve gotten more satisfaction out of this job than other
part-time jobs, and I think that has everything to do with the
responsibility and trust I was given.”
Student employee : Ohio University Campus
Recreation Business Office
Student Voices: Connecting to the
Classroom
“I had a slight idea of what marketing was in taking
this position, but the job taught a lot about real-world
application and the challenges that arise in
developing and implementing marketing programs. I
learned the job as I performed the job, not only
through hands-on working, but in the classroom as
well.”Student employee : Ohio University
Amanda J. Cunningham Leadership
Center
Student Voices
• “This job has provided me the opportunity to be heard by
hundreds of students on campus and impact their lives in a
positive way through the content and messages in the
Center‟s presentations. After taking on three quarters of
restless students, bored organization members, and test
faculty, I feel like I could conquer the world.”
Student employee : Ohio University Amanda
J. Cunningham Leadership Center
Student Voices
• “Over the course of the year, I have developed greatly as a
professional and as a student leader. I have greatly
enhanced my skills of professional communication,
presentation, public-speaking, event planning, and
coaching.”
Student employee : Ohio
University, Amanda J.
Cunningham Leadership Center
Student Voices• “I have learned so much about myself. …how to take responsibility and how to
create new routines for myself and figure out how to do my job in the best and
most efficient manor [sic]. It was overwhelming at first having so many people
come to me asking questions; I initially would freeze up and get nervous. At
this point I can proudly say that I am much more confident and I enjoy
answering questions and guiding anyone who comes to me in the correct
direction. I have also learned quite a bit about working with others. I have
worked on my collaborative skills, active listening skills, and I have realized
that we all have had something great to contribute to the CIC, and when we
combine all of our thoughts and ideas, we can really do impressive things.
This experience has also helped me confirm that I want to eventually work for
a non profit [sic] organization that in general has the goal to give back.”
Student employee : Ohio University Campus
Involvement Center Center
Student Voices
• “Adaptability is the most important lesson I have learned from this
job.”
Student employee : Ohio University
Amanda J. Cunningham Leadership
Center
Questions
• Doug Franklin franklin@ohio.edu
• Kevin Smith- smithk3@ohio.edu