Assessing Learning of Student Employees - Center for the...

Assessing Learning of Student

Employees

Douglas S. Franklin, Ph.D.,

Assistant Dean of Students

Kevin A. Smith, Assistant Director for Leadership &

Community Service

Learning Objectives

• Participants will:

1. gain and apply knowledge of theories relevant to

student employment;

2. learn how to develop learning outcomes relevant

to a student employee work;

3. learn how to construct a rubric for use in

employee performance management;

4. learn how to develop a survey to assess student

employee perceptions of growth.

Project Goal

• Define, establish learning objectives,

and assess the achievement of learning

through student employment as it

relates to the institutional core values of

commitment, community, citizenship,

civility, and character.

Why Student Employment?

• 6 Years of Student

Employment Experience at

University of Arkansas

• 20 Years as U.S. Naval

Reserve Officer

Our responsibilities!

• Empower students through intellectual and practical skills

• Inform students about conditions that affect their lives.

• Recognize student learning occurs in the active part of student‟s lives.

“The conception of mind as a purely isolated possession of

the self is at the very antipodes of the truth…the self is not

a separate mind building up knowledge anew on its own

account. (Dewey, 1958 1929, p. 317)

Common Perceptions of Student Employees

• “Many students in work-study programs seek library

positions because they think they will be able to study while

on the job. …"We're not looking for people just to pay

students to study” (Sharon Welsh, director of student employment at

Rutgers University)

• “I cleaned out grease traps for minimum wage. I received no

guidance, took home my paycheck and bought beer” (Student

employee, Ohio University

• “Students can‟t even answer the telephone correctly” (Anonymous School Director, Ohio University)

Reality

• About 41 % of full-time and 76 percent of

part-time college students ages 16–24 were

employed (NCES, 2009)

“postsecondary education has become our

(nation‟s) core workforce-development

system” Carnevale, A.P (2008)

Percentage of 16- to 24-year-old college students who were

employed, by attendance status and hours worked per week:

October 1970 through October 2009

Source: NCES, 2011

Academic Impact: Mixed Bag• Little impact on GPA between working

and nonworking groups but student

grades declined with increasing hours of

work (Gleason, „93; Hay & Lindsay,‟69)

• Employment had little impact on cognitive

growth, reading comprehension,

mathematics, and critical thinking,

(Pascarella et al., „94)

• Increasing hours of student employment

associated with decreased likelihood of

regular class attendance and less regular

review of class materials (Lammers et al.

,2001)

• Student employment increased amount of

time required to graduate (Canabal,

1998; Gleason, 1993).

• No overall negative relationship between

student employment and educational

performance, although the author

suggested a negative link for African

American and Hispanic students. Canabal

(„98)

• Students working less than 15 hours

weekly did not demonstrate a statistically

significant negative impact on GPA (Henry,

„67; Kaiser & Bergen, „68;Merritt, „70).

• Students working 1–15 hours weekly had

significantly higher GPA than students

working 16 or more hours. Students

working 1–15 hours weekly had a higher

GPA than nonworking students, NCES

(„94)

Career Development: Mostly Positive

• Working students were as persistent, and had as high of a professional

commitment, as nonworking students” (Fjortoft, „95)

• Work or internship experiences during college …have a positive net

influence on the development of career-related skills and the likelihood of

being employed immediately after college” (Pascarella & Terenzini, „05)

• Graduates with work or internship experience produced higher quality

work, accepted supervision and direction more willingly, demonstrated

better time management skills, and were better able to interact with

coworkers on team projects (Casella & Brougham, „95)

• Student employees demonstrated a lack of Moral and Alienative

Commitment which is indicative of a lack of commitment to the

organization (Pereles, „07)

Learning Through Work

• Billett (2001) examined student employment concluded

learning through workplace experiences include:

– access to authentic work activities (i.e. authentic activities,

novel and routine);

– observation and listening (cues and clues – indirect guidance);

– access to more experienced co-workers (direct guidance –

development of heuristics);

– practice (opportunities to reinforce, refine and hone).

“Learning that is personally transformative turns

out to be the learning that involves membership in a

community of practice” Wenger, 1999, p. 32

Developing educational worth from work

(practiced based) experiences (Adapted from Billett, S. 2009)

• Identify experiences to develop, sustain and utilize

students‟ personal epistemologies, including their

critical engagement and reflection.

• Identify and acknowledge the pedagogic potential of

practice experiences,

– How can experiences be integrated into the curricula to

maximize students‟ learning?

• Prepare, position, sequence and identify the most

appropriate duration of experiences

All Employment Not the Same

• Astin (1993)

– observed off-campus work (both full-time and part-time) is

“uniformly negative” but noted on-campus employment has a

completely different impact on students.

– Effects of on-campus employment were characterized as positive in

nature and had the same positive outcomes as other student

involvement.

• Anderson (1981)

– compared with nonworking students who lived on campus, the

author concluded that students who worked on campus (work

study) and lived at home were more likely to persist, while students

who lived and worked off campus were less likely to remain in

school.

Theoretical Framework

• Astin‟s Theory of Involvement

– “students learn more the more they are

involved in both the academic and social

aspects of the collegiate experience.”

“The world‟s problems are not going to be solved by

math and science and technology; they are human

problems, problems of beliefs and values and feelings.

It would behoove higher education to begin to attend

more to these aspects of student development” (Astin,

2003)


• Bridges Transition Theory1. Ending, Losing, Letting Go – “Letting go of old ways and old

identities”

2. The Neutral Zone – “Going through an in-between time when

the old is gone but the new isn‟t fully operational”

3. The New Beginning – “Coming out of the transition and

making a new beginning”

“It isn‟t the changes that do you in,

it‟s the transitions” Bridges, 2003


• Bloom's Taxonomy of Learning (Domains)

– Cognitive: mental skills (Knowledge)

– Affective: growth in feelings or emotional areas

(Attitude)

– Psychomotor: manual or physical skills (Skills)

“There is more than one type of learning”

Bloom, 1956

http://www.nwlink.com/~donclark/learning/learning.html

Student Employment at OHIO

Future Research? What is the effect of campus location (urban verses rural)

and type (commuter verse residential) on student employment?

Assessing the 5C‟s

• Creating definitions

– Used current higher education literature

– Sample learning outcomes developed from

learning outcome domains and dimensions

from the Council for the Advancement of

Standards (CAS).

5C‟s- Character

• Definition- The moral and ethical qualities of persons as well as the demonstration of those qualities in their emotional responses, reasoning, and behavior.

– Associated with such qualities as integrity, responsibility, trustworthiness.

• CAS Domain- Interpersonal

• SLO- Establishes mutually beneficial relationships with others

5C‟s-Civility

• Definition: Implies respect for others, a

willingness to hear and seek others views,

and the exercise of restraint in criticizing.

– Creates an environment where the views and

actions of others are tolerated.

• CAS: Domain: Cognitive Complexity

• CAS: Dimension: Critical Thinking

• SLO: Assesses assumptions and considers

alternative perspectives and solutions

5C‟s: Community

• Definition: Supportive environment where

members with shared values work

collaboratively toward a common purpose or

engage in a common experience.

• CAS Domain: Practical Competence

• CAS Dimension: Social Responsibility

• SLO: Appropriately challenges the unfair,

unjust, or uncivil behavior of other individuals

or groups

5C‟s-Citizenship

• Definition: Process whereby an individual and the collaborative group become responsibly connected to the community and society; to be a good citizen is to work for positive change on the behalf of others and the community

• CAS Domain: Knowledge acquisition, constraint, interjection, or application

• CAS Dimension: Constructing Knowledge

• SLO: Makes meaning from text, instruction, and/or experience

5C‟s: Commitment

• Definition: the level of one‟s personal

investment

– Associated with high levels of intention,

persistence and resolve.

• CAS: Domain: Intrapersonal Development

• CAS: Dimension: Identity Development

• SLO: Identifies and commits to important

aspects of self.


• Measuring Commitment

• Self Assessment (Indirect measure)

– Pre and post testing (Rate the impact

employment had on your ability to:

• Work to overcome obstacles that hamper goal

achievement (Very Low to Very High)


• Testing (Direct measure)

– How many times was student late or no-

show? (not implemented)

• Rubric (Example)SLO Approaches Meets Exceeds

Establishes a

work ethic

Is tardy less than once

a month and misses

work less than once

per quarter

Not tardy and

does not miss

work

Identifies scheduling

problems and seeks to fill

operational weakness

Limitations and Delimitations• Scope is limited by:

– Lack of cohesive support (ad hoc)

– Lack of experience

– Inconsistent distribution of survey

– Use of indirect measures of proxy pre-post

instruments

• Scope is limited to:

– Ohio University Division of Student Affairs

Common Ground: Developing

Learning Outcomes

• As a result of (describe work experience) the student

will (action verb) (knowledge, skill or attitude).

• As a result of working as a supervisor at the Ping

Center, the student will demonstrate effective verbal

communication skills with patrons.

“Objectives indicate what we want our students to learn;

they are explicit formulations of the ways in which

students are expected to be changed by the educative

process” (A taxonomy for learning, teaching, and

assessing . Editors Anderson & Krathwohl, „01)

Bloom‟s Taxonomy

Blooms_Taxonomy_Action_Verbs.pdf

Multipronged Approach

Personal Reflection

Perception Surveys

Performance Rubric

(Peer & Pro)

What‟s a Rubric?

Dimension Scale Level 1 Scale Level 2 Scale Level 3

1

2

3

4

“…a rubric is a scoring tool that lays out the specific

expectations for an assignment” Stevens (2005),

Introduction to Rubrics, p. 3

Learning from the Samurai?Learning outcome

from participation /

employment

Supportive

developme

ntal or

educationa

l theory

Departmental

Vision and

philosophy

Activity

Description

Outcome Measures

Approaching Meets Exceeds

“…a life of service, discipline and compassion…what could be more necessary?” (The Last Samurai, 2003)

Leadership

Development:

Supervisor

demonstrates

a thorough

understanding

of complex

facility and

program

operations

Social

Change

Model:

See

FALDO,

pg. 94

“Learning-

centered

environment

that

promotes

the growth

of all

students”

Facility or

Program

Supervisor

Seeks to

broaden

understan

ding and

acquires

knowledge

Analyzes

and

applies

knowledge

Seamlessly

integrates

actions

into daily

life.

Samurai_Knowledge.wmv

Samurai_Application.wmv

Samurai_synthesis.wmv

Learning Outcomes Rubric

• Response

– 10 Res Housing

– 70 Campus Recreation

• Data not analyzed yet

.

Student Affairs Employee Learning Outcomes.pdf

Performance Management Rubric

• Rubric for Campus Involvement Center

Student Employee Evaluation Form ACLC.pdf


Personal Reflection

Perception Surveys

Performance Rubric

Peer and Pro

Proxy Pre-Post Test

• Distributed Electronically (Survey Monkey)

• Adapted from the Measuring Outcomes from Recsports

Experiences (M.O.R.E.) (Haines & Fortman, 2008)

– Demonstrated construct validity through parallel model design (Carini, Kuh, & Klein, 2006; Cheln, Li, Gliklich, Chu, Liang, & Wang, 2002;

Elliot, Smith, Smith, & Chambers, 2000; Gonyea, Kish, Kuh, Muthia & Thomas,

2003, Gordon, Ludlum, & Hoey, 2006)

Descriptive Statistics

Area Worked Distribution Response % Response Response

Rate

Campus Involvement Center 21 5 5.5% 23.8%

Campus Recreation 426 50 55.5% 11.7%

Event Services 291 18 20% 6.1%

Career Services NA 4 4.4%

Residential Housing 14 14 15.5% 100%

University Judiciaries (DOS) 6 3 3.3% 50%

Responded 106 13.9%

Answered All Questions 90 12.3%

Skipped Questions 16


Current Job Count % of Response

Resident , Facility or Program Assistant 28 29.8%

Supervisor 20 21.2%

Manager or Student Director 17 18%

Specialists (Sports Official, Life Guard, Instructor, Personal Trainer 11 11.7%

Support (Technical Assistants and Receptionists) 16 17%

Graduate Assistants 4 4.2%

Answered Question 94

Skipped Question 12


Ethnicity Count % of

Response

Institutional

Data 2009

African American 4 4.25% 5%

Native American 1 1% .4%

Asian American 0 0.0% 1.3%

Hispanic 1 1% 1.9%

Caucasian 84 89.3% 84.6%

Multiple 1 1%

Other or unknown 1 1%

Prefer not to respond 4 4.25%


Skipped Question 12


Relationship to major Response Count % of Response

Directly 21 23.1%

Not Directly 72 79.1%


Skipped Question 15

Gender Response

Count

% of

Response

Institutional

Data 2009

Male 36 40% 48.3%

Female 58 60% 51.7%


Skipped Question 12

Major Areas (Domains)

• Knowledge of Policies and Procedures

• Decision Making

• Customer Service

• Teamwork (Community, Citizenship, Civility)

• Pursuing Goals (Commitment)

• Effective Communication (Civility)

• Managing Personal Affairs (Commitment)

• Commitment to Ethics and Integrity (Character)

Knowledge of Policies and ProceduresLearning Outcome Pre Ave Post Ave Significance

Ability to use work related policies 105 4.38 90 5.36 ***

Ability to find information 105 4.68 90 5.42 ***

Adherence to policies 105 4.89 90 5.44 ***

Ability to offer helpful information w/o provocation 104 4.42 89 5.46 ***

Decision Making

Learning Outcome Pre Ave. Post Ave Significance

Making decisions alone 105 4.50 90 5.36 ***

Understanding the „big picture‟ 105 4.70 90 5.47 ***

Demonstrating good judgment 104 4.98 90 5.57 ***

Asking superiors for help when necessary 105 4.88 88 5.47 ***


Demonstrating respect when interacting with others 105 5.30 90 5.60 ***

Answering questions without hesitation 105 4.55 90 5.39 ***

Understanding the reasoning behind rules 104 4.92 90 5.46 ***

Customer Service

Teamwork (Community, Citizenship

and Civility)


Giving constructive feedback 105 4.37 90 5.23 ***

Receiving constructive feedback 105 4.53 90 5.23 ***

Ability to handle problems w/o arguments 105 4.90 90 5.47 ***

Supporting others decisions 105 4.91 90 5.47 ***

Raising concerns when necessary 105 4.60 89 5.40 ***

Pursuing Goals (Commitment)


Pursuing individual goals 105 4.91 90 5.46 ***

Articulates rationale for personal or

educational goals

105 4.73 90 5.38 ***

Makes plans to achieve goals 105 4.80 90 5.42 ***

Works to overcome obstacles that hamper

goal achievement

105 4.83 89 5.43 ***

Effective Communication (Civility)


Conveys meaning to others through writing 105 4.54 90 5.20 ***

Articulates meaning to others 104 4.59 90 5.32 ***

Responds after reflection 105 4.60 90 5.24 ***

Demonstrates the ability to persuade 105 4.50 89 5.17 ***

Demonstrates the ability to explain abstract ideas 105 4.50 90 5.21 ***

Uses appropriate grammar 105 4.89 90 5.46 ***

Appears comfortable when presenting to others 103 4.46 89 5.25 ***

Managing Personal Affairs

(Commitment)


Demonstrates self-reliance 105 4.88 89 5.51 ***

Manages time effectively 105 4.61 90 5.46***

Meets routine obligations w/o assistance 104 4.96 90 5.62***

Avoids conflict between work and personal

obligations

105 4.93 90 5.43

***

Is productive in personal affairs 105 4.90 89 5.48***

Commitment to Ethics and Integrity

(Character)


Incorporating ethical reasoning into actions 104 4.88 90 5.48 ***

Considering personal values in decision -making 103 4.95 90 5.50 ***

Aligning personal values with actions 104 4.85 90 5.51 ***

Demonstrates dependability 104 5.06 90 5.61 ***

Demonstrates honesty 103 5.15 90 5.66 ***

Accepting responsibility for actions 104 5.10 90 5.62 ***

Top 9 Learning OutcomesLearning Outcome Ave Significance

Demonstrates honesty (Character) 5.66***

Meets routine obligations w/o assistance (Commitment) 5.62 *** Accepting responsibility for actions (Character) 5.62 *** Demonstrates dependability (Commitment) 5.61 *** Demonstrating respect when interacting with others (Civility and

Community)

5.60 ***

Demonstrating good judgment (Character) 5.57 ***

Demonstrates self-reliance (Commitment) 5.51 ***

Aligning personal values with actions (Character) 5.51 *** Considering personal values in decision -making (Character) 5.50

***

Job Satisfaction: Ping Exit Survey n=20

• Goal Setting, Time

management

• Adjustment to change

• Mentor training

• Appropriate training &

supervision

• Responsiveness (Pro staff, GA,

Sup, Mgr

• Fit with whole experience

• Scheduling

• Fairness of evaluation

• Appropriateness of orientation

and staff meetings

• Equipment condition

• Respect (Pro staff, GA, Sup,

Mgr.)

• Co-worker interactions

• Ability to express opinions

• Value of input

Learning Outcomes Ping Exit Survey CAS Learning Domain (2007) n=15/14 4 pt scale

Communication 3.53

Collaboration 3.36

Interpersonal Relationships 3.20

Social Responsibility 3.20

Satisfying and Productive Lifestyles 3.14

Leadership Development 3.13

Independence 3.11

Self-esteem 3.07

Healthy Behaviors 3.07

Intellectual growth 2.86

Career Choices 2.80

Self appraisal 2.73

Personal and Educational Goals 2.67

Appreciation for Diversity 2.64

Clarification of Values 2.57

Spiritual Awareness 2.00


Personal Reflections

Perception Surveys

Performance Rubric

(Peer & Pro)

Student Voices

• “Time management, problem solving, „handy‟

knowledge, communication”

• “Leadership skills, organizational skills,

professional skills”

• “Communication, adapting to changing situations,

first hand first aid experience”

Students employees from the Ping Center when answering

the question “List 3 transferable skills learned during

employment in Campus Recreation”

Student Voices: Informal Learning

“I …appreciated the growing responsibilities I acquired

throughout the four years that I‟ve worked here… I‟ve been

entrusted with information …and I‟m relied upon by the

accountants to help complete deposits. It‟s certainly helped to

make me feel more adult and responsible… something that helps

the transition from Athens to the real world easier…On the whole,

I can say I‟ve gotten more satisfaction out of this job than other

part-time jobs, and I think that has everything to do with the

responsibility and trust I was given.”

Student employee : Ohio University Campus

Recreation Business Office

Student Voices: Connecting to the

Classroom

“I had a slight idea of what marketing was in taking

this position, but the job taught a lot about real-world

application and the challenges that arise in

developing and implementing marketing programs. I

learned the job as I performed the job, not only

through hands-on working, but in the classroom as

well.”Student employee : Ohio University

Amanda J. Cunningham Leadership

Center

annualReport-marketing.pdf



Student Voices

• “This job has provided me the opportunity to be heard by

hundreds of students on campus and impact their lives in a

positive way through the content and messages in the

Center‟s presentations. After taking on three quarters of

restless students, bored organization members, and test

faculty, I feel like I could conquer the world.”

Student employee : Ohio University Amanda

J. Cunningham Leadership Center

Student Voices

• “Over the course of the year, I have developed greatly as a

professional and as a student leader. I have greatly

enhanced my skills of professional communication,

presentation, public-speaking, event planning, and

coaching.”

Student employee : Ohio

University, Amanda J.

Cunningham Leadership Center

Student Voices• “I have learned so much about myself. …how to take responsibility and how to

create new routines for myself and figure out how to do my job in the best and

most efficient manor [sic]. It was overwhelming at first having so many people

come to me asking questions; I initially would freeze up and get nervous. At

this point I can proudly say that I am much more confident and I enjoy

answering questions and guiding anyone who comes to me in the correct

direction. I have also learned quite a bit about working with others. I have

worked on my collaborative skills, active listening skills, and I have realized

that we all have had something great to contribute to the CIC, and when we

combine all of our thoughts and ideas, we can really do impressive things.

This experience has also helped me confirm that I want to eventually work for

a non profit [sic] organization that in general has the goal to give back.”

Student employee : Ohio University Campus

Involvement Center Center

Student Voices

• “Adaptability is the most important lesson I have learned from this

job.”

Student employee : Ohio University

Amanda J. Cunningham Leadership

Center

Questions

• Doug Franklin [email protected]

• Kevin Smith- [email protected]

mailto:[email protected]

mailto:[email protected]

Assessing Learning of Student Employees - Center for the...

Documents

Transcript of Assessing Learning of Student Employees - Center for the...