Lessons in Implementing a Post- Graduation and Learning-Gains Questionnaire Sara Lazenby...

Lessons in Implementing a Post-Graduation and Learning-Gains Questionnaire

Sara [email protected]

University of Wisconsin - MadisonMay 21, 2013

AIR Annual Forum

Why implement an institutional exit questionnaire?

• Prior to 2008, UW-Madison had no institutional exit questionnaire for Bachelor’s degree recipients

• The need to implement a questionnaire arose due to:– Voluntary System of Accountability

(VSA) question on future plans– Higher Education Opportunity Act of

2008– Essential Learning Outcomes adopted in

2008– Priority of campus leadership

What is the Voluntary System of Accountability?

• A voluntary initiative developed by the higher education community to meet the following objectives:– Demonstrate accountability and stewardship to

the public– Support institutions in the measurement of

educational outcomes and facilitate the implementation of effective practices as part of institutional improvement efforts.

– Assemble and disseminate information that is transparent, comparable, and understandable

– Provide a useful too for students during the college search process

• Sponsored by the Association of Public and Land-grant Universities and the American Association of State Colleges and Universities

• More information available at: http://www.voluntarysystem.org

• Viewable on the College Portrait: http://www.collegeportraits.org/

http://www.voluntarysystem.org/

http://www.collegeportraits.org/

Developing Criteria for an Exit Questionnaire – Phase 1

• Include the required VSA question

• Create a cost-effective and sustainable questionnaire

• Integrated into the student record

• Census approach has potential for high response rates

• Reporting considerations

Importance of the questionnaire design

• Developer coded to be part of our Student Portal

• As part of the Student Record, student responses are protected by FERPA

• Placement in ISIS/PeopleSoft means the questionnaire is easily connected to other student information

• No need to ask additional questions for demographic/academic data

- Module (annoyingly) stays live in student’s MyUW portal until student responds

- During the questionnaire period, the data is extracted into an InfoAccess data warehouse table and refreshed nightly. This is available to anyone who also has access to the relevant tables

-12 Days

Questionnaire goes live

End of Term (December and May)

+21Days

-11 Days:E-mail invitation goes out

-4 Days: Reminder e-mail sent

Questionnaire deactivated

Questionnaire Timeline

Many contributions to this project’s success:

Questionnaire DevelopmentVoluntary System of AccountabilityAAU Data Exchange Senior Exit Survey TeamNSSE University of IllinoisUW-Madison Convergence TeamAACU’s LEAPUW-Madison Assessment FundJohn Stevenson, UW Survey CenterJennifer Dykema, UW Survey CenterJeremy Beach, UW Survey CenterStudent participants in focus groups

ELOs and Post-Grad Plans

Technical Development and ImplementationConnie Chapman, Registrar’s OfficeMichael Fay, DoITJames Helwig, DoITRob Lauer, DoITWanjiru Pontes, DoITStephan Wahe, DoITBob Mayville, DoITBetty McIlwee, DoITJim Steele, Registrar’s OfficeKay Steiner, InfoAccess, DoIT

Example of Portal Appearance

Examples of the Questionnaire

Examples of the Questionnaire – Required VSA Question

Examples of the Questionnaire

Early Lessons Learned

• Quick resolutions needed for technical issues

• Varying response rates by question

• Some questions did not produce usable data– Salary– Workplace/Next Institution

Developing Criteria for an Exit Questionnaire – Phase 2

• Need for indirect measures of the Essential Learning Outcomes

• Questionnaire fatigue considerations

• Removal of questions that weren’t useful

http://www.provost.wisc.edu/content/WI_Exp_ELOs.pdf

- Indirect measures of learning, student perceptions

- ELOs, UI-UC, NSSE were all influences

- UW Survey Center did review and pre- testing, including layout and syntax

- Added questions to the post-graduation plans questionnaire in December 2010 and into the future

Question Development

Fall 2008 Spring 2009




Fall 20120%

10%

20%

30%

40%

50%

60%

70%

45%

29%

62%

52%47%

26%

41%

58%54%

Response Rates to VSA Question by Term, Post Graduation Plans Questionnaire

Source: Stdnt_Post_Grad_Plans_Ugrd, Retention_Awards_Main, Retention_Awards_Subplan, Stdnt_Acad_Plan_Attributes

Web Survey

My UW Portal Survey

What are our response rates?

Response Rates – Effects of Reminder E-mails

December 2012 – Number of responses by day.

May 2012 –

Number of responses by day.

-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 210

50

100

150

200

250

300

-12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 210

100

200

300

400

500

600

700

800

900

1000

Communicating Findings

• Time lag before degrees are posted

• Challenge: Displaying the results in a way that campus can easily understand.

• What do the results mean?

Analyzing and Disseminating data

• Campus Wide

• Schools/Colleges

• Individual Majors for Program Review

• Disciplinary “Divisions”– Arts and Humanities– Biological Sciences– Physical Sciences– Social Studies

16%

7%

77%

38%

16%

47%

9%

28%

6%

57%

7%

24%

8%

61%

Post Graduation Plans – ALL GRADUATES Fall 2011 and Spring 2012 Bachelor’s Degree Recipients

Other includes: graduate school part-time, additional coursework, military service, volunteering, and unspecified activities. Based on 3083 responses (54% response rate)

If employment selected, which best describes your plans at this time?

Based on 2,689 responses (47% response rate)

Work Full-Time

Work Part-Time

Graduate School Full-Time

Other

What is most likely to be your principal activity/activities after graduation?

(Unduplicated) Have accepted a position

Considering one or more

offers

Currently searching for

a position

Will begin searching for a

position after graduation

Do you plan to work in Wisconsin after graduation?

Yes

Undecided

No

Do you have a valid passport?

Yes, from the United States

Yes, from another country

No

SL, Academic Planning and Institutional Research, 10/12Source: Post Graduation Plans Questionnaire Responses)Based on 1853responses (87% response rate among those with work as their primary activity)

Post Graduation Plans – Campus Summary Trend in VSA question responses over time

Other

Graduate School Full-Time

Work Part-Time

Work Full-Time

0 10 20 30 40 50 60 70

2008-09 2009-10 2010-11 2011-12

Percent of Respondents

What is most likely to be your principal activity/activities after graduation? (Unduplicated)

• Eighteen questions that provide a self-assessment at graduation, that maps to the essential learning outcomes at UW-Madison.

• Respondents asked to assess their ability upon entering UW-Madison and at graduation.

• Results are tabulated as the non-null average for each question

Post Graduation Plans – Essential Learning Outcomes

All this data, now how do we communicate findings?

• Display Challenges– Need something visual to help

people interpret the data

• Understanding differences– A lower learning gain on a

particular metric wasn’t always “negative”

– Putting learning gains in the context of our general education requirements

– Differences in learning outcomes match our expectations given major content.

Learn on your ownApply skills and knowledge of your chosen major(s)

Find, organize, and evaluate information from multiple sources

Draw conclusions after weighing evidence, facts, and ideas

Develop a personal code of ethics and values

Empathize w/ind. differences based on culture, ethnicity, disability or sexual orientation

Write Effectively

Apply knowledge and skills in real world settings

Use computers and electronic technologyUnderstand that science is relevant to everyday life

Lead others effectively

Work collaboratively in groups

Speak Effectively

Understand culture and society within the United States

Contribute to the welfare of others

Understand cultures and societies outside of the United States

Appreciate the arts such as literature, music, and fine arts

Communicate in a language other than English

1

2

3

4

5

When Entering At Graduation

SL, Academic Planning and Institutional Research, 10/12Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –ExtremelyResponses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 2,828 to 2,886 responses, depending on question.

Post Graduation Plans – ALL GRADUATES“How able were/are you to …”

Post Graduation Plans – Biological Sciences Division 11-12“How able were/are you to …”

SL, Academic Planning and Analysis, 8/11Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –ExtremelyResponses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 423 to 431 responses, depending on question.






Write Effectively





Speak Effectively






1

2

3

4

5







Write Effectively





Speak Effectively






1

2

3

4

5


Post Graduation Plans – A Biological Science Major“How able were/are you to …”

SL, Academic Planning and Institutional Research, 10/12Scale: 1 –Not at all, 2 –A little, 3 –Somewhat, 4 –Very, 5 –ExtremelyResponses sorted clockwise from the question with the highest response score at graduation for all UW-Madison students. Based on 104 to 107 responses, depending on question.

Post Graduation Plans – Arts and Humanities Division 11-12“How able were/are you to …”







Write Effectively





Speak Effectively






1

2

3

4

5







Write Effectively





Speak Effectively






1

2

3

4

5


Post Graduation Plans – An Arts and Humanities Major“How able were/are you to …”


Post Graduation Plans – Physical Sciences Division 2011-12“How able were/are you to …”







Write Effectively





Speak Effectively






1

2

3

4

5







Write Effectively





Speak Effectively






1

2

3

4

5


Post Graduation Plans – A Physical Science Major“How able were/are you to …”


Post Graduation Plans – Social Studies Division 11-12“How able were/are you to …”







Write Effectively





Speak Effectively






1

2

3

4

5


Post Graduation Plans – A Social Studies Major“How able were/are you to …”






Write Effectively





Speak Effectively






1

2

3

4

5



Majors with at least 50 responses to question about plans after graduation

• Biochemistry• Biology• Business: Accounting• Business: Finance, Investment, and

Banking• Business: Management and Human

Resources• Business: Marketing• Business: Risk Management and

Insurance• Civil Engineering• Communication Arts• Computer Sciences• Economics

• English• Environmental Studies• History• International Studies• Journalism• Mathematics• Mechanical Engineering• Nursing• Political Science• Psychology• Sociology• Spanish• Zoology

Lessons Learned

• Have worked with campus partners to try and increase response rates

• Reminder e-mails!• Overlap with “exit surveys” within

School/College and Departments• Low cost, not a lot of opportunity for changes• Stability of questionnaire has advantages• Popularity of handheld/tablet devices impact

response rates

Opportunities and Challenges

• Going forward …• Opportunities to make part of the graduation process• Changes in regulations may necessitate a different

approach• Continued collaboration with other units, trying to

help them understand the value of the questionnaire

•Involved a lot of creative people with a wide range of skills and experience.

•Inspiration from our peers/competitors.

•Made use of our existing enterprise systems ISIS/PeopleSoftMyUW student portalInfoAccess (data warehouse)

•Census approach has potential for high response rate.

•Cost-effective, sustainable collection and reporting system.

So far, it’s working!

Questions and Discussion

[email protected] https://apir.wisc.edu/pgp.htm

mailto:[email protected]

Lessons in Implementing a Post- Graduation and Learning-Gains Questionnaire Sara Lazenby...

Documents

Transcript of Lessons in Implementing a Post- Graduation and Learning-Gains Questionnaire Sara Lazenby...