May 2010

FAIR Best Paper: Using Technology to Efficiently and Effectively Gather

Information from Recent Alumni and Employers

May 2010

Association of Institutional Research50th Annual Forum

http://www.spcollege.edu/

AIR 50th Annual Forum May 2010

May 31, 2010 Association of Institutional Research 2

Alumni and Employer Surveys

Presenters Dr. James Coraggio, Director of Academic

Effectiveness and Assessment

Other Authors Not Present Maggie Tymms, Assessment Coordinator for

Academic Programs Nicole Carr, Former Assessment Coordinator for

Academic Programs



SPC Background

SPC, established in 1927, is the oldest 2-year college in Florida First Community College in Florida to offer

4 year degrees (2002) 9 Campuses throughout the county 2008-09 FTE: 16,966.8 (LD), 1,676.7 (UD) Opening Fall 2009 credit enrollment: 29,560 Annual 2008-09 headcount: 61,414



Institutional Effectiveness

“Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.”

Operating ResultsBudget PlanningStrategic & Unit

PlanningEvaluations/ AssessmentsMission

“Closing the Loop”



Performance Improvement

From Compliance to Performance Improvement

Data Information ImprovementKnowledge

Improved OutcomesSPCReportsAssessment Data

CommunityCollege LeadershipProgram Directors

FacultyStudents



Agenda

Student Assessment Points Purpose of Follow-up Surveys Alumni/Employer Survey Process Dissemination of Survey Results and

Reports



Student Assessment Points

Students surveyed multiple times



Purpose of Follow-up Surveys

Recent Alumni Survey was designed to: Gain insight into alumni’s perceptions of

the quality of general education and career preparation

Determine the current activities of former students such as employment information and educational status




Employer Survey was designed to: Gain insight into an employer’s perceptions

of hired SPC graduates in the areas of key general education competencies and foundational skills

Provide the employee’s major job responsibilities and wage information, as well as employer’s willingness to hire another SPC graduate

Identify employer’s willingness to support various college activities




State Requirements Survey data are also collected for state

required annual Placement & Follow-up Report (FETPIP: Florida Education Training Placement Information Program) and supplemental information to meet workforce criteria



Alumni/Employer Survey Process

Previous model (Before 2005) Labor intensive Manually tabulated paper surveys Telephone interviews conducted

using overtime hours




Issues with previous model (Before 2005) Costly survey process Declining college budget Increasing demand for accountability Outdated contact information Students over-surveyed Declining response rates Conducted in isolation




Goals of new survey processA. Improve data collection processB. Enhance quality of survey formsC. Employ industry best practicesD. Include discipline specific

informationE. Increase response rates



A. Improve Data Collection Process

Data Collection Process Eliminated telephone calls Automated the process (Remark Office

OMR and email merge) Employed online survey submission

(ZoomerangTM) Obtained accurate email addresses Used multiple points of contact




Survey Schedule Recent Alumni Survey

Graduates surveyed six months after the end of their last term (Three times per year)

Employer Survey Employer of graduates surveyed at the

end of academic year (Once per year)




First email Second email Third email

First mailing Second mailing

2 weeks

2 weeks

2 weeksRecent Alumni Survey Multiple Points of Contact

2 weeks



B. Enhance Quality Of Survey Forms

Improved Survey Forms Better ‘look and feel’ of survey format Developed scannable paper forms Used online branching models to reduce

survey length Designed items to gather only pertinent

data



C. Employ Industry Best Practices

Don Dillman’s Five Elements Respondent-friendly questionnaire Up to five contacts with the respondent Inclusion of stamped envelope Personalized correspondence A token financial incentive that is sent

with the survey request



Online Survey: Email and Form



Paper Survey: Letter and Form



C. Include Program Specific Information

Program specific information Met with Deans/Program Directors Discussed requirements for program

accreditation and timelines Included their necessary program

specific items



D. Increase Response Rates

Create awareness and convey importance Involved program faculty in the process Requested faculty assistance to inform

students Included a flyer with graduation

materials Coordinated with programs that hand-

carry surveys to local employers



Alumni Survey Response Rates

Summer 2007




Fall 2007




Spring 2008




Summer 2008




Fall 2008




Spring 2009




Summer 2009*

*Summer 2009 in progress




Source of Responses 2008-09




6.6%

23.5%

45.5%51.5%

54.9% 57.1%

0%

20%

40%

60%

80%

100%

2004-05 2005-06 2006-07 2007-08 2008-09 2009-10*

Percent of Responses Online





Overall Response Rate





Graduate Year

Surveyed

Number Surveye

d

Number Respond

ed

Percent Responde

d

Number of Online

Responses

Percent of Responses

Online

2008-09 4,181 1,134 27.1% 623 54.9%

2007-08 3,701 1,013 26.5% 522 51.5%

2006-07 3,953 658 16.7% 300 45.5%

2005-06 3,756 982 26.1% 231 23.5%

2004-05 3,417 1,494 43.7% 100 6.6%

Note: For 2009-10, only partial Summer 2009 results have been received.



Response Rates - Employer

OtherResponses

Recent Alumni Surveys Sent

Employer Surveys Sent

Is a 44% Employer Response Rate Good?

COE Responses

Permission RequiredResponses

Permission Not Required



Disseminating Information

College wide access to reports http://www.spcollege.edu/Central/Surveys/SurveyList.htm

Program Specific Reports Published on the Educational Outcomes

website http://it.spcollege.edu/edoutcomes

http://www.spcollege.edu/Central/Surveys/SurveyList.htm



Program Specific Reports



Recent Improvements

Survey invitations from program administrators (signatures)

Additional program specific reports (i.e., baccalaureate programs and PSAV)

Improved oversight (and QC) of printing functions



Lessons Learned

Do not expect that students will remember their ID#s

Administer the survey soon after the end of the session (six months)

Use multiple points of contact (two weeks) Find the most recent contact information Partner with programs to acquire up-to-date

information



Lessons Learned

Make students aware of the importance of the survey process (survey announcements)

Provide contact updates to college data systems (PeopleSoft, Alumni Foundation)

Monitor results and make changes as necessary Use automated processes for a quicker turn

around of results and more accurate data



Future Direction

Our ultimate goal is to provide Program Administrators ‘timely’, ‘relevant’ ‘accurate’, and ‘interpretable’ information regarding their programs and their performance outcomes Multi-measure ‘Dashboard’ style reports

(Business Intelligence/Performance Measurement)



Questions/Next Steps

FAIR Best Paper: Using Technology to Efficiently and Effectively Gather

Information from Recent Alumni and Employers

May 2010

Association of Institutional Research50th Annual Forum

http://www.spcollege.edu/

May 2010

Documents

Transcript of May 2010