So You Want to Do a Survey! University of California Assessment Network (UCAN) Gregg Thomson,...

So You Want to Do a Survey!University of California Assessment

Network(UCAN)

Gregg Thomson, Executive Director

Office of Student Research And Campus Surveys

University of California, Berkeley

May 5, 2009

A Quick Survey on Surveys:

I am of the opinion that for my work designing and conducting student surveys

is:

CHECK ALL THAT APPLY An essential tool OK, but who has the time! A way to score points with the

boss Only tell us what we already know A distraction from our real work Should be left to experts like

David Radwin Symptomatic of misguided worship

of numbers; surveys are part of the problem, not the solution

EXAMPLE OF STUDENT SERVICES SURVEY:

ELIMINATING THE PAPER VERSION OF THE

SCHEDULE OF CLASSES

OSR-OR SURVEY COLLABORATION

• OR identifies issues• Partners with OSR to

do survey• Approves survey• Provides funds

• OSR has survey expertise

• Collaborates on questionnaire

• Draws sample• Sends emails• Compiles results

QuickTime™ and a decompressor

are needed to see this picture.

What we will cover today

Transformation of Student Surveys Major Factors Associated with Surveys Decision Points Along the Way The Three Purposes of Customer Surveys Target Audience & Numbers Some Survey Design Techniques The Power of Open-Ended Questions Common Pitfalls When Survey Monkey and When OSR

The Transformation of Student Survey Capacity

Student Surveys before the Web The Transition to Web-Based

Surveying OSR as Center for Campus Surveys UCUES - Surveying All Students! Survey Monkey, Zoomerang, etc. The Good News - It’s so easy! The Bad News - It’s so easy! Demands for accountability

Initial Questions Associated with Student Surveys

Why are we surveying? Ostensible Need

What’s on the survey? Who are we surveying? How many? How and for how long? Results, then what?

Decision Points along the Way

Who or what is making us do a survey? What action, specifically, is to be

taken? What decisions have to be made? What information do we really need?

Questions that need to be answered How will each piece of information that we

obtain be used? What are our constraints?

Time Money Expertise

After we get the results?????

Three Purposes of Student Surveys

Exploratory or Learning What We Do Not Yet Know

Conclusive or Making Decisions or Resource Allocations on the Basis of What is Known

Performance Monitoring or How are We Doing this Year?

Who is My Target Audience?

• Your Whole Student Population• Pre-Determined Segments

• Linked to the survey results, or• Asked on the survey itself

• Categories that emerge from the survey

• Focus on satisfaction or dissatisfaction, “best practices” or what is not working?

How Large a Sample Size Do I Need for My Survey?

The traditional concern (cost) The “law of large numbers” Confidence Levels and Confidence

IntervalsExample: 95%, +- 3%, all

UC undergraduates (160,000)

Internet Sample Size calculators www.surveysystem.com/sscalc.htm

http://www.surveysystem.com/sscalc.htm

The New Rules: Size Doesn’t Matter

Size & costs - the changing landscape

Why not just survey everyone? Survey fatigue? Non-overlapping samples

What is a Good Response Rate?

What’s your guess? Examples Response bias 20% - 40% - 60% “rule” How many reminders? The role of incentives

Response Rates in the OR Survey

• 189 new freshmen - 51% response rate• 85 new transfers - 46%• 465 continuing freshmen - 37%• 88 continuing transfers - 34% • 155 doctoral students - 37% • 104 masters students - 29%

Constructing Items

Likert Items

QuickTime™ and aTIFF (LZW) decompressor


Check lists and variations



Pull-down menus



Reporting Behaviors



Demographics



Use branching effectively



Open-Ended Questions







“Please comment on the Online Schedule of Classes, including any advice on making it more useful to you”

SAMPLE STUDENT COMMENTS• Paper is out, online is in!• I use the online schedule of exclusively, and I have never seen a paper copy in my two years at Berkeley

• The online is more useful than the printed version that goes out of date the moment you buy it.

• I think it is great—who needs to use paper anymore?

• After using it, I cannot imagine why anyone uses the paper schedule.

• Please get rid of the printed version, as I feel UC Berkeley is just wasting paper when we have such great computer resources.

TOP 7 SUGGESTIONS FOR IMPROVEMENTS

1. Update information in real time like Tele-BEARS enrollment information and course restrictions.

2. Provide searches that filter out cancelled classes and full classes for both primary and secondary sections.

3. Modify the search results format to distinguish between primary and secondary sections, i.e., different font, size or color.

4. Include “none” in search for Restrictions.5. Replace Status/Date to Status/Last Changed on

search results.6. Include day/date/time next to final exam group

number on search results.7. Add Class Schedule Planner.

KISS and Common Pitfalls in the Design of Web-based Student Service Surveys

• A survey is just a KISS away ….• Everything & the kitchen sink

mentality• Assumption of centrality and saliency• Lack of filters and branching• Complicated and confused survey

wording; double-barreled, leading questions, etc. • Agree-Disagree: “This new web site is

easier to find and has the links I need.”

• Asking too much, e.g., ranking of choices

Survey Monkey and OSR: http://www.surveymonkey.com/

• Use Survey Monkey when:• You know what questions you need

answered• Simple target populations• You need answers fast• You have no money

• Use OSR when:• More involved, complicated surveys• You want more control over the

process, results, analyses; follow-up anticipated

• More lead time• You have some money

Please contact me:

Gregg ThomsonExecutive DirectorOffice of Student Research

and Campus SurveysUC Berkeley

Email is best: [email protected]

So You Want to Do a Survey! University of California Assessment Network (UCAN) Gregg Thomson,...

Documents

Transcript of So You Want to Do a Survey! University of California Assessment Network (UCAN) Gregg Thomson,...