CAUCE DIAL AND DISCUSS TUESDAY, 23 FEBRUARY 2010 STANLEY VARNHAGEN University of Alberta KAREN...

CAUCE DIAL AND DISCUSSTUESDAY, 23 FEBRUARY 2010

STANLEY VARNHAGENU n i v e r s i t y o f A l b e r t a

KAREN HAYWARDU n i v e r s i t y o f S a s k a t c h e w a n

Using Surveys to Maximize Strategic Planning

Dial and Discuss: Using Surveys

2

Overview

Surveys and the Planning ProcessUnderstanding SurveysDelivery MethodsClosing Tips

23 February 2010


3

Surveys and the Planning Process

23 February 2010


4

Surveys and Planning

23 February 2010

Are surveys needed/useful for planning or do they just get in the way (slow things down)?

Depends on the timing of the planning Useful for evidence-based decisions Surveys can be an efficient means of collecting data Properly done, they can be a valuable tool that

increases the chances of success


5

Surveys are useful when…

23 February 2010

information is needed to examine population characteristics

a larger response pool is neededexamining subgroup data is important (e.g.,

gender or age differentials)breadth of response is more important than

depth


6

Surveys & Evaluations

23 February 2010

Surveys are effective tools with most types of evaluations including: Needs assessments

Collecting information about the target audience Formative evaluations (midterm)

Collecting information that helps make adjustments & improvements

Summative evaluations Collecting information that helps determine if goals and

objectives are being met (merit, value, & worth) Maintenance evaluations

Collecting information a period of time after whatis being evaluated is complete, to see if the objectives are still being met


7

Surveys and Strategic Planning

23 February 2010

Surveys can help us:Better understand potential

audiences/users/students (consider different segments)

Collect baseline data that can be compared with subsequent survey data

Test assumptions or preliminary products (trial balloons)

Examine existing and potential markets


8

Understanding Surveys…

23 February 2010


9

23 February 2010

What Surveys Can Offer

Collect suggestions to improveCatch problemsTrack trendsGet anonymous feedbackGet buy-in

Gain understandingIncrease awareness

2 way processEasy and effective way to collect dataWith an appropriate sample and response

rate quantitative data can help generalize to the population


10

23 February 2010

Strengths of Surveys

Easy to createEasy to analyzeRelatively inexpensiveCommon/respondent familiarityHandle both quantitative and qualitative

dataCan be relatively time efficient to takeReal time resultsInstant Feedback


11

Additional Reasons to Use Surveys

23 February 2010

Can be cost savings, to allow feedback before considerable time and resources spent Storyboards/scripts

Allow feedback while modification is still possible Helpful to the developer and the user

Determine if goals and objectives have been met


12

23 February 2010

Weaknesses of Surveys

Lack of follow-upNot best suited for collecting qualitative

dataLimited flexibilitySampling problems not always obviousCan miss key results

Results are dependent on the questions/options that are presented

Garbage In Garbage Out (GIGO)


13

Delivery Methods

23 February 2010


14

Survey Delivery Methods

23 February 2010

Surveys can be conducted using different delivery methods These methods have different strengths and

weaknesses that should be considered The three different delivery methods considered here

are: Online Paper-based Telephone

Currently Online Surveys are probably themost popular method


15

23 February 2010

Online

Advantages: Minimal distribution

costs Automatic data entry Automatic

compilation possible Branching possible Potentially quickly

modified or updated Real time data

collection potential Can use

multimedia

Disadvantages: SPAM Filters/Broken

Links Duplication a concern User competency

dependent (submit) More complicated Anonymity a concern Potential user cost (Long

distance/ISP) Access


16

Other Online Considerations

23 February 2010

Response rateLengthItem types

Forced choicesMaking changesNavigation

Going to previous pages

Submitting surveyAccommodating

schedules

Different browsers Colors Size

Exactly who is responding

Location of the survey server

Online survey fatigue


17

23 February 2010

Paper and Pencil

Advantages: Simple (for

respondents and evaluator) No special training

required Easy to go back and

change responses Can have a higher

response rate

Disadvantages: Distribution

Can be slow Branching Following

instructions Usually required

manual data entry Reading writing

Qualitative responses Costs


18

23 February 2010

Phone

Advantages Can sometimes

provides higher response rates

Ability to branch Limited ability to

follow up response Allows assessment of

response quality Longer surveys

possible

Disadvantages Cost

Interviewer Long distance Persistence

More obtrusive Can be perceived

anonymity problems Selective responses Caller ID & Voicemail

Telemarketers


19

General Survey Considerations

23 February 2010

Use (needs assessment, formative, or summative)

Data entry methodNumber of

response options 5, 7, 99, or unlimited Even or odd

Effective scale

Anchors Continuum versus

descriptorsStatement

questionsNormative

referencedValidityReliability


20

Other Considerations

23 February 2010

Completion Time Written versus

telephone Class versus own

timeMixed methodsTriangulationPiloting

IncentivesRoutingFundingBiasTerminology

AcronymsReading ability


21

23 February 2010

Other Considerations (cont’d)

Anonymity Aggregating data Removing identifiers Looking at trends across surveys

Differential responses Only the “keeners” complete or vice-versa Some other important characteristic determines

who respondsSamplingSocial desirabilityTime survey is available


22

Bias

23 February 2010

Can affect results, but is often not obviousQuestion wording

Leading questionsOptions available

Missing optionsQuestion order

Specific to general Carry over

Option orderGeneral survey expectations


23

Creating Surveys… Examples

23 February 2010

Dial and Discuss: Using Surveys 23 February 2010

25

Item Types

Likert - Interval data how many options even or odd additional options direction anchors

I feel very comfortable taking online distance courses.1 2 3 4 5Strongly Disagree Neutral Agree Stronglydisagree agree


26

23 February 2010

End-of-course Evaluations

Uses Likert ItemsFive optionsStandardizedNormative

Effective scaleSocial desirabilityHow would we know when assumptions have

been violated?Intuitive meaning

Dial and Discuss: Using Surveys 23 February 2010

27

Item Types (continued)

Modified Likert Not agree/disagree items Continuum (options) changes Can use fewer anchors

2, 3, or 5

How comfortable do you feel online distance courses?1 2 3 4 5Not at all Somewhat Verycomfortable comfortable comfortable


28

23 February 2010


Multiple choice Nominal data

Reporting results Picking the correct

choice Instructions

Select best Select all that apply

Missing options

Where do you plan to complete most of your online coursework?

a)Homeb)Workc) Computer labsd)Not suree)Other (please specify):


29

23 February 2010


Rankings Following instructions Can force distinctions Can be harder to interpret results

Fill-in-the-blanks Can be time consuming to code Generalization

Open ended Use open-ended questions carefully

Exploratory versus confirmatory Time Potentially rich


30

Closing Tips

23 February 2010


31

23 February 2010

Survey Writing Tips

Demographic information Focus on what information will be useful for

analysis Recognize many consider the information given

to be sensitive Caution about subject identification

Trade-offs Don’t make responses too easy


32

23 February 2010

Survey Tips (cont’d)

Instructions Clarify purpose/reasons Anonymity Voluntary Withdrawal Ethics

Implied Consent Dissemination

When and where results will be found


33

23 February 2010

Survey Pitfalls

Want to get the largest proportion possible to respond Reduced the chance of self-selection Increases likelihood of accurately generalizing

resultsIncreasingly difficult to get high response

rates Frequency being surveyed

Survey fatigue Perceived need/relevance


34

Tips to Increase Response Rates

23 February 2010

Well written cover letter from meaningful individual Explaining survey purpose and relevance

Honorarium – given with survey Draw (e.g., gift certificate)

Keep survey as short as possibleProvide reasonable deadlineReminders


35

23 February 2010

Reference

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009) Internet, Mail, and Mixed-Mode

Surveys: The Tailored Design Method (Third Edition). New York: John Wiley & Son.


36

Questions?

23 February 2010


37

23 February 2010

Contacting Us

Stanley Varnhagen University of Alberta phone: (780) 492-3641 email: [email protected]

Karen Hayward University of Saskatchewan phone: (306) 966-7384 email: [email protected]

mailto:[email protected]




CAUCE DIAL AND DISCUSS TUESDAY, 23 FEBRUARY 2010 STANLEY VARNHAGEN University of Alberta KAREN...

Documents

Transcript of CAUCE DIAL AND DISCUSS TUESDAY, 23 FEBRUARY 2010 STANLEY VARNHAGEN University of Alberta KAREN...