Data Collection & Technology

EDUC 894 Week 6

Plan for Today• Next Week

▫ No class, consultations available ▫ E-mail me by next Monday to set up a time▫ Reading handouts

• Instrument Focus: WebSurveys▫ What NOT to do: An Example

How many problems can you find?▫ What To Do: Principles

Designing a Valid & Reliable Survey▫ The Wario Survey▫ Working with WebSurvey data

• Instrument Focus: Eye-tracking & Brainwaves▫ Field trip to the Engrammetron

-----------------------------Break-----------------------------• Group Work

▫ Honing down your instruments

Survey Design

•(Most) surveys are conducted from an objectivist perspective

•Writing a survey is all about making decisions:1. Recognizing when a choice is being made2. Making choices that will create a reliable

instrument that will help you make valid inferences to answer your research questions

3. It’s (usually) not about making the “best” decision in some abstract sense, but making a choice that will be most useful for your purposes

Limitations of Surveys

•Self-report data▫Notoriously unreliable▫Indicates only what we think, not reality

•Data at one point in time▫Limited Sample

•Retrospective questions rely on memory•The Devil is in the Details

▫Question form, wording, and answer choices can inadvertently dictate responses

•Sampling ▫Who responds isn’t random

Image Source: http://www.nynpa.com/images/NYNPA0081.jpg

Survey Decisions

•Topics Covered•Number & Focus of Items per Topic•Item Sequence & Presentation•Item Type / Format•Item Wording

•Always think about what you will do with the answers you get – i.e. how will you analyze the data?

What NOT to do

•Take the survey posted here (link sent via email)

https://my.sfu.ca/cgi-bin/WebObjects/WebSurvey.woa/wa/survey?17437110

•How many problems in each category can you find? ▫Topics Covered▫Number & Focus of Items per Topic▫Item Sequence & Presentation▫Item Type / Format▫Item Wording

Survey Decisions Topics Covered• Remember the idea of content-related validity?

▫Goal:Determine the most popular animals▫Assumed “like” indicates popularity

What pet you had as a child may or may not relate Are the plants in you garden a better indicator?

Feelings about gardening don’t give relevant info Caveat: Questions to lay a foundation

• Define your terms ▫Are you concerned with popularity of animals (in

general) or only pets? Can ask a similar question about plants / garden

plants▫What about ability self-esteem (in general) or

self-esteem as related to academic work? How would this affect your survey design?

Survey Decisions Topics Covered

Self Esteem

Social

Academic

Athletic

Survey

Survey Decisions Number & Focus of Items per Topic•Number: Do you have enough questions

on each topic to give a reliable measure?▫Three questions is usually the absolute

minimum given for a scale•Focus: Do questions give you info you can

actually triangulate?▫How will you compare the names of plants

reported to be in their garden with Likert scales given for specific plants?

▫What will you do with the question about liking “trees without flowers”?

Survey DecisionsItem Sequence & Presentation• Question sets and introductions• Broad to narrow progression

• How satisfied are you with service in the food court?• Do servers in the food court smile when taking your

order?• Will you group or distribute similar content items?

• Putting similar items together will help learners think about what you want them to

• But, if you have multiple items try and get convergence on a construct, putting them near each other will lead to a false reliability

• Pagination• No one likes to scroll forever• No one likes unnecessary clicking

• The “required item” dilemma

Survey DecisionsItem Type / Format• Open versus Closed Ended Questions

▫It all depends on what you are looking for, but from an objectivist perspective:

“Use close ended questions where at all possible, if you can limit the possible answers it can really speed up the data crunching when you

are finished”

▫You may need open-ended questions when you don’t know the range of possible answers But then you will have to go through and code

these answers for themes▫Thus, don’t ask an open-question with a limited

number of possible answers or where answers range predictably “Do you like to garden?” “How much do you like dogs?”

Survey DecisionsOpen Ended Question Guidelines•Be specific

▫What did you think of class? BAD▫What 3 things did you like best about class? GOOD▫ What 3 things did you like best about the class content?

BETTER•Leave people enough space to write

▫A philosophy of life, love and happiness is not 64 characters or less!

•Don’t ask too many open-ended questions▫Your participants will tire and it will show in their

responses G. W. Bush example

Survey DecisionsClosed Ended Question GuidelinesQuestion Types 1

▫Yes/No▫Lists

• Single choice (“best answer”, radio button)

• Multi (“all that apply”, checkbox)• Can do a series of these in a grid, but

dangers of confusion and auto-pilotListen Speak Read Write

English □ □ □ □

Spanish □ □ □ □

French □ □ □ □

Italian □ □ □ □

Survey DecisionsClosed Ended Question Guidelines

Question Types 2▫ Rating Scales

• Ordinal • E.g. Rank the following from best to worst

• Likert • E.g. Levels of agreement

• Likert-style • E.g. Levels of approval, satisfaction etc.

• Frequency • E.g. Often, sometimes, almost never

• Bipolar • E.g. Outgoing to introverted

Survey DecisionsClosed Ended Question Guidelines

•Focus on Rating Scales▫ Graphic vs. Finite Choice

▫ Think about the following:• Does your scale goes to the extremes?• Does your scale have enough choices?

• Participants are less likely to choose the anchors (n-2)

• Does your scale imply interval spacing?• Will your scale have a “neutral” option?

Mark an X on the line to show your level of agreement

Disagree Agree

Indicate your level of agreement by circling your choice

Strongly Disagree Disagree Neutral Agree Strongly Agree

Survey DecisionsItem Wording 1•Clear & Concise

▫What do you do when you get a headache? UNCLEAR

▫What headache medicine do you use? CLEARER

•Concrete▫Did you enjoy the book? ABSTRACT

▫Did you recommend the book to a friend? CONCRETE

•One idea per question▫Should BC continue its urban density and needle

exchange programs? CONFOUNDED

•Be careful when asking personal questions▫Use categories to avoid emotional responses

Survey DecisionsItem Wording 2•Avoid leading questions

▫Do you think Barak Obama will continue to be a successful candidate?

▫Do you think Barak Obama will continue to be plagued by challenges about his level of experience?

•The challenge of reversed questions▫Can be used to break up question sequence or to get

at something in a different way, but big reliability issues I do not like venus fly-traps EXPLICIT

Self-Esteem Survey: Sometimes I doubt myself IMPLICIT

Survey DecisionsThe Big Picture

•Every time you have to make a decision, always think about 3 things1. How will your decision affect

participant responses?2. How will your decision affect your

analysis?3. What does your decision imply about

the underlying construct?

The Wario Survey

Working with WebSurvey Data

Field trip to the Engrammetron

Break

Group Work