APF orlando diy survey workshop 071114 final

46
www.aperioinsights.com Revealing Insights for Clear Decisions July 11, 2014 APF Orlando

description

DIY Survey workshop presented to APF (Association of Professional Futurists) pro-dev meeting July 2014. Covers basics of DIY quant survey tips.

Transcript of APF orlando diy survey workshop 071114 final

Page 1: APF orlando diy survey workshop 071114 final

www.aperioinsights.com

Revealing Insights for Clear Decisions

July 11, 2014APF Orlando

Page 2: APF orlando diy survey workshop 071114 final

2 Revealing Insights for Clear Decisions

Welcome to DIY Survey Research

Page 3: APF orlando diy survey workshop 071114 final

3 Revealing Insights for Clear Decisions

Mike Courtney Researcher | Ethnographer | Futurist

Educated and trained in both traditional research and long range scenario planning.

MMR degree in research, published author, angel investor.

Data Analysis | Ethnography Ideation | End User RequirementsNew Product Validation | Roadmaps

Page 4: APF orlando diy survey workshop 071114 final

4 Revealing Insights for Clear Decisions

Do YOU Research?

Term Research is vague Research? Sure! I use Google almost every day.

– Many don’t know if they need it.

We don’t really need research. What we need is sales and marketing help.– Our ads aren’t getting the response we expected. We might need to look

for a new ad agency. – Competition just stole our top 3 customers. We need a better sales

trainer or maybe a new sales contest.– I already did the research. Both my mom AND dad loved the idea. My

golfing buddies also think its awesome. Now I just need a couple million $ in funding so I can hire someone to do the marketing and sell it.

– Sales haven’t yet taken off, I think I just need better SEO.

Page 5: APF orlando diy survey workshop 071114 final

5 Revealing Insights for Clear Decisions

Why Do Research?

Three Simple Reasons To reduce risks

To discover opportunities

To improve business results

One Metric for Success — Answers that Lead to Actions

“After conducting thousands of marketing research studies and asking hundreds of thousands of questions, the author has come to understand one thing: There are no bad questions, only irrelevant ones. In other words, the majority of questions asked are irrelevant. That is, they don't result in answers that lead to actions.”The Secret to Great Marketing Research: Ask the Right Questions by Robert J. Kaden

Page 6: APF orlando diy survey workshop 071114 final

6 Revealing Insights for Clear Decisions

Research has Changed

COSTS • Even straightforward projects cost big $

• Simple projects are inexpensive or free

TIMING • Took months, hard to predict completion date

• Often done in weeks if not days or hours

TOOLS • Tools were basic – skilled experts needed

• Tools are powerful and geared towards non-experts

OUTPUT • Hard to change final output once done

• Easier to adjust or change final output

BACK THEN NOW

Page 7: APF orlando diy survey workshop 071114 final

7 Revealing Insights for Clear Decisions

Today’s Research Tools are Easy and Powerful

But could they be dangerous?

Page 8: APF orlando diy survey workshop 071114 final

8 Revealing Insights for Clear Decisions

Safe Surveys –What We Will Cover

When to use Quant vs. Qual

Research brief basics– capturing the objectives upfront (Team Exercise) Turning objectives into questions – subtle differences that add depth

Common Scales and other answer options – how to choose, examples, etc

Common mistakes and traps – rules of thumb (Featuring Dr. Bishop handout)

Survey question practice - you can do it, we can help (Team Exercise) – Show and tell – share your questions and get reactions/critiques

I need how many? – The significance of sample size

Finding Respondents - Survey Sample Options

Survey and analysis tools - low cost options

Final Q & A

Page 9: APF orlando diy survey workshop 071114 final

9 Revealing Insights for Clear Decisions

Refresher — Qualitative vs. Quantitative

Qualitative (Qual):– Discover and understand the “why” or the rationale behind actions– Explore the range of possible perceptions and beliefs– Understand the language used and how it impacts understanding– Helps develop ideas and hypothesis to test

Quantitative (Quant):– Measure the percentage who think a certain way– Project findings onto larger population– Gather feedback from a larger audience– Benchmark and periodically re-measure

Page 10: APF orlando diy survey workshop 071114 final

10 Revealing Insights for Clear Decisions

Research Brief Keeps Everyone on Target

Background – Relevant background to understand context of the request including the scope of project & information currently available:

Existing Research – what data/knowledge do you already have? Primary, secondary, anecdotal

Decision[s] to be made – What are the key decisions that will be made? What are the primary reasons to consider conducting this research?:

Specific objectives/Hypotheses – Are there any theories about what will be learned? What information will affect the decision[s]? What do you need to prove/find out? Ideally, what will you know when the research is completed?:

Additional decision data required – If the research IS successful in gathering the required information, what other data might ALSO be needed before making the decision[s] in question?:

Target respondents – What criteria (demographic, product usage and other specific requirements) must the respondents meet in order to qualify for this study? Please note any hard quotas or splits:

Justification – $ implications / ROI / reasons why project is important:

Deadline – For final results:

Page 11: APF orlando diy survey workshop 071114 final

11 Revealing Insights for Clear Decisions

Team Exercise: Use a Research Brief to Outline your Objectives

Team Task– Complete the research brief and outline the specific survey objectives

that will form the outline of a survey. Pick a survey topic or use below.

Example Survey Topic– As an APF member you understand the value a good futurist can bring to

an organization. – Unfortunately there are still many people in the world who have no idea

what you do and how it might help them. – Imagine you are working on a project to identify potential NEW clients to

approach in order to educate, work with, etc. – Your team has been asked to create a survey to understand and measure

WHO may be interested in working with a futurist.

Page 12: APF orlando diy survey workshop 071114 final

12 Revealing Insights for Clear Decisions

TEAMS DEVELOP AND PRESENT OBJECTIVES

OBJECTIVE = SOMETHING YOU WANT TO LEARN/CONFIRM/MEASURE

(TEN MINUTES? DON’T OVERTHINK IT)

Page 13: APF orlando diy survey workshop 071114 final

13 Revealing Insights for Clear Decisions 13

Watch out for too many or too few response options

“Capital punishment should be reintroduced for “serious crimes”

1 = Agree 2 = Disagree1 = Very, Very Strongly Agree 7 = Slightly Disagree2 = Very Strongly Agree 8 = Disagree3 = Strongly Agree 9 = Strongly Disagree4 = Agree 10 = V. Strongly Disagree5 = Slightly Agree 11 = V, V Strongly Disagree6 = Neutral

Page 14: APF orlando diy survey workshop 071114 final

14 Revealing Insights for Clear Decisions 14

Example: How could this question be improved?

How old are you?___ 18-20___ 20-22___ 22-30___ 30 and over

Page 15: APF orlando diy survey workshop 071114 final

15 Revealing Insights for Clear Decisions 15

Are you satisfied with your marriage and your job?__________________________

Example: How could this question be improved?

Page 16: APF orlando diy survey workshop 071114 final

16 Revealing Insights for Clear Decisions 16

You didn’t think the food was very good, did you?

_____ Yes _____ No

Example: How could this question be improved?

Page 17: APF orlando diy survey workshop 071114 final

17 Revealing Insights for Clear Decisions 17

Environmental issues have become increasingly important when choosing respectable hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation?

____ Yes ____ No

Example: How could this question be improved?

Page 18: APF orlando diy survey workshop 071114 final

18 Revealing Insights for Clear Decisions 18

What information sources did you use to locate your restaurant for today’s meal?(please tick appropriate spaces) ____ yellow pages ____ Internet ____ word of mouth

Example: How could this question be improved?

Page 19: APF orlando diy survey workshop 071114 final

19 Revealing Insights for Clear Decisions

Lenovo Survey Example

Page 20: APF orlando diy survey workshop 071114 final

20 Revealing Insights for Clear Decisions

Lenovo Survey Example (Cont)

Page 21: APF orlando diy survey workshop 071114 final

21 Revealing Insights for Clear Decisions

Lenovo Survey Example (Cont)

Remember – pick only FIVE items to rate

Page 22: APF orlando diy survey workshop 071114 final

22 Revealing Insights for Clear Decisions

SURVEY QUESTION CRASH COURSE: TIPS, TRICKS AND RULES OF THUMB

Page 23: APF orlando diy survey workshop 071114 final

23 Revealing Insights for Clear Decisions

Survey Templates Readily Available

Page 24: APF orlando diy survey workshop 071114 final

24 Revealing Insights for Clear Decisions

Basic Questions Get You Started

Page 25: APF orlando diy survey workshop 071114 final

25 Revealing Insights for Clear Decisions

Subtle Question Differences Add Depth (PS -don’t use this many)

Management and FairnessMy manager treats all his/her employees fairly.

My manager knows what is going on in my work group.Favoritism is not an issue in raises or promotions.

My manager is always consistent when administering policies concerning employees.I am always treated fairly by my manager.Everybody is treated fairly in this organization.My manager establishes plans and work objectives with me.My manager gives me clear instructions.

My manager is available when I need advice.

My manager (immediate supervisor) trusts me.

My manager helps me to improve myself.My manager takes prompt and fair corrective action on employees who fail to perform their work satisfactorily.

I feel free to talk openly and honestly to my manager.My manager praises me when I do a good job.My manager holds regular meetings with my work groups.My manager is effective in making decisions.

TIP: Build and fine tune questions over time; share question lists with other DIY researchers.

Page 26: APF orlando diy survey workshop 071114 final

26 Revealing Insights for Clear Decisions

Survey Tips

Whose opinions matter? Need to ask a sample of respondents who you think might be worth targeting in the future. Don’t ask mom if you doubt she would be a future customer etc.

Understand without Influence - Trying to get true perceptions – not trying to influence, sway, convince etc.

Bias comes in many forms – question wording, concern for interviewer, those who know you, those who take pity on you and want to help, fatigue, low category involvement etc.

Behavior then perception – ask first about what they have done, they what they think about it

KISS principal – Simple wording is often best.

Define Timelines - Have you …how much do you…how often do you….ALL require some sort of timeline.

Define categories & avoid mental math– What % of your beverages are coca cola products? Does % mean number of drinks per day? Are they asking about % as in how many ounces of product or bottles? Cups? Fountain? Should I include bottled water sold by coca cola corp? etc.

Chocolate Cake Factor - Most real world decisions involve many factors. Need to tell the rest of the story – or enough to make sense. Don’t forget to ask about the major factors that would influence behavior.

Page 27: APF orlando diy survey workshop 071114 final

27 Revealing Insights for Clear Decisions

Basic tips for Choosing Scales

Don’t recreate the wheel Choose question stems carefully Reverse word some of the items Use the same scales each time data is collected Avoid mixing too many scales in one survey Format scales for one direction – negative to positive etc.

Balance Scales – Agreement, quality and satisfaction scales use an equal number of options on either side (positive vs. negative) of a scale, each with similar wording.

Scales vs. Checklists – Sometimes it’s better to use a checklist instead of having everyone mark the same scale item multiple times.

https://www.case.edu/president/cir/wordfiles/tips_choosingscales.docx

Page 28: APF orlando diy survey workshop 071114 final

28 Revealing Insights for Clear Decisions28

I need how many?The significance of sample size

Page 29: APF orlando diy survey workshop 071114 final

29 Revealing Insights for Clear Decisions

Example Survey Flow

Screeners – Questions that confirm the respondent meets some minimum criteria

Category Involvement – Questions that explore the level of activity in the category

Stated Needs & Pain Points – What do they like/not like about their current___?

Concept Introduction – Imagine there is a new __ available. Here’s a description of it...

5 Key Questions – Appeal, unique/different, satisfies a need, believability, value for money/Interested in learning more etc.

Suggestions – How would you improve or change it?

Demographics – Age, gender, income, occupation, education, etc.

Bizographics – Company size, industry sectors, locations, etc.

Psychographics – Personality, values, attitudes, interests, or lifestyles

Page 30: APF orlando diy survey workshop 071114 final

30 Revealing Insights for Clear Decisions

Common Questionnaire Mistakes – Dr. Bishop’s ListIssue Definition Example

Ambiguous terms Terms that can mean more than one thing. Do you believe in freedom for all?

AnchoringThe response is biased by the last information presented to the respondent, usually dealing with numerical information.

The U.S. has over 39 million without health insurance. How many uninsured does Germany have? versus

The U.S. has over 200 million with health insurance. How many insured does Germany have?

Biased wordingTerms that indicate the attitude of the questioner and influence the respondent to answer the same way.

Do you agree that worthless people should be excluded from welfare payments?

Complex questions Items that are too difficult to answer because the question is too complicated.

For each of the following courses, indicate whether the course should be offered, when it should be offered, and by whom it should be offered. In the last column, show whether the course should be offered by adjuncts or full-time professors and whether the courses should include only undergraduates, graduates, or both.

Double-barreled questions Asking two questions in one item. Do you think the President should cut defense spending and reduce the national debt?

Incomplete list Response categories that do not contain all possible responses. Protestant, Baptist, Catholic, Jewish

Jargon Terms that are unique to a particular field. All health education curricula should contain courses on RVRBS.

Lack of a filter Questions that apply to only a subgroup without filtering. What was your major in college?

Lack of knowledge The respondent does not have the information requested. When was the last time a President visited Japan?

Mutually inclusive categories Response categories that overlap and contain the same material. Protestant, Baptist, Catholic, Jewish

Negative questions Items that contain negations. Do you agree that the U.S. should not bomb the Serbs in Bosnia?

Spurious exactitude Items that ask respondents to make distinctions that are too fine. Rank the following 20 items how important they are for the future.

Page 31: APF orlando diy survey workshop 071114 final

31 Revealing Insights for Clear Decisions

Survey question practice - you can do it, we can help (Team Exercise)

TEAM TASK–SELECT 2 SPECIFIC SURVEY OBJECTIVES AND

CREATE 2-3 SURVEY QUESTIONS FOR EACH.–USE OBJECTIVES FROM LIST PROVIDED OR

YOU MAY USE YOUR OWN OBJECTIVES.–QUESTION EXAMPLES FROM EACH TEAM

WILL BE PRESENTED AND CRITIQUED.

Page 32: APF orlando diy survey workshop 071114 final

32 Revealing Insights for Clear Decisions

Examples of objectives: (see handout) Screening Criteria: Understanding who should be surveyed/not surveyed. Will you target any particular industry, sector

or organization type? Your list of emails to survey is a big hot mess, should anyone be screened out? Targeting/Sampling: Which job roles or titles are most likely to be relevant to this study? Who would normally deal with

long term strategy and scenarios? Who would benefit and/or be a potential champion for the process? General Background - What background or previous experience will the respondents have? What about their

background would be useful to understand? Awareness of Futurists and Foresight. Do these organizations worry about the future? How much? History of Involvement - Experience or history with doing this type of process? Do these respondents have any

knowledge of, or experience working with futurists? Current State/Situation - Rate of change being experienced by org/industry now. Key issues they are dealing with that

create disruption/risk. Does org experience much change? Where are the major sources of change coming from? Do different rates of change impact interest in working with futurists?

Outsourcing attitude – how often do they hire an outside group for help of any kind? How they view Futurists vs working with some other type of consultant/agency Do they often (ever?) use outside help/consultants?

Time Horizons - What time horizons do these organizations typically deal with? What does short vs long term typically mean to their organization or industry?

Understanding/Common Lexicon - Do they understand the same terms, acronyms and concepts you do? Are you sure you are speaking the same language?

Internal Capabilities/Readiness - What parts of the process would they most like help with vs feel confident to do themselves? Is their organization ready to take on a project that focuses on the future?

Attitudes/Perceptions - Does their mindset or attitude towards your process matter? What would success look like and how would they measure results of a project?

Reaction to the Concept – Once the basics of the process is explained, what are their reactions to it? Do they understand what is possible, services offered, value potential, etc.?

Sources of learning/knowledge – How do they keep up to date on new methods? Where do they look to learn about new topics or approaches to organizational issues?

Alternatives - If they wanted to think about the future, where would they look for help? If they didn’t hire a futurist to help think about the future, who WOULD they likely hire?

Page 33: APF orlando diy survey workshop 071114 final

33 Revealing Insights for Clear Decisions

TEAMS DEVELOP AND PRESENT SURVEY QUESTIONS

(15 MINUTES?)

Page 34: APF orlando diy survey workshop 071114 final

34 Revealing Insights for Clear Decisions

DIY RESEARCH TOOLS

Page 35: APF orlando diy survey workshop 071114 final

35 Revealing Insights for Clear Decisions

Rapid Rise of DIY Research Tools

Tools for Easier Data Capture and Analysis Quant survey and qualitative interviewing software

Analysis tools for the visual display of data

Tools that offer new hybrid methodologies

Features Anyone Can Use Survey tools with more analysis features—built in cross tabs

Automate common tasks—graph output to PowerPoint and stat testing

Templates, canned reports and standard graphics

Tools marketed directly to non-research end users

Page 36: APF orlando diy survey workshop 071114 final

36 Revealing Insights for Clear Decisions

Finding Respondents - Survey Sample Options

Competition among traditional sample sources continues to reduce price points. Some offer DIY “self-serve” sample tools for easy access. Non-traditional sample sources such as Google and Mechanical Turk can appear

cheap but are not always nationally representative, or the best value depending on the survey length. Respondent quality is often and unexpectedly uneven for both sources and data cleaning is critical to meaningful results.

FULL FEATURE, FULL PRICE• GreenBook.org – List of

sample/panel companies such as: • Survey Sampling

• Cint• ResearchNow

• Wide-open, nationally representative sample is currently running $3-6 per complete. Lower incidence studies can still easily top

$10-14 per complete.

SCRAPPY ALTERNATIVE

• Mechanical Turk• Google Sample

Page 37: APF orlando diy survey workshop 071114 final

37 Revealing Insights for Clear Decisions

Quantitative Survey Tools

Many companies have, but don’t fully or frequently use their expensive software licenses. If you really need the full featured versions consider partnering with or

consolidating your survey software activity among several companies.

FULL FEATURE, FULL PRICE

• Qualtrics – Lots of features, popular with academic crowd

($5-10K+/year)

• ConfirmIt – Industrial strength feature set, popular with larger

research shops ($10K+/year)

• SawTooth Software – The standardfor classic conjoint and MaxDiff

($3-10K 1x fee)

SCRAPPY ALTERNATIVE

• SurveyGizmo – Free version (8 question types & 350 responses) or

Basic version $19/month (28+ question types & unlimited

responses)

• SawTooth Software – Make friends (and a deal) with someone who

owns a license

Page 38: APF orlando diy survey workshop 071114 final

38 Revealing Insights for Clear Decisions

Quantitative Analysis Tools

Many survey tools now have built-in analytics capabilities that can handle the basics. If you really need the full featured versions consider partnering with or consolidating

your survey software activity among several companies.

FULL FEATURE, FULL PRICE

• SPSS – One of the original standard analytic tools. Lots of statistical

features, poor graphics. Now IBM owned and 5-10x the old prices

($5-40K+)

• MarketSight – Super easy to use online analytics tool. Can create PPT

charts in batches from crosstabs ($99/year academic license, $1,600 standard license, free 30 day trial)

SCRAPPY ALTERNATIVE

• R Project for Statistical Computing –Popular open source stats package, tricky to install with basic graphics

(FREE)

• Excel – Easy for basic crosstabs and many analysis templates exist for FREE. Most of us already have an

old copy laying around right? Or use Google docs

Page 39: APF orlando diy survey workshop 071114 final

39 Revealing Insights for Clear Decisions

Visualization Tools

Information overload means effective researchers need to have valuable data AND be capable of using it to tell a clear and compelling story. Visualization tools are useful

for exploring data in the analytics phase as well as turning key findings into easily understood and eye catching output.

FULL FEATURE, FULL PRICE

• Tableau Software – Drag and drop visual analytics tool. Capable of handling various data types and

creating great visual outputs with a little practice ($999/year hosted

edition, $1,999 pro version)

SCRAPPY ALTERNATIVE

• RAW –Newly announced online visualization tool with 7 graphic

styles and ability to export finished result (FREE)

• Tableau Public – “Lite” version of drag and drop visual analytics tool, main caveat is your output must be

publicly shared

Page 40: APF orlando diy survey workshop 071114 final

40 Revealing Insights for Clear Decisions

Consider Methods Like MaxDiff

Requires either SawTooth OR Survey

Analytics survey software

Page 41: APF orlando diy survey workshop 071114 final

41 Revealing Insights for Clear Decisions

When Should I Use All This?

Trend is towards conducting smaller, more frequent studies that provide a constant stream of feedback instead of a few large studies

Common Research Touch Points Include: – Idea or concept stage, before any development has begun– MVP or proto stage, testing a working version of the core offer – Development milestones as features are added/perfected– Friendly user stage – full product but pre-full public release– Post launch – periodic testing to track how expectations are

evolving as users move past novelty and into routine use phases– Timing also dependent on pace of change and response times in

the industry sector

Page 42: APF orlando diy survey workshop 071114 final

42 Revealing Insights for Clear Decisions

Keeping Score

Don’t look at just one score

Averages vs. Segmentation

Page 43: APF orlando diy survey workshop 071114 final

43 Revealing Insights for Clear Decisions

Q & A

Page 44: APF orlando diy survey workshop 071114 final

www.aperioinsights.com

Revealing Insights for Clear Decisions

For more information, contact:

Mike Courtneym: 469-363-0109e: [email protected]

Page 45: APF orlando diy survey workshop 071114 final

45 Revealing Insights for Clear Decisions

Links! Survey basics and Tips

https://explorable.com/course/the-survey-guide

http://www.slideshare.net/jtneill/survey-design-workshop?qid=307afda6-63e0-45ce-a197-3fda3491513f&v=default&b=&from_search=2

http://blog.askyourtargetmarket.com/2011/04/survey-design-tips-bad-questions-made.html

http://www.marketingprofs.com/short-articles/2109/three-errors-that-lead-to-bad-survey-questions

http://www.focusgrouptips.com/marketing-research-surveys.html

http://www.focusgrouptips.com/qualitative-research.html

http://www.b2binternational.com/publications/questionnaire-design/

http://www.qualtrics.com/blog/writing-survey-questions/

http://www.researchplan.com/blog/?p=51

http://www.surveysystem.com/sscalc.htm

http://www.slideshare.net/horatjitra/designing-questionnaire-and-survey-research

Page 46: APF orlando diy survey workshop 071114 final

46 Revealing Insights for Clear Decisions

Links! Scales and Sample Frame basics

Scaleshttp://www.360degreefeedback.net/media/ResponseScales.pdfhttp://www.uwex.edu/ces/4h/evaluation/documents/Wordingforratingscales.pdfSample frame PPT’shttp://www.qualtrics.com/blog/determining-sample-size/http://www.slideshare.net/heenadhawan86/sampling-bigslideshttp://www.slideshare.net/hafizahhajimia/research-method-sampling?qid=d8d874cd-dae6-4b76-bcde-51ffe3a168f6&v=default&b=&from_search=6http://www.slideshare.net/faizan-anjum/project-of-sampling-types?qid=d8d874cd-dae6-4b76-bcde-51ffe3a168f6&v=default&b=&from_search=17http://www.slideshare.net/nirbhay4shalu/sampling-14703106

Tool mentioned during APF sessionhttp://www.sensemaker-suite.com/smsite/index.gsp