APF orlando diy survey workshop 071114 final

of 46 /46
www.aperioinsights.com Revealing Insights for Clear Decisions July 11, 2014 APF Orlando

Embed Size (px)


DIY Survey workshop presented to APF (Association of Professional Futurists) pro-dev meeting July 2014. Covers basics of DIY quant survey tips.

Transcript of APF orlando diy survey workshop 071114 final

  • www.aperioinsights.com Revealing Insights for Clear Decisions July 11, 2014 APF Orlando
  • 1 Revealing Insights for Clear Decisions Welcome to DIY Survey Research
  • 2 Revealing Insights for Clear Decisions Mike Courtney Researcher | Ethnographer | Futurist Educated and trained in both traditional research and long range scenario planning. MMR degree in research, published author, angel investor. Data Analysis | Ethnography Ideation | End User Requirements New Product Validation | Roadmaps
  • 3 Revealing Insights for Clear Decisions Do YOU Research? Term Research is vague Research? Sure! I use Google almost every day. Many dont know if they need it. We dont really need research. What we need is sales and marketing help. Our ads arent getting the response we expected. We might need to look for a new ad agency. Competition just stole our top 3 customers. We need a better sales trainer or maybe a new sales contest. I already did the research. Both my mom AND dad loved the idea. My golfing buddies also think its awesome. Now I just need a couple million $ in funding so I can hire someone to do the marketing and sell it. Sales havent yet taken off, I think I just need better SEO.
  • 4 Revealing Insights for Clear Decisions Why Do Research? Three Simple Reasons To reduce risks To discover opportunities To improve business results One Metric for Success Answers that Lead to Actions After conducting thousands of marketing research studies and asking hundreds of thousands of questions, the author has come to understand one thing: There are no bad questions, only irrelevant ones. In other words, the majority of questions asked are irrelevant. That is, they don't result in answers that lead to actions. The Secret to Great Marketing Research: Ask the Right Questions by Robert J. Kaden
  • 5 Revealing Insights for Clear Decisions Research has Changed COSTS Even straightforward projects cost big $ Simple projects are inexpensive or free TIMING Took months, hard to predict completion date Often done in weeks if not days or hours TOOLS Tools were basic skilled experts needed Tools are powerful and geared towards non-experts OUTPUT Hard to change final output once done Easier to adjust or change final output BACK THEN NOW
  • 6 Revealing Insights for Clear Decisions Todays Research Tools are Easy and Powerful But could they be dangerous?
  • 7 Revealing Insights for Clear Decisions Safe Surveys What We Will Cover When to use Quant vs. Qual Research brief basics capturing the objectives upfront (Team Exercise) Turning objectives into questions subtle differences that add depth Common Scales and other answer options how to choose, examples, etc Common mistakes and traps rules of thumb (Featuring Dr. Bishop handout) Survey question practice - you can do it, we can help (Team Exercise) Show and tell share your questions and get reactions/critiques I need how many? The significance of sample size Finding Respondents - Survey Sample Options Survey and analysis tools - low cost options Final Q & A
  • 8 Revealing Insights for Clear Decisions Refresher Qualitative vs. Quantitative Qualitative (Qual): Discover and understand the why or the rationale behind actions Explore the range of possible perceptions and beliefs Understand the language used and how it impacts understanding Helps develop ideas and hypothesis to test Quantitative (Quant): Measure the percentage who think a certain way Project findings onto larger population Gather feedback from a larger audience Benchmark and periodically re-measure
  • 9 Revealing Insights for Clear Decisions Research Brief Keeps Everyone on Target Background Relevant background to understand context of the request including the scope of project & information currently available: Existing Research what data/knowledge do you already have? Primary, secondary, anecdotal Decision[s] to be made What are the key decisions that will be made? What are the primary reasons to consider conducting this research?: Specific objectives/Hypotheses Are there any theories about what will be learned? What information will affect the decision[s]? What do you need to prove/find out? Ideally, what will you know when the research is completed?: Additional decision data required If the research IS successful in gathering the required information, what other data might ALSO be needed before making the decision[s] in question?: Target respondents What criteria (demographic, product usage and other specific requirements) must the respondents meet in order to qualify for this study? Please note any hard quotas or splits: Justification $ implications / ROI / reasons why project is important: Deadline For final results:
  • 10 Revealing Insights for Clear Decisions Team Exercise: Use a Research Brief to Outline your Objectives Team Task Complete the research brief and outline the specific survey objectives that will form the outline of a survey. Pick a survey topic or use below. Example Survey Topic As an APF member you understand the value a good futurist can bring to an organization. Unfortunately there are still many people in the world who have no idea what you do and how it might help them. Imagine you are working on a project to identify potential NEW clients to approach in order to educate, work with, etc. Your team has been asked to create a survey to understand and measure WHO may be interested in working with a futurist.
  • 12 Revealing Insights for Clear Decisions 12 Watch out for too many or too few response options Capital punishment should be reintroduced for serious crimes 1 = Agree 2 = Disagree 1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral
  • 13 Revealing Insights for Clear Decisions 13 Example: How could this question be improved? How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over
  • 14 Revealing Insights for Clear Decisions 14 Are you satisfied with your marriage and your job? __________________________ Example: How could this question be improved?
  • 15 Revealing Insights for Clear Decisions 15 You didnt think the food was very good, did you? _____ Yes _____ No Example: How could this question be improved?
  • 16 Revealing Insights for Clear Decisions 16 Environmental issues have become increasingly important when choosing respectable hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No Example: How could this question be improved?
  • 17 Revealing Insights for Clear Decisions 17 What information sources did you use to locate your restaurant for todays meal? (please tick appropriate spaces) ____ yellow pages ____ Internet ____ word of mouth Example: How could this question be improved?
  • 18 Revealing Insights for Clear Decisions Lenovo Survey Example
  • 19 Revealing Insights for Clear Decisions Lenovo Survey Example (Cont)
  • 20 Revealing Insights for Clear Decisions Lenovo Survey Example (Cont) Remember pick only FIVE items to rate
  • 22 Revealing Insights for Clear Decisions Survey Templates Readily Available
  • 23 Revealing Insights for Clear Decisions Basic Questions Get You Started
  • 24 Revealing Insights for Clear Decisions Subtle Question Differences Add Depth (PS -dont use this many) Management and Fairness My manager treats all his/her employees fairly. My manager knows what is going on in my work group. Favoritism is not an issue in raises or promotions. My manager is always consistent when administering policies concerning employees. I am always treated fairly by my manager. Everybody is treated fairly in this organization. My manager establishes plans and work objectives with me. My manager gives me clear instructions. My manager is available when I need advice. My manager (immediate supervisor) trusts me. My manager helps me to improve myself. My manager takes prompt and fair corrective action on employees who fail to perform their work satisfactorily. I feel free to talk openly and honestly to my manager. My manager praises me when I do a good job. My manager holds regular meetings with my work groups. My manager is effective in making decisions. TIP: Build and fine tune questions over time; share question lists with other DIY researchers.
  • 25 Revealing Insights for Clear Decisions Survey Tips Whose opinions matter? Need to ask a sample of respondents who you think might be worth targeting in the future. Dont ask mom if you doubt she would be a future customer etc. Understand without Influence - Trying to get true perceptions not trying to influence, sway, convince etc. Bias comes in many forms question wording, concern for interviewer, those who know you, those who take pity on you and want to help, fatigue, low category involvement etc. Behavior then perception ask first about what they have done, they what they think about it KISS principal Simple wording is often best. Define Timelines - Have you how much do youhow often do you.ALL require some sort of timeline. Define categories & avoid mental math What % of your beverages are coca cola products? Does % mean number of drinks per day? Are they asking about % as in how many ounces of product or bottles? Cups? Fountain? Should I include bottled water sold by coca cola corp? etc. Chocolate Cake Factor - Most real world decisions involve many factors. Need to tell the rest of the story or enough to make sense. Dont forget to ask about the major factors that would influence behavior.
  • 26 Revealing Insights for Clear Decisions Basic tips for Choosing Scales Dont recreate the wheel Choose question stems carefully Reverse word some of the items Use the same scales each time data is collected Avoid mixing too many scales in one survey Format scales for one direction negative to positive etc. Balance Scales Agreement, quality and satisfaction scales use an equal number of options on either side (positive vs. negative) of a scale, each with similar wording. Scales vs. Checklists Sometimes its better to use a checklist instead of having everyone mark the same scale item multiple times. https://www.case.edu/president/cir/wordfiles/tips_choosingscales.docx
  • 27 Revealing Insights for Clear Decisions 27 I need how many? The significance of sample size
  • 28 Revealing Insights for Clear Decisions Example Survey Flow Screeners Questions that confirm the respondent meets some minimum criteria Category Involvement Questions that explore the level of activity in the category Stated Needs & Pain Points What do they like/not like about their current___? Concept Introduction Imagine there is a new __ available. Heres a description of it... 5 Key Questions Appeal, unique/different, satisfies a need, believability, value for money/Interested in learning more etc. Suggestions How would you improve or change it? Demographics Age, gender, income, occupation, education, etc. Bizographics Company size, industry sectors, locations, etc. Psychographics Personality, values, attitudes, interests, or lifestyles
  • 29 Revealing Insights for Clear Decisions Common Questionnaire Mistakes Dr. Bishops List Issue Definition Example Ambiguous terms Terms that can mean more than one thing. Do you believe in freedom for all? Anchoring The response is biased by the last information presented to the respondent, usually dealing with numerical information. The U.S. has over 39 million without health insurance. How many uninsured does Germany have? versus The U.S. has over 200 million with health insurance. How many insured does Germany have? Biased wording Terms that indicate the attitude of the questioner and influence the respondent to answer the same way. Do you agree that worthless people should be excluded from welfare payments? Complex questions Items that are too difficult to answer because the question is too complicated. For each of the following courses, indicate whether the course should be offered, when it should be offered, and by whom it should be offered. In the last column, show whether the course should be offered by adjuncts or full- time professors and whether the courses should include only undergraduates, graduates, or both. Double-barreled questions Asking two questions in one item. Do you think the President should cut defense spending and reduce the national debt? Incomplete list Response categories that do not contain all possible responses. Protestant, Baptist, Catholic, Jewish Jargon Terms that are unique to a particular field. All health education curricula should contain courses on RVRBS. Lack of a filter Questions that apply to only a subgroup without filtering. What was your major in college? Lack of knowledge The respondent does not have the information requested. When was the last time a President visited Japan? Mutually inclusive categories Response categories that overlap and contain the same material. Protestant, Baptist, Catholic, Jewish Negative questions Items that contain negations. Do you agree that the U.S. should not bomb the Serbs in Bosnia? Spurious exactitude Items that ask respondents to make distinctions that are too fine. Rank the following 20 items how important they are for the future.
  • 31 Revealing Insights for Clear Decisions Examples of objectives: (see handout) Screening Criteria: Understanding who should be surveyed/not surveyed. Will you target any particular industry, sector or organization type? Your list of emails to survey is a big hot mess, should anyone be screened out? Targeting/Sampling: Which job roles or titles are most likely to be relevant to this study? Who would normally deal with long term strategy and scenarios? Who would benefit and/or be a potential champion for the process? General Background - What background or previous experience will the respondents have? What about their background would be useful to understand? Awareness of Futurists and Foresight. Do these organizations worry about the future? How much? History of Involvement - Experience or history with doing this type of process? Do these respondents have any knowledge of, or experience working with futurists? Current State/Situation - Rate of change being experienced by org/industry now. Key issues they are dealing with that create disruption/risk. Does org experience much change? Where are the major sources of change coming from? Do different rates of change impact interest in working with futurists? Outsourcing attitude how often do they hire an outside group for help of any kind? How they view Futurists vs working with some other type of consultant/agency Do they often (ever?) use outside help/consultants? Time Horizons - What time horizons do these organizations typically deal with? What does short vs long term typically mean to their organization or industry? Understanding/Common Lexicon - Do they understand the same terms, acronyms and concepts you do? Are you sure you are speaking the same language? Internal Capabilities/Readiness - What parts of the process would they most like help with vs feel confident to do themselves? Is their organization ready to take on a project that focuses on the future? Attitudes/Perceptions - Does their mindset or attitude towards your process matter? What would success look like and how would they measure results of a project? Reaction to the Concept Once the basics of the process is explained, what are their reactions to it? Do they understand what is possible, services offered, value potential, etc.? Sources of learning/knowledge How do they keep up to date on new methods? Where do they look to learn about new topics or approaches to organizational issues? Alternatives - If they wanted to think about the future, where would they look for help? If they didnt hire a futurist to help think about the future, who WOULD they likely hire?
  • 32 Revealing Insights for Clear Decisions TEAMS DEVELOP AND PRESENT SURVEY QUESTIONS (15 MINUTES?)
  • 33 Revealing Insights for Clear Decisions DIY RESEARCH TOOLS
  • 34 Revealing Insights for Clear Decisions Rapid Rise of DIY Research Tools Tools for Easier Data Capture and Analysis Quant survey and qualitative interviewing software Analysis tools for the visual display of data Tools that offer new hybrid methodologies Features Anyone Can Use Survey tools with more analysis featuresbuilt in cross tabs Automate common tasksgraph output to PowerPoint and stat testing Templates, canned reports and standard graphics Tools marketed directly to non-research end users
  • 35 Revealing Insights for Clear Decisions Finding Respondents - Survey Sample Options Competition among traditional sample sources continues to reduce price points. Some offer DIY self-serve sample tools for easy access. Non-traditional sample sources such as Google and Mechanical Turk can appear cheap but are not always nationally representative, or the best value depending on the survey length. Respondent quality is often and unexpectedly uneven for both sources and data cleaning is critical to meaningful results. FULL FEATURE, FULL PRICE GreenBook.org List of sample/panel companies such as: Survey Sampling Cint ResearchNow Wide-open, nationally representative sample is currently running $3-6 per complete. Lower incidence studies can still easily top $10-14 per complete. SCRAPPY ALTERNATIVE Mechanical Turk Google Sample
  • 36 Revealing Insights for Clear Decisions Quantitative Survey Tools Many companies have, but dont fully or frequently use their expensive software licenses. If you really need the full featured versions consider partnering with or consolidating your survey software activity among several companies. FULL FEATURE, FULL PRICE Qualtrics Lots of features, popular with academic crowd ($5-10K+/year) ConfirmIt Industrial strength feature set, popular with larger research shops ($10K+/year) SawTooth Software The standard for classic conjoint and MaxDiff ($3-10K 1x fee) SCRAPPY ALTERNATIVE SurveyGizmo Free version (8 question types & 350 responses) or Basic version $19/month (28+ question types & unlimited responses) SawTooth Software Make friends (and a deal) with someone who owns a license
  • 37 Revealing Insights for Clear Decisions Quantitative Analysis Tools Many survey tools now have built-in analytics capabilities that can handle the basics. If you really need the full featured versions consider partnering with or consolidating your survey software activity among several companies. FULL FEATURE, FULL PRICE SPSS One of the original standard analytic tools. Lots of statistical features, poor graphics. Now IBM owned and 5-10x the old prices ($5-40K+) MarketSight Super easy to use online analytics tool. Can create PPT charts in batches from crosstabs ($99/year academic license, $1,600 standard license, free 30 day trial) SCRAPPY ALTERNATIVE R Project for Statistical Computing Popular open source stats package, tricky to install with basic graphics (FREE) Excel Easy for basic crosstabs and many analysis templates exist for FREE. Most of us already have an old copy laying around right? Or use Google docs
  • 38 Revealing Insights for Clear Decisions Visualization Tools Information overload means effective researchers need to have valuable data AND be capable of using it to tell a clear and compelling story. Visualization tools are useful for exploring data in the analytics phase as well as turning key findings into easily understood and eye catching output. FULL FEATURE, FULL PRICE Tableau Software Drag and drop visual analytics tool. Capable of handling various data types and creating great visual outputs with a little practice ($999/year hosted edition, $1,999 pro version) SCRAPPY ALTERNATIVE RAW Newly announced online visualization tool with 7 graphic styles and ability to export finished result (FREE) Tableau Public Lite version of drag and drop visual analytics tool, main caveat is your output must be publicly shared
  • 39 Revealing Insights for Clear Decisions Consider Methods Like MaxDiff Requires either SawTooth OR Survey Analytics survey software
  • 40 Revealing Insights for Clear Decisions When Should I Use All This? Trend is towards conducting smaller, more frequent studies that provide a constant stream of feedback instead of a few large studies Common Research Touch Points Include: Idea or concept stage, before any development has begun MVP or proto stage, testing a working version of the core offer Development milestones as features are added/perfected Friendly user stage full product but pre-full public release Post launch periodic testing to track how expectations are evolving as users move past novelty and into routine use phases Timing also dependent on pace of change and response times in the industry sector
  • 41 Revealing Insights for Clear Decisions Keeping Score Dont look at just one score Averages vs. Segmentation
  • 42 Revealing Insights for Clear Decisions Q & A
  • www.aperioinsights.com Revealing Insights for Clear Decisions For more information, contact: Mike Courtney m: 469-363-0109 e: [email protected]
  • 44 Revealing Insights for Clear Decisions Links! Survey basics and Tips https://explorable.com/course/the-survey-guide http://www.slideshare.net/jtneill/survey-design-workshop?qid=307afda6-63e0-45ce- a197-3fda3491513f&v=default&b=&from_search=2 http://blog.askyourtargetmarket.com/2011/04/survey-design-tips-bad-questions- made.html http://www.marketingprofs.com/short-articles/2109/three-errors-that-lead-to-bad- survey-questions http://www.focusgrouptips.com/marketing-research-surveys.html http://www.focusgrouptips.com/qualitative-research.html http://www.b2binternational.com/publications/questionnaire-design/ http://www.qualtrics.com/blog/writing-survey-questions/ http://www.researchplan.com/blog/?p=51 http://www.surveysystem.com/sscalc.htm http://www.slideshare.net/horatjitra/designing-questionnaire-and-survey-research
  • 45 Revealing Insights for Clear Decisions Links! Scales and Sample Frame basics Scales http://www.360degreefeedback.net/media/ResponseScales.pdf http://www.uwex.edu/ces/4h/evaluation/documents/Wordingforratingscales.pdf Sample frame PPTs http://www.qualtrics.com/blog/determining-sample-size/ http://www.slideshare.net/heenadhawan86/sampling-bigslides http://www.slideshare.net/hafizahhajimia/research-method-sampling?qid=d8d874cd- dae6-4b76-bcde-51ffe3a168f6&v=default&b=&from_search=6 http://www.slideshare.net/faizan-anjum/project-of-sampling-types?qid=d8d874cd- dae6-4b76-bcde-51ffe3a168f6&v=default&b=&from_search=17 http://www.slideshare.net/nirbhay4shalu/sampling-14703106 Tool mentioned during APF session http://www.sensemaker-suite.com/smsite/index.gsp