An Introduction to Research Methods: Needs Assessment, Surveys, Focus Groups and Personas

An Introduction to Research Methods: Needs Assessment, Surveys, Focus Groups and Personas

Prepared for the ULS Leadership Program by Luke Ferdinand, John Fudrow, Karen Calhoun and Jeff Wisniewski

20 November 2013

This work is licensed under aCreative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License.

http://creativecommons.org/licenses/by-nc-nd/3.0/deed.en_US








Agenda Topic TimeArrival 10:25-10:35 amIntroduction to the workshop 10:35-10:45 amNeeds assessment - Luke 10:45 am-12:00 pmBox lunches 12:00 -12:30 pmSurveys – John 12:30-1:30 pmShort break 1:30-1:40 pmFocus groups 1:40 -2:40 pmShort break 2:40 –2:50 pmPersonas research – Jeff 2:50 – 3:50 pmWrap up and get on shuttle 3:50 – 4:00 pm

2

Needs AssessmentDefining and Addressing The Needs of an Organization and Its Audience

Luke FerdinandULS Leadership Program20 November 2013

ULS Research Methods Workshop

4

Objectives Define Needs in Organization Define Needs Assessment Understand the methods and tools used in Needs

Assessment Know potential risks in Needs Assessment methods


5

What is a Needs Assessment?"A systematic process for collecting information and making justifiable decisions" - Ryan Watkins

Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naintroduction.html,

http://ryanrwatkins.com/na/naintroduction.html




6

What is a need? “Needs are the differences between your current

achievements and your desired accomplishments.”

Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naintroduction.html,





7

Needs Assessments Help Identify Data that define your needs Prioritization of your needs Criteria for implementing solutions Information necessary to justify selection of one or

more activities to improve performance

Ryan Watkins, 2013, “Needs = Gaps in Results”, http://ryanrwatkins.com/na/naessentials.html


http://ryanrwatkins.com/na/naessentials.html


8

Why? “Provide a systematic process to guide decision-making in

organizations Provide justification for decisions before they are made. Scalable for any size project, time-frame, or budget. Offer a replicable model that can be applied by novices or

experts. Provide a systemic perspective for decision-makers. Allow for interdisciplinary solutions for complex problems.

“Ryan Watkins, 2013, Needs = Gaps in Results”, http://ryanrwatkins.com/na/naessentials.html,





9

Where can we use a needs assessment?Strategic Action 9a : Ground Floor Needs Assessment (obviously) Opportunity to rethink the ground floor Many assumptions and ideas for what's best Gather information from stakeholders, relevant

resources Make an informed decision to best meet the needs of

organization and community Other opportunities?

http://bts.library.pitt.edu/ULSPlanning/PBC/Lists/pbctasklist/Task/displayifs.aspx?List=008c5b68-d9c2-4ed0-a906-53f0856eb934&ID=130&ContentTypeId=0x010800E23A6C459C71C54BAD9D63F3CC8CB184


10

12 StepsBrought to you by the National Oceanic & Atmospheric Administrationhttp://csc.noaa.gov/needs/

http://csc.noaa.gov/needs/


11

Needs Assessment Steps1. Confirm the Issues &

Audience2. Establish the Planning Team3. Establish the Goals and

Objectives4. Characterize Your Audience5. Conduct Information &

Literature Search6. Select Your Data Collection

methods

7. Determine Your Sampling Scheme

8. Design and Pilot the Collection Instrument

9. Gather & Report Data10. Analyze Data11. Manage Data12. Synthesize Data & Create

Report


12

1: Confirm Issues & Audience• Summary:

• Establish purpose of assessment and summarize• Questions:

• Is this a new issue or audience for the organization?• Is there agreement up the organizational chain this issue or

audience needs to be addressed?• Risks:

• Unknown stakeholders• Communication issues• LACK OF SUPPORT


13

2. Establish the Planning Team• All steps should be conducted with planning team• Ideally members of stakeholder groups and experts• Communication is essential• Consider resource needs: time, expertise etc.• Questions:

• Are the stakeholders new or well known?• How will geography impact planning?• Expertise within the team?

• Risks• Sense of ownership• Communication• Planning can overtake action


14

3. Establish Goals and Objectives Summary

Identify the desired and actual levels of knowledge or skill Identify cause(s) for the lack of knowledge or skill Devise solution(s)

Questions: Are the goals widely shared by the audience? Are your objectives measurable? Will the project be considered a success if the objectives are met?

Risks: Potential to disengage by different team members who have different priorities Not knowing if goals have been achieved and to what degree Long term-support for doing needs assessments

Play Time


16

4. Characterize Your Audience Summary

Sample size Skill & knowledge level Educational Level Attitudes and biases

Questions: How long have you worked with the audience? How much variation is there within the audience?

Risks: Assumptions can backfire Broadly generalizing and audience can lead to failure


17

Topics for Audience Characterization Knowledge Training Tools & Techniques Benefits Attitudes & Biases Ability to Attend or Access Cultural Characteristics


18

5. Information & Literature Search Summary

Environmental Scan Readings Surveys to address questions raised in review

Questions: Has this audience or issue been surveyed in the past? What other ways might we find useful information about this audience or issue?

Risks: Results may be outdated Redundant effort, wasted time Missing important information that may make results easier or better


19

6. Select Your Data Collection methods Summary

Observation Personal Interviews Surveys Focus Groups

Questions: Have all methods been considered? Consider audience characteristics when drafting collection methods How much expertise is there in-house? Seek help!

Risks: Can be intrusive or upsetting Time spent designing vs time spent analyzing


20

7. Determine Your Sampling Scheme Summary

Sample more than you think you need Questions:

What is statistically recommended? What is the population size of audience?

Risks: Likely not scientifically sound Too many or too few in sample


21

Exercise: Develop Your GoalsDiscuss how you would approach

Audience CharacterizationData Collection MethodsSample SizeMeasure of Success


22

8. Design and Pilot Your Collection Instrument Summary

Pilot first to identify weaknesses Questions:

How will you pilot your instrument? What kind of expertise is on your planning team? How important is statistical precision? How will data collection be standardized?

Risks: Instrument will not be clear or gather necessary data The audience will not be receptive to the survey instrument Asking too many questions may irritate the respondents


23

9. Gather and Record Data Summary

Find means to incentivize respondents (survey prizes, etc) Ensure anonymity if this is a priority to audience Coordinate with assessment team If audience is outside of library, make use of External Communications group

Questions: Will you recruit people outside planning team to assist in data gathering? Are you getting the desired response rate?

Risks: Data will be biased Language or vocabulary issues Invalid Study


24

10. Analyze Data Summary

Keep findings and interpretation of findings separate in reporting Questions:

Seek assistance in interpreting data if necessary Risks:

Under or over-whelming audience with report Missing trends and patterns Not accounting for possible critical barriers Letting bias slip into the process


25

11. Manage Data Summary

This step involves determining how data will be organized and archived. The importance of this step is often not recognized until it is too late.

Questions: Will this data ever need to be referred to again? Will the raw data be stored? How will it be stored? Can/should the data be shared?

Risks: There may be unforeseen reasons that would necessitate the data being

used again Inadequate metadata


26

12. Synthesize Data and Create Report Always address your goals and objectives in synthesis. Report must

include problems or errors with the design and the implementation of the survey. An executive summary is often helpful.

Questions: Who is the audience for the report? Did you address your objectives?

Risks: Potential to disengage by different team members who have different

priorities Not knowing if goals have been achieved and to what degree Long term-support for doing needs assessments


27

Gather your Data/Report


28

Thank you!Works Cited:

http://www.cscnoaa.gov/needshttp://ryanrwatkins.com/na (includes link to free NA eBook)

Additional Reading:http://en.wikipedia.org/wiki/Needs_assessment

http://www.cscnoaa.gov/needs

http://ryanrwatkins.com/na

http://en.wikipedia.org/wiki/Needs_assessment

Survey Basics and Survey MonkeyJohn FudrowULS Leadership Program20 November 2013


30

When to Use a Survey Explore a Topic Discussion with Target Population Gather Objective Data on Subject Benchmarking of Service Levels


31

Survey Monkey Templates Over 150 pre-made templates Variety of topics

Community (2) Customer Feedback (7) Demographics (9) Education (20) Events (7) Healthcare (14) Human Resources (18) Industry Specific (58) Just for Fun (9) Market Research (12) Non-profit (5) Political (11)


32

Surveys We Have Applied ULS General Survey Library Instruction Survey E-book Survey Event Feedback Surveys Internal Planning


33

Types of Survey Questions Open-Ended Closed-Ended


34

Types of Survey Questions Open-Ended

i. Single Textbox ii. Multiple Textboxes iii. Comment/Essay Box iv. Numerical Textboxes v. Demographic (U.S. or International) vi. Date and/or Time


35

Types of Survey Questions Close-Ended

i. Multiple Choice (Only One Answer Allowed) ii. Multiple Choice (Multiple Answers Allowed) iii. Rating iv. Ranking v. Matrix of Choices (Only One Answer Per Row) vi. Matrix of Choices (Multiple Answers Per Row) vii. Matrix of Drop-down menus


36

Likert Scales and Ratings What is a Likert Scale?

A scaled response of a respondents feelings toward a topic based on a presented scale.


37

Likert Scales and Ratings Clear wording of response choices. Number of choices to relevancy

5 to 7 choices allow for a consistent distribution. The third choice should be the undecided or neutral

decision. Be careful not to force ranking by the wording of the

base question.


38

Anonymity• IRB Authorization

Service Improvement Anonymous Ratings

• Do not store data publically


39

Survey Delivery Target your audience Multiple avenues for large samples Create Multiple links to analyze effectiveness of

delivery methods DON’T SPAM You cannot mass email


40

Statistical Relevance and You We are not statisticians

We don’t have the resources nor time to run full analysis Our reports are not statistically relevant Our analysis is focused on report fulfillment

We don't properly sample Our audience is often “expert” library users

This alters the influence of their input We don't run via SPSS

Statistical Package for the Social Sciences


41

Using Survey Monkey (Free Version) https://www.surveymonkey.com/ Create a free account

https://www.surveymonkey.com/


42

Survey Monkey (Free Version) Limitations

10 Questions 100 Responses No Page Logic 20 Less Templates No Customization


43

Survey Question Creation Create 2 questions you would like to ask of your

department or patrons Think about how you would use the results and write

that idea in the page description. We will look at several and discuss their efficacy.

Focus Groups: the Method and How to Use ItKaren CalhounULS Leadership Program20 November 2013


45

Learning objectives1. Know the advantages and disadvantages of focus

groups and when to use them 2. Be able to select and recruit focus group interviewees3. Know how to prepare a focus group interview guide4. Have information about:

The roles of the facilitator and recorder How to analyze focus group interview data


46

Focus Groups, Defined Qualitative social science research method Feature open, interactive, in-depth small group

discussion (typically 6 to 10 people), led by a trained moderator/facilitator

Participants carefully selected Evaluate participant thoughts, opinions, practices,

values, beliefs, feelings in a nonthreatening, semi-structured setting


47

Purpose of a focus group IT IS TO:

Collect information and ideas on a pre-selected topic IT IS NOT:

To achieve consensus To answer participants’ questions (facilitator ≠ sage on stage) To solve one or more problems To make decisions To provide a forum for participants to gripe or vent frustration

(group therapy)


48

Advantages of Focus Groups Generate insight into not

just what people think or feel, but why

Can be comparatively low cost method research method

Interactive, not one way Can be used alone or in

combination with other research methodsCC BY NC Francois Proulx

http://www.flickr.com/photos/91569742@N00/404909051/


49

Disadvantages/Risks of Focus Groups Findings not representative of entire population of

interest Quality of results highly dependent on:

Appropriate preparation Skill of moderator Skill of recorder Skill of analyst (preparation of report)

Can be challenging to evaluate responses to open-ended questions


50

Steps to Set Up Focus Group Interviews Decide how many focus group interviews to hold (usually from two to a

dozen) Select appropriate facility (see next slide) and time(s) for the event(s) Decide on participant incentives Recruit participants (usually 6 to 10 people in each focus

group) Prepare interview guide/script Assign moderator and recorder Conduct the focus group(s) Analyze and report results


51

Focus Group Facility – Minimum Requirements

Comfortable room in a convenient location for participants Date and time convenient for participants (faculty or peers =

during work hours?; undergraduates = evening?) Table and chairs set up for group conversation Food, refreshments Permission to record the session Suitable means for moderator to conduct the interview (e.g., flip

charts, markers, laptop, projector, screen …) The means to record the session (notes + audio recording is one

way; another is streaming video to a separate room where notetakers/observers sit)


52

Focus Group Facilities – High End

Text: Summary of information provided by a commercial focus group facility – a local one is Direct Feedback Inc. (Pittsburgh area):http://www.dfresearch.com/focus.htm

Facility and technical capabilities:•Video streaming•Wireless•Comfortable, well equipped respondent room•On-staff moderating•Assistance with recruiting, audio and video recording …•Client viewing room with one-way mirror and closed-circuit television

http://www.dfresearch.com/focus.htm

http://www.dfresearch.com/focus.htm


53

Recruiting Participants – Selection Choose participants who will be comfortable with one another (but ideally,

do not know each other) – “homogeneous strangers” Potential selection criteria:

Knowledge of topic Comfortable discussing the topic with others Demographics - gender, age, etc. - if relevant to the topic Status , if relevant to the topic (role/target audience, e.g. faculty, student … or

user of service, non-user of service … ) If group members are known to each other, avoid having all members of a

clique in the same focus group if you can Also in groups known to each other, try to avoid having an employee and

his/her boss in the same group


54

Recruiting Participants – Recruitment Process

Participants might be: Nominated Randomly selected Members of a definable group Have same job/title Volunteers who fit selection criteria Other

Screen potential participants/assign to groups

Once you have grouped recruits, confirm participation (see sample letter, next slide)

Send reminder 2 days before Over-invite by 10 to 20% (to account for

no-shows)

Sample flyer recruiting volunteers

Extract from Eliot and Associates 2005


55

Sample confirmation letter Track invitation emails and responses

(along with demographic info you may need later)

Gain consent to record session (“Although the session will be recorded, your responses will remain anonymous and no names will be mentioned in the report.”)

Ideas for incentives (besides refreshments): Monetary Coupons/gift cards Door prize (drawing at end of session) …? Extract from Eliot and Associates 2005


56

Exercise #1 Situation

You’ve been given the following assignment: Conduct focus groups of the

Pitt community to examine attitudes toward, awareness and usage of library instruction services (inside and outside the library, onsite and virtual)

Exercise Working as a group at your table,

take 10 minutes to sketch out:1. How will you determine who should

be in the focus groups (i.e. the selection criteria)?

2. Once you have selected the criteria, what methods might you use to recruit participants and assign them to groups?

Report out (1-2 minutes each table)


57

Preparing and Using an Interview Guide(Script) Prepare pre-group paperwork – consent forms, brief

demographic information1. Opening – engagement questions

Introductions, ground rules (see sample, next slide), opening question (make it an easy one; could be round robin)

2. Exploration questions (2 to 4 key questions)3. Exit/ending questions (e.g., “Of all the things we

discussed, what is the most important to you?” and/or “Have we missed anything in our discussion today”?)


58

Sample Introduction and Ground Rules (Interview Guide)

Extract from Eliot and Associates 2005


59

Interview Guides Use Open-Ended Questions Open-ended questions = inquiries that produce rich

data = they start conversations and keep them going “When you think about green energy, what comes to

mind?” “What do you like most about coming to the library?”

Closed-ended questions = impose answers (yes or no, or a choice from a list) Help clarify and confirm Typically used in quantitative research like surveys


60

Other Possibilities for Designing Focus Group Interview Guides They don’t always have to be questions …

Can show a brief video/make a proposal then start asking questions

Can introduce alternatives and ask them to choose Fill in the blank Draw a picture More


61

Exercise #2Situation

You’ve been given the following assignment: Conduct focus groups to

evaluate user reaction to the high-tech group study rooms on the second floor of Hillman

Exercise Working at your table, and using

the handout from Richard Krueger: Take 5 minutes to select or create 3

open-ended questions for the focus group interview guide

Take another 5 minutes to consider how you will open and close the focus group interview

Each group report out (1-2 minutes)


62

Help for Facilitators and Notetakers

Preparation for October 11 “Birds of a Feather” EventKaren CalhounOctober 7, 2013

Available from Behind the Scenes, FY14 Planning and Budget Cte. Site

This PPT covers the roles of facilitators and recorders;See also “further reading” at the end


63

You, Moderating the Next Focus Group


64

Analyzing Focus Group Interview Data Is often not easy Devise a method of capturing the comments from the

notes/recordings Capture the comments Look for common categories or themes Assign categories to comments Sort comments by category or theme Synthesize Summarize findings for each category/theme Prepare and present report One method described @ Eliot and Associates 2005 (see last slide)


65

Recommended Reading for Focus Group Designers, Facilitators, Recorders and Analysts Connaway, Lynn Silipigni, Debra Wilcox Johnson, and Susan E. Searing. 1997. “Online

Catalogs from the Users’ Perspective: The Use of Focus Group Interviews.” College & Research Libraries 58 (5): 403–420.

Dixon, Jill. 2005. “Focus Group Facilitation Guidelines”. Centre for Higher Education Quality. Monash University. http://www.uwsuper.edu/cipt/exsite/upload/Focus_Group_Guidelines.pdf

Eliot and Associates. 2005. “Guidelines for Conducting a Focus Group.” http://assessment.aas.duke.edu/documents/How_to_Conduct_a_Focus_Group.pdf

FocusGroupTips.com. 2012. “Focus Group Questionnaire Fundamentals: Basic Questions.” http://www.focusgrouptips.com/focus-group-questionnaire.html

Krueger, Richard A. 2002. “Designing and Conducting Focus Group Interviews.” http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf

Walden, Graham R. 2006. “Focus Group Interviewing in the Library Literature: A Selective Annotated Bibliography 1996-2005.” Reference Services Review 34 (2) (April 1): 222–241. doi:10.1108/00907320610669461.

http://www.uwsuper.edu/cipt/exsite/upload/Focus_Group_Guidelines.pdf

http://assessment.aas.duke.edu/documents/How_to_Conduct_a_Focus_Group.pdf

http://www.focusgrouptips.com/focus-group-questionnaire.html

http://www.eiu.edu/~ihec/Krueger-FocusGroupInterviews.pdf

PersonasJeff WisniewskiULS Leadership Program20 November 2013


67

Road Map What’s a persona? Benefits of personas How to create personas How to use personas


68

Personas Personas are “stand ins” or hypothetical archetypes

created to represent the primary user segments for your web site

Each persona represents a key user type that shares demographic characteristics, needs, behaviours, and environment


69

Personas Imaginary, derived from user research Each has a name and personal details


70

Amy (Soo-Jin)

second year graduate student in Biological Engineering

currently splits her time between class work, time in the lab, and studying from home

since much of her work is either course driven or in the lab, she does not consider herself a particularly heavy library user

tends to rely on lectures rather than library resources

uses company websites quite frequently for information on the lab products she uses and uses professional association sites for recent papers and information on developments in her field

uses the popular search engines initially to get a sense of what types of materials are out there then moves to Web of Science and Compendex for access to journal articles


71

if she can avoid going to the library, she will

at home she always connects through remote access to get access to full articles through the databases for which the library has subscription

interested in doing exhaustive searches for journal articles on her dissertation topic

no one has shown her how to use the full breadth of the resources and functionality of e-Journal on the library web site; she has a sense there are more resources and tools than she knows about

uses ILL often to gain access to articles that she cannot access through Pitt subscriptions


72

Customer Segments to Personas Originated in the 1930 when marketers started using

fictional characters to represent a customer segment Alan Cooper, a software developer, coined a related

term and similar practice: personas. His book The Inmates are Running the Asylum

popularized the use of personas and designing for “archetypal users”.


73

Benefits of Personas


74

#1 “User” Centered Design Way to have users attend all your design meetings Each persona has the weight Personas are based on and embody what we know

about our library’s web site users

Personas keeping it about the user


75


76

#2. Support Evidence Based Decision Making Way to harness the user research data to inform web

site development Easier to remember a persona than pages of facts and

figures: path data, survey results, interview summaries etc.

Share abstract data in a compelling and memorable way Personas

encapsulate evidence


77

#3 Where to Put Design Effort Personas spell out what the site must do to support each personas’ goals and tasks

Personas provide focus


78

#4. Communicate to Stakeholders in a Language Understood by Everyone Easy and fun way to communicate design decisions Keeps the focus on the user Avoid “geek” speak

Personas speak to everyone


79

#4 Build Consensus and Commitment to the Design Communicate a common direction Reduce the need for extremely detailed specifications.

Nuances of behaviours and preferences are captured in the persona and narratives

Personas build shared vision


80

How to Create PersonasResearch!

Environmental scan Interviews Ethnographic research


81

Environmental Scanning Identify true peers Literature review Web search Provides a framework


82

Ethnographic Research Gorillas in the Mist Time consuming Expensive Highly useful!


83

Interviews Useful for creating from scratch and for local validation

of “borrowed”


84

Finding Subjects Leverage your networks General advertising not useful


85

Logistics Voice recorder of smartphone app Transcription Analysis


86

Interview Intro Explain hat you’re doing and why

You’re helping us build a better website Be candid

General computer usage habits When you start your browser where is the first place you

go? Favorite sites, and why?


87

If you need to: Find books to take on vacation Write a paper…Where’s the first place you’d go?

Information-Seeking Habits


88

How often do you go to the library? How often do you go to the library website? When do you go to the library website? What do you

do when you’re there? (Take them to site) What immediately draws your attention? What information did you look for but not find? Is there something you looked for on the homepage

but didn’t find?

Library Questions


89

Now What?

Create

Refine

Interview


90

How Many Personas ? Primary constituencies 5-7 generally recommended


91

How To Use the Personas In the room Frame discussions


92

ExerciseIn groups, create a persona based on one of the group members

Scenario: the personas created will be used to guide a user centered redesign of the ULS’ SharePoint site


93

Questions?


94

Thanks for coming!

Please fill out the post-workshop evaluation survey at:https://www.surveymonkey.com/s/FS8PFXK

An Introduction to Research Methods: Needs Assessment, Surveys, Focus Groups and Personas

Documents

Transcript of An Introduction to Research Methods: Needs Assessment, Surveys, Focus Groups and Personas