Adding Focus Groups to Your Assessment Plan

Nathan Lindsay November 12, 2013

The Assessment Cycle1) Develop or revise divisional,

departmental, or program goals and outcomes

4) Identify, develop, and administer measure(s)

5) Review assessment

results

6) Use results to inform decisions

and practices; provide evidence

for student learning

3) Determine assessment method(s)

2) Provide learning experiences (programs, services, activities, classes, etc.)

What type of data do you need?

Quantitative• Focus on numbers/numeric values

• Easier to report and analyze

• Can generalize to greater population with larger samples

• Less influenced by social desirability

• Can be less of a time commitment, and less expensive

Qualitative• Focus on text/narrative from

respondents

• More depth/robustness

• Ability to capture “elusive” evidence of student learning and development

• Specific sample

Selecting focus groups as a method

Pros•Understand perceptions, beliefs, or opinions of participants

•Direct and indirect method

•More data in a shorter period of time

•Members build off of each other’s ideas

•Increased understanding of specific topic

•Less expensive

•Larger number of participants

Cons•Less useful when statistical data/reports are needed

•Group dynamics/mix can be difficult to manage

•Not generalizable

•Time needed for training and analysis

•Facilitation requires skill

•Lack of control over discussion

(Stage, 1992; Stage & Manning, 2003)

Steps to using focus groups

1. Determine what type of focus groups will work best.

2. Determine how many focus groups you will conduct and select appropriate times and locations. 

3. Write your script/questions for the focus group sessions.

4. Select the students who will be invited to participate.

5. Conduct the focus group.

6. Debrief with other facilitators.

7. Compile the data.

8. Write the report.

The Assessment Cycle1) Develop or revise divisional,

departmental, or program goals, objectives, and outcomes

4) Identify, develop, and administer measure(s)

5) Review assessment

results

6) Use results to inform decisions

and practices; provide evidence

for student learning

3) Determine assessment method(s)

2) Provide learning experiences (programs, services, activities, classes, etc.)

Resources Needed

Staff: One facilitator, one note taker (minimum)

Facilities: Eye contact, privacy, no interruptions

Materials:• Audio or video recorder (recommended)• Note-taking materials (e.g., paper/pen, laptop)• Nametags• Consent form• Incentives

Time: 60-90 minutes for each session

Participants: 6 – 12 per group

Rounds: 2 – 5 (until redundancy is reached)

(Krueger, 1998b; Stage, 1992; Schuh, Upcraft, & Associates, 2001; Vaughn, Schuman, & Sinagub, 1996)

Types of questions

• Experience and behavior questions“How does your organization utilize the Student Center?”

• Opinion and values questions“Why do you think the University should consider including a multipurpose space in the Student Center expansion? Why shouldn’t we?”

• Feeling questions (Intro about inclusivity) “How do you feel when in the Student Center?”

• Knowledge questions“What Student Center resources are you aware of?”

• Sensory/environmental questions“What do you see when you enter our Student Center?”

• Background and demographic questions”Please tell us your name, class year, major, and what student organizations you are associated with.”

Developing questions

• Be clear.• Avoid biased, loaded or leading questions.• Avoid making significant memory demands.• Ask only one question at a time. • Keep the question short.• Don’t make assumptions.• Define terms and concepts.• Consider whether they will feel comfortable answering

honestly.• Let participants admit they don’t know or can’t remember.• No closed-ended.

Sequence of questions

Trust focus1. Ice breakers 2. “Safe” questions 3. Opinions,

interpretations, feelings4. Knowledge questions

Logic focus1. Introductory2. Transition questions3. Key questions4. Summary questions

(Kruger, 1998)

Example ProtocolIntroductory questions:• Please tell us your name, your class year, major, and what student

organizations you are associated with.• How does your organization utilize the Student Center? What are some of the

reasons that students, faculty, and staff utilize the Student Center?Transition questions:• What are some events for which you would like to utilize the Student Center

but don’t? Why don’t you utilize the Student Center?Key questions:• Why do you think the University should consider including a multipurpose

space in the Student Center expansion?• Why shouldn’t the University consider including a multipurpose space in the

Student Center expansion?• What would your organization use the multipurpose space for?• After providing operational definition of the multipurpose space: What is

missing from the definition? Are there other ways that your organization would utilize the space?

Summary questions:• Do you have any final comments?

Selecting Participants

1. Purposeful sampling:• Demographics (representative of population, or targeted

population)• Experience and knowledge (ability to provide rich data)

2. Convenience (who will show up?)

Want to have commonalities among participants but be aware of existing relationships

(Vaughn, Schuman, & Sinagub, 1996; Rea & Parker, 1997; Stage, 1992)

Facilitating focus groups

• Consider who should be the facilitator• Proximity to topic• Relationship to participants

• Establish ground rules• Goal is to generate discussion among participants, not to

come to a consensus or to answer to facilitator• Necessary to establish rapport with subjects

(Stage, 1992)

Focus Groups: Establishing Rapport

• Speak clearly• Show interest• Control negative body language• Remember your role• Stay on track• Seek clarification

(Assessing Student Learning and Development, Bresciani et al., 2004, p. 55)

The “Dynamic” Element

synergistic group effect:

Interactions among and between group members stimulate discussions in which one group member reacts to

comments made by another

(Berg, 1998, p. 101; Upcraft and Schuh, 1996; Stage & Manning, 2003)

Ensuring rigor

Trustworthiness• Credibility: data, interpretations, and conclusions are credible to the

participants and others• Transferability: the study might be useful in another setting• Dependability: changes over time are accounted for• Confirmability: the data and conclusions can be confirmed by

someone other than the researcher

Strategies • Triangulation• Peer debriefing• Inter-rater reliability • Respondent debriefing• Verification

(Lincoln & Guba, 1985)

The Assessment Cycle1) Develop or revise divisional,

departmental, or program goals, objectives, and outcomes

4) Identify, develop, and administer measure(s)

5) Review assessment

results

6) Use results to inform decisions

and practices; provide evidence

for student learning

3) Determine assessment method(s)

2) Provide learning experiences (programs, services, activities, classes, etc.)

Analysis

After data collection is complete:

1. Frame the discussion/context

2. Review the data (tapes, written documents, notes)

3. Take one: Identify categories or themes

4. Discuss themes that have been identified

5. Take two: Categorize data into themes; Refine themes

6. Measure the frequency of theme iteration

Debrief

• What themes or issues were discussed?• How did these differ from what we expected?• How did these differ from what occurred in earlier focus

groups?• What points need to be included in the report?• What quotes should be remembered and possibly

included in the report?• Should we do anything differently for the next focus

group?

(Kruegger, 1998)

Reporting

A summary report lists the key themes under each topic, along with a verbatim quote or two that illustrates the theme and some indicators of frequency, extensiveness, and intensity.

• Frequency -- how often was the theme (or comment) heard?

• Extensiveness -- how many different students expressed the same theme?

• Intensity -- how strongly was the opinion expressed?

Remember to acknowledge limitations and possible bias!

Focus Groups at the University of North Carolina Wilmington (UNCW): 4

Examples

1)Association for Campus Entertainment (ACE) Executive Board

2) Student Veterans Organization

3)Follow-up to Housing Benchmarking Survey

4) New Staff Members’ Perspectives on Diversity

Example #1: Campus Entertainment Executive Board• Focus groups conducted in fall 2008 and spring 2009

• Likert scale questionnaire of 13 items completed at the beginning of the focus groups. Examples included:• I have learned to work better as a part of a group/team.• I am able to hold others accountable for their actions.

• Focus group questions mirrored the 13 outcomes:• Have you learned to work better as a part of a

group/team? Why/why not?• Are you able to hold others accountable for their

actions? Why/why not?

Campus Entertainment Executive Board

• Student responses recorded by note taking and a digital recorder

• Graduate student took the lead in data analysis

• Very useful to compare related quantitative and qualitative data

• The students gave valuable insights into several areas, including:• The ability to develop leadership skills and work as a team

• Accepting responsibility with completing and executing tasks

• Effectively communicating thoughts and ideas confidently in any situation

Example #2: Student Veterans Organization

• There were 400 active military or veteran students at UNCW

• A focus group was conducted with the Student Veterans Organization, which built upon a prior survey of veteran students.

• We learned that you need to balance convenience with location (i.e., the lounge area can be noisy!)

• It was incredibly helpful to have direct quotes from the veterans for report and grant writing

• The veterans’ feedback included the following:• Please realize that we are our own group—we are not just non-

trads or adult students• “Veterans talk to veterans”—marketing and information sharing

is more effective coming from within the group

Applying the Student Veterans Focus Group Feedback

• More proactive in educating faculty and staff on student veterans’ issues and resources.

• Increased academic and personal support resources for student veterans.

• Special session at Orientation created for veterans.

• Make Veterans Day a more significant event on campus

• Create more mentoring programs, social gatherings, and leadership opportunities available to the Student Veterans Organization.

Example #3: Follow-up to a Housing Benchmarking Survey

• Benchmarking survey in fall 2007 produced many positive results, but also some items of concern for Housing administrators and the Vice Chancellor for Student Affairs

• Focus groups conducted in spring 2008 with a broad range of students in Housing and Residence Life

• A document analysis of other institutions’ practices (especially regarding safety) was also conducted to see how we compared with our peers

Housing and Residence Life Focus Group Themes

Housekeeping: • Students repeatedly identified issues with the work habits of housekeeping

staff and a general neglect of duties (excessive time on breaks, sleeping or watching television in hall lounges, and lack of effort in cleaning bathroom areas) being the most reported issue.

• Students identified weekends as a better use of time/effort for cleaning staff.

 Maintenance/Work Order Reporting: • Students identified timeliness of repairs and a lack of knowledge about

timetables for follow-up visits as major concerns. • Students requested a mechanism for identifying that a maintenance item was

addressed, was in the process of being addressed, or was being forwarded to physical plant for a more specialized level of response.

Safety/Security Issues: • Students were complimentary of residential security initiatives including door

access, front desk operations, and staff responses (including the presence of University Police officers in the halls).

• A troubling percentage of students responded that they do not lock the door on a normal basis when leaving their room.

Example #4: New Staff Members’ Perspectives on Diversity

• Purpose: To understand how aspects of diversity had played a role in the hiring process (including in the ad, phone interview, campus visit, and experiences since being hired)

• Participants included 11 staff members in student affairs who had been hired in the last two years

• Focus Group findings:• Ensure that we are conveying adequate and correct

information about our student and staff diversity• Questions regarding diversity in the interview process

need to be more consistent across candidates

The Benefits of Focus Groups in Mixed Methods

Assessments• Drill Down

• Quantitative =What are the issues/numbers?• Qualitative = Why are they issues?

• Greater Validity of Reports (You’ve gone the extra mile)

• High Touch Opportunities

• Students feel that their voice is heard, and you’re able to access students’ “meaning making”

• Schuh (2009) contends that mixed methods assessments will become more widely used

Generating Interest, Participation, and Ownership in Focus Groups

• Many of the same principles outlined for quantitative assessment apply

• For sampling, a purposeful sample is often the best strategy, but captive audiences can also be used

• Incentives (t-shirts, food, drinks) are helpful in recruiting participants

• Provide opportunities to discuss positive/negative feedback

To Foster Greater Ownership: Include the Students

When developing and administering a focus group assessment, reviewing the data, and discussing potential changes/action steps, think about:

• Resident assistants• Peer mentors• Orientation leaders• Student club/organization leaders• Tutors/Supplemental Instruction leaders• Student employees

References

Berg, B. (1998). Qualitative research methods for the social sciences. Needham Heights: Allyn & Bacon.

Bresciani, M.J., Zelna, C.L., & Anderson, J.A. (2004). Assessing student learning and development: A handbook for practitioners. Washington, D.C.: National Association of Student Personnel Administrators.

Kruegger, R.A. (1998a). Analyzing and reporting focus group results. Thousand Oaks, CA: Sage Publications.

Kruegger, R.A. (1998b).Moderating focus groups. Thousand Oaks, CA: Sage Publications.

Lincoln, Y., & Guba, E. (1985). Naturalistic inquiry. New York: Sage.

Marshall, C. & Rossman, G. (1999). Designing qualitative research (3rd edition). Thousand Oaks, CA: Sage Publications.

Patton, M. (1990). Qualitative evaluation and research methods. Thousand Oaks, CA: Sage Publications.

References (cont.)

Schuh, J. (2009). Assessment methods for student affairs. San Francisco: Jossey Bass.

Schuh, J.H., Upcraft, M.L., & Associates (2001). Assessment practice in student affairs. San Francisco: Jossey-Bass.

Stage, F.K. and associates. (1992). Diverse methods for research and assessment of college students. Alexandria, VA: ACPA.

Stage, F. K. & Manning, K. (2003). Research in the college context. New York: Brunner-Routledge.

Subramony, D. P., Lindsay, N., Middlebrook, R.H., & Fosse, C. (2002). Using focus group interviews. Performance Improvement, 41(8): 38-45. Silver Spring, MD: International Society for Performance Improvement.

Upcraft, M. L. & Schuh, J. H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass Publishers.

Vaughn, S., Schuman, J. & Sinagub, J. (1996). Focus group interviews in education and psychology. Thousand Oaks, CA: Sage Publications.