COM564 Research methods for professionals final case study€¦ · DAVID T. GREEN Washington State...

DAVID T. GREEN Washington State University Fall 2016

COM564 RESEARCH METHODS FOR PROFESSIONALS

FINAL CASE STUDY

Background………………………………………………………………………………………….1

Overview of the Focus Group……………………………………………………………1-2

Overview of the Survey…………………………………………………………………….2-3

Overview of the Experiment………………………………………………………………..3

Focus Group………………………………………………………………………………………..4

Research…………………………………………………………………………………….4

Set Up…………………………………………………………………………………………4

Moderator’s Guide………………………………………………………………….5-7

Reliability and Accuracy……………………………………………………………..8

Cover Letter………………………………………………………………………………………..9

Survey…………………………………………………………………………………………. 10-13

Pretest Results……………………………………………………………………..14-15

Reliability and Accuracy……………………………………………………………16

Experiment……………………………………………………………………………………….17

Design………………………………………………………………………………………17

Function…………………………………………………………………………………..18

Survey…………………………………………………………………………………18-20

Reliability and Accuracy……………………………………………………….…..21

Conclusion………………………………………………………………………………………..22

1 | P a g e

We have been hired by the College of Engineering and Architecture at Western State

University to design a comprehensive research program to help CEA public relations staff

members communicate more effectively with College alumni. Our goal is to be able to help

development staff members increase the effectiveness of CEA communication and fundraising

efforts.

Many CEA graduates end up employed in similar industries and a high percentage chose

to remain in the Pacific Northwest. As a result, they tend to live in or near the state’s urban

centers and a significant number end up working for large firms with engineering or related

functions as a core part of their business. As a result, there is potential to reach out to CEA

alumni who work in specific industries, who are employed by key corporations and/or who live

in specific geographic regions of the state.

Our plan is to use focus groups from three major urban areas in the state of Washington.

The cities picked for this study are Seattle, Spokane, and Vancouver. We feel that the need for

focus groups is vital in determining the accuracy of the previous observations made by the CEA.

We already have learned about the alumni’s attitudes and actions towards their charitable

contributions to the Western State College of Engineering and Architecture. However, it is

2 | P a g e

imperative that there are research gaps that need to be addressed, meaning there are still

questions to be answered to make sure that the study is accurate in dissecting the strengths and

weaknesses of the approach that the current public relations group is involved in.

As mentioned earlier, these focus groups will take place in Seattle, Spokane, and

Vancouver. We will use a random sample from the list of CEA alumni living in these urban

areas. This way we can learn more about such topics as demographics, previous contributions,

and even more importantly, we can get a better idea about how the alumni feel about the College

of Engineering and Architecture. We are also looking to see if/how alumni are currently involved

with the alumni association.

One of our major goals with this study is to determine which approaches are working

with alumni, which are not working, and if there are any additional approaches that they would

like included. To get this information, we will develop a quantitative survey presented to CEA

alumni from across the United States. We want a sample size of 750 CEA alumni with the

assumption that we will receive approximately 600 responses (See page 16 for calculations). We

will use a stratified random sampling of CEA alumni from across the country. We will not

include any alumni who have international addresses, due to time and expense.

3 | P a g e

The accompanying cover letter will be presented in a friendly tone and kept as short as possible,

while giving the prospective participant all of the information they will need regarding this

survey.

Our experiment will look at how likely an alumnus will give to the CEA if he/she is kept

up to date and given more information about the alumni association than those who are given

basic information. What we will look at here is if the graduates’ knowledge of the exciting things

going on with the CEA will facilitate them to give back to the school in the form of donations.

The experiments will take place in all three target cities of Seattle, Spokane, and Vancouver. We

will be using twelve participants. All participants will be shown a video, talking about the alumni

association in general, how the association functions, the association officers will be introduced,

and the group will be given a general idea of how donations can help. After this, the participants

will be given a 15 minute break and then split into two groups of six. The control group will

continue to learn general information about the association. The test group will be visited by

members of the board of the association who will delve into much more pertinent information,

such as the specifics about how the donations are being used and what benefits the alumnus can

receive from giving back to the department. They will also take questions from the participants.

At the end of their meetings, both groups will be asked about their inclination to donate. The idea

here is to find out if learning more about how the donations are being used and two-way

conversation with members of the alumni will make them more inclined to donate to the

department.

4 | P a g e

When we look at the information presented to us, we can focus on age being a major

factor in whether an alumnus donates, and how much they donate. “As they age, many alumni

recognize their WSU education as a critically important aspect of their personal development and

one of the most significant personal investments they made as young adults.” This can be a great

place to start when assembling our focus groups.

The Research:

Therefore, we will conduct six focus groups, two each in Seattle, Spokane, and

Vancouver. We will use the CEA alumni list to gather 6-12 former CEA students. We will split

the groups up by age in each city, with one group being those who graduated between 1995 and

2016, and one group from those who graduated before 1995. For these groups, we want to use

our demographic search wisely. We want an equal number of male and female participants. We

will conduct these focus groups in hotel meeting rooms in the three cities.

Setting up for the Focus Groups:

We want the participants of our focus groups to be welcomed at the sessions by having

snacks set up on a table, plus letting them mingle a while with other members of their alumni. At

each seat, we will have bottled water available. At each seat will be consent forms which are to

be read and signed by all participants before the focus group begins. Each focus group will be

filmed and shown to other members of the research staff, live via closed circuit television.

5 | P a g e

The Moderator’s Guide to the Focus Group:

Joe Smith (Moderator):

I want to welcome you to this focus group, brought to you by the College of Engineering

and Architecture. As you probably already know from reading the letter we sent you by mail, this

discussion will focus on the various methods of communication with the CEA and the alumni

association. We will be discussing the CEA’s website, magazine and fundraising efforts. Our

goal is to learn about how your experience has been so far with the association and to get some

feedback on what it is that you would like to see improved with the methods of communication

used to keep you in touch with the CEA. This is to be an open and friendly discussion and I want

to tell you up front that there are no right or wrong answers. Please speak freely and let us know

your honest opinions. If you feel the need to leave the room at any time, please let me know and

there will be no problem with this. Are there any questions that you may have before we begin?

Okay, let’s get started. I want to first ask you about your current interactions with the

CEA. Look up hear at the flip chart and when I talk about a method of communication that you

use, please raise your hand, highly please. Do you ever read about CEA from a magazine? The

CEA alumni monthly newsletter that is mailed to you? The CEA alumni website? Any CEA

social media page, i.e. Facebook, Twitter, etc.?

Next I want to ask you this: Do you receive any information from the association? By

this, I am asking, “Are you on any mailing lists?” Do you keep the association informed if you

change your address, or any other contact information?

Now, let’s take a look at the CEA’s social media sites. First, we will look at the website

(Showing live website up on the screen). As I scroll through the home page, are there any initial

6 | P a g e

thoughts that come to mind? This may be a good time to ask – Have you ever been to this site

before? If not, why haven’t you visited this site?

Now we will take a look at the CEA’s social media sites (shows live feeds from social

media sites). Which of these peeks your interest? Do you find any of these unappealing? Why?

After viewing the website, and social media sites, which one(s) would make you want to

find out more about?

What is it about either the website, social media, or the magazine do you enjoy? Are there

certain elements in any of these sites that you like more than others? What could be missing from

these sites that you feel could be of importance to a member of the CEA alumni?

Do you know who is on the board of the CEA alumni association? If not, would you like

to know who they are and learn about them?

Let’s move on to your current connection with the association. Studies have shown the

approximately 20 percent of the 20,000+ CEA alumni donate to the association. By a show of

hands, do you currently donate? Have you donated in the past? Remember, all of this is

confidential and will not be published. What is it that gets you to donate? Is there a particular

event or story that gets you excited about donating?

For those of you that do donate – let’s talk about how it is that you go about donating. Do

you send in a check? Do you donate online? Do you have donations automatically sent in each

month or year?

7 | P a g e

For those of you who have not donated yet – is there a reason you don’t donate? Is there

something that you can think of that the association could do to make it so you felt good about

supporting the CEA?

Finally, are you happy with the communication you receive from the CEA? How do you

think it could be improved?

I want to thank you all for joining us here tonight. The information and ideas that you

have provided are vital for the CEA.

8 | P a g e

Due to the fact that the questions were set up to elicit responses dealing with the

participant’s communications with the CEA, we can expect acceptable internal validity. As a

rule, reaching good, even acceptable, external validity using focus groups is a problem. And with

this focus group, we have to say that the external validity would be rated as low, at best. Do the

alumni who live in these three urban areas represent the larger population of the state that does

not live in urban areas? Do alumni that have moved away from the state have different ideas

about their communication with the association? One of the reasons we focused on age in this

study was to see if there were any differences in the way they looked at communication. Would

the older group prefer their information by U.S. Mail? Would the younger group ignore the same

style of communication? The best way to assess this kind of focus group is to take the ideas they

give and incorporate them into a broader survey. Validity aside, these groups give us some

valuable feedback and are useful for our study in this case.

9 | P a g e

College of Engineering and Architecture

Western State University

P.O.Box 714

Pullman, WA 99163

Alumna

123 State Ave.

Your Town, WA 98642

Dear Alumnus,

The College of Engineering and Architecture would appreciate it if you would take the time to

complete the enclosed survey. Your feedback is very important and is valued by the CEA in our

continuing efforts to improve the communication between the CEA and its alumni.

It is extremely important to us here at the CEA to bring you important and timely information

about the school. With the results we gather from this survey, we can keep you up to date on

events, innovations, changes that happen within the school, and much more. We want you to be

an active part of the CEA family, and that means we need your input to make this happen. Your

individual answers to this survey will be completely confidential.

Please do not hesitate to contact us if you have any questions about the survey, as we want you to

be a big part of the future of the CEA.

Thank you in advance for participating.

Sincerely,

David T. Green

Public Relations and Communications Manager

Western State University

10 | P a g e

Demographics

Please answer all questions by checking the box next to your response:

Q1 - My Age Is:

Under 22

22-25

26-35

36-45

46-55

56-65

Over 65

Q2 - I am a:

Male

Female

Q3 - I currently live in the following zip code ___________

Q4 - I currently receive the monthly CEA mailing:

Yes

No

Q5 - I prefer communication electronically (i.e. e-mail):

Yes

No

Q6 - I currently receive e-mail from the CEA:

Yes

No

11 | P a g e

Q7 - I donate to the CEA:

Yes (If yes, continue to question 8)

No (If no, continue to question 9)

Q8 – The total amount I donate to the CEA in a year is:

Less than $50

$50 - $100

$101 - $500

$501 - $1000

More than $5000 __________

Q9 – The reason why I haven’t donated to the CEA yet is:

Lack (or poor) communication with association

Don’t have the extra money

Donate to other organizations

I Don’t Donate

Q10 – When was the last time you visited the CEA website:

Today

Within the last week

Within the last month

Within the last year

I have never visited this website

Q11 – What would be the best way for you to receive information about the CEA:

U.S. Mail

E-Mail

Social Media

Magazine

I prefer not to receive any information about the CEA

Q12 – Are alumni relations important to you?

Yes, extremely important

Somewhat important

Slightly important

Not important at all

We have an alumni association?

12 | P a g e

Communication with the College of Engineering and Architecture

In this section, please circle the appropriate response to each question:

Q13 - The alumni association has great communication with its members.

Strongly agree 1 2 3 4 5 6 7 8 9 10 Strongly disagree

Q14 – I would like to know how the alumni’s donations are being used by the CEA


Q15 – I visit the CEA website often


Q15 – When I see a piece of mail from the CEA, I always open it


Q16 – When I see an email from the CEA, I always open it


Q17 – Staying in contact with the CEA is important to me


Q18 – I follow the college on their social media sites


Q19 – I am interested in non-paper communication


Q20 – I like to keep up on what the CEA is doing


13 | P a g e

Q21 – Being a member of the CEA alumni is important to me


Q22 – I would volunteer for CEA alumni sponsored events if approached to do so


Q23 – I read anything that pertains to the CEA


Q24 – I realize that donations are important to the CEA


Q25 – Being able to have two-way communication with the CEA would make me consider

donating to the association


14 | P a g e

I presented our cover letter and survey to ten people between the ages of 22-70. It was

decided that five would be female respondents and five would be male respondents. All have

graduated from college with at least a Bachelor’s degree, and they were instructed to treat this

cover letter and survey as though they would have received it from their very own school. All

reported that the cover letter and survey were easy to understand and to navigate. The survey

averaged between 6-8 minutes to complete.

The feedback from the survey was interesting, as all have experienced their alumni

associations’ communication at one time or another. Two of the ten people stated that they

actively give to their alumni association. Four others indicated that they have given to the

association at one time or another. The remaining four have never given to the alumni

association for their school. They were able to understand the pretense of what the survey was all

about, merely by the scope of the questions asked. When asked if there were any questions that

they could think of that should have been included in the survey, they could think of none. They

agreed that the cover letter was friendly and straight to the point, which they appreciated since as

one participant put it, “these types of letters usually are full of too much information.”

As far as the survey questions were arranged, I had more questions to ask in the

demographic section, such as annual income and ethnicity, but I felt these questions would make

some respondents a little tentative about answering.

All in all, I feel that this pretest covers everything the association would be looking for.

From the answers given, they can form opinions on how to better communicate with the alumni,

15 | P a g e

which in turn will help them reach their goal of receiving donations from 35 percent or more of

the alumni.

16 | P a g e

Compared to the focus group, this method will be high in both reliability and accuracy.

Using the stratified random sampling method to select the participants of the survey is an

excellent method to produce results with high projectability; and as learned earlier – makes them

a good source of interpretations upon which to base future campaigns. I used the following

calculations to produce a sample size of 600 respondents, which will produce a 95% confidence

level with a +/- 4% error. To get to this number, we will send out surveys to 750 potential

participants.

n = (cl/ci)^2 (v)(1-v)

n = number of participants needed

confidence level (cl) = 1.96 sd

confidence interval (ci) = .04

variance (v) = .5

n= (1.96/.04)^2 (.5)(1-.5)

n= (2401)(.5)(.5)

n= 600

We will send out surveys to 750 CEA alumni members, knowing that the response rate

will be dictated by those who are non-responsive. We assume that those who are non-responsive

are probably some of the same people who do not read any of the CEA promotional materials.

We are predicting that we will end up with 600 CEA respondents that will answer the questions

and give us feedback with high validity that will benefit the CEA and its goals.

17 | P a g e

The Design

What we are trying to determine with this experiment is whether a group that is exposed

to more information about an organization, along with two-way conversation with staff, will

increase the possibility of the person donating to the association, compared with the person who

is shown general information only. We are working with independent variables here of increased

information about the association, as well as the two-way conversation with staff. The dependent

variable is the person’s likeliness to donate. The results of the experiment will be reported to the

department so that they can develop a communication plan that will encourage donations to the

program.

The sample will come from the CEA’s own list of alumni using a stratified random

sampling method. As with the focus groups, these experiments will take place in hotel meeting

rooms is Seattle, Spokane, and Vancouver. After all participants watch a video which shares

basic information on what the association is all about, they will take a 15 minute break and when

they come back, they will be split into two groups, one a control group and the other a test group.

There will be 200 respondents in each location. The experiment will be run under supervision of

CEA staff. The experiment will be best served under a controlled environment.

This experiment will help determine whether the survey could be expanded into other

sections of Washington State and other sections of the country.

18 | P a g e

How the Experiment will work

As stated earlier, participants will be in two groups of 100, with one a control group and

the other a test group. When the experiment begins, all participants (200) will sit in on a video

presentation, describing general ideas about what the association is, events it sponsors, number

of alumni, where the association is located, etc. They will then be a survey about their thoughts

on donating to the association. After a 15 minute break, the participants will be split into two

groups of 100. When the experiment continues, the control group will continue to see the video

of general CEA information, the test group will be joined by officers from the board of the

association and allowed to ask any questions they want concerning the association. They will

also see a video, but it will contain much more detailed information about the association. All

participants will then be asked a series of questions about donating to the association. The results

will indicate whether the “personalized” attention will be effective in promoting donations from

the participants.

The Survey

First I want to thank you all for participating in this survey. Your feedback is vital in improving

communications between the CEA and you – its alumni. There will be two parts of this survey –

Part 1 and Part 2. Please answer the questions after each session to the best of your ability.

Part 1

Q1 – I graduated from the CEA in the”

Pre 1950’s

1950’s

1960’s

1970’s

1980’s

19 | P a g e

1990’s

After 2000

Q2 – I found the information in the videos to be useful in teaching me what actually happens

with CEA

Yes

No

Q3 – I donate to CEA:

Monthly

Annually

In a Lump Sum

I have never donated to the CEA

Q4 – After watching this presentation, I know more about the CEA now. I would love to learn

even more.

Yes, I’m interested

No thanks

Q5 – The things that interested me the most in the presentation was (check all that apply):

CEA Activities

Chances to Volunteer

Research

How to donate

CEA Website

CEA Social Media Sites

Other _______________

Q6 – The CEA contributed to where I am today with my career


Q7 – I am all for helping current and future CEA students to the best of my ability


20 | P a g e

Q8 – I want to help CEA. This coming year I will donate

Less than $50

$50 - $100

$101 - $500

$501 - $1000

More than $5000

Q9 – When donating to a cause, I usually donate

Monthly

Annually

In a Lump Sum

I have never donated to a cause

Q10 – I understand that my financial support could have a great impact on the future of the CEA


Q11 – I want the CEA to stay in contact with me


Q12 – It is really important to me that the CEA thrives and is successful


Part 2

Welcome back – I hope you enjoyed your break. Please sit down and get ready for part 2 of our

session.

After participants are separated into both groups. The control group is shown more general

material about the CEA, while the test group is visited by association board members who take

their questions. When time is up – the participants of both groups are given the same survey as

they completed after part 1.

We want to thank you for the time you have given us today. Your feedback is very important and

it will be used to improve communications between the CEA and its alumni.

21 | P a g e

I like the idea that the experiment should be rationally valid in both internal and external

validity, but the strength of external validity will probably not be nearly as strong as the

quantitative survey taken earlier. Also, can we be sure that the answers about donating from the

participants are completely valid. Even though they know that their participation and their

responses will be private, there is a chance that they will think they are on the hook for donating

if they say that they will or might in the future. The amount they say they donate, both to the

program and by donating outside the program, may be overinflated by social pressures.

Also, are we to believe that the respondents that live in Seattle, Spokane, and Vancouver

are a true representation of alumni in places such as Walla Walla or Longview? We can’t answer

that at this time. However, we do know that the experiment will give the promotion, marketing

and public relations departments of CEA something more to work with.

22 | P a g e

The CEA wanted a consultant to come in and study the ways that that the public relations

department of Western State University could more effectively communicate with 20,000 CEA

alumni. They also wanted the consultant to help discover ways to increase the percentage of

alumni who give back to the college to at least 35 percent. By directing questions to the

participants of the focus groups, quantitative survey, and experiment groups, we were able to

establish the idea that better communication would mean better donations. By instituting two-

way conversation with the alumni, as well as increasing college and alumni ties through stronger

communication, the college should be able to reach their goal of receiving donations of 35% or

more in the future.

COM564 Research methods for professionals final case study€¦ · DAVID T. GREEN Washington State...

Documents

Transcript of COM564 Research methods for professionals final case study€¦ · DAVID T. GREEN Washington State...