Frequently Asked Evaluation Questions

Tobacco Control Evaluation Center

Jeanette Treiber, Robin Kipke and Travis Satterlund

April 28, 2011

2

Where Do We Start?

Each evaluation activity is derived from the plan:

Objective plan type outcome/process data

How will data be used to move the objective forward?

Who should the data inform or convince?

Map out what pieces of information are needed, and how best to collect it (method, from whom, where, when)

3

What Is End-use Strategizing?

Identify your evaluation goals

Know the kind of information needed and specific purpose it should serve

Understand the best source for your data

Figure out when the data should be collected

Recognize what type of method is most useful

Decide an appropriate sample size

4

How Can We Use Our Data?

Your data can inform project staff by: Measuring how your message is being

perceived by various stakeholders (subgroups)

Indicating whether further community education efforts are needed

Assessing which policy options to pursue Identifying where your campaign might

encounter opposition Signaling that perhaps it is not yet the right

time to pursue policy

5

Are There Other Uses For Our Data?

Through a variety of reporting strategies, programs can use data to:

Indicate public/stakeholder readiness for a policy to decisionmakers

Generate “buzz” by sharing results in meetings, presentations, fact sheets, press releases

Build a groundswell of support and motivate more people to join the effort

Counteract critics

6

Should We Create New Instruments?

Adapt existing instruments (based on end use strategizing):

Not always necessary to reinvent the wheel

Adapt other data collection instruments for your project’s purposes

Start with high quality instrument Add questions you need data on Delete unnecessary questions Make sure it will collect what you need

7

Where Can We Find Instruments?

TCEC website has model instruments on limited number of objectives

Contact TCEC to access repository of instruments created/used by other projects

Ask other projects working on similar objectives to share their instruments

8

What Other Sites Have Instruments?

Download instruments from statewide sites

CA Health Interview Survey – http://www.chis.ucla.edu/questionnaires.html

CA Healthy Kids Survey – http://www.wested.org/cs/chks/view/chks_s/17?x-layout=surveys

CDC – http://cdc.gov/tobacco/data_statistics/surveys/

YTPS on STORE Website – http://www.tcsstore.org/stages/1_document/t_yps.pdf

9

How Can We Reach a Large Sample?

Utilize high traffic venues/locations Back-to-school packets Church service programs Grocery store flyers Classroom teachers Entertainment tickets (e.g. movie

theaters) Organizational newsletters Fairs and local events

10

What Are Other Distribution Avenues?

Online surveys (SurveyMonkey) Invite participation via email, social

networking media (Facebook, Twitter) Your organization’s website (and

those of coalition members, agencies that serve target population)

11

What Are the Pros of Online Surveys?

Cost effective Faster delivery Quick response time Better at addressing sensitive issues Ease of layout design and analysis

12

How Can We Increase Response Rates for Online Surveys?

Some tips for maximizing the response rates:

Request participation in advance

Include a cover letter

Allow sufficient time frame to complete the survey

Provide survey instructions

Design a clear survey

Send reminders to those who still have not completed the survey

Offer incentives for completion

13

Do We Need to Pilot Test the Instrument? Evaluate instrument’s effectiveness Get a sense of how long it takes to complete Create checklist of concerns to address with

the pre-test Do respondents/data collectors understand the

instrument? Is the wording clear to all segments of sample

population? Do any items produce irritation or embarrassment? Have important issues been overlooked?

14

What Should Our Key Informant Interview Protocol Look Like? Identify your goals beforehand

Know the kind of information needed and specific purpose it should serve

Create a structure to your interview

Probe, probe, probe (write probes into your interview protocol)

15

What Kind Of Outcome Data Do We Need For A MUH Objective?Depending on the objective:

Observed presence and placement of signage

Observation data of smoking and tobacco litter

Confirmed smoke-free language in leases

Tabulated count of signed leases with smoke-free language

16

How Can We Tap Into Volunteer Labor?

Develop relationships with local universities, high schools, programs or clubs

Recruit students looking for research experience to do data collection

Offer training, internships, course credit, other incentives

Entice coalition members to join in the activities

17

How Do We Make Use Of Our Volunteers?

To field test instruments/materials for appropriateness to data sources

To translate instruments/materials into other languages

To collect data To provide access to other distribution

points

18

How Do We Write Up the Results From Our Data Collection?

Summarize as you go Write “mini-analyses” See if the data can help you meet your

objective Incorporate your summaries and

analyses into your reports

19

How Can We Utilize TCEC’s Resources?

Online coalition satisfaction survey administration and analysis service

Additional model data collection instruments on a variety of topics will be developed in coming months

Model instruments translated into other languages (Spanish, Vietnamese, Hmong and Russian)

We’re working with other statewide projects to develop tools that will help you use and share your data more effectively for your policy work

20

When Should We Contact TCEC?

For individualized assistance with: evaluation planning development of data collection

instruments advice on analyzing & interpreting data guidelines for reporting & sharing info

21

Resources You Can Tap

TCEC archive of data collection instruments Evaluation associates at TCEC who can advise

you on developing instruments Recorded webinars on survey development,

sampling decisions, conducting POS Tips & Tools on creating interview questions,

sampling and end-use strategizing Electronic quarterly newsletter Your colleagues working on similar objectives