w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

JSGS 824: HEALTH PROGRAM PLANNING &

EVALUATION -WINTER 2018

UNIVERSITY OF REGINA CAMPUS UNIVERSITY OF SASKATCHEWAN CAMPUS

INSTRUCTOR: Jasmin Bhawra, MSc

PHONE: Skype: jasmin.bhawra

E-MAIL: jasmin.bhawra@uregina.ca

OFFICE HOURS: Web Delivered

OFFICE LOCATION: WEB delivered course

TERM: Winter 2018

ROOM:

DATE AND TIME: January 8 - April 13, 2018

CALENDAR DESCRIPTION

This course is designed to provide the key principles of health program planning and evaluation to

health professionals who aspire to inform program development in their respective fields of expertise.

The areas of focus for this course include program planning models; design of evaluation plans; program

theory; mixed-methods data collection; indicators and measures of program outcomes; special topics

including Indigenous evaluation and policy evaluation, as well as evaluation reporting. By the end of the

course, students will be familiar with a variety of tools used in program planning and evaluation, and will

have gained experience in developing an evaluation plan.

LEARNING OBJECTIVES

Understand the relationship between program planning, research, and evaluation Develop a knowledge base of evaluation that could be used within multiple contexts of health

planning and programming Explore pros/cons and application of multiple evaluation approaches and types Demonstrate knowledge of mixed methods in conducting program evaluation Design an evaluation plan for a public health program, including the development of a logic model

(Insert learning objectives)

ATTRIBUTES OF JSGS GRADUATES

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

1. Management, Governance, and Leadership: Ability to inspire support for a vision or course of

action and successfully direct the teams, processes, and changes required to accomplish it.

2. Communication and Social Skills: Ability to communicate effectively and build enduring, trust-

based interpersonal, professional relationships.

3. Systems Thinking and Creative Analysis: Ability to identify key issues and problems, analyze

them systematically, and reach sound, innovative conclusions.

4. Public Policy and Community Engagement: Ability to understand how organizational and public

policies are formulated, their impact on public policy and management and how to influence

their development.

5. Continuous Evaluation and Improvement: Commitment to on-going evaluation for continuous

organizational and personal improvement.

6. Policy Knowledge: Ability to analyze and contribute content to at least one applied policy field.

COURSE CONTENT AND APPROACH

All course material will be posted on the course website: https://urcourses.uregina.ca/. PowerPoint

presentations will be posted on Wednesday every week (except for topics spanning two weeks).

Students are encouraged to use the discussion board to ask questions and engage in conversations

with their peers about the weekly modules (Insert course content and approach)

COURSE OUTLINE AND ASSIGNMENTS

MODULE 1- Introduction to Program Planning and Evaluation Week of January 8

Topics:

Characteristics of program planning and evaluation

Key definitions (i.e., program, stakeholder, target group, etc.)

Rationale for program planning/evaluation

Approaches to program planning

Social, political, and economic context considerations

Theories, frameworks and models

Program planning cycle

Difference between research and evaluation

Readings:

1. Fraser Health. 2009. A Guide to Planning and Conducting Program Evaluation. Section 2.0 – Background on Program Evaluation p. 10-15.

2. The Health Communication Unit - Centre for Health Promotion University of Toronto. 2001. Introduction to Health Promotion Program Planning. p.1-7.

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

3. Kegler MC, Rigler J, Honeycutt S. 2011. The role of community context in planning and implementing community-based health promotion projects. Evaluation and Program Planning, 34:246-253.

4. Levin-Rozalis M. 2003. Evaluation and research: Differences and similarities. Canadian Journal of Program Evaluation, 18(2):1-31.

5. Crosby R, Noar SM. 2011. What is a planning model? An introduction to PRECEDE-PROCEED. Journal of Public Health Dentistry, 71:S7-S15.

6. Tramm R, McCarthy A, Yates P. 2011. Using the Precede-Proceed model of health program planning in breast cancer nursing research. Journal of Advanced Nursing, 68(8):1870-1880.

MODULE 2- Planning a Program and an Evaluation Week of January 15

Topics:

Question(s)/problems, target populations, program goals and objectives,

Stakeholder relationships

Evaluability assessment

Internal vs. external evaluation

Introduction to logic models

Readings:

1. Dunet DO & Reyes M. 2006. Stakeholder-focused evaluation of an online course for health care providers. Journal of Continuing Education in the Health Professions, 26(4):257-67.

2. March of Dimes (2004). “SMART” objectives. 3. Fraser Health. 2009. A Guide to Planning and Conducting Program Evaluation: Section 3.0 –

Preparing for Evaluation and Section 4.0 – Develop an Evaluation Plan. p. 16-35. 4. Craig P, Campbell M. 2015. Evaluability assessment: A systematic approach to deciding whether

and how to evaluate programs and policies. What Works: Scotland. 5. Additional reading (in-depth description of the steps from the slides): Public Health Ontario.

2015. Planning health promotion programs: Introductory workbook. (4th Edition). Toronto, ON: Queens Printer for Ontario.

MODULE 3- Program Theory and Logic Models Week of January 22

Topics:

What is program theory

How evaluation questions flow from program theory

Program theory assessment

Development of logic models

Examples of theories relevant to health practice

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

Readings:

1. Tudor-Locke C, Myers AM, Rodger NW. 2001. Development of a theory-based physical activity intervention for individuals with Type 2 Diabetes. The Diabetes Educator, 27(1):85-93.

2. Astbury B & Leeuw FL. 2010. Unpacking black boxes: mechanisms and theory building in evaluation. American Journal of Evaluation, 31(3):363-381.

3. Chen H. 2003. Theory-driven approach for facilitation of planning health promotion or other programs. The Canadian Journal of Program Evaluation, 18(2):91-113.

4. Chapter 1: Introduction to Logic Models. 5. W.K. Kellogg Foundation. 2004. Logic Model Development Guide.

MODULE 4- Evaluation Approaches Week of January 29

Topics:

Overview of evaluation approaches

Use and purpose of approaches in guiding program evaluation

Examples of evaluation approaches:

Utilization-focused evaluation

Participatory evaluation

Appreciative inquiry

Developmental evaluation

Readings:

1. Patton MQ in Evaluation Models. (2000). Chapter 23: Utilization-focused Evaluation. 2. Patton MQ. (2013). Utilization-focused Evaluation Checklist. 3. Sette C. Participatory Evaluation. Better Evaluation website. Available at:

http://www.betterevaluation.org/en/plan/approach/participatory_evaluation 4. Minian N et al. Ontario Tobacco Health Unit: Using the principles of participatory evaluation to

evaluate tobacco smoking in Toronto’s Church and Wellesley neighbourhood. PowerPoint Presentation.

Additional readings:

5. Ramirez R & Brodhead D. (2013). Utilization-focused evaluation: A primer for evaluators. 6. MacLellan-Wright MF, Patten S, M. dela Cruz A, Flaherty A. (2007). A participatory approach to

the development of an evaluation framework: process, pitfalls, and payoffs. CJPE, 22(1):88-124. 7. Coghlan AT, Preskill H, Tzavaras Catsambas T. (2003). An overview of appreciative inquiry in

evaluation. New Directions for Evaluation, 100:5-20. 8. Reed J, Pearson P, Douglas B, Swinburne S, Wilding H. (2002). Going home from hospital – an

appreciative inquiry study. Health and Social Care in the Community,10(1):36-45. 9. Krasner MS et al. (2009). Association of an educational program in mindful communication with

burnout, empathy, and attitudes among primary care physicians. JAMA, 302(12): 1284-93. 10. Dozois E, Langlois M, Blanchet-Cohen N. (2010). A practitioner’s guide to developmental

evaluation. JW McConnell Family Foundation; International Institute for Child Rights and Development.

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

MODULE 5- Types of Evaluation Week of February 5 and February 12

(spans 2 weeks)

Topics:

Formative evaluation

Needs assessments

Process (implementation) evaluation

Outcome (effect) evaluation

Cost evaluations and economic analyses

Examples from the field of public health

Readings:

1. Langevin CM. 2001. An evaluation framework for the maison decision house substance abuse treatment program. Canadian Journal of Program Evaluation, 16(1):99-129.

2. Mercer NJ. 2009. Cost analysis of public health influenza vaccine clinics in Ontario. Canadian Public Health Association, 100(5):340-343.

3. Saunders RP, Evans MH, Praphul J. 2005. Developing a process-evaluation plan for assessing health promotion program implementation: a how-to guide. Health Promotion Practice, 6:134-147.

4. Lim S.S. Et al. 2010. India’s Janani Suraksha Yojana, A conditional cash transfer programme to increase births in health facilities: an impact evaluation. The Lancet, 375:2009-23.

5. Andresen MA, Boyd, N. 2009. A cost-benefit and cost effectiveness analysis of Vancouver’s supervised injection facility. International Journal of Drug Policy, doi:10.1016/j.drugpo.2009.03.004.

6. Boyd NR, Windsor RA. 2003. A formative evaluation in maternal and child health practice: The partners for life nutrition education program for pregnant women. Maternal and Child Health Journal, 7(2):137-143.

7. Types of Evaluation. 2015. CDC: National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention.

8. Gobin M, Verlander N, Maurici C, Bone A, Nardone A. 2013. Do sexual health campaigns work? An outcome evaluation of a media campaign to increase chlamydia testing among young people aged 15-24 in England. BMC Public Health, 13:484.

9. Quality Improvement and Innovation Partnership. 2009. Needs Assessment Resource Guide.

OPTIONAL ASSIGNMENT DEADLINE (EVALUATION PLAN PART 1): FEBRUARY 19

READING WEEK: FEBRUARY 20-24

MODULE 6 - Indicators and Measuring Program Outcomes Week of February 26

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

Topics:

Measurement and monitoring of program outcomes

Identification of relevant indicators

Assessment of program impact

Detection, interpretation, and analysis of program effects

Readings:

1. Adamchak et al. 2000. A guide to monitoring and evaluating adolescent reproductive health programs. Tool Series 5. Chapter 4: Indicators.

2. Rogers T, Chappelle EF, Wall HK et al. 2011. Using DHDSP outcome indicators for Policy and Systems Change for Program Planning and Evaluation. Atlanta, GA: Centers for Disease Control and Prevention.

3. National Resource Center. 2010. Measuring Outcomes guidebook. Available at: http://strengtheningnonprofits.org/resources/guidebooks/MeasuringOutcomes.pdf

4. University of Kansas. 2016. The Community Tool Box. Section 9: Developing Baseline Measures of Behaviour. Available at http://ctb.ku.edu/en/tablecontents/chapter_1003.htm

MODULE 7- Evaluation Design, Methods and Data Collection Tools Week of March 5 & March 12

(spans 2 weeks)

Topics:

Evaluation design Quantitative methods (i.e., surveys, google analytics) Qualitative methods (i.e., focus groups, content analysis, goal attainment scaling, Photovoice) Sources of data for health programs

Readings:

1. University of Kansas. 2015. Community Tool Box. Section 4, Chapter 37 – Selecting an appropriate design for the evaluation. Available at: http://ctb.ku.edu/en/table-of-contents/evaluate/evaluate-community-interventions/experimental-design/main

2. Measurement, Learning and Evaluation Project. 2013. Types of Evaluation Designs: Available at: https://www.urbanreproductivehealth.org/toolkits/measuring-success/types-evaluation-designs

3. Fraser Health. 2009. A Guide to Planning and Conducting Program Evaluation: Section 4.5, p. 35-40.

4. University of Kansas. 2015. The Community Tool Box. Available at http://ctb.ku.edu/en/tablecontents/chapter_1003.htm

Section 6: Conducting Focus Groups

Section 7: Needs Assessment Survey

Section 12: Conducting Interviews

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

Section 13: Conducting Surveys

Section 15: Qualitative Methods to Assess Community Issues

Section 24: Arranging Assessments that Span Jurisdictions 5. Kramer L, Schwartz P, Cheadle A, Rauzon S. 2013. Using photovoice as a participatory evaluation

tool in Kaiser Permanente’s Community Health Initiative. Health Promotion Practice, 14(5):686-94.

6. Rao V, Woolcock M. 2003. Integrating qualitative and quantitative approaches in program evaluation. Available at: http://siteresources.worldbank.org/SOCIALANALYSIS/1104890-1120158274352/20566665/Integratingqualitativeandquantapproachesraoandwoolcock.pdf

7. Mailloux Z, May-Benson TA, Summers CA, et al. 2007. Goal attainment scaling as a measure of meaningful outcomes for children with sensory integration disorders. American Journal of Occupational Therapy, 61(2):254-59.

8. Office of Quality Improvement, University of Wisconsin-Madison. 2010. Survey Fundamentals: A guide to designing and implementing surveys. Available at : http://oqi.wisc.edu/resourcelibrary/uploads/resources/Survey_Guide.pdf

9. Optional: Katapally Longo webinar on innovative methods

OPTIONAL ASSIGNMENT DEADLINE (EVALUATION PLAN PART 2): MARCH 12

MODULE 8- Special Topic: Conducting Evaluation in Indigenous Communities

Week of March 19

Topics:

Considerations for conducting evaluation in Indigenous communities/of Indigenous-focused programs

Appropriate evaluation approaches

Case example

Readings:

1. Jamieson et al. (2012). Ten principles relevant to health research among Indigenous Australian populations. MJA, 197(1): 16-18.

2. LaFrance J, Nichols R. (2010). Reframing evaluation: defining an Indigenous evaluation framework. CJPE, 23(2):13-31.

3. Chouinard JA, Cousins JB. (2007). Culturally competent evaluation for Aboriginal communities: a review of the literature. JMDE, 4(8):40-57.

Additional readings:

4. Katz I, Newton BJ, Bates S, Raven M. (2016). Evaluation theories and approaches; relevance for Aboriginal contexts.

5. Inuit Tapiriit Kanatami & Nunavut Research Institute. (2007). Negotiating research relationships with Inuit communities: a guide for researchers.

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

6. LaFrance, J., Nichols, R., & Kirkhart, K. E. (2012). Culture writes the script: On the centrality of context in indigenous evaluation. In D. J. Rog, J. L. Fitzpatrick, & R. F. Conner (Eds.), Context: A framework for its influence on evaluation practice. New Directions for Evaluation, 135, 59–74.

7. Public Health Agency of Canada. (2015). Ways tried and true: Aboriginal methodological framework for the Canadian Best Practices Initiative.

MODULE 9- Challenges and Ethical Considerations Week of March 26

Topics:

Professional standards and ethical conduct

Management of multiple stakeholders

Measurement bias and validity issues

Readings:

1. Canadian Evaluation Society Code of Ethics: Available at: https://evaluationcanada.ca/sites/default/files/ces_ethics_ethique_sce.pdf

2. American Evaluation Association Guiding Principles for Evaluators. Available at: http://www.eval.org/p/cm/ld/fid=51

3. Mohan R, Sullivan K. 2006. Managing the politics of evaluation to achieve impact. New Directions for Evaluation, 112:7-23

4. http://www.evaluatod.org/assets/resources/evaluation-guides/evaluationethics-2-09.pdf 5. Ray M. 2006. Choosing a truly external evaluator. American Journal of Evaluation, 27(3):372-77.

MODULE 10 – Policy Evaluation Week of April 2

Topics:

The role of evaluation in evidence-informed decision and policymaking

Approaches in conducting policy evaluation (key questions, data sources, methods)

Example of a policy evaluation

Readings:

1. National Center for Injury Prevention and Control (CDC). Brief overview of policy evaluation. Available at: http://www.cdc.gov/injury/pdfs/policy/Brief%201-a.pdf

2. National Collaborating Centre for Healthy Public Policy. 2012. A framework for analyzing public policies: Practical Guide. Available at: http://www.ncchpp.ca/docs/Guide_framework_analyzing_policies_En.pdf

3. World Health Organization. 2007. Monitoring and evaluation of mental health policies and plans. Mental Health Policy and Service Guidance Package. Available at: http://www.who.int/mental_health/policy/services/14-monitoring%20evaluation_HKprinter.pdf –

Section 2: Framework for Setting up and Conducting Evaluation of the Policy and Plan

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

Section 3: Case Study: An Evaluation of a National Mental Health Policy and Plan in a Hypothetical Country

MODULE 11 – Evaluation Reporting and Wrap Up Week of April 9

Readings:

1. Mueller NB, Burke RC, Douglas AL, et al. 2008. Getting the word out: multiple methods for disseminating evaluation findings. Journal of Public Health Management and Practice. 14(2):170-176.

2. Fraser Health. 2009. A Guide to Planning and Conducting Program Evaluation: Section 5.0 Conduct Evaluation and Disseminate Findings. p. 45 – 54.

3. Centers for Disease Control and Prevention. Evaluation Reporting: A Guide to Help Ensure Use of Evaluation Findings. Atlanta, GA: US Dept of Health and Human Services; 2013. Available at: http://www.cdc.gov/dhdsp/docs/Evaluation_Reporting_Guide.pdf

4. Health Canada. 2015. Evaluation of the nutrition policy and promotion program 2009-2010 to 2014-2015. Executive Summary. Full text available at: http://www.hc-sc.gc.ca/ahc-asc/performance/eval/nppp-evaluation-pppn-eng.php

5. Reference/supplementary reading only: Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health, Division of Nutrition, Physical Activity and Obesity. (2013). Developing an effective evaluation report: Setting the course for effective program evaluation. Available here: http://www.cdc.gov/eval/materials/Developing-An-Effective-Evaluation-Report_TAG508.pdf

FINAL EVALUATION PLAN ASSIGNMENT DEADLINE: APRIL 13

RECOMMENDED READINGS

All of the reading material for this course will be provided through the course website. This course has

no required textbooks, however optional reference books are listed below:

Issel LM. Health program planning and evaluation. 2nd or 3rd Edition. Toronto: Jones and Bartlett, 2008.

Green LW and Kreuter MW. Health Program Planning: An Educational and Ecological Approach. 4th

edition. New York: McGraw-Hill, 2005.

Myers A. Program evaluation for exercise leaders. 1st edition. Human Kinetics, 1999.

(Insert required readings)

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

EVALUATION

Assessment Weight Deadline

Participation in online discussion board 20% Ongoing

Evaluation Plan

- Part 1: Program description, evaluation purpose, and stakeholder

summary (20%)

- Part 2: Program logic model (30%)

- Part 3: Mixed methods evaluation design and draft survey or

interview guide (30%)

Students will be given the option to either:

1) Submit this assignment in its entirety on April 13 or;

2) Submit each section on its respective due date and receive

feedback prior to continuing with the rest of the assignment.

80%

Part 1 – February 19

Part 2 – March 12

Part 3 – April 13

*All written assignments will be assessed for overall grammar, clarity, presentation/formatting, and writing

style. A total of 5% of your assignment grade will be for writing style. Please consult University of Regina

Writing Services if you require help with writing or proofreading your assignments.

Participation

The discussion board will be used as a venue to post questions and engage in discussions with your

classmates. For each module, the course instructor will post a question or discussion topic for students

to discuss on the board for participation marks. Students are encouraged to share their experiences and

post any questions they may have about the material, however the grading will only be based on

responses to the question/discussion topic posted.

Discussion forum posts do not need to include references. There will be a word limit of 200 words to

help you keep responses succinct and incorporate readings/content from the module where

appropriate.

Live webinars will be held once per month to provide students with an opportunity to interact and ask

questions. Webinars will be scheduled using a poll after the term has commenced. Attendance for

webinars are optional and will not be graded for participation.

Evaluation Plan Assignment

This assignment will test students’ ability to apply the course concepts and design an evaluation for a

health program. Students will be given the option to choose between three different programs. The

assignment can be submitted in three parts, or in its entirety at the end of the course. More detailed

instructions and templates are available on the course website.

Policy on Late Assignments: It is essential that assignments are submitted by the time and date

specified. Late assignments will be penalized by deduction of 5% per day. In extenuating circumstances,

please contact the course instructor.

w w w . s c h o o l o f p u b l i c p o l i c y . s k . c a

STUDENTS WITH SPECIAL NEEDS

University of Regina (U of R): Students in this course who, because of a disability, may have a need

for accommodations are encouraged to discuss this need with the instructor and to contact the

Coordinator of Special Needs Services at (306) 585-4631.

U OF S: Students in this course who, because of a disability, may have a need for accommodations

are encouraged to discuss this need with the instructor and to contact Disability Services for

Students (DSS) at 966-7273.

Students Experiencing Stress

University of Regina (U of R): Students in this course who are experiencing stress can seek assistance

from the University of Regina Counselling Services. For more information, please see the attached

document, visit this website: http://www.uregina.ca/student/counselling/contact.html, or call (306)

585-4491 between 8:30 a.m. to 4:30 p.m. Saskatchewan time Monday to Friday.

ACADEMIC INTEGRITY AND CONDUCT

U of R: Ensuring that you understand and follow the principles of academic integrity and conduct as

laid out by the University of Regina (available at http://www.uregina.ca/gradstudies/grad-

calendar/policy-univ.html) is vital to your success in graduate school. Ensuring that your work is your

own and reflects both your own ideas and those of others incorporated in your work is important:

ensuring that you acknowledge the ideas, words, and phrases of others that you use is a vital part of

the scholarly endeavour. If you have any questions at all about academic integrity in general or

about specific issues, contact your course instructor to discuss your questions.