Post on 04-Jun-2018
8/13/2019 6 Steps 4 Standards Program Evaluation
1/26
Program Evaluation
The How
6-Steps & 4 StandardsAdapted from
http://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tips
http://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tips8/13/2019 6 Steps 4 Standards Program Evaluation
2/26
A Framework for Program Evaluation
8/13/2019 6 Steps 4 Standards Program Evaluation
3/26
Program Evaluation
Effective program evaluation is a systematic
way to improve and account for actions.
Evaluation involves procedures that are useful,
feasible, ethical, and accurate.
A practical, non-prescriptive tool, the
evaluation framework summarizes and
organizes the stepsand standardsfor effectiveprogram evaluation.
http://www.cdc.gov/evaL/steps/index.htmhttp://www.cdc.gov/evaL/standards/index.htmhttp://www.cdc.gov/evaL/standards/index.htmhttp://www.cdc.gov/evaL/steps/index.htm8/13/2019 6 Steps 4 Standards Program Evaluation
4/26
Program Evaluation
Definitions
Evaluation is the systematic investigation of themerit, worth or significance of an object (Scriven,
1999), hence assigning value to a programsefforts means addressing those three inter-relateddomains:
Merit (or quality)
Worth (or value, i.e., cost-effectiveness)
Significance (or importance)
8/13/2019 6 Steps 4 Standards Program Evaluation
5/26
Program Evaluation contA strong evaluation approach ensures that the following questions will
be addressed as part of the evaluation so that the value of programefforts can be determined and judgments about value can be made onthe basis of evidence:
1. What will be evaluated? (i.e., what is "the program" and in whatcontext does it exist?)
2. What aspects of the program will be considered when judgingprogram performance?
3. What standards (i.e., type or level of performance) must bereached for the program to be considered successful?
4. What evidence will be used to indicate how the program hasperformed?
5. What conclusions regarding program performance are justified bycomparing the available evidence to the selected standards?
6. How will the lessons learned from the inquiry be used to improvepublic health effectiveness?
8/13/2019 6 Steps 4 Standards Program Evaluation
6/26
Evaluation Framework
Evaluation framework provides a systematic
way to approach and answer these questions
using a set of 6 steps and 4 standards.
8/13/2019 6 Steps 4 Standards Program Evaluation
7/26
Six Steps of Program Evaluation
Program planning and evaluation go together.
These six steps can help put your organization
on the right track for continuous quality
improvement.
8/13/2019 6 Steps 4 Standards Program Evaluation
8/26
6-Step Process
8/13/2019 6 Steps 4 Standards Program Evaluation
9/26
Step 1: Define your stakeholders
Your stakeholders are supporters,implementers, recipients, and decision-makersrelated to your program.
Getting them involved early on will help youget different perspectives on the program andestablish common expectations.
This helps to clarify goals and objectives of theprogram youll evaluate, so everyoneunderstands its purpose.
8/13/2019 6 Steps 4 Standards Program Evaluation
10/26
Step 2: Describe the program
Taking the time to articulate what your program doesand what you want to accomplish is essential toestablishing your evaluation plan. Your descriptionsshould answer questions like: What is the goal of our
program? Which activities will we pursue to reach ourgoal? How will we do it? What are our resources? Howmany people do we expect to serve?
Articulating the answers to those questions will not
only help with accountability and quality improvement,but it will also help you promote the program to itsbeneficiaries.
8/13/2019 6 Steps 4 Standards Program Evaluation
11/26
Step 3: Focus the design of your
evaluation
Evaluations can focus on process, means,resources, activities, and outputs. They can focuson outcomes or how well you achieved your goal.You may also choose to evaluate both process
and outcomes. As you begin formulating your evaluation, think
about the specific purpose of the evaluationwhat questions are you trying to answer? How
will the information be used? What information-gathering methods are best suited for collectingwhat our organization needs to know?
8/13/2019 6 Steps 4 Standards Program Evaluation
12/26
Step 4: Gather evidence
Qualitative and quantitative data are the two main forms ofdata you may collect.
Qualitative data offers descriptive information that maycapture experience, behavior, opinion, value, feeling,knowledge, sensory response, or observable phenomena.Three commonly used methods used for gatheringqualitative evaluation data are: key informant interviews,focus groups, and participant observation.
Quantitative methods refer to information that may bemeasured by numbers or tallies. Methods for collectingquantitative data include counting systems, surveys, andquestionnaires.
8/13/2019 6 Steps 4 Standards Program Evaluation
13/26
Step 5: Draw conclusions
This is the step where you answer the bottom-line question: Are we getting better, gettingworse, or staying the same?
Data comparisons show trends, gaps,strengths, weaknesses.
You can compare evaluation data with targets
set for the program, against standardsestablished by your stakeholders or funders,or make comparisons with other programs.
8/13/2019 6 Steps 4 Standards Program Evaluation
14/26
Step 6: Present findings and ensure use
It is important that all the work you put intoprogram evaluation gets used for qualityimprovement.
When you present your findings andrecommendations, it is important to know thevalues, beliefs, and perceptions of your group;build on the groups background and build on
common ground; and state the underlyingpurpose for your recommendations before youget to the details.
8/13/2019 6 Steps 4 Standards Program Evaluation
15/26
Standards of Program Evaluation
8/13/2019 6 Steps 4 Standards Program Evaluation
16/26
Program Evaluation Standards
developed by the American Joint Committee onStandards for Educational Evaluation (AJCSEE)
increasingly been promoted through professionalevaluation associations, including the Americanand African evaluation associations.
These standards can be used both as a guide formanaging the evaluation process and to assess anexisting evaluation.
The standards highlight the considerations thatmust be weighed in formulating an evaluationdesign.
8/13/2019 6 Steps 4 Standards Program Evaluation
17/26
Program Evaluation Standards
1. Utility: seek to ensure that an evaluation will servethe information needs of intended users.
2. Feasibility: seek to ensure that an evaluation will berealistic, prudent, diplomatic, and frugal.
3. Propriety: seek to ensure that an evaluation will beconducted legally, ethically, and with due regard forthe welfare of those involved in the evaluation, aswell as those affected by its results.
4. Accuracy: seek to ensure that an evaluation willreveal and convey technically adequate informationabout the features that determine the worth or meritof the program being evaluated.
8/13/2019 6 Steps 4 Standards Program Evaluation
18/26
1. UtilityA. Stakeholder Identification
Persons involved in or affected by the evaluation should beidentified so their needs can be addressed.
B. Evaluator Credibility
Persons conducting the evaluation should be bothtrustworthy and competent to perform the evaluation so its
findings achieve maximum credibility and acceptance.C. Information Scope and Selection
Information collected should be broadly selected toaddress pertinent questions about the program and beresponsive to the needs and interests of clients and other
specified stakeholders.D. Values Identification
The perspectives, procedures, and rationale used tointerpret the findings should be carefully described so thebases for value judgements are clear.
8/13/2019 6 Steps 4 Standards Program Evaluation
19/26
1. Utility cont
E. Report Clarity
Evaluation reports should clearly describe the programbeing evaluated, including its context, purposes,procedures, and findings so that essential information isprovided and easily understood.
F. Report Timeliness and Dissemination
Significant interim findings and evaluation reports shouldbe disseminated to intended users so they can be used in atimely fashion.
G. Evaluation Impact Evaluations should be planned, conducted, and reported in
ways that encourage follow-through by stakeholders toincrease the likelihood that the evaluation will be used.
8/13/2019 6 Steps 4 Standards Program Evaluation
20/26
2. Feasibility
A. Practical Procedures The evaluation procedures should be practical to keep
disruption to a minimum while needed information isobtained.
B. Political Viability
The evaluation should be planned and conducted withanticipation of the different positions of various interestgroups so their co-operation may be obtained, and possibleattempts by any of these groups to curtail evaluationoperations or to bias or misapply the results can be averted
or counteracted.C. Cost Effectiveness
The evaluation should be efficient and produce informationof sufficient value so the resources expended can be
justified.
8/13/2019 6 Steps 4 Standards Program Evaluation
21/26
8/13/2019 6 Steps 4 Standards Program Evaluation
22/26
3. Propriety contE. Complete and Fair Assessment
The evaluation should be complete and fair in its examination andrecording of strengths and weaknesses of the program beingevaluated so that strengths can be built upon and problem areasaddressed.
F. Disclosure of Findings
The formal parties to an evaluation should ensure that the full set
of evaluation findings along with pertinent limitations are madeaccessible to the persons affected by the evaluation, and any otherswith expressed legal rights to receive the results.
G. Conflict of Interest
Conflict of interest should be dealt with openly and honestly so it
does not compromise the evaluation processes and results.H. Fiscal Responsibility
The evaluator's allocation and expenditure of resources shouldreflect sound accountability procedures, and otherwise be prudentand ethically responsible to ensure they are accounted for andappropriate.
8/13/2019 6 Steps 4 Standards Program Evaluation
23/26
4. AccuracyA. Program Documentation
The program being evaluated should be described and documentedclearly and accurately.
B. Context Analysis
The context of the program should be examined in enough detail soits likely influences can be identified.
C. Described Purposes and Procedures The purposes and procedures of the evaluation should be
monitored and described in enough detail so they can be identifiedand assessed.
D. Defensible Information Sources
The sources of information used in a program evaluation should bedescribed in enough detail so their adequacy can be assessed.
E. Valid Information
The information-gathering procedures should be chosen ordeveloped and implemented to ensure that the interpretation isvalid for the intended use.
8/13/2019 6 Steps 4 Standards Program Evaluation
24/26
4. Accuracy contF. Reliable Information
The information-gathering procedures should be chosen ordeveloped and implemented to ensure that theinformation is sufficiently reliable for the intended use.
G. Systematic Information
The information collected, processed, and reported in anevaluation should be systematically reviewed, and anyerrors found should be corrected
H. Analysis of Quantitative Information
Quantitative information should be appropriately and
systematically analysed so evaluation questions areeffectively answered.
I. Analysis of Qualitative Information
Qualitative information should be appropriately andsystematically analysed so evaluation questions are
effectively answered.
8/13/2019 6 Steps 4 Standards Program Evaluation
25/26
4. Accuracy cont
J. Justified Conclusions
The conclusions reached in an evaluation should beexplicitly justified so stakeholders can assess them.
K. Impartial Reporting
Reporting procedures should guard against distortioncaused by personal feelings and biases of any party tothe evaluation so that evaluation reports fairly reflectthe evaluation findings.
L. Meta-evaluation
The evaluation itself should be formatively andsummatively evaluated against these and otherpertinent standards so that its conduct is appropriatelyguided, and, on completion, stakeholders can closelyexamine its strengths and weaknesses.
8/13/2019 6 Steps 4 Standards Program Evaluation
26/26