6 Steps 4 Standards Program Evaluation

8/13/2019 6 Steps 4 Standards Program Evaluation

1/26

Program Evaluation

The How

6-Steps & 4 StandardsAdapted from

http://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tips
http://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tipshttp://www.nwcphp.org/evaluation/learn-evaluation/program-evaluation-tips


2/26

A Framework for Program Evaluation


3/26

Program Evaluation

Effective program evaluation is a systematic

way to improve and account for actions.

Evaluation involves procedures that are useful,

feasible, ethical, and accurate.

A practical, non-prescriptive tool, the

evaluation framework summarizes and

organizes the stepsand standardsfor effectiveprogram evaluation.
http://www.cdc.gov/evaL/steps/index.htmhttp://www.cdc.gov/evaL/standards/index.htmhttp://www.cdc.gov/evaL/standards/index.htmhttp://www.cdc.gov/evaL/steps/index.htm


4/26

Program Evaluation

Definitions

Evaluation is the systematic investigation of themerit, worth or significance of an object (Scriven,

1999), hence assigning value to a programsefforts means addressing those three inter-relateddomains:

Merit (or quality)

Worth (or value, i.e., cost-effectiveness)

Significance (or importance)


5/26

Program Evaluation contA strong evaluation approach ensures that the following questions will

be addressed as part of the evaluation so that the value of programefforts can be determined and judgments about value can be made onthe basis of evidence:

1. What will be evaluated? (i.e., what is "the program" and in whatcontext does it exist?)

2. What aspects of the program will be considered when judgingprogram performance?

3. What standards (i.e., type or level of performance) must bereached for the program to be considered successful?

4. What evidence will be used to indicate how the program hasperformed?

5. What conclusions regarding program performance are justified bycomparing the available evidence to the selected standards?

6. How will the lessons learned from the inquiry be used to improvepublic health effectiveness?


6/26

Evaluation Framework

Evaluation framework provides a systematic

way to approach and answer these questions

using a set of 6 steps and 4 standards.


7/26

Six Steps of Program Evaluation

Program planning and evaluation go together.

These six steps can help put your organization

on the right track for continuous quality

improvement.


8/26

6-Step Process


9/26

Step 1: Define your stakeholders

Your stakeholders are supporters,implementers, recipients, and decision-makersrelated to your program.

Getting them involved early on will help youget different perspectives on the program andestablish common expectations.

This helps to clarify goals and objectives of theprogram youll evaluate, so everyoneunderstands its purpose.


10/26

Step 2: Describe the program

Taking the time to articulate what your program doesand what you want to accomplish is essential toestablishing your evaluation plan. Your descriptionsshould answer questions like: What is the goal of our

program? Which activities will we pursue to reach ourgoal? How will we do it? What are our resources? Howmany people do we expect to serve?

Articulating the answers to those questions will not

only help with accountability and quality improvement,but it will also help you promote the program to itsbeneficiaries.


11/26

Step 3: Focus the design of your

evaluation

Evaluations can focus on process, means,resources, activities, and outputs. They can focuson outcomes or how well you achieved your goal.You may also choose to evaluate both process

and outcomes. As you begin formulating your evaluation, think

about the specific purpose of the evaluationwhat questions are you trying to answer? How

will the information be used? What information-gathering methods are best suited for collectingwhat our organization needs to know?


12/26

Step 4: Gather evidence

Qualitative and quantitative data are the two main forms ofdata you may collect.

Qualitative data offers descriptive information that maycapture experience, behavior, opinion, value, feeling,knowledge, sensory response, or observable phenomena.Three commonly used methods used for gatheringqualitative evaluation data are: key informant interviews,focus groups, and participant observation.

Quantitative methods refer to information that may bemeasured by numbers or tallies. Methods for collectingquantitative data include counting systems, surveys, andquestionnaires.


13/26

Step 5: Draw conclusions

This is the step where you answer the bottom-line question: Are we getting better, gettingworse, or staying the same?

Data comparisons show trends, gaps,strengths, weaknesses.

You can compare evaluation data with targets

set for the program, against standardsestablished by your stakeholders or funders,or make comparisons with other programs.


14/26

Step 6: Present findings and ensure use

It is important that all the work you put intoprogram evaluation gets used for qualityimprovement.

When you present your findings andrecommendations, it is important to know thevalues, beliefs, and perceptions of your group;build on the groups background and build on

common ground; and state the underlyingpurpose for your recommendations before youget to the details.


15/26

Standards of Program Evaluation


16/26

Program Evaluation Standards

developed by the American Joint Committee onStandards for Educational Evaluation (AJCSEE)

increasingly been promoted through professionalevaluation associations, including the Americanand African evaluation associations.

These standards can be used both as a guide formanaging the evaluation process and to assess anexisting evaluation.

The standards highlight the considerations thatmust be weighed in formulating an evaluationdesign.


17/26

Program Evaluation Standards

1. Utility: seek to ensure that an evaluation will servethe information needs of intended users.

2. Feasibility: seek to ensure that an evaluation will berealistic, prudent, diplomatic, and frugal.

3. Propriety: seek to ensure that an evaluation will beconducted legally, ethically, and with due regard forthe welfare of those involved in the evaluation, aswell as those affected by its results.

4. Accuracy: seek to ensure that an evaluation willreveal and convey technically adequate informationabout the features that determine the worth or meritof the program being evaluated.


18/26

1. UtilityA. Stakeholder Identification

Persons involved in or affected by the evaluation should beidentified so their needs can be addressed.

B. Evaluator Credibility

Persons conducting the evaluation should be bothtrustworthy and competent to perform the evaluation so its

findings achieve maximum credibility and acceptance.C. Information Scope and Selection

Information collected should be broadly selected toaddress pertinent questions about the program and beresponsive to the needs and interests of clients and other

specified stakeholders.D. Values Identification

The perspectives, procedures, and rationale used tointerpret the findings should be carefully described so thebases for value judgements are clear.


19/26

1. Utility cont

E. Report Clarity

Evaluation reports should clearly describe the programbeing evaluated, including its context, purposes,procedures, and findings so that essential information isprovided and easily understood.

F. Report Timeliness and Dissemination

Significant interim findings and evaluation reports shouldbe disseminated to intended users so they can be used in atimely fashion.

G. Evaluation Impact Evaluations should be planned, conducted, and reported in

ways that encourage follow-through by stakeholders toincrease the likelihood that the evaluation will be used.


20/26

2. Feasibility

A. Practical Procedures The evaluation procedures should be practical to keep

disruption to a minimum while needed information isobtained.

B. Political Viability

The evaluation should be planned and conducted withanticipation of the different positions of various interestgroups so their co-operation may be obtained, and possibleattempts by any of these groups to curtail evaluationoperations or to bias or misapply the results can be averted

or counteracted.C. Cost Effectiveness

The evaluation should be efficient and produce informationof sufficient value so the resources expended can be

justified.


21/26


22/26

3. Propriety contE. Complete and Fair Assessment

The evaluation should be complete and fair in its examination andrecording of strengths and weaknesses of the program beingevaluated so that strengths can be built upon and problem areasaddressed.

F. Disclosure of Findings

The formal parties to an evaluation should ensure that the full set

of evaluation findings along with pertinent limitations are madeaccessible to the persons affected by the evaluation, and any otherswith expressed legal rights to receive the results.

G. Conflict of Interest

Conflict of interest should be dealt with openly and honestly so it

does not compromise the evaluation processes and results.H. Fiscal Responsibility

The evaluator's allocation and expenditure of resources shouldreflect sound accountability procedures, and otherwise be prudentand ethically responsible to ensure they are accounted for andappropriate.


23/26

4. AccuracyA. Program Documentation

The program being evaluated should be described and documentedclearly and accurately.

B. Context Analysis

The context of the program should be examined in enough detail soits likely influences can be identified.

C. Described Purposes and Procedures The purposes and procedures of the evaluation should be

monitored and described in enough detail so they can be identifiedand assessed.

D. Defensible Information Sources

The sources of information used in a program evaluation should bedescribed in enough detail so their adequacy can be assessed.

E. Valid Information

The information-gathering procedures should be chosen ordeveloped and implemented to ensure that the interpretation isvalid for the intended use.


24/26

4. Accuracy contF. Reliable Information

The information-gathering procedures should be chosen ordeveloped and implemented to ensure that theinformation is sufficiently reliable for the intended use.

G. Systematic Information

The information collected, processed, and reported in anevaluation should be systematically reviewed, and anyerrors found should be corrected

H. Analysis of Quantitative Information

Quantitative information should be appropriately and

systematically analysed so evaluation questions areeffectively answered.

I. Analysis of Qualitative Information

Qualitative information should be appropriately andsystematically analysed so evaluation questions are

effectively answered.


25/26

4. Accuracy cont

J. Justified Conclusions

The conclusions reached in an evaluation should beexplicitly justified so stakeholders can assess them.

K. Impartial Reporting

Reporting procedures should guard against distortioncaused by personal feelings and biases of any party tothe evaluation so that evaluation reports fairly reflectthe evaluation findings.

L. Meta-evaluation

The evaluation itself should be formatively andsummatively evaluated against these and otherpertinent standards so that its conduct is appropriatelyguided, and, on completion, stakeholders can closelyexamine its strengths and weaknesses.


26/26

6 Steps 4 Standards Program Evaluation

Documents

Transcript of 6 Steps 4 Standards Program Evaluation