Considerations, Analysis of Approaches, and Guidelines

Small Group 3 10/15-10/26

Overview• Welcome to our presentation on Chapters 9-11

discussing Considerations, Analysis of Approaches, and Guidelines for evaluations.

• Content for each section will be delivered in multiple media based on the most effective means for presentation.

• Please review the summary content and respond to the discussion questions at the end of each section using the Blackboard discussion threads.

Table of Contents

Chapter 9

Other Current Considerations: Cultural Competence and Capacity Building

Chapter 10

Analysis of Approaches

Chapter 11

Clarifying the Evaluation Request and Responsibilities

Other Current Considerations: Cultural Competence and Capacity Building

Chapter 9

Presented by: Kaylea Howarth

NINECultural Competence & Capacity Building

• Two factors that influence evaluation practice, but transcend particular evaluation approaches: • the need to build cultural competence• evaluation capacity building or mainstreaming evaluation

• We need to give consideration to the context of our evaluations because they take place in both an organizational context as well as a broader cultural context. These contexts influence, or should influence, our evaluation choices.

“If we do not consider these issues from the beginning of our evaluation, the validity, power, and use of the

evaluation will be diminished.” (Fitzpatrick, Sanders, & Worthen, 2011, p. 231)

Cultural Competence

“A systematic, responsive inquiry that is actively cognizant, understanding, and appreciative of the cultural context in which the evaluation takes place; that frames and articulates the epistemology of the evaluative endeavor; that employs culturally and contextually appropriate methodology; and that uses stakeholder-generated, interpretive means to arrive at the results and further use of the results” (SenGupta, Hopson, & Thompson-Robinson, 2004, p. 13).

Cultural Competence• The issue: As evaluators, we often

conduct evaluations within cultural contexts that differ from our own.

“This lack of familiarity affects our ability to appropriately direct and guide the evaluation, to conceptualize variables and ways to collect data on them,

to analyze and interpret results, and disseminate them to stakeholder groups in ways they can understand and use” (Fitzpatrick et al, 2011, p.

231)

Cultural Competence

Why is it important?• To identify the needs of stakeholders• To consider multiple perspectives of program success• To increase the legitimacy of the evaluation to all stakeholders• To increase the usefulness of the results

How can we achieve it?• Self-examination• Awareness of your own cultural norms• Make inclusion a priority• Quiet observation, respectful interactions, and reflection

Cultural Competence

Example: A pipeline company assigns an internal evaluator from head office to launch an evaluation of the company’s community investment program. This requires cultural competence in recognizing the context of each rural community and responding appropriately to each stakeholder’s needs.

Evaluation Capacity Building

• Learning

• Analysis

Evaluation

• Decisions

• Changes

ResultsOrganizatio

n

Process Use

• Evaluations within an organizational context have impact beyond programs, but on the organization itself.

• Process Use: • “Individual changes in thinking and behavior, and program or

organizational changes in procedures and culture, that occur among those involved in evaluation as a result of the learning that occurs during the evaluation process” (Patton, 1997c, p. 90).

Evaluation Capacity• The recognition of this impact lead to the movement to

mainstream evaluation in organizations. • Mainstreaming Evaluation:

• “The process of making evaluation an integral part of an organization’s everyday operations. Instead of being put aside in the margins of work, evaluation becomes a routine part of the organization’s work ethic if it is mainstreamed. It is part of the culture and job responsibilities at all levels of the organization” (Sanders, 2002, p. 254).

Workflow

Workflow

Evaluation

Evaluation

Non-Mainstreamed Mainstreamed

Evaluation Capacity Building• In parallel to

mainstreaming, others have maintained a focus on evaluation capacity building (ECB) in organizations.

Evaluation Capacity Building (ECB)“A context-dependent, intentional action system of guided processes and practices for bringing about and sustaining a state of affairs in which quality program evaluation and its appropriate uses are ordinary and ongoing practices within and/or between one or more organizations/programs/sites” (Stockdill, Baizerman, & Compton, 2002, p. 8).

• Evaluations in organizations have two forms: • project-based, designed to provide information on the

program or policy being evaluated• ongoing, with the evaluator working to sustain an

environment conducive to evaluation and its use within the organization

Evaluation Capacity Building

Strategies for building evaluation capacity and mainstreaming: • Stakeholder Involvement in an Evaluation• Coaching/Mentoring• Technology• Written Materials• Training• Communities of Practice• Meetings

Related ResourceClick on the item to reviewhttp://www.youtube.com/watch?v=uW-D9K0Ohl0

• Example: A financial trade company recognizes a need to evolve their programs over time to remain competitive within their sector. Yet as opposed to maintaining an evaluation group separate from the internal programs and processes to be evaluated, senior leadership pushes to mainstream the evaluation processes so that each job role and workgroup is involved in program improvement.

Evaluation Capacity Building

Limitations of mainstreaming evaluation and building evaluation capacity:• Need to build competence and skills within the

organization• Danger of reducing quality of formal evaluations• Organization members are not evaluation

professionals

Chapter 9 Discussion Questions

1. Cultural Competence: refer to example 1 on slide 9

You are the lead evaluator for the pipeline company’s Community Investment program evaluation. Develop a strategy for how you will enact cultural competence in deal with the regional contexts within your evaluation. Use the text for examples and guidelines.

2. Mainstreaming Evaluation: refer to example 2 on slide 15

You are a senior leader at the financial trade organization. Develop a strategy for how you will mainstream evaluation. Use the text as well as the Preskill presentation on slide 14 (specifically 10:14 – 14:30) for examples and guidelines.

Please post your responses in the designated Chapter 9 thread on Blackboard.

A Comparative Analysis of Approaches

Chapter 10

Presented by: Carolina Sanchez-Lopez & Reynaldo Lopez

TENA Comparative Analysis of Approaches

• Please view the following Prezi for a summary of Chapter 10:

Chapter 10: A Comparative Analysis of Approaches Prezi

http://prezi.com/7sdnxf_99ake/a-comparative-analysis-of-approaches/?utm_source=share&utm_campaign=shareprezi&utm_medium=email

Chapter 10 Discussion Questions1. Should we attempt to synthesize the various approaches into

one? What would be the advantages of doing so? 2. Adams Elementary School has started a volunteer program in

which parents are encouraged to help out in the classroom. The goal of the program is to not only provide the teacher with assistance, but also to get parents more involved in the school and their children’s education. The principal hopes to boost the learning of the children who are achieving below grade level by getting their parents more involved in their children’s education through volunteer efforts in the classroom. Contrast using a decision-oriented, participant-oriented, and program-oriented approach.

Please post your responses in the designated Chapter 10 thread on Blackboard.

Clarifying the Evaluation Request and Responsibilities

Chapter 11

Presented by: Carole Poche

ELEVENClarifying the Evaluation Request & Responsibilities

• Please view the following Prezi for a summary of Chapter 11:

Chapter 11: Clarifying the Evaluation Request & Responsibilities

http://prezi.com/recommend/5q1wxgdcqfsh

Chapter 11 Discussion Questions• In getting down to some solid planning for an evaluation –

what concerns must one address?• When might you refuse to do an evaluation? What types of

reasons exist not to evaluate?• What types of research results can not be used?• What reasons are given for the creation of evaluability

assessment? How does a program become evaluable?

Please post your responses in the designated Chapter 11 thread on Blackboard.

References• Fitzpatrick, J.L., Sanders, J.R., & Worthen, B.R. (2011). Program Evaluation:

Alternative Approaches and Practical Guidelines (4th Ed.). Pearson. Allyn & Bacon.• Patton, M. Q. (1997c). Utilization-focused evaluation: The new century text. Thousand

Oaks, CA: Sage.• Preskill, H. Effective Strategies for Facilitating Evaluation Capacity Building. [video]

Retrieved from http://www.youtube.com/watch?v=uW-D9K0Ohl0• Sanders, J. (2002). Presidential address: On mainstreaming evaluation. American

Journal of Evaluation, 23(3), 253-259. • Stockdill, S. H., Braizerman, M., & Compton, D. W. (2002). Toward a definition of the

ECB process: A conversation with the ECB lliterature. In D.W. Compton, M. Braizerman, & S. H. Stockdill (Eds.), The art, craft, and science of evaluation capacity building. New Directions for Evaluation, No. 93, 7-26. San Francisco: Jossey-Bass.

• SenGupta, S., Hopson, R., & Thompson-Robinson, M. (2004). Cultural competence in evaluation: An overview. In M. Thompson-Robinson, R. Hopson, & S. SenGupta(Eds.), In search of cultural competence in evaluation: Toward principles and practices. New Directions for Evaluation No. 102, 5-20. San Francisco: Jossey-Bass.

• Trevisan, M. And Huang, Y. (2003) Evaluability Assessment: A Primer, Practical Assessment, Research & Evaluation. Retrieved from http://pareonline.net/getvn.asp?v=8&n=20

Images retrieved from iStock Royalty Free Images