EVALUATION, DEMOCRACY, AND SOCIAL CHANGE€¦ · Evaluation, Democracy, and Social Change119...

Educational research [and evaluation] cannever be value-free. To the extent it approachesvalue-freedom in its self-perception, it is to thatextent dangerous … [and] in fact … useless …[Moreover] I take it as a given that democraticvalues are prominent among those that educa-tional research [and evaluation] ought to incor-porate, a premise not likely to be challenged inthe abstract (Howe, 2003, pp. 133–134).

From almost the beginning of the contempo-rary history of program evaluation, therehave been theorists and practitioners whoanchor their work in an intentional commit-ment to democratic social justice, equality, orempowerment. These evaluators reject thevery possibility of value neutrality in evalua-tion and instead fully embrace the intertwine-ment of values with evaluative practice.Moreover, these evaluators go beyond a value-relative stance, which acknowledges andengages the plurality of values that inhabitevaluation contexts, to a value-committed

stance, through which evaluation purpose-fully advances particular values (Schwandt,1997). The most defensible values to pro-mote, in the reasoning of these evaluators,are those intrinsic to political democraticideals, namely, social justice, equality,empowerment, and emancipation.

The rationales offered by the theoristsin this evaluative tradition for their value-committed stances are complex. They rest onboth epistemological arguments regardingthe nature and purpose of social knowledgeand political arguments regarding the loca-tion and purpose of evaluation in society.And they rest on varied conceptualizations ofdemocracy, equality, and justice. Moreover,these arguments are less about particularevaluation designs and methods than theyare about evaluative processes and evaluatorroles, stances, and commitments. That is,these theories about democratically orientedevaluation do not emphasize prescriptions

EVALUATION,DEMOCRACY, ANDSOCIAL CHANGE

Jennifer Greene

5

Shaw-3369-Chapter-05.qxd 2/24/2006 6:08 PM 18

about the technical aspects of evaluationpractice. Rather, these theories focus on(1) the macro positioning of evaluation insociety, specifically addressing issues related towhich purposes and whose interests evalua-tion should serve, and (2) the micro characterof evaluation practice, in particular therelationships evaluators establish with othersin a given context and the processes andinteractions that enact these relationships.Clearly, these emphases spill over into moretechnical issues of establishing priority evalu-ation questions, criteria for judging quality,utilization, and reporting procedures, as wellas evaluation design and methods. But, aswill be illustrated throughout this chapter,democratically oriented evaluators’ ideasabout the technical facets of evaluation prac-tice are most importantly rooted in theirunderstandings of evaluation’s location insociety and the evaluator’s location in thestudy at hand.

Using this heuristic framework of themacro politics and the micro relationships ofevaluation, this chapter first presents thehistorically influential theories of BarryMacDonald and Ernest House, then importantfacets of additional theories that have shapedthe landscape of democratically oriented eval-uation, and finally contemporary develop-ments that continue the tradition. For eachtheorist or group of theorists, the discussionincludes key concepts and rationales – bothepistemological and political – as well as keyimplications for evaluation practice. Examplesand critiques of these approaches are inter-spersed throughout the chapter, with a con-cluding summary critique. And although thediscussion takes place almost exclusively in thepublic sector, involving evaluations of publiclyfunded programs, democratically orientedevaluative theory is certainly relevant to thenon-profit, civil sectors and even in some casesto private enterprise as well.1

Historical Legacies in DemocraticallyOriented Evaluation

Democratically oriented traditions in evalua-tion have their genesis in Barry MacDonald’soriginal formulation of “democratic evalua-tion” for the field of education in England(MacDonald, 1976) and Ernest House’s long-standing commitment to social justice forevaluation in the US (House, 1980, 1993;House & Howe, 1999).

Barry MacDonald’s“Democratic Evaluation”

MacDonald offered a “political classificationof evaluation studies” as a way of helpingevaluators choose their “allegiances and priori-ties,” because evaluators inevitably confront“the distribution and exercise of power” in theirwork (MacDonald, 1976, p. 125). Evaluation isinherently a political activity with potentialpolitical influence. “Evaluators not only live inthe real world of educational politics; they actu-ally influence its changing power relationships”(MacDonald, 1976, p. 132).

MacDonald’s political classification hadthree types. First, bureaucratic evaluation is anunconditional service to government agenciesalready empowered to allocate educationalresources and determine policy directions. Thebureaucratic evaluator’s role is one of man-agement consultant, and his/her work isneither independent nor available for publicscrutiny. Bureaucratic evaluation clientsretain control over the products of this work.Second, autocratic evaluation is a conditionalservice to the same governmental agencies.The autocratic evaluator retains indepen-dence as an outside expert adviser and thusretains ownership of the evaluation products.His/her work is validated by the scientificresearch community and thus, when valid,serves to defend existing policy directions.

Evaluation, Democracy, and Social Change 119

Shaw-3369-Chapter-05.qxd 2/24/2006 6:08 PM Page 119

In contrast, the democratic evaluator recog-nizes value pluralism in service of the publicright to know. In democratic evaluation, themethods and results must be presented inways accessible to multiple non-specialistaudiences, in a report that aspires to “bestseller” status. Moreover, all participants in theevaluation are guaranteed control over therelease and form of the information they pro-vide. In short, the democratic evaluator servesthe public interest in education, in addition tothe established interests of policy-makers andexperts.

Rationale

MacDonald’s turn to a democratically ori-ented approach to evaluation arose from hisconcerns about “Who controls the pursuit ofnew knowledge, and who has access to it?”(MacDonald & Walker, 1977, p. 185). Hesought primarily to democratize knowledge inevaluation – to broaden the evaluation ques-tions addressed and thus the interests servedbeyond established decision-makers andexperts to include the citizenry at large, andalso to disseminate evaluation findingsequally broadly so as to engage the public ininformed discussion of key policy issues anddirections – thus positioning evaluation inservice of an informed citizenry.2 MacDonaldalso envisioned evaluation as an opportunityfor policy critique, rather than an activity con-strained by the boundaries of a particular pro-gram (which is an enactment of a policy) withthe assumptions and values of the policy leftunexamined. Evaluation can serve as a “disin-terested source of information about theorigins, processes, and effects of social action …challenging monopolies of various kinds – ofproblem definition, of issue formulation, ofdata control, of information utilization. We arenot just in the business of helping some peopleto make educational choices within their

present responsibilities and opportunities. Weare also in the business of helping all ourpeoples to choose between alternative societies”(MacDonald, 1978, p. 12). With this collectiveand pluralistic vision of evaluation in democra-tic service for policy-makers, experts, and thepublic alike, accountability also becomesmutual and collective (Ryan, 2004).

Major Implications for Practice

MacDonald’s political turn to democratic val-ues in evaluation was accompanied by amethodological turn to the case study for edu-cational evaluation.3 Case studies can renderportrayals of educational programs “moreknowable to the non-research community[and] more accessible to diverse patterns ofmeaning, significance and worth throughwhich people ordinarily evaluate social life”(MacDonald, 1977, p. 50). Case studies take“the experience of the programme partici-pants as the central focus of investigation[and they] convey images of educational activ-ity which both preserve and illuminateits complexity” (MacDonald, 1977, p. 51).Within MacDonald’s democratic evaluation,the case study method focuses on practice andon practitioners’ own language and under-standings of or theories about the program(Simons, 1987), and further serves to encour-age critical self-reflection about the quality ofprogram implementation and its connectionsto policy intent.

In conjunction with the case study method,MacDonald’s democratic evaluation requiresthat evaluators themselves act democratically,primarily in reference to control over, accessto, and release of evaluative information. Thisis because the personalized information gener-ated in case study evaluations (in contrast tothe anonymous information generated inother evaluation approaches) can be impor-tantly consequential for case study participants.

120 HANDBOOK OF EVALUATION—CHAPTER 5


Principles of fairness, relevance, and accuracyguide all negotiations between evaluators andstudy participants regarding the content anddissemination of all evaluation reports.

An Example of MacDonald’sDemocratic Evaluation

Robin McTaggart, an active member of theCARE–CIRCE network and a renowned propo-nent of action research, offers a thoughtful cri-tique of the promise and peril of MacDonald’sdemocratic evaluation through reflections on acase example (McTaggart, 1991). The pro-gram evaluated was an Australian LanguageCurriculum Project, which sought to providespecialized instruction for students with identi-fied weaknesses in language skills. The evalua-tion was self-consciously democratic in theMacDonald tradition, focusing thus onprocesses related to information control. In facta written set of Principles of Procedure –designed to make “an externally commissionedevaluation as democratic as possible by givingparticipants considerable control over theinterpretation and release of information”(McTaggart, 1991, p. 10) – was shared with allevaluation participants and used to guide theevaluators’ actions and decisions regardinginformation release and especially reporting.4

The troublesome incident in this caseexample involved a male school principal andthe female language teacher (hired specificallyfor the program and thus not on a tenuretrack) in one of the program sites. When inter-viewed, the principal offered glowing supportfor the program but was not aware of any of theprogram’s operational details, encouraging theevaluation team to consult the teacher directlyinvolved. When the language teacher was inter-viewed, she offered significant criticism of theprogram primarily with reference to its organi-zation and management. For example, thestudents who showed up for the program were

not the kinds of students the program wasintended to serve, nor did they come from theschools designated as participants in the pro-gram. As per the Principles of Procedure, boththe principal and the teacher reviewed theirinterview records and agreed that the data,with minor corrections, could be included inthe evaluation report. Yet, as the draft reportwas circulating again for approval and release,the teacher recanted and withdrew much of herinterview data from the evaluation, notably thedata critical of the language program. Whentelephoned, the teacher said that the principalhad asked her to change her interview dataso that the data were not critical of theprogram’s organization and management(because this would reflect badly on the princi-pal). The principal even hinted that the teacher’sjob could be at stake. This then created a dilemmafor MacDonald’s democratic evaluation: “Shouldthe public’s ‘right to know’ take precedence overthe individuals’ rights to ‘own the facts abouttheir own lives’” (McTaggart, 1991, p. 15) and todecide for themselves what risks to incur? And,“however democratic the Principles of Proceduremay have seemed to be, they still gave the evalu-ators considerable control” (p. 20).

Critique

MacDonald’s democratic evaluation supportsa representative form of democracy in that thepower of elected officials and their appointeesto make decisions is engaged, but not chal-lenged. (The concluding section of this chap-ter offers further discussion of different formsof democracy.) Moreover, within the spaces ofthe evaluation itself, McTaggart’s Principles ofProcedure did actively seek and value theteacher’s views about the program, but did notadequately safeguard other rights of theteacher against the established power of theprincipal. Nor were any evaluation participantsactually empowered to speak for themselves,



as the evaluator, in the role of “informationbroker,” retained authorship of the finalreport. MacDonald’s approach to democraticevaluation thus serves primarily to “givevoice” to and thereby legitimize the perspec-tives and experiences of multiple stakeholders.It endeavors to provide and protect spaceswithin evaluation for multiple accounts ofprogram value, but is inherently limited in itsability to guarantee either provision or protec-tion of stakeholder voice, as power remainswith the evaluator who is positioned in serviceto the established representative government.

Ernest House’s Deliberative DemocraticEvaluation, with Kenneth Howe

For almost as long as Barry MacDonald, ErnestHouse has championed a democratic approachto evaluation that takes particular Americanform as it seeks primarily to address inequitiesof social class and minority culture and toadvance “social justice” in the context at handand in the broader society (House, 1980,1993; House & Howe, 1999). House attendsspecifically to the ways in which evaluation notjust influences but actually serves to constitutepublic decision-making institutions anddiscourses, and thereby policy directions.

Evaluation always exists within some authoritystructure, some particular social system. It doesnot stand alone as simply a logic or a methodol-ogy, free of time and space, and it is certainly notfree of values or interests. Rather, evaluationpractices are firmly embedded in and inextricablytied to particular social and institutional struc-tures and practices. (House & Howe, 2000, p. 3)

Given that evaluation is embedded in the fab-ric of public decision-making rather than anindependent contributor to it, evaluation“should be explicitly democratic” (House &Howe, 2000, p. 4). As such, evaluation canhelp to constitute a more democratic society.

The character of democracy promoted byHouse is one of deliberation in service of socialjustice. Historically, House rejected a pluralistmodel of democracy favored by many liberalsocial scientists for much of the twentiethcentury, because it does not attend seriously tothe interests of the least advantaged.5 In thepluralistic model, “the political system is kept inequilibrium by group elites bargaining for theirconstituencies and government elites reachingaccommodations. There is little need for directparticipation by individuals other than to expresstheir demands to their leaders” (House, 1993,p. 118).6 However, argued House, pluralismoften excludes some stakeholders, usually the“powerless and the poor” (House, 1993, p. 121)because there are no special provisions for theirinclusion. Further, “many critical issues neverarise for discussion, study, or evaluation … . [Inparticular] fundamental issues involving con-flicts of interest often do not evolve into publicissues because they are not formulated”(House, 1993, pp. 121–122). (This discussionof House’s views on democratic theories iscontinued below under “Rationale.”)

In collaboration with philosopher colleagueKenneth Howe, House has most recently pre-sented a deliberative democratic model for eval-uation (House & Howe, 1999, 2000). Thismodel intentionally insures that the interestsof all stakeholders, specifically those of thepowerless and the poor, are respectfullyincluded.7 And it prescribes procedures bywhich stakeholders interests are articulated,shared, and advanced in evaluation, evenwhen, or perhaps especially when, they conflict.These procedures rest on three inter-relatedprinciples: inclusion, dialog, and deliberation.Inclusion means that the interests of all legiti-mate stakeholders are included in the evalua-tion. “The most basic tenet of democracy isthat all those who have legitimate, relevantinterests should be included in decisions thataffect those interests” (House & Howe, 2000,



p. 5). Dialog (among stakeholders) is offered asthe process through which the real or authenticinterests, as compared to the perceived inter-ests, of diverse stakeholders are identified.And deliberation is the rational, cognitiveprocess by which varying, even conflictingstakeholder claims are negotiated. These maybe claims of values, interpretations of evalua-tion results, or action implications. Delibera-tion means that all such claims are subject toreasoned discussion, with evidence andargument. In deliberative democratic evalua-tion thus, the evaluator’s role is crucial andchallenging, as he/she is charged with insur-ing these principles of inclusion, dialogue, anddeliberation through skillful facilitation anddiplomatic leadership.

Rationale

The epistemological rationale underlyingHouse’s ideas about deliberative democraticevaluation fundamentally involves a rejectionof the fact–value dichotomy and thus the pos-sibility of a value-free evaluative science.Instead, “we contend that evaluation incorpo-rates value judgments (even if implicitly) bothin the methodological frameworks [see alsoHouse, 1993, chapter 8 on “methodology andjustice”] and in the concepts employed, con-cepts such as ‘intelligence’ or ‘community’ or‘disadvantaged’” (House & Howe, 1999, p. 5).Also rejecting both extreme relativism (radicalconstructivism) and post modernism as viableframeworks for a value-engaged evaluationpractice, House & Howe (1999) emphasize theimportance of legitimizing values as intrinsicto evaluative knowledge claims, but also sub-jecting them to reasoned deliberation, usingappropriate rules of evidence, argument andnegotiation.

The question then becomes, what valuesshould evaluation promote? In response,House has argued for fundamental democratic

values, namely social justice and equality. Thequotation from Howe at the beginning of thischapter attests to the defensibility of democra-tic value choices. Specifically, House & Howeadvance a modified version of political-moraltheorist John Rawls’ egalitarian formulationof distributive social justice (Rawls, 1971).Rawls’ original principles of justice (a) call forequal liberties for all persons and (b) addresssocial and economic inequalities so that thegreatest benefits accrue to the least advantaged,while also attached to opportunities fairlyopen to all. This conception of justice protectsthe interests of the least advantaged by allow-ing unequal distribution of resources underconditions of “fair equality of opportunity.”Yet, recognizing that these principles excludethe “least advantaged” from defining theirown needs and negotiating for themselves thedistribution of societal goods, the revised egal-itarian position refocuses equality as a princi-ple of democratic participation (rather thanonly one of distribution, Guttmann, 1987).Equality in this view refers not just to the dis-tribution of goods but also to the status andvoice of the participants, in part to enablemeaningful participation in the democraticprocess by all.8 “Goods, along with needs, poli-cies, and practices, are investigated and nego-tiated in collaboration, with democraticdeliberation functioning as an overarchingideal” (House & Howe, 1999, p. 108; see alsoHowe, 1997). And so, deliberative democraticevaluation advances “an egalitarian … con-ception of justice that seeks to equalize powerin arriving at evaluative conclusions” regard-ing effective social programs and policies(House & Howe, 1999, p. 134).9

Major Implications for Practice,with Illustrations

Deliberative democratic evaluation importantlyaims to be “objective,” in the sense of being



impartial and unbiased, that is, equitablyinclusive of all important interests and perspec-tives. “Objective” in this theory further meansto be rational or reasoned through both themethodological canons of the discipline andthrough the interactive and argumentativeprocesses of dialog and deliberation.

Beyond this, guidelines for deliberative demo-cratic practice emphasize evaluative processesand commitments related to the major tenets ofthis theory, rather than any particular ques-tions, methods, or procedures. Specifically, 10questions are offered to guide the deliberativedemocratic evaluator: (1) Whose interests arerepresented? (2) Are major stakeholders repre-sented? (3) Are any major stakeholdersexcluded? (4) Are there serious power imbal-anced? (5) Are there procedures to controlpower imbalances? (6) How do people partici-pate in the evaluation? (7) How authentic istheir participation? (8) How involved is theirinteraction? (9) Is there reflective deliberation?(10) How considered and extensive is thedeliberation? (House & Howe, 1999, p. 113).

Instances of deliberative democratic evalua-tion in practice remain rare, perhaps because itis acknowledged to be an idealized theory(House & Howe, 1999, p. 111), though seeHowe & Ashcraft (in press) for one example. Atthe same time, many evaluators with similarpolitical commitments have both espoused andendeavored to implement particular features ofthis evaluation theory. Tineke Abma, forexample, has conducted a number of evalua-tions featuring stakeholder dialog (Abma,2001a). Some of her work suggests that con-structing narratives to represent evaluationfindings and engaging stakeholders in dialogsabout these narratives is a promising approachto meaningful dialog with authentic stake-holder participation (Abma, 2001b).

Ove Karlsson (1996) has also used dialogs inevaluation, particularly to engage stakehold-ers in developing deeper understandings ofprogram advantages and disadvantages,

especially for intended beneficiaries. Karlsson’swork indicated that a significant challenge inimplementing a meaningful, equitable dialog isthat stakeholders come to the table with differ-ential resources for participation (and see Guba& Lincoln, 1989). In this regard, CherylMacNeil (2000, 2002) has experimented withthe idea of implementing deliberative forumsfor negotiation of important evaluative findingsand action implications, in conjunction withsome advance coaching to prepare for theseforums for stakeholders with limited verbal flu-ency or limited experience in articulating theirown ideas, views, and stances.

Deliberative democratic evaluation is a chal-lenging ideal to implement because existingarrangements of power and privilege renderequitable, authentic participation by all stake-holders difficult to actualize. But, of course, thevery point of this theory is to conduct evalua-tions that help to rearrange (redistribute) powerand privilege in more just and equitable ways.

Critique

House & Howe’s deliberative democratic evalua-tion aspires to help constitute a socially just andequitable society in which all citizens activelyand authentically participate in rationale delib-eration about their common and conflictinginterests toward reasonable agreement aboutappropriate public decisions and directions. Asan important societal institution, evaluation isboth constituted by and helps to constitute thisjust and rational democracy. And the evaluatorcontributes professional methodological skillsbut more importantly facilitation in conductingauthentic dialogs and meaningful deliberationsand strong advocacy for democracy and an egal-itarian conception of justice.

This vision is acknowledged by its ownauthors as idealistic and difficult to implementwholesale in today’s democracies, with theirspecial-interest politics and sound-byte mediadomination. But, to conduct evaluation in the



absence of this kind of democratic vision is “toendorse the existing social and power arrange-ments implicitly and to evade professionalresponsibility” (House & Howe, 1999, p. 111).So, the ideal can still serve as useful guide andframework for evaluation practice.

More substantively, some critics, even withclosely allied evaluation theories (Kushner,2000; Stake, 2000) do not agree that the pro-motion of democracy is the main purpose ofevaluation and further worry about the imposi-tion of the evaluator’s own values in theprocess of judging quality, which they see as aform of advocacy and activism. And advocacyremains irreconcilable with notions ofrespectable evaluation in most evaluation com-munities. (Datta (1999, 2000) has writtenespecially thoughtfully on the intersections ofevaluation and advocacy.) Other critics, notablyArens & Schwandt (2000) express concernsthat the dialogic and deliberative strands of thistheory require further development (along linesof reciprocity, for example) lest they risk “covertdomination – a hegemonic process cloaked inpseudo-participation” (p. 333). (Similar con-cerns are raised about participatory evaluation,as noted below.) And finally, there are thosewho argue that House & Howe do not go farenough in envisioning an evaluation processwith strong potential for meaningful socialchange. As described in the sections that follow,these arguments include challenges to theexpert status and authority of the evaluatorand challenges to the assumptive frameworkwithin which evaluative knowledge is gener-ated, including the critical and actionablestrands of such a framework.

Extending Historical Legacies inIdeologically Based Evaluation:Participation and Social Critique

There are two additional major clusters of con-ceptual ideas in ideologically based evaluation

that have grown up alongside the ideas ofMacDonald and House but have drawn theirprimary inspiration from other traditions insocial research. These two clusters, which relateto stakeholder participation and empowermentand structural social critique, respectively –are interconnected and overlapping, but dis-cussed here separately, highlighting bothcommonalities and differences.

Participatory Evaluation

Originally influenced by trends and develop-ments from outside the evaluation field,notably, participatory research and then partic-ipatory action research, especially in contexts ofinternational development in the southernhemisphere (Fals-Borda, 1980; Freire, 1970;Hall, 1981), participatory approaches to evalu-ation directly engage the micro-politics ofpower by involving stakeholders in importantdecision-making roles within the evaluationprocess itself. Multiple, diverse stakeholders –most importantly, stakeholders from the leastpowerful groups – collaborate as co-evaluatorsin evaluations, often as members of an evalu-ation team. All collaborators in participatoryevaluation share authority and responsibilityfor decisions about evaluation planning (keyevaluation questions, evaluation design andmethods), implementation (data collectionand analysis), and interpretation and actionimplications. The primary intention of suchparticipation is individual and group stake-holder agency and empowerment, towards thebroader ideal of social change in the distribu-tion of power and privilege.10 Participatory“evaluation is conceived as a developmentalprocess where, through the involvement ofless powerful stakeholders in investigation,reflection, negotiation, decision making, andknowledge creations, individual participantsand power dynamics in the socioculturalmilieu are changed” (Cousins & Whitmore,1998, p. 9).



A leading theorist-practitioner of participa-tory evaluation is Elizabeth (Bessa) Whitmore,who has used her social work facilitation skillsin excellent service of participatory evaluationin varied contexts (Whitmore, 1991, 1994,1998). Many of the ideas about participatoryevaluation advanced in this section come fromWhitmore’s work. The empowerment evalua-tion theory of David Fetterman (2001) is alsopart of this tradition. Fetterman’s workextends the concept of empowerment to theideal of self-determination, so that the pri-mary purpose of evaluation becomes individ-ual and group self-determination throughevaluation participation and capacity building.And Egon Guba & Yvonna Lincoln’s (1989)fourth-generation evaluation approach is alsoconnected to this tradition (although it moreaccurately straddles the participatory andcritical/emancipatory traditions). While notan explicitly participatory approach, fourth-generation evaluation seeks authentic, local-ized constructions of program knowledgefrom multiple and diverse stakeholdersthrough a dialogic process in which the evalu-ator serves as negotiator. Legitimizing diversevoices and multiple knowledge construc-tions are core ambitions of fourth-generationevaluation.

Rationale

As noted, participatory evaluation shares sig-nificant history with the frameworks and ide-ologies underlying international development.In the last quarter of the twentieth century,multiple challenges arose to the dominantdevelopment paradigms, which were per-ceived by development workers, advocates,and others as exploitive, fostering dependency,narrowly focused on macro-economics, anddivorced from urgent local problems ofhuman suffering due to poverty, lack ofeducation, and disease. Development workers

and researchers/evaluators alike found wel-come responses to their disillusion with domi-nant development paradigms in the liberatoryideas of adult educator Paolo Freire (1970),the action-oriented ideas of action researcherslike Orlando Fals-Borda (1980) and Budd Hall(1981), and the participatory research ideasof Rajesh Tandon (1981) and, in the US, JohnGaventa (1980), among others. Collectively,these ideas called for people’s own participa-tion in the construction of knowledge regard-ing their own lives, including the experiencesand effects of development interventions ontheir lives. Not only are people legitimateauthors of their own life stories, but enablingsuch authorship can itself generate greaterefficacy and empowerment among those tar-geted by development efforts. Moreover, theknowledge to be constructed should be “action-able” knowledge with intrinsic action implica-tions and directions, in contrast to abstract orconceptual knowledge that requires separateapplication to practice. In these ways, partici-patory evaluation of development efforts canpromote values of respect and equity, serveempowerment aims, and thereby encouragedevelopment programs to do the same.

Beyond these specific political and philo-sophical bases, participatory evaluationshares with critical evaluation (discussed inthe next section) justifications in broader radi-cal and emancipatory traditions of philosophyand ideology. These include the theories ofMarx, Gramsci, Habermas, and other criticalsocial scientists. From these theories, partici-patory evaluators understand that “workingto achieve emancipation requires more than atextured criticism of oppressive structures.Emancipation demands action and radicalchange firmly grounded in, but not obfuscatedby, theory. Activity gives meaning to thetheory, and the melding of both in praxis givesinquiry not only a political but a moral andethical significance.” Further, participatory



evaluators “begin, continue, and end with theindividuals whose lives are at the center of theevaluation … . The ethical starting point isequity in research relationships … . [by whichindividuals can] work collectively towardunderstanding of one’s self, one’s place in theworld, the societal conditions that permitchange” (Brisolera, 1998, pp. 30–31).

So, central to conceptualizations of partici-patory evaluation is the importance of broad-ening the bases and control of knowledgeproduction to include the people who are theobjects of evaluation, thereby facilitating theirempowerment.11 People are empowered, thatis, “through participation in the process ofconstructing and respecting their own knowl-edge (based on Freire’s notion of ‘conscientiza-tion’) and through their understanding of theconnections among knowledge, power, andcontrol” (Cousins & Whitmore, 1998, p. 8).(See Brisolera, 1998, for further discussion ofthese sociopolitical and philosophical roots ofparticipatory evaluation.)

Implications for Practice, with Illustrations

“It’s the process that counts” (Whitmore, 1991).That is, what matters most in participatory eval-uation practice is the process and experience ofstakeholder participation and its enablement ofempowerment. This process is intrinsically val-ued for its empowerment potential, over andabove the evaluative results and reports. Mostimportantly, “participatory evaluation is a set ofprinciples and a process, not a set of tools ortechniques (Burke, 1998, p. 55).

Given its connections to the vast enterpriseof international development, participatoryevaluation has a rich practical history, in con-trast to other democratically oriented evalua-tive approaches. Accompanying this historyare many thoughtful reflections on thepromises and challenges of participatoryevaluation in the field, reflections that honor

its commitment to principles and process.Samples of these follow, as illustrations ofmany of the major themes in this literature.

Reflecting on multiple participatory evalu-ations of local community-based programs(primarily in the fields of education, youthdevelopment and child care provision), Greene(1997) asserted that “in its ideal form, partici-patory evaluation intentionally involves alllegitimate stakeholder interests in a collabora-tive, dialogic inquiry process that enables theconstruction of contextually meaningfulknowledge, that engenders the personal andstructural capacity to act on that knowledge,and that seeks action that contributes todemocratizing social change” (p. 174). Greene’sattention to the consequentialist character ofmeaningful stakeholder participation isechoed by many, as action is directly con-nected to the empowerment agenda of thisapproach. Burke (1998), for example, assertedthat a participatory evaluation process “must …be useful to the program’s end users … [and]rooted in [their] concerns, issues, and prob-lems” (p. 44). And Guba & Lincoln’s fourth-generation evaluation approach is orientedaround the “concerns, claims, and issues” ofparticipating stakeholders.

More broadly in the domain of participa-tory evaluation of development assistance,Reiben (1996) offered a set of criteria for dis-tinguishing genuine from more token forms ofparticipation (though see Gregory, 2000 for acritique of these ideas):

1. Stakeholders must have an active role assubjects in the evaluation process, that is,they identify information needs and designthe terms of reference, rather than have amerely passive role as objects of meresources of data.

2. As it is practically impossible to actively includeall stakeholders in the evaluation process,at least the representatives of beneficiaries,



project field staff, project management, andthe donor should participate.

3. Stakeholders should participate in at leastthree stages of the evaluation process:designing terms of reference, interpretingdata, and using evaluation information.

These criteria can be readily mapped ontoCousins & Whitmore’s (1998) conceptualemphasis on depth of stakeholder participation,range of stakeholders who participate, anddegree of stakeholder vs. evaluator control ofthe evaluation process, respectively, as criticaldimensions of participatory evaluation.

Regarding who should actually participatein participatory evaluation, Mathison (1996)has challenged the field to consider formingevaluation teams with varying numbers ofstakeholders that correspond to each group’soverall size in the context at hand. For example,a participatory evaluation in an educationalcontext would have many more students andparents than teachers or administrators.Mathison also discusses the problematic natureof asking stakeholders to represent the views oftheir group, absent any formal process for suchrepresentation (see also Gregory, 2000). Inmost contexts, participatory evaluators havecome to accept that participating stakeholderscan only represent themselves, rather than theviews of the group to which they belong.12

Concerns about who participates are acommon practical challenge in participatoryevaluation. Far too often, participatory evalu-ations are initiated with but one or two tokenparticipants from the beneficiary group, eventhough this is the group directly targeted forempowerment. The reasons for this challengeare complex and contextual and often includeissues of access, time, location, familiarity andcomfort, language and verbal fluency, andoverall understanding. Even when participa-tion is framed in ways more familiar andcomfortable to beneficiaries, their participation

(and consequent empowerment) are the evalu-ator’s agenda, not theirs. Seigart’s (1999)extraordinary but ultimately unsuccessfulefforts to recruit beneficiaries (parents) for herparticipatory evaluation of a school-basedhealth clinic well illustrate these challenges.

Other practical challenges to participatoryevaluation include the facilitation skillsneeded by the evaluator, possible dissonancebetween the values intrinsic to participatoryevaluation and the values embodied in a givenprogram and its context (Coghlan, 1998;VanderPlaat, 1995), the time demands onprogram staff for participation, and the chal-lenges of conducting an evaluation thatrequires active staff and beneficiary participa-tion in an organization that lacks an evalua-tive culture (Brisolera, 1997) or in a localcontext with conflicting demands from anational evaluation (Biott & Cook, 2000).

Critique

Participatory evaluation in theory aspires to aparticipatory form of democracy, in whichmeaningful participation becomes constitutiveof genuine citizenship, both privileges andresponsibilities therein (Barber, 1984). Thereare significant practical challenges to imple-menting meaningful and effective participatoryevaluation, probably because it is more an ori-entation and commitment to a set of principlesthan a clearly defined set of procedures. But,even in theory, its reach is limited. Participatoryevaluation concentrates on individual empow-erment or on changing individuals, primarilywithin a time-limited evaluation process,with little planned carryover to issues of voiceand power outside the evaluation or after itis over, or few concentrated efforts to changeinstitutions and practices of decision-making.Meanings of empowerment also remainunclear and often unrealized or more imposedthan authentically enabled.



At the same time, participatory evaluationimportantly legitimizes multiple sources ofknowledge and multiple and diverse knowl-edge producers. Participatory evaluation fur-ther knits democratic values into the veryfabric of evaluative work, positioning evalua-tion as itself a democratizing practice servingthe well-being of those least advantaged in ourcontemporary societies.

Critical Evaluation

Loosely clustered under the label of “criticalevaluation” are several other ideologically oriented evaluation approaches that seek toengage the macro-politics of power by focusingevaluation (content and process) around soci-etal critique. These evaluation approaches areinformed by some form of a critical socialscience epistemology (Fay, 1987) and endeavorto conduct social analyses that reveal struc-tural injustices and to generate actions thatcan redress such injustices. Central to all ofthese approaches are principles of collabora-tion, critical theorizing and reflection, andpolitical action with a transformative or eman-cipatory intent. Critical forms share consider-able philosophical and ideological ground withparticipatory evaluation. Among the key dif-ferences are that critical evaluation is rela-tively more oriented toward macro structuralissues, compared to the micro emphasis ofparticipatory evaluation; more attentive to theactual substance of the evaluation, comparedto participatory evaluation’s emphasis onthe process; and politically more radical –compare the agendas of empowerment andemancipation.

Critical forms of evaluation fully situate thesocial and educational practices being evalu-ated (as well as evaluation itself as a socialpractice) in their contested socio-cultural-political contexts. Rejecting the atheoreticalidea that “practice exists as a commodity on its

own that can be separated out for study,” criticalevaluators see practice as “constructed withinlegislative, policy, and funding processesand … shaped through dimensions of class,gender, race, age, sexuality and disability”(Everitt, 1996, p. 174).

Practice is also “continuously negotiated byall those involved [and] people’s interests inpractice … constitute political interests and[thus] may be conceptualized in terms of wholoses and who gains” (Everitt, 1996, p. 178).This perspective disrupts taken-for-grantedways of understanding practice and opens thedoor to evaluative scrutiny of broader societal,especially political structures and discourses,alongside programmatic practices.

The program, project, and practices to be evalu-ated … [are] understood as being constructedthrough discourses, which in turn need to beunderstood in terms of power: whose interestsdo they serve? … Evaluation becomes concernedwith contributing to the deconstruction of dis-courses that serve consistently to render someless powerful than others, and some ways ofknowing the world more credible than others.(Everitt, 1996, p. 182)

Critical forms of evaluation are multi perspec-tival, respecting a diversity of stakeholderinterests and experiences, but they are notcompletely relativistic. Rather, evaluativejudgments of merit or “goodness” are accom-plished through processes of stakeholderengagement, dialog, and critical reflection aboutthe practices being evaluated – intertwinedwith critical theorizing about how power,opportunity, and privilege are constituted, dis-tributed, and maintained in the context athand (often by discourses outside the context).The evaluator’s role is to facilitate stakeholderengagement, dialog, and reflection and, perhapsmost importantly, to contribute the lenses andsubstance of critical theory. Evaluative judg-ments in critical evaluation thereby aim to be



transformative, deconstructing inequitabledistributions of power and reconstructingthem more fairly and justly.

Snapshots of four examples of criticalapproaches to evaluation are offered next,as these abstract ideas gain clarity throughspecificity.

Examples of Critical Theories of Evaluation

The critical evaluation theory of Everitt &Hardiker (1996) offers several principles forcritical evaluation practice in service of differ-entiating judgments of good, poor, and evencorrupt practice. These principles, include, forexample, “scepticism of rational-technicalmodes of practice” and “removal of the ‘other-ness’ that may be attributed to those lower inthe hierarchy, to users and to those relativelypowerless” (Everitt, 1996, pp. 180, 181). Inpractice, this theory relies primarily on reflec-tive and dialogic methods for generating evi-dence, accepts that such evidence cannotreveal the one “truth,” and thus turns to politi-cal considerations as the basis for makingevaluative judgments. “If there are no centersof truth … then there are only working truthsand relative truths. The full participation ofthose involved in making decisions about whatis going on and what should be done is theonly way to define non-oppressive, culturallypertinent truths and working, practical judg-ments” (Howe, 1994, p. 525, cited in Everitt,1996, p. 186). And so, in this theory, a prac-tice is judged as “good” if it is rooted in devel-opment processes and needs identificationthat themselves are democratic and fair –“having equality as [their] underpinningvalue and goal” – and if the practice serves tobring about equality, “enabling all people, irre-spective of their sex, ethnicity, age, economicposition, social class and disability, to flourishand enjoy human well-being” (Everitt, 1996,p. 186). A practice is judged “good enough” if it

is moving in these directions, “poor” if it makesno attempt to meet criteria of democraticequality, and “corrupt” if it is anti-democraticand unfair.

The communicative evaluation theory ofNiemi & Kemmis (1999) is rooted in Habermas’theory of communicative action, specifically thecharacter and purpose of public discoursewithin democracies, and in traditions of partici-patory action research (PAR) (Kemmis &McTaggart, 2000). Communicative evaluationaspires to help establish and nurture democra-tic, public conversational spaces in which “citi-zens can come together to debate and deliberate,creating discourses that may be critical of thestate and that have the potential for contribut-ing to the development of new or differentpublic policies or programs” (Ryan, 2004,p. 451). In practice, communicative evaluationestablishes a local site for stakeholder conversa-tion and practical deliberation about locallyimportant program issues. In addition, drawingfrom PAR traditions, communicative evaluationemphasizes joint ownership of the evaluation,collaboration in evaluation implementation,critical analysis and reflection, and an actionorientation. The communicative evaluator’srole is one of enabling and supporting stake-holder conversation and reflection on action.

Merten’s inclusive evaluation theory (Mertens,1999, 2003) is rooted in a “transformative-emancipatory” paradigm and is especiallyconcerned with discrimination, oppression,and other injustices suffered by people in mar-ginalized groups. Inclusive evaluation inten-tionally seeks to include such people in theevaluation process and to focus key evaluationquestions, and thus designs and methods,around their experiences of injustice. Forexample, an evaluation of an educationalcurriculum would probe the ways in whichgender, age, class, race, ethnicity, sexual pref-erence, and disability status were portrayed inthe curriculum, with an eye to discriminatory



portrayals and another eye on possibilities forchange. Much like House & Howe’s delibera-tive approach, Mertens’ inclusive evaluationstrives for objectivity, defined as lack of biasand “achieved by inclusion of all relevantstakeholders in a way that authentic andaccurate representations of their viewpointsare considered” (Mertens, 2003, p. 95). Alsolike House & Howe, the inclusive evaluatorretains authority and responsibility for ensur-ing that “quality evaluation is planned, con-ducted, and used” (Mertens, 2003, p. 104).

Finally, feminist perspectives on evaluationdraw their inspiration from feminist theoriesand feminist politics (Seigart & Brisolera,2002) and characteristically have two majoremphases. First, a feminist lens is centeredon the well-being of girls and women, or asexpressed in one of Ward’s (2002) key princi-ples for conducting feminist evaluation, “Placewomen and their material realities at thecenter of evaluation planning and analysis[and] … understand the problem context froma feminist perspective” (pp. 44, 47). Thismeans that whether or not the program beingevaluated seeks specifically to benefit females,a feminist evaluation will ask if and how itdoes so, or not (much like Mertens’ concen-trated focus on people from marginalizedgroups). Second, consistent with all ideologi-cally oriented evaluation approaches, feministevaluation attends seriously to the evaluationprocess but gives it a particular feminist cast. Afeminist evaluation process is self-consciouslycollaborative and reciprocal, trusting andcaring, and ideally conducted with humilityand grace, as these are strong feminist values(Beardsley & Miller, 1992; Ward, 1992).

Critique

Critical approaches to evaluation aspire toengender more participatory and deliberativeforms of democratic decision-making, through

a process of assisting people from oppressedgroups to realize, understand, and activelyseek to change the historical conditions oftheir oppression. As such, critical evaluation issubject to critiques similar to those offered fordemocratic and participatory evaluationapproaches, including questions of feasibilityand acceptance. Moreover, critical approachesto evaluation, more than the others, impose aparticular set of values onto the evaluationcontext and invite stakeholders to engage withthese values – those of structural critique andemancipation. Justification for this impositionremains widely sought. In addition, criticalevaluation primarily offers theoretical lensesthrough which existing ways of setting publicpolicies and designing ameliorative programsfor those in need are soundly challenged.Alternatives are not as readily offered in theseapproaches.

At the same time, the critical voice is anessential one. It guards against satisficing andcomplacency.

Contemporary Developments

Finally, two ideologically oriented contemporaryevaluation theories-in-the-making deserve briefmention. Both attend directly to issues of cul-ture and, relatedly, race and ethnicity, andboth seek to supersede historical legacies ofenslavement and colonization with theoriesrooted in once-dominated cultures. The first isprimarily relevant to racial and ethnic minori-ties in the US, and the second to indigenouspeoples in North America and the Pacific.

Culturally and ContextuallyResponsive Evaluation

A group of primarily African American schol-ars in the US has been developing an approachto evaluation that is culturally and contextually



responsive, meaning, consonant with theways of knowing and ways of being particularto a given minority community in the US. Thisgroup includes Henry Frierson (Frierson,Hood, & Hughes, 2002), Rodney Hopson(2000, 2001), Stafford Hood (1998, 2001),and Veronica Thomas and Gerunda Hughes(Thomas & Stevens, 2004a).13

Consistent with other race-conscioustheories, culturally and contextually respon-sive evaluation begins from a standpoint thatsummarily rejects deficit thinking andembraces starting points that emphasize the“strengths” and “assets” of underserved com-munities. Further, problem identification anddefinition must be located within the minoritycommunity to be served, as racist and discrim-inatory habits of mind persist in the largersociety, despite considerable legal progress(Madison, 1992). What constitutes a social“problem” from the vantage point of thedominant society (say, “at risk” youth) is likelyexperienced and understood quite differentlyfrom within the communities where suchyouth live (say, youth without meaningfuleducation, recreation or employment oppor-tunities). For similar reasons, the characterand logic (or theory) of an interventiondesigned to ameliorate an identified problemmust be grounded in the culture of the contextto be served (Madison, 1992). Just who can bea culturally responsive evaluator is an addi-tional issue; significant shared life experiencewith those being evaluated is an essentialqualification, argue many (Hood, 1998,2001; Thomas, 2004).

To date, the most comprehensive approachto culturally and contextually responsive eval-uation has been developed by the group atHoward University. The approach was devel-oped in tandem with the Talent DevelopmentModel of (Urban) School Reform (TD), whichis a major project of the Howard UniversityCenter for Research on the Education of

Students Placed at Risk. The TD program itselfis rooted in cultural responsiveness andrespect, blending elements from critical peda-gogy, school restructuring ideas, and researchon the effective education of children of color(Thomas, 2004).

The TD evaluation approach … seeks to be prac-tical, useful, formative, and empowering for theindividuals being served by TD evaluations andto give voice to persons whose perspectives areoften ignored, minimized, or rejected in urbanschool settings. … [Moreover, the TD] evaluationframework seeks to reposition evaluation in low-income urban contexts as accountable not onlyfor producing accurate and relevant informa-tion on the program being evaluated, but alsofor enabling and contributing to the program’ssocial betterment and social justice intentions.(Thomas & Stevens, 2004b, p. 1)

[Moreover] standards of evidence for evalua-tions of TD projects encompass both scientific-methodological and political-activist criteria.(Thomas, 2004, p. 6)

The TD evaluation approach rests on five inter-related major principles (from Thomas, 2004):

1. Key stakeholders, including students,parents, teachers, and other school personnel,are authentically engaged throughout the eval-uation process. “TD evaluators enter theurban school contexts being studied gently,respectfully, and with a willingness to listenand learn in order to plan and implement eval-uations better” (p. 8).

2. The substance and process of the evalua-tion are co-constructed. “Co-construction isdefined as evaluators’ collaborating and forminggenuine partnerships with key urban schoolstakeholder groups … and TD project designersand implementers in order to conceptualize,implement, and evaluate school reform efforts ina manner that is responsive to the school’s con-text … . Co-construction seeks to democratize



the evaluation process by lessening the implicit,and sometimes explicit, power dynamicsbetween evaluators and project stakeholders”(p. 9). Distinctively, as co-construction suggests,the TD evaluator is an engaged member of alarger evaluation team, all of whom areaccountable to the aims of the intervention.

3. TD evaluation attends meaningfully toculture and context, where context refers to“the combination of factors (including cul-ture) accompanying the implementation andevaluation of a project that might influenceits results, including geographical location,timing, political and social climate, [and] eco-nomic conditions” (p. 11). In this regard, TDevaluators must be culturally competent,preferably sharing the same cultural back-ground as those being studied. Having ashared cultural life experience affords greatersensitivity to and understanding of relevantcontextual issues.

4. “TD evaluations embrace the underlyingphilosophy of responsiveness found in the litera-ture,” notably, the importance of “respecting,honoring, attending to, and representing stake-holders’ perspectives” (p. 13, emphasis added).

5. Finally, TD evaluators use triangulationof perspectives in multiple ways, includingtriangulation of investigators, methods andmeasures, target people, and analyses.Triangulation is valued in this approach pri-marily for its inclusiveness of perspective.

Evaluation by and forIndigenous Peoples

Linda Tuhiwai Smith’s 1999 book entitledDecolonizing Methodologies: Research and indige-nous peoples is continuing its significant influ-ence well into the twenty-first century. Smith isa Maori educational scholar from New Zealand.“The ways in which scientific research is impli-cated in the worst excesses of colonialism

remains a powerful remembered history formany of the world’s colonized peoples” (Smith,1999, p. 1). In this book, Smith reclaims themeanings of research and knowledge, as well asthe right to be a knower, for indigenous peoples.She does so with a relentless critique of Westernresearch on indigenous peoples, followed by anarticulation of an indigenous research visionand agenda. This agenda is centered around thegoal of indigenous peoples’ self-determinationand uses processes of decolonization, healing,mobilization, and transformation, processes“which connect, inform, and clarify the ten-sions between the local, the regional, and theglobal” (p. 116). Using the metaphor of oceans,there are also three tides in this researchagenda – survival (of peoples, languages, spiri-tual practices, art), recovery (of land, indige-nous rights, histories), and development (ofcommunities, economic opportunities, pride) –representing the ebb and flow of conditionsand states of movement on the way to self-determination.

Smith further presents Maori approaches toresearch, called Kaupapa Maori research.“Kaupapa Maori, however, does not mean thesame as Maori knowledge and epistemology.The concept of kaupapa implies a way of fram-ing and structuring how we think about …ideas and practices … . Kaupapa Maori isa ‘conceptualization of Maori knowledge’”(p. 188). Kaupapa Maori research is informedand guided by Maori philosophy and world-view. It “takes for granted the validity andlegitimacy of Maori [and] the importance ofMaori language and culture, [and it] isconcerned with the struggle for autonomyover our own colonial well being” (p. 185).Kaupapa Maori researchers disagree whetheror not non-Maoris can conduct meaningfulKaupapa Maori research.

Reclamation of native epistemologiesas frameworks for social and educationalresearch is happening in other locales around



the globe (Cajete, 2000). And evaluators arebeginning to use these to develop distinctivelyindigenous ways of thinking about and con-ducting evaluation. For example in the US,Joan La France is contributing to efforts toarticulate the meanings of culturally compe-tent evaluation “in Indian country,” whichinclude “the importance of understanding theimplications of sovereignty … , the significanceof an emerging indigenous framework for eval-uation, Indian self-determination in settingthe research and evaluation agenda, and …particular methodological approaches”(LaFrance, 2004, p. 39). Candidates for ele-ments of an indigenous framework for evalua-tion include the importance of trust (ratherthan evidence), community, holistic thinking,and from Cajete (2000), a profound “sense ofplace” and being part of the web of the nat-ural world. In New Zealand, Fiona Cram andcolleagues (Cram, Ormond, & Carter, 2004;Cram et al., 2002) are pursuing conceptualand political questions of research ethics forresearch and evaluation with Maori people, aswell as endeavoring to apply Kaupapa Maoriin evaluation studies, “as a basis to explore apolitical, social and cultural analysis of [forexample] domestic violence within the contextof [domestic violence] programmes” deliveredto Maori people by Maori providers (Cramet al., 2002, p. iii). And across the Pacific,indigenous evaluators from New Zealand andHawaii met together in 2003–2004 to beginto share common visions and possibilities(http://www.kohalacenter.org/ws_pono040116.html and http://www.ksbe.edu/pase/researchproj-evalhui.php).

Reprise

The landscape of democratically orientedapproaches to evaluation is richly textured. Itis rooted in some very important ideas about

knowledge production, legitimacy, and owner-ship; about the character and role of values inevaluative knowledge generation; and aboutthe connections between evaluation anddemocratic principles, practices, and institu-tions. And it is populated today by equallyimportant ideas related to participation andempowerment, dialog and deliberation, publicspheres for communication, emancipationand social critique, cultural responsiveness,and self-determination.

Table 5.1 presents a summary of key ideasfrom each genre of democratically orientedevaluation discussed in this chapter, specificallyas related to philosophical framework, views ofdemocracy, and macro and micro positions forevaluation in society. As with any simplifiedpresentation, this table omits important ideasand suggests sharp lines demarcating one genrefrom another, when actually there are manyshared concepts and commitments amongthem. This table also represents my own sense-making of these complex ideas; others mayhave differing interpretations. Importantsources in the construction of this table wereHanberger (2001) and Ryan (2004).

The meanings of the various kinds ofdemocracy featured in the table are as follows.In a representative democracy, ordinary citi-zens participate primarily by electing elites,who are empowered to carry out and areresponsible for public decision-making. A par-ticipatory democracy emphasizes the impor-tance of people’s direct participation inactivities and decisions that affect their lives.Such participation is viewed as constitutive ofdemocratic citizenship. And a deliberativedemocracy emphasizes the importance ofcommunication, dialog, deliberation aboutpublic issues among free and equal citizens.This democracy features a commitment toreasoned discourse on matters of public policyin spaces free from domination. The differenttraditions of democratically oriented evaluation



variously serve one or more of thesedifferent conceptualizations of democracy,though with a shared agenda of making theprocesses and results of public decision-making more inclusive of multiple stakeholderinterests and values, more broadly based onmultiple stakeholder knowledge, and therebymore likely to provide an equitable and justdistribution of goods and services.

Clearly, there are challenges to the premisesof democratically oriented evaluation. Manyevaluators and evaluation commissioners, espe-cially in the Anglo-American tradition and espe-cially with today’s infatuation with technocraticideas about public accountability, reject out ofhand a value-committed stance for evaluation.Instead, these critics believe that standards ofimpartial objectivity, attained via excellence ofmethod, are needed to support contemporaryaccountability concepts like performanceindicators, results-based management, andevidence-based decision making, all part of thecurrent “climate of control” (McKee & Stake,2002). Also problematic are the meanings ofsuch lofty ideals as democracy itself, inclu-sion, social justice, equity, empowerment, self-determination, along with the meanings ofsuch processes as participation, dialog, deliber-ation, and cultural responsiveness. These con-cepts must be specifically and contextuallydefined if democratic approaches to evaluationare to gain any practically meaningful purchasein the field. They currently offer inspiration,much of which remains unrealized in practice.

There are also substantial practical chal-lenges to the implementation of democrati-cally oriented evaluation. These includechallenges to the acceptability of ideologicallybased evaluation among many stakeholdersand to the feasibility of its implementation inparticular evaluation contexts. Regardingacceptability, persuading evaluation commis-sioners that it is indeed in their interest toshare power more equitably with program

staff and especially beneficiaries is a signifi-cant practical hurdle in most evaluation con-texts. McKee & Stake call this “paying fortrouble making” (2002, p. 134). Persuadingthe disempowered and marginalized people ina given context that it is also in their interest to(a) participate in the evaluation and (b) sharethis participatory space with program staff,managers, and others from more powerfulgroups also poses substantial hurdles in mostevaluation contexts. The intensity of the eval-uation process in these evaluation approachespresents special challenges of practicabilityand feasibility. For example, conducting ameaningful participatory or deliberative eval-uation within a large-scale, multisite study orwithin a time-limited study would be difficultat best and likely of limited democratizingvalue. And even with interested stakeholdersand citizens, participation in an evaluationstudy has to compete with multiple demandsand opportunities already present in their livesfor professional development, personal com-mitment, and civic engagement.

Additional practical challenges for imple-menting democratically oriented evaluationapproaches include meaningfully operational-izing lofty democratic ideals and commitmentsin specific contexts, developing facilitation anddialogic skills in evaluators, creating the timeand spaces needed for messy processes likeparticipation and deliberation, and maintain-ing methodological excellence while advocat-ing for democratic ideals.

Yet, at the end of the day, democraticallyoriented approaches to evaluation offer con-siderable promise. They are anchored in a pro-found acceptance of the intertwinement ofvalues with facts in evaluative knowledgeclaims and the concomitant understandingthat all evaluation is interested evaluation,serving some interests but not others. Theyare also anchored in turn to democratic valuesas the most defensible interests to be served



in evaluation – democratic values enacted inevaluations designed to serve the public rightto know or citizen participation in decisionsthat affect their lives or reasoned deliberationamong diverse stakeholders regarding impor-tant policy directions or self-determinationamong oppressed peoples. These visions posi-tion evaluation as itself a democratizing socialpractice, but not in ways that exclude tradi-tional evaluation audiences like policy-makersand program staff nor reject traditional evalu-ation roles like gathering credible informationuseful for program improvement or organiza-tional learning. For democratically orientedevaluation is an inclusive practice, that is dis-tinctive not for its technical methodology, butrather for its societal location in service ofdemocratic ideals and its concomitant evalua-tor role as a facilitator and advocate of democ-ratic engagement through evaluation.

Notes

1 It should also be acknowledged that many if notmost evaluation theorists position their work inservice to an open and rational democratic society.These many other theorists characteristically con-ceptualize evaluation as providing impartial empiri-cal evidence to help public officials make informedand fair decisions about effective policies and pro-grams. As such, they position evaluation on the side-lines of democratic decision-making. What isdistinctive about the theorists reviewed in this chap-ter is their positioning of evaluation as inherentlyand inevitably entangled with and constitutive of thepolitics and values of such decision-making.

2 The early roots of Lee Cronbach’s vision of aneducational role for evaluation are evident here(Cronbach & associates, 1980).

3 In fact, the turn to the case study precededMacDonald’s articulation of democratic evaluation.The 1972 “Cambridge Manifesto”, drawn up by agathering of evaluators at Cambridge University,sought explicitly to legitimize interpretive methodsfor evaluation and to anchor evaluative work in apublic service obligation (see McKee & Stake,2002). MacDonald was joined in this turn to case

studies and other qualitative methods for evaluationby a number of prominent evaluation theorists,including in the UK Stephen Kemmis, HelenSimons, David Hamilton and Malcolm Parlett, andin the US Robert Stake, Louis Smith, Elliot Eisner,Egon Guba, Yvonna Lincoln, and Ernest House.During this heady, formative era in evaluation, theCentre for Applied Research in Education (CARE) atthe British University of East Anglia and the Centerfor Instructional Research and CurriculumEvaluation (CIRCE) at the American University ofIllinois both served as vital sources of energy andcreativity for new developments in evaluationtheory. Many of these evaluation scholars gathereda number of times at CARE and CIRCE to shareideas and support, thus catalyzing the rich andhighly influential body of work generated duringthis era.

4 Examples of these principles include: “No par-ticipant will have unilateral right or power of vetoover the content of the report.” “The perspectivesof all participants and interested observers have aright to be considered in the evaluation.” “Theprocess of negotiation of accounts will, where nec-essary, be phased to protect participants from theconsequences of one-way information flow. Parts ofa report may first be negotiated with relevant indi-viduals who could be disadvantaged if the reportwere negotiated as whole with all participants.”(McTaggart, 1991, pp. 10–11.)

5 At this time, House maintained that Americansocial scientists had been especially reluctant to“recognize social classes as enduring causal entitiesthat influence life chances in US society” (House,1993, p. 124), while considerable progress hadbeen made in recognizing gender, race, and ethnic-ity (and other historical markers of disadvantageand discrimination). House also rejected the liber-tarian and utilitarian forms of distributive justicewithin liberal democratic thought.

6 This pluralist model underlay the emergenceof stakeholder-based evaluation (Gold, 1983),which explicitly sanctioned the importance ofmultiple stakeholder perspectives in evaluation,although gave little guidance for the resolution ofconflicts among diverse groups (Weiss, 1983).

7 In this theory, “an interest is anything con-ducive to the achievement of an individual’s wants,needs, or purposes, and a need is anything neces-sary to the survival or well-being of an individual.To be free is to know one’s interest; or to possess theability and resources, or the power and opportunity,



to act toward those interests; and to be disposedto do so” (House, 1993, p. 125).

8 House & Howe’s (1999) arguments in favor ofa deliberative, participatory vision of democracyalso include arguments against views they labeltechnocratic (stripped of values), emotivist (valuesdetermined by non-rational means), preferential/utilitarian (all preferences maximized), hyper-egali-tarian (all views count the same), and hyper-pluralist(diversity more desirable than consensus).

9 The meaning of “effectiveness” here is surelyconsonant with the deliberative, participatory con-ceptualization of justice that frames this evaluationapproach – so that a “good” program is one thatadvances the well-being of the least advantaged andone in which the least advantaged themselves equi-tably participate in program definitions, decisions,and directions – although this is not explicitlystated.

10 This discussion focuses on “transformativeparticipatory evaluation” and excludes “practicalparticipatory evaluation” (Cousins & Whitmore,1998). Participation in the latter genre of partici-patory evaluation is instrumental, designed toenhance evaluation utilization, rather than moti-vated by ideological agenda. As ideologicallyoriented evaluation approaches are the focus of thischapter, utilization-oriented participatory evalua-tion will not be discussed. Similarly, because of itslargely instrumental rationale, the early stake-holder model of evaluation (Gold, 1983) is alsoexcluded from this discussion (but see Chapters 3,12, and 15, this volume, for discussions on stake-holder participation in service of utilization).

11 MacDonald sought to broaden the owner-ship of and access to the information generated inan evaluation. Participatory evaluation seeks tobroaden actual authorship of this information.

12 This challenge of representation is relevant to alldemocratically oriented evaluation approaches whichrest on stakeholder participation or engagement.

References

Abma, T. A. (Guest Editor). (2001a). Special issue:Dialogue in evaluation. Evaluation, 7(2).

Abma, T. A. (2001b). Evaluating palliative care:Facilitating reflexive dialogues about anambiguous concept. Medicine, Health Care, andPhilosophy, 4, 261–276.

Arens, S. A. & Schwandt, T. A. (2000). Review ofValues in evaluation and social research, by E. R.House & K. R. Howe. Evaluation and ProgramPlanning, 23(3), 331–333.

Barber, B. (1984). Strong Democracy: Participatorypolitics for a new age. Berkeley, CA: University ofCalifornia Press.

Beardsley, R. M. & Miller, M. H. (1992). Revisioningthe process: A case study in feminist programevaluation. In D. Seigart & S. Brisolera (eds)Feminist Evaluation: Exploration and experiences.New Directions for Evaluation, 96 (pp. 57–70).San Francisco: Jossey-Bass.

Biott, C. & Cook, T. (2000). Local evaluation in anational Early Years Excellence Centres PilotProgramme: Integrating performance manage-ment and participatory evaluation. Evaluation,6(4), 399–413.

Brisolera, S. A. (1997). A critical examination ofparticipatory evaluation through the lens ofAdelante, a women’s microenterprise develop-ment organization in Costa Rica. Unpublisheddoctoral dissertation. Department of HumanService Studies, Cornell University, Ithaca NY.

Brisolera, S. A. (1998). The history of participatoryevaluation and current debates in the field. InE. Whitmore (ed.) Understanding and PracticingParticipatory Evaluation. New Directions forEvaluation, No. 80 (pp. 25–41). San Francisco:Jossey-Bass.

Burke, B. (1998). Evaluating for change:Reflections on a participatory methodology. InE. Whitmore (ed.) Understanding and PracticingParticipatory Evaluation. New Directions forEvaluation No. 80 (pp. 43–56). San Francisco:Jossey-Bass.

Cajete, G. (2000). Natural Science: Natural laws ofinterdependence. Santa Fe, NM: Clear Light.

Coghlan, A. (1998). Empowerment-oriented evalu-ation: Incorporating participatory evaluationmethods to empower Ugandan communities toprevent HIV/AIDS. Unpublished doctoral dis-sertation. Department of Human ServiceStudies, Cornell University, Ithaca, NY.

Cousins, J. B. & Whitmore, E. (1998). Framing par-ticipatory evaluation. In E. Whitmore (ed.)Understanding and Practicing ParticipatoryEvaluation. New Directions for Evaluation, No. 80(pp. 5–23). San Francisco: Jossey-Bass.

Cram, F., Ormond, A., & Carter, L. (2004).Researching our relations: Reflections onethics and marginalization. Paper presented at



the Kamehameha Schools 2004 Conferenceon Hawaiian Well-Being. Hawaii.

Cram, F., Pihama, L., Jenkins, K., & Karehana, M.(2002). Evaluation of programmes for Maori adultprotected persons under the Domestic Violence Act1995. Evaluation report prepared for theMinistry of Justice, Wellington, New Zealand.

Cronbach, L. J. & associates (1980). Toward Reformof Program Evaluation. San Francisco: Jossey-Bass.

Datta, L-E. (1999). The ethics of evaluation neu-trality and fairness. In J. L. Fitzpatrick &M. Morris (eds) Current and Emerging EthicalChallenges in Evaluation. New Directions forEvaluation, No. 82 (pp. 77–88). San Francisco:Jossey-Bass.

Datta, L-E. (2000). Seriously seeking fairness:Strategies for crafting non-partisan evaluationin a partisan world. American Journal ofEvaluation, 21, 1–14.

Everitt, A. (1996). Developing critical evaluation.Evaluation, 2(2), 173–188.

Everitt, A. & Hardiker, P. (1996). Evaluating for GoodPractice. Basingstoke: Macmillan.

Fals-Borda, O. (1980). Science and the commonpeople. Paper presented at the InternationalForum on Participatory Research, Ljubljana,Yugoslavia.

Fay, B. (1987). Critical Social Science. Ithaca NY:Cornell University Press.

Fetterman, D. M. (2001). Foundations of Empower-ment Evaluation. Thousand Oaks, CA: Sage.

Freire, P. (1970). Pedagogy of the Oppressed. New York:Seabury Press.

Frierson, H. T., Hood, S., & Hughes, G. (2002).Strategies that address culturally responsiveevaluation. In The 2002 User-Friendly Handbookfor Project Evaluation. Arlington VA: NationalScience Foundation.

Gaventa, J. (1980). Power and Powerlessness:Quiescence and rebellion in an Appalachian valley.Chicago: University of Chicago Press.

Gold, N. (1983). Stakeholder and program evalua-tion: Characterizations and reflections. InA. Bryk (ed.) Stakeholder-Based Evaluation.New Directions for Program Evaluation, No. 17(pp. 63–72). San Francisco: Jossey-Bass.

Greene, J. C. (1997). Participatory evaluation. InL. Mabry (ed.) Evaluation and the Post-ModernDilemma. Advances in Program Evaluation,Volume 3 (pp. 171–190). Greenwich, CT: JAIPress.

Gregory, A. (2000). Problematizing participation: Acritical review of approaches to participation inevaluation theory. Evaluation, 6(2), 179–199.

Guba, E. G. & Lincoln, Y. S. (1989). Fourth GenerationEvaluation. Thousand Oaks, CA: Sage.

Guttmann, A. (1987). Democratic Education.Princeton, NJ: Princeton University Press.

Hall, B. (1981). Participatory research, popularknowledge and power: A personal reflection.Convergence, 14(3), 6–19.

Hanberger, A. (2001). Policy, program evaluation,civil society, and democracy. American Journalof Evaluation, 22(2), 211–228.

Hood, S. (1998). Responsive evaluation Amistadstyle: Perspectives of one African Americanevaluator. In R. Davis (ed.) Proceedings of theStake Symposium on Educational Evaluation.Urbana IL: University of Illinois.

Hood, S. (2001). Nobody knows my name: In praiseof African American evaluators who wereresponsive. In J. C. Greene & T. A. Abma (eds)Responsive Evaluation. New Directions forEvaluation, No. 92 (pp. 31–43). San Francisco:Jossey-Bass.

Hopson, R. K. (ed.) (2000). How and Why LanguageMatters in Evaluation. New Directions forEvaluation, No. 86. San Francisco: Jossey-Bass.

Hopson, R. K. (2001). Global and local conversationson culture, diversity, and social justice in evalu-ation: Issues to consider in a 9/11 era. AmericanJournal of Evaluation, 22(3), 375–380.

House, E. R. (1980). Evaluating with Validity.Thousand Oaks, CA: Sage.

House, E. R. (1993). Professional Evaluation: Socialimpact and political consequences. ThousandOaks, CA: Sage.

House, E. R. & Howe, K. R. (1999). Values inEvaluation and Social Research. Thousand Oaks,CA: Sage.

House, E. R. & Howe, K. R. (2000). Deliberativedemocratic evaluation. In K. E. Ryan andL. DeStefano (eds) Evaluation as a DemocraticProcess: Promoting inclusion, dialogue, and delib-eration. New Directions for Evaluation, No. 85(pp. 3–12). San Francisco: Jossey-Bass.

Howe, K. R. (1997). Understanding Equal educationalOpportunity: Social justice, democracy, and schooling.Advances in Contemporary Educational Thought,Volume 20. New York: Teachers College Press.

Howe, K. R. (2003). Closing Methodological Divides:Toward democratic educational research. Philosophyand Education, Volume 11. Boston, MA: Kluwer.



Howe, K. R. & Ashcraft, C. (in press). Deliberativedemocratic evaluation: Successes and limita-tions of an evaluation of school choice.Teachers College Record, 107(10).

Karlsson, O. (1996). A critical dialogue in evalua-tion: How can interaction between evaluationand politics be tackled? Evaluation, 2, 405–416.

Kemmis, S. & McTaggart, R. (2000). Participatoryaction research. In N. K. Denzin & Y. S. Lincoln(eds) Handbook of Qualitative Research, 2ndedition (pp. 567–605). Thousand Oaks CA: Sage.

Kushner, S. (2000). Personalizing Evaluation.London: Sage.

LaFrance, J. (2004). Culturally competent evalua-tion in Indian country. In Thompson M. Robinson, R. Hopson, & S. SenGupta (eds)In Search of Cultural Competence in Evaluation. NewDirections for Evaluation, No. 102 (pp. 39–50).San Francisco: Jossey-Bass.

MacDonald, B. (1976). Evaluation and the controlof education. In D. Tawney (ed.) CurriculumEvaluation Today: Trends and implications(pp. 125–136). London: MacMillan Education.

MacDonald, B. (1977). The portrayal of persons asevaluation data. In N. Nigel (ed.) Safari: Theoryin practice (pp. 50–67). Norwich: Centre forApplied Research in Education, University ofEast Anglia.

MacDonald, B. (1978). Evaluation and democracy.Public address at the University of AlbertaFaculty of Education, Edmonton, Canada.

MacDonald, B. & Walker, R. (1977). Case-study andthe social philosophy of educational research.In D. Hamilton, B. MacDonald, C. King,D. Jenkins, & M. Parlett (eds) Beyond theNumbers Game: A reader in educational evalua-tion (pp. 181–189). Berkeley, CA: McCutchanPublising Company.

MacNeil, C. (2000). Surfacing the Realpolitik:Democratic evaluation in an anti-democraticclimate. In K. E. Ryan & L. DeStefano (eds)Evaluation as a Democratic Process: Promotinginclusion, dialogue, and deliberation. New Direc-tions for Evaluation, No. 85 (pp. 51–62). SanFrancisco: Jossey-Bass.

MacNeil, C. (2002). Evaluator as steward of citizendeliberation. American Journal of Evaluation,23, 45–54.

Madison, A. M. (1992). Primary inclusion of cul-turally diverse minority program participantsin the evaluation process. In A. M. Madison(ed.) Minority Issues in Evaluation. New

Directions for Program Evaluation, No. 53(pp. 35–43). San Francisco: Jossey-Bass.

Mathison, S. (1996). The role of deliberation in eval-uation. Paper presented at the annual meetingof the American Evaluation Association,Atlanta GA, November.

McKee, A. & Stake, R. E. (2002). Making evaluationdemocratic in a climate of control. In L. Bresler &A. Ardichivili (eds) International Research inEducation: Experience, theory, and practice(pp. 121–137). New York: Peter Lang.

McTaggart, R. (1991). When democratic evalua-tion doesn’t seem democratic. EvaluationPractice, 12(1), 9–21.

Mertens, D. M. (1999). Inclusive evaluation:Implications of transformative theory for evalua-tion. American Journal of Evaluation, 20(1), 1–14.

Mertens, D. M. (2003). The inclusive view of evalu-ation: Visions for the new millennium. InS. I. Donaldson & M. Scriven (eds) EvaluatingSocial Programs and Problems: Visions for thenew millennium (pp. 91–107). Mahwah, NJ:Lawrence Erlbaum Associates.

Niemi, H. & Kemmis, S. (1999). Communicativeevaluation. Lifelong Learning in Europe, 4, 5–64.

Rawls, J. (1971). A Theory of Justice. Cambridge,MA: Belknap.

Reiben, C. C. (1996). Participatory evaluation ofdevelopment assistance: Dealing with powerand facilitative learning. Evaluation, 2(2),151–172.

Ryan, K. E. (2004). Serving public interestsin educational accountability: Alternativeapproaches to democratic evaluation.American Journal of Evaluation, 25(4),443–460.

Schwandt, T. A. (1997). Reading the “problem ofevaluation” in social inquiry. QualitativeInquiry, 3(1), 4–25.

Seigart, D. (1999). Unpublished doctoral disserta-tion. Department of Human Service Studies,Cornell University, Ithaca NY.

Seigart, D. & Brisolera, S. (eds) (2002). FeministEvaluation: Exploration and experiences. NewDirections for Evaluation, No. 96. San Francisco:Jossey-Bass.

Simons, H. (1987). Getting to Know Schools in aDemocracy: The politics and processes of evalua-tion. New York: Falmer.

Smith, L. T. (1999). Decolonizing Methodologies:Research and indigenous peoples. London: ZedBooks.



Stake, R. E. (2000). A modest commitment to thepromotion of democracy. In K. E. Ryan &L. DeStefano (eds) Evaluation as a DemocraticProcess: Promoting inclusion, dialogue, and delib-eration. New Directions for Evaluation, No. 85(pp. 97–106). San Francisco: Jossey-Bass.

Tandon, R. (1981). Participatory research in theempowerment of people. Convergence, 14(3),20–29.

Thomas, V. G. (2004). Building a contextuallyresponsive evaluation framework: Lessonsfrom working with urban school interven-tions. In V. G. Thomas & F. I. Stevens (eds)Co-constructing a Contextually ResponsiveEvaluation Framework. New Directions forEvaluation, No. 101 (pp. 3–23). San Francisco:Jossey-Bass.

Thomas, V. G., & Stevens, F. I. (eds) (2004a). Co-constructing a Contextually Responsive evalua-tion Framework. New Directions for Evaluation,No. 101. San Francisco: Jossey-Bass.

Thomas, V. G. & Stevens, F. I. (2004b). Editors’ notes.In V. G. Thomas & F. I. Stevens (eds) Co-constructing a Contextually Responsive EvaluationFramework. New Directions for Evaluation, No.101 (pp. 1–2). San Francisco: Jossey-Bass.

VanderPlaat, M. (1995). Beyond technique:Evaluating for empowerment. Evaluation, 1(1),81–96.

Ward, K. (2002). Reflections on a job done: Well?In D. Seigart & S. Brisolera (eds) FeministEvaluation: Exploration and experiences. NewDirections for Evaluation, No. 96 (pp. 41–56).San Francisco: Jossey-Bass.

Weiss, C. H. (1983). Toward the future of stakeholderapproaches in evaluation. In A. Bryk (ed.)Stakeholder-Based Evaluation. New Directionsfor Program Evaluation, No. 17 (pp. 83–96).San Francisco: Jossey-Bass.

Whitmore, E. (1991). Evaluation and empowerment:It’s the process that counts. Empowerment andFamily Support Networking Bulletin, 2(2), 1–7.Ithaca, NY: Cornell University EmpowermentProject.

Whitmore, E. (1994). To tell the truth: Workingwith oppressed groups in participatoryapproaches to inquiry. In P. Reason (ed.)Participation in Human Inquiry. London: Sage.

Whitmore, E. (ed.) (1998). Understanding andPracticing Participatory Evaluation. NewDirections for Evaluation, No. 80. San Francisco:Jossey-Bass.



EVALUATION, DEMOCRACY, AND SOCIAL CHANGE€¦ · Evaluation, Democracy, and Social Change119...

Documents

Transcript of EVALUATION, DEMOCRACY, AND SOCIAL CHANGE€¦ · Evaluation, Democracy, and Social Change119...