Measuring the effectiveness of environmental dispute settlement efforts

Dispute resolution and negotiations

Measuring the effectiveness of environmental

dispute settlement efforts

Susan Todd�

Department of Natural Resources, University of Alaska-Fairbanks, 349 O'Neill Building,

Fairbanks, AK 99775, USA

Received 1 February 1999; received in revised form 1 February 2000; accepted 1 February 2000

Abstract

Many authors have addressed the question of how to define effectiveness in

environmental dispute settlement (EDS). However, to date, we have no standard, measurable

definition of what constitutes effectiveness, and the criteria offered thus far are often more

philosophical than practical. This article reviews and integrates the literature on the criteria

for effectiveness. It translates these criteria into 26 indicators of effectiveness and presents a

pragmatic questionnaire that can be used across a broad array of cases to evaluate the

effectiveness of EDS processes. D 2001 Elsevier Science Inc. All rights reserved.

1. Introduction

Many authors have addressed the question of how to define success in

environmental dispute settlement (EDS). Most agree that the definition of success

must be multifaceted and that no single variable, such as reaching an agreement,

is adequate for evaluating the effectiveness of an EDS effort. However, to date,

the discipline has no common definition of success. According to Blackburn

(1995), the practice of EDS is well developed, but a major challenge to theory

development is `̀ the problem of defining `success' in mediation efforts.''

Because there is no common definition, most case studies present more descrip-

tion than evaluation and often lack empirical evidence (O'Leary, 1995). In

addition, those studies that do attempt to evaluate cases use different yardsticks

0195-9255/01/$ ± see front matter D 2001 Elsevier Science Inc. All rights reserved.

PII: S0 1 9 5 - 9 2 5 5 ( 0 0 ) 0 0 0 5 6 - 1

* Tel.: +1-907-474-6930; fax: +1-907-474-6184.

E-mail address: [email protected] (S. Todd).

www.elsevier.com/locate/eiar

Environmental Impact Assessment Review

21 (2001) 97± 110

in measuring success (Blackburn, 1995; O'Leary, 1995; Kleiboer, 1996). This

makes it difficult, if not impossible, to compare across cases.

There is also a conflict between a desire to promote EDS on the one hand and

a desire to conduct an objective evaluation on the other Ð a serious conflict that

has not been readily acknowledged. For example, Scher (1996) interviewed

several practitioners and found most were uncomfortable with evaluations,

although `̀ even the most skeptical'' conceded that there might be value in

evaluation for improving the practice. Most of the practitioners had little or no

faith that EDS would fare well in an assessment that looked for anything more

than subtle changes in the behavior of participants and the level of trust between

the parties.

Further, Scher postulates that both promotion and evaluation are legitimate

reasons for evaluating EDS cases. He states that evaluations can be used both `̀ to

improve and to promote the practice'' of EDS. Further, he states that two of the

most important steps in designing an evaluation are determining `̀ how the results

should be used'' and `̀ who the evaluation is for.'' He quotes Mike Elliott, a

professor at Georgia Institute of Technology, who perhaps inadvertently pointed

out why evaluations should not be conducted for the purpose of promoting the use

of EDS. Elliott claimed he had `̀ never seen a good evaluation'' of an EDS process

and was frustrated because, even though they might improve the practice of EDS,

`̀ negative evaluations might be damaging to the field.'' Another professor, Judith

Innes from Berkeley, said `̀ the biggest problem with evaluation is the risk that it

will underestimate the benefits of ADR'' (alternative dispute resolution). The

implication is that it is better not to evaluate at all than to risk a negative result.

However, there are also risks involved in failing to evaluate the practice

objectively. Credibility is hard-won and easily lost. To have credibility, evalua-

tions must be done objectively, even though some probably will be negative. We

must conduct them scientifically without concern for `̀ how the results will be

used'' or `̀ who the evaluation is for.'' Paying attention to these concerns implies

that it is okay to report negative results to practitioners, but not okay to report

those same results to politicians or agencies who might be in a position to fund an

EDS process. This would not constitute objective evaluation. We must get the

facts out and let the chips fall where they may.

Negative results do not mean that EDS does not work Ð only that it may not

always work. If a study is done well, and if EDS is as powerful as its promoters

claim, they have nothing to fear from evaluation. Unless EDS processes are

already perfect, we should welcome negative results for that is how the process

will be improved.

This conflict between promoting and evaluating EDS spotlights the need for

standards in judging effectiveness. Allowing each evaluator to select their own

favorite criteria after a case is finished is analogous to shooting the arrow first,

then painting a target around the point where it landed. Instead, we need to set the

target before conducting evaluations, and we can only do so by defining a

common set of factors for identifying effective EDS efforts.

S. Todd / Environmental Impact Assessment Review 21 (2001) 97±11098

This set must also be practical and Ð to the extent possible Ð measurable. It

will not help to have a broad criterion such as `̀ the wisdom of the agreement''

unless we also have practical means of assessing it. Such measurements will help

provide the data that some have found lacking in the studies to date. For example,

O'Leary (1995), in her assessment of the EDS literature, questions the claims of

success in many case studies, which she finds `̀ are too often reported with little,

if any, empirical evidence.'' She concludes that we need quantitative data,

questionnaire research, and additional comparative studies that examine not only

cases of dramatic success, but also less-effective cases. Likewise, Kressel and

Pruitt (1989), in their review of mediation research, call for more `̀ systematic

approaches to self-report data.'' To that end, this article provides a questionnaire

that researchers could apply to a wide variety of cases.

A change in terminology is also proposed here. Instead of `̀ criteria for

success'' or the `̀ definition of success,'' as most of the literature has called it,

this article recommends the phrase `̀ indicators of effectiveness.'' This terminol-

ogy is preferred for two reasons. First, unlike the term `̀ criteria,'' which implies

precision, the word `̀ indicator'' suggests that the factors are indicative Ð but not

a hard and fast measure Ð of success. Second, the word `̀ effectiveness'' is

preferred over the term `̀ success.'' The word `̀ success'' generally denotes

something unambiguous and near-perfect Ð an effort is either a successful or

it is not, and there is little room to equivocate. The word `̀ effectiveness,'' on the

other hand, is less black and white. It connotes something that is adequate and

useful, but not necessarily perfect.

This article reviews and integrates the literature into a fairly comprehensive

yet manageable list of indicators that should be considered in evaluating an EDS

case. Next, it discusses how these indicators could be used in the form of a

relatively simple Likert-scale questionnaire and proposes that the population for

the questionnaire should be those who took part in the negotiations. (I refer to

these active participants as `̀ the negotiators'' and/or `̀ the negotiating group.'')

Such a questionnaire should supplement rather than substitute for the qualita-

tive information and open interviews of the stakeholders that remain vital in

setting the context of a negotiation (these interviews usually include, but are not

limited to, the negotiators). A questionnaire would provide some of the quanti-

tative data that is noticeably lacking to date, it would facilitate the comparison of a

wide diversity of cases, and it would help to standardize the definition of what

constitutes an effective EDS process. The article concludes with a discussion of

who would administer and who would complete this questionnaire.

2. The indicators

Several commonalities exist in the literature on the definition of success. First,

many authors note that it is important to evaluate the effectiveness of both the

outcome and the process of an EDS effort. Perhaps the first to note this distinction

S. Todd / Environmental Impact Assessment Review 21 (2001) 97±110 99

was Bingham (1986), who categorized several of the factors listed below as either

`̀ process'' or `̀ outcome'' indicators. Moore (1996) and Wondolleck and Yaffee

(1994) also used factors that were classified as process or outcome criteria.

The list of indicators proposed here is organized by these `̀ outcome'' and

`̀ process'' categories, as well as several subcategories. Four subcategories are

adapted from Susskind and Cruikshank (1987), who identified four character-

istics of an effective EDS effort: fairness, efficiency, wisdom, and stability.

Several additional subcategories are recommended. These categories were helpful

in building a thorough and systematic list of indicators. Table 1 lists all the

indicators. The quetionaire in Table 1 was first developed based on a review of

the literature, and then it was used in evaluating three different EDS processes

[18]. It has since been refined and used in evaluating four additional cases. It

includes the indicators discussed below, listed as statements with a Likert scale

ranging from strongly agree to strongly disagree. The respondents can simply

mark their level of agreement with each statement. These indicators, and how

they can be used are discussed below.

2.1. Process indicators

2.1.1. Fairness

The best way to determine fairness, in Susskind and Cruikshank's (1987)

approach, is to `̀ evaluate the attitudes and perceptions of the parties most

affected.'' They suggest several tests, including: (1) Was the offer to participate

genuine and were all the stakeholders given a chance to be involved? (2) Was the

process perceived as legitimate after it ended, as well as when it began? and (3) In

the eyes of the community, was a good precedent set? Would the participants use

such a process again?

2.1.2. Efficiency

Susskind and Cruikshank (1987) determine efficiency in terms of whether

the process cost more than traditional processes, whether it took more time, and

whether any joint gains were left on the table. The findings of Buckle and

Thomas-Buckle (1986) corroborate the importance of this indicator: `̀ the most

concrete benefit indicated by the participants as contributing to their perception

that mediation had `succeeded' was reduction in time spent, expenses paid, and

delay encountered in mediation, when compared with adjudication or regula-

tory hearings.''

2.1.3. Changes in relationships

Several authors have stressed that signing a written agreement, although often

the goal, is not in itself an adequate measure of success because changes in

previously adversarial relationships are also important.

For example, the participants in a study conducted by Buckle and Thomas-

Buckle (1986) pointed out Ð to the surprise of their mediators Ð that they


Tab

le1

Indic

ators

of

effe

ctiv

enes

s:a

ques

tionnai

refo

rth

eneg

oti

atin

gte

am

Cat

ego

rya

Su

bca

teg

ory

aIn

dic

ator

Str

ongly

agre

e

(2)

Agre

e

(1)

Neu

tral

or

undec

ided

(0)

Dis

agre

e

(ÿ

1)

Str

ongly

dis

agre

e

(ÿ

2)

Not

appli

cable

(NA

)

Pro

cess

ind

icat

ors

Fai

rnes

s(1

)T

he

pro

cess

has

bee

nfa

ir.

(2)

All

pote

nti

ally

affe

cted

par

ties

wer

e

invit

edto

par

tici

pat

e.

(3)

Th

em

emb

ers

of

the

team

adeq

uat

ely

rep

rese

nte

dth

era

ng

eof

publi

copin

ion

on

this

issu

e.

(4)

The

pro

cess

of

choosi

ng

team

mem

ber

s

was

fair

.

(5)

Iw

ou

ldb

ew

illi

ng

topar

tici

pat

eon

such

ate

amag

ain

.

(6)

Th

isp

roce

ssse

ta

go

od

pre

ceden

tin

the

eyes

of

the

com

mu

nit

y.

Eff

icie

ncy

(7)

The

pro

cess

has

bee

nef

fici

ent

inte

rms

of

tim

ean

dm

on

ey.

Chan

ges

in

rela

tio

nsh

ips

(8)

Th

eef

fort

has

incr

ease

dth

ele

vel

of

trust

bet

wee

nth

ep

arti

es.

(9)

Th

ep

roce

ssim

pro

ved

my

insi

ght

into

the

inte

rest

san

dp

osi

tion

sof

all

the

dis

puta

nts

.

Com

par

edto

alte

rnat

ive

pro

cess

es

(10)

An

alte

rnat

ive

pro

cess

,su

chas

liti

gat

ion

or

leg

isla

tio

n,

wo

uld

not

hav

ew

ork

edas

wel

l

inse

ttli

ng

this

dis

pute

.

Ou

tcom

e

ind

icat

ors

Pu

rpo

se(1

1)

Th

ete

amm

etit

sst

ated

purp

ose

(to

dis

cuss

the

issu

eso

rth

ed

ata,

or

tore

ach

agre

emen

t,et

c).

Con

sen

sus

on

a

wri

tten

agre

emen

t

(12

)T

he

team

reac

hed

conse

nsu

son

a

wri

tten

agre

emen

t.


Cat

ego

rya

Su

bca

teg

ory

aIn

dic

ator

Str

ongly

agre

e

(2)

Agre

e

(1)

Neu

tral

or

undec

ided

(0)

Dis

agre

e

(ÿ

1)

Str

ongly

dis

agre

e

(ÿ

2)

Not

appli

cable

(NA

)

Inte

rest

s(1

3)

My

ow

nin

tere

sts

wer

em

etby

the

pla

n.

(14

)T

oth

eex

ten

tp

oss

ible

,th

ew

ritt

enag

reem

ent

met

ever

yon

e's

inte

rest

s.

Wis

do

mo

fth

e

wri

tten

agre

emen

t

(15)

The

pro

cess

and

the

wri

tten

agre

emen

tat

tem

pte

d

tom

inim

ize

the

risk

sof

bei

ng

wro

ng.

(16

)T

he

pro

cess

consi

der

edth

epote

nti

alen

vir

on-

men

tal

impac

tsof

the

wri

tten

agre

emen

tan

d

fou

nd

them

acce

pta

ble

.

(17

)T

he

pro

cess

consi

der

edth

epote

nti

also

cial

and

eco

no

mic

imp

acts

and

found

them

acce

pta

ble

.

Sta

bil

ity

of

the

wri

tten

agre

emen

t

(18)

The

team

'sw

ritt

enag

reem

ent

isli

kel

yto

last

.It

wil

l`̀

stan

dth

ete

stof

tim

e.''

(19)

The

level

of

contr

over

syhas

dim

inis

hed

asa

resu

lt

of

this

dis

pute

sett

lem

ent

effo

rt.

(20

)T

her

eis

bro

adp

ub

lic

support

for

the

wri

tten

agre

emen

t.

(21

)T

he

effo

rth

asse

ttle

dth

ekey

issu

es.

(22

)I

bel

iev

eth

ew

ritt

enag

reem

ent

isbei

ng

or

wil

l

be

imple

men

ted.

(23)

Ther

ear

epro

vis

ions

for

reneg

oti

atio

nif

pro

ble

ms

aris

e.

Over

all

effe

ctiv

enes

s(2

4)

As

ate

amm

ember

,I

found

the

effo

rt

per

sonal

lyre

war

din

g.

(25

)I

thin

kth

eef

fort

was

asu

cces

s,by

my

ow

n

def

init

ion

of

the

term

.

(26

)P

leas

eex

pla

inb

elow

why

you

thin

kit

was

Ðor

was

no

tÐ

asu

cces

s.

aT

hes

eca

tegori

esan

dsu

bca

tegori

esw

ere

use

ful

inbuil

din

gth

eli

stof

indic

ators

,but

they

nee

dnot

be

incl

uded

on

the

ques

tionnai

refo

rth

eneg

oti

atin

gte

am.


viewed their EDS processes as effective even though no agreement was

reached. They identified several other significant ways in which mediation had

contributed to the successful processing of their conflicts. These included

personal rewards such as an increased personal ability to negotiate, an increase

in their level of trust and understanding, and insight into the interests and

positions of all the disputants. The authors concluded that we should `̀ account

a mediator effective to the degree that he or she enables the parties to increase

their affective and cognitive awareness about their relationship and the matter

at dispute.''

In a study of several EDS cases compiled by Crowfoot and Wondolleck

(1990), citizen members of EDS teams noted that they had `̀ gained valuable new

skills'' and that the processes had `̀ increased the credibility, legitimacy, and trust

between what had before been traditional adversaries.'' Bingham (1986), Lee

(1982), Moore (1996), Wondolleck and Yaffee (1994), and Zubek et al. (1992)

found that similar relationship-oriented criteria were important in participants'

assessments of their EDS processes.

2.1.4. Comparison to alternative processes

Most evaluations are relative. It is hard to judge the success of anything

without also answering the questions: `̀ Effective compared to what? What are the

alternatives?'' Comparing EDS to other alternatives is also needed. Would a

different process, such as litigation or legislation, have been a more effective

forum for settling the dispute? An EDS process may not appear totally effective

by itself Ð but if litigation and/or legislation were tried and failed to settle the

dispute, or if they cost far more in terms of money or time Ð the EDS process

may look quite effective in comparison.

If the stakeholders in a given case did try other forums, researchers should

make an effort to find out what happened and compare that to the EDS results. If

other forums have not yet been tried, then our only option may be to ask the

stakeholders themselves if they think an alternative forum would have worked

better in terms of the same indicators of fairness, efficiency, wisdom, etc.

2.2. Outcome indicators

2.2.1. Purpose

In Susskind and Cruikshank's approach, the fairness and the efficiency

indicators apply to the process, while wisdom and stability apply to the outcome.

However, the wisdom and stability indicators will not apply in every case because

many EDS teams are convened to discuss options or share information and are

not expected to reach an agreement. To deal with these cases, I added the

indicator `̀ Did the group meet its purpose?'' One criterion for the effectiveness of

many endeavors is whether the effort accomplished what it set out to do. If the

purpose was simply to discuss the issues, was that purpose met? If a team set out

to develop recommendations, did they do so?


2.2.2. Consensus

First, a definition is in order. Consensus is often discussed but rarely defined in

the literature, despite the fact that it is often misunderstood by the general public

as an agreement that fully meets the concerns of all the parties to the fullest. I

define consensus as a package agreement Ð often a package of compromises on

the issues and subissues Ð which all members of the negotiating group can live

with because it meets their most important concerns. Unlike the typical vote Ð

which offers essentially two options: `̀ I love it'' or `̀ I hate it'' Ð consensus

allows a third option, if you will, the `̀ I can live with it'' option. This in itself

considerably expands the possibilities for agreement.

Because consensus is so fundamental to an EDS process, it is only natural to

view it as a key criterion of effectiveness. In fact, the pinnacle of effectiveness is

often viewed as a process in which the parties reached consensus on all of the key

issues and were able to see their consensus agreement implemented. However, it is

also quite respectable to reach agreement on many, but not all, of the issues.

2.2.3. Written agreement

In addition to reaching consensus, the literature also emphasizes the impor-

tance of putting the agreement on paper. A written agreement helps to ensure that

all parties are agreeing to the same thing and to eliminate some problems in

recollection and interpretation. In addition, while group members themselves may

move on, a document will remain and can have lasting impact. Therefore, if

reaching an agreement was part of the purpose of an EDS process, then it is

important to include this as an indicator of effectiveness.

If, however, the purpose of the process was simply to discuss information or

develop options, then it would be inappropriate to use a written agreement as a

measure of effectiveness (in which case the next two major criteria Ð the wisdom

and the stability of the agreement Ð would also be inappropriate measures).

2.2.4. Interests

A fundamental concept in EDS and consensus-building generally is that the

parties should attempt to meet their individual interests. The interviewees in

Moore's (1996) study identified this as a key factor in judging effectiveness.

However, as Moore (1996) and Wondolleck and Yaffee (1994) point out, a good

process attempts to meet the interests of all the parties, both individually and as a

group. Thus, the statements `̀ My own interests were met by the plan,'' and `̀ To

the extent possible, the agreement met everyone's interests'' were added to the list

of indicators.

2.2.5. Wisdom of the agreement

The wisdom of an agreement is surely the most difficult and subjective, yet

perhaps the most important, criterion to measure. Certainly, group effectiveness

should be related to the quality of their decision. As an expert in the evaluation of

management teams points out, if a group is established to make a decision, then


effectiveness for that group should be gauged in terms of the merits of their

decision (Guzzo, 1986).

As Susskind and Cruikshank (1987) point out, one difficulty with this criterion

is that it is generally only obvious in hindsight. Given time, we can look back and

evaluate the impacts of a decision. If a team decided to harvest timber on one site

and preserve another, we could examine both areas for many years to monitor the

environmental and social consequences of the decision. However, if little time

has passed, it will be difficult to assess the actual impacts.

However, we can determine whether the EDS process made a systematic

attempt to consider several alternative courses of action and their potential

impacts. Often, a good deal of the controversy on an EDS case results from

differences of opinion regarding the potential impacts of an action. Did the group

consider the possible environmental impacts (including cumulative impacts) or

several alternatives? Did they reach some agreement on the range of likely

impacts? Did they attempt to minimize the risk of being wrong?

In addition to the environmental questions, it is also important to consider the

social consequences. People are a resource, too, and most EDS cases are basically

struggles to balance the biophysical needs of an ecosystem with human needs. A

culture and a way of life can also be endangered. Thus, a question should be

included regarding whether the risks of social and economic impacts were

considered and whether there was an attempt to minimize problems.

2.2.6. Stability of the agreement

Stability is also a key component of effectiveness. `̀ An agreement that is

perceived as fair, is reached efficiently, and seems technically wise is nevertheless

unsatisfactory if it does not endure'' (Susskind and Cruikshank, 1987). Ideally,

the agreement will be implemented, and it will last because it diminishes the

controversy and, at best, settles the key issues. An agreement will also be more

stable if provisions are included for renegotiation, should that become necessary.

These points are discussed in more detail below.

2.2.6.1. Has the level of controversy diminished? Is there broad public

support? The level of controversy and the level of public support for a decision

are really opposite sites of the same coin; one looks for opposition, the other for

approval. It should be noted that at best, the underlying broad conflict (e.g.,

timber harvesting vs. preservation) is unlikely to disappear as a result of an EDS

process (Crowfoot and Wondolleck, 1990). However, it is hoped that the

controversy over the specific dispute (harvesting timber in a small, specific area

vs. preserving that area) can diminish Ð if not disappear Ð if the key issues have

been settled. Most importantly, the process should not create more problems than

it solves. As Carpenter and Kennedy (1988) put it, a basic principle of EDS is to

`̀ do no harm'' (i.e., do not exacerbate the controversy).

If the agreement is supported by a wide range of interests and if that support is

fairly solid, it is more likely to be stable and capable of withstanding attacks from


extremes on both sides. Without broad support, an agreement signed by a few

representatives will not have far-reaching impact.

2.2.6.2. Have the key issues been settled? What could be a finer outcome than

that the key issues are settled? As one negotiator said of the key issue in her EDS

process, `̀ It's not an issue anymore. It's no longer front-page news; it's no longer

even back page news. I guess that's one of the best indicators we could hope for

that we did our job.''

If the key issues have not been settled, the controversy is likely to flare again. The

negotiators may have made significant progress in promoting understanding,

establishing relationships, and sharing information, but as long as key issues

remain, the controversy will not diminish and the agreement is unlikely to be stable.

2.2.6.3. Are provisions included for renegotiation? Because it is rarely possible

to anticipate all the types of problems the settlement may face, it is also important

for the disputants to have a good working relationship that will allow them to

return to the bargaining table to work things out if the need arises. `̀ If disputing

parties build a good working relationship, the prospects for stability are greatly

enhanced . . .. The parties are likely to come back to fix the agreement and make

it work if they feel positively about how they were treated'' (Susskind and

Cruikshank, 1987).

2.2.6.4. Has the agreement been implemented? Have the parties complied? Per-

haps the most common measure of effectiveness is whether the agreement has been

implemented and whether the parties complied with the agreement (Lee, 1982;

Zubek et al., 1992; Pearson and Thoennes, 1989). As Carpenter and Kennedy

(1988) point out, if a written agreement has been reached but not implemented, it

can create a feeling of betrayal among the parties and resentment over apparently

wasted effort.

2.2.7. Overall effectiveness

Finally, it is also useful to simply ask the stakeholders whether their process

was effective according to their own personal definitions of the term. This has

been done in many studies, including Buckle and Thomas-Buckle (1986),

Crowfoot and Wondolleck (1990), and Moore (1996).

3. Using the indicators

3.1. Interviews remain essential

Qualitative data will always have a place in the analysis of EDS cases. In fact,

I would argue for much more `̀ rich, thick description,'' as Geertz (1983) calls it,

in the presentation of case studies. Kressel and Pruitt (1989) also contend that


most investigations fail to provide enough detail about a case to make it clear just

what conflict management procedures were used.

However, qualitative data alone has some disadvantages that could be over-

come by supplementing interview data with a questionnaire completed by each of

the negotiating group members. In the process of evaluating a case, most

researchers conduct in-depth interviews of the major parties involved in the

negotiation (both those at the table and those on the sidelines), as well as any third-

party interveners. Most interviewers take care to avoid leading questions and to

allow the respondents to direct the discussion to their own points of concern. A

broad question such as `̀ Was this process effective and if so, in what ways?'' is

often used early in the interview to probe for the interviewee's own definition of

effectiveness. Such interviews should retain their place in the assessment of EDS

cases. In fact, the indicators suggested here also work well as an interview guide.

In my own work, I use the indicators as the basis for `̀ open-ended'' questions late

in an interview, after the respondent has shared what they feel is important with

minimal prompting or questioning from me. Combining both open-ended inter-

view questions and more structured, limited-response questions on the same topics

has proven useful for corroborating the data and interpreting the oral responses.

However, interview data alone has several disadvantages. First, it is largely

anecdotal and by itself, anecdotal information is not adequate to accurately assess

the effectiveness of EDS. Second, interview responses are not easily compared.

Because the interviews are intentionally loosely structured, the researcher often

asks the same basic question in slightly different ways in each interview. This

could potentially alter the responses, making it difficult to compare them. In

addition, in answering a question such as `̀ Was this process effective?'' respon-

dents will rarely use a simple `̀ yes'' or `̀ no.'' It is much more likely that each

respondent will use different wording Ð and often many paragraphs of colorful,

transcribed text Ð to answer the question. This detail is essential in providing the

context that is so important in qualitative research. Nevertheless, at the same time,

the detail makes it almost impossible to compare responses within a case Ð much

less across cases Ð or to make a statement such as `̀ eight team members thought

the process was fair and three did not.'' Such discreet data are also needed, and are

sorely lacking, in existing evaluations of EDS.

However, if the open interviews are supplemented with a questionnaire that

allows only a limited number of choices, several of these disadvantages are

overcome. First and most important, the questions are presented in exactly the

same words to each respondent and their answers are easily compared both

within a case and across a wide diversity of cases. The researchers is not required

to summarize Ð and in the process, possibly misinterpret Ð an interviewee's

soliloquy. If the questionnaire takes place following the preliminary analysis of

the interviews, it will also be possible to test the accuracy of the developing

conclusions. Finally, for those who object to the limited-response questionnaire,

it is possible to add a few open-ended questions and room for comment at the

end of the questionnaire.


3.2. Who should administer the questionnaire?

By definition, an objective evaluation must be done by people who do not have

a personal stake in either promoting EDS or in the outcome of the negotiations in

the case being evaluated. Those involved in a case, either as an intervener or as a

stakeholder, can add to the store of knowledge by writing their memoirs of the case,

but their personal interests must be acknowledged up front. Further, such firsthand

accounts must not be our only source of information and evaluation of EDS.

People in the discipline are very firm on how important it is for a potential

mediator to be neutral on the issues. It is equally important for an evaluator to be

neutral. As stated above, there is a place for personal accounts, but objective

analysis by people who are not promoters of EDS, has nothing to gain whether

the evaluation is positive or negative, and were not involved in any way in the

case are essential.

3.3. Who should answer the questionnaire?

Very little has been written about the importance of being careful to `̀ sample''

each of the interests equally when choosing those to interview. It would not be

acceptable, for example, to interview only those who favored a proposal and none

of those who opposed it (or vice versa). All of us have probably seen such

`̀ evaluations,'' and they harm the credibility of the field.

With a written questionnaire, where results will be quantified according to

how many agreed, how many disagreed, etc., it is even more vital to ensure that

the sample is balanced. It is also important to define a bounded `̀ population''

from which you will take your sample. One of the easiest ways to determine who

to survey is to limit the `̀ population'' to the negotiating group itself, which

should be balanced in regard to the range of opinion on the issues (and the

questionnaire asks if they felt the group was balanced). This way, the population

is clearly defined, the research is less likely to be criticized as biased due to

selective sampling, and it is obvious what percentage of this small population

your sample represents (and you may have 100%).

The questionnaire in Table 1 was first developed based on a review of the

literature, and then it was used in evaluating three different EDS processes (Todd,

1995). It has since been refined and used in evaluating four additional cases. It

includes the indicators discussed above, listed as statements with a Likert scale

ranging from strongly agree to strongly disagree. The respondents can simply

mark their level of agreement with each statement.

4. Conclusions

Many authors have addressed the question of how to define success in EDS,

but to date, we have no standard, measurable definition of what constitutes


effectiveness. Most of the research has asked the negotiators to assess their

process in their own terms. Since each individual devises a slightly different

yardstick, we cannot compare across cases.

Clearly, it is time to adopt a uniform measure of effectiveness that incorporates

the primary criteria proposed in many different studies. Such a measure need not

be exclusive, but could supplement other approaches. This article reviews and

integrates the literature into a comprehensive yet manageable list of 26 indicators

that should be considered in evaluating an EDS case. These indicators can be

used in the form of a simple Likert-scale questionnaire. This questionnaire would

provide a uniform yardstick and the quantitative data that EDS evaluations

noticeably have been lacking, it would facilitate the comparison of a wide

diversity of cases, and it would help to standardize the definition of what

constitutes an effective EDS process.

References

Bingham G. Resolving environmental disputes: a decade of experience. Washington, DC: The Con-

servation Foundation, 1986.

Blackburn J. Environmental mediation theory and practice: challenges, issues, and needed research

and theory development. In: Blackburn J, editor. Mediating environmental conflicts, theory and

practice. Westport, CT: Quorom Books, 1995.

Buckle LG, Thomas-Buckle SR. Placing environmental mediation in context: lessons from `̀ failed''

mediations. Environ Impact Assess Rev 1986;6(3):55±70.

Carpenter S, Kennedy WJD. Managing public disputes. San Francisco: Jossey-Bass, 1988.

Crowfoot J, Wondolleck J. Environmental disputes: community involvement in conflict resolution.

Washington, DC: Island Press, 1990.

Geertz C. Thick description: toward an interpretive theory of culture. In: Emerson R, editor. Con-

temporary field research. Boston: Little, Brown and Company, 1983. pp. 37± 59.

Guzzo R. Group decision making and group effectiveness in organizations. In: Goodman P, editor.

Designing effective work groups. San Francisco: Jossey-Bass Publishers, 1986. pp. 34± 71.

Kleiboer M. Understanding success and failure of international mediation. J Conflict Resol

1996;40(2):360± 89.

Kressel K, Pruitt D. Conclusion: a research perspective on the mediation of social conflict. In: Kressel

D, Pruitt D, editors. Mediation research: the process and effectiveness of third-party intervention.

San Francisco: Jossey-Bass Publishers, 1989.

Lee K. Defining success in environmental dispute resolution. Resolve 1982;1(Spring).

Moore S. Defining `̀ successful'' environmental dispute resolution: case studies from public land

planning in the United States and Australia. Environ Impact Assess Rev 1996;16(2):151 ±69.

O'Leary R. Environmental mediation: what do we know and how do we know it? In: Black-

burn J, editor. Mediating environmental conflicts, theory and practice. Westport, CT: Quorom

Books, 1995. pp. 17,32.

Pearson J, Thoennes N. Divorce mediation: reflections on a decade of research. In: Kressel D, Pruitt D,

editors. Mediation research: the process and effectiveness of third-party intervention. San Francis-

co: Jossey-Bass Publishers, 1989. pp. 9 ±30.

Scher E. Evaluations: what for, by whom who pays? Consensus. 1996;5.

Susskind L, Cruikshank J. Breaking the impasse. New York: Basic Books, 1987.

Todd S. Designing effective negotiating teams for environmental disputes: An analysis of three wolf

management plans. Doctoral Dissertation, University of Michigan, Ann Arbor, 1995.


Wondolleck J, Yaffee S. Building bridges across agency boundaries: in search of excellence in the

USFS. A research report submitted to the USDA-Forest Service Pacific Northwest Research

Station, in fulfillment of USDA Forest Service Cooperative Agreement #PNW 92-0215. Ann

Arbor: School of Natural Resources and the Environment, The University of Michigan, 1994.

Zubek J, Pruitt D, Pierce R. Disputant and mediator behaviors affecting short term success in media-

tion. J Conflict Resol 1992;36(3):546±72.


Measuring the effectiveness of environmental dispute settlement efforts

Documents

Transcript of Measuring the effectiveness of environmental dispute settlement efforts