B U L L E T I N O c t o b e r 2 0 0 0

NON-PROFIT ORG.U.S. POSTAGE

PAIDACGME

PERMIT NO. 9451Accreditation Council for Graduate Medical Education515 North State Street, Suite 2000Chicago, IL 60610-4322

Change of Address:Requests for achange of addressshould be sent to theeditor at the addressgiven above. Pleaseinclude a copy of themailing label fromthe most recent copyof the ACGMEBulletin along withyour new address.

CHICAGO

I L

INSIDE THIS ISSUE

• Executive Director’s Column:Trust Me, I’m A Doctor

• Enhancing Public Oversight of Accreditation - An Interview with the ACGME’s Public Members

• Resident Retreats as a Way to Facilitate Communication and Learning

• RRC/IRC Column

• Complaints Against Residency Programs

• Two Frequently Asked Questions about the ACGME Outcome Project: Time-Line and Minimum Language

• Other Highlights from the September 2000 ACGME Meeting

• Analyzing Decision-Making in Groups - Findings from the Management Literature with Applications for Medical Education

B U L L E T I N O c t o b e r 2 0 0 0

Accreditation Council for Graduate Medical Education

The ACGME Bulletinis published four timesa year by the AccreditationCouncil for GraduateMedical Education.The Bulletin is distributedfree of charge to over 12,000individuals involved in graduatemedical education. The Bulletinis also available from theACGME’s World Wide Web siteat http://www.acgme.org.Inquiries, comments or lettersshould be addressed to the editor.

Ingrid PhilibertEDITOR

David C. Leach, M.D.CONSULTANT

515 North State Street, Suite 2000 Chicago, IL 60610-4322

Phone: 312/464-4920 FAX: 312/464-4098.

©Accreditation Council for Graduate MedicalEducation, October 2000. This documentmay be photocopied for the non-commercialpurpose of educational advancement.

Executive Director’s Column:

Trust Me, I’m A DoctorThe ACGME's stakeholders include residency programs,residents, medical students, specialty boards, patients, thegovernment and the public. The public is the hardest groupto define and, in many ways, it is the most important ofour stakeholder groups. Its members reasonably expectthat newly minted physicians have acquired state-of-the-artclinical and professional skills and are prepared to meettheir patients' needs by the time they graduate from anaccredited residency program, especially since programsare supported by public funds. What can be done tostrengthen the natural alliance between the medical profes-

sion and the public during these challenging times? Is it enough to say, "Trustme; I'm a doctor." Is it sufficient that someone has graduated from an accreditedprogram? Finally, does the public understandthe role of accreditation in safeguarding thequality of physician education? Do we need tomake our work and their safeguards crystal-clear? If so, what type of information should bepresented? How can it be configured in waysunderstandable and useful to the public? Howcan we engage in a partnership with the publicthat will focus on our common purpose ofimproving education and patient care?

There are more questions than answers. Toexplore the questions and seek some of theanswers, the ACGME has done several thingsto further the dialogue with the public. Recently,it added a third public member to its twenty-six member Board of Directors. Furthermore,it now encourages its public members to attend all, even closed, committee meetingsof the ACGME, its subcommittees and its Residency Review Committees (RRCs). Publicmembers regularly attend the Executive Committee meetings and have begun to attendRRC meetings. They bring intelligence, the insight of non-physician citizensand consumers of health care, and a critical public perspective to deliberations. An articlein this issue of the Bulletin reports some of their thoughts captured in a recent interview.

On June 27, 2000, the ACGME became a separately incorporated entity. This allowsthe Council to benefit to a greater extent from the wealth of experience and the posi-tions of our member organizations and yet holds each of the Directors of the ACGMEto high standards of fiduciary responsibility, standards that place the public's concernsin a prominent position. Directors are free to be objective on issues such as residentwork hours that are of special interest to patients and the general public.

A crucial question that remains is how much and what kind of information theACGME should provide to the public. Presently the accreditation status of each

David C. Leach, MD

“The ACGME’spublic members bring

intelligence, the insight ofnon-physician citizens andconsumers of health care,and a critical public per-

spective to deliberations.”

“A crucial question that remains is how much and what kindof information the ACGME should provide to the public.”

B U L L E T I N O c t o b e r 2 0 0 0

residency program is available on our web site. Thelength of the cycle between reviews is also available.The information missing for the public is what thevarious status designations imply, and what the cyclelength says about a given RRC's comfort level with aprogram's ability to educate residents. Specific citationsare not available. Nor are normative data about theactions of the 26 different RRCs.

Data external to, but capable of illuminating, programsare publicly available. But they are not uniformlyaccessible, and often not conveniently configured.They include pass rates on the board exam; informationfrom resident and recent graduate surveys; generalinstitutional quality indicators; data from patientsurveys; and data on an academic department's grantsand publications. In addition, areas of particularconcern to the public, such as the average number ofresident work hours, could be displayed. How muchdata will be needed to make the public comfortablewith the system that educates its physicians is notexactly known. The ACGME will struggle with this issueuntil we get it right. However, "Trust me I'm a doctor"may have to be replaced with, "In God we trust, allothers must bring data."

Enhancing Public Oversightof Accreditation - An Interviewwith the ACGME'sPublic MembersIngrid Philibert

In the preceding article, Dr. Leach comments on thepublic's trust in the qualification of physicians. Thepublic members play a pivotal role in safeguarding thistrust. The ACGME's mission statement indicates that it".... strives to improve evaluation methods and processesthat are valid, fair, open, and ethical." In keeping withthis, its public members are ultimately responsible forensuring that public expectations for quality andaccountability are met.

In the past, the ACGME had two public members. InSeptember 1999, it appointed Mr. Duncan McDonald,Vice President for Public Affairs and Development forthe University of Oregon, as the third, joining KayHuffman Goodwin and Agnar Pytte, PhD. The members'backgrounds are diverse. Ms. Goodwin has served as thechair of the University of West Virginia System's Boardof Trustees, and sits on the Board of West VirginiaUniversity Hospitals and the West Virginia United HealthSystem. Agnar Pytte, PhD, recently retired from the

Presidency of Case Western Reserve University. Hisresearch career focused on theoretical plasma physicsand nuclear fusion. Mr. McDonald's background combinesjournalism and political science. Ms. Goodwin and Mr.McDonald are familiarwith accreditation. Sheserves as the public mem-ber of the North CentralAssociation of Collegesand Schools West VirginiaAccreditation Committee;he has been Chair ofthe National AccreditingCommittee for Educationin Journalism and MassCommunications. Theinterview included PaulFriedmann, MD, who justconcluded his term asChair of the ACGME,and R. Edward Howell,the current ACGME Chair.

Asked what attracted themto their role, the memberscommented on their deepinterest in education. Ms.Goodwin is the daughter of a family practice physician,and stated that she, "saw the results of an excellentmedical education up close and personal." Dr. Pytteadded he had spent nearly his entire life in educationand his responsibilities as Provost at DartmouthCollege, and later President of Case Western ReserveUniversity gave him insight into the complex natureof physician education. Mr. McDonald stated, "I wasintrigued and flattered when I was asked to be part ofthe organization that accredits graduate medical educa-tion in the United States and has major influence inter-nationally." All noted that the rigor of the accreditationprocess and the dedication with which the work isapproached is gratifying, as is the amazing volunteercommitment by physicians and non-physicians - fromACGME directors to RRC members - to ensure soundeducation and practice in their specialty and in themedical community overall. Ms. Goodwin creditedRobert D'Alessandri, MD, Dean of West VirginiaUniversity School of Medicine and former Chair of theACGME, with the outstanding quality of medical educa-tion in her state. She noted she is honored to assist themedical education community, and is grateful to herfather and Dr. D'Alessandri.

The members feel their role has given them insight intothe impact of rapid change in medicine on education,including how mergers and hospital closures canprofoundly disrupt residents' lives. Mr. Duncan stated,

2

“...one of the

ironies of public

membership

is part of the

attraction: ‘As

public members,

we are

given a very

private insight into

the functioning

of the ACGME.’”

3

"I have learned about the delicate balance of residentcomplement and patient volume - that too many residentsand not enough patients can be as detrimental toeducation as too many patients and not enoughresidents." He added that one of the ironies of publicmembership is part of the attraction: "As publicmembers, we are given a very private insight intothe functioning of the ACGME."

Question: What characterizes effective publicmembership of the ACGME?

Ms. Goodwin: Having public members is importantto the public. Having three makes it possible to have apublic member participate on all committees, to bringthe public's perspective to the table and report to thepublic on the Council's efforts. It is also important thatpublic members familiarize themselves with the workof the RRCs, to enable them to inform the public abouthow this work contributes to educational quality.

Dr. Pytte: Effective public membership should reflecttwo things. First, as non-physicians, the public membershave a different perspective, that of "patient" and"member of society" interested in education and the

quality of care. This is important when the ACGMEconsiders such issues as resident work hours. The publicis concerned that residents provide care while exhausted.The members of the discipline care about work hours,but need to balance this with their concerns thatresidents learn the entire depth and breadth of practicein the specialty. Our perspective is closer to that of thepublic at large, but is informed by our interactions withthe ACGME. The second function is to report to thepublic on the ACGME's activities, to explain what wehave learned, and to ensure that these activities haveappropriate public oversight.

Question: Howdo public memberscontribute to an accredi-tation process that isresponsive to the needsof the public?

Dr. Pytte: I said earlier thatone of the functions of thepublic members is to informthe public about the roleof accreditation. I don'tthink we are doing as muchin this area as we could. Asa former university presi-dent, I speak to individualsand influential groups aboutthe impressive work theACGME is doing. However,I don't reach a meaningfulnumber of the manyAmericans who may beconcerned and need to hearthis message. The publicmembers can do some ofthis, but the ACGME itself needs to do more to connectwith the public.

Mr. McDonald: One of the responsibilities of thepublic members involves advising the ACGME on report-ing information to the public. I understand that not alldata can be public, but my bias as a journalist is,"more is better."

Ms. Goodwin: I agree. More is better.

Dr. Pytte: I disagree somewhat, the public is notinterested in every detail about the accreditation ofmore than 7,700 residency programs. It wants thelarger picture: that there are systems that safeguardphysician education and patient care in hospitalswhere physicians train, and that these are functioningas they should. Also, the public may not completelyunderstand how physicians are educated, and explainingand defending the educational process is somethingpublic members should do.

Question: What could the ACGME and its publicmembers do to define the public's expectations forphysician education quality?

Mr. McDonald: The context for the public's expectationsis its concern for patient care quality and safety. Whenwe discuss the public release of accreditation information,we need to understand this context. At present, thepublic has access to information on programs' accredi-

...as non-physicians,

the public members have a

different perspective, that of

“patient” and “member of society”

interested in education and the

quality of care.”

“The public...

wants the larger

picture: that there

are systems that

safeguard physi-

cian education and

patient care in

hospitals where

physicians train,

and that these are

functioning as

they should.”

B U L L E T I N O c t o b e r 2 0 0 0

4

tation status and cycle length, but does not under-stand the implications of this information withouta more detailed explanation. Another context isprovided by the changes in health care, which createan environment in which it is more difficult to educateresidents. The public hears about this, and we muststress that the ACGME is doing all it can to ensure thatresidents are educated appropriately. Data will play arole in this, but data alone cannot accomplish it.

Dr. Friedmann: Yes, the public is interested how thedata are used. For example, on the issue of work hours,the public wants to know what the ACGME does tohelp improve the state of things after it cites a programfor resident work hours.

Mr. Howell: Work hours are a complex issue. Residentslearn by doing, and the public rightfully asks what isdone to ensure safe patient care. Our attitude must be,"the buck stops here." At the same time, physicians inpractice may work just as many hours, but there is lessconcern about their level of exhaustion and how it mayimpact performance.

Question: How can the ACGME improve its systemsfor communicating with the public?

Mr. McDonald: Communicating with the public isimportant, but potentially much more critical is findingways for the public to communicate with the ACGME,and what role the public members may play in this.Establishing this communication will be vital toassuring the public that it truly has opportunityto provide input into the process.

Mr. Howell: Opening up communication channelsis important. Beyond the general public, two criticalstakeholder groups are residents and medical students.We need to enhance the system that allows them tocommunicate with the ACGME, and will be well servedto have our public members engaged in this process. Iam impressed with the enthusiasm and zeal the publicmembers bring to the ACGME. Going to three publicmembers truly has resulted in more energetic oversightof our accreditation processes.

Dr. Friedmann concluded: What we take away from thisinterview is that we need to do more to inform the publicabout the ACGME's role in ensuring "good learning forgood health care," and we should think of ways for thepublic to communicate with us. In all of this, we can takeadvantage of the knowledge and dedication of our publicmembers. Soliciting their participation to the fullest extentwill contribute to our effectiveness.

Resident Retreatsas a Way to FacilitateCommunication and Learning

Jill Klessig, MD

Being providers of medical education requires academicinstitutions to be responsive to the sometimes conflictingneeds of a number of stakeholders. Residents expectthat a defined set of factual information andclinical/professional skills will be taught to them inan effective atmosphere that is sensitive to their needsas learners and as individuals. Outside agencies, includingthe ACGME and the Joint Commission on theAccreditation of Health Care Organizations (JCAHO),future employers, the federal government and the publichave requirements, evaluative procedures and expectationsfor competency that must be met. While meeting thesegoals, the institution must ensure that the educationprovided to residents reflects the practice environmentthat they will enter on completion of training. Thus,institutions need a mechanism that providesthe opportunity to impart the required and/ordesirable facets of education, that can also be usedto evaluate and, when needed, reform the existingtraining environment. Resident retreats, either onan institutional or program level, provide a possiblemechanism for achieving these goals.

The concept of a retreat isnot novel. Industry has usedretreats for years to achievea wide variety of goals.The major goal, regardlessof organization, is improvedcommunication amongvarious participants in anorganization. Medical edu-cators have found residentretreats an effective meansof achieving educational orprogrammatic goals thatare otherwise difficult toaccomplish. The majority ofmedical education programs

that utilize retreats have found them to be valuable, butthey are not without drawbacks.

Goals for Resident RetreatsThe substantial time and financial resources requiredfor a retreat, and the complex logistics of obtainingcoverage for patient care obligations, make it necessary

“The major goal,

regardless of

organization,

is improved

communication

among various

participants in an

organization.”

5

that the program or institution establish goals forresident retreats that justify the resource expenditure.Two major reasons to sponsor a retreat are listed below.

1. Improved social interactions. Residency training isa stressful time in an individual's life, and group supportamong residents and faculty is essential to achievingsuccessful educational outcomes. A retreat allowsresidents and faculty tointeract in a social setting,and emphasizes that they arecolleagues. The more informalsetting may help improve two-way communication anddecrease barriers to effectiveinteractions throughout theyear. This may lead to bettereducational outcomes as wellas improved patient care. Therelaxed atmosphere also allowsparticipants to renew their enthu-siasm for residency education.

2. Enhanced education, evaluation and/or accredita-tion performance. A retreat is an excellent opportunityto cover ACGME/RRC requirements at the program orinstitutional level. Two major areas that can be tied toRRC requirements are education and evaluation. There aremany educational objectives a retreat can accomplish. Forexample, some topics are less well-suited for teaching ina lecture, attending round, or morning report format.A retreat allows for small group sessions and blocks ofeducational time, as well as having all residents in oneplace at the same time. There are some topics, like deathand dying, that can generate strong feelings in residents.The retreat allows residents and faculty to discuss them,and have time to deal with emotions without having toimmediately go back to patient care. Additionally, aretreat provides an excellent opportunity to conductprogram evaluations. Because all residents are present,the program can distribute written evaluation forms andbe assured that all are returned, without breaching confi-dentiality by requiring that the forms be signed. Withresidents and attendings present, the group can engage ineffective dialogue about the positive and negative aspectsof the program, and what should be changed. Smallgroup sessions can be designed to address specificproblems, especially if they are identified ahead of time.Most important, a retreat format, structured well, allowsthe opportunity for all residents to have input into theprogram, not just vocal or dissatisfied individuals.

Barriers to Resident RetreatsOne of the major barriers to a successful retreat is theexpense involved. Sometimes, the available resourcesdictate the format of the retreat, instead of thereverse. It is important to note however, that a verysuccessful retreat can be held with a minimal financialoutlay. Most programs use a combination of fundingschemes for retreats. Common ones include: having

the residents pay all or partof the cost of the retreat; askfaculty to pay an assessment(on a mandatory or voluntarybasis) to defray residentexpenses; take a percentageof private attendings' clinicalincome; solicit non-restrictededucational grants frompharmaceutical companies;or solicit graduates of theprogram for donations. Thesponsoring institution can alsobe asked to pay for the retreat.

However, if the institution pays for one specialty, it shouldexpect to be asked to pay for all.

There are other factors that must be considered. Coverageis an issue that appears to be an insurmountable obstacle,but can be managed with a variety of schemes. If theretreat is sponsored by a program or institution, there alsois concern that the institution may be legally responsiblefor any adverse outcomes, including accidents, inappropri-ate behavior, or the like. Lastly, the quality of the retreat isimportant, because a poor retreat may worsen existingproblems or resident dissatisfaction with the program.

Structuring Resident RetreatsThe timing and location of retreats are important. Anessential component of a retreat is to take all partici-pants out of their normal working environment, andplace them where they can interact in a positive fashion.Many locations are possible, including beaches, hotels,parks, conference centers and other sites. The timing ofthe retreat is to some extent related to specified goals.If the main goal is to allow residents to get to knoweach other, to introduce programmatic changes, or togive specific educational sessions (such as procedureskills), then the beginning of the academic year maybe best. If evaluating the program is an essential compo-nent, the retreat should be held closer to the end.However, if held later in the year, the retreat must bescheduled far enough in advance of the new academicyear that there is still time to implement any changesthat are suggested. The length of the retreat depends on

“The retreat allows residents

and faculty to discuss difficult

issues, and have time to deal

with emotions without

having to immediately

go back to patient care.”

B U L L E T I N O c t o b e r 2 0 0 0

6

the goals, the distance from the institution, and thefunds available. Most retreats are from one to threedays in length.

The content really depends on the goals that theprogram or institution wants to meet. It is essentialthat the question "what is in it for me?" (the resident)is addressed in planning the event. If the contentmerely focuses on the program's or institution's needs,the meeting will not be successful. Potential contentareas include:

(1) Bonding activities, because one of the most importantgoals is to improve communication. Sample activities include group sports, team building activities, "Resident Olympics," etc.

(2) Panel discussions with patients. These could include patients from different religious or cultural back-grounds discussing aspects of their religion that impact patient care; patients who are dying talking about their feelings; or patients who were dissatis-fied with their care discussing the reasons for this.

(3) Panel discussions with the program's graduates whohave entered various career tracks (private practice, HMOs, academics, pharmaceutical industry) andwho discuss he pros and cons of specific practice opportunities. This gives trainees a more realisticpicture of current employment situations than recruiters do. An added advantage of havinggraduates participate is that they can be involvedin the sessions that assess the program, and discuss what they wish the program had taught them.

(4) Workshops to enhance procedural skills.

(5) Mock trials (this is especially good for riskmanagement issues).

(6) Role playing exercises.

(7) Showing commercial movies or TV excerpts toillustrate points of relevance to the program.

(8) Curriculum and program evaluations.

(9) Teaching techniques with sample exercises.

(10)Having the entire group air issues or hold a"gripe" session.

The Outcome of Retreats

In general, retreats are an effective method ofachieving the goals listed above, and in improvingresident satisfaction with the program. They can letthe residents know that they are valued partners inthe educational process. The impact on a program maybe subtle or very profound. Enhanced interactionamong the participants often results in an environ-

ment more conducive to education and patient care.Specific changes, such as new call schedules ordifferent lectures, can be implemented. Theprogram may be betterprepared for an RRC sitevisit, or the retreat canbe used to explain whycertain changes mustbe made to enhancecompliance withACGME/RRC guidelines.However, merely hold-ing a retreat is notsufficient. The mostcritical element isfollow-up. If residents'suggestions and com-plaints are discussed at aretreat, but no action istaken, increased dissatis-faction and anger mayresult. Not only areproblems left unaddressed, but the residentsmay feel that the discussions were solely "show."In contrast, improved resident satisfaction maylead to more enthusiasm for the educationalprocess, which results in more active participationin education, patient care and institutionalactivities, such as JCAHO visits.

In summary, resident retreats can be an effective wayto enhance communications within a program, conductprogram evaluations, implement curricular changes,deliver educational content, and improve morale. Theyallow this to be done in a non-threatening environmentwhere all residents, and many of the faculty, are presentand in which it may be easier to elicit participation.

Complaints AgainstResidency Programs Marsha A. Miller

In 1998 the ACGME decided to test the hypothesisthat the centralization of complaints along with datacollection would demonstrate patterns and make theprocessing and resolution of residency complaints moreeffective. Before this change, complaints were handledseparately by each specialty's RRC Executive Director.Decentralization made it hard to discern patterns. Inorder to make the residency complaint process moreeffective, changes in the Institutional Requirements,and new policies and procedures for egregious violationsand for residency complaints were implemented, and aresidency complaint data base was developed.

“If residents’

suggestions

and complaints

are discussed at a

retreat, but no

action is taken,

increased dissatis-

faction and anger

may result.”

7

The Institutional Requirements were strengthened andmake the protection of resident rights more explicit.The following Institutional Requirements are especiallyimportant to note. They were approved by the ACGMEat its September 26, 2000, meeting and are effectiveimmediately. Look for the newly approved InstitutionalRequirements on the ACGME website (www.acgme.org.)in November. The wording that is bolded and underlinedhighlights the new language.

Additions to the Institutional Requirementsto Enhance Resident Protection

I.B.3.f.4. Establishment and implementation of fair institutional policies and procedures for adjudication of resident complaints and grievances related to actions which could result in dismissal, non-renewal of agree-ment of appointment, or any other action that could threaten a resident's intended career development.

II.C.3.c. Non-renewal of Agreements of Appointment: Institutions must ensurethat programs provide their residents with a written notice of intent not to renew a resident's agreement of appointment no later than four months prior to the end of the resident's currentagreement of appointment. However, if the primary reason(s) for the non-renew-al occur(s) within the four months prior to the end of the agreement of appoint-ment, institutions must ensure thatprograms provide their residents with as much written notice of the intent not torenew as the circumstances will reason-ably allow, prior to the end of the agreement of appointment. Residents must be allowed to implement theinstitution's grievance procedures as addressed in section I.B.3.f(4), when theyhave received a written notice of intent not to renew their agreements of appointment.

II.C.9 Residency Closure/Reduction: All sponsor-ing institutions must have a writtenpolicy that addresses a reduction in size or closure of a residency program. The policy must specify that if an institution intends to reduce the size of a residency program or to close a residency program, the institu-tion must inform the residents as soon as possible. In the event of such a reduction orclosure, institutions must allow residents

already in the program to complete their education or assist the residents in enrolling in an ACGME-accredited programin which they can continue their education.

The ACGME developed new policy and proceduresfor dealing with "Alleged Egregious AccreditationViolations or Catastrophic Institutional Events." Anegregious violation is an occurrence of an accreditationviolation or a catastrophic institutional event thatbecause of its urgency is addressed outside of theestablished complaint process of the ACGME. TheACGME Executive Director, David C. Leach, MDreviews these allegations initially and then decideswhether to convene a meeting with the InstitutionalReview Committee Chair and the RRC Council Chair.This committee determines whether an immediate on-site survey and consultation should occur. Somecomplaints that have been considered egregious are pro-grams that close without notifying the residents in goodtime and fail to assist the residents in locating otherpositions, programs that operate without supervision,and programs that blatantly do not follow grievanceand due process procedures.

The ACGME rewrote its procedures for handling allcomplaints. Especially important in the rewrite is theemphasis placed uponthe ACGME's responsi-bility to monitor and, ifwarranted, take adverseaction against programsthat violate the ACGMEInstitutional and /o rP rogram Requirements,including probationaryaccreditation or with-drawal from the accredi-tation process. Becausethe ACGME can only citeprograms for accredita-tion violations and/or takean adverse action, it isimportant that residentsfirst exhaust all dueprocess avenues withinthe institution before filinga complaint with the ACGME. The ACGME cannot adjudi-cate disputes between program directors and residents, andonce a resident is dismissed from the program the ACGMEcannot get them reinstated. Sometimes the ResidencyReview Committees (RRC) will permit a temporary increasein resident complement to programs willing to take a resi-dent that has been unfairly dismissed. In order for the RRCto grant a temporary increase, the program must be ingood standing and have adequate resources.

“...in the absence

of a clear violation

of an accreditation

requirement, the

ACGME cannot

adjudicate a

dispute between

a resident and

his/her program

director.”

B U L L E T I N O c t o b e r 2 0 0 0

8

As complaints are received at the ACGME, they aredirected to one person who enters the information intoa database. Each Executive Director can still choose tohandle the complaint or leave it with the "complaintadministrator." If handled by the Executive Director, allinformation, including the outcome, is forwarded tothe complaint administrator for data entry.

The centralization and data collection showed resultsand patterns emerged. The data collection from January1998 through September 2000, displayed in Figure 1,shows the result of 86 individuals filing complaints with

the ACGME. Twenty-five of the complaints were anony-mous, and the ACGME does not handle anonymouscomplaints. The stated reason for anonymity is fear ofreprisal and loss of position although the ACGME keepsthe resident's name confidential. Some of the com-plainants' allegations were not singular but included vio-lations of multiple Institutional and/or Programrequirements. Inadequate or no due process and/orlack of grievance procedures topped the chart, withinadequate work environment and excessive duty hourscoming in second. Figure 2 shows the breakdown ofthe number of complaints for specialties that received

——— Figure 1 ———

Types of Complaints Received by the ACGMEJanuary 1998-September 2000

Complaint Frequency*

Inadequate or no due process/lack of grievance procedures . . . . . . . . . . . . . . . . .25

Inadequate work environment and excessive duty hours . . . . . . . . . . . . . . . . . . .23

Lack of evaluation and feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13

Discrimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Poor Educational Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Inadequate supervision or lack of supervision . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Contract Disputes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

*Numbers do not equal total number of complaints, because some dealt with more than one concern.

——— Figure 2 ———

Breakdown of Complaints among Specialties With More than Five Complaints January 1998-September 2000

Specialty Number of ComplaintsInternal Medicine (includes subspecialties) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

Diagnostic Radiology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Surgery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

Emergency Medicine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8

Anesthesiology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Family Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Pediatrics (includes subspecialties) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6

Psychiatry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5

All Other Specialties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11

Total . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .86

9

more than five complaints, again for the period fromJanuary 1998 to September 2000. Internal Medicine,General Surgery and Diagnostic Radiologyhave the highest number of complaints, withEmergency Medicine, Anesthesiology, close behindGeneral Surgery and Diagnostic Radiology.

Due to the nature of the complaints, there was variationin the way they were handled. Some complaints causedsite visits to be scheduled or the cycle shortened, otherswere placed in the program's file for the site visitor topay particular attention to at the next survey, some weredismissed because of lack of evidence, and some are stillin process and not yet resolved.

Two of the complaints during this time period were con-sidered egregious violations; both had to do with lack ofdue process. One program had probation proposed andboth have been scheduled for an institutional review. Asof this writing, the outcome is not yet known.

Also, the ACGME saw for the first time a pattern. Acomplaint regarding lack of due process came from fourresidents in different programs at the same institution.This caused the ACGME to schedule an immediateInstitutional Review.

Centralization of complaints and data collected supportedthe ACGME’s decision to strengthen its procedures forand processing of residency complaints. The proceduresoutlined above have been implemented and can be foundon the ACGME website at www.acgme.org.

Finally, what can programs do to avoid residencycomplaints? It is simple.

• Develop fair institutional policies and procedures,distribute them to the residents and faculty, and mostimportantly, follow them.

• Provide formal written evaluations and promptfeedback.

• Have residents and faculty sign evaluations.

• Document and keep copies of everything in the resi-dent file and permit the residents access to their files.

The ACGME is committed to graduate medicaleducation and to one of its most important stakeholders– the future doctors of America.

Two Frequently Asked Questionsabout the ACGME OutcomeProject: Time-Line andMinimum LanguageSusan Swing, PhD and Patricia Surdyk, PhD

Time LineThe ACGME is continuing with the phasing-in of itsaccreditation system based on educational outcomes.The Council determined that language addressing thegeneral competencies must be placed in all Programand the Institutional Requirements by July 1, 2001.RRCs may either opt to use the ACGME's MinimumLanguage for the General Competencies, or maydevelop their own language for the competencies andevaluation requirements. The latter must be approvedby the ACGME's Program Requirements Committee.For specialties in which language related to the generalcompetencies has not previously been present in theRequirements, the period from July 1, 2001 to June 30,2002 will allow programs time to phase in the revised

requirements. During this period, these efforts maybe highlighted in site visit reports as "best practices,"but programs will not be held accountable for theadded education and evaluation requirements createdby the competencies. They will not be part of theformal accreditation review. This will provide RRCsand programs some added time to plan for implemen-tation of the competencies and development andincorporation of a growing number of increasinglymore dependable assessment tools.

“During this period,

these efforts may be highlighted

in site visit reports as “best practices,”

but programs will not be held

accountable for the added education

and evaluation requirements

created by the competencies.”

B U L L E T I N O c t o b e r 2 0 0 0

10

Minimum LanguageThis next section provides answers to somefrequently asked questions about the "MinimumLanguage."

Educational Program"The residency program must require its residents toobtain competencies...to the level expected of a newpractitioner" reflects the essence of the OutcomeProject. Not only are residents expected to engage ineducational activities, but, the learning objectives ofthose activities must be achieved.

EvaluationA great deal of background work is currently underwayto help RRCs determine what an "...effective plan forassessing resident performance" might look like in thevarious specialties in which training is accredited bythe ACGME, and which assessment methods might beconsidered "dependable." Within the next severalmonths, results of a joint initiative between the ACGMEand the American Board of Medical Specialties (ABMS)and discussions of the Outcome Project Advisory Groupconcerning model assessment systems will becomeavailable. In addition, the aim of the "Toolbox ofAssessment Methods" is to identify "dependable[assessment] measures."

The statement: "Programs that do not have a set ofmeasures...must demonstrate progress in implementingthe plan" acknowledges that many programs will needtime to do an acceptable job of implementing evalua-tion methods that are increasingly more dependableand useful for improving their educational program.This requirement allows programs to phase inimprovements in evaluation as an expected "outcome"of the Outcome Project.

A major goal of the Outcome Project is to define andimplement an expanded role for continuous improve-

ment in residency programs. Thus, the section entitled"Program Evaluation" outlines the expectation that theresidency program "should have in place a process for

using resident and performance assessment resultstogether with other program evaluation results toimprove the residency program."

Both the Full version and Minimum Languageversions of the competencies, answers to otherfrequently asked questions and a copy of the firstedition of the "Toolbox" can be found by selectingthe Outcome Project link on the ACGME web siteat www.acgme.org. In the weeks to come, this site willcontinue to grow with additional resources to assistprograms in their efforts to integrate the generalcompetencies and improved assessment methodsinto their educational planning.

“A major goal of the

Outcome Project is to define

and implement an expanded role

for continuous improvement in

residency programs.”

11

R R C / I R C C O L U M NR R C / I R C C O L U M NACGME Approves Revised Institutional Requirements The ACGME adopted the revisions of the Institutional Requirements at its September 26, 2000 meeting tobecome effective immediately. The majority of the changes helped clarify existing requirements, particularly thosethat pertain to the internal review process. A few noteworthy changes are shown in Exhibit 1 below.

——— Exhibit 1 ———Changes and Additions to the Institutional Requirements

Effective September 26, 20001. The GME Committee is charged with the responsibility for reviewing on a regular basis all ACGMEletters of accreditation and monitoring correction plans. (I.B.3.c.)2. Regular internal reviews must be conducted of all subspecialty programs. (I.B.3.d.)3. The internal review must be conducted by the GMEC, or a body designated by the GMEC, which mustinclude faculty, residents, and administrators from within the institution but from programs other than theone that is being reviewed. (I.B.3.d.(1)).4. All internal reviews are to be conducted at approximately the midpoint between ACGME programsurveys. (I.B.3.d.(2)).

Aside from these clarifications, there were three new additions:1. All individuals involved in GME (administrators, residents and faculty, etc.) must have access toadequate communication technologies and technological support to include at least computers andaccess to the internet. (I.B.)2. If an institution intends not to renew a resident's contract, it must ensure that the resident is notified inwriting no later than four months prior to the end of the resident’s current contract. However, if the intentnot to renew a contract occurs within the four months prior to the end of the contract, institutions mustensure that programs provide the resident with as much written notice of the intent as circumstances willreasonably allow prior to the end of the contract. Residents must be allowed to implement the institution'sgrievance procedures as addressed in section I.B.3.f.(4) when they have received a written notice of intentnot to renew their contracts. (II.B.4) 3. All sponsoring institutions must have a written policy that addresses professional activities outside theeducational program to include moonlighting that is in compliance with requirement II.C.11.

Changes in the Requirements for Family Practice and PediatricsFor Family Practice, the revisions, including the incorporation of the ACGME competencies that wereavailable on the Website for comment earlier this year and that were discussed at the Program DirectorWorkshop in June, were approved by ACGME in September 2000. A workshop on the practical application of the competencies in programs in Family Practice and Pediatricswill be included in the ACGME Mastering the Accreditation Workshop in March 2001.

Other Changes in Program RequirementsThe ACGME approved revisions to the Program Requirements for Radiation Oncology, Obstetrics-Gynecology, Preventive Medicine, and minor revisions to the Program Requirements for three subspecialtiesof Internal Medicine — Hematology, Hematology-Oncology, and Nephrology. The Council also approved the addition of the General Competencies language to the following ProgramRequirements: Colon and Rectal Surgery, Family Practice, Neurology and Pediatrics.The new requirements, included those with the addition of the General Competencies language, will becomeeffective July 1, 2001. All new program requirements can be found at the ACGME's address on the WorldWide Web (Http://www.acgme.org). Program requirements that have been approved but are not yet in effect,are located within each RRC web page, under the subheading "Approved but not currently in effect.”

B U L L E T I N O c t o b e r 2 0 0 0

12

Other Highlights from theSeptember 2000 ACGME Meeting

Election of ACGME Officers and Recognitionof Outgoing Directors The ACGME elected the following Officers of theACGME for 2001: Daniel H. Winship, MD, AAMC(Vice-Chair), Richard Allen, MD, AMA (Treasurer),John I. Fishburne, Jr., MD, CMSS (Officer), D. DavidGlass, MD, ABMS (Officer).

R. Edward Howell, Chief Executive Officer of theUniversity of Iowa Hospitals and Clinics, and incomingACGME Chair, recognized the contributions to theACGME made by Paul Friedmann, MD, ACGME Chairwhose term ended at this meeting. The ACGMEalso recognized the contributions of theother ACGME Directors whose termsended: David L. Nahrwold, MD, whoserved as a member of the ExecutiveCommittee and Chair of the Committeeon Strategic Initiatives; Carol Berkowitz,MD, who serves as Chair, RRC Council ofChairs; Charles E. Allen, MD, F. StephenLarned, MD, who also served as Chair ofthe Monitoring Committee; Ellison C.Pierce, MD, and Stephen J. Thomas, MD,who completed his term as Vice-Chair,RRC Council of Chairs.

Follow-up on Duty Hour CitationsThis year marked the first time theACGME published data on the frequencyof citations for violations of the require-ment governing resident duty hours.(see ACGME Bulletin, April 2000) Acrossmany of the major disciplines, 20 to 30percent of programs were cited in 1999for violating duty hour requirementsestablished by groups of their peers. Inresponse, the ACGME has decided tostrengthen its mechanisms for respond-ing to work hour citations, by requiringprograms that receive this citation, andtheir sponsoring institution, to respondto the citations immediately with a planfor corrective action.

ACGME Fee Structure for 2001The ACGME approved the revenue andexpense budgets for 2001. The accredi-tation fees for the year 2001 will remainthe same as those for the year 2000.The fee for programs with five residents

or more will thus remain at $2,500 per year; fees forprograms with less than five residents and for inactiveprograms will continue to be $2,000 per year.Application fees remain at $3,000 per application; feesfor late cancellations of a scheduled site visit will contin-ue to be $2,000; and fees for late notice of intent tovoluntarily withdraw a program or place it on inactivestatus will remain at $1,000.

J-1 Visa Holders Training in Non-ACGMEAccredited ProgramsThe ACGME approved the content of a standard letterto be sent to programs that have individuals on J-1 Visaswho want to extend their Visas for additional training ina program not currently accredited by the ACGME orwhere training is not approved by an ABMS memberboard. These programs frequently contact the ACGME

R. Edward Howell (left), incoming Chair of the ACGME, receives the gavelfrom Paul Friedmann, MD, Senior Vice President of Academic Affairs,

Baystate Medical Center, and outgoing ACGME Chair.

13

to request a letter endorsing this action for a particularindividual who wants this added training. The ACGMEfeels that such an endorsement could imply an act ofaccreditation for these programs, which is not the case.The standard response clarifies this for any programrequesting such a letter.

Update on the ACGME Web-BasedAccreditation System The ACGME's Web-Based Accreditation System, whichwill collect programs' and institutions' demographicinformation for the accreditation process, is up and run-ning. Data gathering began on October 9, 2000 at theinstitutional level, with data being collected from twentypreviously identified institutions. This will be followed byphased implementation across the various specialties,beginning mid-November 2000. Data collection will bean ongoing process, and will provide the ACGME withdata submitted in "real time."

RFP Process Results in Submission of MoreThan 65 ProposalsThe Request for Proposals 2000 (RFP 2000) Project to datehas resulted in the receipt of more than 65 proposals.Proposals will be evaluated throughout the comingmonths, and the ACGME plans to highlight the RFP2000 Project submissions in a special issue of theACGME Bulletin, to be published in early 2001.

RRC Resident Council The ACGME approved the recommendation toestablish an RRC Resident Council whose membershad been meeting on a semiformal basis as a standingcommittee of the ACGME. The Chair of the new RRCResident Council will henceforth be invited to attendall ACGME meetings.

September 22, 2000 Joint Meeting of the ABMSand ACGMEA meeting of the ACGME and the American Board ofMedical Specialties was held on September 22, 2000 tocontinue the work of the specialty-specific "Quadrads,"composed of a Program Director, a Resident, an RRCChair and a Medical Specialty Board Member. TheQuadrads worked throughout the summer to developdiscipline-specific language and an initial set of evalua-tion tools for each of the six competencies. The daywas used to determine which competencies spanned alldisciplines and what assessment tools could be used toassess competencies. The work of the Quadrads willbe completed by the end of October 2000.

ACGME Bulletin Editor's Occasional Column:

Analyzing Decision-Makingin Groups - Findings from theManagement Literature withApplication for Medical EducationIngrid Philibert

“Decision-making is a highly contextual, sacred activity surrounded by myth and ritual, and as much concerned with the interpretive order as with the specifics of particular choices." - James G. March

Team-based approaches in health care have increasinglybecome the norm across a wide range of settings. Atthe same time, use of teams in other work settings hasproduced a body of literature on team decision makingand what characterizes successful teams. Characteristicsfrequently mentioned include closed-loop communication;differentiating roles while compensating and backing-upindividual functions; mutual performance monitoring;and cooperative interac-tion toward the pursuitof shared objectives(Kraiger and Wenzel,1997). Most researchstudies "permanentteams," and many charac-teristics of effective teamsare tied to team longevity.Most health care teamsare relatively stable, butsome settings – such asemergency rooms andteaching hospitals – arecharacterized by the useof fluid and flexible"transient teams." Whatfrequently also charac-terizes these settings is a higher degree of complexity,ambiguity and stress. In teaching settings, the presenceof learners adds to complexity, because teaching iscarried out simultaneously with patient care, and addsto ambiguity because learners have to 'learn by doing,'by actively participating in diagnosis and treatmentdecisions being made.

In an article in which he discusses the limits of"sense making" in organizations, Karl Weick (1993)noted that how transient teams deal with difficult,dangerous situations is timely because he states,"the work of organizations is increasingly done in

“...some settings –

such as emergency

rooms and teaching

hospitals – are

characterized by

the use of fluid

and flexible

transient teams.”

B U L L E T I N O c t o b e r 2 0 0 0

14

small temporary outfits in which the stakes are highand where foul-ups can have serious consequences."

It is probably not realistic to expect to find the charac-teristics of successful permanent teams in their tran-sient counterparts. Weick is thus interested in howtransient teams make decisions. Earlier research hadfocused on the effectiveness of techniques thatemphasize rigorous debate among the group mem-bers holding opposing opinions produce decisionsthat are superior to those produced by a harmoniousgroup. Techniques like Devil's Advocacy or DialecticInquiry have been used in management teams(and were incorporated into the decision-makingrepertoire of John F. Kennedy's Cabinet af ter thedisastrous "group think" that produced the Bayof P igs invas ion) . The two techniques differ, butboth help a team to identify and address diverseassumptions that underlie a given issue.

But these techniques require a more permanentteam and time for formal decision-making processes.Weick's focus is on temporary teams andsituations of significant complexity requiring rapiddecisions. He points out that research in the field

has shifted from lookingat decision-making in theclassical sense to anemphasis what he termssense making; the roleof power in decisions;and models thatconsider interpersonaland social relations.Sense making isdefined as an approachthat views reality as "anongoing accomplishmentthat emerges fromefforts to create orderand make retrospectivesense of what occurs."

He analyzes sense making, or rather its limits, usingthe example of a temporary team of U.S. ForestService firefighters fighting the Mann GulchDisaster, made famous by Norman McLean's1992 Book "Young Men and Fire." Based on this,Weick suggests four ways in which a temporaryteam's resilience in complex situations can bestrengthened: (1) improvisation, (2) virtual rolesystems, (3) a new approach toward "wisdom,"and (4) respectful interaction.

The ability to improvise becomes more important asthe uncertainty of a situation increases. Yet, Weickstates when individuals are put under pressure, theyrevert to their "most habituated ways of responding."

He adds that individuals who "habitually" live onthe edge, where improvisation is a necessity, excelin it for just that reason. He defines a virtual rolesystem as each individual being competent in all theroles in a team, to allow them to "assume whateverrole is vacated, pick up the activities, and run acredible version of the group." His definition of"wisdom" is overcoming the limitations posed bythe firefighters' own definition of their purpose,which was "putting out fires so fast they cannotbecome large fires." Thus, they had little experi-ence and little perception of themselves havingexperience in fighting large fires. The parallel tomedicine here is particularly poignant.

In defining his fourth recommendation, respectfulinteraction, Weick suggests that focusing onsocial relationships and networks in difficult situ-ations enables “social construction,” a combinedsearch for meaning. In another article on trust intemporary groups, Weick and collagues (1996)proposes that trust can be generated in transientteams, and that attentiveness to social structuresis vital in this. Individuals "trust" the role theother has been recruited to play – emergencyroom physician, anesthesiologist – and his/hercompetency to carry out the role's responsibili-ties. But this 'role confidence' can be shaken,when the circumstances do not fit the traditionalpatterns. McLean's book illustrates this point. In deal-ing with a grass fire on a ridge, the firemen who aretrained on forest fires, initially refuse to leave behindtheir axes, useless in a grassfire and a weight and hin-drance in climbing the ridge. When the foreman, whowas unfamiliar to most of the team, does somethinghighly unusual – lighting an escape fire, which none ofthe men had seen before – the men refused to joinhim. They did this because they did not know him andthus did not trust his decisions, and "there was nottime to change this," nor was there time for him toexplain. Most of them burn to death. He survives.

“Weick’s

focus is on

temporary teams

and situations

of significant

complexity

requiring rapid

decisions.”

“...to use the various abilities

of the members of a team

effectively requires the group

to identify and synthesize

the varying skills of the members

and use those most appropriate

to the decision at hand.”

A Clarification from the Editor:In the June 2000 ACGME Bulletin, we published a letter by Dr. Jonathan Rhoads, commenting on resident workhours. In my response to Dr. Rhoads, I provided a general restatement of the ACGME standards on work hours.Since that general statement did not reflect the specific standards of the various RRCs, we received a few questionsabout how the ACGME considers the requirement that residents "on average, have one day off in seven, and areon-call no more than every third day."

My general restatement inadvertently left off "on average," making the one day off in seven appear to be anabsolute. This resulted in some confusion and it is worthwhile to clarify this requirement by stating that theRRCs and the Institutional Review Committee treat this requirement by aggregating a two-week or four-weekperiod to produce the average. Averages aggregated across periods longer than that, e.g., situations whereresidents are not permitted four days off per month in some sequence of days, can result in the program beingcited for non-compliance.

In addition, a few questions arose because some specialties, like Surgery and Neurological Surgery, phrase thisrequirement as a "desirable." For example, the Neurological Surgery requirements are as follows: "It is desirablethat residents' work schedules be designed so that, on average, excluding exceptional patient care needs,residents have at least one day out of seven free of routine responsibilities and be on-call in the hospital no morethan every third night." This produced questions on whether these disciplines place less value on the require-ment. It is useful to remember that programs can be cited for failing to comply with a "desirable" requirement.The glossary defines "desirable" as:

A term, along with its companion, "highly desirable," used to designate aspects of an educationalprogram that are not mandatory but are considered to be very important. A program may be citedfor failing to do something that is desirable or highly desirable.

What this unlikely story (yet, frighteningly, manyof us can think of parallel examples from medicine)demonstrates is that, to use the various abilities ofthe members of a team effectively requires the groupto identify and synthesize the varying skills of themembers and use those most appropriate to thedecision at hand.

Weick supports his suggestions with the findingsof related research in each of the four areas. Yet whatremains to be tested is the hypothesis whetherimprovisation, virtual role systems, respectful interaction,and defining "wisdom" as understanding the complexityof unknown domains, are effective in facilitating informa-tion processing in temporary teams in critical situations.Weick's recommendations are not specifically intended toaddress decision-making in medical teams, and need tobe adapted to the medical setting, where team functionscannot easily be exchanged. Other work has explored theimpact of individuals on teams having distributedresources, including specific 'static' knowledge andexpertise. This could be knowledge in a medical discipline,nursing or another field, combined with role expectationson the part of the individual and his or her team mem-bers. Decisions in health care frequently still are not thedomain of the team, but are made by a single individualwith primary responsibility for the patient (the "treatingphysician"), often with input from the group. The dynam-ics of information transfer in these 'input' situations likely

differ from the examples discussed here. At the sametime, for decision-making that truly involves a team, thetwo closest models may be the emergency room, where ateam collectively decides how to diagnose and manage acritical patient and the teaching hospital, where residentsmust learn how to independently diagnose and managepatients but where care must be under the supervisionand coordination of a faculty physician.

Sources:Kraiger, K. and L. H. Wenzel. Conceptual Development and EmpiricalEvaluation of Measures of Shared Mental Models as Indicators of TeamEffectiveness. In Team Performance Assessment and Measurement. M.T. Brannick, E. Salas and C. Prince (Editors). Mahwah, NJ: LawrenceErlbaum Associates, 1997, 63-84.

Kramer, R. M. Divergent Realities and Convergent Disappointments inthe Hierarchic Relation: Trust and the Intuitive Auditor at Work. In Trustin Organizations, R. M. Kramer and T. R. Tyler, Editors, Thousand Oaks,CA: Sage, 1996, 216-245.

Larson, J. R., C. Christensen, T. M. Franz, A. S. Abbott. DiagnosingGroups: The Pooling, Management and Impact of Shared andUnshared Case Information in Team-Based Medical Decision-Making.Journal of Personality and Social Psychology 75:1, 1998: 93-108.

McLean, N. Young Men and Fire, Chicago: University of ChicagoPress, 1992.

Meyerson, D., Weick, K. E., and Kramer, R. M. (1996). Swift trust andtemporary groups. In Trust in organizations, R. M. Kramer and T. R.Tyler, (Eds.), Thousand Oaks, CA: Sage: 166-195.

Weick, K E. The Collapse of Sensemaking in Organizations: The MannGulch Disaster. Administrative Science Quarterly 38, 1993: 628-652.

15