Common Threads Among Catastrophic Mishaps

LessonsNotLearnedVulnerableDesign

WorkmanshipShortcomingsProcessControlFailures

FailuretoControlCri=calMaterialItemsFraud

BrianHughiB OfficeofSafetyandMissionAssurance

PrecursorEventIneffec/veCorrec/veAc/on

VulnerableDesign

Fraud Workmanship/ProcessControlShortcomings

MaterialControlInadequacies

BigDigTunnelCollapse

X X X X

TurkishAirlinesFlight981

X X X

USSThresher

X X X X

Apollo1

X X X X

USSIwoJima

X

Orbi=ngCarbonObservatory/Glory

X X X X

AntaresOrb-3

X X

Falcon9-20

X2

•  7.5milecorridor•  161lanemiles•  5milesoftunnel•  6interchanges•  200bridges•  541,000truckloadsofdirt•  3.8millioncubicyardsofconcrete

Originallyscheduledtobecompletedin1998atacostof$2.6billion, theprojectwasfinallycompletedin2006atacostof$14billion.

3

The Big Dig

OnSeptember9,1999,aconstruc=oncontractoremployeeinstallingven=la=onductworkoverthetunnelceilingno=cedthatseveraloftheanchorshadbeguntopullout.

The Smoking Gun

OnNovember12,1999,aproofloadtestwasperformedononeoftheanchorsthathadshownsignificantnine-sixteenths-inchdisplacement.Theengineernotedthat“theboltheldforafewseconds,thenbegantopulloutwithalmostnoresistance”.

4

The Gi;

5

The Supplier’s Response

Whenthesupplierwascalledtoexaminetheanchordisplacements,theyseemedsurprisedthattheanchorsthathadbeensuccessfullyprooftestedonlyafewmonthsbeforecouldbefailing.Installa=onproblems(e.g.,excessivepreload)werepostulatedasthecause.

Noevidencewasfoundthatthesuppliertookanyfollow-upac=onadertheexamina=on.

-Nofurthertes=ng

-Nofurtherresearch

“At least some supplier officials were aware that their Fast Set epoxy was subject to creep, but this information was apparently not considered or was not known by the representatives who evaluated the failed anchors. Even if the information about poor creep resistance was not common knowledge, a reasonable amount of research would likely have revealed it. The Safety Board would have expected the supplier of a safety critical component to have been more proactive in determining why its product was failing.”

The Builder, Design Agent, and Project Manager Reply

Increasedproofloadtes=ng

Therootcauseforthehangerdisplacementwasneveriden=fied……andsurveillancemonitoringinspec=onswereneverimplemented.

20,000#

15,000#

10,000#

5,000#

0

LbF

Design Service Load Post Installation

Proof Test Finite Element Analysis

2,600 #

Calculated design service load

(2,600 Lb-Force)

20,000#

15,000#

10,000#

5,000#

0

LbF

Design Service Load Post Installation

Proof Test

Finite Element Analysis

2,600 # 3,250 #

After each bolt was installed, a proof test was conducted at 25% higher than design service load

(3,250 Lb.- Force)

20,000#

15,000#

10,000#

5,000#

0

LbF

Design Service Load Post Installation

Proof Test Finite Element Analysis

2,600 #

6,350 #

3,250 #

Later, after slippage was noted, bolts were proof tested to the maximum allowable load

(6,350 Lb-Force)

15,000#

10,000#

5,000#

0

LbF

Design Service Load Post Installation

Proof Test Finite Element Analysis

2,600 # 2,371 # 2,823 #

6,350 #

3,250 #

A finite element analyses determined that the load would be between 2,371 and 2,823 lb force

"You’ve noted the key piece of information that is missing. That is the

cause of the anchor failure and how the repair procedure will overcome that… We are not trying to hold up construction, we are trying to make a determination that the installation is safe…” Design Manager e-mail concerning response to Deficiency Report

“Glaringly absent from the Deficiency Report is any explanation why the anchors failed and what steps are proposed to ensure that this problem does not reoccur.” Structural Engineer e-mail reply

The Gi; (Part 2)

OnDecember17,2001,aqualitycontrolinspectorsubmiBedaNoncomplianceReport,whichstated:

“Severalanchorsappeartobepullingawayfromtheconcrete.Thesubjectanchorswerepreviouslytestedtotherevisedvalueof6,350pounds,allofwhichpassed.[…]Reasonforfailureisunknown.”

“Atthispoint,itshouldhavebeenobvious[…]thattheremedythathadbeendevelopedinresponsetotheanchordisplacementinthe[HighOccupancyVehicle]tunnelin1999hadnotbeeneffec/ve,asanchorsthathadpassedprooftes/ngathighervaluesweres/lldisplacing.Thiswasanotheropportunityto[…]inspectalltheinstalledanchorstodeterminetheextentand,moreimportantly,thecauseoftheanchordisplacement.Instead,thecompaniesapparentlyconsideredthecon/nuingfailuresasisolatedinstancesandtooknoac/ontoaddresstheprobleminasystemicway.”

NTSBAccidentReport

13

Accident Synopsis

At11p.m.onJuly10,2006,a1991Buickpassengercaroccupiedbya46-year-oldmaledriverandhis38-year-oldwifewastravelingeastboundintheI-90connectortunnelinBoston,MA,enroutetoLoganInterna=onalAirport.Asthecarapproachedtheendoftheconnectortunnel,asec=onofthetunnel’ssuspendedconcreteceiling(26tons)detachedfromthetunnelroofandfellontothevehicle,crushingitsrightside.

14

ProximateCause

Useofanepoxyanchoradhesivewithpoorcreepresistance

15

NTSB Accident Report

Creep

ASTMD2990-01 Sincetheproper=esofviscoelas=cmaterialsaredependenton=me…aninstantaneoustestresultcannotbeexpectedtoshowhowamaterialwillbehavewhensubjectedtostressordeforma=onforanextendedperiodof=me.

Epoxyisapolymeranditss=ffnessis=meandtemperaturedependent.Ifaloadisappliedsuddenly,theepoxyrespondslikeahardsolid.Butifthatloadisthenheldconstant,themoleculeswithinthepolymermaybegintorearrangeandslidepastoneanother,causingtheepoxytograduallydeform.Asthedeforma=onincreases,itbecomesirreversible.

Epoxy Secured Bolts

CreepTheUnknownKnown

20

“At least some supplier officials were aware that their Fast Set epoxy was subject to creep, but this information was apparently not considered or was not known by the representatives who evaluated the failed anchors. Even if the information about poor creep resistance was not common knowledge, a reasonable amount of research would likely have revealed it. The Safety Board would have expected the supplier of a safety critical component to have been more proactive in determining why its product was failing.”

NTSB Accident Report

“Thisaccidentinves=ga=onrevealedastrikinglackofawarenessamongdesigners,contractors,managers,andoverseersaboutthenatureandperformanceofpolymeradhesives,evenasthoseadhesiveswerebeingapprovedforuseanapplica=onswhereafailurewouldpresentanimmediatethreattothepublic.

Evenaderbeingpresentedwithevidenceofanchorcreep,projectmanagersandoverseersfailedtorecognizetheinherentweaknessintheepoxyadhesive–aweaknessthatcouldnotbeovercomeevenwiththebestinstalla=onprac=cesorthemostrigorousshort-termprooftes=ng.”

NTSBAccidentReport

TheUnknownKnown(cont)

CogniGve Dissonance

From Wikipedia:

•  A psychological term describing the uncomfortable tension that may result from having two conflicting thoughts at the same time, or from engaging in behavior that conflicts with one's beliefs, or from experiencing apparently conflicting phenomena.

•  In simple terms, it can be the filtering of information that conflicts with what

you already believe, in an effort to ignore that information and reinforce your beliefs.

Wikipedia

Accidents related failures of imagination

Failure of imagination has been invoked in regards to the Apollo 1 fire by astronaut Frank Borman in 1967 when he spoke at the Apollo 1 investigation hearings. It has also been mentioned in reference to design flaws in the RMS Titanic and … the failure of the United States to anticipate the attack on Pearl Harbor.

Failure of imagination From Wikipedia: A failure of imagination is a circumstance wherein something seemingly predictable (particularly from hindsight) and undesirable was not planned for.Failure of imagination is related to unknown unknowns.

ContribuGng Causal Factor

- Inadequate Industry Standards -

ICCAC58:

Eitheradesignsafetyfactorof5.33ora120-daycreeptestisrequiredforFastSetepoxy.

“Given that the ability to sustain a load over a period of time is a typical requirement for almost any type of fastener, the Safety Board is concerned that the ICC has allowed creep testing of epoxy adhesives to be optional. A design engineer should be provided with all of the relevant information about a product before it is used in a safety critical application.” NTSB Accident Report

Consequently…

Tosupportproductqualifica=on,thesupplierprovidedanEvalua=onReport(ER)whichincludedbondstrengthtablesspecifyingasafetyfactorof5.33forFastSetepoxy-nottheresultsofcreeptests*.

*TheSafetyBoardlearnedduringtheinves=ga=onthatFastSetepoxyhadbeentestedforcreep

performancein1995and1996andhadfailedtomeetthestandard

26

27

DesGnaGon Disaster

OnMarch3,1974,TurkishAirlinesFlight981,onarou=neflightfromParistoLondon,crashedinadenseforestinFrance,resul=nginthelossofall346personsaboard.At11,500feet,thedifferen=alpressureinthecabincausedtheadcargodoortoopenandbeblownoff.Thelargeholesuddenlyappearinginapressurehullcreatedanoutwardaccelera=onofairsorapidastoresembleabombexplosion.

Theexplosiondestroyedtheflooringabovethecargohold,severingthecontrolcablesfortherudder,theelevators,andthenumbertwoengine.

28

The proximate cause of the accident was determined to be a faulty latch.

29

30

LatchDesign

“Thereweremul=plecomplexlinkagesbetweentheexternalhandleandthelockingpinbarwhich,inaggregate,werefartooweakandflexible.Ratherthanencounteringanirresis=bleforceifthelockingpinshitthelugsofanunclosedlatch,abaggagehandlerofnormalstrengthcouldpushthehandlefullydown,thinkingthathehadthusinsuredtheclosingofthedoorwhenallhehaddonewasbendtheinternalbarsandrodsoutofshape.”

31

“Itwas,byanysenseofsafetyengineering,agimcrackpieceofdesign.Yet,becauseofdecisionstakenaboutfloorstrengthandcontrol-cableroutes,thesafetyofeveryman,womanandchildwhowentaboardtheDC-10wasdependentupontheefficacyofthelinkagesfromthemomenttheplanewentintoservice.”

PaulEddy

Des=na=onDisaster

32

Contribu=ngCausalFactors DesignChoices

•  DoorConfigura=on•  Rou=ngofcables,hydrauliclines&wire•  Floorstrength•  Latchdesign•  Cockpitindicatorlight•  Ventdoordesign

33

FMEA

In the summer of 1969, Douglas asked Convair to draft a FMEA for the lower cargo door system of the DC-10. Convair produced a document which accurately foresaw the deadly consequences of a cargo-door latch failure. But neither Convair’s draft FMEA, nor anything closely resembling it, was ever shown to the FAA.

The Gi;

OnMay29,1970,duringgroundtes=ngofShip1toprepareitforitsupcomingmaidenflight,theaircondi=oningsystemwasbeingexercisedtobuildupapressuredifferen=alof4to5poundspersquareinch.Suddenly,theforwardlowercargodoorblewopencausingalargesec=onofthecabinfloortocollapseintothehold.

McDonnellDouglasaBributedtheincidentalmosten=relytohumanfailureonthepartofthebaggagehandler.

34

35

TheApplegateMemorandum:

“Thepoten=alforlongtermConvairliabilityhasbeencausingmeincreasingconcernforseveralreasons…theairplanedemonstratedaninherentsuscep=bilitytocatastrophicfailurewhenexposedtoexplosivedecompressionofthecargocompartmentin1970groundtests…Itseemstomeinevitablethatinthetwentyyearsaheadofus,DC-10cargodoorswillcomeopenandcargocompartmentswillexperiencedecompressionandIwouldexpectthistousuallyresultinthelossoftheairplane.”

F.D.Applegate DirectorofProductEngineering

Convair

The Gi; (Part 2)

OnJune12,1972,AmericanFlight96departedDetroit,MIandwasclimbingthrough11,750feetwhentherearcargodoorblewoutcausinganexplosivedecompressionandlossofflightcontrols.ThecrewmanagedtoregaincontroloftheplaneandreturntoDetroit.

“Thedesigncharacteris=csofthelatchingmechanismpermiBedthedoortobeapparentlyclosed,when,infact,thelatcheswerenotfullyengagedandthelockpinswerenotinplace.” Na=onalTransporta=on

SafetyBoardAccidentReport(NTSB) February28,1973

36

TheMidnightGentlemen’sAgreement:

ThepresidentofDouglasspersuadedtheFAAAdministratorthatcorrec=vemeasurescouldbeundertakenasaresultofagentleman’sagreement,therebynotrequiringtheissuanceofanFAAAirworthinessDirec=ve.

“ When you have a well–constructed state with a

well-framed legal code, to put incompetent officials in charge of administering the code is a waste of good laws, and the whole business degenerates into farce.”

Plato Laws (Book IV)

PertheGentlemen’sAgreement,Douglasissuedtwoairlineservicebulle=ns:

1.  Installapeepholeandadecalshowingdiagramma=callywhatthehandlerwouldseeifthelockingpinwassafelyhome.IssuedasaSafetyAlert.

2.Installasupportplatetoholdupthetorquetubejustinsidethehandle.Issuedasarou=neservicebulle=n.

PlanningDepartmentrecordsclearlyshowthatonJuly18,1972,threeinspectorsseeminglyappliedstampsindica=ngthatthesupportplatehadbeeninstalledandthelocktubehadbeenmodified.Thesethreemenwerebroughtforwardandexaminedunderoath.ItemergedthatnotoneofthesethreecouldrecallhavingworkedonthecargodoorofanyDC-10atany=me.Norcouldtheyrecallonanyoccasionwhateveronwhichtheyhadworkedtogether.

Douglasmaintainedtotheendthathumanerrormustaccountforthefalsityoftherecords.

40

A Clear Case of Fraud

“Becausehistoryisanunrepeatableexperiment,wecannotprovethattheextraurgency,legalweight,andpublicitywhichgowithAirworthinessDirec=veswouldnecessarilyhavemadethedifference.Butthecrucialpointisthedetermina=onontheFAAAdministrator’spartthattheDouglascompanyitselfcouldbeledtohandlethemaBerinitsownway.”

PaulEddy

Des=na=onDisaster

Apollo1CommandModule

FirstinItsClass

Itwasmuchlargerandmorecomplexthananypreviousdesign:theleadingedgeofU.S.

spacecradtechnology.42

The Loss of Two Technological Marvels with All Crew Aboard

USSThresher

FirstinItsClassItwasfast,quietanddeepdiving:theleading

edgeofU.S.submarinetechnology.

OnApril10,1963,whileengagedinadeeptestdive220mileseast

ofCapeCod,MA,theUSSThresher

waslostatsea,seBlingatadepth

of8,400feetwithallaboard.

Intheend,112navalofficersand

enlistedpersonneland7civiliansperished.

43

Loss of the USS Thresher

Loss of the Apollo 1 Command Module

OnJanuary27,1967,theApollo1crewenteredthespacecradtoperformalaunchcountdownrehearsaltest.Thetestcommencedwithinstallingthehatchdoorandpurgingthecabinwitha100percentoxygenatmosphere.

Pad34:SiteoftheApollo1Fire

Hourslater,asparkfromfaultywiringinsula=onignitedafireconsuminganabundanceofflammablematerialsinthecockpit.Thefirecreatedanoverwhelmingpressureagainstthehatchdoor,sealingthecrewinside.

Grissom,White,andChaffee

44

USSThresher

Improperlybrazedpipejoint

Apollo1

“Theboardfoundnumerousexamplesinthewiringofpoorinstalla=onand

poorworkmanship.”

PoorlybrazedpipesledtotheelectricalshortagethatledtothelossoftheUSSThresher

Wireswherethefirewassuspectedtohavestarted

45

ContribuGng Causal Factors Inadequate Workmanship

Apollo1KennedySpaceCenterinspectorscitedmul=pleinstancesofdeficientparts,equipment,andworkmanship.

46

ContribuGng Causal Factors IneffecGve CorrecGve AcGon

USSThresherPortsmouthNavalShipyardinspectorsusingnewlydevelopedultrasonictes=ngtechniquesiden=fiednumerousinstancesoffaultybrazedjoints;however,manybrazedjointsontheUSSThresherwerenevertestedusingthenewtechnique.

Apollo1•  Pureoxygenatmosphere

•  Combus=blematerials

•  Inwardopeninghatch•  Inadequateescapeprovisions

47

ContribuGng Causal Factors

Vulnerable Design Inadequate Emergency Recovery

Unforeseen Failure Mode

USSThresher•  Reactorshutdown•  Impairedaccesstovitalequipment

•  Compromisedballasttankblow

WreckagefromtheUSSThresher’ssonardomecanbeseenontheoceanfloor

Fallen Astronauts

Thereasoningbehindtheuseofpureoxygenseemedsoundenough.Inthevacuumofspacethecabinpressureneededtobemaintainedatlessthan6psi.Testshadalreadyprovedthatanyfireatthispressure,eveninapureoxygenenvironment,couldbeeasilycontainedandex=nguished.Duringgroundtests,however,asea-levelpressureof14.7psiwouldenvelopthespacecrad.Iftheoutsidepressureexceededthatinsidethespacecradbymorethan2psi,therewasachancethatthepressurehullcouldrupture,soitwasimpossibletotestthespacecradonthepadusingthe5.2to5.6psithatwouldbestandardoncethespacecradhadachievedorbit.Insteadtheengineerscrankeduptheinteriortomorethan16psitoexceedsea-levelpressure.Itwouldprovetobeafatalerrorofreasoning:thateasilycontainedfireinspaceatlessthan6psiwouldbecomeanexplosiveinfernoat16psi.

ColinBurgess

49

When VulnerabiliGes Line Up, Consequences Can Be DevastaGng

SapaProfilesmayhavealteredmechanicalproper=estestresultsperformedonaluminumextrusionsproducedfromatleast1996to2007,andmayhavereportedalteredtestreportsinmaterialcer=fica=onsgiventoitscustomersduringthat=meframe.

50

Fraud

51

Counterfeit Parts

CombaXng Fraud Heightened Awareness and Understanding

Supplier Oversight and TesGng of Incoming Product

Western Titanium

M&M Metals A&P Alloys

52

“OrbitalATKandNASAdiscoveredthepresenceofadefectintroducedduringmachiningofthebearingborehousing…Forensicinves/ga/onofEngineE17,whichfailedduringATPinMay2014,discoveredthepresenceofasimilarnon-conformingdefectinthehousing”

53

Workmanship/CorrecGve AcGon

JULY 20, 2015

CRS-7 INVESTIGATION UPDATE “Preliminaryanalysissuggeststheoverpressureeventintheupper-stageliquidoxygentankwasini=atedbyaflawedpieceofsupporthardware(a“strut”)insidethesecondstage.SeveralhundredstrutsflyoneveryFalcon9vehicle,withacumula=veflighthistoryofseveralthousand.Thestrutthatwebelievefailedwasdesignedandmaterialcer=fiedtohandle10,000poundsofforce,butfailedat2,000pounds,afive-folddifference.Detailedclose-outphotosofstageconstruc=onshownovisibleflawsordamageofanykind.”

54

Material Control

Muskalsoblamesoverconfidencefortheaccident…“Whenyou’veonlyeverseensuccess,youdon’tfearfailurequiteasmuch.

Highperformingorganiza/onsarepreoccupiedwiththeprospectoffailure.

"What we really learned from the Apollo fire, in the words of [former astronaut] Frank Borman, was the failure of imagination," said William H. Gerstenmaier, NASA's associate administrator for space operations. "We couldn't imagine a simple test on the pad being that catastrophic. "The message to the team is to remember how difficult our business is, the importance of staying focused and using our imaginations to envision what can go wrong."

40 years later, recalling the lessons of Apollo 1 January 28, 2007|Michael Cabbage | Orlando Sentinel

AddiGonal Stuff

58

DemonstratedReliabilityvsInsight/Oversight

USS IWO JIMA Mishap

A Set-up for Failure

BryanO’ConnorChief,OfficeofSafetyandMissionAssuranceNASAHeadquarters,Washington,DCOSMAPOC,BrianHughiB(202)358-1572

(61)

What Happened

-October1990:USSIWOJIMAAmphibiousAssaultShipDeployedtoPersianGulf,Opera=onDESERTSHIELD

-DockedatBahrainshipyardforemergentrepairs

-Asshipwasleavingport-onehouraderpropulsionplantbroughtonline-bonnetfastenersfora4”valvesupplyingsteamtoShipsServiceTurbineGeneratorfailedcatastrophically

-  850degreesuperheatedsteamat600psiescapedintomannedcompartment

-  Ninesailorskilledinstantly,onemorefatallyinjured

(62)

Proximate Cause

Unauthorizedsubs=tu=onofblackoxidecoatedbrassfastenersforhighstrengthsteelfasteners

AlloySteelNutsBlackOxideCoatedNuts

(63)

The Set-Up

-  BOCBF’svirtuallyiden=calinappearancetohighstrengthsteelfasteners.Coa=ngservednofunc=onalpurpose-appliedinordertostandardizecommonparts

-  Fastenersreadilyavailableandeasilyinterchangeable

-  Brassexhibitssignificantlydiminishedstrengthproper=esfromsteelunderelevatedtemperatures

-  ManufacturerLogoonBOCBFbox:“FastenersforHighStrengthApplica=ons”

(64)

ContribuGng Causal Factors

-  Repairspecifica=onsdidnotiden=fyfastenerpartnumbers

-  NoevidencethatGovernmentholdpointinspec=onswereperformed

-  InadequateknowledgeofLevelImaterialcontrolrequirements

-Segrega=onfromnon-LevelI-Segrega=onoflook-alikeparts-Markings-Colorcoding-Documenta=on