Columbia Generating Station INPO AFI Recovery

Dave Weber, Energy NorthwestCMBG June, 2015

CGS - INPO Area For Improvement (AFIs) September 2014

AFI CM.2.1 - Design Calculation Backlog and Quality

2

AFI CM.3.1 - Engineering Service Provider Design Quality

AFI CM.2.1 - Design Calculation Backlog and Quality

CGS - INPO Area For Improvement (AFI) September 2014

AFI CM.2.1 - Design Calculation Backlog and Quality. Some plant critical design bases calculations have not been maintained and some are of poor quality. This has resulted in some design bases without updated supporting calculations and technical evaluations, a large backlog of revisions to low-voltage calculations, and documentation control issues with calculations. Contributing, design basis calculation update standards are not reinforced.

4

INPO 12-013 Performance Objectives and Criteria

OPERATIONAL CONFIGURATION CONTROL

PERFORMANCE OBJECTIVE (CM.2) - Plant operation, maintenance, and testing activities are conducted in a manner consistent with the licensing and design bases and maintain configuration control.

CRITERIA - 1. Conditions that could result in structures, systems, or components being outside of operating and design requirements are evaluated and resolved on a schedule commensurate with the importance to safety and reliability. Special consideration is given to conditions that may challenge design bases or safety analyses.

5

6

January 2014

◘ Identified Design Calculations and > 5 Calculation Modification Records (CMRs) as a negative trend

◘ Created Excellence Plan Focus Area, “Calculation Backlog Improvement”

January – September 2014

◘ Worked on calculation backlog as level of effort July 2014

◘ Created Calculation Backlog Project September 2014

◘ AFI in CM.2-1, Design Calculation Backlog and Quality

◘ CR 315070 Severity Level A (RCE) on 9/27/2014

Background History

7

Root and Contributing Causes

Root Cause

◘ RC1: Design Engineering did not recognize the vulnerability to configuration management with regards to calculations.

The AFI is based on three aspects:

◘ Four legacy calculation discrepancies were identified during the 2013 CDBI Inspection

◘ The design calculation backlog is higher than the industry standard

◘ Approximately 150 calculations were reported during the plant evaluation as missing, illegible, or incomplete.

8

Root and Contributing Causes Aspect one: Four legacy issues involving plant design documents without

supporting calculations or technical evaluations. These issues were not self-identified, not corrected, and therefore challenge the design organization when addressing nonconforming conditions.

◘ CR 292758 – Independence requirements IAW industry standards (IE/Non-1E Loads)

◘ CR 292764 – No technical basis for turbulent mixing of diesel combustion air

◘ CR 292760 – Operation of under voltage relays did not consider all design inputs

◘ CR 292762 – Emergency core cooling system pumps’ limits were not included in emergency operating procedures.

9

Root and Contributing Causes Aspect two: Three critical electrical calculations have a Calculation

Modification Request (CMR) backlog of greater than 100:

◘ Calculation E/I-02-95-01 Calculation for Overcurrent Protective Device Settings and Coordination for 480V Distribution System (146 CMRs),

◘ Calculation 2.06.20 Calculation for Cable Ampacity Verification for Scheduled Power Cables (144 CMRs),

◘ Calculation E/I-02-90-01 Calculation for Low Voltage System Loading (102 CMRs).

◘ At the time of initial investigation, there were 37 calculations that had greater than 5 CMRs (efforts during this RCE reduced this number to 29), 30 calculations that have 5 CMRs, and approximately 3425 calculations that have one to three CMRs. This large backlog challenges the retrieval of the design basis of the affected systems when addressing nonconforming or degraded plant conditions or the development of design changes.

10

Root and Contributing Causes

Aspect three: Approximately 150 design bases calculations were identified as either missing in their entirety (hard and microfiche copies), incomplete (missing pages, appendices, or attachments) or illegible. During the evaluation, the missing calculations were located: however, the issue with missing pages, appendices, or attachments remains.

The plant evaluation team stated “Some plant critical design bases calculations have not been maintained and some are of poor quality.” Based on the examples given and the Design Engineering Manager’s discussions with the evaluator it was determined that ‘poor quality’ relates to the incomplete or illegible calculations listed on the spreadsheet and not technical quality. Therefore the poor quality piece is addressed under documentation control issues. Additionally technical quality issues related to design basis documents is being addressed under Root Cause Evaluation, CR 315069.

11

Root and Contributing Causes

The team identified that there has been a lack of priority applied to upkeep of the design bases calculations

With emphasis placed on issuing design modifications and/or emergent plant needs, configuration management suffers with a culture that has justified the Calculation Modification Request (CMR) process as meeting minimum standards.

12

Corrective Actions

Immediate and Interim Corrective Actions

◘ Revised calculation procedure to clarify timeliness requirements of revising a calculation

◘ Developed a work down curve for the outstanding calculation backlog

RC1/CC1/CC2/CC3 corrective actions

◘ Dedicate Design Engineering resources that are focused on completing calculation updates

◘ Revised calculation procedure to clarify timeliness and corrective action requirements for revising a calculation

13

RCE Corrective Actions

RC1/CC1/CC2/CC3 corrective actions

◘ Create performance evaluation SMART goals for each supervisor

◘ Create calculation backlog project plan

◘ Create a calculation workdown curve performance indicator

◘ Create a schedule addendum that will include calculations and milestone due dates

◘ Drive the calculation backlog >5 CMRs to zero – due 12/31/2015

Effectiveness Review

Perform three Semi-annual effectiveness reviews

◘ Review Supervisory observations, Condition reports and Performance indicators for validation – due 06/30/2016

14

Actions

AE hired to revise calculations with greater than 5 plant implemented CMRs

◘ Weekly challenge calls with the AE to discuss calculation work focused on quality, cost and schedule

Design Calculation Backlog

15

Calculations > 5 CMRs PI

16

Sustainability

Set standards and expectations

◘ Performance Appraisals

◘ Technical Conscience Procedure Improvements

◘ Time based calculation updates not to exceed 1 cycle

◘ Modifications developed to include calculation/ drawing update

◘ Condition reports generated when CMRs >5 Monitor Calculation Backlog with Performance Metrics Annual funding for Configuration Management maintenance Perform periodic effectiveness reviews

17

18

January 2014

◘ Drawing backlogs for those categorized as “Update Upon Request” identified as a negative trend

January – September 2014

◘ Backlog reduction efforts were worked as level of effort September 2014

◘ Drawing Backlog entered into the Design Engineering Excellence Plan

◘ RCE for calculation backlog identified an extent of cause and condition issue regarding drawing backlog

◘ Condition Report generated

INPO Interviews - Drawing Backlog

Background History

19

Actions Completed to Date

Develop Project Plan for resolving the drawing backlog Funds obtained for FY15 Project plan has been developed with schedule, resources and funding

identified Work down curves have been established 7 contractors have been hired to work on drawing backlog Additional AutoCAD and Raster Design licenses purchased to support new

contract Drafters Improved Performance Indicator Funds obtained for FY16 Monitor progress using work down curve and performance indicators

metrics on a weekly and monthly basis

Current status As of May, 2015

◘ 1886 of 10,200 drawings have been drafted

◘ 1869 of 10,200 drawings have been checked

◘ 1813 of 10,200 drawings are approved

Prioritization of System Drawings (Based on Maintenance Department)

◘ Heater Drain 442 total drawings 97% Comp

◘ Condensate 674 total drawings 95% Comp

◘ Bleed Steam 320 total drawings 83% Comp

◘ Seal Steam 111 total drawings 50% Comp

◘ Heater Vents 225 total drawings 29% Comp

◘ Misc Drains 175 total drawings 3% Comp

20

Drawing Backlog PI

21

AFI CM.3.1 – Engineering Service Provider Design Quality

CGS - INPO Area For Improvement (AFI) September 2014

23

AFI CM.3.1 - Engineering Service Provider Design Quality. From November 2012 to September 2014, engineering service provider design packages and products have contained technical errors such as use of incorrect or incomplete design inputs, specifying use of obsolete equipment or components, missing affected drawings and calculations, and frequently omitting critical construction and installation details. Contributing to this, engineering supervisors do not consistently hold engineering service providers accountable to department standards for quality and schedule adherence.

INPO 12-013Performance Objectives and Criteria

DESIGN CHANGE PROCESSES

PERFORMANCE OBJECTIVE (CM.3) - Changes to plant configuration, design, and licensing bases are evaluated, controlled, tested, and implemented while consistency is maintained among the physical plant configuration, design and licensing requirements, and the documented plant configuration.

CRITERIA - 1. The design authority is clearly identified and authorizes all plant design changes through a system of controls. Roles, responsibilities, and processes for developing, reviewing, and approving design and license bases changes are clearly defined and followed.

24

2005 - Decision to use AEs 2008 – AFI in CM.3-1, Weaknesses in oversight of external engineering

products 2010 - Self Assessment, AE Oversight January 2014

◘ Identified Engineering Change Product Quality as a negative trend

◘ Created Excellence Plan Focus Area, “Effectiveness of Design Engineering Changes”

April 2014 - INPO IER L1-14-20 July 2014 - Created Procedure Upgrade Project September 2014

◘ AFI in CM.3-1, Engineering Service Provider Design Quality

◘ CR 315069 Severity Level A Root Cause (RCE) 9/27/14

25

Background History

Root and Contributing Causes

Root Cause

◘ RC1: Design Engineering organization does not consistently demonstrate ownership of design quality to provide early engagement and consistent, accountable resources for AE oversight.

Contributing Causes

◘ CC1: Engineering Management is not enforcing priorities to ensure the quality of design products prepared by the AEs.

◘ CC2: AEs are not consistently holding themselves accountable for providing design products that meet EN standards.

26

Root and Contributing Causes

In 2005, EN transitioned to the use of external AEs to supplement in-house engineering resources (not in same time zone, not on site)

In October 2008, a plant evaluation determined that CGS had a weakness in the oversight of external engineering products (AR-CR 190612).

◘ This ACE resulted in multiple corrective actions that included revising the EN Project Manager qualification, enhancement of design procedures to clarify EN expectations for product in-progress reviews and condition reporting expectations for the AE, better defined roles and responsibilities in design engineering procedures and contracts, and implementation of a Comment Control Form for formal communication between EN and AE firms

◘ Transition AE to on-site presence

27

Root and Contributing Causes

In 2010, a Self-Assessment was completed to determine the effectiveness of the corrective actions from ACE 190612 (AR-SA 222381). The Self-Assessment determined that there were improvements made from implementing the administrative tools such as the Division of Responsibility form, Comment Control Form, and the Columbia Design Guidance Tool. However, the self-assessment also revealed there were still areas for improvement. The key findings determined that EN was still not holding the AEs accountable for product quality and EN had not effectively developed a resource loaded schedule to adequately develop design changes

28

Root and Contributing Causes

In 2014, the first quarter Engineering Department Roll-Up Meeting (DRUM) report identified an adverse trend in the number of field changes. The report indicated the Design Engineering Excellence Plan would address the causes and corrective actions going forward. The Design Engineering Excellence Plan identified six focus areas for the department, one of which was Effectiveness of Design Engineering Changes, identifying that AE design products do not consistently meet EN engineering standards for effectiveness regarding quality, cost, and timeliness

29

Root and Contributing Causes In April 2014, a Level 1 event report was published, IER L1-14-20,

Integrated Risk – Healthy Technical Conscience, relating to integrated risk, technical conscience principles and engineering fundamentals. The report discusses an adverse trend in technical errors and risk management that have resulted in serious organizational and operational impacts. The report shares lessons learned regarding insufficient technical expertise, critical design requirements and loss of margin, and insights from engineering vendors. The recommendation for Engineering Vendor Oversight and Support focuses on adhering to sound project management fundamentals. The attributes of the recommendation include communicating achievable and understandable objectives, assessing technical and project management proficiencies, and ensuring prompt reporting of events during project execution.

◘ EN has approximately 50 assignments to address IER L1-14-20; 24 are complete and the others are scheduled to be completed by 9/25/2015

30

Root and Contributing Causes EN uses AEs to develop engineering products such as design changes,

calculations, and drawings for CGS. The AEs work under a Basic Ordering Agreement (BOA) contract with individual contract releases for each specific project scope. The AEs are expected to develop engineering products in accordance with EN procedures with oversight from EN project managers and design engineers. Design Engineering reviews and evaluates (also known as an owner acceptance review) in-progress and final AE engineering products using score sheets in accordance with DES-3-3, Formal Design Reviews. Once the AEs have completed the engineering products, they are approved by EN and implemented in accordance with applicable procedures.

◘ Any issues or errors that are discovered after EN’s approval are generally resolved by Design Engineering as a design change revision or field change (not the AE). Each revision and field change is designated with one of the following cause codes for trending purposes; planned, scoping, material substitution, design error, construction convenience, installation error, administrative only, and user defined

31

Root and Contributing Causes

Specific Mods

◘ Multiple design change revisions and field changes related to the Radwaste Building Exhaust Air (WEA) and Turbine Building Exhaust Air (TEA) modifications,

Installation difficulties and omissions, translation errors within design documents, Old mod sat on the shelf

◘ Fast track modifications for Motor Control Center (MCC) Buckets and 50DHP-350 3000A (N/S) Breakers resulting in critical path delays

32

Corrective Actions Immediate and Interim Corrective Actions

◘ Briefed design engineers and AEs on gap and importance to EN◘ Interim corrective action – Perform self-assessment of R22 designs for

technical errors

RC1/CC1/CC2 Corrective Actions◘ Change Roles Responsibilities and Accountabilities

Assign a single Energy Northwest (EN) engineer for each AE design change to follow projects from cradle to grave Modify design engineering and project management procedures to clarify roles and responsibilities

33

RCE Corrective Actions RC1/CC1/CC2 corrective actions

◘ Ensure future Basic Ordering Agreements (BOA)/ Contract Releases for the AEs reflect EN standards and quality, schedule, and cost

◘ Engineering VP to provide briefing to Engineering Department to reinforce priorities for Design Engineering and emergent plant response expectations (after procedures revised)

◘ Incorporate a SMART goal in Performance Appraisals for Design Engineering supervision

◘ Onsite Engineering Service Providers to develop and implement a product quality recovery plan that meet EN standards

34

Effectiveness Review

Perform an interim corrective action effectiveness review following R22 refueling outage.

◘ Review Engineering Management focused observations, corrective action program, performance indicators, product quality score sheets, and Comment Control Forms (CCFs) - 06/30/2016

Perform a final corrective action effectiveness review.

◘ Review performance indicators, quality score sheets for design changes, and field changes for implemented design changes - 12/31/2017

35

36

Procedure Upgrade Project Procedures Upgrade Project

◘ Three month effort to make large scale improvements to over 30 Design Engineering Procedures and 20 forms

Significant Upgrades/ Improvements

◘ Implemented a 30% Conceptual Design Review Board

◘ Implemented required quorums for all stakeholders

◘ Implemented scorecards following both the Design Preparation (post 100%) and the modification implementation

◘ Added performance at risk options based on product quality, timeliness, and cost behaviors

◘ Addressed Enterprise Risk

◘ Developed an engineering walkdown procedure Training on these procedure changes is complete

Completed Excellence Plan Actions

Obtained FY15 funding to allow early start for high impact designs in first quarter 2015

Bi-weekly Design Engineering schedule review meeting Weekly meetings between DE manager and AE’s managers to discuss

performance Developed PI metrics on ECs

◘ Engineering Field Changes (including design errors)

◘ Root Cause Evaluation reviewed and approved on 01/06/2015

37

38

Actions Remaining RCE (CR 315069)

◘ Implement corrective actions

◘ Perform effectiveness reviews Monitor progress through PI Metrics

◘ Engineering Change Field Changes (including design errors)

◘ Product Quality Grading scores Excellence Plan

◘ Improve accountability through contract management

◘ Perform snapshot assessment to ensure success

INPO 90-009, Guidelines for the Conduct of Design Engineering, Rev 3

Field change requests are monitored and causes for field changes are identified and analyzed to provide a basis for improving the design change process. Implementing appropriate corrective actions should decrease the number of field changes needed for future modifications. Experience has shown that large numbers of field changes are frequently caused by inadequate walkdowns, inaccurate or incomplete as-built drawings, or incomplete designs resulting from tight scheduling or insufficient planning.

39

Field change requests for ease of construction (Construction Convenience) should be minimized, because the cost of analysis, documentation (e.g. rework to the EC package, WO task package, QC reviews, etc.), and work delays related to the field change often will exceed the expected benefit

40

CGS - Cause Code DefinitionsConstruction Convenience

Category Code Description

1 CC01 Construction Convenience - Installation cost savings. The craft provided a recommended installation solution that deviates from the design but the recommendation will save labor, time and/or material.

1 CC02 Construction Convenience - Dose savings. Reconfigure the original design to reduce radiological exposure.

1 CC03 Construction Convenience - Ease of maintenance. Reconfigure the original design to allow future Maintenance activities to be performed more efficiently.

1 CIE01 Craft Installation Error - EC Rev is provided to revise the design rather than the cost for reworking the installation.

USA Category 1: Engineering Errors - field changes required to correct design/engineering errors which could have (1) prevented installation, (2) resulted in improper installation, (3) resulted in undesirable operation of equipment, or (4) resulted in an unsafe condition to personnel of plant equipment.

USA Category 2: Modification Acceptance Test Errors - field changes that involve testing methodology or the adequacy of testing. Included are inadequate or excessive testing requirements, inadequate interpretations of testing results, changes to testing requirements, and changes to hold points.

USA KPI Cause Categories

41

CGS - Cause Code DefinitionsDesign Error

1 DE01 Design Error - Functional design error such as the Top Tier drawing as designed inhibits proper operation or a Calculation change is needed based on Vendor info or unforeseen needs.

1 DE02 Design Error - Physical design error such as the physical design documents not in agreement with the Functional Design Documents or inadequate tolerances, re-routing or mounting changes.

1 DE03 Design Error - Inadequate walkdown during the design phase caused re-design or missing enough omitted documents that could cause an error likely situation during the planning of Work Order Tasks prior to implementation.

1 DE04 Design Error - Inadequate design reviews or verification during development of the design caused an EC revision due to lack of rigor in preparation of the design package by responsible engineer, review or input by participating SMEs, or overall design verification.

2 DE05 Design Error - Inadequate Modification Test Plan. The Test Plan did not address all elements of the design change necessary to validate that SSCs as modified will perform required design functions as intended and can be returned to unrestricted service and/or the Modification Test Plan did not include necessary provisions for special testing of the design change or special test procedures could be developed for special test evolutions

1 DE06 Design Error - Inadequate Modification Risk Assessment. The Modification Risk Assessment did not identify all the risks and mitigation strategies that caused unforeseen human performance, process and station implementation risks.

1 DE07 Design Error - Inadequate identification and control of Design Inputs. Design inputs from sources such as design basis, regulatory requirements, codes and standards, etc, were lacking to address the technical considerations that bound the design.

2 DE08 Design Error - Inadequate Design Evaluation and Safety Assessment – The present configuration before the modification was not fully addressed or understood which caused errors not considered in the Failure Modes and Effects Analysis, component or system functions. The safety assessment neglected to include certain aspects of the impact to licensing basis, FSAR/Tech Specs or overall impact to plant safety.

42

CGS - Cause Code DefinitionsMaterial Substitution/Scoping

1 MS01 Material Substitution - Changes in material/parts due to availability or cost.

1 MS02 Material Substitution - Changes in vendor manufacturing and or configuration.

1 SCOPING01 SCOPING - Inadequate walkdown during the design phase resulted in a scope change during implementation.

1 SCOPING02 SCOPING - Lack of design team member support from interfacing departments (e.g. Craft, Ops, RP, Chemistry, etc.) contributed to changing the scope of the design.

1 SCOPING03 SCOPING - Implementation is in-progress but will not be implemented by its expiration date and requires re-evaluation.

43

Functional/Physical Definition

Functional Design Documents ‑ Documents which portray design criteria and schematic arrangements for the purpose of establishing and conveying the operational (functional) aspects of Structures, Systems, and Components. Examples: Elementary Diagrams, Flow Diagrams, Logic Diagrams, Safety Related Equipment Lists, Design and Safety Calculations, One‑line Diagrams, Engineering Standards that implement Criteria, and Procurement Specifications.

   Physical Design Documents ‑ Documents which convey physical layout

(configuration, organization of elements) and arrangement requirements. Physical Design Documents interpret the functional requirements given in the Functional Design Documents into configuration requirements of Structures, Systems and Components. Examples: Connection Diagrams, Piping ISOs, Mechanical Layout Drawings, Cable Schedules, Raceway Plan Drawings.

44

EC Field Changes PI (Month to Month)

45

EC Field Changes PI (Month to Month)

46

EC Field Changes PI (6 Months Rolling)

47

Sustainability Set standards and expectations

◘ Performance Appraisals◘ Technical Conscience

Procedure Improvements ◘ Conceptual Design Review Board implementation ◘ Scorecard Improvements establishing performance at risk based on quality, cost

and schedule performance◘ Design Walkdown implementation

Monitor Field Changes through Performance Metrics Perform periodic effectiveness reviews (self-assessments)

Excellence Plan

Calculation Improvement

◘ No calculations >5 CMR’s 12/31/2015

Margin Management◘ Technical Conscience Self Assessment

Regulatory Margin◘ CDBI Preparation/ Downgraded systems

Effectiveness of Design Engineering Changes

◘ Procedures upgraded/ Self Assessments on schedule Design Engineering Backlog

◘ Drawing backlog Reduction 08/31/2017

48

49