Agenda Operating Committee - NERC Filings to FERC DL/Response_… · Operating Committee Work Plan...

116-390 Village Blvd. Princeton, NJ 08540

609.452.8060 | www.nerc.com

Agenda Operating Committee March 12, 2008 | 1–5 p.m. March 13, 2008 | 8 a.m.–noon The Westin City Center 650 North Pearl Street Dallas, Texas 75201 214-979-9000

Item Leader Action

1. Administration Secretary

a. Quorum

b. Procedures

c. Introduction of Members and Guests

d. Agenda Chairman Approve

2. Consent Agenda Chairman Approve

a. Minutes of December 12−13, 2007 meeting

b. State/Municipal Utility Representative

3. FYI

a. NERC Board of Trustees Vice Chairman

b. Operating Committee Work Plan Vice Chairman

c. NERC Board Technology Committee Stan Johnson

d. Operating Committee Appointments Chairman

i. OC/PC Subgroup Nomination Form Chairman

ii. Reliability Fundamentals Working Group Chairman

iii. Integration of Variable Generation Task Force Warren Frost

iv. Reliability Metrics Working Group Jason Shaver

continue…

Operating Committee Meeting Agenda March 12–13, 2008 2

4. Operating Committee Charter Revisions Chairman Approve. Follow-up

from the joint meeting.

5. Critical Infrastructure Protection Stan Johnson

a. NERC Situation Awareness Tool Status Report

b. North American SynchroPhasor Initiative Follow-up from joint meeting

c. National Response Framework Information

6. Reliability Readiness Program

a. Reliability Readiness Program Work Plan Kevin Conway Discussion.

b. Role of the Operating Committee Chairman Discussion

7. Functional Model Working Group Steve Crutchfield Follow-up from joint meeting

8. Organization Registration and Certification Lucius Burris Discussion

9. Reliability-based Control Standard Drafting Team Doug Hils Discussion

10. Training John Taylor Endorse Personnel Subcommittee’s training model

11. Education and Certification Programs Martin Sidor Discussion

12. Events Analysis Bob Cummings Discussion

13. NERC/NAESB Coordination Andy Rodriquez Discussion

a. TLR Procedure

b. ATC/AFC and CBM/TRM Revisions

c. Balancing Authority Controls

14. Real-time Tools Best Practices Task Force Jack Kerr Approve

15. Interchange Subcommittee John Ciza Discussion

a. TSIN Registry

b. Eastern Interconnection Interchange Tool

16. Resources Subcommittee Terry Bilke

a. Time Monitor Reference Document Approve

b. Access to Performance Data Information

c. FERC Request for Access to Inadvertent Data Discussion

17. Operating Reliability Subcommittee

a. Flowgate Administration Reference Document David Zwergel Approve

18. Next meeting – June 4–5, 2008 (Tentatively Toronto, Ontario)

Secretary

Operating Committee Meeting Agenda March 12–13, 2008 10

Item 6. Reliability Readiness Program

Item 6.a Reliability Readiness Program Work Plan Kevin Conway, reliability readiness evaluator, will provide an overview of the Reliability Readiness Program Work Plan.

Item 6.b Role of the Operating Committee Chairman Mayo will lead a discussion of the role of the Operating Committee in the Reliability Readiness Program.

Background (Excerpt of the Operating Committee charter)

3. Support to the Reliability Readiness Program. Provide technical support, guidance, and advice to NERC’s Reliability Readiness Program (see also NERC Rules of Procedure, Section 700, “Reliability Readiness Evaluation and Improvement program, and Appendix 7, “Reliability Readiness Evaluation and Improvement Program Procedure”).

a. General • Develop criteria for measuring program success, and

review the program against those criteria. • Recommend actions to other NERC programs (standards,

compliance, assessments, training, etc.) based on lessons learned and trends from readiness evaluations and examples of excellence

b. Readiness Evaluations • Review readiness evaluations for trends and recommend

new or different types of evaluations or changes in processes or metrics, including:

Readiness criteria Guidelines for reporting and disclosure, and Guidelines for consistency and relevancy of

evaluations: • Between comparable entities, and • Through time

• Provide guidance to the readiness evaluations process c. Examples of Excellence

• Review and discuss the examples of excellence for lessons learned

• Support information exchange within the industry on examples of excellence

Reliability Readiness Evaluation and Reliability Readiness Evaluation and Implementation Program Implementation Program –– Three Year Strategic GoalsThree Year Strategic Goals

Kevin Conway

Dallas, TX March 12-13, 2008

Driver of the Three Year PlanDriver of the Three Year Plan

NERC 2008 Strategic Plan

Inputs to the three year plan• Staff Meetings

• ORS Meeting

• Regional Representative Meeting

• OC/ORS Small Meeting

• OC meeting

ORS ConcernsORS Concerns

Tracking of Recommendations

Interviewing of Staff by Compliance

Evaluation Timing with Compliance Audits

Periodicity

Process to Identify TOs who should be evaluated

EoE Approval Process

Compliance Staff Participation on Evaluations

Reporting of Compliance Violations

Confidentiality

ORSRETF Meeting ORSRETF Meeting

Operation Reliability Subcommittee ReadinessEvaluation Task Force Meeting

March 6, 2008• Jim Castle NYISO• Sam Holeman Duke (representing the OC)• Richard Schneider NERC• Frank Koza PJM• Jim Hartwell NPCC• Ben Deutsch SERC• Mitch Needham NERC • Kevin Conway NERC • Kristin Iwanechko NERC

ORSRETF DiscussionsORSRETF Discussions

Discussed the Mission of the Program• Encourage Excellence

• Provide Service to the Industry

Industry Culture and Ethics

Program Benefits• Objective Review

• Process Based

• Benchmarking

Discussed Issues Raised at the ORS

Recommendation for a White PaperRecommendation for a White Paper

• Role of the Program

• What are the Expectations

• How the Program has Developed

• The Role of the Regions in the Process

• The Role of the Program Sponsors (BOT/OC/ORS)

• Consistency

• Foster Innovative Ideas (EoE)

• Training

• Frequency of Evaluations

• Who are the Volunteers and what are the Qualifications

• What is FERC’s role

• Recommendation Tracking and Metrics

• TO reviews (Regions, Bulk Power Test)

• Program Future

Strategic DirectionStrategic Direction

Staffing

Industry Communications

Reinforce the Program Identity

Provided Readiness Evaluator Training

Communications of Examples of Excellence

Perform Readiness Program Self-Assessments

StaffingStaffing

Attracting qualified applicants

Retention of current evaluators

Reliance on contractors

Succession planning

Program manager

Industry CommunicationsIndustry Communications

Participation in forums• NERC BOT

• NERC Standing Committees

• ORS

• Transmission Owners and Operators Forum

• Regional and Sub-Regional Meetings

Reinforce the Program Identity Reinforce the Program Identity

Readiness and Compliance are organizationally separate within NERC• In Staff

• In Process

• In Timing

Communicate the goals of the program• Encourage operational excellence

• Be positive and proactive influence

Provided Readiness Evaluator TrainingProvided Readiness Evaluator Training

Continue to provide evaluator training

Develop team lead and regional representative training

Continue to improve consistency

Continue to keep evaluators current with standards and practices

Communications of Examples of ExcellenceCommunications of Examples of Excellence

Continue to refine process

Include technology and tools into process

Allow process to better recognize regional and sub-regional practices and tools

Allow EoE process to consider entities of similar characteristics

Better archive and update EoEs

Codify and communicate process

Readiness Program SelfReadiness Program Self--AssessmentsAssessments

Utilize end of year time frame

Review feedback forms

Review lessons learned from staff

Develop more useful metrics

Report to various forums on findings of the program

Continue to refine process and be more dynamic

Questions?Questions?


609.452.8060 | www.nerc.com

Minutes Operating Committee March 12–13, 2008 Dallas, Texas Convene

Meeting Highlights The Operating Committee:

1. Approved a revision of the OC charter to include a process for approving reliability guidelines and fractional voting for regional entity representatives.

2. Approved the Time Monitor Reference Document.

3. Approved version 4 of the Flowgate Administration Reference Document.

4. Reviewed the developing NERC Situation Awareness Tool and a proposal to develop a uniform reliability coordinator “morning” report.

5. Accepted the final report of the Real-Tools Best Practices Task Force.

A regular meeting of the NERC Operating Committee (OC) was held on March 12−13, 2008, in Dallas, Texas. The meeting notice, agenda, and attendance list are affixed as Exhibits A, B, and C, respectively; and individual statements and minority opinions as Exhibits D and E, respectively. The meeting presentations are posted in a separate file at http://www.nerc.com/~filez/ocmin.html.

Operating Committee chair Gayle Mayo convened the meeting at 1 p.m.

Secretary Larry Kezele announced that a quorum was present.

Antitrust Compliance Chair Mayo referred the committee to the NERC Antitrust Compliance Guidelines included in the meeting agenda.

Consent Agenda Derek Cowbourne and Julien Gagnon proposed amending the draft minutes of the December 12–13, 2007 meeting. By unanimous consent the committee approved the amended meeting minutes.

In addition, by unanimous consent, the committee elected Ralph Anderson as a State/Municipal Utility sector representative.

FYI Vice Chair Holeman reviewed:

1. The NERC Top Ten Goals for 2008 (Exhibit F) and the committee’s role in achieving those goals,

2. A summary of the February 2008 Board of Trustees (BoT) meeting (Presentation 1), and

3. The committee’s three-year work plan.

http://www.nerc.com/

Operating Committee Meeting Minutes March 12–13, 2008

2

Stan Johnson, manager of Situation Awareness and Infrastructure Security, summarized the actions of the Board of Trustees Technology Committee (Presentation 2). The Technology Committee has reviewed all of the existing and proposed NERC reliability tools (e.g., Reliability Coordinator Information System and the Interchange Distribution Calculator), and has also approved guidelines for the development of reliability tools. Mr. Johnson outlined a draft process for developing a new reliability tool. Chair Mayo stated that, until a specific transition plan is drafted and implemented, NERC will continue to support all existing reliability tools. In addition, she asked that each of the subcommittees review the draft development process flow chart for further discussion at the June 2008 OC meeting.

Operating Committee Administrative Topics 1. Committee Elections ⎯ The secretary reported that nominations and elections must begin

in late May or early June to replace those committee members with one-year terms. Chair Mayo asked the secretary to send the list of members with one-year terms to the committee.

2. Operating Committee and Planning Committee Subgroup Nomination Form ⎯ Chair Mayo stated that the OC and PC officers developed the subgroup nomination form to foster the formation of special assignment working groups or task forces. The nomination form is posted at http://www.nerc.com/committees/operating.html.

3. Committee Recognition ⎯ Chair Mayo acknowledged Derek Cowbourne for his years of service to the OC and to the industry.

4. Nominating Committee ⎯ Derek Cowbourne moved to elect Mark Fidrych as chair of the Nominating Committee and Tom Irvine to serve on the Nominating Committee. The OC approved the motion.

5. Reliability Fundamentals Working Group (RFWG) ⎯ Chair Mayo stated that the OC approved the scope of the RFWG at its December 2008 meeting. She encouraged those interested in serving on this working group to complete the subgroup nominations form referenced in item 2 above.

6. Integrating Variable Generation Task Force (IVGTF) ⎯ Warren Frost, chair of the IVGTF, and Mark Lauby, NERC facilitator for the IVGTF, responded to questions resulting from Mr. Frost’s presentation at the joint meeting. The task force will address variable generation in general; however, integration of wind generation is its primary focus. The task force expects to have a report available for OC and PC review in September 2008.

7. Reliability Metrics Working Group (RMWG) ⎯ Jason Shaver, chair of the RMWG, responded to questions resulting from his presentation at the joint meeting. Howard Tarler asked how the work of the RMWG will dovetail with the NERC top ten goals discussed earlier, especially with regard to expected completion dates. The OC expressed its desire to review, if not approve, proposed reliability metrics. (Secretary’s note: Following his discussion with the Planning Committee, Mr. Shaver reported that the PC directed the RMWG to initially focus on the development of metrics related to the long-tern reliability assessment.)

http://www.nerc.com/committees/operating.html


3

OC Charter Revisions Chair Mayo reviewed the proposed revisions to the OC charter, highlighting new Appendix 3, which documents the committee’s process for approving reliability guidelines. Ed Schwerdt recommended codifying fractional voting of regional entity representatives, and suggested using the language in the charter for the Compliance and Certification Committee as a model. Tom Bowe moved to endorse the revised OC charter, as amended by the addition of fractional voting for regional entity representatives. The committee approved the motion. The approved OC charter is attached as Exhibit G.

Critical Infrastructure Protection

Situation Awareness Tool (SAT)Situation Awareness Tool (SAT)

S it u a t i o n A w a r e n es s

NERC Situation Awareness Tool (SAT) ⎯ Stan Johnson provided a progress report of the development of the NERC SAT (Presentation 3). He summarized Section 1001 (Situation Awareness) of the NERC Rules of Procedure and outlined NERC’s goals to accomplish its mission

e

n ormation related to major bulk

power systems events.

North American SynchroPhasor Initiative (NASPI) ⎯ As a follow-up to his presentation at the joint meeting, Mr. Johnson discussed NASPI from a system operator’s perspective (Presentation 4). Training materials are

for this program.

He also stated that NERC and the regional entitles are working to develop a uniform reliability coordinator “morning” report of system status. The committee suggested thdevelopment of a formal communications policy, procedure, and protocol for use ireporting inf

available at http://www.naspi.org/resources/training/training.stm.

National Response Framework ⎯ Mr. Johnson explained that the National Response Framework presents the guiding principles that enable all response partners to prepare for and provide a

t

• rconnection disturbance

September 18, 2007 MRO disturbance

unified national response to disasters and emergencies ⎯ from the smallest incident to the largescatastrophe (Presentation 5). The electricity sector is supported in its response to emergencies by the Department of Energy.

Events Analysis Bob Cummings reviewed four topics (Presentation 6):

1. Lessons learned from events analysis and summary of emerging and ongoing trends.

2. Update of events being analyzed

August 4, 2007 Eastern Inte

•


4

• February 14, 2008 WECC PacificCorp-East disturbance

ings explained that NERC can issue three levels of alerts (advi y

Eric Senkow onclusions resulting from FRCC’s analysis of the Februa 2 ion 7). He explained that the initia 3 yed clearing, which resulted in th o mately 2,500 MW of generation in soureppropag om FRCC disturbance (Video 1). The

k

tion

e

s

., the Boaand to solicit input to further enhance the value of the

d focus of the evaluation review process has 003 blackout. He believes that the evaluations should nd that the entity being evaluated should expect to

• February 26, 2008 FRCC disturbance

3. NERC Alert System – Mr. Cummsor , recommendation, or essential action).

icz reviewed the preliminary findings and cry 6, 2008 underfrequency load shedding event (Presentatl 1 8 kV singe phase fault led to a three phase fault with delae l ss of 25 transmission lines and approxi

thern Florida. FRCC established an events analysis team and expects to develop a disturbance ort within eight to twelve weeks. Mr. Cummings narrated a video of the frequency wave

ation across the Eastern Interconnection resulting frvideo was produced by Virginia Tech using data captured by its frequency monitoring system.

Reliability Readiness Program Kevin Conway, reliability readiness evaluator, reviewed the Reliability Readiness Program worplan and outlined the program’s strategic direction to address 1) staffing, 2) industry communications, 3) reinforcement of the program’s identity, 4) readiness evaluator training, 5) communications of examples of excellence, and 6) performance of readiness

ORS ConcernsORS Concerns

program self-assessments (Presenta Tracking of Recommendations

Periodicity

Process to Identify TOs who should be evaluatedEoE Approval Process

Compliance Staff Participation on EvaluationsReporting of Compliance ViolationsConfidentiality

8). Mr. Conway stated that the Operating Reliability Subcommittee (ORS) and a small committee of the OC and ORS has identified several areas of potential improvement andis developing recommendations to address those concerns (as shown at right).

Mr. Conway and Jim Castle reported that thORS and the Readiness program will develop a white paper that further explorethe identified concerns and that details therole of the program stakeholders (e.gThis white paper will be used for discussion program.

Interviewing of Staff by ComplianceEvaluation Timing with Compliance Audits

rd of Trustees, the regional entities, and the OC).

Vice Chair Holeman suggested that the tone anchanged since its inception following the 2address fewer topics, but in greater depth, areceive benefit from the evaluation. The committee also discussed:

1. Compliance staff participation on readiness evaluations


5

2. Extensiveness of system operator questions, especially if only one operator is availabparticipate in the evaluation

3. Need to trend recommendations and identify lessons learned

le to

6. ring evaluations for training purposes

Cha essed the committee’s desire for

ReTerry Bilk(Pr

ty Subcommittee developed the reference document in sions to reliability standard BAL-004 (Time Error Correction).

tandard . The

proved the motion.

a t

3. vertent Data ⎯ The RS received a request from FERC

FlowgDavid Zwergel, chair of the Operto tAdmini

Midwee a certified balancing authority

orning at 8:03 a.m.

4. Whether the readiness evaluations are mandatory

5. Encourage broad input into readiness program evaluation and improvement process

Participation of observers du

ir Mayo encouraged the development of the white paper and exprstatus report at its June 2008 meeting.

sources Subcommittee e, chair of the Resources Subcommittee (RS) discussed the following topics

esentation 9):

1. Time Monitor Reference Document ⎯ The RS, the Reliability Coordinator Working Group, and the Operating Reliabilisupport of proposed reviChair Mayo explained the Board of Trustees actions taken related to reliability sBAL-004. Tom Bowe moved to approve the Time Monitor Reference Documentcommittee ap

2. Access to Performance Data ⎯ Mr. Bilke explained that RS members are required to sign nondisclosure agreement (NDA) prior to reviewing control data; however, the NDA has yeto be developed, which prevents the RS from completing its responsibilities. Chair Mayowill follow up with NERC staff to resolve this issue.

FERC Request for Access to Inadfor access to inadvertent data. Derek Cowbourne reviewed the intent of the committee’s resolution regarding the provision of information to federal, state, or provincial agencies (see Exhibit H to the December 2008 OC meeting minutes).

ate Administration Reference Document ating Reliability Subcommittee, explained the proposed revisions

he reference document. Thomas Stuchlik moved to approve version 4 of the Flowgate stration Reference Document. The committee approved the motion.

st ISO Balancing Authority Certification David Zwergel reviewed the status of MISO’s efforts to becom(Presentation 10). The certification team expects to issue the final report to MISO on March 14, 2008.

Adjourn and Reconvene The committee adjourned at 5:24 p.m. CDT and reconvened the following m


6

Functional Model Working Group As a follow-up to his presentation at the joint meeting, Steve Crutchfield encouraged the OC to

on the proposed revisions to the functional model.

tandard Drafting Team

ents identified in the reliability-based control standard authorization request ). The reliability-based control field trial continues, and the representatives of

up (RCWG), the ORS, and the standard drafting team

cing

ganizations, 3) final authority egion coordination. The subcommittee awaits

tml

provide comments

Reliability-based Control SDoug Hils, chair of the Reliability-based Control Standard Drafting Team, reviewed the five purpose statem(Presentation 11the Reliability Coordinator Working Gromeet monthly by conference call to review results. The standard drafting team expects to post a technical reference for comment in July 2008.

Organization Registration and Certification Lucius Burris, chairman of the Organization Registration and Certification Subcommittee, highlighted the history of the subcommittee’s efforts to develop reliability coordinator, balanauthority and transmission operator certification standards (Presentation 12). Currently the subcommittee is proposing to revise Section 5 (Organization Registration and Certification) and Appendix 5 (Organization Certification and Registration Manual) of the NERC Rules of Procedure. Industry stakeholders asked the subcommittee to clarify or address the 1) specific requirements for certification, 2) delegation and joint registration orregarding certification and appeals, and 4) multiple rNERC direction before posting the revised rules of procedure for comment.

Real-Time Tools Best Practices Task Force Jack Kerr, chairman of the Real-Time Tools Best Practices Task Force (RTBPTF), stated that thetask force completed its report and that it is posted at http://www.nerc.com/~filez/rtbptf.h

of situational awareness due to

-

re

(Presentation 13). The report is in response to Recommendation 22 of the U.S./Canada 2003 Blackout report. Recommendation 22 addressed the lack

13

Five Focus Areas

Situational Awareness

inadequate reliability tools and backup Practicescapabilities as a root cause of the

blackout. As shown in the presentation excerpt at right, the task force focused on five major areas by developing an online, Web-based, questionnaire containing approximately 2,000 questions. Survey responses wereceived from all reliability coordinatorsand 43 balancing authorities or transmission operators, who are not also reliability coordinators.

Real-Time Data Collection

Real-Time Data Collection

Tools Used for Support and Maintenance

Tools Used for Support and Maintenance

Modeling Practices

Modeling Practices

Situational Awareness

Reliability Tools for Situational

Awareness

Practices

Reliability Tools for Situational

Awareness


7

The task force recommended 1) 35 revisions to the reliability standards, 2) the development of 1operating guides, 3) the consideration of several examples of excellence, 4) 18 areas requiring additional anal

5

ysis, and 5) eight major issues for further industry discussion. A key RTBPTF recommendation identifies the need for a “reliability toolbox,” which includes five mandatory reliability tools and the development of performance and availability metrics and maintenance practices for those tools. The presentation excerpt at right identifies the five mandatory reliability tools.

Mr. Kerr stated that several groups have already started using the RTBPTF report to guide their future activities (e.g., the Reliability Coordinator Standards Drafting Team and the Data Exchange Working Group).

Following the committee’s discussion, Tom Bowe moved to accept the Real-Time Tools Best Practices Task Force Report. The committee approved the motion.

Training and Personnel Certification Martin Sidor, director of Training, Education, and Personnel Certification, explained the program goals related to system operator certification, continuing education, and education (Presentation 14). The education program area anticipates developing industry-wide education and learning opportunities for compliance auditors and reliability readiness evaluators. Chair Mayo stated that the OC identified the development of a Training chapter within the Reliability Concepts documents framework and assigned this task to the Personnel Subcommittee.

NERC/NAESB Coordination Andy Rodriquez, manager of Business Practice Coordination, reviewed several ongoing NERC/NAESB coordination efforts (Presentation 15). NERC and NAESB are exploring the roles of the two organizations to identify overlap and synergistic opportunities (e.g., merger of working groups). The committee suggested that the industry needs to be educated as to significance of a FERC-approved NAESB business practice.

Interchange Subcommittee John Ciza, chair of the Interchange Subcommittee (IS), reviewed the development status of the e-Tag registry and the Eastern Interconnection Interchange Tool (Presentation 16). The IS established a task force to evaluate the benefits of creating an Eastern Interconnection decision-making organization similar to those existing in WECC and ERCOT. The task force will also evaluate if the WECC Interchange Tool is feasible for implementation in the Eastern Interconnection.

15

Reliability Toolbox for Real-Time Operation

Visualization Tools

Telemetry D

ata System

Alarm

Tools

Network Topology Processor

State Estimator

Contingency Analysis

Other Real-Time Tools

Critical Applications/Facilities Monitor

System Operator

Real-Time Data:• Telemetry Data• ICCP Data• Other Real-

Time Data

NOTE:RedRed font indicates mandatory tools


8

Mr. Ciza stated that the Board of Trustees and the NAESB executive committee agreed to transfer the e-Tag registry from NERC to NAESB. The Joint Interchange Scheduling Working Group developed a functional specification, which is currently posted for comment.

Next Meeting The next meeting of the Operating C

Adjourn There being no further business beforeThursday, March 13, 2008 at 10:53 a

Larry Keze

o

the Operating Committee, chair Mayo adjourned the meeting on .m.

le

mmittee will be on June 4–5, 2008 in Toronto, Ontario.

Larry Kezele Secretary


609.452.8060 | www.nerc.com

Agenda Operating Committee June 4, 2008 | 1–5 p.m. June 5, 2008 | 8 a.m.–noon Marriott Toronto Airport 901 Dixon Road Toronto, Ontario, Canada 416-674-9400

Item Leader Action


a. Quorum

b. Procedures




a. Minutes of March 12−13, 2008 meeting

3. FYI

a. NERC Board of Trustees Chairman

b. NERC Meeting Locations Chairman

c. Member Elections Secretary

d. Reliability Fundamentals Working Group Mark Fidrych

e. Integration of Variable Generation Task Force Mark Lauby

f. Reliability Metrics Working Group Jason Shaver

g. Standards Development Work Plan David Taylor

continued…

Operating Committee Meeting Agenda June 4–5, 2008 2

4. Reliability Readiness Program

a. Reliability Readiness Program Work Plan and Whitepaper

Jessica Bain and Kevin Conway

Discussion

b. Role of the Operating Committee Chairman Discussion

5. Reliability Assessments Mark Lauby Discussion

6. Critical Infrastructure Protection Stan Johnson

a. Reliability Tool Development Process Approve

b. Situation Awareness Monitoring Center Information

c. Reliability Coordinator Daily Report Information

7. Functional Model Working Group Scope Revision Steve Crutchfield Approve

8. Operator “Tools” Access Chairman Approve

9. Interchange Authority Registration David Zwergel and Terry Bilke

Discussion

10. Backup Facilities Standard Drafting Team Tom Bowe Discussion

11. System Restoration and Blackstart Standard Drafting Team

Richard Kafka Discussion


13. Operating Reliability Subcommittee David Zwergel Information

14. Underfrequency Loading Shedding Relays Vice Chairman Discussion

15. DOE – Bulk Electric System Situational Awareness Jim McGlone Information

16. Next meeting – September 10–11, 2008 (Seattle, WA)

Secretary

Item 4. Reliability Readiness Program

Item 4.a Reliability Readiness Program Work Plan and Whitepaper

Kevin Conway, reliability readiness evaluator, will provide an update on a white paper being written by members of the Operating Reliability Subcommittee, NERC Readiness Program Staff, and the OC. At its March meeting, Mr. Conway presented the strategic goals of the Readiness Program. He discussed the current efforts of the program to solicit input from the industry, and reported on the decision to develop a white paper to further explore how to increase value to the industry. The ORS, OC, and Readiness Program ad hoc group proposed the white paper after input from several groups. This effort will assist the program to develop additional goals, meaningful metrics, and better tools to serve the needs of the industry. Mr. Conway will also introduce Jessica Bian, the new director of reliability readiness and benchmarking. Ms. Bian will give an update of the program, and be available to answer questions on the current status.

Item 4.b Role of the Operating Committee Chairman Mayo will lead a discussion of the role of the Operating Committee in the Reliability Readiness Program.

Background (Excerpt of the Operating Committee charter)

3. Support to the Reliability Readiness Program. Provide technical support, guidance, and advice to NERC’s Reliability Readiness Program (see also NERC Rules of Procedure, Section 700, “Reliability Readiness Evaluation and Improvement program, and Appendix 7, “Reliability Readiness Evaluation and Improvement Program Procedure”).

a. General • Develop criteria for measuring program success, and review the program

against those criteria. • Recommend actions to other NERC programs (standards, compliance,

assessments, training, etc.) based on lessons learned and trends from readiness evaluations and examples of excellence

b. Readiness Evaluations • Review readiness evaluations for trends and recommend new or different

types of evaluations or changes in processes or metrics, including: Readiness criteria Guidelines for reporting and disclosure, and Guidelines for consistency and relevancy of evaluations:

• Between comparable entities, and • Through time

• Provide guidance to the readiness evaluations process c. Examples of Excellence

• Review and discuss the examples of excellence for lessons learned • Support information exchange within the industry on examples of

excellence


609.452.8060 | www.nerc.com

Minutes Operating Committee June 4–5, 2008 Toronto, Ontario, Canada Convene


1. Reviewed critical dates related to member elections.

2. Reviewed the status of the Backup Facilities and the System Restoration and Blackstart Standard Drafting Teams.

3. Approved a motion related to the proposed Readiness Assistance Program (see discussion on pages 2–4 and 7).

4. Began preliminary discussions to determine its role in the development and approval of seasonal reliability assessment. (Note: These functions are currently performed by the Planning Committee and its Reliability Assessments Subcommittee.)

5. Approved a revision to the scope of the Functional Model Working Group.

A regular meeting of the NERC Operating Committee (OC) was held on June 4−5, 2008, in Toronto, Ontario. The meeting notice, agenda, and attendance list are affixed as Exhibits A, B, and C, respectively; and individual statements and minority opinions as Exhibits D and E, respectively. The meeting presentations are posted in a separate file at http://www.nerc.com/~filez/ocmin.html.

Operating Committee chair Gayle Mayo convened the meeting at 1 p.m.


Antitrust Compliance The secretary referred the committee to the NERC Antitrust Compliance Guidelines included in the meeting agenda.

Consent Agenda The committee approved the March 12–13, 2008 meeting minutes.

FYI Board of Trustees ⎯ Chairman Mayo summarized discussions at the May 7, 2008 Board of Trustees meeting, especially those related to the NERC 2009 business plan and budget and, more specifically, the reliability readiness program.

Meeting Locations ⎯ The secretary stated that, in its attempt to hold down escalating meeting costs, NERC is considering the development of a pre-approved list of meeting locations and hotels. The expectation is that half of the committee’s meetings would be held at these locations.

Member Elections ⎯ The secretary reviewed the critical dates related to committee member elections. July 7, 2008 is the target date for completing the selection process. NERC would submit the slate of new committee members to the Board of Trustees at its July 30, 2008 meeting.


Operating Committee Meeting Minutes June 4–5, 2008

2

Reliability Fundamentals Working Group ⎯ Chairman Mayo noted that Mark Fidrych has the assignment to solicit members to serve on the Reliability Fundamentals Working Group. Mr. Fidrych will provide a detailed report at the committee’s next meeting.

Integration of Variable Generation Task Force ⎯ John Kehler, representing Warren Frost, chairman of the Integration of Variable Generation Task Force, provided an overview of the efforts to prepare 1) a concepts document that includes the philosophical and technical considerations for integrating variable resources into the interconnection, and 2) specific recommendations for practices and requirements, including reliability standards, that address the planning, operations planning, and real-time operating time frames (Presentation 1). The task force anticipates presenting a draft final report to the committee at its September 2008 meeting.

Reliability Metrics Working Group ⎯ Jason Shaver, chairman of the Reliability Metrics Working Group, noted that the working group, at the request of the Planning Committee, is focusing its activities on developing metrics related to the Long-Term Reliability Assessment (Presentation 2). The working group will turn its attention to developing metrics related to the definition of Adequate Level of Reliability in 2009.

Standards Development Work Plan ⎯ David Taylor, manager of standards development, was available to respond to questions following his presentation at the joint meeting. The work plan is currently posted on the standards Web site (https://standards.nerc.net/) and Mr. Taylor encouraged the submission of comments.

NERC Press Release Related to Compliance Enforcement ⎯ Chairman Mayo summarized NERC’s press release related to its submission of the first set of violation notices in the United States to the FERC. Violation notices are posted at http://www.nerc.com/~filez/enforcement/index.html.

Reliability Assessments As a follow-up to Scott Helyer’s presentation at the joint meeting regarding the review and approval process for seasonal assessments (Presentation 3), Mark Lauby, manager of reliability assessments, noted that the Reliability Assessments Subcommittee (RAS) is looking to the OC to provide a more in-depth operational perspective to its seasonal assessments. The committee suggested providing a liaison to the RAS, developing operating metrics that might capture operational trends, and providing an opportunity for committee review and comment prior to release. Chairman Mayo will discuss these thoughts with the chairs of the Operating Reliability Subcommittee (ORS) and the Reliability Coordinator Working Group.

Reliability Readiness Program Jessica Bian, director of reliability readiness and benchmarking, provided an overview of the reliability assistance program’s mission and objectives as reflected in a draft proposal for discussion by the Board of Trustees Finance and Audit Committee at its June 6, 2008 meeting (http://www.nerc.com/~filez/finance.html) (Presentation 4). Ms. Bain highlighted the integrated reliability information help desk, on-site consulting for small to midsize entities, voluntary reliability excellence program, information sharing (i.e., positive observations and examples of excellence), and an alliance/partnership program that promotes dialogue on reliability.

https://standards.nerc.net/

http://www.nerc.com/%7Efilez/enforcement/index.html

http://www.nerc.com/%7Efilez/finance.html


3

Kevin Conway, reliability readiness evaluator, summarized the status of development of a white paper that will present recommendations for improvement to the reliability readiness evaluation program.

Following their presentation, the committee excused Ms. Bian and Mr. Conway and continued its discussion of the reliability readiness evaluation program. The committee’s discussion is summarized into the following general categories:

Initially Valuable 1. Program initially served a great purpose. With NERC now into compliance, benefit of the

program is significantly lessened. 2. There was significant benefit to having the first evaluation. Follow-up evaluations are not

nearly as beneficial.

Outlived Its Usefulness/Retire Program 1. Option 1 (Retain Program as is), take this off the table. Dollars are getting squeezed. One

region would like to kill. No support for the program in this region after the first cycle. General consensus in this specific region is that the program should go away. Prioritizing of programs and dollars would indicate that.

2. There are budget and manpower constraints. Program has outlived its time frame of usefulness.

3. Concerned about the cost effectiveness of program. NERC is providing a service that is very resource intensive. Standards development process is very resource intensive as well, and more beneficial. Other forums, such as Transmission Owners and Operators Forum (TOOF) provide this service.

4. Don’t see the incremental benefit of continuing going forward. 5. The program could have achieved better results if the opportunity of open dialog existed,

but because of the risk of findings of noncompliance, open dialog was lacking. With compliance at the forefront, evaluations outlived usefulness. Need to get NERC out of this process.

6. Readiness evaluation will not help address cyber standard implementation.

Retain Some/All of Program 1. Some residual value of program (e.g., certification of new entities, reviewing entities that

score especially high (for Examples of Excellence (EoE)) or low in metrics). Stop short of killing the program.

2. Readiness evaluations of cyber standards or other new standards would be beneficial. 3. Readiness evaluations focused on specific standards may be beneficial. 4. Some value to evaluations. Current time frame is probably not appropriate. Program does

need to be scaled back. 5. TOOF is not open to everyone. Need something for nonmembers of TOOF.

Combine Compliance and Readiness Audits 1. Benefit to combining compliance and readiness programs and audits. 2. May be an opportunity to use the experience of the ORS discussions of the Readiness

Evaluation Program to improve the compliance audits.


4

3. During compliance audits look for best practices. Don’t need to perform a duplicative readiness evaluation.

Help Desk 1. Help desk function at NERC may not provide answers that are consistent with regional

enforcement. 2. NERC help desk would fail. 3. If an entity has a standards question go to standards program or a compliance question go

to compliance program. Not go to single source. 4. Help desk not likely to work because questioners need definitive answers. 5. Help desk questionable. 6. Scope of help desk and on-site consulting needs to be very clear. 7. Help desk concept is noble, but belongs in the compliance area.

Voluntary Evaluations 1. If an entity wants an evaluation that entity would pay for all expenses. Evaluations should

be voluntary. 2. NERC should not compete with others to provide fee-based reviews. 3. Pay as you go is a good idea. 4. Most companies already have mandatory (financial) audits. Auditor also provides best

practices. Company pays for those audits. If IRS compliance becomes an issue, the company goes to the auditor for help. Should have similar model for reliability evaluations.

5. NERC as a compliance consultant does not make any sense.

Redeploy Resources 1. Refocus readiness personnel into other programs such as events analysis. 2. There should be something in the program that NERC can build on and re-channel staff to

work on, such as situational awareness. 3. Better use of resources ⎯ include operations lessons learned, and not just engineering

lessons learned, from event analysis. 4. Several committee members mentioned providing more resources to event analysis.

Examples of Excellence (EoE) 1. There is value in EoE. Don’t allow residual value to die. 2. EoE could be absorbed into compliance. 3. EoE are great and need to be pursued. However, they become outdated.

Other 1. As great as the program has been, there may be a conflict if something is discovered that is

not reported to compliance. 2. The NERC proposal seems to be more of a holding pattern option while waiting to

determine what to do with the NERC employees in the readiness evaluation program. 3. Even with the proposed change, there is no reduction in the NERC budget. 4. Need innovation and better ways to perform duties ⎯ concerned that we are taking our

eyes off of the ball in terms of reliability and security of the system.


5

5. Several committee members mentioned questions around entity certification (initial) and how that would work without readiness reviews.

6. ORS members mentioned that ORS could take on the initial entity certification readiness review responsibilities.

Adjourn and Reconvene The committee adjourned at 5:45 p.m. EDT and reconvened the following morning at 8:04 a.m.

Functional Model Working Group Scope Steve Crutchfield, standards development coordinator, summarized proposed revisions to the scope of the Functional Model Working Group. He also noted that the working group is drafting version 4 of the functional model and its associated technical reference document. Jerry Rust moved to approve the revised scope of the Functional Model Working Group. The committee approved the motion.

Reliability Readiness Program (cont’d) The committee returned to its discussion of the reliability readiness program. Jerry Mosier moved to eliminate the readiness evaluation effort as an independent program and redirect the resources into 1) enhanced event analysis, 2) situational awareness, and 3) augmented compliance audits. Following a brief discussion, the committee did not approve the motion.

Dennis Florom moved to suspend the readiness evaluation effort after 2008 until further direction can be determined. Following a brief discussion, Jerry Rust moved to postpone the motion indefinitely. The committee approved the motion.

Operator “Tools” Access Chairman Mayo stated that the Nuclear Regulatory Commission has requested access to the Resource Adequacy (ACE and Frequency) reliability tool. She noted that NERC staff would like to provide access within industry approved guidelines. She also opined that providing such access should not require a revision of “Vetting Criteria for Granting Access to NERC Reliability Tools,” approved by the committee in December 2007. Chairman Mayo requested the Resources Subcommittee and the Operating Reliability Subcommittee to draft reliability tool access guidelines for committee review.

Interchange Authority Registration David Zwergel summarized a letter regarding interchange authority registration from the Midwest ISO Balancing Authority Committee to NERC, SERC, MRO, and RFC. He noted that there does not appear to be a common understanding of what an interchange authority does. In some regions the interchange authority is merely a tool that helps in interchange management. The Interchange Subcommittee drafted a standard authorization request, which was submitted to the Standards Committee for consideration. In addition, the Functional Model Working Group is addressing industry concerns related to the Interchange Authority as it drafts version 4 of the functional model.


6

Operating Reliability Subcommittee David Zwergel, chairman of the Operating Reliability Subcommittee, reported that Frank Koza, PJM, was elected to serve as the subcommittee’s vice chairman. Chairman Mayo approved Mr. Koza’s appointment. Mr. Zwergel also noted that the ORS executive committee drafted responses for standard interpretation for EOP-002 (Capacity and Energy Emergencies) and for EOP-001 (Emergency Operations Planning). Both interpretations are posted for 30-day pre-ballot review.

Mr. Zwergel stated that the ORS approved minor revisions to the MISO and ERCOT reliability plans and conducted a preliminary review of a revised WECC regional reliability plan.

Mr. Koza summarized the on-going market flow field trial and stated that the Standards Committee approved extending the field trial to October 31, 2008 and increasing the market flow curtailment threshold from 3% to 5% effective June 1, 2008.

Mr. Koza also summarized ORS discussions related to its “Gap” analysis of SOL and IROL reliability standards (Presentation 5). The ORS analysis began with the collection of each reliability coordinator’s IROL selection criteria and an analysis of their operations planning methodology. Mr. Koza noted that the ORS would develop a standard authorization request to address the issue of defining outcome based scenarios to determine if an IROL exists. In addition, the ORS will discuss the criteria for reporting IROL exceedances to NERC and FERC.

Ben Li informed the committee of a FERC Order on Rehearing and Clarification, dated June 2, 2008, wherein FERC stated in paragraph 10:

…The statement in Order No. 705 regarding NERC’s definition of IROL Tv was not intended to prejudge the results of the survey of IROL practices that the Commission directed NERC to perform in Order No. 693. The Commission believes that operating the system within IROL limits under normal system conditions and exceeding IROL limits only after a contingency and subsequently returning the system to a secure condition as soon as possible, but no longer than 30 minutes, may be appropriate. This mode of operation will help ensure the system can withstand the single largest contingency without cascading failures. However, until the Commission acts on further information received in the study ordered in Order No. 693, it is not a violation of IRO-005-1 to exceed an IROL limit during normal operations, i.e., prior to a contingency, provided that corrective actions are taken within the required 30 minute time frame under IRO-005-1.


7

System Restoration and Blackstart Standard Drafting Team Richard Kafka, chairman of the System Restoration and Blackstart Standard Drafting Team, summarized the team’s progress is reviewing and revising several emergency preparedness and operations reliability s(see presentation excerpt at right (Presentation 6). The drafting team is currently developing responses to comments received from the posting of draft three of these standards. The team anticipates posting another draft of the standards for comment prior to pre-ballot review and balloting.

SRBSDT SRBSDT –– Project StandardsProject Standards

Original set: • EOP-005: System Restoration Plans • EOP-006: Reliability Coordination – System Restoration • EOP-007: Establish, Maintain, and Document a Regional

Blackstart Capability Plan • EOP-009: Documentation of Blackstart Generating Unit Test

Results

Proposed set: • EOP-005: System Restoration from Blackstart Resources • EOP-006: System Restoration Coordination • EOP-007 & EOP-009 will be retired. Requirements have been

incorporated into EOP-005 or EOP-006 or are no longer needed.

tandards

Reliability Readiness Program (cont’d) The committee returned to its discussion of the reliability readiness program. Tom Bowe moved that, although the NERC readiness evaluation program filled an important role in enhancing the industry’s reliability following the August 14, 2003 blackout, the Operating Committee believes that many of these functions have become redundant within the new ERO and compliance environment, and that most of the benefits of the program were achieved by the initial series of these evaluations. Additionally, the committee has serious concerns with the current proposal for the readiness assistance program. As such, the committee requests the opportunity to provide greater direction and recommends the future suspension of the current readiness evaluation program. The committee will provide this direction to NERC staff by July 1, 2008. This direction will at least address the current Board of Trustees Finance and Audit Committee proposal, mandatory readiness evaluations, information sharing, and best practices. The intent of this additional direction would be to establish a complete transition plan for the program by September 1, 2008. Following a brief discussion, the committee approved the motion. Chairman Mayo appointed a task force (Sam Holeman (chairman), Jerry Rust, Jerry Mosier, Tom Bowe, Dennis Florom, Jim Griffith, Ken Saathoff, Jacquie Smith, and Ron Donahey) to develop the transition plan as required in the above motion.

Backup Facilities Standard Drafting Team Tom Bowe, a member of the Backup Facilities Standard Drafting Team, summarized proposed revisions to reliability standard EOP-008 (Plans for Loss of Control Center Functionality) (Presentation 7). The drafting team is currently reviewing comments received during the first posting of the proposed standard and anticipates posting anther draft in the third quarter of 2008.

Critical Infrastructure Protection Reliability Tool Development Process ⎯ Stan Johnson, manager of situation awareness and infrastructure security, sought input on the proposed NERC reliability tool development process flow chart. At the OC’s March 2008 meeting, Chairman Mayo asked each of its subcommittees to review the proposed flow chart. The results of those reviews were summarized in the agenda


8

background. Furthermore, the committee recommends that the board Technology Committee (TC) should 1) address risk, reliability, and costs of operating reliability tools, 2) recognize that a potential conflict of interest may exist in providing reliability tools, and 3) annually review each reliability tool to ensure that it continues to meet its guidelines for reliability tools. The committee also recommends that NERC continue to support its reliability tools until a transition plan is developed.

Jim Griffith moved that as contracts related to existing reliability tools come up for renewal, those tools should be considered for transitioning from NERC. Following a brief discussion, Jim Griffith moved to postpone the motion indefinitely. The committee approved the motion.

Chairman Mayo will share the committee’s input with the board TC at its next meeting.

Reliability Coordinator Daily Report ⎯ Stan Johnson summarized NERC’s efforts to develop a reliability coordinator daily report, which NERC and FERC would use for situation awareness. He anticipates completing the development and implementation process by the end of June 2008.

Next Meeting The next meeting of the Operating Committee will be on September 10–11, 2008 in Seattle, Washington.

Adjourn There being no further business before the Operating Committee, Chairman Mayo adjourned the meeting on Thursday, June 5, 2008 at 11:36 a.m.

Larry Kezele Larry Kezele Secretary

NERC Reliability Readiness ProgramNERC Reliability Readiness Program

White Paper Development Update

Joint NERC Readiness Program /Operating Reliability Subcommittee

June 4-5, 2008

2

ObjectiveObjective

Mission of Readiness Program• Promote reliability and operations excellence

Continuous Improvement• Develop performance-based criteria

• Provide assistance (reliability information help desk, on-site consultation, webinars, reliability conference)

• Database search through NERC website

• Seek OC support and guidance

3

OC SupportOC Support

Provide technical support, guidance and advice to the Readiness Program

Develop criteria for measuring program success, and review the program against those criteria

Review readiness evaluations for trends and recommend new or different types of evaluations or changes in processes or metrics

4

On-Site Evaluations

Alliance/PartnershipIntegrated ReliabilityHelp Desk

Information Sharing

Ag

OC Guidance and Advice

- Needs or risk-based- Requested by entities seeking specific assistance

- Web-based database- Webinars- Reliability Conference

- Coordinate with REs- Provide complete and accurate answers- Not a Compliance Hotline- No legal advice

- Collaborative relationship- Promote dialogue on reliability

Details available at: ftp://ftp.nerc.com/pub/sys/all_updl/bot/finance/FAC-0530cca.pdf

5

Ad Hoc Review Group Formed• Four NERC employees

• Three ORS representatives

• One OC representative

• One Regional Entity representative

Met March 6, 2008, in Charlotte, NC

Subsequent teleconference on April 16, 2008

Pre-ORS meeting telecon on May 1, 2008

ORS meeting status update on May 7, 2008

Progress ReportProgress Report

6

Progress to DateProgress to Date

Outline of proposed white paper developed

Additional subjects added

Initial drafts developed and combined• One from NERC staff

• One from ORS member

• Additional input also received, various conversations

Combined effort reviewed April 25, 2008

Collected additional ORS input from the May 7th meeting

7

Readiness Evaluation and BenchmarkingReadiness Evaluation and Benchmarking

Combines two existing functions within NERC

Provides good impetus to continue to grow and evolve the readiness program

Will lead to a proactive development of partnerships among evaluated entities

Cycle III will begin a new emphasis on the evaluation products

Program assessment was part of the 2008 work plans

8

Initial ORS ObservationsInitial ORS Observations

The ORS consensus is:• That the program provides a valuable contribution to

reliability

• Industry should work together in a cooperative effort to improve the program

9

Key Items for DiscussionKey Items for Discussion

Program Identity

Confidentiality of Readiness Evaluation Reports

Focus / Scope of Readiness Evaluations

Consistency of Evaluations• Recommendations for improvement (raise the bar)

• Examples of Excellence

Industry Assistance (aka Mentoring)

10

White Paper OutlineWhite Paper Outline

Program Overview

Program Origin / History

Program Evolution• Readiness / Compliance

• Examples of Excellence Process

• Addition of Transmission Owners

• NERC Staffing and Communications

• Program Consistency

11

White Paper OutlineWhite Paper Outline

Program Future

Topics for Discussion• Identity

• Confidentiality

• Focus/Scope

• Consistency

• Industry Assistance

Conclusion

12

Program OverviewProgram Overview

Purpose — evaluates the quality and effectiveness of entity operations and processes

Approach — conducted by industry peers serving as subject matter experts

Focus — focuses on evaluating the quality of entity practices

Scope — broad review of system operations functions

Result — highlights positive observations; offers recommendations to entities for improving practices.

13

Program OriginProgram Origin

August 14, 2003 blackoutRecommendation 18 – “Support and strengthen NERC’s Reliability Readiness Program.”NERC adapted the WECC RMS compliance and evaluation program.• First Readiness Audit performed the week of March 8, 2004• Three year cycle due to large demand on industry participants

(rather than proposed 2 year cycle)

Principles of the program: Open, Fair, Transparent, Inclusive• Reports are published on the NERC website sans critical

infrastructure information

14

Evolution in an Environment of Evolution in an Environment of Mandatory ComplianceMandatory Compliance

Perception with respect to compliance audits• Changed to ‘Evaluations’ in 2007

Audit relies on evidence

Evaluation asses performance

• De-emphasis of standards and hard evidenceShifted to a more conversational model

• Program redefined in 2007 based on five primary program groupings

Corporate Culture; Fundamentals of Operations; Fundamentals of Maintenance; Fundamentals of Operational Planning; Fundamentals of Training

Focusing on control center activities and related infrastructure

15

Program EvolutionProgram EvolutionAddition of Transmission OwnersAddition of Transmission Owners

Originally referred to as ‘Local Control Centers’• Term has specific meaning in one Regional Entity

Generally involved in performing delegated functions

Increased program scope via registration information• Debate is currently underway with regard to the

frequency of evaluations

16

Program EvolutionProgram EvolutionNERC Staffing and CommunicationsNERC Staffing and Communications

Better communications needed as indicated from feedback received from NERC staff, Regional Entities, team volunteers, and companies• Identification of team products for the entity as well as

the industry

• Increased presence at industry meetings

Staffing increased at the end of 2007 to a total of five full time readiness evaluators

17

Program EvolutionProgram EvolutionProgram ConsistencyProgram Consistency

NERC has developed computer based training modules for evaluation team volunteers• Modules include interview skills, observation

techniques, dealing with different personalities

Objective is to increase the consistence given the diverse nature of the individual evaluation teams• Without limiting debate and forward thinking

18

Program FutureProgram Future

2009 to 2011 business plan• Staffing

• Identity

• Training

• Examples of Excellence

• Self Assessment

• Industry Assistance

• Communications

19

Topics for DiscussionTopics for DiscussionProgram IdentityProgram Identity

Timing of readiness evaluations and compliance audits• Schedules are determined in conjunction with the Regional

Entities

• Inference of compliance based on a readiness evaluation is not a viable planning process

• For entities seeing their first round of evaluations and audits,readiness might be a good way to guide preparation

Compliance personnel on evaluation teams• NERC relies on industry volunteers and Regional Entities to field

evaluation teams

20

Program IdentityProgram Identity

Identification of probable compliance violations during a readiness evaluation• The NERC Rules of Procedure, as approved, require reporting for

resolution through the Regional EntitySelf reporting during the onsite visit is encouraged

Teams do not seek evidence of violations

Not part of the evaluation report unless a recommendation is associated

Recommendation from the ORS to alter the NERC rules of procedure to remove the reference to probable violations (Paragraph C.3, page 5)

Recommendation Tracking• Seeking to determine the value of the process

• Prior recommendations are not formally discussed as part of a current evaluation

21

Resolution on ConfidentialityResolution on Confidentiality

Posting a report is appropriate• Process is open, and transparent

What should be included in the posting?• Team products are important to the industry

Critical infrastructure information is always redacted from the published report

Provide better access to information (better database or tool)

22

Program Focus and ScopeProgram Focus and Scope

Flexibility to request a more targeted evaluation• Narrow but deep evaluation might be requested by an

entity (perhaps) in addition to the standard evaluation process – part of the NERC business plan for Cycle III

• Completion of Cycle II of the evaluations will provide a good baseline of experiences

23

ConsistencyConsistency

Common themes from feedback include overall consistency, recommendations, examples of excellence, program metrics• Training and continuity of leadership will improve

consistencyTraining is now required of all evaluation team members

• Team leaders and regional representatives must help evaluation teams develop meaningful recommendations and examples of excellence

Team discussions are very dynamic, and based largely on the expertise of the team members

24

Industry AssistanceIndustry Assistance

There is a need for a mechanism to partner entities receiving recommendations with high performers in the specific program area• Development of a higher level report format is

suggested

NERC should conduct technical conferences to present examples of excellence to the industry• Requires care to identify the correct audience

25

Industry AssistanceIndustry AssistanceExamples of ExcellenceExamples of Excellence

Codify and communicate process• Including alternative nomination process (self nomination)• Consideration of a fall ‘Reliability Summit’ where presentations

could be made for EoEs• Establish criteria for nomination

Emphasis on technology and toolsAllow process to better recognize regional and sub-regional practices and toolsAllow EoE process to consider entities with similar characteristicsBetter archive and update EoEs

26

ConclusionConclusion

Identify those areas of consensus and agreement

Highlight areas where the industry should continue to debate and give input

Create a roadmap for the future, that can be incorporated into the next set of NERC business plans

27

In SummaryIn Summary

Cooperative Effort• Directed by the NERC OC

• Knowledgeable small group

• Significant ORS input

Provides a good base for continued discussion

Memorializes information about the program

Will result in better service to the industry

28

Questions?

Summary of Operating Committee Discussion of Readiness Evaluation Program June 4-5, 2007

Initially Valuable: Program initially served a great purpose. With NERC now into compliance, benefit of program is significantly lessened. There was significant benefit to having the first evaluation. Follow-up evaluations are not nearly as beneficial. Outlived Its Usefulness/ Kill Program: Option 1 [Retain Program as is], take this off the table. Dollars are getting squeezed. Region X would like to kill. No support for program in Region X after first cycle. General consensus in Region X is that the program should go away. Prioritizing of programs and dollars would indicate that.

There are budget and manpower constraints. Program has outlived its time frame of usefulness.

Concerned about the cost effectiveness of program. NERC is providing a service that is very resource intensive. Standards development process is very resource intensive as well, and more beneficial. Other forums such as TOOF provide this service.

Program has out lived its usefulness. Don’t see the incremental benefit of continuing going forward. The program could have achieved better results if the opportunity of open

dialog existed, but because of the risk of findings of non-compliance, open dialog was lacking. With compliance at the forefront, evaluations outlived usefulness. Need to get NERC out of this process.

Readiness evaluation will not help address cyber standard implementation. Retain Some/All of Program: Some residual value of program: certification of new entities, reviewing entities that score especially high (for Examples of Excellence (EoE)) or low in metrics. Stop short of killing the program. Readiness evaluations of cyber standards or other new standards would be beneficial. Readiness evaluations focused on specific standards may be beneficial. Some value to evaluations. Current timeframe is probably not appropriate. Program does need to be scaled back. TOOF is not open to everyone. Need something for non-members of TOOF. Combine Compliance and Readiness Audits: Benefit to combining compliance and readiness programs and audits. May be an opportunity to use this experience [of the ORS discussions of the Readiness Evaluation Program] to improve the compliance audits. During compliance audits look for best practices. Don’t need to perform a duplicative readiness evaluation.

1

Help Desk: Help desk function at NERC may not provide answers that are consistent with regional enforcement. NERC help desk would fail. If an entity has a standard question go to standards program or a compliance question go to compliance program. Not go to single source. Help desk not likely to work because questioners need definitive answers. Help desk questionable. Scope of Help desk and on-site consulting needs to be very clear. Help desk concept is noble, but belongs in the compliance area. Voluntary Evaluations:

If you want an evaluation you pay for all expenses. Evaluations should be voluntary.

NERC should not compete with others to provide fee-based reviews. Pay as you go is a good idea. Most companies already have mandatory [financial] audits. Auditor also

provides best practices. You pay for those audits. If IRS compliance becomes an issue, you go to the auditor for help. Should have similar model for reliability evaluations.

NERC as a compliance consultant does not make any sense. Redeploy Resources:

Refocus the readiness folks on other programs such as events analysis. There should be something in the program that NERC can build on and re-

channel staff to work on, such as situational awareness. Better use of resources – include operations lessons learned, and not just engineering lessons learned, from event analysis.

Several committee members mentioned providing more resources to event analysis. Examples of Excellence (EoE): There is value in EoE. Don’t allow residual value to die. EoE could be absorbed into compliance. EoE are great and need to be pursued. However, they become outdated. Other: As great as the program has been, there may be a conflict if something is discovered that is not reported to compliance. NERC proposal seems to be more of a holding pattern option while waiting to determine what to do with the NERC employees in the readiness evaluation program.

Even with the proposed change, there is no reduction in the NERC budget. Need innovation and better ways to perform duties – concerned that we are

taking our eyes off of the ball in terms of reliability and security of the system. Several committee members mentioned questions around entity certification

(initial) and how that would work without readiness reviews. ORS members mentioned that ORS could take on the initial entity certification

readiness review responsibilities.

2

file:///C|/...s%20and%20Settings/hawkinsh/Desktop/DR%20Documents/OC%20Motion%20re%20Readiness%20Evaluation%20Program.htm[3/15/2009 1:42:26 PM]

From: Larry J Kezele [[email protected]]Sent: Thursday, June 05, 2008 10:20 AMTo: [email protected]: OC Motion re Readiness Evaluation ProgramTo: Operating Committee The following motion was approved by the Operating Committee at its June 4-5 meeting: Tom Bowe moved that although the NERC Readiness Evaluation Program filled an important rolein enhancing the industry’s reliability following the August 14, 2003 Blackout, the NERC OperatingCommittee believes that many of these functions have become redundant within the new ERO andcompliance environment, and that most of the benefits of the program were achieved by the initialseries of these evaluations. Additionally, the OC has serious concerns with the current proposal for the Readiness AssistanceProgram. As such, the NERC OC requests the opportunity to provide greater direction and recommends thefuture suspension of the current Readiness Evaluation Program. The OC will provide this directionto NERC staff by July 1, 2008. This direction will at least address the current FAC proposal,mandatory readiness evaluations, information sharing, and best practices. The intent of thisadditional direction would be to establish a complete transition plan for the program by September1, 2008. Larry KezeleManager - Reliability Support ServicesWork (609) 452-8060Cell (609) 273-0839

NOTICE:This e-mail and any of its attachments may contain NERC proprietary information that is privileged, confidential, orsubject to copyright belonging to NERC. This e-mail is intended solely for the use of the individual or entity to whichit is addressed. If you are not the intended recipient you are hereby notified that any dissemination, distribution,copying, or action taken in relation to the contents of and attachments to this e-mail is strictly prohibited and may beunlawful. If you have received this message in error, please notify the sender immediately and permanently delete theoriginal and any copy of this e-mail. ---You are currently subscribed to oc_plus as: [email protected] unsubscribe send a blank email to [email protected]


609.452.8060 | www.nerc.com

Agenda Operating Committee September 10, 2008 | 1–5 p.m. September 11, 2008 | 8 a.m.–noon Doubletree Hotel Seattle Airport 18740 International Boulevard Seattle, Washington 206-246-9600

Item Leader Action


a. Quorum

b. Procedures




a. Minutes of June 4−5, 2008 meeting

3. FYI

a. NERC Board of Trustees Chairman

b. Reliability Fundamentals Working Group

Mark Fidrych

4. Committee Charter Revisions Secretary Approve

5. NERC Budget – Impact of Committee and Subgroup Meetings

Chairman Discussion

6. Operating Committee Work Plan Vice Chairman Discussion

7. Long-Term Reliability Assessment William Bojorquez and Mark Lauby

Approve

continued…

Operating Committee Meeting Agenda September 10–11, 2008 2

Item Leader Action

8. Functional Model Version 4.0 Jim Cyrulewski and Mark Fidrych

Approve

9. Integration of Variable Generation Task Force Report – Operational Perspective

Warren Frost Discussion

10. Future Path of Transmission Loading Relief Tom Mallinger Discussion

11. Reliability Readiness Program Jessica Bian and Kevin Conway

Discussion

12. Reliability Metrics Working Group Jason Shaver and Jessica Bian

Discussion

13. Continuing Education Program Martin Sidor Information

14. System Personnel Training Standard Drafting Team

Patricia Metro Discussion

15. WECC Reliability Plan Linda Perez Approve


17. Resources Subcommittee Terry Bilke

a. Frequency Trends Information

b. NAESB Time and Inadvertent Interchange Management

Information

c. Performance Standards Reference Document

Approvel

18. Underfrequency Loading Shedding Relays Vice Chairman Discussion

19. Next meeting – December 3–4, 2008 (Orlando, FL)

Secretary

Item 11. Reliability Readiness Program Discussion Jessica Bian, director of Reliability Readiness and Benchmarking, and Kevin Conway, reliability readiness evaluator, will provide a status report of the Readiness Evaluation and Improvement Program. The program has reduced the remaining 2008 evaluations scheduled based on a criteria developed to bring the program to a close by the end of the first quarter of 2009.

Background (Excerpt from the Board of Trustees Approved 2009 NERC Business Plan and Budget) The NERC Readiness Evaluation and Improvement Program began in response to FERC directives following the August 14, 2003 blackout. The goal was to increase transparency on operating practices and to assess the industry’s overall preparedness to minimize the likelihood of another major blackout.

When FERC approved NERC as the electric reliability organization and began adopting NERC standards as mandatory in the United States, NERC created an associated compliance audit program (Compliance Monitoring and Enforcement Program — CMEP). The readiness program, which had been asking compliance-type questions, shifted its full effort to encouraging entities to improve reliability performance and achieve excellence in their assigned functions and responsibilities.

The readiness program has been working with industry experts to conduct on-site evaluations of all balancing authorities, transmission operators, reliability coordinators, and other entities that support the reliable operation of the bulk power system in North America to determine their readiness to maintain safe and reliable operations.

While feedback from entities that have participated in readiness evaluations has generally been favorable, industry comments on the first draft of the NERC 2009 Business Plan and Budget suggest that: (1) the value of the readiness program is diminishing; (2) the focus and resources of NERC should be squarely on standards and compliance particularly during this critical time; and (3) the mandatory enforcement role at NERC creates a conflicted environment for the readiness program, suggesting the role can be better undertaken by others (such as the Transmission Owners and Operators Forum). This leads to the overall conclusion that the assessment-oriented evaluation process in its current form be discontinued.

Based on the direction provided by the NERC Board’s Finance and Audit Committee, the readiness evaluation program will complete the remaining scheduled readiness reviews and close at the end of the first quarter of 2009.


609.452.8060 | www.nerc.com

Minutes Operating Committee September 10–11, 2008 Doubletree Hotel Seattle Airport Seattle, Washington Convene


1. Approved a revised committee charter.

2. Reviewed the status of the System Personnel Training Standard Drafting Team.

3. Approved the initial membership of the Reliability Fundamentals Working Group tasked with overseeing the maintenance of the Reliability Concepts document.

4. Discussed the impacts of the committee and its subgroups on the NERC budget and recommended ways of reducing costs.

5. Began preliminary discussions related to the development of the committee’s 2009-2011 work plan.

6. Approved the technical content of version 4 of the Functional Model.

7. Approved the WECC reliability plan, subject to WECC receiving reliability coordinator certification.

8. Reviewed the proposed Operational Reliability section of the Long-Term Reliability Assessment.

A regular meeting of the NERC Operating Committee (OC) was held on September 10−11, 2008, in Seattle, Washington. The meeting notice, agenda, and attendance list are affixed as Exhibits A, B, and C, respectively; and individual statements and minority opinions as Exhibits D and E, respectively. The meeting presentations are posted in a separate file at http://www.nerc.com/~filez/ocmin.html.

Operating Committee chairman Gayle Mayo convened the meeting at 1 p.m.


Antitrust Compliance The secretary referred the committee to the NERC Antitrust Compliance Guidelines included in the meeting agenda.

Consent Agenda The committee approved the June 4–5, 2008 meeting minutes.

FYI Board of Trustees ⎯ Chairman Mayo summarized discussions at the July 30, 2008 Board of Trustees meeting, especially those related to the Electricity Sector Steering Group (ESSG) charter, staffing a senior executive as the “Chief Security Officer” who will serve as a single point of contact for the industry, the ESSG, and government stakeholders seeking to communicate with NERC on cyber and infrastructure security matters, and approval of a change to the charter of the Transmission Owners and Operators Forum to increase diversity of membership.

Reliability Fundamentals Working Group ⎯ At its June 2008 meeting, chairman Mayo tasked the Nominating Subcommittee with soliciting members to serve on the Reliability Fundamentals Working Group (RFWG). Subcommittee chairman Mark Fidrych reported that the subcommittee recommends appointing Stephen Lee (EPRI), Steve Myers (ERCOT), Francis Halpin (BPA), Ali Chowdhury (University of Missouri – Rolla), Peter Brandien (ISO-NE), and Jerry Rust (Northwest


Operating Committee Meeting Minutes September 10–11, 2008

2

Power Pool) to the RFWG. Chairman Mayo accepted the subcommittee’s recommendations and noted that the RFWG scope was included in the agenda material.

Committee Charter Revisions The secretary reviewed proposed revisions to the committee’s charter related to approval of the Functional Model and committee membership (e.g., terms and proxies). Jim Griffith moved to approve the revisions to the Operating Committee charter. The committee approved the motion.

NERC Budget – Impact of Committee and Subgroup Meetings Chairman Mayo summarized discussions of the Standing Committees Coordinating Group and the committee’s executive committee related to the reduction of costs associated with conducting committee and subgroup meetings. Examples include:

• Eliminating some joint meetings and/or conducting some joint meetings by WebEx

• Holding one-day committee meetings

• Recommending that subgroups hold their meetings at company or Regional Entity offices, thus eliminating hotel meeting room costs

• Merging the Operating and Planning Committees

Following a brief discussion of the merits of each of the above examples, chairman Mayo stated that the committee’s executive committee would further address this topic.

Operating Committee Work Plan Chairman Mayo stated that the committee’s current work plan was approved at its December 2007 meeting and that she asked vice chairman Holeman to lead a discussion of this topic at this meeting (Presentation 1). Following a brief discussion related to the need to revise the work plan and topics that the work plan should encompass, chairman Mayo appointed vice chairman Holeman, Jerry Rust, Tom Bowe, Jim Griffith, Mark Fidrych, and Lorne Midford to review andrevise the work plan for discussion at the committee’s next meeting.

Future Path of Transmission Loading Relief Tom Mallinger, MISO, summarized the history of the Eastern Interconnection’s transmission loading relief (TLR) procedure (Presentation 2). Mr. Mallinger noted that the TLR procedure:

• Is the primary congestion management procedure used over the last 10 years, and that only minor modifications have been made to the procedure during this time.

• Is used when significant, externally induced parallel flows make local procedures insufficient to control facility loading.

• Relies on e-Tag curtailments to curtail non-firm usage and a combination of e-Tag and network and native load (NNL) relief obligations to curtail firm usage (share-the-pain approach).

However, use of the TLR procedure has resulted in industry complaints related to:


3

• Implementation of the TLR procedure can result in a large number of e-Tag curtailments for a small amount of relief; thus, use of TLR is disruptive to the markets and has resulted in entities scheduling around bottlenecks.

• The NNL relief obligation is based on a static set of assumptions contained in the interchange distribution calculator (IDC). As such, the IDC does not rely on real-time generator, load or net interchange information.

• Because the NNL calculation is based on static assumptions, reliability coordinators lack visualization of the magnitude and source of parallel flows when they experience congestion.

To address these industry complaints, Mr. Mallinger stated that MISO, PJM, and SPP submitted a standard authorization request (SAR) that would require all Eastern Interconnection reliability coordinators to report their generator-to-load impacts to the IDC on a real-time basis. Hence, a reliability coordinator experiencing congestion would have visualization of the magnitude and source of all flows affecting their congested flowgate using information in the IDC.

Mr. Mallinger noted that the Operating Reliability Subcommittee (ORS) and the Reliability Coordinator Working Group discussed the proposed SAR. While neither group endorsed the concept, they did ask for a cost/benefit analysis of the proposal.

Functional Model Version 4 As a follow-up to the Functional Model Working Group’s report at the joint session, chairman Mayo asked if the committee had any questions of working group chairman Jim Cyrulewski. The committee expressed concerns related to the Interchange Authority, which Mr. Cyrulewski noted would be more thoroughly addressed by the working group during its development of version 5. Mark Fidrych moved to approve the technical content of version 4 of the Functional Model and the Functional Model Technical Document. The committee approved the motion.

Events Analysis Bob Cummings, director events analysis and information exchange, highlighted the following events currently being analyzed (Presentation 3):

1. Eastern Interconnection Frequency Disturbance – August 4, 2007 ⎯ Analysis team is editing the final report, which will have limited distribution due to critical infrastructure sensitivities. Mr. Cummings outlined the follow-up recommendations resulting from the analysis.

2. MRO System Separation – September 18, 2007 ⎯ Analysis team is writing semi-final report. Although the report is not yet final, Mr. Cummings outlined the preliminary follow-up recommendations from the analysis.

3. FRCC South Florida Disturbance – February 26, 2008 ⎯ FRCC analysis team is writing the final report.

4. RFC PEPCO Disturbance – June 13, 2008 ⎯ ReliabilityFirst is forming an event analysis team.


4

5. SPP Southwestern Public Service Company Separation – June 17, 2008 ⎯ The analysis of this event is just beginning.

Integration of Variable Generation Task Force Report – Operational Perspective As a follow-up to the Integration of Variable Generation Task Force report at the joint session, task force chairman Warren Frost noted that the Planning Committee requested a summary of best practices be included in the final report. The committee suggested that the task force report will help educate various industry stakeholders (e.g., industry regulators) regarding the technical challenges of implementing variable generation resources. The committee also requested the task force to address the operational details of all limited energy resources (e.g., wind generation and demand-side management). The task force anticipates presenting its preliminary report to the committee at its December 2008 meeting. Chairman Mayo noted that the committee will take action on the proposed recommendations after it has received the final report.

WECC Reliability Plan Linda Perez, WECC managing director of reliability coordination, summarized the WECC revised reliability plan and efforts to consolidate the three WECC reliability coordinators (Pacific Northwest Security Coordinator, Rocky Mountain–Desert Southwest Reliability Coordinator, and California-Mexico Reliability Coordinator) into one reliability coordinator with operating desks in Loveland, Colorado, and Vancouver, Washington (Presentation 4). Ms. Perez noted that there will be visual links between the two operating centers and that the transition is scheduled to occur on January 1, 2009.

Operating Reliability Subcommittee vice chairman Frank Koza stated that the ORS reviewed the revised WECC reliability plan and endorsed it for presentation to the Operating Committee. Tom Bowe moved to endorse the WECC Reliability Coordinator Plan, contingent upon WECC receiving NERC reliability coordinator certification prior to the expected January 1, 2009 startup. The committee approved the motion.

Boreas Vulnerability Working Group Chairman Mayo reviewed a memorandum from Michael Assante, NERC’s chief security officer, regarding a request for assistance in the formation of a Boreas Vulnerability Working Group. The request notes that NERC, in its function as the ES-ISAC, issued an advisory dated August 27, 2008, describing Boreas (a computer vulnerability that affects industrial control systems). The working group, in conjunction with the Department of Homeland Security and the National Cyber Security Division, will develop a greater understanding of the technical issues, evaluate potential technical impacts, and assess the need for providing additional information. Chairman Mayo asked for committee volunteers to serve on the working group.

Continuing Education Program Martin Sidor, director of training, education, and personnel certification, highlighted efforts to revise the Continuing Education Administrative Manual (Presentation 5). The manual outlines the continuing education program process and rules, including provider rules. The manual was last updated in 2007. Mr. Sidor will post the manual for a 30-day comment period in late September 2008 and inform the committee of the results of that posting at its December meeting.


5

Underfrequency Load Shedding Relays Vice chairman Holeman stated that the 2003 blackout report contained recommendations related to the implementation and coordination of underfrequency load shedding relay programs. He noted that some university researchers are asking if such programs are coordinated across Regional Entity boundaries. The committee noted that Regional Entities are coordinating the implementation of underfrequency load shedding relays within their Region in accordance with reliability standards. Furthermore, requirement R1.1 of PRC-006-0 (Development and Documentation of Regional UFLS Programs) requires “…coordination of UFLS programs within the subregions, Regional Reliability Organization and, where appropriate, among Regional Reliability Organizations.”

Adjourn and Reconvene The committee adjourned at 5:09 p.m. PDT and reconvened the following morning at 8:02 a.m.

Resources Subcommittee Terry Bilke, chairman of the Resources Subcommittee, lead the committee in a discussion of the following topics (Presentation 6):

• Frequency Performance ⎯ Mr. Bilke noted that the increase in the number of smaller frequency excursion is a direct result of a continuing decline in the Eastern Interconnection frequency response. In addition, an analysis of the data suggests that the majority of larger frequency excursions (greater than 0.05 Hz) occur during time corrections.

• NAESB Time and Inadvertent Management ⎯ Mr. Bilke highlighted FERC concerns, as expressed in Order 693, with inadvertent balances, and NERC and NAESB efforts to address those concerns. For example, NAESB is considering a different approach to implementing time error corrections.

• Process for Addressing FERC Requests and Questions ⎯ As chairman of the Resources Subcommittee, FERC has requested access to NERC reliability tools and access to data generated by those tools. Chairman Mayo noted that FERC cannot be granted access to NERC reliability tools until the Canadian data issue is resolved. She suggested that the committee’s executive committee review and respond to each request prior to action being taken by any of the committee’s subgroups.

• Performance Standard Reference Document ⎯ Mr. Bilke reviewed proposed revisions to the Performance Standard Reference Document and explained that the purpose of the document is to provide instruction for calculating a balancing authority’s control performance. The committee noted that, while the document contains a compliance disclaimer, the word “compliance” is used repeatedly. The committee debated whether the document should be a technical reference document to support implementation of a reliability standard, or a committee-approved guideline. Chairman Mayo asked the secretary to research each of these alternatives and to report his findings to the executive committee.


6

Long-Term Reliability Assessment Mark Lauby, manager of reliability assessments, provided an overview of the new operational reliability section of the Long-Term Reliability Assessment (Presentation 7). The objectives of the operational reliability section include:

• Collect and monitor industry-wide data that can be used to assess industry trends in maintaining reliable performance of the bulk power system.

• Assess the reliability significance and causes of any statistically significant adverse industry trends, determine if the trends represent an actual degradation in overall industry reliability performance, and respond appropriately to any reliability issues that may be identified.

• Communicate industry-level information to stakeholders in an effective and timely manner.

The operational reliability section was reviewed by the Reliability Metrics Working Group and the Reliability Assessments Subcommittee. Some of the proposed metrics include the number of bulk power system disturbances by severity and year, the number of transmission loading relief events resulting in interruption to firm load, and the number of energy emergency alert events. The committee expressed concerns that the operational reliability section was Eastern Interconnection centric, and suggested other potential sources of data (i.e., market monitor reports) that address areas of congestion management in significantly more detail. Mr. Lauby will redraft the operational reliability section for further committee review.

Reliability Readiness Program Kevin Conway, reliability readiness evaluator, summarized the expected transition of the reliability readiness program through the first quarter of 2009 (Presentation 8). In July 2008 the Board of Trustees approved version 3 of the NERC budget, thus approving the phase out of the reliability readiness program by the end of the first quarter of 2009. All reliability coordinator, balancing authority, and transmission operator reliability readiness evaluations scheduled during the fourth quarter of 2008 were cancelled. Mr. Conway noted that program staff will continue working with the Operating Reliability Subcommittee to review and finalize for posting potential examples of excellence.

Chairman Mayo thanked the Mr. Conway and Ms. Jessica Bian for their quality work since the program was initiated in response to the 2003 blackout. The committee acknowledged their accomplishments with applause and congratulations.

Reliability Metrics Working Group As a follow-up to his presentation at the joint session, Jason Shaver, chairman of the Reliability Metrics Working Group (RMWG), reviewed the working group’s recommended metrics to support the Long-Term Reliability Assessment (Presentation 9). Mr. Shaver noted that in 2009 the RMWG would 1) continue its development of a metrics framework using the draft white paper “Toward Ensuring Reliability: Reliability Performance Metrics” as input, and 2) identify adequate level of reliability metrics and associated underlying data requirements.


7

System Personnel Training Standard Drafting Team Patti Metro, chairman of the System Personnel Training Standard Drafting Team, reviewed proposed reliability standard PER-005-1 (System Personnel Training) (Presentation 10). The purpose of PER-005-1 is to ensure that system operators performing real-time, reliability-related tasks on the North American bulk power system are competent to perform those reliability related tasks. When PER-005-1 becomes effective, PER-002-0 (Operating Personnel Training) and certain requirements in PER-004-1 (Reliability Coordinator – Staffing) will be retired. The drafting team anticipates posting PER-005-1 for 30-day pre-ballot review in September 2008 followed by a 10-day ballot.

Next Meeting The next meeting of the Operating Committee will be on December 3–4, 2008 in Orlando, Florida.

Adjourn There being no further business before the Operating Committee, chairman Mayo adjourned the meeting on Thursday, September 11, 2008 at 9:33 a.m. PDT.

Larry Kezele Larry Kezele Secretary

RReeaaddiinneessss EEvvaalluuaattiioonn aanndd IImmpprroovveemmeenntt PPrrooggrraamm 22000088 WWhhiittee PPaappeerr

Month Day, 2008

Chapter # — Chapter Title

Name of Report Month 20XX 2

TTaabbllee ooff CCoonntteennttss Introduction..................................................................................................................................... 3

Ad-Hoc Review Group ................................................................................................................... 3

Program Overview.......................................................................................................................... 4

Program Origin ............................................................................................................................... 5

Program Evolution.......................................................................................................................... 7

Program Future ............................................................................................................................. 10

Topics for Discussion ................................................................................................................... 11 Identity ...................................................................................................................................... 11 Confidentiality .......................................................................................................................... 13 Focus/Scope .............................................................................................................................. 13 Consistency ............................................................................................................................... 14 Industry Assistance ................................................................................................................... 17

Conclusion .................................................................................................................................... 20



IInnttrroodduuccttiioonn On March 6, 2008 an ad hoc meeting was held in Charlotte, North Carolina among members of NERC’s Operating Reliability Subcommittee (ORS), the manager of readiness from SERC, the vice chairman of the NERC Operating Committee, and members of NERC Reliability Readiness Evaluation and Improvement Program staff to discuss aspects of the readiness program. This and other meetings prompted this paper. The Reliability Readiness Evaluation and Improvement Program is advised with industry input through the NERC Operating Committee (as a sponsoring committee). The Operating Committee has specified this role in its charter. The NERC ORS is a subcommittee of the Operating Committee and, as such, has been given some latitude to advise the readiness program on technical issues, identification of examples of excellence, and methods for providing increased value to the industry. During the March meeting, a number of concepts were chosen to be explored further. The goal is to continue to bring benefit to the industry through this program, but the program will need to change to meet the changing needs of the industry. This white paper was deemed as a way to provide the baseline information needed to allow informed and productive discussions regarding the purpose, direction, and value of the readiness program. Through these discussions, increased value to the industry will be realized. AAdd--HHoocc RReevviieeww GGrroouupp Richard Schneider NERC Director of Reliability Readiness Sam Holeman DUKE Vice Chairman, NERC Operating Committee James Castle NYISO Member, NERC ORS Ben Deutsch SERC Manager, Readiness and Situation Awareness James Hartwell NPCC Member, NERC ORS Frank Koza PJM Member, NERC ORS Kevin Conway NERC Reliability Readiness Evaluator Mitch Needham NERC Reliability Readiness Evaluator Kristin Iwanechko NERC Reliability Readiness Technical Analyst



PPrrooggrraamm OOvveerrvviieeww The Reliability Readiness Evaluation and Improvement Program works with industry experts to conduct on-site evaluations to independently review the culture, tools, facilities, and processes of all balancing authorities, transmission operators, reliability coordinators, and other entities that support the reliable operation of the bulk power system in North America to determine their readiness to maintain safe and reliable operations. The evaluations identify areas of excellence in operations as a product for the industry in general and areas in need of improvement, from the evaluation team’s perspective, as a product for the company being evaluated. NERC uses the results of these evaluations to give entities ideas to improve their operations and move beyond just meeting standards. A readiness evaluation offers value to an entity and the industry as a consulting exercise:

Purpose — evaluates the quality and effectiveness of entity operations and processes. Approach — conducted by industry peers serving as subject matter experts to point out positive aspects of an entity’s operation and offer recommendations for improving practices. Focus — focuses on evaluating the quality of entity practices and determining entity readiness and preparedness for present and future operations. Scope — broad review of system operations functions with the assumption that standard requirements are being met (a caveat is in order: according to the NERC Rules of Procedure, an evaluation team is obligated to report evidence “of possible noncompliance with a reliability standard” to NERC “for resolution through the applicable compliance enforcement program”; please note, teams do not seek such items). Result — highlights examples of excellence and positive observations; offers recommendations to entities for improving practices.



PPrrooggrraamm OOrriiggiinn The NERC Reliability Readiness Evaluation and Improvement Program (originally referred to as a readiness audit program) began in response to Federal Energy Regulatory Commission (FERC) directives following the August 14, 2003 blackout. As a result of the blackout, FERC, NERC, and the U.S.-Canada Power System Outage Task Force (the group that investigated the blackout) all recognized a need to assess the vulnerability of similar events happening again on the bulk power system in North America. On February 10, 2004, the NERC Board of Trustees adopted the recommendations of the U.S.-Canada Power System Outage Task Force final report. Recommendation 18 particularly related to readiness:

“18. Support and strengthen NERC’s Reliability Readiness Audit Program. A. Readiness Audits

NERC: In its directives of February 10, 2004, NERC indicated that it and the regional councils would jointly establish a program to audit the reliability readiness of all reliability coordinators and control areas within three years and continuing thereafter on a three-year cycle. Twenty audits of high-priority areas will be completed by June 30, 2004, with particular attention to efficiencies identified in the investigation of the August 14 blackout. Task Force: Recommends that the remainder of the first round of audits be completed within two years, as compared to NERC’s plan for three years.

B. Public Release of Readiness Audit Reports

Task Force: Recommends that NERC require all readiness audit reports to be publicly posted, excluding portions pertaining to physical and cyber security. Reports should also be sent directly to DOE, FERC, and relevant authorities in Canada and state commissions. Such reports should draw clear distinctions between serious and minor violations of reliability standards or related requirements.”

NERC quickly responded to this request and instituted a reliability readiness audit program to assess the readiness of bulk power operations throughout North America (adapted from a WECC program already in place). The first audit took place the week of March 8, 2004. Absent national enforceable and mandatory NERC standards, the intent of the program was to increase transparency on operating practices and take the industry’s temperature on overall preparedness to minimize the likelihood of another major blackout. By leveraging publicly posted and balanced reports, NERC could encourage the best practices of the industry, identify industry trends, and become a proactive force in encouraging national reliability practices. The program has been one of the many driving forces contributing to increased reliability of the bulk power system.

Comment [kc1]: Verify this as a quote, and correct as nessesary. Need to include a footnote.



It is important to point out that the Recommendation 18 identified reliability coordinators and control areas as entities to be audited. A three-year audit cycle was chosen to be retained over the recommended two-year cycle due to the demand on existing resources and the logistics needed for program expansion. (Currently, reliability coordinators, balancing authorities, transmission operators, and other key entities are evaluated). Operating under the principles of being open, fair, transparent, and inclusive, the audit (since renamed evaluation) reports are posted on the public Web site. Critical infrastructure information is not publicly posted but is communicated to the entity in a private, confidential version of the report.



PPrrooggrraamm EEvvoolluuttiioonn In February 2007, the Reliability Readiness Evaluation and Implementation Program began its second three-year cycle of evaluations. The program incorporates many of the lessons learned from industry feedback obtained during the first three-year cycle. In addition, the program quickly acknowledged the impacts of the compliance audit process on the NERC community and has further enhanced the evaluation process to continue to provide benefit to the industry. A major issue for readiness has been establishing its own identity, as opposed to being perceived as having a connection with compliance. The program has and will continue to promote its purposes. Readiness v. Compliance As a result of the United States Congress enacting the Energy Policy Act of 2005, FERC approved NERC as the Energy Reliability Organization (ERO) and adopted NERC standards as mandatory in the United States. The enforcement of mandatory standards compliance in the United States began on June 18, 2007. (The efforts to recognize these standards throughout Canada and Northern Mexico are ongoing.) Mandatory compliance, and the creation of an associated NERC compliance audit program, has impacted the readiness program in a number of ways. One ongoing issue with the readiness program’s identity is the appearance of the duplication of effort with the compliance program. When readiness began, the program asked compliance-type questions patterned after the mandatory compliance program already established in the Western Interconnection. NERC’s voluntary standards were used as a guideline for how to approach evaluations, and the program evaluated entities readiness to meet its obligations as a control area or reliability coordinator. This is in recognition of the fact that standards cannot prescribe every aspect of reliable operations. Standards set the minimum threshold for performance. Now that NERC has a compliance program based on mandatory standards, readiness has dropped all references to standards compliance and shifted its full effort to evaluating operations and planning processes. The readiness program now focuses entirely on the subjectivity of encouraging excellence in utility practices. Readiness and compliance are different, each with unique goals. The compliance program focuses on whether an entity complies with approved written standards, while the readiness program evaluates how prepared an entity is from a process view. The compliance audit is objective and backward looking, while the readiness evaluation is subjective and forward looking. The readiness effort is a proactive method for encouraging operational excellence. The specific focus of the program is on reliability coordinators, transmission operators, balancing authorities, and transmission owners with delegated tasks that could affect the operation of the bulk power system. Shaking the compliance perception, however, is ongoing. One visual link to compliance was the term “audits.” Both the compliance and readiness groups were using the term. In order to further emphasize the difference between readiness and compliance, the readiness program stopped using audit and adopted “evaluation.”



Examples of Excellence Process During the FERC hearings on mandatory compliance, one issue of concern was repeated: the Commission as well as others wanted to maintain objective and enforceable standards but were concerned that the standards would result in “lowest common denominator” practices. This issue relates to the role of the readiness program. Readiness is rooted in the recognition that standards cannot prescribe all aspects of reliable operations and that NERC standards present a threshold, not a target, for performance and excellence in the industry.

NERC wanted to promote “best practices,” but there were issues with the use of that particular term. There was a fear that best practices could lead to de facto standards; a decision was made to use the term “example of excellence.” The examples of excellence process continues to be an important element of the readiness program. If an evaluation team deems a practice especially effective or unique, the team will nominate the practice as a “potential” example of excellence. Readiness evaluators vote to determine if the practice should be considered an actual example of excellence. (The ORS is now involved in the process, which is discussed further in the Topics for Discussion section.) If the example is approved, NERC drafts a one- to two-page essay for posting on NERC’s Web site. Addition of Transmission Owners During the first round of readiness evaluations, it became obvious that a number of major transmission owners, who were performing certain delegated reliability tasks for their host transmission operators, were not being evaluated. In 2005, the program was expanded to evaluate these transmission owners (originally referred to as local control centers), which are identified through collaboration with each Regional Entity. Communications and Staffing Through feedback from NERC staff, Regional Entities, evaluation team volunteers, and evaluated entities, it became clear that the program needed to communicate better with the industry. Unfortunately, several factors kept the program changes from being effectively communicated to the industry. With the industry’s overwhelming attention to preparing for and participating in the implementation of the mandatory standards, little attention was given to the ongoing process of readiness evaluations. In addition, Readiness staff availability did not allow participation at the various NERC committee meetings, and agendas were usually dominated by ERO issues, including roles of the various committees under the ERO and mandatory standards implementation. Little time was given to program reports to the committees; conversely, there was much speculation about the program from the committees and the industry. By the close of 2007, the program added three additional full-time NERC readiness evaluators for a total of five; the program is currently supplemented by several part-time contract evaluators. These additional full-time staff members helped develop the three-year business plan, revamped materials used during the evaluation process, and began to communicate the program status and direction to the industry, including the NERC OC and ORS.



By March of 2008, the program had reconnected with the NERC ORS and the NERC Operating Committee. In addition, there has been an increase in communication with the regional entities. The program was in a position to solicit and implement desired industry changes. It was clear that the industry need was changing, and the program needed to modify its products to continue to benefit the industry. Using input from the committees, the program refined its three-year plan. This feedback and improvement process has been captured and is now inherent in the program. Consistency Feedback from the first cycle of evaluations highlighted concerns of consistency, and program staff has refocused on increasing the level of consistency of all evaluations. In 2007, program staff worked with the NERC training program to develop and implement a course to train all members of the evaluation teams. To date, feedback on the course has been very encouraging. Other initiatives include increased staff meetings, revamped evaluator guides (interview/note-taking documents used on site), and an improved final report template. The program does acknowledge, however, that the use of volunteer team members makes complete consistency a challenge. This issue is discussed in depth in the Topics for Discussion section.



PPrrooggrraamm FFuuttuurree NERC has begun writing three-year business plans as part of an overall strategic planning process. This will be a yearly process, with the first year of the business plan being very clear and the following years being somewhat less detailed. Accordingly, readiness staff has drafted a 2009 to 2011 business plan. The 2009 to 2011 readiness business plan is designed to be general, in order to position the program for the flexibility needed to provide a product that is useful and desired by the NERC community. The draft plan was discussed with the NERC ORS at its February 2008 meeting. The ORS confirmed that the draft plan was in line with its ideals. The ORS provided additional feedback to the plan and the group’s concerns with the program, and this information was used to refine the draft plan. The plan was also shared with Regional Entity representatives, who were pleased that the program was openly soliciting confirmation on the program direction and input from the industry. Using the input from the various industry groups, the three-year business plan has been submitted to NERC senior management for review. This is not the end of the process, and the program is looking forward to further refinements. The 2009 to 2011 business plan addresses appropriate staffing level, communication of the program purposes to the industry, reinforcement of the program’s identity, evaluator training, expansion of the examples of excellence process, development of a similar “excellent tools” process, proactive self assessments of program effectiveness, and development of a process to provide assistance to NERC members. Many aspects of the plan are discussed in the Topics for Discussion section of this report. Self-assessment A self-assessment of the readiness program was listed as a goal for 2008. Time will be set aside in the late December through January, when there are typically no evaluations, to perform a detailed self-assessment, review the previous year’s accomplishments, and develop future goals. Staff will be assigned specific tasks to review and improve program processes. Some improvements have already taken place. Enhancements to the tracking of evaluation recommendations, evaluation processes, and reporting to the Board of Trustees have helped improve overall efficiency. In early 2008, the program’s evaluator guides were edited for context and content. As this assessment process continues, questionnaires, report templates, and organizational processes will be reviewed, streamlined, and improved to promote efficiency. A 2009 target will be to revise the entity questionnaires. Streamlining the final report process through the use of technology is included in 2010 and 2011 goals.



TTooppiiccss ffoorr DDiissccuussssiioonn In late 2007, the readiness program requested feedback from previously evaluated entities. Unfortunately, the program received a limited number of responses, which was not necessarily a good representation of the industry. The limited response indicated apathy towards the program due to the higher priority of compliance issues being dealt with in the industry. Common themes were identified, however, and this input helped shape the first draft of the three-year business plan for the readiness program. After incorporating readiness staff input and refining the draft business plan, a presentation was made for the NERC ORS that included program direction and feedback from the industry. The presentation prompted an enlightening discussion indicating the program’s business plan was headed in the correct direction. Additionally, the program was able to benefit from an open and honest exchange on the issues, including program feedback and a number of others the ORS identified. This list of issues was discussed in greater depth in an ad hoc meeting with representatives from the readiness program and the NERC Operating Committee. After the ORS meeting, but prior to the ad hoc meeting, the readiness program staff was able to meet with Regional Entity representatives. As part of this meeting, the program business plan was presented, with changes to that point. The Regional representatives were also asked for input on the ORS list of concerns. After lengthy discussion, the feedback from the group was that the program was moving in the desired direction, was showing signs of serious input for improvement, and that the desired outcome was continued improvement in communications with the various groups. Entity feedback and input received from the ORS and Regional Entity representatives centered around five major program topics: identity, confidentiality, focus/scope, consistency, and industry assistance. The following pages detail the feedback and input (discussion) and readiness program response regarding these topics. Identity Discussion: Industry and ORS feedback emphasized that the readiness and compliance programs are entirely different and should be kept separate. The following are a few statements received:

1. Readiness evaluations should precede a compliance audit.

2. Compliance personnel should not participate on readiness evaluation teams.

3. If a readiness evaluation team comes across a potential compliance violation, it should not be reported to the NERC compliance officer or the region. The organization should be responsible to follow up and resolve the issue prior to its next compliance audit.



4. Recommendations are not compliance violations; they are simply recommendations with appropriate action completely up to the entity under review. Therefore, tracking recommendation progress is not necessary.

Response: Scheduling and Staffing There has been some confusion over the purpose of the readiness program, and this has been compounded by readiness evaluations being combined or closely coupled with compliance audits conducted by the Regional Entities. The readiness program has been working with Regional Entities to schedule an entity’s readiness evaluation 6 to 18 months from its compliance audit. This will not be a schedule that can be accomplished quickly, since entities are evaluated for both readiness and compliance on a three-year cycle. Scheduling all readiness evaluations to precede compliance audits would possibly send the message that readiness evaluations are pre-compliance audits, which is not the case; therefore, creating a time separation should be the main focus as opposed to order. It is true that some Regional Entities have compliance staff attend both readiness evaluations and compliance audits. Separating the functions by staff is a good goal, and NERC supports this concept. Current staffing at some Regional Entities, however, curbs their ability to provide independent readiness staff and evaluation participants. The two programs must grow apart gradually in this regard. Potential Compliance Violations The NERC Rules of Procedure require an evaluation team to report evidence “of possible noncompliance with a reliability standard” to NERC “for resolution through the applicable compliance enforcement program.” Teams do not seek potential compliance violations. NERC, as the ERO, has a fiduciary responsibility to report possible violations. Recommendation Tracking Tracking the acceptance and implementation of evaluation team recommendations is a good first measure of the efficacy of the overall program. When the evaluation event is properly viewed as an independent analysis by a team of industry experts (similar to consultants but less costly), an obvious product of such an analysis is improvement possibilities. NERC encourages entities to carefully consider the implementation of all team recommendations, understanding some situations might ultimately preclude such implementation (cost to benefit analysis, local or state laws in place, etc.). In those instances, expecting an entity to document their determinations does not seem onerous. A secondary issue that NERC has addressed is whether a new evaluation team should spend time analyzing an entity’s response to prior recommendations. At the urging of members of the Regional Entities and the ORS, NERC has decided to cease discussion of previous recommendations with current teams unless it is brought up by a current member. Prior evaluation (or audit) reports are publicly posted, so it is conceivable that a team member might read through the prior report as part of his or her participation in an evaluation. It is then up to the individual team to determine whether to spend time in this activity, but the NERC leads are not instigating the discussion.



Other Efforts to Reinforce Identity NERC has recognized internally the need to reinforce the identity of the readiness program. Much of the industry has the perspective that NERC is a regulator, interested only in enforcing compliance through sanctions. In many ways, NERC’s is a paradox between FERC and the industry. FERC has tasked NERC with enforcing compliance as well as developing a readiness program to provide guidance to entities for achieving operational excellence1; however, the industry provides guidance to NERC to carry out this mission. The readiness program will continue to seek ways to distance itself from the compliance process. In addition to working with the regions on scheduling, the readiness program stopped using the term “audit” and adopted “evaluation.” Based on the 2008 programs goals, readiness staff will regularly attend NERC Operating Committee, the Transmission Owners and Operators Forum, and NERC ORS meetings to enhance communication. Evaluators will also seek opportunities at the regional, subregional, and association level to discuss and answer questions about the readiness program and its vision. Feedback received through this effort will further the efforts to bring increased value to the industry. Confidentiality Discussion: Promoting operational excellence through the readiness program requires honest and meaningful discussion between the review team and the individuals representing the entity being reviewed. It has been suggested that NERC consider readiness evaluation reports confidential, as confidential reports may prompt more candid discussions. Update: Structuring the program such that the evaluations and the subsequent reports are confidential would be difficult to accomplish given that NERC operates under the principles of being open, fair, transparent, and inclusive. The NERC team leaders have not experienced any significant “holding back” in the course of conducting the evaluations. Evaluation reports are one of the primary products of the readiness program, and publishing them fulfills an important role for the industry. Each evaluation team determines positive observations and potential examples of excellence, items that could be valuable and of interest to other companies, particularly those seeking to implement improvements to a specific area. Critical infrastructure information is not publicly posted but is communicated to the entity in a private, confidential version of the report. Focus/Scope Discussion: Some feedback related to the focus and scope of the evaluations, such as the following response: 1 See FERC Order 672



o Many entities that are being evaluated are very experienced in the functions they are

performing. For those entities, a general readiness evaluation may be of value for the industry by confirming processes and procedures are being executed properly by their neighbors. However, an experienced entity may benefit by having a more focused evaluation. Therefore, experienced entities should be able to request an in-depth evaluation of certain areas along with the standard evaluation.

Feedback specific to transmission owners included the following statements: o Evaluations for transmission owners should be scheduled on the same six-year cycle as for

compliance audits.

o NERC should create a procedure for determining whether a registered transmission owner is qualified to have a readiness evaluation. It may require the Region Entities to make that determination and provide written justification to NERC supporting that decision.

Update: NERC is in its second three-year cycle of evaluations; therefore, entities have been evaluated one to two times. By the end of 2009, entities will have had two evaluations. The exception is transmission owners, for which evaluations began in 2005. At the end of the second cycle, NERC will have an effective baseline of experiences and will be in a position to analyze what areas merit more attention. As listed in the 2009 to 2011 business plan, the readiness program will take specific requests from entities to include areas where specific focus is requested, areas that may not have been identified through past evaluations. The idea is to conduct more focused evaluations, adopting a “narrow but deep” concept as suggested by the ORS. The evaluation schedule for transmission owners has been discussed and is currently under review. NERC agrees that a definitive procedure should be developed to determine which registered transmission owners should be the subject of a readiness evaluation. (These are issues the ad-hoc group can evaluate and make recommendations). Consistency Discussion: A significant portion of the feedback focused on consistency. Common themes included overall consistency, recommendations, examples of excellence, and program metrics. Training and established criteria were specifically mentioned as keys to moving towards consistency. Below is a paraphrase of the feedback. Evaluations To maximize success of the readiness program it is of paramount importance to attain and maintain a high degree of consistency among evaluation teams and among evaluations themselves. Achieving the level of consistency demanded by the industry requires significant



training for the teams and team leaders. Providing training in team dynamics to the team leaders should provide the tools and authority for the leader to keep the team focused on the reliability issues and industry norms. Recommendations The recommendation process is vital for maintaining program credibility in the industry. Professionals conducting a readiness review must remain focused on reliability readiness from a qualitative view point drawing on their experience, and understanding of industry norms, to assess entity practices and performance. Recommendations should be meaningful and clearly written, and team leadership and NERC management should provide guidance and strive for consistency. When contemplating a recommendation, the review team should utilize criteria to guide their decision. Factors should include reliability risk, whether implementation would measurably reduce the reliability risk at this entity, evidence of the recommendation being effective in other areas or at other entities, and team consensus. Regarding consensus, if teams recognize an opportunity for improvement that does not rise to the level of a recommendation, it can be included in the body of the report as a suggestion or consideration. Examples of Excellence Examples of excellence have the potential to be the catalyst for meaningful technical discussions that could enhance the operational excellence in the electric industry. To realize its full potential, the process may need to expand beyond the readiness program: a potential example of excellence (EoE) should be reviewed to gain industry consensus and support to ensure it reaches the level of an EoE. One possibility is to put the EoE process under the purview of the Operating Committee and the ORS. Readiness teams would continue to identify potential EoEs and encourage the entity to pursue full EoE status. The entity would develop a white paper and submit to the ORS for review, then present the paper to the ORS and lead a discussion. If ORS supports the paper as an EoE, it would recommend Operating Committee approval. Separately, other entities such as NERC, Regional Entities, Transmission Owners and Operators Forum, and the entity itself could also identify potential EoEs and pursue full EoE status without a readiness review. Criteria should be used for consistency purposes. Factors listed included enhancement to reliability, uniqueness of process, practicality of being implemented at similar entities, and measurability of enhancement to reliability if widely implemented. Update: As mentioned previously, feedback from the first cycle of evaluations also highlighted concerns of consistency. Program staff has refocused on increasing the level of consistency of all evaluations. Some initiatives geared towards consistency include increased staff meetings, revamped evaluator guides (interview/note-taking documents used on site), and an improved final report template. NERC agrees, however, that training is critical to achieving consistency.



In the 2007 NERC business plan, the development of a training program was identified as a goal. Program staff worked with the NERC training program to develop and implement a course to train all members of the evaluation teams. All evaluation team members (regional staff participants, FERC staff participants, and industry volunteers) are expected to complete this training prior to their participation on any 2008 evaluation teams. If an evaluation team member does not complete the training in the allotted time, he or she will be replaced with another volunteer who has completed the training. To date, feedback on the course has been very encouraging. The 2008 goals include development of a focused training program for NERC evaluators and senior regional representatives. The program is to focus on team leadership, organizational skills, and reliability knowledge improvement. This program is planned for implementation in 2009. Going forward, yearly training goals will be set for each evaluator. By the end of 2009, all NERC evaluators and senior regional representatives will have completed the first session of advanced training. This will add to the efficiency, consistency, and effectiveness of the reliability evaluations. Criteria The concept of criteria was mentioned as a concern for both examples of excellence and recommendations. Readiness staff has discussed criteria for these aspects as well as positive observations and agrees that criteria need to be developed. With the inclusion of select members of the ORS, additional industry input is being sought. The evaluation team leader and the director of reliability readiness seek to coach the teams with regard to scope of recommendations as well as guidance in the identification of potential examples of excellence. Readiness evaluations consist of teams with varying backgrounds and expertise. Ultimately, the evaluation report, including areas for improvement, is the product of the team itself – and most if not all teams are very vested in the process and the products. Recommendation Guidelines As a way to move towards consistency, program staff has developed a number of guidelines. Though these instructions are not necessarily criteria, they give evaluators a foundation for communications. Below is a sampling of our guidelines:

1. Begin each recommendation with a verb indicating the action the team wants the entity to take (ex. develop training resources…).

2. Create concise, specific, and actionable recommendations.

3. Limit the use of recommendations to “complete” or “evaluate”:

a. Complete implies the entity is already taking care of the issue, which does not necessarily qualify as a recommendation. If the team feels the project or action is in jeopardy of stalling or is something that needs to occur as soon as possible, use “expedite, to the extent possible,” or similar wording.



b. In general, evaluate is not definitive. Use “add” or another word indicating the action recommended. If the action can be viewed as a “maybe,” list it in the text as a suggestion instead of a recommendation.

4. Include a rationale for recommendations, when possible.

Examples:

a. Provide information to the system operators to identify the limiting element associated with each transmission facility rating to aid in the understanding of various limits and to assist in evaluating appropriate actions and response times.

b. Implement a formal document management process that includes information on ownership, author, approvals, revision dates, distribution, and acknowledgement of receipt and understanding to make sure operators can access and use the most up-to-date policies and procedures.

Examples of Excellence Traditionally, NERC evaluators voted on potential examples of excellence. Program staff discussed with the ORS options of acquiring broader input on the selection of these examples. In 2007, the process was changed to include ORS input. As part of a pilot program, a sub-team of the ORS was included in the evaluation process. Though this practice broadened the initial decision making group, the process was slowed significantly, partly due to the volume of e-mail communications. After a preliminary experiment with the revised process, it was decided that potential examples of excellence should be screened through the NERC evaluation leaders first. Then, only potential examples with support of the readiness evaluators will be forwarded to the ORS sub-team. Industry Assistance Discussion: Feedback and input regarding industry assistance included the following: Sharing the good work some entities are doing would foster improvements in the industry. Similarly, it is beneficial for entities to be exposed to recommendations that are being included in evaluation reports. Promoting operational excellence might be better served if NERC periodically incorporated pertinent information into a master report, providing one consolidated place for the industry to review results of readiness evaluations rather than reviewing each individual report. NERC should conduct periodic technical conferences presenting EoEs to the industry. Care must be taken to identify relevant parameters where the EoE may be effective. For example an EoE might work for a transmission operator but not a reliability coordinator, or one might work for a market system but not a bilateral system, etc. Create an archival policy related to outdated readiness reports and examples of excellence.



Update: One of the program’s founding principles was to provide assistance to NERC entities. To date, this assistance has mostly been comprised of providing entities with an impartial, outside view of their operation. Constructive criticism is given on ways to improve, and positive practices are highlighted and shared with the industry. After several years of evaluations, and the onset of mandatory compliance, the program needs to provide further assistance. Many companies use the program as a benchmarking tool to help evaluate their policies and practices against a changing culture. It has become evident that the program needs to recognize how the industry is using the results and modify how it manages the results of its efforts. It then needs to provide the requested additional services that many entities desire. In October 2008, the program will begin to work with NERC ORS and Operating Committee input to begin a process to provide assistance to entities who request help with their processes. Staff will attempt to broker requests for assistance, possibly providing direct representation, to help implement practices at requesting entities. Posting the results of the readiness program on the NERC Web site has not met the current needs of the industry. Entities have commented that it is difficult to review the reports for targeted needs. Beginning in 2008, the readiness program will inventory all examples of excellence and positive observations from the readiness reports. This inventory (slated for completion by the end of 2008) will then lead to the development of a Web-based database (2009) that allows a user to query for positive practices identified through the program. The company can then identify the contact information to pursue further investigation and possible implementation of these practices. Examples of Excellence Expansion and Communication The examples of excellence process continues to be one of the more popular elements of the program. By December 2007, over 90 examples of excellence had been identified and posted to NERC’s Web site. To increase exposure, NERC has moved from using a quarterly examples of excellence bulletin to announcing examples in the widely read monthly NERC News. With participation and guidance from the ORS, the process to identify these practices will continue to be refined and communicated to the industry. Refinements include streamlining the process, developing a matrix to better identify solutions that fit specific niches, and improving the indexing of examples to allow interested parties better access. In addition to uncovering excellent practices, the readiness evaluators are in a unique position to identify the use of new tools and emerging technology. Many times, the application of technology itself is not cited as an example of excellence; however, it is listed as a positive observation. Though NERC cannot be an advocate of any particular vendor or supplier, the readiness program can identify entities that have successfully implemented tools that are unique and have solved particular problems with which the industry struggles. The example of excellence program can highlight these cases and provide interested entities with contact information to learn more about these tools. The goal for the readiness program will be to codify



and refine the examples of excellence process to include tools and further educate the industry about the examples. Archiving The program is reviewing archiving concepts. A three-year posting for reports and examples of excellence has been proposed and is still under review. The decision has been made to remove an evaluation report from the Web site when a more current one is posted.



CCoonncclluussiioonn WILL WRITE AFTER FEEDBACK IS RECEIVED

Agenda Operating Committee - NERC Filings to FERC DL/Response_… · Operating Committee Work Plan...

Documents

Transcript of Agenda Operating Committee - NERC Filings to FERC DL/Response_… · Operating Committee Work Plan...