Compliance_Evaluation_Report_121509

8/7/2019 Compliance_Evaluation_Report_121509

1/130

Submitted to:North American Electric Reliability Corporation116-390 Village Boulevard

Princeton, New Jersey 08540

Report prepared by:Crowe Horwath LLP70 West Madison Street, Suite 700Chicago, Illinois 60602-4903

November 23, 2009

Compliance Enforcement, Registration, andCertification Program

Process Evaluation Report


2/130

Compliance Enforcement, Registration and Certification


AFFILIATES Crowe Horwath LLP is a member of Crowe Horwath International, a Swiss association. Each member firm of Crowe Horwath

International is a separate and independent legal entity. Crowe Horwath LLP and its affiliates are not responsible or liable for any acts or

omissions of Crowe Horwath International or any other member of Crowe Horwath International and specifically disclaim any and all

responsibility or liability for acts or omissions of Crowe Horwath International or any other member of Crowe Horwath International. Crowe

Horwath International does not render any professional services and does not have an ownership or partnership interest in Crowe Horwath

LLP. Crowe Horwath International and its other member firms are not responsible or liable for any acts or omissions of Crowe Horwath LLP and

specifically disclaim any and all responsibility or liability for acts or omissions of Crowe Horwath LLP. 2009 Crowe Horwath LLP

Table of Contents

Executive Summary ....................................................................................................................................... 3

Section 1: Overview ...................................................................................................................................... 8Project Background ...................................................................................................................................... 8Process Evaluation Methodology ............................................................................................................... 14Purpose of Report ...................................................................................................................................... 16Document Overview ................................................................................................................................... 17Disclaimer of Confidentiality ....................................................................................................................... 17

Section 2: Observations and Recommendations Summary ................................................................... 18Introduction ................................................................................................................................................. 18The Process-driven Organization............................................................................................................... 18Process Governance and the Process Foundation Summary Observations ............................................ 20Overarching Observations and Recommendations ................................................................................... 21Categorization of Recommendations ......................................................................................................... 37

Section 3: Cross-Functional Areas Evaluation ........................................................................................ 42Introduction ................................................................................................................................................. 423.1. Compliance Program Confidentiality Requirements .......................................................................... 423.2. Developing and Overseeing the Compliance Training Program........................................................ 433.3. Developing and Disseminating Compliance Process Directives and Bulletins .................................. 443.4. Processing Reliability Standards Violations ....................................................................................... 45

Section 4: Functional Area Evaluation ...................................................................................................... 47Introduction ................................................................................................................................................. 474.1. Compliance Program Planning .......................................................................................................... 484.2. Overseeing Registration of Owners/Users/Operators of the Bulk Power System ............................. 544.3. Overseeing Certification of Owners/Users/Operators of the Bulk Power System ............................. 604.4. Overseeing Compliance Activities of Regional Entities (excluding CVIs) .......................................... 654.5. Overseeing Enforcement Activities of Regional Entities .................................................................... 76

4.6. Analyzing and Reporting Compliance Information ............................................................................. 834.7. Conducting Reviews of Regional Entities Compliance and Enforcement Programs ........................ 884.8. NERC Involvement in Compliance Inquiries and Violation Investigations ......................................... 944.9. Handling Complaints ........................................................................................................................ 1014.10. Executing Compliance Enforcement Authority Responsibilities .................................................... 105

Appendix I Functional Area to Processes and Procedures Crosswalk ............................................. 114

Appendix II Process Questionnaire ....................................................................................................... 117

Appendix III Observations and Recommendations from Development of Agreed-UponProcedures .................................................................................................................................................. 118

Appendix IV Excerpt from Management Letter to NERC ..................................................................... 127


3/130

Compliance Enforcement, Registration and Certification


AFFILIATES Crowe Horwath LLP is a member of Crowe Horwath International, a Swiss association. Each member firm of Crowe Horwath

International is a separate and independent legal entity. Crowe Horwath LLP and its affiliates are not responsible or liable for any acts or

omissions of Crowe Horwath International or any other member of Crowe Horwath International and specifically disclaim any and all

responsibility or liability for acts or omissions of Crowe Horwath International or any other member of Crowe Horwath International. Crowe

Horwath International does not render any professional services and does not have an ownership or partnership interest in Crowe Horwath

LLP. Crowe Horwath International and its other member firms are not responsible or liable for any acts or omissions of Crowe Horwath LLP and

specifically disclaim any and all responsibility or liability for acts or omissions of Crowe Horwath LLP. 2009 Crowe Horwath LLP

Table of Figures

TABLE 1PROJECT APPROACH PHASE 1................................................................................................................ 9

TABLE 2PROJECT APPROACH PHASE 2.............................................................................................................. 10

TABLE 3CERCPPROCESS EVALUATION FINAL SCOPE.......................................................................................... 12

TABLE 4CMEPPROCESSES AND PROCEDURES................................................................................................... 13

FIGURE 1LEVEL OF EVALUATION...................................................................................................................... 14

TABLE 5POLICY,PROCESS, AND PROCEDURE DEFINED ........................................................................................ 15

TABLE 6THE INFRASTRUCTURE FOR PROCESS SUCCESS ........................................................................................ 19

TABLE 7RECOMMENDATION CATEGORIES......................................................................................................... 37

TABLE 8RECOMMENDATIONS SUMMARY BY CATEGORY OF RECOMMENDATION ..................................................... 40

TABLE 9RECOMMENDATIONS COUNT BY SECTION, BY CATEGORY......................................................................... 41


4/130

Compliance Enforcement, Registration and Certification 3


Executive Summary

Project Objectives

North American Electric Reliability Corporation (NERC) determined the need for a project to

provide NERC with an evaluation of its Compliance Enforcement, Registration and Certification

Program (CERCP) processes and procedures. NERC engaged Crowe Horwath LLP to perform

this evaluation and Crowe completed this project between July and October, 2009.

The project was initiated to assist NERCs Compliance area (NERC Compliance or the NERC

Compliance Department) in achieving its overall objectives for effective implementation of the

CERCP, including adequate management controls . The project objective, therefore, was to

identify and document whether the program has adequately implemented applicable CERCP

processes and procedures in accordance with the applicable law, FERC orders, and NERC Rules

of Procedure. Additionally, Crowe reviewed the internal processes and procedures used by the

Compliance Department in carrying out its duties for consistency with the Rules of Procedure

and for completeness and effectiveness.

Project Approach

For purposes of planning, tracking, and execution, the project was divided into two separate,

sequential phases where the outputs from Phase I became key inputs to Phase II activities.

Phase I of the project primarily involved (i) conducting necessary project initiation and planning

activities, and (ii) gathering information from NERC Compliance personnel concerning the

processes that NERCs Compliance Department has in place over the compliance with and

enforcement of approved electric reliability standards. Phase II of the project involved (i)

performing analysis and review of process and procedure information and artifacts gathered in

Phase I, (ii) preparation of the public report and the confidential letter to management, (iii)review and revisions to the reports based upon feedback, and (iv) final delivery of the reports

and project closeout.

Project Scope

Four cross-functional areas and ten functional areas comprise the final scope of the CERCP

process evaluation and, therefore, the scope of this report. Cross-functional areas are areas

that underlie all CERCP processes for example, confidentiality requirements. Functional areas

represent groupings of related processes, frequently for purposes of mapping related processes

back to a unit or basic responsibility of the program for example, registration, certification,

CVIs, and enforcement are all functional areas. The 37 processes defined by the NERC

Compliance Departments CMEP Processes and Procedures Manual are all encompassed within

these 14 cross-functional and functional areas. The CMEP Processes and Procedures Manual is

an internal set of procedures developed and maintained by NERCs Compliance department to

assist in the implementation of the compliance enforcement, registration and certification

program.

Cross-Functional Areas

1. Compliance Program Confidentiality Requirements

2. Developing and Overseeing the Compliance Training Program

3. Developing and Disseminating Compliance Process Directives and Bulletins


5/130



4. Processing Reliability Standards Violations

Functional Areas

1. Compliance Program Planning

2. Overseeing Registration of Owners/Users/Operators of the Bulk Power System

3. Overseeing Certification of Owners/Users/Operators of the Bulk Power System

4. Overseeing Compliance Activities of Regional Entities (excluding CVIs)

5. Overseeing Enforcement Activities of Regional Entities

6. Analyzing and Reporting Compliance Information

7. Conducting Reviews of Regional Entities Compliance and Enforcement Programs

8. NERC Involvement in Compliance Inquiries and Compliance Violation Investigations

9. Handling Complaints

10.Executing Compliance Enforcement Authority Responsibilities

Purpose of Report

The purpose of this report is to provide NERC with an evaluation of its CERCP processes and

procedures across the 14 cross-functional and functional areas identified above. This report,

submitted by Crowe Horwath LLP, represents the culmination of activities performed on the

project.

The primary objective of the report is to provide observations as to whether the program has

adequately developed and implemented applicable CERCP processes and procedures, where

adequacy is defined by those criteria identified in the Process Evaluation Methodologysection

of this document, and to make recommendations where the implementation of the CERCPprocesses and procedures can be improved.

Process Governance and the Process Foundation

In summary, our observations regarding the governance and foundational layers of the NERC

Compliance process environment are as follows:

As a regulatory entity, NERC is by its very nature compelled to maintain an environmentfocused on the creation, compliance, and enforcement of its standards and rules. We

observed that the NERC CERCP program generally has the governance and tone at the top

to be successful with its processes. Our assessment of individual functional areas indicates

that process objectives are typically well known and well understood and that there is

clearly a culture of policy and process adherence.

As part of our analysis we placed NERCs CERCP into appropriate context from the standpointthat NERCs Compliance organization and the purpose, roles, and scope of responsibilities

for that organization has existed in their current state only for a relatively very short period

of time. The relative immaturity of the organization certainly has a bearing on the

expectations for its level of process maturity. For example:

o We observed in our analysis that the organizational structure, and the resulting roles

and responsibilities within that structure, continue to mature and change fairly

frequently as the Compliance area has undergone numerous structural changes within

the past two to four years. Three years ago the Compliance organization shifted from a


6/130


7/130



term solutions built on enterprise-level platforms with the foundation of IT controls

required of such systems.

Overarching Observations and Recommendations

As part of this project, Crowe identified observations in different functional areas and cross-functional areas within the Compliance Department. In doing so, seven themes surfaced that

impact the Compliance Department as a whole, as opposed to a specific team, process, or

functional area. These seven themes are important to the NERC Compliance Departments

maturity as a process-driven organization. We provide an overview of these themes below.

Each is addressed in further detail this report:

1. We recommend to NERC that a number of changes to the ROP (including its related

appendices). These changes should be implemented to ensure a solid foundation for NERCs

compliance program. We observed a number of issues with the ROP whereby it could be

strengthened by adding to it (address areas of Regional Entity accountability e.g.

Compliance Inquiry process), changing it (address areas where Regional Entities differ in

practice from the ROP as documentede.g. terminology such as guidelines and notices ofviolation), or deleting from it (removing redundancies).

2. We recommend to NERC that CMEP Process and Procedures documents should be

completed, reviewed, and approved, including incorporating more defined roles,

responsibilities, timelines, and outcomes where these were found to be lacking. We

observed that process documents lacked consistency in form and included some conflicting

information, and at times did not contain obvious tie-backs to the ROPs by virtue of the

process used to develop them. The individual documents requiring completion, review, and

approval are captured within the detailed recommendations of this report.

3. We observed that the Compliance Department was not consistently meeting a number of its

internal process goals for timeliness. NERC Compliance indicated to us that, with theircurrent staff resources, they often had to adjust timelines in order to ensure the quality of

their work. It is our observation, therefore, that staffing levels may not be appropriately

aligned for the workload required. However, it is also our observation that there are other

contributing factors (process inefficiency issues, deficiencies in the process infrastructure,

effort-based metrics) which may also contribute heavily towards NERCs ability to meet its

goals in certain compliance enforcement, registration and certification process areas. The

lack of activity level, effort-based metrics impeded the ability to fully assess whether staffing

levels are adequate relative to workload and/or to assess the degree to which staff levels

are required to meet certain levels of desired timeliness and quality.

4. We observed that problems with the consistency of outputs from Regional Entities (in terms

of the level of quality of outputs and the timeliness of those outputs) and differences inprofessional opinion between NERC, the Regional Entities, and FERC impacted the timelines

for the Compliance Departments work and the quantity of work that can be accomplished

(i.e. as measured by the number of enforcement actions processed within established time

frames). For example, one manager noted that Regional Entities often submitted Notices of

Confirmed Violations that contained errors in dates and judgments that NERC did not find

appropriate, such as classifying an issue as a documentation error rather than a failure to

perform, when the standard required documentation of performance. Another manager

stated that NERC and FERC periodically had different opinions on applications of reliability

standards on Compliance Violations Investigations.


8/130



5. We observed that processes within some functional areas were not adequately monitored

because there were few interim checkpoints being taken during the overall duration of the

process. For example, the functional areas Analyzing and Reporting Compliance Violation

Information and NERC Compliance Enforcement Authority Responsibilities had no

monitoring in place or planned. We also observed that for those functional areas that weremonitored, there was often not adequate follow up when process deviations were found. In

the functional area Overseeing Compliance Activities of Regional Entities, for example, we

observed that staff was given reminders of the need to meet timeliness goals, but no other

actions were taken when these goals were not met.

6. We observed several processes that involve handling large amounts of information and

documentation. NERC had begun to address these issues through the development of new

technologies, but it was our observation that until these are fully implemented, the volume

of data and documentation will continue to be an impediment to accomplishing the

Compliance Departments goals in a timely manner.

7. We identified some issues with the level of controls over data security, confidentiality, andphysical security. Confidential information has been removed from this public version and

has been provided under separate cover to NERC management.

Document Overview

This report takes a top-down approach towards presenting the detailed observations and

recommendations. The Overviewsection provides a more detailed look at the objectives, scope,

and approach of this Process Evaluation.

The subsequent section (Section 2) titled Observations and Recommendations Summary

provides a summary level view across all observations and recommendations. As part of this

project and the methodology used, Crowe Horwath LLP developed a scorecard for evaluating

the various functional and cross-functional areas. The summary contains the summarized levelview of that scorecard. The summary also contains a number of overarching recommendations.

These recommendations are summary-level findings that in many cases present macro-level

observations made across functional areas or within functional areas across multiple criteria.

The next section of the document, Section 3, Cross-Functional Areas Evaluation, contains the

observations and recommendations as they relate to the four cross-functional areas.

Finally, Section 4, Functional Area Evaluation, contains the observations and recommendations

as they relate to the ten functional areas evaluated. Especially relevant to the functional area

evaluations are appendices I and II. Appendix I contains a crosswalk of the functional areas back

to the actual CERCP processes and procedures as defined by the NERC CMEP Processes and

Procedures manual. As most analysis will be documented as the functional area level, it is

important to note which processes and procedures comprise each functional area.

Appendix II contains the criteria used to evaluation each functional area. Appendix III contains

detailed observations and recommendations regarding changes to the ROP. These observations

and recommendations were developed by Crowe as part of its development of the Agreed-Upon

Procedures. Appendix IV contains an excerpt from the Management Letter to NERC from the

results of recently completed Agreed-Upon Procedures project for a regional entity. The excerpt

contains key recommendations regarding the ROP and the CMEP Processes and Procedures.


9/130



Section 1: Overview

Project Background

Project Objectives

North American Electric Reliability Corporation (NERC) determined the need for a project to

provide NERC with an evaluation of its Compliance Enforcement, Registration and Certification

Program (CERCP) processes and procedures. NERC engaged Crowe Horwath LLP to perform

this evaluation and Crowe completed this project between July and October, 2009.

The project was initiated to assist NERCs Compliance area in achieving its overall objectives for

effective implementation of the CERCP, including adequate management controls. The project

objective, therefore, was to identify and document whether the program meets the

requirements of the implementing rules established by FERC for the Energy Policy Act (i.e. the

NERC Rules of Procedure and subsequent FERC orders), and if the NERC implementation has

adequately implemented applicable CERCP processes and procedures.

More specifically, the intent of this engagement was to:

1. Assess the core internal processes of the NERC CERCP implementation through interviews of

NERC Compliance employees and inspection of documentary evidence, using criteria found in

the following program documents from the ROP, and applicable sections from 18 CFR Part 29 as

the primary basis for the evaluation:

a. Section 400 Compliance Enforcement

b. Appendix 4B Sanction Guidelines of the North American Electric Reliability

Corporation

c. Appendix 4C Compliance Monitoring and Enforcement Program

d. Section 500 Organization Registration and Certification

e. Appendix 5 Organization Registration and Certification Manual

f. Section 1500 Confidentiality of Information

2. Provide an independent Process Evaluation Report (i.e. this report) for public use to align

with NERCs need to be transparent, stating process efficiency, resource, or other improvement

recommendations identified (if applicable) during the process evaluation.

3. Provide a Confidential Letter to Management (i.e. a separate letter from this report) for any

process efficiency, resource, or other improvement recommendations that for the purposes ofcommunicating such information must include the identification of confidential information,

including but not limited to company names, data, NERC confidential information or personnel

identification. NERC assisted Crowe Horwath LLP with identification of such information.

Project Approach

For purposes of planning, tracking, and effective execution, the project was divided into two

separate, sequential phases where the outputs from Phase I became key inputs to Phase II

activities. The purpose, scope, activities, and outcomes of the two phases are described below.


10/130



Phase I - Planning and Data Gathering

Purpose Phase I of the project involved (i) conducting necessary project initiation and planning

activities, and (ii) gathering information from Compliance personnel concerning the

processes that NERCs Compliance Department has in place over the compliance withand enforcement of approved electric reliability standards. The activities included a

review of the criteria contained in the applicable sections of the Rules of Procedure,

developing questionnaires for data gathering, scheduling and conducting interviews

with NERC Compliance staff, and reviewing information received from NERC

Compliance staff and other documentary evidence regarding the execution of the

CERCP processes.

Activities 1. Conduct project initiation activities including, but not limited to, project kickoff

meetings to coordinate all project stakeholders and to ensure that there is a

common understanding for the project objectives, scope, approach, schedule, and

responsibilities.

2. Plan and establish the operating model for the project. Planning included the

creation and coordination of the project schedule of activities, resource schedules

and availability, project communications and status reporting.

3. Create a crosswalk of NERC compliance processes and procedures back to

functional areas that effectively group and map the processes and procedures

back to areas of organizational responsibility (see Appendix I).

4. Conduct initial interviews with functional area owners (primarily CERCP Managers

and Directors) to confirm understanding of the scope of the functional area, key

interactions with other functional areas, and the processes and resources

implemented within the area. Identify key documents and information supporting

the implementation of the CERCP processes and procedures.

5. Request, collect, and review documents and information supporting the

implementation of the CERCP processes and procedures (received from functional

areas owners and key subject matter experts).

6. Conduct formal interviews with functional area owners and functional area staff

(primarily analysts, investigators, administrators, and auditors) using common

functional area evaluation criteria to determine the status of the CERCP

implementation with respect to the criteria (note: the interview criteria are

included as Appendix II to this report).

7. Conduct final functional area interviews to confirm understanding and answer

final questions regarding processes, procedures, documents, and process artifacts.

Interviews included, in some cases, observation of various supporting IT systems.

Outputs Project Operating Model and Project Schedule

CMEP Process and Procedure-to-Functional Area Crosswalk (included as Appendix Ito this report)

Process review criteria and interview template (included as Appendix II to thisreport)

Documents and Artifacts Log

Table 1 Project Approach Phase 1

Phase II - Data Analysis and Reporting

Purpose Phase II of the project involved (i) performing analysis and review of information

gathered in Phase I, (ii) preparation of the public report and the confidential letter to


11/130



management, (iii) review and revisions to the reports based upon feedback, and (iv)

final delivery of the reports and project closeout.

Activities 1. Prepare preliminary process write-ups for functional areas and conduct follow-up

interviews and communications to confirm understanding and address openquestions.

2. Perform cross-process analysis to identify overarching findings (e.g. trends) and

recommendations and prepare draft report sections for cross-functional areas and

overarching items.

3. Prepare a draft of the report overview section and executive summary.

4. Combine report sections and prepare initial draft of the Confidential Letter to

Management, including the CERCP Process Evaluation Report.

5. Conduct an internal (that is, internal to Crowe Horwath LLP) quality assurance

review cycle to fully review and discuss content and revise as necessary for initial

external review.

6. Prepare and conduct a preliminary report presentation (deliver draft report,

communicate the preliminary evaluation results, explain and confirm the quality

review and report acceptance process). Discuss approach for the public report and

confidential management letter (e.g. identify confidential aspects of the draft

public report).

7. Facilitate external quality assurance review cycle (distribute draft report, collect

and vet feedback, make applicable changes to draft report and letter).

8. Issue final evaluation report (public) and confidential management letter (non-

public).

9. Conduct project closeout (turnover of project assets, final project assessment and

feedback, etc.)

Outputs CERCP Process Evaluation Report (public/non-confidential)

CERCP Process Evaluation Confidential Letter to Management

Table 2 Project Approach Phase 2

Project Scope

The Engagement Letter for this Process Evaluation project established that The intent of this

engagement is to assess the core processes of the CMEP [plus other compliance enforcement

areas+ using criteria found in the following program documents as the primary basis for the

evaluation:

a. Section 400 - Compliance Enforcement

b. Appendix 4B - Sanction Guidelines of the North American Electric Reliability Corporation

c. Appendix 4C - Compliance Monitoring and Enforcement Program

d. Section 500 Organization Registration and Certification

e. Appendix 5 - Organization Registration and Certification Manual

f. Section 1500 Confidentiality of Information


12/130



To that end, the Engagement Letter identified eleven internal processes related to NERCs

compliance enforcement, registration and certification goals that we used as the initial basis for

the scope of this Process Evaluation:

1. Compliance program planning2. Following compliance program confidentiality requirements

3. Registration of users, owners, and operators of the bulk power system

4. Certification of users, owners, and operators of the bulk power system

5. Overseeing the compliance activities of Regional Entities

6. Overseeing the enforcement actions of Regional Entities

7. Reporting to the Federal Energy Regulatory Commission (FERC) or other Applicable

Governmental Authorities

8. Conducting reviews of Regional Entities compliance and enforcement programs

9. Conducting Compliance Violation Investigations and other monitoring and oversight

methods

10.Processing reliability standard violations

11.Handling complaints received on the hotline and via the Web site and those

communicated by the Regional Entities appropriately

During the course of the project this list of eleven initial processes evolved to more accurately

reflect the scope of all CERCP responsibilities and the alignment of these processes to the CERCP

as functionally implemented by NERCs Compliance organization. Crowe discovered that NERC

has defined and documented 37 different internal compliance enforcement, registration and

certification processes and procedures and that the initial list of eleven processes in factrepresents eleven different groups of processes. We termed these groups of processes

functional areas to avoid confusion on the project because we were using the term process

liberally whereby it could mean too many things a policy or rule, a procedure, a group of

processes, etc.

In an effort to ensure that the scope of the assessment fully covered the applicable processes

and procedures, Crowe created a crosswalk of the 37 CMEP Processes and Procedures back to

the original process list of 11 items. The CMEP Processes and Procedures Manual is an internal

set of procedures developed and maintained by NERCs Compliance department to assist in the

implementation of the compliance enforcement, registration and certification program. The

result of that crosswalk is contained in Appendix I of this report.

As the list of areas evolved, Crowe also recognized that some of these functional areas

represent responsibilities that are shared across processes in essence these areas are core or

foundational elements across CERCP processes. Through reviews of NERCs process

documentation and discussions with management in NERCs Compliance Department, we

identified four such areas that are cross-functional in nature: Compliance Program

Confidentiality, Developing and Overseeing the Compliance Training Program, Developing and

Disseminating Compliance Process Directives and Bulletins, and Processing Reliability Standards

Violations. Because these cross-functional areas are not necessarily processes or groups of

processes in and of themselves, but rather requirements and policies with responsibilities

spread throughout the organization and across processes, we redefined the list of areas and

conducted project activities using the following breakout:


13/130



Cross-Functional Areas

1 Compliance Program Confidentiality Requirements

2 Developing and Overseeing the Compliance Training Program

3 Developing and Disseminating Compliance Process Directives and Bulletins

4 Processing Reliability Standards Violations

Functional Areas

1 Compliance Program Planning

2 Overseeing Registration of Owners/Users/Operators of the Bulk Power System

3 Overseeing Certification of Owners/Users/Operators of the Bulk Power System

4 Overseeing Compliance Activities of Regional Entities (excluding CVIs)

5 Overseeing Enforcement Activities of Regional Entities

6 Analyzing and Reporting Compliance Information

7 Conducting Reviews of Regional Entities Compliance and Enforcement Programs

8 NERC Involvement in Compliance Inquiries and Compliance Violation Investigations

9 Handling Complaints

10 Executing Compliance Enforcement Authority Responsibilities

Table 3 CERCP Process Evaluation Final Scope

These four cross-functional areas and ten functional areas comprise the final scope of the CERCP

process evaluation that is, the areas assessed as part of the evaluation and, therefore, the

scope of this report. The 37 processes defined by NERC CMEP Processes and Procedures manual

are all encompassed within these 14 areas. The list of processes is as follows:

NERC

Process

Identifier

NERC CMEP Processes and Procedures Manual

Process Name Relevant ROP Section

NPP-CME-101 Organization Certification Process Procedure ROP 500; ROP Appx 5

NPP-CME-102 Organization Registration Appeals Procedure ROP 500; ROP Appx 5

NPP-CME-103 Organization Certification Appeals Procedure ROP 500; ROP Appx 5

NPP-CME-200 CMEP Development and Maintenance Process ROP 401.1

NPP-CME-201 CMEP Implementation Plan Process ROP 402.1.1; CMEP 4.0

NPP-CME-202 Training Process ROP 402.9

NPP-CME-204

Monitoring and Facilitating Effectiveness of the

CMEP ROP 402; ROP 404

NPP-CME-205 Compliance Process Bulletins/Directives None

NPP-CME-300 Compliance Inquiry Process None

NPP-CME-301 Complaint Process CMEP 3.8

NPP-CME-302 Compliance Violation Investigation Process CMEP 3.4


14/130



NERC

Process

Identifier

NERC CMEP Processes and Procedures Manual

Process Name Relevant ROP Section

NPP-CME-303 Evidence Handling Process CMEP 3.4

NPP-CME-400 Observation of RE-led Compliance Audits CMEP 3.1.5

NPP-CME-401 Regional Entity-led Compliance Audit Process CMEP 3.1.6

NPP-CME-402

Procedure for the Regions to Self-Certify Adherence

to the ROP and CMEP during and Audit None

NPP-CME-403 Regional Entity Spot Check Process None

NPP-CME-404

NERC Audit of Regional Entity Adherence to the

CMEP ROP 402.1.3; ROP 404.3

NPP-CME-500 Remedial Action Process CMEP 7.0

NPP-CME-501

Compliance Violation and Penalty Process - Regional

Entity CEA CMEP 5.1, 5.2, 5.4, 5.6

NPP-CME-502 Settlement Process - Regional Entity CEA CMEP 5.4

NPP-CME-503 Mitigation Process - Regional Entity CEA CMEP 6.0

NPP-CME-504 Mitigation Process - NERC CEA CMEP 6.0

NPP-CME-505 Appeals and Hearing Process CMEP 5.3, 5.5

NPP-CME-506 Penalty Guidance Process Appx 4B

NPP-CME-602 Registered Entity Audit Process Procedure CMEP 3.1

NPP-CME-603 Self-Report Procedure CMEP 3.5

NPP-CME-604 Spot Check Procedure CMEP 3.3

NPP-CME-605 Mitigation Plan Procedure CMEP 6.0

NPP-CME-606 Self-Certification Procedure CMEP 3.2

NPP-CME-607 Data Reporting and Disclosure Procedure CMEP 8.0

NPP-CME-608 Exception Reporting Procedure CMEP 3.7

NPP-CME-609 Periodic Data Submittal Procedure CMEP 3.6

NPP-CME-610 Implementation and Tracking Procedure CMEP 5.1; CMEP 6.0; CMEP 7.0

NPP-CME-611 Remedial Action Directive Procedure - CEA CMEP 7.0

NPP-CME-700 Data Management, Evaluation, and Analysis Process ROP 408; CMEP 8.0

NPP-CME-701 Compliance Data Reporting Process CMEP 8.0

NPP-CME-800 Document Management and Control

ROP 402.8; ROP 404.3; ROP

1500; CMEP 9.0

Table 4 CMEP Processes and Procedures

The evaluation and the results documented within this report are focused at the level of the

cross-functional and functional areas, as demonstrated below, because this was the level of

evaluation most closely tied to the scope and intent of the project as expressed by the

engagement letter. We used individual internal process documents and comparisons to the

Rules of Procedure and other policies for making our evaluations. We also rolled up

observations and recommendations at any individual process level to the relevant functional


15/130


16/130



Definition Applicable

Artifacts

Policy Policies are concise, formal and mandatory statements

of principles and rules formulated or adopted by ordictated to an organization to reach its objectives and

perhaps its goals. They are designed to influence all

major decisions and actions and to set all boundaries for

all activities that take place within the scope set by them.

Applicable Rules

of Procedure(ROP) sections

FERC Orders and

related decisions

Applicable laws

and regulations

Processes

and

Procedures

Defines what is to be done and describes how (that is,

the steps involved) the activities are to be performed.

The mandatory steps and specific methods required to

implement and comply with a policy to meet its intentand perform the operations of the organization.

Processes and procedures must ensure (i.e. put controls

in place) that a point of view held by the governing body

of an organization (that is, the policies) is translated into

steps that result in an outcome compatible with that

view.

Note: while there are subtle, technical differences

between the termsprocess (typically refers only to the

what is to be done) andprocedure (typically refers to

the how it is to be done), we do not attempt to

differentiate these terms or use them to infer specificmeaning by their usage which is to say, they are used

interchangeably throughout this document per the

definition above.

NERC CMEP

(internal)

Processes and

Procedures

Manual

NERC

Compliance

Directives and

Bulletins

Table 5 Policy, Process, and Procedure Defined

Adequacy of Implementation

For each of the functional areas within the scope of the project, Crowe Horwath analyzed the

information obtained through interviews and review of documentation to assess the following

for each process within each functional area:1. Whether the objective of the process is known and documented

2. Whether the process is accurately documented that is, the process as documented

matches how the process is most commonly executed by practitioners

3. Whether the roles and responsibilities in executing the process are documented and

whether responsibilities in executing the process are understood

4. Whether necessary inputs are available and in place to support appropriate execution of the

process


17/130



5. Whether an appropriate processenvironment is in place to support appropriate execution

of the process (e.g. this would include, but not be limited to, governance, organizational

priorities, support resources like tools and technologies, etc.)

6. Whether the process appears to accomplish its desired objective within the time (duration),cost, and resource/material usage limits (that is, within the control limits)

7. Whether the process is applied and/or executed consistently (i.e., it is controlled to the

extent that it consistently executes without significant deviations in procedures)

8. Whether the process is measured (observation and reporting of process execution results

can be real-time or after-the-fact)

9. Whether the process is monitored (ongoing, real-time observation of in-process scenarios to

detect when execution is deviating from plan, requirements, or objectives)

10.Whether the process appears to be efficient, to the extent that unnecessary steps,

iterations, resources, and delays have been eliminated

11.Whether process exceptions are recorded and root causes are assessed for systematic

improvement of the process

12.Whether personnel responsible for executing the process have awareness and

understanding of the process (as documented), and capability to execute the process (i.e.

they are trained and possess appropriate levels of authority)

13.Whether process documentation and supporting tools, technologies, resources, and process

inputs are made readily available

14.Whether the process documentation is made available, as required, and is controlled.

Crowe Horwath performed additional analysis for functional areas that had deficiencies to

determine, where possible, the key factors (e.g. root causes) contributing to the noteddeficiencies. Crowe Horwath identified best practices and developed recommendations that if

implemented may correct any performance deficiencies noted. Crowe Horwath synthesized the

results of the evaluations across all functional areas into an overall summary and identified any

trends or overall issues common throughout functional areas. The results of these efforts are

included in this report.

As noted above, the cross-functional areas in many cases are not in and of themselves processes

as much as they are core or foundational elements across CERCP processes. As such, the

methodology used to assess those areas and make recommendations was limited to those

criteria from the above list that were deemed to be applicable. The methodology used for

cross-functional areas also contemplated the extent to which the area supports or is

implemented by the individual functional areas.

Purpose of Report

The purpose of this report is to provide NERC with an evaluation of its CERCP processes and

procedures. This report, submitted by Crowe Horwath LLP, represents the culmination of

activities performed on the project per the Project Approach and methodology described above.

The primary objective of the report is to document observations as to whether the program has

adequately implemented applicable CERCP processes and procedures, where adequacy is

defined by those criteria identified in the Process Evaluation Methodology section of this


18/130



document, and to make recommendations where the implementation of the CERCP processes

and procedures can be improved.

Document Overview

The following report takes a top-down approach towards presenting the observations and

recommendations. The subsequent section (Section 2) titled Observations and

Recommendations Summary provides a summary level view across all observations and

recommendations. As part of this project and the methodology used, Crowe Horwath LLP

developed a scorecard for evaluating the various functional and cross -functional areas. The

summary contains the summarized level view of that scorecard. The summary also contains a

number of overarching recommendations. These recommendations are summary-level findings

that in many cases present macro-level observations made across functional areas or within

functional areas across multiple criteria.

The next section of the document, Section 3, Cross-Functional Areas Evaluation, contains the

observations and recommendations as they relate to the four cross-functional areas.

Finally, Section 4, Functional Area Evaluation, contains the observations and recommendations

as they relate to the ten functional areas evaluated. Especially relevant to the functional area

evaluations are appendices I and II. Appendix I contains a crosswalk of the functional areas back

to the actual CERCP processes and procedures as defined by the NERC CMEP Processes and

Procedures manual. As most analysis will be documented at the functional area level, it is

important to note which processes and procedures comprise each functional area.

Appendix II contains the criteria used to evaluation each functional area. Appendix III contains

detailed observations and recommendations regarding changes to the ROP. These observations

and recommendations were developed by Crowe as part of its development of the Agreed-Upon

Procedures. Appendix IV contains an excerpt from the Management Letter to NERC from theresults of a recently completed Agreed-Upon Procedures project for a regional entity. The

excerpt contains key recommendations regarding the ROP and the CMEP Processes and

Procedures.

Disclaimer of Confidentiality

This report contains no confidential information. Confidential information gathered or shared

as part of Crowes process evaluation has been shared with NERC management in a separate

confidential letter.


19/130



Section 2: Observations and RecommendationsSummary

Introduction

During our data gathering process, we used a Process Questionnaire (Appendix II) and other

methods to identify observations in different functional areas and cross-functional areas within

the Compliance Department. This section presents a summary of our analysis conducted across

the functional and cross-functional areas.

The Process-driven Organization

Background

In the pre-ERO era of NERC as a Council, the predecessor department to NERCs Compliance

Department could be characterized generally as a service provider organization that responded

predominantly to unique, frequently one-off, situations or requests by a constituency of

voluntary stakeholders, or to the Regions (now NERCs delegated authorities the Regional

Entities) who themselves were also and similarly service providers to those same stakeholders.

However, beginning before and certainly since certification of NERC as the ERO in 2006 NERC

CMEP has been transformed into a regulatory and regulated organization that is significantly

dependant upon development and implementation of thorough and complete processes to

succeed in its primary task/goal, which is consistent monitoring and fair enforcement. NERCs

CMEP implementation must do this in a significantly-prescribed, uniform manner, which is to

say the basis for NERCs CMEP implementation has become significantly more process-driven.

Basis for Observations

Before we summarize the observations made across the various functional areas it is worthwhile

to understand the basis for the observations. In observing the process areas within NERC

Compliance we apply concepts from process engineering and classical process

improvement/process optimization techniques and theories such as Lean, Six Sigma, TQM, etc.

We assessed NERC Compliance processes and procedures across three tiers or layers

comprising the elements critical for organizations to be successful with their processes:

Process

Governance

Organizational success with process starts at the top. Management must

create and instill an environment whereby the organization will operate and

guide its decisions within the policies and processes set by management or

dictated externally by laws or regulations.

The Process

Foundation

In order for policies to be followed and processes to be successful in an

organization, management must, through whatever means available to it,

provide foundational elements that enable the organization to carry out its

mission and operate within the policies and processes. Organizations

frequently fail to achieve process efficiency and/or control process exceptions

(that is, process results outside of the results desired and/or considered

within tolerances set by policy) when they lack one or more foundational

elements that are required to enable processes. Such items include, but are


20/130


21/130



Process Governance and the Process Foundation Summary Observations

Before we summarize the observations made across the various NERC CERCP functional process

areas it is worthwhile to note our observations regarding the governance and foundational

layers of the NERC process environment.

As a regulatory entity, NERC by its very nature is compelled to maintain an environmentfocused on the creation, compliance, and enforcement of its standards and rules. We

observe that the NERC CERCP program generally has the governance and tone at the top

to be successful with its processes. Our assessment of individual functional areas indicates

that process objectives are typically well known and well understood and that there is

clearly a culture of policy and process adherence.

As part of our analysis we placed NERCs CERCP into appropriate context from the standpointthat NERCs Compliance organization and the purpose, roles, and scope of responsibilities

for that organization has existed in their current state only for a relatively very short period

of time. The relative immaturity of the organization certainly has a bearing on theexpectations for its level of process maturity. For example:

o We observed in our analysis that the organizational structure, and the resulting roles

and responsibilities within that structure, continue to mature and change fairly

frequently as the Compliance area has undergone numerous structural changes within

the past two to four years. Three years ago the Compliance organization shifted from a

Service Organization whose purpose was to provide technical assistance to a

Regulatory Organization whose purpose was to regulate (i.e. compliance

enforcement, in addition to the role of registration and certification). The changes in

scope of responsibilities and assignment of responsibilities within an organization

certainly create challenges when attempting to get to a level of process maturity.

o We observed that the NERC Compliance Director/Manager-level positions are staffed, inmost cases, by personnel that are relatively new to the NERC Compliance organization.

Of the six (6) Director/Manager-level positions reporting up through the Vice President

of Compliance the average length of tenure for the personnel is less than 40 months. If

you filter out the one Manager with significant tenure (i.e. greater than five years), we

find that the average Director/Manager in Compliance has been with the organization

just over two years (i.e. approximately 25 months).

o The newness of staff to their respective positions certainly impacts expectations with

respect to process documentation. Organizational and process problems and

inefficiencies are being addressed by NERC compliance personnel (e.g. Compliance has

stood up 35+ processes in the past two years), but organizational and process best

practices emerge typically once some degree of longevity and critical mass has been

achieved. Procedurally, NERCs Compliance area has achieved a great deal despite their

relatively short existence as an organization.

We observe a number of areas (explained further in subsequent sections of this report)where the NERC CERCP can improve its process foundation. It is our observation that a

number of these areas are a result of the NERC Compliance areas relatively short duration

of existence and immature organizational infrastructure and, therefore, process

infrastructure. For example:

o Both the Rules of Procedures (ROP) and the NERC CMEP Processes and Procedures

Manual can be significantly upgraded to provide a more solid operational foundation. A


22/130



number of enhancements and changes to the ROP are recommended and we outline

those in this report. We also find that the internal CMEP Processes and Procedures are

substantially less mature than the ROP and will require a great deal of attention to reach

a point where they are documented in a manner where the tieback to the ROP is more

obvious, consistent across the Processes and Procedures themselves, and adequate toprovide the ultimate level of management control needed. Generally, the CMEP

Processes and Procedures Manual needs better defined roles and responsibilities,

timelines, and outcome-based measurements.

o While existing systems/processes to measure some results and provide statistics, it is

our observation that tools, systems, and technologies can be leveraged to provide

greater degrees of control and security over both public and private/confidential assets,

to enhance process efficiency and effectiveness, and to assist with the creation of a

continuous process improvement environment. For example, we observe that the

CERCP program generally requires a great deal of monitoring, in large part because

there are a number of reporting requirements that must be met and, therefore, requires

significant levels of rigor in terms of tracking and measuring process execution.

However, with that said, we also observe that the systems and technologies available to

Compliance personnel are largely a collection of non-enterprise level solutions created

by various means (e.g. grassroots) to support the needs of the departments.

Generally speaking, some of these critical monitoring, measuring, reporting systems are

currently not structured as long term solutions built on enterprise-level platforms with

the foundation of IT controls required of such systems.

Overarching Observations and Recommendations

Introduction

During our data gathering process, we used a Process Questionnaire (Appendix II) and othermethods to identify observations in different functional areas and cross-functional areas within

the Compliance Department. In doing so, seven themes emerged that impact the Compliance

Department as a whole, as opposed to a specific team, process, or functional area. These seven

themes are important to the NERC Compliance Departments maturity as a process-driven

organization. We provide an overview of these themes below and address each in further detail

in subsequent sub-sections:

1. We recommend to NERC that a number of changes to the ROP (including its related

appendices). These changes should be implemented to ensure a solid foundation for NERCs

compliance program. We observed a number of issues with the ROP whereby it could be

strengthened by adding to it (address areas of Regional Entity accountability e.g.

Compliance Inquiry process), changing it (address areas where Regional Entities differ inpractice from the ROP as documentede.g. terminology such as guidelines and notices of

violation), or deleting from it (removing redundancies).

2. CMEP Process and Procedures documents should be completed, reviewed, and approved,

including incorporating more defined roles, responsibilities, timelines, and outcomes where

these were found to be lacking. We observed that process documents lacked consistency

and at times did not contain obvious tie-backs to the ROPs by virtue of the process used to

develop them. The individual documents requiring completion, review, and approval are

captured within the detailed recommendations of this report.


23/130



3. We observed that the Compliance Department was not consistently meeting a number of its

internal process goals for timeliness. NERC Compliance indicated to us that, with their

current staff resources, they often had to adjust timelines in order to ensure the quality of

their work. It is our observation, therefore, that staffing levels may not be appropriately

aligned for the workload required. However, it is also our observation that there are othercontributing factors (process inefficiency issues, deficiencies in the process infrastructure,

effort-based metrics) which may also contribute heavily towards NERCs ability to meet its

goals in certain compliance enforcement, registration and certification process areas. The

lack of activity level, effort-based metrics impedes the ability to fully assess whether staffing

levels are adequate relative to workload and/or to assess the degree to which staff levels

are required to meet certain levels of desired timeliness and quality.

4. We observed that problems with the consistency of outputs from Regional Entities (in terms

of the level of quality of outputs and the timeliness of those outputs) and differences in

professional opinion between NERC, the Regional Entities, and FERC impacted the timelines

for the Compliance Departments work and the quantity of work that could be accomplished

(i.e. as measured by the number of enforcement actions processed within establish time

frames). For example, one manager noted that Regional Entities often submitted Notices of

Confirmed Violations that contained errors in dates and judgments that NERC did not find

appropriate, such as classifying an issue as a documentation error rather than a failure to

perform, when the standard required documentation of performance. Another manager

stated that NERC and FERC periodically had different opinions on application of reliability

standards on Compliance Violations Investigations.

5. We observed that processes within some functional areas were not adequately monitored

because there were few interim checkpoints being taken during the overall duration of the

process. For example, the functional areas Analyzing and Reporting Compliance Violation

Information and NERC Compliance Enforcement Authority Responsibilities had no

monitoring in place or planned. We also observed that for those functional areas that were

monitored, there was often not adequate follow up when process deviations were found.

In the functional area Overseeing Compliance Activities of Regional Entities, for example, we

observed that staff was given reminders of the need to meet timeliness goals, but no other

actions were taken when these goals were not met.

6. We observed several processes that involved handling large amounts of information and

documentation. NERC had begun to address these issues through the development of new

technologies, but it was our observation that until these are fully implemented, the volume

of data and documentation will continue to be an impediment to accomplishing the

Compliance Departments goals in a timely manner.

7. We identified some issues with the level of controls over data security, confidentiality andphysical security. Confidential information has been removed from this public version and

has been provided under separate cover to NERC management.

Underlying each of these themes are several overarching observations that we made during our

data gathering and analysis process. As appropriate, we also made recommendations to

address these observations. The following sub-sections provide our observations for each of the

seven key areas followed by our recommendations for each area.


24/130



Recommended Changes to the Rules of Procedure

Observations

The Rules of Procedure and its related appendices make up the foundation of NERCs

compliance program. Without a solidly developed ROP1, NERCs ability to oversee andenforce compliance with reliability standards diminishes. For example, if the ROP does not

include a requirement for Regional Entities to submit draft spot check reports to NERC, then

NERC Compliance has no immediate visibility over whether those spot checks were carried

out as scheduled and in a consistent manner. See Overarching Recommendation ROP-01.

During the process of developing the agreed-upon procedures used as a part of NERCs audit

procedures of Regional Entity compliance programs, Crowe identified almost 50 additions,

deletions, and revisions to the ROP that would improve NERCs ability to carry out its

compliance and enforcement functions. These observations are listed and included as

Appendix III to this report. NERC should review these observations and consider the

applicable changes to the ROP. See Overarching Recommendation ROP-01.

While performing the agreed-upon procedures at one of three Regional Entities, we also

made a number of observations and recommendations related to improvements needed to

the ROP. These observations and recommendations are listed and included as Appendix IV

to this report. NERC should also review these observations and recommendations and

consider the related changes to the ROP. See Overarching Recommendation ROP-01.

Since developing the agreed-upon procedures, we found that NERC issued a number of

Compliance Directives, which NERC expected different parties, particularly Regional Entities,

to follow. Some of these were one-time directives that NERC did not expect to be

performed on an ongoing basis or that NERC expected to possibly change in the future.

However, others were permanent requirements, and not all of these permanent

requirements had been incorporated into the ROP. As a result, there is a higher risk that theone-time directives and/or permanent requirements will not be followed, because they

were not in a single reference location and they may not have been viewed by the Regional

Entities as being required or as important as the ROP. Therefore, we recommend that NERC

consider a formal review of bulletins and Compliance Directives to determine those that

should be permanent requirements of the ROP. For those determined to be permanent

we recommend that NERC incorporate those changes into the ROP. See Overarching

Recommendation ROP-01.

1In this report, where we refer to the ROP, we are also referring to its appendices, including Appendix 4C (the Compliance

Management Enforcement Program or CMEP).


25/130



During this project, we recommended several other changes to the ROP, which are

described below. See Overarching Recommendation ROP-01.

o A section should be added to the CMEP to describe the rules governing the

Compliance Inquiry process. We observed that there was no reference to thisprocess in the ROP, although NERC expected Regional Entities to follow it. See

Recommendation CVI-01 in the Functional Area Evaluation NERC Involvement in

Compliance Inquiries and Compliance Violation Investigations.

o References to Transitional Certification in ROP Appendix 5 should be deleted,

because this process has never been implemented. It should be replaced with the

Provisional Certification process. Note at the time ofour observations, a revision

of Appendix 5 was pending that would incorporate these changes, but it was not yet

approved. See Recommendation CER-01 in the Functional Area Evaluation

Overseeing Certification of Owners, Operators, and Users of the Bulk Power

System.

o NERC Compliance Staff have identified a gap in the RoP and CMEP concerningviolation dismissals. In order to exercise appropriate and expected oversight there

needs to be developed both an internal process for the review of dismissals prior to

approval and appropriate changes to RoP and CMEP to ensure due process for the

industry, regional entities and NERC. We observed that NERC must review Notices

of Confirmed Violations prior to filing a Notice of Penalty with FERC, but not before

this stage. As a result, NERC has spent a great deal of time working with Regional

Entities at this end phase after the Regional Entities had already presented their

findings and had significant points of contact with the violating Registered Entities.

See Recommendation ENF-03 in the Functional Area Evaluation Overseeing

Enforcement Activities of Regional Entities.

When revisions to the ROP are made, other documents, such as implementation plans,

delegation agreements, report templates, documents in the Compliance Departments

Processes and Procedures Manual, training materials, and systems may need to be revised

as well. Once the ROP changes are implemented, NERC should undergo a process to ensure

that other updates are made to related documents and systems as well. See

Recommendation ROP-02.


26/130



Recommendations

ROP-01 Perform an assessment of ROP changes recommended as part of this evaluation

(along with changes that may by otherwise queued up within NERCs own

assessment of the ROP) and then develop and implement a plan to incorporate thefollowing into the Rules of Procedure and related appendices (that is, where there

is concurrence on the need for the change):

Observations on the ROP that Crowe made while developing the Regional

Entity AUPs,

Observations on the ROP that Crowe made while performing the Regional

Entity AUPs,

Required Compliance Directives that are meant to be followed on an

ongoing basis and that have not already been incorporated into the ROP,

and

Recommended changes to the ROP that Crowe identified during the

process evaluation project.

As part of the plan, include a schedule for reviewing the ROP revisions internally,

drafting the revised ROP, obtaining necessary input from outside parties, obtaining

BOTCC approval, and issuing the revised ROP.

ROP-02 Based on the ROP changes that are made, determine what changes need to be

made to other documents, including implementation plans, templates used by

NERC and Regional Entities, the Compliance Departments Policy and Procedure

Manual, and any internal systems (tracking, reporting, etc.) if applicable. We

recommend that NERC Compliance develop and implement a plan to incorporate

necessary changes.

ROP-03 Based upon observations made while executing recommendations ROP-01 andROP-02, we recommend that NERC Compliance should establish and implement a

formal internal change control process whereby changes to the ROP, delegation

agreements, implementation plans, templates, the Compliance Departments

Policy and Procedure Manual, training materials, and any internal systems can be

fully managed, coordinated, and tracked to completion in a consistent manner.

Managing internal change in a consistent, methodical manner is critical towards

assuring consistency between all of these pieces that are ultimately critical

towards the effective implementation of the CERCP. The internal change process

would accommodate externally-driven changes (e.g. changes to the ROP and FERC

orders) and ensure that these changes appropriately permeate throughout the

organization and would also accommodate internal changes to ensure consistencybetween the process assets (process documentation, training assets, templates,

etc.)

Process Documentation Development

Observations

The NERC Compliance Department underwent a concerted effort to document its internal

policies, processes, and procedures in a Processes and Procedures Manual. Each team within

the Department contributed to this effort, in addition to performing its regular duties, and a lot

was accomplished, with over 50 documents drafted. However, we observed that NERC


27/130



Compliance had a fairly substantial amount of progress to make before its process documents

could be considered mature and reflective of a process-driven organization.

Certain compliance-related internal processes that NERC performs had not yet been

documented. Specifically:o No document had been drafted of the CMEP Development and Maintenance

Process, meaning that NERC Compliance did not have a documented tool to guide

the development, coordination, or management of changes to the ROP. (See the

Functional Area Evaluation Compliance Program Planning, Criterion 1.)

o No document had been drafted for Penalty Guidance beyond the Sanction

Guidelines contained in the ROP. As a result, NERC Compliance had no documented

practice for the review of penalties assessed by Regional Entities. In particular,

there was no formal process for ensuring consistent application of penalties across

Regional Entities. This is a key NERC responsibility under the CMEP and Appendix 4B

to the ROP. (See the Functional Area Evaluation Overseeing the Enforcement

Activities of Regional Entities, Criterion 1.)

Because the ROP did not specify how to carry out these processes, documented internal

processes are essential to assure consistent achievement of NERCs compliance goals. See

Recommendation PPM-01.

Of the Processes and Procedures Manual documents that have been drafted, only five -

NPP-CME-301 (Complaint Process); NPP-CME-303 (Evidence Handling Process); NPP-CME-

400 (Observation of RE-led Compliance Audits); NPP-CME-403 (RE Spot Check Process); NPP-

CME-404 (NERC Audit of RE Adherence to the CMEP)have been finalized and reviewed by

the Vice President and Director of Compliance or his designee. We observed that several of

the documents were still in very early draft form, with unresolved details blanked out or

unanswered comments and questions. These included the CMEP Implementation PlanProcess (NPP-CME-201) in the functional area Compliance Program Planning; the Training

Process, (NPP-CME-202) in the cross-functional area Developing and Overseeing the

Compliance Training Program; and, several processes within the functional area Overseeing

Regional Entity Enforcement Programs. As a result, the Compliance Department may not

have been executing the processes in a manner consistent with management s goals. See


We observed that the documents in the Processes and Procedures Manual did not clearly

distinguish between policies, processes, and procedures. Often the terms were used

interchangeably. For example, documents such as the Auditor Training Process, Data

Management, Evaluation, and Analysis Process and the Evidence Handling Process did

not really have a process flow, but were more like policy documents. As noted above,policies form the underlying rules and principles of an organization, while processes provide

a general framework for implementing those policies (what is to be done), and procedures

provide the specific steps for executing the processes (how it is to be done). As a best

practice, NERC Compliance should ensure that its Processes and Procedures Manual follows

the appropriate hierarchy of policies, processes, and procedures. See Recommendation

PPM-03.

Several of the processes did not document well-defined roles and responsibilities (these are

detailed throughout the report). We observed that they often noted that steps were to be

performed by NERC, or they may have assigned general responsibility for a process to a

certain manager, without identifying what team members are responsible for what parts of


28/130



the process. Examples of processes where these types of issues were identified included the

Regional Entity-led Compliance Audit Process (NPP-CME-401), within the functional area

Overseeing Regional Entity Compliance Programs, and the Data Management Evaluation

and Analysis Process within the functional area Analyzing and Reporting Compliance

Information. (See Criterion 3 in the functional area evaluations.) Organizational flexibility iscritical, and generally it is not necessary to assign a specific individual to be responsible for a

specific process step. For example, a process could refer to a designated member of the

Enforcement and Mitigation team, or a Regional Entity Compliance Auditor, or the

Manager or Organization Registration and Certification or his designee. Essentially,

Compliance staff should be aware of what roles they have, or might have, within certain

processes. This is especially important as new staff are hired who would not be as familiar

with NERCs policies, processes, and procedures as the current Compliance Department

staff, many of whom were involved in the actual development of these documents. See


We observed that some processes lacked adequate information on how they were to be

carried out. We found this to be especially true when the process involved reviewing or

observing the work of Regional Entities. For example, we observed that NERCs role while

observing Regional Entity compliance audits and NERCs role in reviewing compliance

violation investigations led by Regional Entities were not well defined. (See Criterion 3 in

the functional area evaluations Overseeing Compliance Activities of Regional Entities and

NERC Involvement in Compliance Inquiries and Compliance Violation Investigations.) In

addition, the enforcement process for when NERC is acting as the Compliance Enforcement

Authority was not fully documented. (See Criterion 1 in the functional area evaluation NERC

Compliance Enforcement Authority Responsibilities.) See Recommendation PPM-05.

We observed that a number of processessuch as the Organization Registration Process

(NPP-CME-100) and the Compliance Violation and Penalty Process (NPP-CME-501)did

not include adequate timelines or other measurable outcomes, other than those required

by the ROP. (See Criterion 6 within the functional area evaluations.) Admittedly, this

timelines are often dependent on receiving information from outside parties who cannot be

held to deadlines not specified in the ROP or other policy directives. However, for purposes

of better measuring and monitoring of the processes, and for communicating process norms

to staff, key measurements should be built into the process documents. See


We observed that many of the processes that we reviewed were not developed with the

ROP as a starting point. Instead, Compliance staff related to us that they developed the

processes based on how they carried out their functions at the time or how the processes

had been historically executed. Staff noted that they kept the ROP requirements in mind

while drafting the documents. However, in instances we observed process documents that

were not based on ROP requirements, such as the process documents related to

Compliance Inquiries, and ROP requirements that did not have an associated process

document prepared, such as NERCs reviews of penalties and sanctions. We did not observe

any obvious or direct conflicts between the process document contents and the ROP

requirements, largely because the ROP was generally non-specific on the way many of

NERCs compliance duties are to be carried out. See Recommendation PPM-07.

As part of the review cycle of this process evaluation report it was noted that there were

inconsistent uses of the term CMEP (i.e. Compliance Monitoring and Enforcement Program).

It was NERCs observation of our initial report draft that the scope of the processes


29/130



contained within this report, and likewise within NERCs Compliance Department, was

broader than CMEP, using the ROPs definition of CMEP (which is identified and defined by

Appendix 4C of the ROP). As an example, NERCs Compliance Department refers to its

processes and procedures as the CMEP Processes and Procedures Manual, when this

document contains items that map back to other sections of the ROP (e.g. registration,certification, confidentiality). Similarly, the use of the term RE was noted to be ambiguous

to the extent that this can refer to both regional entities and registered entities. See


In this report, we made other recommendations to improve the quality of the process

documents themselves. These are specific to certain cross-functional and functional areas, and

for purposes of providing an easy cross reference to these related recommendations, these

consist of the following recommendations within the sections listed:

o Recommendations TRA-01 and TRA-02 within the Cross-Functional Area Evaluation

Developing and Overseeing the Compliance Training Program,

o Recommendation PRO-01 within the Cross-Functional Area Evaluation ProcessingReliability Standards Violations,

o Recommendations IMP-01 and IMP-02 in the Functional Area Evaluation

Compliance Program Planning,

o Recommendations REG-01 and REG-02 in the Functional Area Evaluation

Overseeing Registration of Users, Owners, and Operators of the Bulk Power

System,

o Recommendations CER-02, CER-04, and CER-05 in the Functional Area Evaluation

Overseeing Certification of Users, Owners, and Operators of the Bulk Power

System,

o Recommendations COM-01, COM-03, COM-04, COM-05, and COM-06 in the

Functional Area Evaluation Overseeing Compliance Activities of Regional Entities,

o Recommendations ENF-01 and ENF-02 in the Functional Area Evaluation

Overseeing Enforcement Activities of Regional Entities,

o Recommendation REP-03 in the Functional Area Evaluation Analyzing and

Reporting Compliance Information,

o Recommendations REV-01 and REV-03 in the Functional Area Evaluation

Conducting Reviews of Regional Entities Compliance and Enforcement Programs,

o Recommendations CVI-02 and CVI-03 in the Functional Area Evaluation NERC

Involvement in Compliance Inquiries and Compliance Violation Investigations, and

o Recommendations CEA-01, CEA-02, and CEA-04 in the Functional Area Evaluation

NERC Compliance Enforcement Authority Responsibilities.

Recommendations

PPM-01 Develop internal process documents for the CMEP Development and Maintenance

Process and the Penalty Guidance Process. Include procedures for cross-regional

comparisons in the Penalty Guidance Process. Develop a due date for completion

of these drafts.


30/130



PPM-02 Finalize all internal process documents and have them reviewed by the

appropriate Compliance team manager and by the Vice President and Director of

Compliance or a designee. Reviewers of the process documents should ensure

that the Recommendations PPM-04, PPM-05, PPM-06, and all functional area-

specific recommendations made in this report to improve the quality of theprocess documentation are incorporated. All processes should be finalized and

reviewed before FERC begins requesting information for its audit of NERC.

PPM-03 In the internal Processes and Procedures Manual documents, classify the policies,

processes, and procedures into a hierarchy. Note that for some purposes, policies

- and sometimes even processes - may be the underlying ROP or FERC orders,

which would not need to be repeated in their entirety within the documents.

PPM-04 We noted as a recommendation in many of the functional area evaluations, that

NERC should consider the definition of roles and responsibilities within its process

documents. As such, there are many references in the functional area evaluations

to this recommendation (i.e. Recommendation Id PPM-04). We recommend that

NERC should consider designating who is responsible for executing each step

within the related processes and that these designations should continue to be

tied to roles within the organization, as opposed to specific names of individuals.

As individuals are frequently added to the organization, leave the organization, or

change roles within the organization, best practices dictate that designating

responsibilities tied to roles eliminates the need to maintain process documents as

people change.

PPM-05 Where processes were found not to be clear or well-defined (see references to this

recommendation, that is, Recommendation Id PPM-05 in the functional area

evaluations), we recommend that NERC Compliance specify in greater detail whatsteps are to be followed within the processes. In keeping with Recommendation

PPM-04, designate who (by role) is responsible for these process steps.

PPM-06 Where noted as an issue in the functional area evaluations (see references to this

Recommendation, i.e. PPM-06), we recommend that NERC Compliance consider

identifying key milestones (perhaps in many cases, more detailed milestones)

wit

Compliance_Evaluation_Report_121509

Documents

Transcript of Compliance_Evaluation_Report_121509