Post on 08-Apr-2018
8/7/2019 Compliance_Evaluation_Report_121509
1/130
Submitted to:North American Electric Reliability Corporation116-390 Village Boulevard
Princeton, New Jersey 08540
Report prepared by:Crowe Horwath LLP70 West Madison Street, Suite 700Chicago, Illinois 60602-4903
November 23, 2009
Compliance Enforcement, Registration, andCertification Program
Process Evaluation Report
8/7/2019 Compliance_Evaluation_Report_121509
2/130
Compliance Enforcement, Registration and Certification
Process Evaluation Report
AFFILIATES Crowe Horwath LLP is a member of Crowe Horwath International, a Swiss association. Each member firm of Crowe Horwath
International is a separate and independent legal entity. Crowe Horwath LLP and its affiliates are not responsible or liable for any acts or
omissions of Crowe Horwath International or any other member of Crowe Horwath International and specifically disclaim any and all
responsibility or liability for acts or omissions of Crowe Horwath International or any other member of Crowe Horwath International. Crowe
Horwath International does not render any professional services and does not have an ownership or partnership interest in Crowe Horwath
LLP. Crowe Horwath International and its other member firms are not responsible or liable for any acts or omissions of Crowe Horwath LLP and
specifically disclaim any and all responsibility or liability for acts or omissions of Crowe Horwath LLP. 2009 Crowe Horwath LLP
Table of Contents
Executive Summary ....................................................................................................................................... 3
Section 1: Overview ...................................................................................................................................... 8Project Background ...................................................................................................................................... 8Process Evaluation Methodology ............................................................................................................... 14Purpose of Report ...................................................................................................................................... 16Document Overview ................................................................................................................................... 17Disclaimer of Confidentiality ....................................................................................................................... 17
Section 2: Observations and Recommendations Summary ................................................................... 18Introduction ................................................................................................................................................. 18The Process-driven Organization............................................................................................................... 18Process Governance and the Process Foundation Summary Observations ............................................ 20Overarching Observations and Recommendations ................................................................................... 21Categorization of Recommendations ......................................................................................................... 37
Section 3: Cross-Functional Areas Evaluation ........................................................................................ 42Introduction ................................................................................................................................................. 423.1. Compliance Program Confidentiality Requirements .......................................................................... 423.2. Developing and Overseeing the Compliance Training Program........................................................ 433.3. Developing and Disseminating Compliance Process Directives and Bulletins .................................. 443.4. Processing Reliability Standards Violations ....................................................................................... 45
Section 4: Functional Area Evaluation ...................................................................................................... 47Introduction ................................................................................................................................................. 474.1. Compliance Program Planning .......................................................................................................... 484.2. Overseeing Registration of Owners/Users/Operators of the Bulk Power System ............................. 544.3. Overseeing Certification of Owners/Users/Operators of the Bulk Power System ............................. 604.4. Overseeing Compliance Activities of Regional Entities (excluding CVIs) .......................................... 654.5. Overseeing Enforcement Activities of Regional Entities .................................................................... 76
4.6. Analyzing and Reporting Compliance Information ............................................................................. 834.7. Conducting Reviews of Regional Entities Compliance and Enforcement Programs ........................ 884.8. NERC Involvement in Compliance Inquiries and Violation Investigations ......................................... 944.9. Handling Complaints ........................................................................................................................ 1014.10. Executing Compliance Enforcement Authority Responsibilities .................................................... 105
Appendix I Functional Area to Processes and Procedures Crosswalk ............................................. 114
Appendix II Process Questionnaire ....................................................................................................... 117
Appendix III Observations and Recommendations from Development of Agreed-UponProcedures .................................................................................................................................................. 118
Appendix IV Excerpt from Management Letter to NERC ..................................................................... 127
8/7/2019 Compliance_Evaluation_Report_121509
3/130
Compliance Enforcement, Registration and Certification
Process Evaluation Report
AFFILIATES Crowe Horwath LLP is a member of Crowe Horwath International, a Swiss association. Each member firm of Crowe Horwath
International is a separate and independent legal entity. Crowe Horwath LLP and its affiliates are not responsible or liable for any acts or
omissions of Crowe Horwath International or any other member of Crowe Horwath International and specifically disclaim any and all
responsibility or liability for acts or omissions of Crowe Horwath International or any other member of Crowe Horwath International. Crowe
Horwath International does not render any professional services and does not have an ownership or partnership interest in Crowe Horwath
LLP. Crowe Horwath International and its other member firms are not responsible or liable for any acts or omissions of Crowe Horwath LLP and
specifically disclaim any and all responsibility or liability for acts or omissions of Crowe Horwath LLP. 2009 Crowe Horwath LLP
Table of Figures
TABLE 1PROJECT APPROACH PHASE 1................................................................................................................ 9
TABLE 2PROJECT APPROACH PHASE 2.............................................................................................................. 10
TABLE 3CERCPPROCESS EVALUATION FINAL SCOPE.......................................................................................... 12
TABLE 4CMEPPROCESSES AND PROCEDURES................................................................................................... 13
FIGURE 1LEVEL OF EVALUATION...................................................................................................................... 14
TABLE 5POLICY,PROCESS, AND PROCEDURE DEFINED ........................................................................................ 15
TABLE 6THE INFRASTRUCTURE FOR PROCESS SUCCESS ........................................................................................ 19
TABLE 7RECOMMENDATION CATEGORIES......................................................................................................... 37
TABLE 8RECOMMENDATIONS SUMMARY BY CATEGORY OF RECOMMENDATION ..................................................... 40
TABLE 9RECOMMENDATIONS COUNT BY SECTION, BY CATEGORY......................................................................... 41
8/7/2019 Compliance_Evaluation_Report_121509
4/130
Compliance Enforcement, Registration and Certification 3
Process Evaluation Report
Executive Summary
Project Objectives
North American Electric Reliability Corporation (NERC) determined the need for a project to
provide NERC with an evaluation of its Compliance Enforcement, Registration and Certification
Program (CERCP) processes and procedures. NERC engaged Crowe Horwath LLP to perform
this evaluation and Crowe completed this project between July and October, 2009.
The project was initiated to assist NERCs Compliance area (NERC Compliance or the NERC
Compliance Department) in achieving its overall objectives for effective implementation of the
CERCP, including adequate management controls . The project objective, therefore, was to
identify and document whether the program has adequately implemented applicable CERCP
processes and procedures in accordance with the applicable law, FERC orders, and NERC Rules
of Procedure. Additionally, Crowe reviewed the internal processes and procedures used by the
Compliance Department in carrying out its duties for consistency with the Rules of Procedure
and for completeness and effectiveness.
Project Approach
For purposes of planning, tracking, and execution, the project was divided into two separate,
sequential phases where the outputs from Phase I became key inputs to Phase II activities.
Phase I of the project primarily involved (i) conducting necessary project initiation and planning
activities, and (ii) gathering information from NERC Compliance personnel concerning the
processes that NERCs Compliance Department has in place over the compliance with and
enforcement of approved electric reliability standards. Phase II of the project involved (i)
performing analysis and review of process and procedure information and artifacts gathered in
Phase I, (ii) preparation of the public report and the confidential letter to management, (iii)review and revisions to the reports based upon feedback, and (iv) final delivery of the reports
and project closeout.
Project Scope
Four cross-functional areas and ten functional areas comprise the final scope of the CERCP
process evaluation and, therefore, the scope of this report. Cross-functional areas are areas
that underlie all CERCP processes for example, confidentiality requirements. Functional areas
represent groupings of related processes, frequently for purposes of mapping related processes
back to a unit or basic responsibility of the program for example, registration, certification,
CVIs, and enforcement are all functional areas. The 37 processes defined by the NERC
Compliance Departments CMEP Processes and Procedures Manual are all encompassed within
these 14 cross-functional and functional areas. The CMEP Processes and Procedures Manual is
an internal set of procedures developed and maintained by NERCs Compliance department to
assist in the implementation of the compliance enforcement, registration and certification
program.
Cross-Functional Areas
1. Compliance Program Confidentiality Requirements
2. Developing and Overseeing the Compliance Training Program
3. Developing and Disseminating Compliance Process Directives and Bulletins
8/7/2019 Compliance_Evaluation_Report_121509
5/130
Compliance Enforcement, Registration and Certification 4
Process Evaluation Report
4. Processing Reliability Standards Violations
Functional Areas
1. Compliance Program Planning
2. Overseeing Registration of Owners/Users/Operators of the Bulk Power System
3. Overseeing Certification of Owners/Users/Operators of the Bulk Power System
4. Overseeing Compliance Activities of Regional Entities (excluding CVIs)
5. Overseeing Enforcement Activities of Regional Entities
6. Analyzing and Reporting Compliance Information
7. Conducting Reviews of Regional Entities Compliance and Enforcement Programs
8. NERC Involvement in Compliance Inquiries and Compliance Violation Investigations
9. Handling Complaints
10.Executing Compliance Enforcement Authority Responsibilities
Purpose of Report
The purpose of this report is to provide NERC with an evaluation of its CERCP processes and
procedures across the 14 cross-functional and functional areas identified above. This report,
submitted by Crowe Horwath LLP, represents the culmination of activities performed on the
project.
The primary objective of the report is to provide observations as to whether the program has
adequately developed and implemented applicable CERCP processes and procedures, where
adequacy is defined by those criteria identified in the Process Evaluation Methodologysection
of this document, and to make recommendations where the implementation of the CERCPprocesses and procedures can be improved.
Process Governance and the Process Foundation
In summary, our observations regarding the governance and foundational layers of the NERC
Compliance process environment are as follows:
As a regulatory entity, NERC is by its very nature compelled to maintain an environmentfocused on the creation, compliance, and enforcement of its standards and rules. We
observed that the NERC CERCP program generally has the governance and tone at the top
to be successful with its processes. Our assessment of individual functional areas indicates
that process objectives are typically well known and well understood and that there is
clearly a culture of policy and process adherence.
As part of our analysis we placed NERCs CERCP into appropriate context from the standpointthat NERCs Compliance organization and the purpose, roles, and scope of responsibilities
for that organization has existed in their current state only for a relatively very short period
of time. The relative immaturity of the organization certainly has a bearing on the
expectations for its level of process maturity. For example:
o We observed in our analysis that the organizational structure, and the resulting roles
and responsibilities within that structure, continue to mature and change fairly
frequently as the Compliance area has undergone numerous structural changes within
the past two to four years. Three years ago the Compliance organization shifted from a
8/7/2019 Compliance_Evaluation_Report_121509
6/130
8/7/2019 Compliance_Evaluation_Report_121509
7/130
Compliance Enforcement, Registration and Certification 6
Process Evaluation Report
term solutions built on enterprise-level platforms with the foundation of IT controls
required of such systems.
Overarching Observations and Recommendations
As part of this project, Crowe identified observations in different functional areas and cross-functional areas within the Compliance Department. In doing so, seven themes surfaced that
impact the Compliance Department as a whole, as opposed to a specific team, process, or
functional area. These seven themes are important to the NERC Compliance Departments
maturity as a process-driven organization. We provide an overview of these themes below.
Each is addressed in further detail this report:
1. We recommend to NERC that a number of changes to the ROP (including its related
appendices). These changes should be implemented to ensure a solid foundation for NERCs
compliance program. We observed a number of issues with the ROP whereby it could be
strengthened by adding to it (address areas of Regional Entity accountability e.g.
Compliance Inquiry process), changing it (address areas where Regional Entities differ in
practice from the ROP as documentede.g. terminology such as guidelines and notices ofviolation), or deleting from it (removing redundancies).
2. We recommend to NERC that CMEP Process and Procedures documents should be
completed, reviewed, and approved, including incorporating more defined roles,
responsibilities, timelines, and outcomes where these were found to be lacking. We
observed that process documents lacked consistency in form and included some conflicting
information, and at times did not contain obvious tie-backs to the ROPs by virtue of the
process used to develop them. The individual documents requiring completion, review, and
approval are captured within the detailed recommendations of this report.
3. We observed that the Compliance Department was not consistently meeting a number of its
internal process goals for timeliness. NERC Compliance indicated to us that, with theircurrent staff resources, they often had to adjust timelines in order to ensure the quality of
their work. It is our observation, therefore, that staffing levels may not be appropriately
aligned for the workload required. However, it is also our observation that there are other
contributing factors (process inefficiency issues, deficiencies in the process infrastructure,
effort-based metrics) which may also contribute heavily towards NERCs ability to meet its
goals in certain compliance enforcement, registration and certification process areas. The
lack of activity level, effort-based metrics impeded the ability to fully assess whether staffing
levels are adequate relative to workload and/or to assess the degree to which staff levels
are required to meet certain levels of desired timeliness and quality.
4. We observed that problems with the consistency of outputs from Regional Entities (in terms
of the level of quality of outputs and the timeliness of those outputs) and differences inprofessional opinion between NERC, the Regional Entities, and FERC impacted the timelines
for the Compliance Departments work and the quantity of work that can be accomplished
(i.e. as measured by the number of enforcement actions processed within established time
frames). For example, one manager noted that Regional Entities often submitted Notices of
Confirmed Violations that contained errors in dates and judgments that NERC did not find
appropriate, such as classifying an issue as a documentation error rather than a failure to
perform, when the standard required documentation of performance. Another manager
stated that NERC and FERC periodically had different opinions on applications of reliability
standards on Compliance Violations Investigations.
8/7/2019 Compliance_Evaluation_Report_121509
8/130
Compliance Enforcement, Registration and Certification 7
Process Evaluation Report
5. We observed that processes within some functional areas were not adequately monitored
because there were few interim checkpoints being taken during the overall duration of the
process. For example, the functional areas Analyzing and Reporting Compliance Violation
Information and NERC Compliance Enforcement Authority Responsibilities had no
monitoring in place or planned. We also observed that for those functional areas that weremonitored, there was often not adequate follow up when process deviations were found. In
the functional area Overseeing Compliance Activities of Regional Entities, for example, we
observed that staff was given reminders of the need to meet timeliness goals, but no other
actions were taken when these goals were not met.
6. We observed several processes that involve handling large amounts of information and
documentation. NERC had begun to address these issues through the development of new
technologies, but it was our observation that until these are fully implemented, the volume
of data and documentation will continue to be an impediment to accomplishing the
Compliance Departments goals in a timely manner.
7. We identified some issues with the level of controls over data security, confidentiality, andphysical security. Confidential information has been removed from this public version and
has been provided under separate cover to NERC management.
Document Overview
This report takes a top-down approach towards presenting the detailed observations and
recommendations. The Overviewsection provides a more detailed look at the objectives, scope,
and approach of this Process Evaluation.
The subsequent section (Section 2) titled Observations and Recommendations Summary
provides a summary level view across all observations and recommendations. As part of this
project and the methodology used, Crowe Horwath LLP developed a scorecard for evaluating
the various functional and cross-functional areas. The summary contains the summarized levelview of that scorecard. The summary also contains a number of overarching recommendations.
These recommendations are summary-level findings that in many cases present macro-level
observations made across functional areas or within functional areas across multiple criteria.
The next section of the document, Section 3, Cross-Functional Areas Evaluation, contains the
observations and recommendations as they relate to the four cross-functional areas.
Finally, Section 4, Functional Area Evaluation, contains the observations and recommendations
as they relate to the ten functional areas evaluated. Especially relevant to the functional area
evaluations are appendices I and II. Appendix I contains a crosswalk of the functional areas back
to the actual CERCP processes and procedures as defined by the NERC CMEP Processes and
Procedures manual. As most analysis will be documented as the functional area level, it is
important to note which processes and procedures comprise each functional area.
Appendix II contains the criteria used to evaluation each functional area. Appendix III contains
detailed observations and recommendations regarding changes to the ROP. These observations
and recommendations were developed by Crowe as part of its development of the Agreed-Upon
Procedures. Appendix IV contains an excerpt from the Management Letter to NERC from the
results of recently completed Agreed-Upon Procedures project for a regional entity. The excerpt
contains key recommendations regarding the ROP and the CMEP Processes and Procedures.
8/7/2019 Compliance_Evaluation_Report_121509
9/130
Compliance Enforcement, Registration and Certification 8
Process Evaluation Report
Section 1: Overview
Project Background
Project Objectives
North American Electric Reliability Corporation (NERC) determined the need for a project to
provide NERC with an evaluation of its Compliance Enforcement, Registration and Certification
Program (CERCP) processes and procedures. NERC engaged Crowe Horwath LLP to perform
this evaluation and Crowe completed this project between July and October, 2009.
The project was initiated to assist NERCs Compliance area in achieving its overall objectives for
effective implementation of the CERCP, including adequate management controls. The project
objective, therefore, was to identify and document whether the program meets the
requirements of the implementing rules established by FERC for the Energy Policy Act (i.e. the
NERC Rules of Procedure and subsequent FERC orders), and if the NERC implementation has
adequately implemented applicable CERCP processes and procedures.
More specifically, the intent of this engagement was to:
1. Assess the core internal processes of the NERC CERCP implementation through interviews of
NERC Compliance employees and inspection of documentary evidence, using criteria found in
the following program documents from the ROP, and applicable sections from 18 CFR Part 29 as
the primary basis for the evaluation:
a. Section 400 Compliance Enforcement
b. Appendix 4B Sanction Guidelines of the North American Electric Reliability
Corporation
c. Appendix 4C Compliance Monitoring and Enforcement Program
d. Section 500 Organization Registration and Certification
e. Appendix 5 Organization Registration and Certification Manual
f. Section 1500 Confidentiality of Information
2. Provide an independent Process Evaluation Report (i.e. this report) for public use to align
with NERCs need to be transparent, stating process efficiency, resource, or other improvement
recommendations identified (if applicable) during the process evaluation.
3. Provide a Confidential Letter to Management (i.e. a separate letter from this report) for any
process efficiency, resource, or other improvement recommendations that for the purposes ofcommunicating such information must include the identification of confidential information,
including but not limited to company names, data, NERC confidential information or personnel
identification. NERC assisted Crowe Horwath LLP with identification of such information.
Project Approach
For purposes of planning, tracking, and effective execution, the project was divided into two
separate, sequential phases where the outputs from Phase I became key inputs to Phase II
activities. The purpose, scope, activities, and outcomes of the two phases are described below.
8/7/2019 Compliance_Evaluation_Report_121509
10/130
Compliance Enforcement, Registration and Certification 9
Process Evaluation Report
Phase I - Planning and Data Gathering
Purpose Phase I of the project involved (i) conducting necessary project initiation and planning
activities, and (ii) gathering information from Compliance personnel concerning the
processes that NERCs Compliance Department has in place over the compliance withand enforcement of approved electric reliability standards. The activities included a
review of the criteria contained in the applicable sections of the Rules of Procedure,
developing questionnaires for data gathering, scheduling and conducting interviews
with NERC Compliance staff, and reviewing information received from NERC
Compliance staff and other documentary evidence regarding the execution of the
CERCP processes.
Activities 1. Conduct project initiation activities including, but not limited to, project kickoff
meetings to coordinate all project stakeholders and to ensure that there is a
common understanding for the project objectives, scope, approach, schedule, and
responsibilities.
2. Plan and establish the operating model for the project. Planning included the
creation and coordination of the project schedule of activities, resource schedules
and availability, project communications and status reporting.
3. Create a crosswalk of NERC compliance processes and procedures back to
functional areas that effectively group and map the processes and procedures
back to areas of organizational responsibility (see Appendix I).
4. Conduct initial interviews with functional area owners (primarily CERCP Managers
and Directors) to confirm understanding of the scope of the functional area, key
interactions with other functional areas, and the processes and resources
implemented within the area. Identify key documents and information supporting
the implementation of the CERCP processes and procedures.
5. Request, collect, and review documents and information supporting the
implementation of the CERCP processes and procedures (received from functional
areas owners and key subject matter experts).
6. Conduct formal interviews with functional area owners and functional area staff
(primarily analysts, investigators, administrators, and auditors) using common
functional area evaluation criteria to determine the status of the CERCP
implementation with respect to the criteria (note: the interview criteria are
included as Appendix II to this report).
7. Conduct final functional area interviews to confirm understanding and answer
final questions regarding processes, procedures, documents, and process artifacts.
Interviews included, in some cases, observation of various supporting IT systems.
Outputs Project Operating Model and Project Schedule
CMEP Process and Procedure-to-Functional Area Crosswalk (included as Appendix Ito this report)
Process review criteria and interview template (included as Appendix II to thisreport)
Documents and Artifacts Log
Table 1 Project Approach Phase 1
Phase II - Data Analysis and Reporting
Purpose Phase II of the project involved (i) performing analysis and review of information
gathered in Phase I, (ii) preparation of the public report and the confidential letter to
8/7/2019 Compliance_Evaluation_Report_121509
11/130
Compliance Enforcement, Registration and Certification 10
Process Evaluation Report
management, (iii) review and revisions to the reports based upon feedback, and (iv)
final delivery of the reports and project closeout.
Activities 1. Prepare preliminary process write-ups for functional areas and conduct follow-up
interviews and communications to confirm understanding and address openquestions.
2. Perform cross-process analysis to identify overarching findings (e.g. trends) and
recommendations and prepare draft report sections for cross-functional areas and
overarching items.
3. Prepare a draft of the report overview section and executive summary.
4. Combine report sections and prepare initial draft of the Confidential Letter to
Management, including the CERCP Process Evaluation Report.
5. Conduct an internal (that is, internal to Crowe Horwath LLP) quality assurance
review cycle to fully review and discuss content and revise as necessary for initial
external review.
6. Prepare and conduct a preliminary report presentation (deliver draft report,
communicate the preliminary evaluation results, explain and confirm the quality
review and report acceptance process). Discuss approach for the public report and
confidential management letter (e.g. identify confidential aspects of the draft
public report).
7. Facilitate external quality assurance review cycle (distribute draft report, collect
and vet feedback, make applicable changes to draft report and letter).
8. Issue final evaluation report (public) and confidential management letter (non-
public).
9. Conduct project closeout (turnover of project assets, final project assessment and
feedback, etc.)
Outputs CERCP Process Evaluation Report (public/non-confidential)
CERCP Process Evaluation Confidential Letter to Management
Table 2 Project Approach Phase 2
Project Scope
The Engagement Letter for this Process Evaluation project established that The intent of this
engagement is to assess the core processes of the CMEP [plus other compliance enforcement
areas+ using criteria found in the following program documents as the primary basis for the
evaluation:
a. Section 400 - Compliance Enforcement
b. Appendix 4B - Sanction Guidelines of the North American Electric Reliability Corporation
c. Appendix 4C - Compliance Monitoring and Enforcement Program
d. Section 500 Organization Registration and Certification
e. Appendix 5 - Organization Registration and Certification Manual
f. Section 1500 Confidentiality of Information
8/7/2019 Compliance_Evaluation_Report_121509
12/130
Compliance Enforcement, Registration and Certification 11
Process Evaluation Report
To that end, the Engagement Letter identified eleven internal processes related to NERCs
compliance enforcement, registration and certification goals that we used as the initial basis for
the scope of this Process Evaluation:
1. Compliance program planning2. Following compliance program confidentiality requirements
3. Registration of users, owners, and operators of the bulk power system
4. Certification of users, owners, and operators of the bulk power system
5. Overseeing the compliance activities of Regional Entities
6. Overseeing the enforcement actions of Regional Entities
7. Reporting to the Federal Energy Regulatory Commission (FERC) or other Applicable
Governmental Authorities
8. Conducting reviews of Regional Entities compliance and enforcement programs
9. Conducting Compliance Violation Investigations and other monitoring and oversight
methods
10.Processing reliability standard violations
11.Handling complaints received on the hotline and via the Web site and those
communicated by the Regional Entities appropriately
During the course of the project this list of eleven initial processes evolved to more accurately
reflect the scope of all CERCP responsibilities and the alignment of these processes to the CERCP
as functionally implemented by NERCs Compliance organization. Crowe discovered that NERC
has defined and documented 37 different internal compliance enforcement, registration and
certification processes and procedures and that the initial list of eleven processes in factrepresents eleven different groups of processes. We termed these groups of processes
functional areas to avoid confusion on the project because we were using the term process
liberally whereby it could mean too many things a policy or rule, a procedure, a group of
processes, etc.
In an effort to ensure that the scope of the assessment fully covered the applicable processes
and procedures, Crowe created a crosswalk of the 37 CMEP Processes and Procedures back to
the original process list of 11 items. The CMEP Processes and Procedures Manual is an internal
set of procedures developed and maintained by NERCs Compliance department to assist in the
implementation of the compliance enforcement, registration and certification program. The
result of that crosswalk is contained in Appendix I of this report.
As the list of areas evolved, Crowe also recognized that some of these functional areas
represent responsibilities that are shared across processes in essence these areas are core or
foundational elements across CERCP processes. Through reviews of NERCs process
documentation and discussions with management in NERCs Compliance Department, we
identified four such areas that are cross-functional in nature: Compliance Program
Confidentiality, Developing and Overseeing the Compliance Training Program, Developing and
Disseminating Compliance Process Directives and Bulletins, and Processing Reliability Standards
Violations. Because these cross-functional areas are not necessarily processes or groups of
processes in and of themselves, but rather requirements and policies with responsibilities
spread throughout the organization and across processes, we redefined the list of areas and
conducted project activities using the following breakout:
8/7/2019 Compliance_Evaluation_Report_121509
13/130
Compliance Enforcement, Registration and Certification 12
Process Evaluation Report
Cross-Functional Areas
1 Compliance Program Confidentiality Requirements
2 Developing and Overseeing the Compliance Training Program
3 Developing and Disseminating Compliance Process Directives and Bulletins
4 Processing Reliability Standards Violations
Functional Areas
1 Compliance Program Planning
2 Overseeing Registration of Owners/Users/Operators of the Bulk Power System
3 Overseeing Certification of Owners/Users/Operators of the Bulk Power System
4 Overseeing Compliance Activities of Regional Entities (excluding CVIs)
5 Overseeing Enforcement Activities of Regional Entities
6 Analyzing and Reporting Compliance Information
7 Conducting Reviews of Regional Entities Compliance and Enforcement Programs
8 NERC Involvement in Compliance Inquiries and Compliance Violation Investigations
9 Handling Complaints
10 Executing Compliance Enforcement Authority Responsibilities
Table 3 CERCP Process Evaluation Final Scope
These four cross-functional areas and ten functional areas comprise the final scope of the CERCP
process evaluation that is, the areas assessed as part of the evaluation and, therefore, the
scope of this report. The 37 processes defined by NERC CMEP Processes and Procedures manual
are all encompassed within these 14 areas. The list of processes is as follows:
NERC
Process
Identifier
NERC CMEP Processes and Procedures Manual
Process Name Relevant ROP Section
NPP-CME-101 Organization Certification Process Procedure ROP 500; ROP Appx 5
NPP-CME-102 Organization Registration Appeals Procedure ROP 500; ROP Appx 5
NPP-CME-103 Organization Certification Appeals Procedure ROP 500; ROP Appx 5
NPP-CME-200 CMEP Development and Maintenance Process ROP 401.1
NPP-CME-201 CMEP Implementation Plan Process ROP 402.1.1; CMEP 4.0
NPP-CME-202 Training Process ROP 402.9
NPP-CME-204
Monitoring and Facilitating Effectiveness of the
CMEP ROP 402; ROP 404
NPP-CME-205 Compliance Process Bulletins/Directives None
NPP-CME-300 Compliance Inquiry Process None
NPP-CME-301 Complaint Process CMEP 3.8
NPP-CME-302 Compliance Violation Investigation Process CMEP 3.4
8/7/2019 Compliance_Evaluation_Report_121509
14/130
Compliance Enforcement, Registration and Certification 13
Process Evaluation Report
NERC
Process
Identifier
NERC CMEP Processes and Procedures Manual
Process Name Relevant ROP Section
NPP-CME-303 Evidence Handling Process CMEP 3.4
NPP-CME-400 Observation of RE-led Compliance Audits CMEP 3.1.5
NPP-CME-401 Regional Entity-led Compliance Audit Process CMEP 3.1.6
NPP-CME-402
Procedure for the Regions to Self-Certify Adherence
to the ROP and CMEP during and Audit None
NPP-CME-403 Regional Entity Spot Check Process None
NPP-CME-404
NERC Audit of Regional Entity Adherence to the
CMEP ROP 402.1.3; ROP 404.3
NPP-CME-500 Remedial Action Process CMEP 7.0
NPP-CME-501
Compliance Violation and Penalty Process - Regional
Entity CEA CMEP 5.1, 5.2, 5.4, 5.6
NPP-CME-502 Settlement Process - Regional Entity CEA CMEP 5.4
NPP-CME-503 Mitigation Process - Regional Entity CEA CMEP 6.0
NPP-CME-504 Mitigation Process - NERC CEA CMEP 6.0
NPP-CME-505 Appeals and Hearing Process CMEP 5.3, 5.5
NPP-CME-506 Penalty Guidance Process Appx 4B
NPP-CME-602 Registered Entity Audit Process Procedure CMEP 3.1
NPP-CME-603 Self-Report Procedure CMEP 3.5
NPP-CME-604 Spot Check Procedure CMEP 3.3
NPP-CME-605 Mitigation Plan Procedure CMEP 6.0
NPP-CME-606 Self-Certification Procedure CMEP 3.2
NPP-CME-607 Data Reporting and Disclosure Procedure CMEP 8.0
NPP-CME-608 Exception Reporting Procedure CMEP 3.7
NPP-CME-609 Periodic Data Submittal Procedure CMEP 3.6
NPP-CME-610 Implementation and Tracking Procedure CMEP 5.1; CMEP 6.0; CMEP 7.0
NPP-CME-611 Remedial Action Directive Procedure - CEA CMEP 7.0
NPP-CME-700 Data Management, Evaluation, and Analysis Process ROP 408; CMEP 8.0
NPP-CME-701 Compliance Data Reporting Process CMEP 8.0
NPP-CME-800 Document Management and Control
ROP 402.8; ROP 404.3; ROP
1500; CMEP 9.0
Table 4 CMEP Processes and Procedures
The evaluation and the results documented within this report are focused at the level of the
cross-functional and functional areas, as demonstrated below, because this was the level of
evaluation most closely tied to the scope and intent of the project as expressed by the
engagement letter. We used individual internal process documents and comparisons to the
Rules of Procedure and other policies for making our evaluations. We also rolled up
observations and recommendations at any individual process level to the relevant functional
8/7/2019 Compliance_Evaluation_Report_121509
15/130
8/7/2019 Compliance_Evaluation_Report_121509
16/130
Compliance Enforcement, Registration and Certification 15
Process Evaluation Report
Definition Applicable
Artifacts
Policy Policies are concise, formal and mandatory statements
of principles and rules formulated or adopted by ordictated to an organization to reach its objectives and
perhaps its goals. They are designed to influence all
major decisions and actions and to set all boundaries for
all activities that take place within the scope set by them.
Applicable Rules
of Procedure(ROP) sections
FERC Orders and
related decisions
Applicable laws
and regulations
Processes
and
Procedures
Defines what is to be done and describes how (that is,
the steps involved) the activities are to be performed.
The mandatory steps and specific methods required to
implement and comply with a policy to meet its intentand perform the operations of the organization.
Processes and procedures must ensure (i.e. put controls
in place) that a point of view held by the governing body
of an organization (that is, the policies) is translated into
steps that result in an outcome compatible with that
view.
Note: while there are subtle, technical differences
between the termsprocess (typically refers only to the
what is to be done) andprocedure (typically refers to
the how it is to be done), we do not attempt to
differentiate these terms or use them to infer specificmeaning by their usage which is to say, they are used
interchangeably throughout this document per the
definition above.
NERC CMEP
(internal)
Processes and
Procedures
Manual
NERC
Compliance
Directives and
Bulletins
Table 5 Policy, Process, and Procedure Defined
Adequacy of Implementation
For each of the functional areas within the scope of the project, Crowe Horwath analyzed the
information obtained through interviews and review of documentation to assess the following
for each process within each functional area:1. Whether the objective of the process is known and documented
2. Whether the process is accurately documented that is, the process as documented
matches how the process is most commonly executed by practitioners
3. Whether the roles and responsibilities in executing the process are documented and
whether responsibilities in executing the process are understood
4. Whether necessary inputs are available and in place to support appropriate execution of the
process
8/7/2019 Compliance_Evaluation_Report_121509
17/130
Compliance Enforcement, Registration and Certification 16
Process Evaluation Report
5. Whether an appropriate processenvironment is in place to support appropriate execution
of the process (e.g. this would include, but not be limited to, governance, organizational
priorities, support resources like tools and technologies, etc.)
6. Whether the process appears to accomplish its desired objective within the time (duration),cost, and resource/material usage limits (that is, within the control limits)
7. Whether the process is applied and/or executed consistently (i.e., it is controlled to the
extent that it consistently executes without significant deviations in procedures)
8. Whether the process is measured (observation and reporting of process execution results
can be real-time or after-the-fact)
9. Whether the process is monitored (ongoing, real-time observation of in-process scenarios to
detect when execution is deviating from plan, requirements, or objectives)
10.Whether the process appears to be efficient, to the extent that unnecessary steps,
iterations, resources, and delays have been eliminated
11.Whether process exceptions are recorded and root causes are assessed for systematic
improvement of the process
12.Whether personnel responsible for executing the process have awareness and
understanding of the process (as documented), and capability to execute the process (i.e.
they are trained and possess appropriate levels of authority)
13.Whether process documentation and supporting tools, technologies, resources, and process
inputs are made readily available
14.Whether the process documentation is made available, as required, and is controlled.
Crowe Horwath performed additional analysis for functional areas that had deficiencies to
determine, where possible, the key factors (e.g. root causes) contributing to the noteddeficiencies. Crowe Horwath identified best practices and developed recommendations that if
implemented may correct any performance deficiencies noted. Crowe Horwath synthesized the
results of the evaluations across all functional areas into an overall summary and identified any
trends or overall issues common throughout functional areas. The results of these efforts are
included in this report.
As noted above, the cross-functional areas in many cases are not in and of themselves processes
as much as they are core or foundational elements across CERCP processes. As such, the
methodology used to assess those areas and make recommendations was limited to those
criteria from the above list that were deemed to be applicable. The methodology used for
cross-functional areas also contemplated the extent to which the area supports or is
implemented by the individual functional areas.
Purpose of Report
The purpose of this report is to provide NERC with an evaluation of its CERCP processes and
procedures. This report, submitted by Crowe Horwath LLP, represents the culmination of
activities performed on the project per the Project Approach and methodology described above.
The primary objective of the report is to document observations as to whether the program has
adequately implemented applicable CERCP processes and procedures, where adequacy is
defined by those criteria identified in the Process Evaluation Methodology section of this
8/7/2019 Compliance_Evaluation_Report_121509
18/130
Compliance Enforcement, Registration and Certification 17
Process Evaluation Report
document, and to make recommendations where the implementation of the CERCP processes
and procedures can be improved.
Document Overview
The following report takes a top-down approach towards presenting the observations and
recommendations. The subsequent section (Section 2) titled Observations and
Recommendations Summary provides a summary level view across all observations and
recommendations. As part of this project and the methodology used, Crowe Horwath LLP
developed a scorecard for evaluating the various functional and cross -functional areas. The
summary contains the summarized level view of that scorecard. The summary also contains a
number of overarching recommendations. These recommendations are summary-level findings
that in many cases present macro-level observations made across functional areas or within
functional areas across multiple criteria.
The next section of the document, Section 3, Cross-Functional Areas Evaluation, contains the
observations and recommendations as they relate to the four cross-functional areas.
Finally, Section 4, Functional Area Evaluation, contains the observations and recommendations
as they relate to the ten functional areas evaluated. Especially relevant to the functional area
evaluations are appendices I and II. Appendix I contains a crosswalk of the functional areas back
to the actual CERCP processes and procedures as defined by the NERC CMEP Processes and
Procedures manual. As most analysis will be documented at the functional area level, it is
important to note which processes and procedures comprise each functional area.
Appendix II contains the criteria used to evaluation each functional area. Appendix III contains
detailed observations and recommendations regarding changes to the ROP. These observations
and recommendations were developed by Crowe as part of its development of the Agreed-Upon
Procedures. Appendix IV contains an excerpt from the Management Letter to NERC from theresults of a recently completed Agreed-Upon Procedures project for a regional entity. The
excerpt contains key recommendations regarding the ROP and the CMEP Processes and
Procedures.
Disclaimer of Confidentiality
This report contains no confidential information. Confidential information gathered or shared
as part of Crowes process evaluation has been shared with NERC management in a separate
confidential letter.
8/7/2019 Compliance_Evaluation_Report_121509
19/130
Compliance Enforcement, Registration and Certification 18
Process Evaluation Report
Section 2: Observations and RecommendationsSummary
Introduction
During our data gathering process, we used a Process Questionnaire (Appendix II) and other
methods to identify observations in different functional areas and cross-functional areas within
the Compliance Department. This section presents a summary of our analysis conducted across
the functional and cross-functional areas.
The Process-driven Organization
Background
In the pre-ERO era of NERC as a Council, the predecessor department to NERCs Compliance
Department could be characterized generally as a service provider organization that responded
predominantly to unique, frequently one-off, situations or requests by a constituency of
voluntary stakeholders, or to the Regions (now NERCs delegated authorities the Regional
Entities) who themselves were also and similarly service providers to those same stakeholders.
However, beginning before and certainly since certification of NERC as the ERO in 2006 NERC
CMEP has been transformed into a regulatory and regulated organization that is significantly
dependant upon development and implementation of thorough and complete processes to
succeed in its primary task/goal, which is consistent monitoring and fair enforcement. NERCs
CMEP implementation must do this in a significantly-prescribed, uniform manner, which is to
say the basis for NERCs CMEP implementation has become significantly more process-driven.
Basis for Observations
Before we summarize the observations made across the various functional areas it is worthwhile
to understand the basis for the observations. In observing the process areas within NERC
Compliance we apply concepts from process engineering and classical process
improvement/process optimization techniques and theories such as Lean, Six Sigma, TQM, etc.
We assessed NERC Compliance processes and procedures across three tiers or layers
comprising the elements critical for organizations to be successful with their processes:
Process
Governance
Organizational success with process starts at the top. Management must
create and instill an environment whereby the organization will operate and
guide its decisions within the policies and processes set by management or
dictated externally by laws or regulations.
The Process
Foundation
In order for policies to be followed and processes to be successful in an
organization, management must, through whatever means available to it,
provide foundational elements that enable the organization to carry out its
mission and operate within the policies and processes. Organizations
frequently fail to achieve process efficiency and/or control process exceptions
(that is, process results outside of the results desired and/or considered
within tolerances set by policy) when they lack one or more foundational
elements that are required to enable processes. Such items include, but are
8/7/2019 Compliance_Evaluation_Report_121509
20/130
8/7/2019 Compliance_Evaluation_Report_121509
21/130
Compliance Enforcement, Registration and Certification 20
Process Evaluation Report
Process Governance and the Process Foundation Summary Observations
Before we summarize the observations made across the various NERC CERCP functional process
areas it is worthwhile to note our observations regarding the governance and foundational
layers of the NERC process environment.
As a regulatory entity, NERC by its very nature is compelled to maintain an environmentfocused on the creation, compliance, and enforcement of its standards and rules. We
observe that the NERC CERCP program generally has the governance and tone at the top
to be successful with its processes. Our assessment of individual functional areas indicates
that process objectives are typically well known and well understood and that there is
clearly a culture of policy and process adherence.
As part of our analysis we placed NERCs CERCP into appropriate context from the standpointthat NERCs Compliance organization and the purpose, roles, and scope of responsibilities
for that organization has existed in their current state only for a relatively very short period
of time. The relative immaturity of the organization certainly has a bearing on theexpectations for its level of process maturity. For example:
o We observed in our analysis that the organizational structure, and the resulting roles
and responsibilities within that structure, continue to mature and change fairly
frequently as the Compliance area has undergone numerous structural changes within
the past two to four years. Three years ago the Compliance organization shifted from a
Service Organization whose purpose was to provide technical assistance to a
Regulatory Organization whose purpose was to regulate (i.e. compliance
enforcement, in addition to the role of registration and certification). The changes in
scope of responsibilities and assignment of responsibilities within an organization
certainly create challenges when attempting to get to a level of process maturity.
o We observed that the NERC Compliance Director/Manager-level positions are staffed, inmost cases, by personnel that are relatively new to the NERC Compliance organization.
Of the six (6) Director/Manager-level positions reporting up through the Vice President
of Compliance the average length of tenure for the personnel is less than 40 months. If
you filter out the one Manager with significant tenure (i.e. greater than five years), we
find that the average Director/Manager in Compliance has been with the organization
just over two years (i.e. approximately 25 months).
o The newness of staff to their respective positions certainly impacts expectations with
respect to process documentation. Organizational and process problems and
inefficiencies are being addressed by NERC compliance personnel (e.g. Compliance has
stood up 35+ processes in the past two years), but organizational and process best
practices emerge typically once some degree of longevity and critical mass has been
achieved. Procedurally, NERCs Compliance area has achieved a great deal despite their
relatively short existence as an organization.
We observe a number of areas (explained further in subsequent sections of this report)where the NERC CERCP can improve its process foundation. It is our observation that a
number of these areas are a result of the NERC Compliance areas relatively short duration
of existence and immature organizational infrastructure and, therefore, process
infrastructure. For example:
o Both the Rules of Procedures (ROP) and the NERC CMEP Processes and Procedures
Manual can be significantly upgraded to provide a more solid operational foundation. A
8/7/2019 Compliance_Evaluation_Report_121509
22/130
Compliance Enforcement, Registration and Certification 21
Process Evaluation Report
number of enhancements and changes to the ROP are recommended and we outline
those in this report. We also find that the internal CMEP Processes and Procedures are
substantially less mature than the ROP and will require a great deal of attention to reach
a point where they are documented in a manner where the tieback to the ROP is more
obvious, consistent across the Processes and Procedures themselves, and adequate toprovide the ultimate level of management control needed. Generally, the CMEP
Processes and Procedures Manual needs better defined roles and responsibilities,
timelines, and outcome-based measurements.
o While existing systems/processes to measure some results and provide statistics, it is
our observation that tools, systems, and technologies can be leveraged to provide
greater degrees of control and security over both public and private/confidential assets,
to enhance process efficiency and effectiveness, and to assist with the creation of a
continuous process improvement environment. For example, we observe that the
CERCP program generally requires a great deal of monitoring, in large part because
there are a number of reporting requirements that must be met and, therefore, requires
significant levels of rigor in terms of tracking and measuring process execution.
However, with that said, we also observe that the systems and technologies available to
Compliance personnel are largely a collection of non-enterprise level solutions created
by various means (e.g. grassroots) to support the needs of the departments.
Generally speaking, some of these critical monitoring, measuring, reporting systems are
currently not structured as long term solutions built on enterprise-level platforms with
the foundation of IT controls required of such systems.
Overarching Observations and Recommendations
Introduction
During our data gathering process, we used a Process Questionnaire (Appendix II) and othermethods to identify observations in different functional areas and cross-functional areas within
the Compliance Department. In doing so, seven themes emerged that impact the Compliance
Department as a whole, as opposed to a specific team, process, or functional area. These seven
themes are important to the NERC Compliance Departments maturity as a process-driven
organization. We provide an overview of these themes below and address each in further detail
in subsequent sub-sections:
1. We recommend to NERC that a number of changes to the ROP (including its related
appendices). These changes should be implemented to ensure a solid foundation for NERCs
compliance program. We observed a number of issues with the ROP whereby it could be
strengthened by adding to it (address areas of Regional Entity accountability e.g.
Compliance Inquiry process), changing it (address areas where Regional Entities differ inpractice from the ROP as documentede.g. terminology such as guidelines and notices of
violation), or deleting from it (removing redundancies).
2. CMEP Process and Procedures documents should be completed, reviewed, and approved,
including incorporating more defined roles, responsibilities, timelines, and outcomes where
these were found to be lacking. We observed that process documents lacked consistency
and at times did not contain obvious tie-backs to the ROPs by virtue of the process used to
develop them. The individual documents requiring completion, review, and approval are
captured within the detailed recommendations of this report.
8/7/2019 Compliance_Evaluation_Report_121509
23/130
Compliance Enforcement, Registration and Certification 22
Process Evaluation Report
3. We observed that the Compliance Department was not consistently meeting a number of its
internal process goals for timeliness. NERC Compliance indicated to us that, with their
current staff resources, they often had to adjust timelines in order to ensure the quality of
their work. It is our observation, therefore, that staffing levels may not be appropriately
aligned for the workload required. However, it is also our observation that there are othercontributing factors (process inefficiency issues, deficiencies in the process infrastructure,
effort-based metrics) which may also contribute heavily towards NERCs ability to meet its
goals in certain compliance enforcement, registration and certification process areas. The
lack of activity level, effort-based metrics impedes the ability to fully assess whether staffing
levels are adequate relative to workload and/or to assess the degree to which staff levels
are required to meet certain levels of desired timeliness and quality.
4. We observed that problems with the consistency of outputs from Regional Entities (in terms
of the level of quality of outputs and the timeliness of those outputs) and differences in
professional opinion between NERC, the Regional Entities, and FERC impacted the timelines
for the Compliance Departments work and the quantity of work that could be accomplished
(i.e. as measured by the number of enforcement actions processed within establish time
frames). For example, one manager noted that Regional Entities often submitted Notices of
Confirmed Violations that contained errors in dates and judgments that NERC did not find
appropriate, such as classifying an issue as a documentation error rather than a failure to
perform, when the standard required documentation of performance. Another manager
stated that NERC and FERC periodically had different opinions on application of reliability
standards on Compliance Violations Investigations.
5. We observed that processes within some functional areas were not adequately monitored
because there were few interim checkpoints being taken during the overall duration of the
process. For example, the functional areas Analyzing and Reporting Compliance Violation
Information and NERC Compliance Enforcement Authority Responsibilities had no
monitoring in place or planned. We also observed that for those functional areas that were
monitored, there was often not adequate follow up when process deviations were found.
In the functional area Overseeing Compliance Activities of Regional Entities, for example, we
observed that staff was given reminders of the need to meet timeliness goals, but no other
actions were taken when these goals were not met.
6. We observed several processes that involved handling large amounts of information and
documentation. NERC had begun to address these issues through the development of new
technologies, but it was our observation that until these are fully implemented, the volume
of data and documentation will continue to be an impediment to accomplishing the
Compliance Departments goals in a timely manner.
7. We identified some issues with the level of controls over data security, confidentiality andphysical security. Confidential information has been removed from this public version and
has been provided under separate cover to NERC management.
Underlying each of these themes are several overarching observations that we made during our
data gathering and analysis process. As appropriate, we also made recommendations to
address these observations. The following sub-sections provide our observations for each of the
seven key areas followed by our recommendations for each area.
8/7/2019 Compliance_Evaluation_Report_121509
24/130
Compliance Enforcement, Registration and Certification 23
Process Evaluation Report
Recommended Changes to the Rules of Procedure
Observations
The Rules of Procedure and its related appendices make up the foundation of NERCs
compliance program. Without a solidly developed ROP1, NERCs ability to oversee andenforce compliance with reliability standards diminishes. For example, if the ROP does not
include a requirement for Regional Entities to submit draft spot check reports to NERC, then
NERC Compliance has no immediate visibility over whether those spot checks were carried
out as scheduled and in a consistent manner. See Overarching Recommendation ROP-01.
During the process of developing the agreed-upon procedures used as a part of NERCs audit
procedures of Regional Entity compliance programs, Crowe identified almost 50 additions,
deletions, and revisions to the ROP that would improve NERCs ability to carry out its
compliance and enforcement functions. These observations are listed and included as
Appendix III to this report. NERC should review these observations and consider the
applicable changes to the ROP. See Overarching Recommendation ROP-01.
While performing the agreed-upon procedures at one of three Regional Entities, we also
made a number of observations and recommendations related to improvements needed to
the ROP. These observations and recommendations are listed and included as Appendix IV
to this report. NERC should also review these observations and recommendations and
consider the related changes to the ROP. See Overarching Recommendation ROP-01.
Since developing the agreed-upon procedures, we found that NERC issued a number of
Compliance Directives, which NERC expected different parties, particularly Regional Entities,
to follow. Some of these were one-time directives that NERC did not expect to be
performed on an ongoing basis or that NERC expected to possibly change in the future.
However, others were permanent requirements, and not all of these permanent
requirements had been incorporated into the ROP. As a result, there is a higher risk that theone-time directives and/or permanent requirements will not be followed, because they
were not in a single reference location and they may not have been viewed by the Regional
Entities as being required or as important as the ROP. Therefore, we recommend that NERC
consider a formal review of bulletins and Compliance Directives to determine those that
should be permanent requirements of the ROP. For those determined to be permanent
we recommend that NERC incorporate those changes into the ROP. See Overarching
Recommendation ROP-01.
1In this report, where we refer to the ROP, we are also referring to its appendices, including Appendix 4C (the Compliance
Management Enforcement Program or CMEP).
8/7/2019 Compliance_Evaluation_Report_121509
25/130
Compliance Enforcement, Registration and Certification 24
Process Evaluation Report
During this project, we recommended several other changes to the ROP, which are
described below. See Overarching Recommendation ROP-01.
o A section should be added to the CMEP to describe the rules governing the
Compliance Inquiry process. We observed that there was no reference to thisprocess in the ROP, although NERC expected Regional Entities to follow it. See
Recommendation CVI-01 in the Functional Area Evaluation NERC Involvement in
Compliance Inquiries and Compliance Violation Investigations.
o References to Transitional Certification in ROP Appendix 5 should be deleted,
because this process has never been implemented. It should be replaced with the
Provisional Certification process. Note at the time ofour observations, a revision
of Appendix 5 was pending that would incorporate these changes, but it was not yet
approved. See Recommendation CER-01 in the Functional Area Evaluation
Overseeing Certification of Owners, Operators, and Users of the Bulk Power
System.
o NERC Compliance Staff have identified a gap in the RoP and CMEP concerningviolation dismissals. In order to exercise appropriate and expected oversight there
needs to be developed both an internal process for the review of dismissals prior to
approval and appropriate changes to RoP and CMEP to ensure due process for the
industry, regional entities and NERC. We observed that NERC must review Notices
of Confirmed Violations prior to filing a Notice of Penalty with FERC, but not before
this stage. As a result, NERC has spent a great deal of time working with Regional
Entities at this end phase after the Regional Entities had already presented their
findings and had significant points of contact with the violating Registered Entities.
See Recommendation ENF-03 in the Functional Area Evaluation Overseeing
Enforcement Activities of Regional Entities.
When revisions to the ROP are made, other documents, such as implementation plans,
delegation agreements, report templates, documents in the Compliance Departments
Processes and Procedures Manual, training materials, and systems may need to be revised
as well. Once the ROP changes are implemented, NERC should undergo a process to ensure
that other updates are made to related documents and systems as well. See
Recommendation ROP-02.
8/7/2019 Compliance_Evaluation_Report_121509
26/130
Compliance Enforcement, Registration and Certification 25
Process Evaluation Report
Recommendations
ROP-01 Perform an assessment of ROP changes recommended as part of this evaluation
(along with changes that may by otherwise queued up within NERCs own
assessment of the ROP) and then develop and implement a plan to incorporate thefollowing into the Rules of Procedure and related appendices (that is, where there
is concurrence on the need for the change):
Observations on the ROP that Crowe made while developing the Regional
Entity AUPs,
Observations on the ROP that Crowe made while performing the Regional
Entity AUPs,
Required Compliance Directives that are meant to be followed on an
ongoing basis and that have not already been incorporated into the ROP,
and
Recommended changes to the ROP that Crowe identified during the
process evaluation project.
As part of the plan, include a schedule for reviewing the ROP revisions internally,
drafting the revised ROP, obtaining necessary input from outside parties, obtaining
BOTCC approval, and issuing the revised ROP.
ROP-02 Based on the ROP changes that are made, determine what changes need to be
made to other documents, including implementation plans, templates used by
NERC and Regional Entities, the Compliance Departments Policy and Procedure
Manual, and any internal systems (tracking, reporting, etc.) if applicable. We
recommend that NERC Compliance develop and implement a plan to incorporate
necessary changes.
ROP-03 Based upon observations made while executing recommendations ROP-01 andROP-02, we recommend that NERC Compliance should establish and implement a
formal internal change control process whereby changes to the ROP, delegation
agreements, implementation plans, templates, the Compliance Departments
Policy and Procedure Manual, training materials, and any internal systems can be
fully managed, coordinated, and tracked to completion in a consistent manner.
Managing internal change in a consistent, methodical manner is critical towards
assuring consistency between all of these pieces that are ultimately critical
towards the effective implementation of the CERCP. The internal change process
would accommodate externally-driven changes (e.g. changes to the ROP and FERC
orders) and ensure that these changes appropriately permeate throughout the
organization and would also accommodate internal changes to ensure consistencybetween the process assets (process documentation, training assets, templates,
etc.)
Process Documentation Development
Observations
The NERC Compliance Department underwent a concerted effort to document its internal
policies, processes, and procedures in a Processes and Procedures Manual. Each team within
the Department contributed to this effort, in addition to performing its regular duties, and a lot
was accomplished, with over 50 documents drafted. However, we observed that NERC
8/7/2019 Compliance_Evaluation_Report_121509
27/130
Compliance Enforcement, Registration and Certification 26
Process Evaluation Report
Compliance had a fairly substantial amount of progress to make before its process documents
could be considered mature and reflective of a process-driven organization.
Certain compliance-related internal processes that NERC performs had not yet been
documented. Specifically:o No document had been drafted of the CMEP Development and Maintenance
Process, meaning that NERC Compliance did not have a documented tool to guide
the development, coordination, or management of changes to the ROP. (See the
Functional Area Evaluation Compliance Program Planning, Criterion 1.)
o No document had been drafted for Penalty Guidance beyond the Sanction
Guidelines contained in the ROP. As a result, NERC Compliance had no documented
practice for the review of penalties assessed by Regional Entities. In particular,
there was no formal process for ensuring consistent application of penalties across
Regional Entities. This is a key NERC responsibility under the CMEP and Appendix 4B
to the ROP. (See the Functional Area Evaluation Overseeing the Enforcement
Activities of Regional Entities, Criterion 1.)
Because the ROP did not specify how to carry out these processes, documented internal
processes are essential to assure consistent achievement of NERCs compliance goals. See
Recommendation PPM-01.
Of the Processes and Procedures Manual documents that have been drafted, only five -
NPP-CME-301 (Complaint Process); NPP-CME-303 (Evidence Handling Process); NPP-CME-
400 (Observation of RE-led Compliance Audits); NPP-CME-403 (RE Spot Check Process); NPP-
CME-404 (NERC Audit of RE Adherence to the CMEP)have been finalized and reviewed by
the Vice President and Director of Compliance or his designee. We observed that several of
the documents were still in very early draft form, with unresolved details blanked out or
unanswered comments and questions. These included the CMEP Implementation PlanProcess (NPP-CME-201) in the functional area Compliance Program Planning; the Training
Process, (NPP-CME-202) in the cross-functional area Developing and Overseeing the
Compliance Training Program; and, several processes within the functional area Overseeing
Regional Entity Enforcement Programs. As a result, the Compliance Department may not
have been executing the processes in a manner consistent with management s goals. See
Recommendation PPM-02.
We observed that the documents in the Processes and Procedures Manual did not clearly
distinguish between policies, processes, and procedures. Often the terms were used
interchangeably. For example, documents such as the Auditor Training Process, Data
Management, Evaluation, and Analysis Process and the Evidence Handling Process did
not really have a process flow, but were more like policy documents. As noted above,policies form the underlying rules and principles of an organization, while processes provide
a general framework for implementing those policies (what is to be done), and procedures
provide the specific steps for executing the processes (how it is to be done). As a best
practice, NERC Compliance should ensure that its Processes and Procedures Manual follows
the appropriate hierarchy of policies, processes, and procedures. See Recommendation
PPM-03.
Several of the processes did not document well-defined roles and responsibilities (these are
detailed throughout the report). We observed that they often noted that steps were to be
performed by NERC, or they may have assigned general responsibility for a process to a
certain manager, without identifying what team members are responsible for what parts of
8/7/2019 Compliance_Evaluation_Report_121509
28/130
Compliance Enforcement, Registration and Certification 27
Process Evaluation Report
the process. Examples of processes where these types of issues were identified included the
Regional Entity-led Compliance Audit Process (NPP-CME-401), within the functional area
Overseeing Regional Entity Compliance Programs, and the Data Management Evaluation
and Analysis Process within the functional area Analyzing and Reporting Compliance
Information. (See Criterion 3 in the functional area evaluations.) Organizational flexibility iscritical, and generally it is not necessary to assign a specific individual to be responsible for a
specific process step. For example, a process could refer to a designated member of the
Enforcement and Mitigation team, or a Regional Entity Compliance Auditor, or the
Manager or Organization Registration and Certification or his designee. Essentially,
Compliance staff should be aware of what roles they have, or might have, within certain
processes. This is especially important as new staff are hired who would not be as familiar
with NERCs policies, processes, and procedures as the current Compliance Department
staff, many of whom were involved in the actual development of these documents. See
Recommendation PPM-04.
We observed that some processes lacked adequate information on how they were to be
carried out. We found this to be especially true when the process involved reviewing or
observing the work of Regional Entities. For example, we observed that NERCs role while
observing Regional Entity compliance audits and NERCs role in reviewing compliance
violation investigations led by Regional Entities were not well defined. (See Criterion 3 in
the functional area evaluations Overseeing Compliance Activities of Regional Entities and
NERC Involvement in Compliance Inquiries and Compliance Violation Investigations.) In
addition, the enforcement process for when NERC is acting as the Compliance Enforcement
Authority was not fully documented. (See Criterion 1 in the functional area evaluation NERC
Compliance Enforcement Authority Responsibilities.) See Recommendation PPM-05.
We observed that a number of processessuch as the Organization Registration Process
(NPP-CME-100) and the Compliance Violation and Penalty Process (NPP-CME-501)did
not include adequate timelines or other measurable outcomes, other than those required
by the ROP. (See Criterion 6 within the functional area evaluations.) Admittedly, this
timelines are often dependent on receiving information from outside parties who cannot be
held to deadlines not specified in the ROP or other policy directives. However, for purposes
of better measuring and monitoring of the processes, and for communicating process norms
to staff, key measurements should be built into the process documents. See
Recommendation PPM-06.
We observed that many of the processes that we reviewed were not developed with the
ROP as a starting point. Instead, Compliance staff related to us that they developed the
processes based on how they carried out their functions at the time or how the processes
had been historically executed. Staff noted that they kept the ROP requirements in mind
while drafting the documents. However, in instances we observed process documents that
were not based on ROP requirements, such as the process documents related to
Compliance Inquiries, and ROP requirements that did not have an associated process
document prepared, such as NERCs reviews of penalties and sanctions. We did not observe
any obvious or direct conflicts between the process document contents and the ROP
requirements, largely because the ROP was generally non-specific on the way many of
NERCs compliance duties are to be carried out. See Recommendation PPM-07.
As part of the review cycle of this process evaluation report it was noted that there were
inconsistent uses of the term CMEP (i.e. Compliance Monitoring and Enforcement Program).
It was NERCs observation of our initial report draft that the scope of the processes
8/7/2019 Compliance_Evaluation_Report_121509
29/130
Compliance Enforcement, Registration and Certification 28
Process Evaluation Report
contained within this report, and likewise within NERCs Compliance Department, was
broader than CMEP, using the ROPs definition of CMEP (which is identified and defined by
Appendix 4C of the ROP). As an example, NERCs Compliance Department refers to its
processes and procedures as the CMEP Processes and Procedures Manual, when this
document contains items that map back to other sections of the ROP (e.g. registration,certification, confidentiality). Similarly, the use of the term RE was noted to be ambiguous
to the extent that this can refer to both regional entities and registered entities. See
Recommendation PPM-08.
In this report, we made other recommendations to improve the quality of the process
documents themselves. These are specific to certain cross-functional and functional areas, and
for purposes of providing an easy cross reference to these related recommendations, these
consist of the following recommendations within the sections listed:
o Recommendations TRA-01 and TRA-02 within the Cross-Functional Area Evaluation
Developing and Overseeing the Compliance Training Program,
o Recommendation PRO-01 within the Cross-Functional Area Evaluation ProcessingReliability Standards Violations,
o Recommendations IMP-01 and IMP-02 in the Functional Area Evaluation
Compliance Program Planning,
o Recommendations REG-01 and REG-02 in the Functional Area Evaluation
Overseeing Registration of Users, Owners, and Operators of the Bulk Power
System,
o Recommendations CER-02, CER-04, and CER-05 in the Functional Area Evaluation
Overseeing Certification of Users, Owners, and Operators of the Bulk Power
System,
o Recommendations COM-01, COM-03, COM-04, COM-05, and COM-06 in the
Functional Area Evaluation Overseeing Compliance Activities of Regional Entities,
o Recommendations ENF-01 and ENF-02 in the Functional Area Evaluation
Overseeing Enforcement Activities of Regional Entities,
o Recommendation REP-03 in the Functional Area Evaluation Analyzing and
Reporting Compliance Information,
o Recommendations REV-01 and REV-03 in the Functional Area Evaluation
Conducting Reviews of Regional Entities Compliance and Enforcement Programs,
o Recommendations CVI-02 and CVI-03 in the Functional Area Evaluation NERC
Involvement in Compliance Inquiries and Compliance Violation Investigations, and
o Recommendations CEA-01, CEA-02, and CEA-04 in the Functional Area Evaluation
NERC Compliance Enforcement Authority Responsibilities.
Recommendations
PPM-01 Develop internal process documents for the CMEP Development and Maintenance
Process and the Penalty Guidance Process. Include procedures for cross-regional
comparisons in the Penalty Guidance Process. Develop a due date for completion
of these drafts.
8/7/2019 Compliance_Evaluation_Report_121509
30/130
Compliance Enforcement, Registration and Certification 29
Process Evaluation Report
PPM-02 Finalize all internal process documents and have them reviewed by the
appropriate Compliance team manager and by the Vice President and Director of
Compliance or a designee. Reviewers of the process documents should ensure
that the Recommendations PPM-04, PPM-05, PPM-06, and all functional area-
specific recommendations made in this report to improve the quality of theprocess documentation are incorporated. All processes should be finalized and
reviewed before FERC begins requesting information for its audit of NERC.
PPM-03 In the internal Processes and Procedures Manual documents, classify the policies,
processes, and procedures into a hierarchy. Note that for some purposes, policies
- and sometimes even processes - may be the underlying ROP or FERC orders,
which would not need to be repeated in their entirety within the documents.
PPM-04 We noted as a recommendation in many of the functional area evaluations, that
NERC should consider the definition of roles and responsibilities within its process
documents. As such, there are many references in the functional area evaluations
to this recommendation (i.e. Recommendation Id PPM-04). We recommend that
NERC should consider designating who is responsible for executing each step
within the related processes and that these designations should continue to be
tied to roles within the organization, as opposed to specific names of individuals.
As individuals are frequently added to the organization, leave the organization, or
change roles within the organization, best practices dictate that designating
responsibilities tied to roles eliminates the need to maintain process documents as
people change.
PPM-05 Where processes were found not to be clear or well-defined (see references to this
recommendation, that is, Recommendation Id PPM-05 in the functional area
evaluations), we recommend that NERC Compliance specify in greater detail whatsteps are to be followed within the processes. In keeping with Recommendation
PPM-04, designate who (by role) is responsible for these process steps.
PPM-06 Where noted as an issue in the functional area evaluations (see references to this
Recommendation, i.e. PPM-06), we recommend that NERC Compliance consider
identifying key milestones (perhaps in many cases, more detailed milestones)
wit