Nodal EDW Project Reporting Requirements Content Update

Nodal EDW Project Reporting Requirements Content Update

EDW Project Team

Sept. 24, 2007

2Nodal EDW Archive and Reporting Requirements04/19/23

Contents

The purpose of this presentation is to provide information on the structure of the “Nodal EDW Archive & Reporting

Requirements” document and an update on the progress of the information gathering.

CONTENTS• Review of EDW archival & reporting criteria

• EDW Data Archival and Reporting Requirements Content Update

• EDW Business Agreement on Deliverables Update


EDW Data Archival and Reporting Criteria (EIS systems)

• Criteria for archiving Source System Data– Source System data must be replicated to the ODS system when:

1. The source system does not maintain at least 4 years of complete history online.2. The data is required for the EDW reporting function to ERCOT business or any external

entity (Market, IMM, PUCT, etc.).3. The source system does not capture all changes to data throughout the operating day (data

gets overwritten).

• Criteria for using EIS systems for reporting– Reporting will be delivered from the EIS systems (ODS or CDW) when:

1. The report information is not needed by the recipient within 9 hours of creation of the data. (This allows time for the data to be archived and validated prior to reporting).

– Additionally one of the following should apply1. Report must be generated from historical data no available from the source system. 2. The volume of the report information is greater than 1,000 rows per report.3. The generation of the report involves resource intensive processing of the data.4. The report information must be integrated from two or more source systems.5. The report information requires significant alteration, aggregation, or de-normalization of

the data6. Creation of the report could/will adversely affect the performance or stability of the source

system.

• Criteria for archiving files in the Market Information Repository (MIR)– Files will be archived to the EIS MIR system for:

1. All reports, extracts, web service files, and operational data files sent to the MIS for consumption by the Market.

2. All data sets that have a requirement to be capable of reposting at a future time.


Preliminary Source System Replication Requirements

• NMMS– Data Content – No replication of database tables (NMMS keeping history)

Daily archiving of all output files into MIR» NOMCR Header files » NOMCR Status Change files » Incremental CIM XML files » Full CIM XML Model files » Contingency file » SPS/RAP/MAP/etc. files » Settlement Point file » Electrical Bus list file » As Built Report files » NOMCR attachments » Methodology files

» Ratings changes files

– SLA – TBD (Expected to be within 24 hours of file generation)– Schedule – Replication to begin 3/20/2008– Method – Tibco bus

• MMS– Data Content –

• Dimensional Data – Replicate all tables• Transactional Data – Still reviewing - currently 5 min. periodicity is minimum. (TBD)

– SLA – Within 9 hours of data generation– Schedule – Replication to begin 2/21/2008– Method – Standard Goldengate replication


Preliminary Source System Replication Requirements

• EMS– Data Content –

• Dimensional Data – Replicate all tables• Transactional Data - State Estimator & Telemetry Summary Data,

additional data TBD – SLA – Within 9 hours of data generation– Schedule – Partial replication to begin 1/17/2008, complete replication to

begin 9/15/2008– Method – E-Terra Archival from EMS to EDB (Oracle), Standard

Goldengate replication from EDB

• PI– Data Content –

• Dimensional Data – TBD (will be driven by reporting requirements)• Transactional Data – Telemetry Data @ 15 min interval, additional

data TBD– SLA – Within 9 hours of data generation– Schedule – Partial replication to begin 1/17/2008, complete replication to

begin 10/15/2008– Method – Investigating alternatives (ODBC push from PI system)


Source System Replication Requirements

• CRR– Data Content –

• Dimensional Data – Replicate all tables• Transactional Data – Replicate all tables


• SIEBEL– Data Content –


– SLA – Within 9 hours of data generation– Schedule – Replication currently in production, additional changes expected 1/30/2007– Method – Standard Goldengate replication

• LODESTAR– Data Content –



• CMM– Data Content –




Preliminary Reporting Requirements (EIS Systems only)

• NMMS– Market Reports – No reports generated from EIS systems.

– Market Extracts – No extracts generated from EIS systems.

– Market Web Services – No EIS web services of NMMS data provided from EIS systems.

• MMS– Market Reports –

• 17 (21) Reports identified for nodal market, meeting with business to provide details for requirements.

• 7 Outage Scheduler reports identified for nodal market, meeting with business to provide details for requirements.

• 6 Reports possibly transitioning from zonal market

– Market Extracts –

• 10 Extracts identified for nodal market, meeting with business to provide details for requirements.

• 1 Outage Scheduler extract identified for nodal market, meeting with business to provide details for requirements.

• 7 Extracts possibly transitioning from zonal market

– Market Web Services – No EIS web services of MMS data provided from EIS systems.



• EMS– Market Reports –


• 0 Reports transitioning from zonal market


• 18 (17) Extracts identified for nodal market, meeting with business to provide details for requirements.

• 5 Extracts possibly transitioning from zonal market

– Market Web Services – No EIS web services of EMS data provided from EIS systems.

• PI– Market Reports –

• 4 Reports identified for nodal market, meeting with business to provide details for requirements.




• 0 Extracts transitioning from zonal market

– Market Web Services – No EIS web services of PI data provided from EIS systems.



• CRR– Market Reports –






– Market Web Services – No EIS web services of CRR data provided from EIS systems.

• SIEBEL– Market Reports –


• 1 (0) Reports transitioning from zonal market




– Market Web Services – No EIS web services of SIEBEL data provided from EIS systems.



• LODESTAR– Market Reports –


• 21 (1) Report transitioning from zonal market (13 are for ERCOT Finance)



• 20 (7) Extracts transitioning from zonal market

– Market Web Services – 17 (15) EIS web services of Lodestar data provided from EIS systems.

• CMM– Market Reports –






– Market Web Services – No EIS web services of CMM data provided from EIS systems.


Document Archival Requirements for MIR

• NMMS– 12 daily replication files archived in MIR

• MMS– 44 report files to be archived in MIR (34)

• EMS– 54 report files to be archived in MIR (47)

• PI– 0 report files archived in MIR

• CRR– 9 report files archived in MIR (11)

• SIEBEL– 4 report files archived in MIR (2)

• LODESTAR– 64 report files archived in MIR (51)

• CMM– 3 report files archived in MIR (0)


Data Archival and Reporting Requirements – Tracking Docs

The following documents are used to organize and track all items associated with gathering the EDW data archiving and reporting requirements. These documents will be updated frequently until the requirement doc is approved by TPTF and baselined.

• Nodal EDW Business Agreement Matrix 08/30/2007.xls– A listing of all nodal functional areas with there respective business owners and nodal project

teams that provides the basis of agreement between the EDW project, Nodal source system projects, and the ERCOT business groups.

• Nodal Data Services Master List v 2.9.xls– A comprehensive list of all zonal, retail, and nodal data services provided by ERCOT to market

participants. The purpose of this listing is to better ensure the Nodal EDW, MIS, and EIP teams deliver the expected data services from the perspectives of the Nodal Program, ERCOT Business, and Market Participants.

*These files can be accessed (due to be posted by 9/25/2007) at:

http://nodal.ercot.com/docs/pd/edw/index.html#req


Questions


Supplemental slides of backup info


Data Archival and Reporting Requirements Structure

• Source System Replication Requirements (ODS)• Source System

– Data content to be replicated» Dimensional Data» Transactional Data (specify granularity)

– Replication SLAs (driven by reporting requirements)– Replication timelines (driven by source system availability)– Replication method

• Reporting Requirements• Source System

– Market Reports (CDW)» New reports for nodal market» Reports transitioning from zonal market

– Market Extracts (ODS)» New extracts for nodal market» Extracts transitioning from zonal market

– Market EIS Web Services (ODS/CDW)

• Document Archival Requirements (MIR)• Source System


Data Archival and Reporting Requirements

The “Data Archival and Reporting Requirements” document will describe data archival to the following level of detail:

• Source System Replication Requirements (reflects business/Nodal Project agreement for the delivery of replication requirements)– Data replication process– Data set descriptions – Transactional Data granularity/periodicity– Replication SLAs– Replication timelines


Data Archival and Reporting Requirements

• New Market Reports– Report Name– Report Data Set– Data Source– Generating System– Generating Frequency (timeliness SLAs)– Archival System– Format(s)– Display or Delivery System– ERCOT Report owner– Intended Recipients– Report Definition

• New Market Extracts– Extract Name – Extract Data Set– Data Source– Generating System– Generating Frequency (timeliness SLAs)– Archival System– Format(s)– Display or Delivery System– ERCOT Extract owner– Intended Recipients– Extract Definition

• New Market Web Services– Web Service name – Web Service possible data set– Data Source– Generating System– Archival System– Format– Display or Delivery System– ERCOT Web Service owner– Intended Recipient– Web Service Definition

• Market Extracts and Reports transitioning from Zonal

– Extract/Report Name – Extract/Report Data Set– Data Source– Generating System– Generating Frequency (timeliness SLAs)– Archival System– Format(s)– Display or Delivery System– ERCOT Extract/Report owner– Intended Recipients– Extract/Report cut over schedule– Extract/Report Definition

The “Data Archival and Reporting Requirements” document will describe data reporting to the following level of detail:

Market Data Products (reflects business/Nodal Project agreement for the delivery of reporting requirements)

Nodal EDW Project Reporting Requirements Content Update

Documents

Transcript of Nodal EDW Project Reporting Requirements Content Update