Information Technology Report

Trey FeltonManager, IT Service Delivery

October 2011

ERCOT Public

2 ERCOT PublicOctober 18, 2011

Highlights

Service Availability: Market Operations IT systems met all SLA targets Market Data Transparency IT systems met all SLA targets

Retail Market IT systems missed one SLA target (Retail Processing Business Hours)

Retail Market IT system outage with impact to SLA:

•Data Center migration to the new T3 site resulted in an incorrect Internet Protocol (IP) address configuration (9/26)

• Improperly routed traffic caused messages to queue and slow processing which resulted in 60 minutes of delayed transactions and slow MarkeTrak performance

• Correct IP configuration change was applied to resolve the issue

Retail Market unplanned outages:

•Two parallel processes encountered contention that affected Electronic Data Interchange (EDI) (9/6)

• Retail transaction processing was unavailable for 28 minutes (outside of business hours)

• Short-term resolution: Restarted one of the processes in contention

• Long-term: Configure processes to run on different systems and eliminate contention. The fix will be implemented with the move to new data center by 10/30

•Hardware failure caused outage of Enterprise Data Warehouse (EDW) (9/26)

• Caused Get Reports and other components of TML and MIS to be unavailable for 75 minutes

• Short-term: Manually took component out of service and used redundant component (NIC)

• Long-term: Replaced hardware to resolve issue

3

Highlights Cont’d

Planned Outages:

• Weekend Retail and Market Operations maintenance activity (9/11 and 9/25)

• Outages lasted 1,471 minutes which is within the 1,800 minutes allowed via SLA

SLA Exception Request - October

• An extended retail outage requested through RMS and COPS to execute retail system moves to the new T3 Data Center in Taylor (10/22, 10/23, 10/29, 10/30)

ERCOT PublicOctober 18, 2011

4

SLA Exception Request

• ERCOT requests a 36-hour Release window be granted for October 22-23 (Saturday noon through Sunday midnight), and again for the same period on October 29-30 

• These two windows are necessary to complete the migration (and verification) of Production systems as part of the Data Center Migration Project 

• There are no Release windows outlined in the SLA for 2011 due to Nodal Stabilization– Currently there are only Sunday maintenance windows, which

are insufficient for the volume of activities with the Data Center Migration 

TDTWGOctober 2011

5 ERCOT Public

2011 Net Service Availability (Retail and Market Ops)

October 2011

2011 Net Service AvailabilityYear to Date

Retail & Market Operations

Transaction Processing SLA Target: - Business Hours (BH): 99.9%- Off Business Hours (Off BH): 99%Texas Market Link (TML) SLA Target: 99%

MarkeTrak SLA Target: 98%TML Report Explorer SLA Target: 99%Retail API SLA Target: 99%Congestion Revenue Rights SLA Target: 98%Market Information System SLA Target: 99%

99%

98%

97%

94%

95%

96%

93%

92% and below

100%

Transaction Processing

(BH)

TML MarkeTrak Retail API

TML Report Explorer

99.95% 99.88%99.92% 99.85% 99.80%

CRR

99.96%

Transaction Processing ( Off BH)

99.91%

MIS

99.87%

6

2011 Data Center Availability

ERCOT PublicOctober 2011

2011 Data Center AvailabilityYear to Date

Outage prevented due to Tier 3 redundancy

Target – 99.982%

99.99%

99.98%

99.97%

99.94%

99.95%

99.96%

99.93%

99.92%

100.0%

Taylor 1 Austin Bastrop

100% 100%100%

Taylor 2

100%

7 ERCOT Public

September 2011 Net Service Availability

October 2011

99%

98%

97%

94%

95%

96%

93%

92% and below

100%

September 2011 Net Service Availability

Transaction Processing

(BH)

TML MarkeTrak Retail API

TML Report Explorer

99.90% 100%100% 100% 99.82%

CRR

100%

Retail & Market Operations

Transaction Processing ( Off BH)

99.61%

MIS

99.91%

Transaction Processing SLA Target: - Business Hours (BH): 99.9%- Off Business Hours (Off BH): 99%Texas Market Link (TML) SLA Target: 99%

MarkeTrak SLA Target: 98%TML Report Explorer SLA Target: 99%Retail API SLA Target: 99%Congestion Revenue Rights SLA Target: 98%Market Information System SLA Target: 99%

8

September 2011 Data Center Power Availability

ERCOT PublicOctober 2011

Target – 99.982%

99.99%

99.98%

99.97%

99.94%

99.95%

99.96%

99.93%

99.92%

100.0%

Taylor 1 Austin Bastrop

100% 100%100%

Taylor 2

100%

September 2011 Data Center Availability

9

YTD Availability – Retail Market IT Services

ERCOT PublicOctober 2011

Retail Transaction Processing (Off Business Hours)

99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

Jan Feb MayAprMar JulyJune Aug Sep

Texas Market Link

Target – 99%99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

Jan Feb MayAprMar

YTD – 99.92%

JulyJune Aug Sep

MarkeTrak

Target – 98%

99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

Jan Feb MayAprMar

YTD – 99.88%

JulyJune Aug Sep

Retail Transaction Processing (Business Hours)

Target – 99.9%

99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

Jan Feb MayAprMar

YTD – 99.91%

June July Aug Sep

10

YTD Availability – Market Operations

ERCOT PublicOctober 2011

Retail API

Target – 99%99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

Jan Feb MayAprMar

YTD – 99.85%

JulyJune Aug Sep

TML Report Explorer

Target – 99%99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

Jan Feb MayAprMar

YTD – 99.80%

JulyJune Aug Sep

Market Information System

Target – 99%99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

Jan Feb MayAprMar

YTD – 99.87%

JulyJune Aug Sep

Congestion Revenue Rights

Target – 98%

99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

Jan Feb MayAprMar

YTD – 99.96%

JulyJune Aug Sep

11

Retail Transaction Processing Availability

ERCOT PublicOctober 18, 2011

Retail Transaction Processing Availability (Business Hours)

September 2011 Retail Transaction Processing Availability Summary (Business Hours)

Target – 99.9%

99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

September 2011 Retail Transaction Processing Availability (BH) – 99.61%

9/26 (60 minutes) – Incorrect IP address configuration during weekend maintenance led to delayed transaction processing

12

Retail Transaction Processing Availability

ERCOT PublicOctober 18, 2011

Retail Transaction Processing Availability (Off Business Hours)

September 2011 Retail Transaction Processing Availability Summary(Off Business Hours)

Target – 99%99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

September 2011 Retail Transaction Processing Availability (Off BH) – 99.9%

9/6 (28 minutes) – Process contention from known software bug caused an outage of Electronic Data Interchange

13 ERCOT Public

TML Report Explorer Availability

October 18, 2011

September 2011 TML Report Explorer Availability Summary

Target – 99%

September 2011 TML Report Explorer Availability – 99.82%

99.00%

98.00%

97.00%

94.00%

95.00%

96.00%

93.00%

92.00%

100.0%

TML Report Explorer Availability

9/26 (75 minutes) – Hardware failure (Network Interface Card) led to intermittent outage of TML Report Explorer

14

Release Management Metrics (Releases)

ERCOT PublicOctober 2011

Awaiting slide

15

Release Management Metrics (Changes)

ERCOT PublicOctober 2011

Awaiting slide

16

ERCOT Public Website Metrics

ERCOT PublicOctober 2011

Awaiting slide

17

ERCOT Public Website Metrics

ERCOT PublicOctober 2011

Awaiting slide