Meter Data Management Presentation

The Hidden Challenge in Meter Data

Management: What You Don't Know Will Hurt

You!

An InformationWeek Webcast

Sponsored by

Webcast Logistics

Today’s Presenters

Jill Feblowitz, Vice President, Utilities and Oil and Gas,

IDC Energy Insights

Kevin Brown, Chief Architect,

IBM Informix Database

Dealing with Smart Grid Data Jill Feblowitz, Vice President, IDC Energy Insights

© IDC Energy Insights

Agenda

Smart Meter Outlook

Management Challenges

IT Challenges

The Way Forward

Recommendations


It’s All About More Interaction


Smart Meter Outlook

A Decade of Global Growth in Smart Metering

2001 Enel begins AMI deployment

2002 First AMI deployments begin in North America

2005 North American utilities begin large-scale AMI deployments

2009 North America overtakes Europe in Smart Meter shipments

2015 AP and LA will drive the global smart meter marketplace


Smart Meter Outlook

Data Deluge

85 million installed meters by 2015 according to the IDC Energy Insights Smart Meter Tracker

Each meter has the potential of producing a vast amount of interval data: 12 consumption events for a year vs. 8760 billing events with a 1 hour interval

Not to mention other on-going communications and transactions


Smart Grid Outlook

IT Spending on Smart Grid, 2011 Hardware Services

Software It is not just about smart meters at the home – it‟s about making the grid smarter, too. Smart meters at the

transformer Intelligence

electronic devices.



Change in Business Process… Meter to Cash Before Smart Meters

Meter (residential)

Customer information System

(CIS)

Customer Relationship

Management (CRM)

Data Collection Bill Calc Fulfillment

Bill Print

Bill Mail

Electronic Bill Payment and Presentment

(EBPP)

Payment & Collections

EBPP

Lock Box

Credit and Collections

Kiosk, Local Office,

Mail

Meter Reading Accounting Accounting Accounting

Customer Service

Meter data management

Complex Billing

Advanced Meter (C&I)

60 to 90 days


Payment & Collections

Smart Phone

Web Portal

EBPP

Lock Box

Credit and Collections

Kiosk, Local Office,

Mail

Management Challenges (and Opportunities)

Change in Business Process… Meter to Cash Future

Smart Meter CIS

CRM

Data Collection Bill Calc Fulfillment

Smart Phone

Web Portal

In-home Display

EBPP

Bill Print

Bill Mail

Network Operating Center - Data

Analysis

Accounting Accounting Accounting

Customer Service

MDM

Advanced Meter (C&I)

.

7 to 90 days

Data Transport and Assessment

Advanced Metering

Infrastructure

Customer Sees The Data



The business case must be supported.

Type of Benefit Source of Benefit

Revenue Assurance • Remote connect/disconnect

• Theft and tamper detection

• Analysis of billing data to detect unbilled accounts

• Pre-payment

Operational

Efficiency

• Reduced truck rolls for connect/disconnect, high bill

complaints

• More efficient deployment of workforce in outage

based on data

Deferred Capital

Investment

• Support for demand response in capacity constrained

areas

• Prioritization of equipment replacement

Increased Reliability • Predictive analytics applied to condition-based

monitoring

• Automated switching routines to minimize outage

impact



There Much More Left to Do with the Meter Data

Source: IDC Energy Insights, Utility CIO Survey, 2010

n=26

n=26



Supporting New Pricing, Services

Still to come…..fast charge rates?

Expanded offerings: energy efficiency, demand response, green energy rate

New Pricing: Time-based, critical peak pricing

New Relationships: “Prosumer” and net metering for PV

Tools to Build Relationship and Awareness: carbon footprint, energy efficiency and savings

Pre-pay and budget notification


IT Challenges

Data explosion calls for data storage and handling and much more.

Managing Meter Data

Managing T&D Grid Data Achieving acceptable levels of production for

billing and customer presentment

Determining the balance between centralized or

distributed (device) computing

Optimizing performance and utilization of storage

specific to the workload

Optimizing performance and utilization of storage

specific to the workload

Making the right data available for production

(billing and customer presentment) and analytics Making the right data available for operations

and analytics

Retaining and archiving billing data to meet

regulatory requirements and satisfy business

continuity

Securing the smart grid telecommunications

network from incursion by hackers

Managing data, alerting about data irregularities,

and resolving inconsistencies

Managing data, alerting about data irregularities,

and resolving inconsistencies

Protecting privacy of customer data

Integrating old and new communication

infrastructure to support secure data

communications

Establishing consistent data synchronization, data

models, and protocols

Establishing consistent data synchronization, data

models, and protocols

Securing the AMI network from incursion by hackers Minimizing network traffic with high device data production

given new devices that produce high data volumes


IT Challenges

Examples of Volumes in our Study

Number of

Meters Currently

Deployed

(million)

Intervals Frequency of

Data Collection

(per day)

Data Processed

on a Daily Basis

(gigabytes)

Rention in Active

Database

(years)

Utility A 1.32 15 minute 3 times 4.752 1.5

Utility B 1.40 Hourly 2 times 12 3.0

Utility C 0.70 Hourly 1 time 4 2.0

Utility D 2.00 15 minute 6 times 70 1.1

“Utilities are not accustomed to managing or processing this much data, I mean there‟s been no reason to…Probably the largest amount of data they work with has been in the billing world and maybe some GIS, but this volume is much larger, especially on a daily basis, so it really pushes on how well you architect something and you‟re using the right tools in your systems and your databases are properly architected."


IT Challenges

Details, details, details…examples Time to process and speed of processing

– Re-interrogating the meters to get missing reads and avoid duplication

– “ If a system goes down or the network is down and you don’t get it resolved quickly, then when you start piling up two days’ worth of data, then it becomes a challenge because it takes longer.”

– Service levels mandated by PUCs or service level agreements between IT and the business

8:00 AM presentment of previous day’s usage

Need to respond within four hours if system goes down

Target of first time read success and presentation at SLA of 99.5%

– “Where the bigger challenge is going to be is managing flows on the system if everybody wants to do load control simultaneously, that’s going to be a little more challenging to us and it’s going to be interesting to see how those challenges work out.”

– “We’ve had everything on the MDM from a bad index to a bad spot on the disk space to we needed more horsepower.”

Ease of access

– “If you wanted to see a month’s worth of data on the meter, you’d have to go through 30 different files. It’s not very user friendly.”

– “We have to go through essentially 60 millions rows of data per day.”


Looking Forward

Approaches

Operational data stores

– To bring in other data

– To off load data so there is no impact on “production” of the MDM

Changed business process

Developed retention and archiving protocols

“Changed how EDI moves data from one platform to another”

“Changed database schemas on mainframe”

Re-partitioning – “low hanging fruit”

Adding servers and storage. Cluster servers.

Considering high end compression techniques

“[We have been working with meter data for about 3 years, and in that time, we have] “changed out systems two or three times.”

“We‟ve been pretty aggressive in providing a quality system to this market, so we have already taken the steps to get what we think is the right level of hardware and the software products…We‟ve spent some money, unfortunately.”


Looking Forward

Managing the Information: Wish List

Shorten processing time to meet regulatory and internal SLAs

Build analytics into the database for faster querying

Reduce hardware and software costs related to server, storage and RDBMS.

“Loss less” storage is useful in areas where time series data must keep its fidelity, such as predictive maintenance


Recommendations

Develop a data retention policy and investigate what needs to be kept online and for how long. Do not forget to include customer needs for presentment. Start early to evaluate how others in your organization (load research, capital planning, etc.) will access data.

Ask vendors for a “proof of concept”. Most vendors are willing to help by running test data sets using their technology.

Start by understand what current and future requirements for processing speed, storage and data access will be for your company and how these demands will ramp up over time.

Do your due diligence. Based on scenarios, investigate your options for processing and storage and the total cost of ownership associated with these. Do not assume that by adding more servers and storage is the most cost effective approach.

© 2011 IBM Corporation

Information Management

Powering Large Volumes of Meter Data with Informix Kevin Brown, IBM Informix Chief Architect [email protected]

mailto:[email protected]



22

Changing Storage Requirements

Monthly Meter

Reads

Daily Meter

Reads 15 Minute Meter

Reads

350.4B

3.65B

120M

Changing Workloads For 10 Million Smart Meters:

Today – Each meter read once per month

Very soon – Each meter read once every 15 minutes

Regulations – Need to keep data on line for 3 years (PUC) and, perhaps, save for 7 years

# of Records

Per Year for

10M meters

Smart Meter Interval Data

Frequency

of reads



23

Keeping up with Smart Meter Data

Large amounts of data causes problems in 2 areas:

1) Storage management • Large storage = Expensive

• Cumbersome to maintain

– Requires sophisticated partitioning schemes

– Reorganization often required, which leads to downtime

2) Query performance • Compliance Reports must be completed before the end of each day

• Customer portal queries must be handled in a timely manner

• Customer billing must be completed each day



24

Informix TimeSeries: The Solution for Managing Meter Data

Time Series:

– A logically connected set of records ordered by time

Performance – Extremely fast data access

• Up to 60 times faster than competition

– Handles operations hard or impossible to do in standard SQL

Space Savings – Saves at least 50% over standard relational layout

Toolkit approach – Develop algorithms that run directly in the database

Easier – Extremely low maintenance



25

1 1-1-11 12:00 Value 1 Value 2 …….. Value N



… … … … …….. …




… … … … …….. …

Typical Relational Schema for Smart Meters Data

Smart_Meters Table

Index

• Each row contains exactly one record = billions of rows in the table

• Additional indexes are required for efficient lookups

• Data is appended to the end of the table as it arrives

• Meter ID’s stored in every record

• No concept of a missing row

Table

Gro

ws

KWH Voltage ColN Time Meter_id



26

Same Table using an Informix TimeSeries Schema (logical view)

1 [(1-1-11 12:00, value 1, value 2, …, value N), (1-1-11 12:15, value 1, value 2, …, value N), …]




… …

• Each row contains a growing set of records = one row per meter

• Data append to end of a row rather than to the end of the table

• Meter IDs stored once rather than with every record

• Data is clustered by meter id and sorted by time on disk

• Missing values take no disk space, missing interval reads take 2 bytes

Smart_Meters Table

Table grows

Meter_id Series



27

Virtual Table Interface makes Time Series data appear Relational

mtr_id Series (int) timeseries(mtr_data)

SM_vt

1 Tue Value 1

1 Wed Value 1

... ...

3 Mon Value 1

3 Tue Value 1

3 Wed Value 1 ... ... ... ...

1 Mon Value 1 Value 2

col_1 col_2 date mtr_id

Smart_meter

...

...

...

...

...

...

...

...

TimeSeries Table TimeSeries Virtual Table

Execute procedure tscreatevirtualtable

[(Mon, v1, ...)(Tue,v1…)]

(„SM_vt‟, „Smart_meter‟); 8

7

6

5

4

3

2

1

[(Mon, v1, ...)(Tue,v1…)]

[(Mon, v1, ...)(Tue,v1…)]

[(Mon, v1, ...)(Tue,v1…)]

[(Mon, v1, ...)(Tue,v1…)]

[(Mon, v1, ...)(Tue,v1…)]

[(Mon, v1, ...)(Tue,v1…)]

[(Mon, v1, ...)(Tue,v1…)]

...

Value 2

Value 2

Value 2

Value 2

Value 2

...



28

100 Million Smart Meter Benchmark: Goals

1. Measure processing times for data collected over a 31 day period for:

100 million meters at 30-minute intervals in an 8 hour day.

2. Demonstrate consistent processing times over the 31 day period.

3. Demonstrate linear storage growth of data stored over the 31 day period.

4. Complete one day’s billing cycle while simultaneously processing and loading meter data

for 100M within an 8 hour time period.

5. Demonstrate all processing can be done using a low-cost combination of commercially

available hardware, storage, and software.



29

Operations performed each day

– Load interval and register data for 100M meters at 30 minute intervals

(49 records/day/meter)

• 49 records/day/meter * 100M = 4.9 Billion records/day

– Perform VEE on the data (validation/estimation/editing)

– Run a daily billing cycle on 6% of the meters

– Gather results for 31 days

100 Million Smart Meter Benchmark: Operations

Perform Validation, Estimation & Editing

Run daily billing cycle

Gather Results

Load & Register Data (100 million meters)

Processed over 30 minute intervals over 31 Days

Pro

ce

ssin

g



30

100 Million Smart Meter Benchmark: Components

Sto

rage

Are

a N

etw

ork

(SA

N)

IBM Power P750 32 cores (3.5 GHz) – 16 active 500 GB RAM 1 GB LAN Fiber dual port adapter - 1 active 2 X 8GB FC dual port adapter - 4 active ports of storage

XIV Storage System 15 X 2TB storage 6 X 6 FC connections @ 4GB

4 X 8GB 6 X 4GB

Software Stack

Informix v11.70.xC3 with TimeSeries version 5.0 AMT-Sybex Affinity Meterflow Meter AIX v7.1

Hardware Stack

Upstream Management Systems

Knowledge Applications

Data Management

MDM, DMS, NMS

IBM System P Series &

Storage

AIX v7.1

Informix 11.70

Monitor & Admin

AMT-Sybex Affinity

Meterflow



31

An “end to end” run of 100 Million Meters at 30 minute intervals was performed for 31 days of data

The result: all data was prepared, loaded, validated as well as a billing cycle run in less than 8 hours

The average time to do these operations remained consistent over the 31 days

– Performance remained constant even as storage increase

The billing cycle completed in less than 5 hours and ran concurrently

100 Million Smart Meter Benchmark: Load Results

Process Avg. Elapsed Time Avg. Throughput Rate

Preparation and Technical

Verification

2 hrs 10 min 628,205 records/sec

Data Load 3 hrs 14 min 420,962 records/sec

Validation, Estimation, and Editing

(VEE)

2 hrs 11 min 623,409 records/sec

Total Time: 7 hours and 35 minutes!



32

Tota

l Tim

e -

Min

ute

s

No. of Days

Total Process Time over 31 Days 100 Million Meters @ 30 minute intervals

Load Time over 31 Days 100 Million Meters @ 30 minute intervals

Tota

l Tim

e -

Min

ute

s

No. of Days

Load Performance: Storage Comparison over Time D

isk

Spac

e in

TB

No. of Days

Storage in TB over 31 Days 100 Million Meters @ 30 minute intervals



33

Summary: Informix is the Answer for Smart Meter Data

1) The enormous volume of smart meter data can be

overwhelming

2) Most smart meter data is time series data

3) Not all database servers are equal, choose one that

handles relational and time series queries equally well

For more information on our 100M Smart

Meter Benchmark: http://bit.ly/pfu2RW

http://bit.ly/pfu2RW



34

(Kevin Brown – [email protected])

(Jill Feblowitz - [email protected])






35

Meter Data Management Presentation

Documents

Transcript of Meter Data Management Presentation