Post on 30-May-2020
SESSION #301
Ontario Canada’s IESO and Smart
Metering System Experiences Przemek Tomczak, Director, Smart Metering,
Independent Electricity System Operator
Ontario At a Glance
Population 13.5 million
(38.7 % of Canada)
Nominal GDP
(2012)
$673 billion
(37% of Canada)
Installed Capacity 33,771 MW
Record Summer Peak 27,005 MW
Record Winter Peak 24,979 MW
Total Annual domestic Energy use
140.7 TWh (2013)
Low Volume Retail Customers
4.8 million
(all with smart meters)
Ontario Import Capability
6,000 MW
Transmission Lines 30,000 km
The IESO is the reliability coordinator for Ontario and works closely with other jurisdictions to ensure energy adequacy across North America.
Ontario’s Smart Metering System
3
Ontario’s Smart Metering System
• One of the largest shared services for utilities
in the world, supporting 72 distribution
utilities and over 4 million customers
• Processes and stores over 100 million meter read intervals per day
• Delivers billing quantity data to LDCs for billing
customers on “Time-of-Use” rates
Security and Privacy
Slide Title (Helvetica 36pt) Service Delivery Points
Residential: 3.5m, Small General Service: .4m
AMI Supported Sensus, Elster, Trilliant, Tantalus, SmartSynch (5,15,30, 60 minute intervals; most 60 minute)
Services 26 Validation, Estimation and Editing Services 28 Framing Structures
Interval Reads ~ 100m – 120m per day (processing 20 to 28 million per hour)
Billing Requests 200K to 300K per day
Data Retrieval Requests
15K to 60K per hour
Reports Delivered 2,300 per day
Situation
Data Retrieval Requests
Research Studies
Ad-Hoc Queries
Recent Data Historical Data
Simultaneous
Availability Req’ts
99.8% 24x7x365
Challenges
Managing 200+ billion row database and growing
Challenges
Retention Requirements (7+ years)
New Demands & Requirements
Diverse Customer Base (72 Utilities)
Processing Transactions vs. Queries/Analytics
Our Journey
Research & Proof of Concepts (3 months)
•Requirements
•Analysis and Recommendations
•Validation of Recommendations
Planning & Procurement
(3 months)
• Software Procured
•Vendors Engaged
Architecture & Design (2 months)
•Presentation
•Application
•Data Schema
•Data Mart
•Extract, Load, Transform
• Infrastructure
Build (9 months)
•Detailed Design
•Data Analysis
•Development
•Unit Testing
•Extract, Load, Transform
• Infrastructure
•Hosting
Beta Launch & Pilots
We are Here
Functional, Integration,
and Performance Testing
11
Solution Considerations
Time-Series
Database
Performance
At Scale
Total Cost of Ownership
Availability of Talent
Vendor and Product Maturity
Data Integration
Security & Privacy
Proof of Concept
Scale and Performance Testing
Records/ Day/SDP
Years Days SDPs Recs/ SDP
Total Records
Interval, Usage, Register
29 5 1,826 5,000,000 52,954 264,770,000,000
(264.8B)
Data Model Test Cases
24 Test Cases, run 7 times • Retrieval, Aggregations,
Analytics • No caching; controlled
Proof of Concept Testing Results Sample Tests SDPs Run Time* Retrieve 1 day (29 records) to 730 days (21K records) of data 1 4 to 332ms
Retrieve 365 days of records for 10K SDPs (random across 5 years); 105 million records
10K 1.85 min
Aggregate interval consumption by hour for 1 day for all SDPs; group by utility and VEE Service
5000K 8.6 sec
Aggregate interval consumption by hour for 365 days for all SDPs
5000K 2.5 min
Find SDPs whose daily consumption differs by more than 10% of their 30-day moving average
100K 5.7 sec
Re-process interval consumption and calculate impact of changing “time-of-use” buckets for 1 month
5000K 1.2 min
Compute the monthly consumption usage for a specified month and 1 year prior and rank SDPs by their usage change
750K 7.1 sec
* Tests used historical database only with caching turned off; actual results will vary with use of cache and in-memory database
Proof of Concept Testing Results
Most of the tests were completed in
seconds Tests involving 40+ billion rows of
meter read data took
two minutes
IESO - Meter Data Management and Repository
Meter Data Management System
Distributors
CIS / AMI System
Consumers
Meter Data Management System Overview
Billing Statement
Web Presentment
Meter Data
SYNC
DC & VEE
Billing
Reports
Meter Reads Retrieval
Web Services
Master Data
Meter Data
Meter Data
Billing Request
Billing Response
Web Service Request
Web Service Response
Reports
Master Data
Ad-hoc Queries / Bulk Retrievals
(SQL)
Data Mart Solution Overview
Meter Data Management System
Distributors
CIS / AMI System
Consumers
Billing Statement
Web Presentment
Meter Data
Meter Reads Retrieval
Web Services
Master Data
Meter Data
Meter Data
Billing Request
Billing Response
Web Service Request
Web Service Response
Reports
Master Data
Transformation & Loading (KDB+)
Bulk Data Extract
Change Data Capture
(Real Time)
Initial Extraction
Intra-Day Extraction
Applications: JAVA, Logos and KDB+; Database: KDB+
Ad-hoc Queries & Analysis (q, KDB+)
Data Mart (KDB+)
Data Mart Solution Overview
> 26 months compressed
Historical Tier 2
0 to 26 months Historical Tier 1
1 to 2 weeks + Master Data
Real-Time In Memory Tier 0
Near-Real Time Feed from MDM
Transformation
Data Retrievals
Ad-Hoc Queries/Analysis
On-Line Archive
Business Benefits
Improved Service Availability
• Available on Sunday evenings
• Available during maintenance
outages of the transaction system
Improved Service Performance
Previous Solution Data Mart
Web-services response time
Up to 5 seconds Milliseconds
Number of requests supported
15-30 per second Hundreds to thousands per second
Ad-hoc queries, aggregations, bulk extracts
Days and weeks Seconds and minutes
Improved Agility and Visibility
• Ability to implement enhancements to query
and data retrieval with no impact to the
transaction system
• Enable enhanced insights from data
submissions and processing
Lessons Learned
Check out other
industries and
companies
Lessons Learned
Validate Claims
Test at Scale
Do More Testing
Benchmarks
Independent Research
Lessons Learned
Know Your Data
Users
Uses
Sources
Change
Next Steps Fulfill Data
Requests
Seek Feedback
Analyze Feedback
& Use
Enhance
Thanks for Attending
Please fill out an evaluation form and drop it in the collection
basket located at the back of the room.