An Architectural Approach to Building a Scalable and Secure Communications Network for the Smart Grid

Vinay Dua, Head Business Development, Industry Solutions, Digital Transformation Office

Cisco India & SAARC

An Architectural Approach

Multi-ServiceIPv6 communications over an IEEE 802.15.4g-based wireless RF Mesh network.

• IP convergence reduces

technology risk

• High Availability

• QoS

• RF Mesh 865MHz-867MHz

• PLC

• LoRaWAN

• Ethernet/Fiber

• Serial

• 3G/LTE

• AMI + DA + Smart City

Applications

• One Network : Multiple

Applications

Security

• Authentication and

Encryption

• Hardening the

network

• 802.1x

• Key mgmt 802.11i,

• Firewall

• Netflow

• ACLs

Fog Computing

• Distributed Intelligence

SCADA protocol

translation

• SDK and Open APIs

• Application Layer

Gateway / Protocol

Translation

• Application Data

Processing

• Distributed Control

• Application Layer

Security

• BYOI – Enable 3rd Party

Communication Modules

Management and Automation

• Scalable to 10M

endpoints,

• Zero Touch

Deployment

• Geo-location

• RF Mesh, PLC &

cellular analytics &

troubleshooting

• Management

protocol for low BW

and lossy links

Standards & Interoperability

• Full WiSUN” compliance

– not just layer-1, but

layers 1-4.

• Prevents Vendor Lock

In/Ensures

Interoperability across

Equipment from different

vendors

PartnerEcosystem

• Meter OEM

Partners with CGE

SDK/Hardware

Reference Design

• DevNet

• Large set of System

Integrators

• Industry Standard

Skills

Multi-Service Field Area Network SolutionMulti-service & secure IPv6 communications over an IEEE 802.15.4g-based wireless mesh network

Hybrid Network 865-867MHz RF and PLC Mesh

Neighborhood Area Network

EV Charging Infrastructure

Substation

WA

N T

ier

Cisco IoT Field Network Director

Cisco IR809

Ethernet,WiMAX, WiFi

AMI Metering/ HAN Gateway

Transformer Monitoring

Distribution Automation

Cisco IR809

Cisco ConnectedGrid Endpoint

Distribution Automation

Gas/Water Meters

Distributed Energy

Resources

SCADA Protectionand Control Network

Direct Load Control

Work Force Automation

Outdoor Lighting

NA

N T

ier

2G/3G/LTE/Ethernet/Fiber/MPL

S

Certificate Authority

Access Control

Directory Services

Secure Network Infrastructure

NMS

Intrusion PreventionSIEM

AMI Head-End

Dist. Planning

Distribution Management SystemDEREVSE Mgmt. HER Data Center, Enterprise Apps

MDM CIS HistorianIWC FLISR SCADA

Cisco IR809

Cisco CGR2010

WiFi

Cisco IR809 LPWA

Faulted Circuit Indicator

Cisco Confidential 4© 2010 Cisco and/or its affiliates. All rights reserved.

Metering & Data SCADA

Public or Private

IP WAN

Secure handheld with utility engineer

NAN(RF Mesh & PLC)IEEE 802.15.4g

DA devices (Ethernet / Serial)

Cisco CG-NMS

Certificate Authority

Directory Services

AAA Server

AAA Server: scalable, high-performance policy system for

authentication, user access, and administrator access; ECC for meters

OMS DMS

Firewall + IPS Appliance: primary firewall for securing the head-end infrastructure; optional use of IPS module

NTP Appliance: acts as precision timing source

Active Directory(AD) & Certificate Authority (CA): for

user & device identity management along with CA for certificate

management Supports Cryptography: ECC keys for

certificate-based authentication

Oracle

CG-NMS (FND): Network & Security Management: supports browser based clients, interface with ASR 1K, IR 809 and End

Points

CG-NMS DB (Oracle) Stores all operational state, device configuration, network event alarm, performance metric, etc

IPv4 / IPv6Adv. Scalable Routing

FlexVPN, IKEv2Application Visibility

SIEMMeter Data Management

DMSGISDMSOMS

IPAM, DHCPv6 and DNS: IPv4/IPv6 address allocation and naming: scale

up to +10M endpoints

IPAM DHCP

ASR 1K/903

Cisco Confidential 5© 2013-2014 Cisco and/or its affiliates. All rights reserved.

Field Network Applications

AMI Generic Telemetry Applications

EV Charging Station

Distributed Generation

Distribution/Secondary Substation/ Feeder Automation, RecloserControls, Cap Bank Controls, Voltage Regulator, RMUs, RTUs, Sensors

Field/Neighborhood Area Network (FAN/NAN)

Transmission and

Distribution Substations

Field Area Networks - Top Use Cases

Use Case Use Case Title

1

Advanced Metering Infrastructure (AMI):

• OPEX Reduction: Remote meter reading, Connect/Disconnect - Pre-payment, Demand Response

• Customer Service: Power Outage / Restoration reporting, customer portals for usage data

2

Distribution Automation (DA) – Grid Reliability:

Self Healing Feeder Network:

• Fault Location, Isolation and Restoration (FLIR) or

• Fault Isolation and Service Restoration (FISR)

3

Distribution Automation (DA) – Grid Efficiency:

• Integrated Volt / Var Control (IVVC)

• Conservation Voltage Reduction (CVR)

4

Distribution Automation (DA) – Grid Visibility and Control:

• Distribution-Supervisory Control and Data Acquisition (DSCADA)

• Remote Asset Monitoring

5

Remote Workforce Management – Secure Access from the Field to:

• FAN Network Devices

• Remote Experts and Applications (Fixed and Mobile Hotspots)

• Device hardening with 802.1AR and ACT2

security chip

• Network hardening tools

• Certificate-based identities, user names &

passwords

• Role based Access Control

• 802.1x-based access control for meters,

routers, grid devices

• Link-layer encryption in RF Mesh

• Group-based key generation and management

(mesh)

• Network-layer encryption for WAN Backhaul

(IPSec)

Time-stamped logs, correlation

at SIEM

Separation of AMI vs. non-AMI

traffic, segmentation

Mobile Workforce

FAN Aggregation Layer

within Substation

Automation Network

Neighborhood Area Network

(RF/PLC Mesh)

RF/PLC Devices

AMI/DA Head-End

NMS HES

AAA Server

Certificate

Authority

Intrusion

Prevention

Directory

Services

SIEM

Security Services

Secure storage for encryption keys

Secure encryption keys

Network-layer encryption (IPSec)

Link-layer encryption (AES-128)

Field Area

Router (FAR)

CGR 1000

Series

Public or Private

WAN

Security

Secure Device Identity via

Digital Certificates

Strong user identities with

Role-Based Access

FAN Security Architecture

DMZ

Field-Area Network Head-EndUtility Private Network

Registration

Authority (RA)

Tunnel

Proxy

Server

(TPS)

Certification

Authority (CA)

Server

SCEP

HTTPS

Cisco CG-NMS Oracle Database

Cisco Prime

Access Registrar

(AAA/RADIUS)

Cisco

Head-End RouterSCEP HTTPS

SSH

Cellular WAN

Cisco IOS Router

Fog Infrastructure for Running Apps Close to Things

Rich Service Capabilities: APIs

Application Lifecycle ManagementEcosystem PartnersFog Node Choice

Cisco Fog Computing

Platforms: CGR, 8X9 Series Eco System PartnersAPIs: IOx

Device Compute Management

Device Compute Ecosystem

Fog Director

WHAT IS IT

CISCO VISION

CISCO PORTFOLIO

Cisco Confidential 10© 2013-2014 Cisco and/or its affiliates. All rights reserved.

IoT Requires Distributed Computing

App

Traditional Compute Model(Terminal-Mainframe, Client-Server, Web)

ENDPOINT

DATACENTER/CLOUD

Cisco Confidential 11© 2013-2014 Cisco and/or its affiliates. All rights reserved.

IoT Requires Distributed Computing

ENDPOINT

DATACENTER/CLOUD

FOG

AppAppAppAppApp

BYOA: Bring Your Own Applicationat the edge of the network

(data->event transformation, data

aggregation, correlation, etc)

IoT Compute Model(Local control loops, Data Volume, Security, Resiliency, Latency, Scale)

Cisco Confidential 12© 2013-2014 Cisco and/or its affiliates. All rights reserved.

Fog Computing Use Cases

Application Layer Gateway / Protocol Translation

Application Data Processing

Distributed Control

Application Layer Security

BYOI – Enable 3rd Party Communication Modules

Virtual RTU running on Cisco IOx

FOG Intelligence with BitStew on IOx

OMICRON CMC 356 AS 25 kV POWER SYSTEM SIMULATOR

SYSTEM DIAGRAM FOR IOx FOG Intelligence Demo

THREE PHASE DIGITAL CAPACITOR BANK CONTROLBECKWITH M-6283A

CISCO 500 SERIES WPAN ROUTER

DNP3 TCP/IP

ETHERNET

3 3

3 PHASE VOTLAGE SENSING3 PHASE CURRENT SENSINGNEUTRAL CURRENT SENSING

POWER SYSTEM CONTROL ANDCISCO 500 CSMP GUI

CISCO UCS C210 M2

ETHERNET

TELNET INTO BITSTEW MIx CORE

CISCO CGR 1240IOS SOFTWAREVERSION 15.4(2) CG

IOS SOFTWAREVERSION 15.4(2) CG

GOS YOCTO 9.0.1BITSTEW MIX

NETFLOWCOLLECTOR():RECEIVING NETFLOW REPORTS, STORING/MANAGING IN MEMORY.

TRAFFIC APP_TRIGGERS():(under development)

TRAFFIC APP_RESPONDER():(under development)

TRAFFIC APP_INTELLIGENCE():(under development)

TRAFFIC APP_REPORTER():SEND APP_INTELIGENCE() REPORT TO CENTRALIZED GRID DIRECTOR

GRID DIRECTOR

CISCO INTEGRATED FIELD NETWORK HEADEND

NETFLOW():MONITORING TRAFFIC AND PUSHING <SOURCE IP, DEST IP,TIME, > TO COLLECTORETHERNET

SERIAL

https://www.youtube.com/watch?v=2XzLD0oaFIA

DevNet Members To Date

Shanghai Runpower

Borui Electric

IOT Endpoints

Cisco Confidential 16© 2013-2014 Cisco and/or its affiliates. All rights reserved.

Open Standards Reference Model

Open Standards: At All Levels to Ensure Interoperability and Reduce Technology Risk for Utilities

Future Proofing: Common Application Layer Services Over Various Wired/Wireless Communication Technologies

IPv6/IPv4

UDP/TCP

IEEE 802.15.4e MAC Enhancements

IPv6 RPL

Web Services, EXI, SOAP, RestFul,HTTPS/CoAP

802.1x/EAP-TLS and IEEE 802.11i-Based Access Control

Physical Layer

IEEE 802.15.4g2.4GHz, 915, 868MHz

DSSS, FSK, OFDM

IEEE 1901.2 NB-PLCOFDM

IEEE 802.11 Wi-Fi

2.4, 5 GHz, Sub-GHz

IEEE 802.3 Ethernet UTP, FO

2G, 3G, LTECellular

IEEE 802.16WiMAX

1.x, 3.xGHz

Data Link Layer

IEEE 802.15.4Including FHSS

IEEE 1901.2 802.15.4 Frame

Format

IEEE 802.11 Wi-Fi

IEEE 802.3 Ethernet

2G, 3G, LTECellular

IEEE 802.16WiMAX

6LoWPAN (RFC 6282) IPv6 over Ethernet (RFC 2464)IPv6 over PPP

(RFC 5072)IP or Ethernet

Convergence SubL.

NetworkLayer

TransportLayer

ApplicationLayer

Addressing, Routing, Multicast, QoS, Security

Security (DTLS/TLS)

DNS, NTP, IPfix/Netflow, SSHRADIUS, AAA, LDAP, SNMP,…

(RFC 6272 IP in Smart Grid)

MeteringIEC 61968 CIM, ANSI C12.22,

DLMS/COSEM,…

SCADAIEC 61850, 60870

DNP3/IP, Modbus/TCP,…

LLC

MAC

Mgmt

Cisco Confidential 17© 2013-2014 Cisco and/or its affiliates. All rights reserved.

865-867 MHz RF Mesh

VISION: Drive industry to embrace open standards and interoperability

Reduce technology risk

Facilitate connectivity for third-party devices and applications

Quality of Service (QoS)

Enterprise network security

Scalable network management

Spatial reuse for more effective bandwidth

IEEE 802.154g Smart Utility Network (SUN)

IPv6 RF Mesh

ZERO

TOUCH

SCALE SECURE LOW

BANDWIDTH

ANALYTICS TOOLS

IoT Field Network DirectorManagement

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19

Field Area Network NMS Key Features

Geographic Information System (GIS) map-based, visualization, monitoring, troubleshooting,

and alarm notifications

Group-based configuration management for field-area routers and smart meter endpoints

Rule-engine infrastructure for customizable threshold-based alarm processing and event

generation

Inventory of all Endpoints and FARs

Registrations from Endpoints and FARs

Automatically provision FARs and Head-end routers with configuration (for tunnels, mesh

interface, etc)

Automatically provision Endpoints with basic configuration

Collect metrics and events from field area routers, head-end tunnel routers, and mesh endpoints,

and store them in DB

Network status monitoring and diagnosis for issues

Update firmware on groups of FARs and Endpoints

North-bound integration API for transparent integration with utility head-end and operational systems, i.e.

Outage Reporting System

Create work order authorizations for field tool

Scalable, High availability and disaster recovery configuration

CertificateAuthority

1. IR 809 Communicates with RA to ask for a new certificate

Registration Authority

DMZ

ASR HeadendRouter

IR 809India AMI Gw

Data Center

Zero Touch Deployment – Simple Certificate Enrollment Process (SCEP)

LDAP

1

2

3

<meter-list>

Blah

Blah

Blah

…

</meter-list>

4

AAA

WAN Backhaul

5

2. RA Requests a new certificate on behalf of the IR

3. MS CA authenticates this IR against AAA

4. AAA server refers to the LDAP to confirm this IR’s ID

5. The new “LDevID” certificate is generated and passed back

to the IR via the RA

LoRa

WAN

LTE

RF

MeshLoRa

WAN

WiFi

LTE

AMI + Smart City Applications…It’s going to take a lot of connectivity WiFi

Zigbee

Blue

toothRFID

NFC

Zigbee

Blue

toothRFID

NFC

Use Case Preference• Less concerns to power consumption to endpoint• High deployment density in geo• High data rate applications

AMI DistributionSubstation

RenewableEnergy

EV Charging Infrastructure

City FacilitiesWater/Gas/Waste

SmartParking

SmartEnvironment

Asset VisionAnd Tracking

Topology Mesh

Applicable End Point Always active, constant external power supply

Standard IEEE-802.15.4g/e WPAN

PHY/MAC FSK (and OFDM, DSSS)

Protocol 6LoWPAN/IPv6/RPL

Coverage 1.5 km per hopping

Data Rate 10’s to 100’s kbs, modulation dependent

RF Mesh/PLC

Topology Star

Applicable End Point Wake-up and dormant, powered by battery

Standard Semtech proprietary, LoRa Alliance

PHY/MAC LoRa modulation, DSSS and FHSS

Protocol LoRa MAC Spec v3.0, 6LoWPAN possible

Coverage 10s km in rural and 2 km in downtown

Data Rate 100 bs to 50 kbs, adaptive to radio propagation

LoRa

Use Cases Covered with RF Mesh/PLC/LoRA

• Power consumption very sensitive toendpoint

• Massively scattered deployment in geo• Low data rate applications

LPWA Use Cases

Cisco E2E LoRaWAN Solution…Same Infrastructure (AMI + Smart City Applications)

Service Provider Enterprise

LoRaWA Network

Server

LoRaWAN App

Router

BSS/OSS Portal

VPN Connectivity

LoRaWAN

GW

MGMT

AAA/CA

Field Network Director LoRaWAN Beck-end

Internet

LoRa Endpoints

Private Network

Cisco LoRaWAN GW

IR809/829 with

LoRaWAN module

LoRa Endpoints

Public NetworkUnlicensed ISM Radio Unlicensed ISM Radio

GW ZTD and LRR

Initial Provisioning

LoRaWAN Network Mgmt.

and Application

Enablement

Cisco LoRaWAN GW

IR809/829 with

LoRaWAN module

Cisco Value Proposition - Summary

Fog Computing

Management and Automation

Multi-Service Security Standards