1

VADER: Visualization and Analytics for DistributedEnergy Resources

Raffi Avo Sevlian1,3, Jiafan Yu1,3, Yizheng Liao1,4, Xiao Chen1,4, Yang Weng1,3, Emre Can Kara2, MichelangeloTabone1,4, Srini Badri2, Chin-Woo Tan1, David Chassin2, Sila Kiliccote2 and Ram Rajagopal1,4

Abstract—Enabling deep penetration of distributed energyresources (DERs) requires comprehensive monitoring and controlof the distribution network. Increasing observability beyondthe substation and extending it to the edge of the grid isrequired to achieve this goal. The growing availability of datafrom measurements from inverters, smart meters, EV chargers,smart thermostats and other devices provides an opportunity toaddress this problem. Integration of these new data poses manychallenges since not all devices are connected to the traditionalsupervisory control and data acquisition (SCADA) networks andcan be novel types of information, collected at various samplingrates and with potentially missing values. Visualization andanalytics for distributed energy resources (VADER) system andworkflow is introduced as an approach and platform to fusethese different streams of data from utilities and third parties toenable comprehensive situational awareness, including scenarioanalysis and system state estimation. The system leveragesmodern large scale computing platforms, machine learning anddata analytics and can be used alongside traditional advanceddistribution management system (ADMS) systems to provideimproved insights for distribution system management in thepresence of DERs.

I. INTRODUCTION

Supporting even moderate amounts of DERs such as rooftopphotovoltaics (PV) in the distribution system is challenging.Regions with deep DER penetration such as Hawaii, wherethe increase of distributed PV up to 10% of the minimumdaily load are facing various integration issues such as volt-age violations, protection tripping and increased transmis-sion requirements [33], [15]. Fundamentally, this problemoccurs because the distribution system was designed to servepassive loads that could be monitored in the aggregate atthe substation. Aggregate loads have low variability and areforecastable with high accuracy [25]. In contrast, rooftop PVoutput experiences large scale correlations in disturbances anddo not lead to increased smoothness when aggregated [14].Moreover, active controls in the distribution networks and byconsumers introduces additional variability and bidirectionalpower flows.

In order to mitigate these problems, voltage control andlocal energy dispatching problems have been proposed fordistribution system management. (See [11], [36], [31], [20]and [12] and references within). These methods assume theavailability of extensive models of the distribution network.Approaches relying on complex controls of a large numberof heterogenous devices connected to a network require some

This research was supported part by DOE SunShot Office Project AwardNumber: de-ee00031003. (1) Stanford Sustainable Systems (S3L) Lab, Stan-ford University. (2) Grid Integration, Systems and Mobility (GISMO) Lab,SLAC National Accelerator Laboratory (3) Electrical Engineering, StanfordUniversity (4) Civil and Environmental Engineering, Stanford University

basic level of system observability, but achieving this levelof situational awareness in distribution networks utilizingproprietary SCADA connected sensing maybe cost prohibitive.The lack of distribution grid visibility is a clear limiting factorin widespread DER integration.

The state-of-the art in distribution system management relieson Advanced Distribution Management Systems (ADMS) forsituational awareness and scenario analysis. Existing ADMSfollow the overall paradigm in transmission systems for sens-ing and data analytics. This paradigm requires all end points,whether for sensing or actuation, to be instrumented withhigh bandwidth dedicated communication links to a centraloperation center. The analytics relies on data that is uniformlysampled, with low latency/jitter and high availability.

The analytics itself have almost exclusively relied on powersystem state estimation and power flow studies to determinethe current system state and scenario analysis in planning.Therefore, ADMS software offers distribution system stateestimation (DSSE) and load flow models based on finely tunedmodels of three phase power flow. This traditional physicsbased approach, when deployed at higher spatial granularityin distribution networks, suffers from various problems.Partial Observability: The large capital costs of the ADMSstyle control paradigm is justifiable in the context of missioncritical transmission system control. However, deploying largescale SCADA networks on the distribution system are imprac-tical. Typical distribution systems have partial observability ofvariables of interest, such as substation voltage and currents.New Devices, Data Sources and Models: The deep DERenvironment will have many new device types with unknownsystem models. These include rooftop photovoltaic (PV) sys-tems, local storage devices, electric vehicles (EV), smartcontrol devices and many others. At this granular level, thesystem models may be difficult to characterize or may beproprietary. Additionally, their dynamics may be governed byclosed loop systems which are completely unknown. In sucha system, no simple steady state model can be incorporated inthe traditional steady state power flow equations.Heterogenous Sources and Streams: Many independent devicesare acting on the network, without centralized communicationand control between them. These data exist in disparate silos.The primary types of grid data are utility owned SCADAand non-SCADA information, third party data from DERproduct and service providers, and publicly available data. Theheterogeneity of sources leads to a collection of streams wherethe output is not delivered as part of a synchronized SCADAnetwork for further analysis.

In order to address these challenges, we propose the VADERsystem. The system is meant to be used along side the

arX

iv:1

708.

0947

3v1

[cs

.CY

] 3

0 A

ug 2

017

2

traditional ADMS in providing distribution system situationalawareness. The focus of VADER is data fusion of disparatedata sources and the application of machine learning andstatistical inference techniques to understand the system mod-els and state to provide scenario analysis for DER planningdecisions. The paper describes the overall VADER conceptand gives an overview research questions which are currentlybeing investigated.

II. VADER CONCEPT OVERVIEW

ServicesVisualiza.onRepor.ng

VirtualSCADA

DataPlug

U.lity

Public

3rdParty

WhatNow

WhatIf

Tradi.onal:SE/PF

Direct:ML/Stat

RawDataAccessInges.onAnaly.csApplica.ons

Fig. 1: VADER ingestion pipeline. Raw data accessed via variousAPI’s are cleansed into ‘virtual-SCADA’ stream which are then usedin DER motivated power systems analytics.

An overview of the VADER workflow is shown in Figure1. The main challenges addressed by this workflow are howto combine multiple streams of data, align them and developanalytics for the distribution network related to DERs. Ingeneral, analytics applications require a combination of tradi-tional approaches and metrics with concepts such as machinelearning to effectively incorporate heterogeneous data andsystem models that don’t perfectly capture third party devicedynamics.

The workflow is organized into modules as follows. Rawdata is classified into to the three types of data available toany system and broadly classified as public, utility and thirdparty. In the Access phase, this data is collected and processedvia the Data Plug utilizing third party APIs provided byvendors. The resulting heterogeneous sources include specificdevices, historical databases, etc. In Ingestion, streams areprocessed to obtain a time aligned virtual-SCADA for usein subsequent analytics. In the Analytics phase, the virtual-SCADA stream is utilized to provide situational awareness(what now) and scenario (what if ) analytics that correspond totraditional state estimation and power flow. There are multipleApplications that utilize these analytics such as visualization ofperformance metrics from the distribution network, locationalnet benefits analysis and optimal placement of resources. Aplatform implements the workflow has been developed utiliz-ing the latest technology in cloud computing and analytics.The remainder of the paper explains the steps in the workflowand the platform in detail.

III. VADER DATA INGESTION

A. Available DataIdeally, a distribution system operator has access to a

distribution network where every node is instrumented with

Data Type Spatial Temporal Accuracy Latency

AMI Customer 15 m - Hr. Med. Hr. - DayEV Customer 15 m - Hr. Med. Hr. - DayPV Customer 15 m - Hr. Med. Hr. - Day

µPMU Sparse sub-sec. High 5 min.Line Sense. Sparse 1-5 sec. High 5 min.Substation Sparse 1-5 sec. High 5 min.Weather Regional 1 Hr - 1D High Daily

Solar Proxy Sparse 1-15 m. High minutesSatellite Regional Daily Med. Daily

TABLE I: Available sensor deployments in Distribution Systems.

high sampling rate, low latency sensors. Yet in practice weare restricted by instrumentation options and their limitations.Table I summarizes various types of sensors and their perfor-mance. In Appendix A we detail each sensor further.

B. Data Ingestion and ‘Virtual-SCADA’

?

STREAM1

STREAM2

STREAM3

VirtualSCADAGenera7on

?

SCADACLOCK

Fig. 2: ‘Virtual-SCADA’ concept used to generate time-alignedstreams for subsequent processing.

The aim of a utility SCADA system in transmission anddistribution systems is to enable real-time access to mea-surements from all sensor-ed points in the system. This isrealized by dedicated high bandwidth data connections thatsupport synchronized low latency and high-data rate collectionof measurements.

The task of ‘virtual-SCADA’ generation is the fusion ofnon-SCADA data formats into the analysis process. Figure 2shows a simple model of the output ‘virtual-SCADA’ stream.First a fixed time scale for analysis called a ‘SCADA-clock’will be used for time alignment. Then given the incoming data,the desired signal value for all times, (up to present) will begenerated regardless of 1) missing data, 2) non-uniform dataarrival, 3) non-uniform sampling 4) delays. This is motivatedby similar data cleansing engines seen in various real timeanalytics engines. A classical example of this is detailed in [7],[5], for the PEMS traffic management system, where manytechniques are employed for imputation, forecasting, back-casting and other tasks.

The difficulty of this process depends heavily on the actualdata type. To infer missing data, imputation and interpolationtechniques must be used. For dealing with delays, forecastingmethods relying on traditional time series analysis or moreadvanced machine learning are used. For non-uniform data,

3

interpolation and smoothing can be used. In all cases, the es-timated time series will have uncertainty quantification whichdepends on the particular methods used, which can be used insubsequent algorithms.

IV. BASIC ANALYTICS

A. Data Abstraction via System Primitives

A set of system primitives is defined in VADER to unifythe various basic elements in a distribution network and aidingin abstracting the DER analytics data requirements. ‘Virtual-SCADA’ generation then provides a temporally aligned mea-surements of a subset of the primitives. Basic data analyticsfunctions in VADER are then defined by which primitives aretaken as inputs and outputs and the class of computations per-formed in these primitives. We organize the basic primitives torepresent the network, demand/supply and device interactionsas well along with the system state.

1) Network Primitives: Network primitives include infor-mation on distribution system lines, lengths, phases, voltagesand transformers connected to them. Other information is theconnectivity information of AMI meters (phase and locationalong network), as well as estimates of the 3 phase generalimpedance model of the full network. Information regardingconnectivity of switches, breakers, fuses and sectionalizingdevices are also included.

2) Demand Primitives: Demand primitives include but arenot limited to cleansed, time-aligned energy consumption pro-files at each node in the network. Forecasts for each load fromthe historic data can be considered as well. Other parametersinclude estimates of load models (such as ZIP parameters)which may be determined from advanced smart meter typesif available.

3) Supply Primitives: Supply Primitives quantify all un-derstanding and models related to DERs on the distributionsystem. For example, in the context of DER’s such as res-idential solar, this can include information such as: sizing,location, system parameters and exact or approximate circuitmodels of the system used in detailed power flow analysis.Other parameters of potential use may include possible smartinverter parameters that may or may not be known but canpossibly be estimated from utility owned sensor data. Animportant primitive which is generally not known is theactual production profile of all individual PV systems. This isbecause net-metering tariffs usually require that only a home’snet consumption be measured. Estimating these primitives interms of historic, real time and predicted value is essential.

4) Device Primitives: Devices primitives quantify all mod-els and understanding of discrete controllers, storage or othermiscellaneous devices on the network. Examples of suchdevices and potential modeling parameters include, distributedvoltage controllers, load tap changing transformers, capacitorbanks and residential transformers. Information on storagedevices operated by utilities or customers is considered as well.

5) Distribution System State: Important goals of distribu-tion system management are determining the current systemstate and utilizing this information to evaluate various sce-narios in planning decisions. In VADER, the system state isdefined as a collection of primitives for different types ofinformation:

Network state is the electrical arrangement of sections ofthe network which can be altered due to switch or breakeroperation.

Power system state is the set of voltage magnitudes, powerinjections and phases on each node in the network.

Both the network state and power system state are needed todescribe the full system. Other metrics of interest that plannersand operators can use to understand the state of the entiresystem can be included as part of the composite distributionsystem state primitives.

B. What-Now and What-If

RealTime(WN)Opera&onsOp&miza&onControl

Day-of(WI)DA-ForecastControlsetpoint

Months(WI)ResourcePlacementLNBA

Years(WI):Investment+InfrastructurePlanningForecast

YearsRealTime

VADERProjectScope

Fig. 3: The ‘what-if’ (WI) and ‘what-now’ (WN) analytics fit withina utilities analytics time horizon. Real time analysis requires ‘what-now’ analytics by definition. Medium to long term scenario analysisrequires ‘what-if’ analytics.

These computational models are used in DER analytics usecases which can be categorized as ’what now’ and ‘what if’.Figure 3 shows some example analytics, and their requiredhorizons.

What Now: Before the utility can take any actions, it musthave the best estimate of the state of the distribution system,given all information so far. This situational awareness canbe determined via traditional approaches or more data drivenmethods.

What If : When taking any strategic actions on the distri-bution system, the utility will want to run short to mediumterm scenario analysis given all the information available. Themodels used are based on the best estimates of the underlyingsystem parameters or based on data driven models of inputand output of the system.

C. Computational ModelThe two main computational methods used in VADER are

traditional, and direct learning approaches as shown in Figure4.

1) Traditional Approach: The traditional computationmodel relies on physical models of power flow equationsto constrain the various system primitives (Figure 4). Therequired data is assumed to be uniformly sampled and fullyavailable set of system primitives with uncertainty quantifica-tion. The typical examples are state estimation and load flowstudies.

State Estimation methods use physical models relating ob-servations of voltages, currents and physical parameters suchas impedances and topology to improve estimation of variousquantities. The models are based on standard Kirchoff’s law

4

DirectLearning:DataDrivenModelingMachineLearningSta3s3calInference

Tradi/onal:PhysicsBasedStateEs3ma3on/LoadFlow

Primi/ves/State

Primi/ves

RawData

VirtualSCADA Primi/ves/State

Primi/ves

Fig. 4: The two computational models used in VADER involve thetraditional physics based modeling approach and a more data drivenmodeling approach.

formations, and device characteristics. Sensor readings andstatistics on their nominal uncertainties are used to set amaximum likelihood estimate of the underlying parameters.Variations involve determining the actual network state fromsensor information, in the case of Generalized State Estimation(GES).

Load flow methods are used to predict the set of voltage,and currents on the system under some set of hypotheticalconditions. In this analysis, the physical models and simula-tion conditions are used to solve for all remaining unknownparameters on the system.

2) Direct Learning Approach: To contrast with the tradi-tional physics based methods, VADER relies on more datadriven approaches to solving many of the required problems.This approach is feasible and needed because:

1) Partial observability of all measurements in system;2) Unknown, non-linear system behavior from unknown

devices.The traditional approach cannot be applied in high fidelitystate estimation models unless almost all the required systemprimitives for the state estimator or load flow solver aredetermined.

Replacing finely tuned physical models with approximatedata driven models has been done in a other fields suchas robotics [30]. In robotics, the rigid body dynamics hasbeen traditionally determined by careful physics based modelsfrom CAD drawings of all the parts. However, in realisticsituations, many non-linearities and un-modelled dynamicsmust be taken into account when systems become too large.Therefore data driven models have been successfully proposedand implemented. This analogy motivates the use of thesetypes of models even when working in problem domains withwell defined physics based models.

V. EXAMPLES OF BASIC ANALYTICS

This section describes some of the basic analysis performedin VADER which can be categorized as follows:

Primitives Generations: The traditional approach to deter-mining the system primitives has involved instrumenting allelements or laboriously performing ad-hoc data validationtasks. Given the data limitations discussed in Section III-A,one of the main data science tasks in the VADER system isto automate the estimation of many of these system primitivesto enable advanced analytics.

System State Generation: The DER use cases rely on gen-erating the full system state in various scenarios using eithertraditional of direct learning approaches. Also, this systemstate need not be restricted to the distribution system statebut can include other metrics of interest.

The following projects discussed in remaining sections arecurrent research projects by the author and others working onVADER.

A. Generating Network Primitives: AMI ConnectivityIn order to incorporate AMI data in any practical oper-

ational role, the topological connectivity and phasing mustbe determined with utmost accuracy. This may seem to bean issue of GIS record cross referencing, but with the totalnumber of AMI’s deployed and accounting for human error,many records are incorrect. In private discussions with someutilities, it is estimated that about one third of all AMI haveincorrect phasing labels, therefore making them useless in anyreal power flow modeling for DER integration.

The labor cost to verify all connectivity information isfeasible, but may be prohibitively expensive. Therefore animportant question is whether the large amount of AMI (powerand voltage time series), can be used to recover the correctconnectivity and phasing, correcting human errors in the GISdata along the way.

HistoricalVoltageTimeseries

DetectLoopsbyUsingtheExtendedMutualInforma?on

ComputePairwiseMutualInforma?on

SortMutualInforma?on

Loop(s)?

MaximumWeightSpanningTreeUsingMutualInforma?on

(a)

Network V |V |IEEE 8 0% 0%IEEE 123 0% 1.62%EPRI 13 0% -EPRI 34 0% -EPRI 37 0% -EPRI 2998 0% -

(b)

Fig. 5: 5(a) Connectivity detection algorithm using AMI voltage timeseries. 5(b) Summary of connectivity detection performance undervarious test scenarios as reported in [32]. Abbreviations: IEEE (I)and EPRI (E).

In [32], this problem is solved with little assumptions mak-ing use of parallels to structure learning in graphical models.For example, a distribution system can be interpreted as agraphical model with unknown connectivity, but with end nodevoltage observations. Therefore determining the connectivityfrom sensor measurements is equivalent to determining thestructure of the graphical model.

Assuming a model of independent power consumption ineach home, the voltage at any metered point is v = (I +vpay1)/y2 where I is the current drawn at the home load, vpais the voltage in the parent node (upstream AMI) while y1 andy2 are impedances that need not be known. This model leadsto a probability distribution of the entire vector of voltage timeseries satisfying

p(v) =

n∏i=2

p(vi|vpa(i)). (1)

The key insight in this solution is to treat each node as arandom variable in the graphical model where between any

5

two nodes, conditional independence is determined by com-puting the time-series mutual information metric and apply theChow Liu Algorithm [8]. The algorithm is shown in Figure5(a) where the mutual information between all time series iscomputed, and the maximum weight spanning tree recoveredthe correct AMI connectivity. This work has shown significantimprovement over existing correlation and heuristic analysistechniques found in [6], [9], [19]. Table 5(b) reproducesconnectivity detection results reported in [32], illustrating thestrength of such methods.

B. Generating Supply Primitives: Estimating Real Time andHistoric Behind the Meter Solar Production

Outputs Load and solar at each meter

+ =

Inputs AMI data Sparse # of solar

sensors

(a)

Input Data from load aggregation point (substation) Solar from sensor

Outputs Solar at each meter

(b)

Fig. 6: (a) The historical disaggregation problem is separating thelocal PV production from using net metered AMI data with moder-ately high sample rates (1 minute - 15 minute), and a sparse numberof ground truth solar proxy irradiance sensors. (b) The real timedisaggregation problem uses training data from the historic problemalong with real time SCADA information and solar irradiance todetermine the real time solar generation profile at each end-point.

In many distribution system, utilities may not know thelocation and capacity of installed PV on the network. Thisworks in the context of net-metering, but leads to difficulties inutility operation in terms of modeling and understanding wheredifficulties will arise in terms of localized overproduction. In[17], the problem is addressed as a problem of solar disaggre-gation given a very large number of net-metered residentialunits and sparse solar proxies that a utility can control. Solarproxy information can be high data rate SCADA connected PVsites or irradiance sensors in a region. The is formulated asa source separation problem, which borrows much from loaddisaggregation literature [34] and is composed of a historicand real time problem:

Historical Disaggregation: (Figure 6(a)) Separating the sup-ply and demand of energy from batch analysis of net-meteredAMI and solar proxy information.

Real Time Disaggregation: (Figure 6(b)) Separating thesupply and demand of energy from real time SCADA mea-surements of net load and real time solar proxy information.

Historical separation can be used in planning analysis, whilereal time disaggregation can be used in an operational setting.

C. Generating Demand Primitives: Multi-Hour and DayAhead AMI Load Forecasting

The real time load at all points is a primitive for runningmany different analysis. Therefore, the task of load forecastingcan be seen as a primitive generating procedure. Load forecastsare crucial to running a power flow solutions. In the caseof AMI, the forecast horizons are on the order of 2-3 hrsup to 1 day ahead, depending on the delays introduced onthe network, and data collection systems. Various studieshave shown the difficulty of forecasting AMI data at thelevel of individual homes or premises. However, the accuracycan improve considerably if the end-point in question is asmall aggregate of loads. The forecast accuracy improveswith the aggregation (Figure 7(a)). Note this primitive canbe interpreted as a a feed through of the ‘virtual-SCADA’generation process. Recent work on forecasting of individualcustomers, are the following [10], [37], [29], [13], [21], [24],[28]. Various methods have shown varying success, and havehinted at the scaling which is described in Figure 7(a).

D. Direct Learning of Network State: Topology and OutageDetection

The network state defines the current status of all discreteswitching, or protective devices which are used to disconnector move loads between regions. This is shown in Figure 7(b),where a single feeder can supply a toy network power inmultiple configurations, leading to various topologies. Underany topology, any number of outage conditions can occurwhich must be detected quickly to update real time models.

In the case of Transmission systems, this network stateidentification fits into the larger problem of Generalized StateEstimation (GES) [22]. The GES formulation for network andstate detection is formulated as follows:

{v, w} = argminvi∈C, wi∈{0,1}

‖h−H (v,w,Y) ‖2

st. G (v,w) = 0

Here h is the set of all observations and H(•) is a mapping be-tween the power system state v, network state w, unbalanced3-phase impedance matrix Y and the sensor observations.Function G (v,w) = 0 represents the power flow equations.Previous work in this field, relied on the traditional GESframework. In [18], the authors relax and round the solution tothe GES formulation attaining close to error free results. In [4],the authors use voltage PMU measurements and exhaustiveenumeration in finding the maximum likelihood solution.

This general formulation is difficult to adapt in distributionsystems for a number of reasons.

1) Unknown load models and 3 phase impedance matricesbesides nominal parameters.

2) Sparsity of sensing beyond AMI and few SCADAmeasurements. State estimators generally require highlyoverdetermined measurements [3], since the main goal isnot network state detection, but improving the estimateof the power system state v.

6

3) The feasible set of breakers and switches are difficultto enumerate for fully connected and outage situations.The 2|w| enumeration can be extremely large.

A robust ‘flow based’ solution is given in [27] and [26] fordetecting connected topologies and disconnected outage statesas part of the combined network state estimate as shown inFigure 7(b). This approach combines real time line flows thatare measured by commonly deployed line sensors and loadforecasts for medium sized aggregates of load which will typ-ically be easier to forecast. This approach is promising since itrelies on flows in determining discrete network changes, whichare much more robust than using voltage measurements ascommonly assumed in GES formulations. Additionally, flowsare typically available in the form of sensing since all homeshave AMI interval load readings and sparse switch SCADAcurrent and power flow measurements.

Feeder

Secondary Transformer

Single Home

1

(a)

…

G F F T1 F TN

F T1

… F O1 F OM

(b)

Fig. 7: (a) Load forecasting error (coefficient of variation) withrespect to mean load of AMI time series. (b) The network stateidentification is a combination of detecting the proper radial topology,where all loads a connected and degenerate outage case where someloads may not be connected. The proper radial topology Ti must bedetermined as well as any potential outage state Oi.

E. Direct Learning of Power System State: Data-DrivenPower Flow Modeling

Fig. 8: The black box modeling approach for solving distributionsystem power flow, assumes high sensor density and little GIS orverified information regarding individual devices. In this situation, thedata alone is used to build an approximate power flow model usingappropriate machine learning techniques for training and validation.

An alternative approach from many of the directions dis-cussed here is to adopt a fully data driven power flow model.Similar work was proposed in [23], but restricted to onlycalibration of the system impedance matrix via AMI voltagemeasurements. A more analytical approach to this problem ispresented in the [35]. This approach makes the following clear

observation that it is possible to bypass inference of many ofthe system primitives such as device models, impedances andothers if enough sensor training data can be used to train somenon-linear machine learning model.

The inherent mapping function that steady state power flowprovides can be represented as:

(p,q) = f(v). (2)

The system transfer function is estimated f(•) directly fromlarge amounts of training data of real and reactive power (p,q)and voltage phasor (v), skipping individual models for eachload, supply and control device. This is used to estimate theinverse function g := f−1, which can be used in estimatingvoltage magnitudes in all points in the network from loadpseudo-measurements.

In the case of steady state power flows and constant powerloads, this reduces to a conventional estimation of the busmatrix Y. However, in the regime of medium voltage dis-tribution systems, with large quantities of sensor data, poorload/sensor/controller/supply models, a data driven approachcan be extremely useful in providing insight when propermodels are missing.

In [35], our team has proposed a data-driven power flowmodel based on linear and support vector regression techniquesand evaluated the performance of predicting bus voltages withrespect to varying real power injection ranges. The compara-tive results shown in Figure 8 highlight the capabilities of suchmodels in predicting bus voltages measured in mean absoluteerror (MAE).

VI. DER USE CASES

Here, we focus on a problem that utilities deal with inDER integration which is Locational Net Benefits Assessment(LNBA) of various resources. The main aim of this applicationis to judge the cost of integration of some particular DERwhich has asked to connect to the system. Because of thepotential infrastructure change requirements or adverse affectson the network, a specific resource must be scored so thatappropriate decisions can be made on what actions must betaken.

Determining the locational benefit of various DER resourcescan be done in a seamless fashion in the VADER platform.This requires batch analytics that will have access to all nec-essary data required to perform the required scenario analysisfor individual feeders through the system primitives and powerflow modeling.

Evaluating the locational net benefit for PV siting requirespower flow simulation with time-series inputs from historicPV sites, home loads, along with potential install sites. Thelocational benefit can be determined, looking at DER siteswhich do not violate a set of constraints for power flowsalong certain lines or line thermal limits [16]. Scenarios arescored by evaluating the system state, under different potentialscenarios using the data available.

Computing these requires knowledge of the power systemsstate under various hypothetical scenarios, including the pro-posed DER site, network configuration and other parameters.Figure 9 shows how this can be implemented with the variousraw data streams, primitive generation and ‘what-if’ modules.First, we assume that the data has been cleansed and imputed

7

AMIConnec*vity

SolarDisaggrega*on

GIS AMI

NetworkPrimi*ves

DERTimeseriesEs*mates

PowerFlowModeling-DataDriven-PhysicalModelBased

PVSolarProxy

ScenarioGenera+onTopologySwitchingProposedDERs

+

Loca*onalNetBenefitAnalysis

+

Fig. 9: Sample application use case for providing locational netbenefit assessment using multiple modules discussed.

given our ‘virtual-scada’ ingestion step. The first module isdetermining the connectivity of AMI loads given GIS andAMI voltage time-series analysis. A second module used inthe data flow is historic solar disaggregation problem, whichas described in Section V-A. This uses the AMI net meteringinformation as well as solar proxy information to separatethe total generation of each PV site. Now that the networkprimitives, loads and sources are fully determined, powerflow models can be determined using: (1) Given high sensordensity, a data driven model for the full power flow equationscan be learned. (2) A physics based model using estimates ofthe full impedance matrix.

VII. VADER COMPUTING ARCHITECTURE

PersistentStorage

MessagingQueue

DeviceAPI

StreamAnaly8cs

BatchAnaly8cs

DataExplora8on+Development:DataScien*st

SystemsEngineer

Dashboard:WorkCrew

Opera*onsEngineer

Fig. 10: Overview of VADER software platform. Open source toolsused in high bandwidth data ingestion, storage along with batch andreal time processing. Interfacing with compute resources done viadashboards for operators and analytics interfaces for engineers anddata scientists.

The following section describes the software tools andresources needed to develop the system. The main feature ofthe VADER system is that it is designed for high throughputstreaming of many data sources and support of offline andreal time analytics. For this reason the system leverages manyof the modern large scale computing tools. VADER is imple-mented with EpiData, which is an Internet of Things (IoT)Analytics Platform. An overview of the software implemen-tation is given in Figure 10. This combination of messagingqueue, real time analytics and batch analytics is commonlyreferred to as a λ architecture. The platform in development

uses this is a starting point, with many modifications to fit theneeds of the IoT application. The platform relies on mostlyopen source software components [2].

1) API Gateway: Data ingestion could take place throughAPIs or user interface. For instance, IoT sensor data would beingested using APIs, offline data would be ingested via webinterface, and third-party web services data would be queriedand ingested directly.

2) Message Queues: Enabling a large number of disparatedevices to send data to a central controller can be solved usingmodern distributed computing infrastructure such as messagequeue. Messaging Queues will instantiate multiple in-memorybuffers where any number of consumers and producers ofinformation can interact. These queues generally run in a fault-tolerant and distributed fashion so that there is no single pointof failure. In this system, each device will act as a producerof information and post to the queue. Various subscribers willconsume the sensor information and subsequent analytics. Anumber of message queueing systems satisfying these require-ments are available such as ActiveMQ, RabbitMQ and Kafka.The VADER system is based on Kafka due to its performancewith respect to other messaging queue’s.

3) Stream Analytics: For near real time analytics from largenumbers of streaming sources, open source tools are availableethat can handle high throughput rates which are fault tolerant.Some examples of this are Apache Storm, Samza, and SparkStreaming. Streaming capability can be used to implementmuch of the ‘virtual-SCADA’ functionality desired in termsof data cleansing and real time analytics.

4) Persistent Storage: Data will be stored in a persis-tent database. Several open-source databases were considered,and few have been evaluated for the project. These includeMySQL, MongoDB, HBase and Cassandra databases. Thesystem used in VADER is Cassandra for two reason: (1)almost all of the data required is time series in nature; (2)Cassandra is a high performance database with scalability andhigh availability [1].

5) Batch Processing: In development of data analytics(‘what-if’, ‘what-now’) functionality, a distributed computingenvironment is required. Analytics platforms such as ApacheSpark, PySpark, or RSpark, are typically used for data process-ing. Scientists and engineers will be able to run their analyticsusing Python, R and Apache Spark (PySpark or RSpark),Matlab or other platforms. Apache Spark is a fast and generalprocessing engine for large amounts of data. It can processdata in HDFS, HBase, Cassandra, Hive and any Hadoop inputformat. It is designed to perform both batch processing andstream processing.

6) Visualization, Reporting and Alerting: The data explo-ration is supported via Jupyter Notebooks, with Python and Ras high-level programming languages. Data can be queriedthrough REST APIs or a Spark-Cassandra connector, andanalyzed using statistical or machine learning packages orstandard engineering languages like Matlab. Data visualizationis implemented using Python (Django) and JavaScript (D3.js).The visualization interacts with the backend using a RESTAPI.

VIII. CONCLUSION

We introduce the concept behind the VADER project in in-tegrating various disparate data streams from utilities and third

8

party DER service providers to enable heightened situationalawareness on distribution systems. The platform is describedas well as the data science challenges in generating insightwhere traditional SCADA systems are no longer available. Thebasic system primitives needed along with the analysis usedto generate a subset of them are described along with otherprogress made in the platform development.

REFERENCES

[1] Apache Cassandra NoSQL Performance Benchmarks. https://academy.datastax.com/planet-cassandra/nosql-performance-benchmarks.

[2] Apache Software Foundation: https://projects.apache.org/projects.html.[3] A. Abur and A.G. Exposito. Power System State Estimation: Theory

and Implementation. CRC press, 2004.[4] R. Arghandeh, M. Gahr, A. von Meier, G. Cavraro, M. Ruh, and G. An-

dersson. Topology Detection in Microgrids with Micro-Synchrophasors.CoRR, abs/1502.06938, 2015.

[5] P.J. Bickel, C. Chen, J. Kwon, J. Rice, E. V. Zwet, and P. Varaiya.Measuring Traffic. Statistical Science, pages 581–597, 2007.

[6] S. Bolognani, N. Bof, D. Michelotti, R. Muraro, and L. Schenato. Iden-tification of power distribution network topology via voltage correlationanalysis. In 52nd IEEE Conference on Decision and Control, pages1659–1664, Dec 2013.

[7] C. Chao. Freeway Performance Measurement System (PeMS). Califor-nia Partners for Advanced Transit and Highways (PATH), 2003.

[8] C. Chow and C. Liu. Approximating Discrete Probability Distributionswith Dependence Trees. IEEE Transactions on Information Theory,14(3):462–467, May 1968.

[9] D. Deka, S. Backhaus, and M. Chertkov. Structure learning andstatistical estimation in distribution networks - part i. ArXiv e-prints,jan 2015.

[10] R. E. Edwards, J. New, and L. E. Parker. Predicting Future HourlyResidential Electrical Consumption: A Machine Learning Case Study.Energy and Buildings, 49:591–603, 2012.

[11] M. Farivar, R. Neal, C. Clarke, and S. Low. Optimal inverter var controlin distribution systems with high pv penetration. In 2012 IEEE Powerand Energy Society General Meeting, pages 1–7, July 2012.

[12] L. Gan, N. Li, U. Topcu, and S. H. Low. Exact Convex Relaxationof Optimal Power Flow in Radial Networks. IEEE Transactions onAutomatic Control, 60(1):72–87, Jan 2015.

[13] M. Ghofrani, M Hassanzadeh, M. Etezadi-Amoli, and MS. Fadali. SmartMeter Based Short-term Load Forecasting for Residential Customers.In North American Power Symposium (NAPS), 2011, pages 1–5. IEEE,2011.

[14] T. E. Hoff and Richard Perez. Quantifying PV Power Output Variability.Solar Energy, 84(10):1782–1793, 2010.

[15] IER. Hawaiians learning about the pitfalls of solar power, September2015. [Online; 11-September-2015].

[16] Electric Power Research Institute. Stochastic Analysis to DetermineFeeder Hosting Capacity for Distributed Solar PV. Technical report,Electric Power Research Institute, Palo Alto Ca, June 2012.

[17] E.C. Kara, M. Tabone, C. Roberts, S. Kiliccote, and E.M. Stewart.Estimating behind-the-meter solar generation with existing measurementinfrastructure: Poster abstract. In Proceedings of the 3rd ACM Interna-tional Conference on Systems for Energy-Efficient Built Environments,BuildSys ’16, pages 259–260, New York, NY, USA, 2016. ACM.

[18] G. N. Korres and N. M. Manousakis. A State Estimation Algorithm forMonitoring Topology Changes in Distribution Systems. In Power andEnergy Society General Meeting, 2012 IEEE, pages 1–8. IEEE, 2012.

[19] P. Joshua L. Wenpeng, M. Maras, J. Lo, and H. Brian. Smart meterdata analytics for distribution network connectivity verification. IEEETransactions on Smart Grid, 6(4):1964–1971, 2015.

[20] J. Lavaei, D. Tse, and B. Zhang. Geometry of power flows andoptimization in distribution networks. IEEE Transactions on PowerSystems, 29(2):572–583, March 2014.

[21] P. Mirowski, S. Chen, H. Kam, T. Chun-Nam, and Y. Chun-Nam.Demand Forecasting in Smart Grids. Bell Labs Technical Journal,18(4):135–158, 2014.

[22] A. Monticelli. Electric Power System State Estimation. Proceedings ofthe IEEE, 88(2):262–282, 2000.

[23] J. Peppanen, M.J. Reno, M. Thakkar, S. Grijalva, and R. G. Harley.Leveraging ami data for distribution system model calibration andsituational awareness. IEEE Transactions on Smart Grid, 6(4):2050–2059, July 2015.

[24] M. Vasirani S. Humeau, T.K. Wijaya and K. K. Aberer. Electricityload forecasting for residential customers: Exploiting aggregation andcorrelation between households. In Sustainable Internet and ICT forSustainability (SustainIT), 2013, pages 1–6, Oct 2013.

[25] R. Sevlian and R. Rajagopal. Short term electricity load forecasting onvarying levels of aggregation. ArXiv e-prints, March 2014.

[26] R. Sevlian and R. Rajagopal. Feeder topology identification. CoRR,abs/1503.07224, 2015.

[27] R. Sevlian, Y. Zhao, A. Goldsmith, Ram Rajagopal, and H. Vincent Poor.Outage detection in power distribution networks. CoRR, abs/1503.08244,2015.

[28] P.G. Da Silva, D. Ilic, and S. Karnouskos. The Impact of Smart GridProsumer Grouping on Forecasting Accuracy and its Benefits for LocalElectricity Market Trading. IEEE Transactions on Smart Grid, 5(1):402–410, 2014.

[29] R. P. Singh, P. X. Gao, and D. J. Lizotte. On Hourly Home PeakLoad Prediction. In IEEE International Conference on Smart GridCommunications, 2013.

[30] J. Ting, M. Mistry, J. Peters, S. Schaal, and J. Nakanishi. A BayesianApproach to Nonlinear Parameter Identification for Rigid Body Dynam-ics. In Robotics: Science and Systems, pages 32–39, 2006.

[31] K. Turitsyn, P. Sulc, S. Backhaus, and M. Chertkov. Local controlof reactive power by distributed photovoltaic generators. In 2010First IEEE International Conference on Smart Grid Communications(SmartGridComm), pages 79–84, Oct 2010.

[32] Y. Weng, Y. Liao, and R. Rajagopal. Distributed energy resourcestopology identification via graphical modeling. IEEE Transactions onPower Systems, PP(99):1–1, 2016.

[33] E. Wesoff. How Much Solar Can HECO and Oahu Grid Really Handle?,February 2014. [Online; 10-February-2014].

[34] M. Wytock and Z. Kolter. Contextually supervised source separationwith application to energy disaggregation. ArXiv e-prints, dec 2013.

[35] J. Yu, Y. Weng, and R. Rajagopal. Mapping rule estimation for powerflow analysis in distribution grids. ArXiv e-prints, February 2017.

[36] B. Zhang, A. Y. S. Lam, A. D. Domínguez-García, and D. Tse.An optimal and distributed method for voltage regulation in powerdistribution systems. IEEE Transactions on Power Systems, 30(4):1714–1726, July 2015.

[37] H. Ziekow, C. Goebel, J. Str§ker, and H. A. Jacobsen. The potentialof smart home sensors in forecasting household electricity demand. In2013 IEEE International Conference on Smart Grid Communications(SmartGridComm), pages 229–234, Oct 2013.

APPENDIX

A. Utility DataUtility data can be classified according to performance. The

two classes of streaming sensor data are SCADA and non-SCADA. SCADA data have high sample rates, low latencyand sparse measurements throughout the network. Non-SADAdata have low sample rates, high latency and widespreadmeasurements throughout the network.

Line Sensing: High data rate, low latency wirelessly con-nected devices outputting voltage/current along distributionlines. Ideally, these devices can be deployed everywhere, butunfortunately their deployment has been limited to strategiclocations in the network.

Substation Monitoring: High data rate, low latency voltageand current measurements at local substations. Possibility ofPhasor measurement units installed at each location.

AMI Smart Metering: Ideally, a DSO has real time loadinformation in the form of voltage and current phasor inevery home to perform various controls. Unfortunately, mostAMI are connected through proprietary wide spread ad-hocnetworks for security and reliability reasons. Due to thesecommunication constraints, the data is of low sample rate,high latency, high availability measuring interval power usagein every end node of the distribution system.

Geographic Information System (GIS): Standard data avail-able to utilities for network maps, circuit connectivity andother physical devices. Ideally, this data reflects the exact

9

system parameters of the distribution system, unfortunatelyAMI connectivity, and other topology information is usuallyplagued with errors in any practical setting.

B. Third Party DataThe following non-utility data are needed for DER control

applications:PV Metering: Ideally, this data reports the units output

power at a high rate to a local controller which will incorporatethe information for planning and operations. Currently, PVunits produce an output which is owned by the user or thirdparty installer.

EV Metering: Ideally, a central coordinator will have accessto data obtained from these vehicles in the form of dailycharging profiles and user mobility patterns. The data, likethat of residential smart meters will have sample rates on theorder of minutes and report peak and average power alongwith interval energy consumption, and possibly voltage level.Currently, these forms of data are silo-ed by charging unitproviders or electric vehicle manufacturers.

C. Publicly Avaliable Data

Weather: Course sensor data on order of days or hoursuseful in load and DER forecasting

Irradiance: High data rate irradiance useful for PV fore-casting, or disaggregation. This data can be used for DERmodeling, forecasting.

Satellite Imagery: High resolution satellite imagery coveringmost of world landmass. Can re-image multiple times a dayat 3-5 meter per pixel.

Other Data: Demographic and other social datasets can beused to develop models of customer base for various scenarioanalysis. Non-power social media datasets such as GoogleStreet View or streaming social media datasets can be usedas well.