A Multi-domain Co-Simulator for Smart Grid: Modeling Interactions in Power, Control ... ·...

A Multi-domain Co-Simulator for Smart Grid: Modeling Interactions in Power, Control and Communications

by

Mellitus Okwudili Ezeme

A thesis submitted in conformity with the requirements for the degree of Master of Applied Science

Department of Electrical and Computer Engineering University of Toronto

© Copyright by Mellitus Okwudili Ezeme 2015

ii

A Multi-domain Co-Simulator for Smart Grid: Modeling

Interactions in Power, Control and Communications

Mellitus Okwudili Ezeme

Master of Applied Science

Department of Electrical and Computer Engineering University of Toronto

2015

Abstract

Smart grid studies have been generating a lot of interest both in the business and research

community as it is now seen as the future power grid. However, understanding the interactions

amongst different components that form the smart grid is a Herculean task. Researchers rely on

simulations as one of the means of understanding the effects of coupling the distributed control

systems, communication networks and power system to form the smart grid, but synchronization

of the simulation times of the various simulators, and data exchange amongst the different

components are challenges that must be overcome. Fixed-time-step and queuing system

synchronization schemes have been proposed, but the fixed-time-step synchronization scheme

suffers from accumulation errors and the proposed queuing method does not scale well. In this

thesis, we present a synchronization scheme that eliminates accumulated synchronization errors,

and efficient data exchange scheme which enhances the scalability of the platform.

iii

Acknowledgements I am grateful to my family for the emotional and financial support they gave me during this

program. Their enduring love, compassion and encouragement kept me going even when I

questioned my own ability. I am also indebted to my research group members (Eman, Abdallah,

Pirathayini, and Joseph) for sparing their time any time I needed help.

This acknowledgement would not be complete without the mention of my Supervisor, Professor

Deepa Kundur, who I owe so much for giving me the platform to do this degree, and advising me

both like a mother and a professor. I will forever remain indebted to her for allowing me take a

sip from her intellectual cup.

iv

Table of Contents Acknowledgements ........................................................................................................................ iii

Table of Contents ........................................................................................................................... iv

List of Tables ............................................................................................................................... viii

List of Figures ................................................................................................................................ ix

Chapter 1 ......................................................................................................................................... 1

1 Introduction ................................................................................................................................ 1

1.1 Background ......................................................................................................................... 1

1.2 Why co-simulator ................................................................................................................ 4

1.2.1 Synchronization ...................................................................................................... 5

1.3 Literature Review ................................................................................................................ 6

1.4 Statement of the Problem and Research Objectives ......................................................... 11

1.5 Dissertation Layout ........................................................................................................... 12

Chapter 2 ....................................................................................................................................... 14

2 Co-simulation in Smart Grid .................................................................................................... 14

2.1 Smart Grid System ............................................................................................................ 14

2.1.1 Components of Smart Grid ................................................................................... 16

2.2 Smart Grid Co-Simulator .................................................................................................. 17

2.2.1 Types of co-simulator ........................................................................................... 17

2.2.2 Components of smart grid co-simulator ............................................................... 20

2.2.3 Methods of co-simulation ..................................................................................... 23

2.3 Smart grid co-simulator modeling approaches ................................................................. 23

2.3.1 Mathematical models ............................................................................................ 24

2.3.2 Simulation modeling ............................................................................................. 26

2.4 Synchronization in smart grid co-simulation .................................................................... 27

2.4.1 Periodic Synchronization ...................................................................................... 28

v

2.4.2 Event-based Synchronization ................................................................................ 29

2.5 Data Exchange in Smart Grid Simulation ......................................................................... 29

2.5.1 Pipes ...................................................................................................................... 30

2.5.2 Shared Memory ..................................................................................................... 30

2.5.3 Sockets .................................................................................................................. 31

2.6 Tool Sets Discussion ......................................................................................................... 31

2.6.1 PSCAD/EMTDC ................................................................................................... 31

2.6.2 OMNeT++ /INET ................................................................................................. 32

2.6.3 MATLAB/Simulink .............................................................................................. 32

2.7 Applications of Smart Grid multi-domain simulator ........................................................ 33

2.7.1 State Estimation .................................................................................................... 34

2.7.2 Small Signal Stability ........................................................................................... 34

2.7.3 Voltage Stability Analysis .................................................................................... 34

2.7.4 Postmortem Analysis ............................................................................................ 34

2.7.5 Transient Analysis ................................................................................................ 34

2.8 Summary ........................................................................................................................... 35

Chapter 3 ....................................................................................................................................... 37

3 Software Implementation ......................................................................................................... 37

3.1 The Power System Module ............................................................................................... 37

3.1.1 Power System Design in PSCAD ......................................................................... 38

3.1.2 EMTDC and C Interface ....................................................................................... 39

3.2 Energy Management System (EMS) ................................................................................ 42

3.3 Scheduler ........................................................................................................................... 43

3.3.1 Create Table Module () ......................................................................................... 45

3.3.2 Create Default Status () ......................................................................................... 45

3.3.3 Rate Module () ...................................................................................................... 45

vi

3.3.4 Search Send Flag Module () ................................................................................. 45

3.3.5 Socket Module () ................................................................................................... 46

3.3.6 Write Database Module () ..................................................................................... 46

3.3.7 Read Database Module () ...................................................................................... 46

3.3.8 Update Output Module () ...................................................................................... 47

3.4 The Communication System ............................................................................................. 47

3.4.1 OMNeT++ ............................................................................................................ 48

3.4.2 INET Framework .................................................................................................. 48

3.4.3 C++ Interface ........................................................................................................ 52

3.5 Summary ........................................................................................................................... 53

Chapter 4 ....................................................................................................................................... 54

4 Test Cases: Microgrid Control and Communication ............................................................... 54

4.1 Study System: Microgrid .................................................................................................. 54

4.1.1 Power Management .............................................................................................. 57

4.1.2 Frequency Control and Synchronization .............................................................. 58

4.1.3 Local Controllers .................................................................................................. 58

4.2 Communication Architecture ............................................................................................ 59

4.2.1 Geni LAN .............................................................................................................. 60

4.2.2 Energy Management System ................................................................................ 60

4.2.3 Communication Channel ...................................................................................... 61

4.3 Simulation Results ............................................................................................................ 62

4.3.1 Performance metrics ............................................................................................. 62

4.3.2 Single-domain simulation ..................................................................................... 63

4.3.3 Fixed-delay Single-domain simulation ................................................................. 63

4.3.4 Multi-domain simulation ...................................................................................... 64

4.4 Impact evaluation .............................................................................................................. 64

vii

4.4.1 Effect of sampling rate on controller .................................................................... 64

4.4.2 Discussion of results ............................................................................................. 69

4.5 Summary ........................................................................................................................... 73

Chapter 5 ....................................................................................................................................... 75

5 Conclusion ................................................................................................................................ 75

5.1 Summary and Conclusions ............................................................................................... 75

5.2 Challenges and Limitations ............................................................................................... 76

5.3 Contributions ..................................................................................................................... 77

5.4 Future Research Directions ............................................................................................... 77

viii

List of Tables Table 2-1 Synchronization methods in some reviewed works. .................................................... 28

Table 2-2: Smart Grid Applications Classified According to Latency Requirement. .................. 33

Table 2-3: Smart grid co-simulation approaches with the pros and cons of each method ........... 35

Table 4-1 Microgrid Data showing the ratings of the DERs, load and line impedance. .............. 56

Table 4-2: Packet Structure of transmitted data, channel capacity, channel types and their

characteristics ............................................................................................................................... 61

ix

List of Figures Figure 1-1: Continuous-time simulation sequence of PSCAD ....................................................... 3

Figure 1-2 Discrete time simulation depicting the execution of queued events which can jump in

time ................................................................................................................................................. 4

Figure 2-1: Smart grid system showing the renewable and non-renewable energy sources, the

control center, and the communication network overlay............................................................. 16

Figure 2-2: Diagram showing the interaction of the components of the smart grid co-simulator20

Figure 2-3: Interactive co-simulation depicting the bi-directional information exchange. ......... 27

Figure 2-4: Non-interactive co-simulation showing unidirectional flow of information. ............ 27

Figure 3-1: System architecture showing the interconnection of the different components that

make up the smart grid. ................................................................................................................ 37

Figure 3-2: Software Architecture of the Co-simulator with all the key components .................. 38

Figure 3-3: Pseudo code for the dynamic buffer managing traffic from different sources going

into the communication network. .................................................................................................. 40

Figure 3-4: Pseudo code of the dynamic buffer managing traffic reception from the

communication network. ............................................................................................................... 41

Figure 3-5: Flowchart showing the working principle of the scheduler with Communication

Network and EMS as black boxes. ................................................................................................ 44

Figure 3-6 TCP Signaling diagram showing connection establishment, connection maintenance

and connection teardown ............................................................................................................. 50

Figure 4-1 Complete diagram of the study microgrid identifying the loads, the DERs, and the

main grid. ...................................................................................................................................... 56

Figure 4-2 Sawtooth Waveform used to regulate the output of the DERs. .................................. 58

https://d.docs.live.net/58d69afe822da748/Thesis%20material/Ezeme_Mellitus_MASc_thesisfirstdraft-OkwudilChukwu.doc%23_Toc428030874







x

Figure 4-3: Communication system equivalent of the microgrid showing the interconnection

amongst the key components of the system. .................................................................................. 60

Figure 4-4: Single domain simulation showing the response of the DGs to transient instability

(load shedding) ............................................................................................................................. 65

Figure 4-5:Non-interactive simulation showing the response of the DGs to transient instability

(load shedding) ............................................................................................................................. 66

Figure 4-6: Full co-simulation showing the response of the DGs to transient instability (load

shedding) ....................................................................................................................................... 67

Figure 4-7: Real time simulation of the wireless channel and the power components of the DERs

....................................................................................................................................................... 68

Figure 4-8: Non-real time wireless channel simulation and the reaction of the DERs to load

connection and disconnection ...................................................................................................... 69

Figure 4-9: Pseudocode showing the data structure used by the scheduler during

synchronization ............................................................................................................................. 71







1

Chapter 1

Introduction

1.1 Background Virtually every device around us today is becoming “smart.” Even physical infrastructure like

the power grid is not left behind. The advancement of power systems, to provide greater

consumer-centricity, more efficiency and a smaller carbon-footprint, has motivated the need for

power grid cyber-enablement [1]. Cyber-enablement will no doubt enable high fidelity data

acquisition and processing for effective decision-making. However, for the legacy power grid to

become a smart grid, it must undergo system integration with advanced sensors, high speed

communications and distributed control, which present significant challenges. Two non-

functional challenges of information and communication technology ICT integration are

interoperability and cyber security. According to [2], interoperability represents the ease by

which a system can exchange information with another system. For our discussion here, it

suffices to say that interoperability issues increase with the “distance” between the measurement

source and the application. The distance here is not only geographical, but also has a context or

organizational dimension; thus, if data transverses several networks within, or outside a

regulatory scope, additional interoperability issues may be experienced due to protocol

conversions or communication gateways and network configurations. The second non-functional

aspect of ICT integration is cyber security. In [3], cyber security is extensively discussed and

provides the best example of the dangers of focusing ICT development merely on the function to

be fulfilled.

Researchers and engineers aiming to perform this form of integration often must study the

impact of bringing two such systems together, and this again is not easy because of the inherent

diverse nature of the two systems. The power grid exhibits analog system behavior while the

cyber system is discrete in nature. Continuous-time simulators of power systems are considered

to mimic analog behavior because of the use of fixed time-step with no jump in time during the

simulation process. This means that there is an event for every time-step in the simulation. Also,

2

it is worthy of note that the time-step is determined and set by a user based on the features of the

power system being studied. This behavior is illustrated in Figure 1-1. In contrast, as shown in

Figure 1-2, communication simulators that often model queued events typically can jump in time

and employ a varying time-step that is beyond user control. Here, the system determines when an

event is created and determines whether to cancel or execute the event based on the outcome of

previous events.

The distinct characteristics of each type of simulator makes exploration of a cyber-enabled smart

grid difficult within a common existing framework. Hence, there is a need to develop co-

simulators capable of solving the integration challenges while maintaining the independence of

each of the systems under study, which is the topic of this thesis. This usually involves using a

power simulator, a communication simulator and a simulator capable of rigorous control

behavior manipulations if the need arises. Enabling these separate simulators to work together

and still depict acceptable real system behavior involves overcoming the challenges of

synchronization of the simulation times of the simulators, and data exchange amongst the

systems.

The synchronization problem arises because of the nature of the simulators used in the cross-

domain simulation. As highlighted above, most network simulators such as NS21, NS3,

OPNET, OMNeT++, etc., are discrete event simulators. This means that they are driven by

queued events, which occur at a particular time and marks a change of system state. Between two

consecutive events, it is assumed that the system state is unchanged [4]. This makes it possible

for the system to jump over time from one event to the next. On the other hand, power simulators

such as PSCAD2, PSLF3, etc., are continuous-time simulators that solve differential equations at

fixed time-steps. This step by step execution, otherwise known as continuous-time simulation as

cited above, means that every time-step must be accounted for [4]. Hence, in every fixed time-

1 NS2 and NS3 are communication network simulators; NS3 represents a newer version of NS2.

2 Power System Computer Aided Design (PSCAD) is the graphical user interface for the power simulator.

3 Positive Sequence Load Flow (PSLF) is continuous-time power simulator used mostly for load flow studies.

3

step, an event is executed. This difference in simulation time behavior presents a synchronization

problem which is elaborated more in Section 2.4 of this work.

Figure 1-1: Continuous-time simulation sequence of PSCAD

4

The rest of this chapter is organized as follows. Section 1.2 discusses why we need a co-

simulator and plain explanation of synchronization. Section 1.3 reviews the numerous literature

in this field with emphasis on the merits and limitations of each. Section 1.4 presents the

statement of the problem and objectives. Section 1.5 presents the organization of the thesis.

1.2 Why co-simulator In [5], a microgrid system using only a power simulator and assuming zero end-to-end delay is

implemented. Also, in [6], a communication network simulator is used to simulate the

communication needs of a virtual power system. The difference in the simulation domain of the

two systems as highlighted in [7] has made it difficult to use one simulator to accomplish both

needs. But we also note that while most considerations in power systems is via mathematical

equations, the communication system is a stochastic system, and the randomness it introduces in

the overall behavior of cyber-physical system like microgrid can only be captured by using a co-

Figure 1-2 Discrete time simulation depicting the execution of queued events which can jump in time

5

simulator. Hence, the high interest by researchers in this area. The use of co-simulator also

allows us to achieve the following:

• Domain specific modeling: With multi-domain co-simulator, we have the flexibility of

modeling and designing each of the systems involved in the integration in their respective

domain, thereby making our work easier and modular in nature.

• Well-developed simulators: Each of the simulators involved in the integration has

evolved over time and has large research community that keeps developing and

improving the simulators. Hence, it affords us the benefit of having well tested simulators

with known consistent behavior than developing another one from scratch.

• Component reuse: Another benefit of this integration is the ability to reuse components

that have been developed by others in this field. An example is the INET modules in

OMNeT++ which would have been built from scratch if we had to build our own

simulator.

1.2.1 Synchronization

As highlighted in Section 1.1, the behavior of the federating simulators with different execution

method makes modelling the interaction amongst the different entities difficult as the simulation

time of the various simulators involved in the federation are different. Hence, the co-simulation

design tries to model the practical behavior of the systems involved by ensuring that events are

applied at exactly when they are supposed to happen in reference to the time-stamp. Processing

times by sensors and actuators, propagation and transmission delays incurred by communication

systems all affect the true behavior of the co-simulated system. Hence, application of an event

with a time-stamp less than the simulation time will halt the simulation prematurely, and

triggering an event before the simulation time introduces an error to the modelling process.

Therefore, synchronization is a process of harmonizing the simulation time of the federating

simulators to ensure that events are applied according to their time-stamps without halting the

simulation prematurely or accumulating errors, thereby depicting the behavior of the modelled

systems closely.

6

1.3 Literature Review For the past two decades, there has been growing interest in the development of a smart grid

system as evidenced by the literature in this area [4, 6-16]. However, the topic of smart grid

(power-communications-control) multi-domain simulation remains an open field of study, to the

best of the knowledge of the author. In this section, we will highlight some research that is

closely related to the work in this thesis.

The earliest approach is entitled MESCOS ̶ A Multi-energy System Co-simulator for City district

Energy Systems [17] which is a multi-domain simulation platform that analyzes city district

energy systems with the aim of serving as a tool for the design and control of energy

management systems. By multi-domain, the author refers to the fact that the integrated

simulators can simultaneously run power system and rigorous control system analyses required

for efficient power system management. The work claims to take into account the various

internal and external sources available for city-scale energy systems, including internal

conversion systems and electrical grids. Moreover, the authors state the work can provide

analysis for a variety of energy sources like solar, wind, biogas and the main grid, which is a

positive feature given that the smart grid requires integration of both the traditional power

sources and distributed energy resources (DER’s)4 [18]. The work however does not take into

account the effect of the communication network infrastructure needed to coordinate the

operation of the both the energy management system and the actuators that implement the

control decisions. Therefore, the work lacks the ability to do cyber-related analysis of the power

grid.

Another work in this area is [19]. Here, wireless communication links for microgrid control are

studied. The author investigates the impacts of bit error rate (BER)5, number of hops traversed

and packet drop rate in microgrid control. The power and communication system is simulated in

MATLAB/Simulink. Hence, no integration is involved. The lack of integration, however, makes

it difficult to model and study practical integration challenges. Although MATLAB/Simulink is a

4 DER includes energy sources like Biogas, Solar power, Wind power, etc.

5BER is the tolerable error rate in bits in the communication channel.

7

powerful general-purpose computational tool, it does not have the much-needed functionality of

advanced communication simulators including event driven communication toolboxes.

In [14], the PowerNet co-simulator is proposed using the Modelica power simulator and NS2 as

the communication simulator. The platform can implement real-time co-simulations to

investigate security, reliability and performance of different components of the smart grid.

Although the systems used in the co-simulation have real-time capabilities, they are not allowed

to run simultaneously because of the need to synchronize the simulation time. The design choice

is to use the NS2 to control the Modelica, so that NS2 determines all the time instants at which

communication between Modelica and NS2 should occur. This choice of design has the

following implications:

• The unpredictability and indeterminism of communication networks is supported.

• The unpredictability of the power system is eliminated. This is because, in practice, it is

easier to predict the behavior of physical system than that of communication networks.

Therefore, the time instants can be hard coded into NS2.

The limitation of this design approach is due to the fact that Modelica cannot initiate data

delivery to NS2, which governs this operation. Therefore, exchanging data between Modelica

and NS2 in response to events initiated by the power system alone is not supported. Hence, the

method fails to exploit the parallelism inherent in the co-simulation approach.

Greenbench [4] is another approach to smart grid co-simulation that is related to this thesis.

Here, PSCAD and OMNET++ are used as power and communication simulators respectively.

The co-simulation platform is used to investigate data-centric attacks on a smart grid system. The

attacks are on the confidentiality, integrity and availability of the data. Synchronization between

the simulators is designed to carefully eliminate accumulated errors which would occur if an

event occurred in between the predetermined synchronization time instants. This is achieved by

utilizing the timing capability of self message in OMNET++, which is a timer that helps to set a

pre-determined time for an event. This behavior makes OMNET++ behave like a continuous-

time simulator. To fully utilize this feature, the researchers used a customized software called

interactor which is responsible for switching the operation of the simulators each time the self

message times out. This helps to ensure that the simulation times are synchronized, but it

8

requires practice and experience to set the duration of the self message for every behavior of the

system under study. For instance, for transient study of the power system, the timer duration has

to be shorter than that of state estimation. Also, the data exchange that Greenbench uses is

simple, and it employs the use of buffer files to read and write data between the simulators,

which is more applicable to small-scale power systems. Each physical device in the system has a

buffer file associated with it for the purpose of exchanging data with OMNET++. Now,

employing this technique for a power system with thousands of buses is simply difficult, hence,

scalability is limited.

Another related work is that of the electric power and communication simulator (EPOCHS) [9],

which is a distributed simulation environment composed of NS2, PSLF and PSCAD/EMTDC.

The NS2 serves as a communication simulator, the PSLF as an electromechanical transient

simulator, and PSCAD/EMTDC as an electromagnetic transient simulator. It solves the

synchronization problem among the federating6 simulators by developing a software mediator

called runtime infrastructure (RTI), which is responsible for synchronizing the simulation time of

each of the simulators in every time-step. Though difficult to implement, it is a good approach as

the simulators are equally single threaded simulation engines and can easily adapt to this

approach. One of the setbacks of using this approach is that there must be an exchange of

information between each simulator and the runtime infrastructure in every time-step of the

simulation. So, for a transient analysis in which the time-step is small, it means that the

simulation time for a few seconds, especially for medium to large systems will be significant.

Also, because the synchronization is a simple time-stepped method, if certain events happen

which requires interacting among those simulators, the event has to be buffered in a cache and

wait to be processed until the next synchronization point. This creates room for accumulated

errors and reduce the fidelity of the simulation, especially for time-critical applications.

In order to improve the synchronization mechanism in EPOCHS, the authors in [16] develops a

power system model using discrete events system formalism (DEVS) and integrates it with NS2

so that the federated simulators collectively behave like a discrete event simulator. The goal of

this technique is to make the power system mimic the characteristics of the communication

6 The merging of different simulators to achieve a set objective.

9

simulator so as to make the overall behavior of the two systems compatible and create a seamless

synchronization process. Ordinarily, DEVS would appear to be an improvement in terms of

synchronization. However, the DEVS package employed is a general discrete event system, and

not specific power system modeling. Therefore, users would have to develop their own code

which conforms to the DEVS specification of power system dynamic simulation. This will affect

the reliability of the models and the scalability of the system. Also, since most off-the-shelf

simulators do not use this approach in power system modeling of dynamic simulation, then this

approach cannot be used in distributed simulations.

In [20], a co-simulation platform is developed using OPNET as a communication simulator and

MATLAB/Simulink as a power simulator. The impact of the architecture of information and

communication technology (ICT) on the reliability of wireless area monitoring systems (WAMS)

applications is studied. Here, hierarchical type of control and monitoring is modelled in OPNET,

and the communication delays are tuned to study the sensitivity of phasor measurement unit

based applications. The authors’ detailed modeling of the processes relating to the phasor data

collection is impressive. However, the paper does not address the synchronization method used,

as well as the actual integration of the two simulators.

Reference [21] is an extension of OPNET to simulate wide area communication networks in

power systems. In this scheme, the power system dynamic simulation is modified as a virtual

demander which works in the following sequence;

• Whenever the virtual demander requests to transmit data on the network, it suspends

itself and creates a packet in OPNET.

• Then, OPNET simulates the total communication delay of this packet and report it back

to the virtual demander.

• Finally, the virtual demander reactivates itself and simulates for the duration of the

communication delay before further processing.

While this method is a good way of eliminating accumulated synchronization errors, it is limited

only to one agent, one request scenario. Hence, scalability is an issue as the single-threaded

nature of the platform cannot support complex hybrid system.

10

In [6], the integration of the Virtual Test Bed (VTB) software and OPNET is developed for

simulating remotely controlled power devices in a power system. The synchronization method

used in the paper is a pre-determined time-step synchronization in which a software mediator is

responsible for assigning same time-step for each of the simulators involved in the federation and

allows the simulators to run in turn in every time-step. Here, a software mediator synchronizes

the simulation time of each of the simulators involved in the integration and samples values from

each of the simulators based on a global simulation time-step, making it susceptible to

accumulation errors as in the case of EPOCHS.

Another recent work in this area is [7], which describes Global Event-Driven Co-Simulation

(GECO). GECO introduces the notion of a global event queue, in which all events occurring in

the simulation are stored and queued in a single event list. This method eliminates the problem

experienced in EPOCHS [9] and other similar approaches, because the data exchange occurs at

the exact time it is intended to occur instead of at a synchronization point in the temporal vicinity

of the event. This increases the accuracy and scalability of the co-simulation. GECO uses NS2

and PSLF as the federating simulators and employs middleware called global event queue

written in programming languages C++ and Tcl, to develop the interface for each of PSLF and

NS2 respectively. GECO is used to study the power transmission grid, with emphasis on fault

detection schemes using distance relays. This kind of application is time-critical, so the time

accuracy and the global event queue suits it best. The only drawback to GECO is the complexity

involved in integrating other communication and power simulators to it as it requires modifying

both the simulators and the GECO framework.

Reference [12] is another co-simulation platform that makes use of a global scheduling approach

to solve synchronization challenges encountered by federating simulators. The paper approaches

the synchronization problem by building a queue that mixes the communication and power

events together and determines which simulator is run based on the time stamp of each event. It

involves a novel approach of adjusting the time-step dynamically depending on the accuracy of

the simulation as it progresses. This is a solid approach, but the time adjustment algorithm

requires substantial modification of the simulation engine of the federating simulators, which

requires access to the source code of these simulators, which may be unrealistic. Hence,

integrating other simulators to the platform will be challenging.

11

Thus, based on the former research in this field, we deduce that the following challenges require

the most significant attention for co-simulator design:

• Synchronization of the simulation time amongst the federating simulators: This is

necessary to avoid crashing the simulation by trying to execute events with timestamps

that have either passed the simulation time or ahead of the simulation time.

• Effective data exchange mechanisms between the simulators that is scalable and ease to

use.

1.4 Statement of the Problem and Research Objectives As discussed in the literature review, most of the known methods have the following limitations:

• The method is implemented in a single simulator where one of the systems (either power

or communication network) is made “virtual” thus limiting its practical applicability. A

case in point is [19] in which the power and communication modules were developed in

MATLAB/Simulink, which are not discrete event simulators appropriate for

communication network modeling.

• In addition, some references such as [20] do not account for synchronization of the

simulation time of the simulators, hence, ignoring a major integration issue.

• Most of the techniques involve using a pre-determined time-step for synchronization as in

[9] which results in events being delayed before they are processed. Hence, accumulating

errors, which could reduce the fidelity of the simulation for time-critical simulations like

transient analysis.

• Also, as stated in the literature review, some work such as [4] employ a buffer file for

every physical device in the model for the bi-directional data communication. Hence,

applying this model will not be scalable in terms of the volume of data exchange

required.

• Most of the control decisions are limited only to elementary digital or logic control.

12

• Finally, most of the co-simulation platform developed thus far does not account for the

role of the energy management system or is simply assumed to be working ideally, which

is not practical.

With the above limitations in mind, this research seeks to develop an ICT-Power system

integrator with the following goals:

• To develop an efficient and scalable queued event mechanism that eliminates

accumulated errors inherent in using pre-determined time-step method.

• To add flexibility to the already existing approaches by jointly considering power,

communication and control modeling for multi-domain simulation of smart grid that can

be used to study transient behavior in both distribution and transmission grid overlaid

with communication system. This way, we are able to observe the impact of cyber-

attacks in power systems in real-time, unlike physical systems modeling technique

described in [22] or the networking approach used in [23].

• To develop an effective data exchange mechanism which uses a dynamic buffer with

socket communication. This will ensure that the scale of the system that can be analyzed

can only be limited by the physical attributes of the computer used in the simulation.

• To facilitate efficient data exchange between the simulators by developing a system that

has a short history with the aid of an open source database called SQLite.

1.5 Dissertation Layout The rest of this dissertation is organized as follows.

In Chapter 2, our aim is to identify all the vital components that make up the smart grid system

and their features. We also discuss co-simulation in smart grid with emphasis on real time and

off-line co-simulation. Components of a smart grid co-simulator are then discussed in details.

The chapter also proposes the modeling approaches used in developing a co-simulator for smart

grid. The problem of synchronizing the simulators is discussed along with the various data

exchange approaches available in co-simulation. The chapter then concludes with discussions of

13

the tool sets used in this thesis as well as the various applications of smart grid co-simulator in

smart grid studies.

In Chapter 3, we discuss the detailed implementation of the components that make up our smart

grid co-simulator. Basically, it is divided into four distinct modules: the power system module,

the energy management module, the scheduler and the communication module.

Chapter 4 is where we demonstrate the use of the co-simulator in micro-grid analysis. Many test

cases are run to show the workability of the co-simulator and the advantage of employing full co-

simulation over single domain simulation for a study that involves more than one domain.

In Chapter 5, we critically summarize our contributions, position our work within the field of

study and discuss future research directions.

14

Chapter 2

Co-simulation in Smart Grid When more than one simulator is integrated and executed to achieve a particular modeling

objective, the process is referred to as co-simulation. In this arrangement, there must be a means

for each simulator to communicate with one another. Moreover, based on whether the

simulations are real time or off-line, there may be a need to synchronize simulation time of the

simulators involved in the simulation to avoid trying to execute past events or forcing sequential

time simulators like the power simulator to jump in time to execute future event as this will halt

the simulation. In smart grid studies, this federation of simulators represents a promising

scientific contribution as it enables researchers to study a variety aspects of smart grid operation

which hitherto was not possible as a result of the limitations of single domain simulators. This

discussion is the focus of this chapter.

2.1 Smart Grid System Smart grid presents an idea of a future power system that is designed and built as an

interconnected network of power and electricity systems with a communication infrastructure

overlay. Generally, smart grid is defined as a network for electricity transmission and

distribution systems that uses a bidirectional communication between customer, power plant and

control center in order to improve efficiency by optimizing the power supply and demand. The

smart grid is developed based on Internet protocols, open standards, and compatibility principles.

In order to appreciate the concept of smart grid, it is better to try and understand the

shortcomings of the traditional power system. The primary duty of the electrical power system is

the production and distribution of electricity to customers in order to satisfy the load demands.

But of note is the fact that the electrical power system is made up of both centralized and

decentralized power plants, transmission and distribution grids as well as loads. And a traditional

power system has a vertically integrated scheme with centralized generation, distributed

consumption, limited interaction and regulatory frameworks that are not coordinated for mutual

advantage. The traditional grid can be characterized by:

15

• Large conventional power plants;

• Technically optimized for regional power adequacy;

• Centralized control;

• Limited cross-border interaction;

• Technology that is almost 100 years old.

Now, the liberalization of the electricity market means there is a need for improved data

exchanges between these components in order to guarantee efficient management of the system.

But because the present power system was built as a centralized system due to the ease of

controlling the large power plants which are responsible for the production of electricity,

balancing power supply and regulating system frequency [10], this has become an onerous task

to achieve. Now, in order to incorporate a more flexible and decentralized power system, the

concept of a smart grid has been introduced. This seeks to develop more distributed energy

resources from mostly renewable sources and integrate them into the traditional power system.

Unfortunately, these renewable sources are intermittent in nature. Therefore, they require robust

control and coordination among the participating plants. These changes in power system

structure result in three main goals:

• Reduction in greenhouse gas emission;

• Increased usage of renewable energy sources;

• Improved energy efficiency.

16

2.1.1 Components of Smart Grid

Figure 2-1: Smart grid system showing the renewable and non-renewable energy sources, the control center, and the communication network

overlay

2.1.1.1 The Power Infrastructure

As can be seen from Figure 2-1, the smart grid is made up of a combination of both electrical and

communication systems. The electrical system is comprised of generating stations, transmission

systems and distribution systems with other ancillary power equipment for facilitating control

decisions. The ancillary devices include smart meters, which are new digital meters that collect

periodic data of energy-usage from the individual consumers and transmits the data via (often)

wireless radio signals to the electric utility. This helps both the utility providers and consumers to

make informed decisions about the operation of the grid in order to save costs, improve

efficiency and reduce the chances of failure. There are also data concentrators commonly called

supervisory control and data acquisition (SCADA)7 whose function is to collect the data from a

large quantity of smart meters (about 500) and monitor the power flow on the grid. Finally, there

are intelligent devices for monitoring the energy usage and are able to switch off the selected

loads, when the electricity tariffs are too expensive or the energy is needed in another place.

7 SCADA serves as a data concentrator for receiving and transmitting signals from/to multiple devices.

Hydro power Plant Off-shore Wind Plant

Comm. Network Overlay

Natural gas plant Solar power plant

Control center

17

From the foregoing, it’s obvious that these devices need some means of communication

considering they are usually located some distance away from one another. And even when they

are close, they may not be operating on the same protocol. And that is the discussion of Section

2.1.1.2.

2.1.1.2 The Cyber Infrastructure

Overlaying the power infrastructure is the communication network which is responsible for the

smooth operation of the power grid by ensuring efficient data exchange amongst the various

devices and generators connected to the power grid. The communication could be realized via a

variety of media including radio, fiber optics, power line carriers, telecommunication cables or a

combination. The cyber infrastructure will ideally possess some features like plug and play

capability, possibilities for mapping to different physical layers, and expandability of the data

models and introduction of new models in accordance with the new and enhanced

communication tasks. In Figure 2-1, it demonstrates how a communication overlay can run

across the span of the power grid to provide communication and coordination amongst the DERs

in the power grid. This helps to make the smart grid a horizontally configured system. Hence, all

the DERs can participate in meeting the power demand on the grid and as well regulate the

frequency of the power system to ensure a stable operation. This capability enables engineers

and researchers to design a hierarchical control approach in which different levels of control can

be applied, hence, optimizing the operation of the system.

2.2 Smart Grid Co-Simulator

2.2.1 Types of co-simulator

A smart grid co-simulator is a software Integrator; that helps to simulate the practical smart grid

operation by enabling different federated simulators to run concurrently or serially, and in

coordination with each other to depict a desired scenario of the smart grid under study. This is

important as the two main components that form the smart grid have different simulation

modeling characteristics as we have highlighted in the introduction of the thesis. However, the

software Integrator is classified into two types based on whether it is real-time and off-line.

18

2.2.1.1 Real-time co-simulator

For decades, simulators played a major role in the design and planning of electrical power

systems. From aerospace studies, transmission line layout design, home electronics design to

optimization of the motor drives in transportation, simulation has helped us to imagine, plan,

design and implement our thoughts before implementing the ideas physically. However, the

rapid evolution of simulation tools, primarily due to decreasing cost of computing devices and

increasing capability of these devices to solve more complex problems with less time, greatly

provides a new vista in the simulation world. The reduced cost has equally made simulation tools

more accessible by a wide variety of audiences. According to [24], a simulation is the

representation of the operation or features of a system through the use or operation of another.

During real-time simulation, the accuracy of the computations not only depends upon precise

dynamic representation of the system, but also on the length of time used to produce results [24].

For a real-time simulation to be valid, the real-time simulator used must accurately produce the

internal variables and outputs of the simulation within the same length of time that its physical

counterpart would. In fact, the time required to compute the solution at a given time-step must be

shorter than the wall clock duration of the time-step. This permits the real-time simulator to

perform all operations necessary to make a real time simulation relevant, including driving inputs

and outputs (I/O) to and from externally connected devices.

For a given time-step, any idle-time preceding or following simulator operations is lost; as

opposed to accelerated simulation, where idle time is used to compute the equations at the next

time-step. In such a case, the simulator waits until the clock ticks to the next time-step. However,

if simulator operations are not at all achieved within the required fixed time-step, the real-time

simulation is considered erroneous. This is commonly known as an “overrun.” Based on these

basic definitions, it can be deduced that a real-time simulator is performing as expected if the

equations and the states of the simulated system are solved accurately, with an acceptable

resemblance to its physical counterpart, without the occurrence of overruns. Real-time digital

simulation is based on discrete time-steps where the simulator solves model equations

successively. Proper time-step duration must be determined to accurately represent the system

frequency response up to the fastest transient of interest. Simulation results can be validated

when the simulator achieves real-time without overruns. Basic functions performed in each time-

19

step include; read input and generate output, solve model equations, exchange results with other

simulation nodes, and wait for the start of the next step. Therefore, in real time co-simulator,

both the power and communication simulator have the capability of doing real time simulation.

This type of simulation helps us to understand two main issues:

• Whether the proposed design or algorithm in the simulation can be digitally implemented

in an industrial hardware platform.

• The effects of practical issues, including noise, non-idealities associated with analogue-

to-digital, sampling rate of digital controllers, which ordinarily would not manifest in off-

line simulation.

2.2.1.2 Off-line co-simulator

For digital simulation, it is assumed that a simulation with discrete-time and constant step

duration is performed. During discrete-time simulation, time moves forward in steps of equal

duration. This is commonly known as fixed time-step simulation [24]. It is important to note that

other numerical solving techniques exist that use variable time-steps. Such techniques are used

for solving high frequency dynamics and nonlinear systems because of its accuracy and ability to

solve algebraic loops. The advantages derived from using variable-step solvers are obtained on

the condition that the solvers take the required processing time to achieve the specified accuracy,

hence, the processing time cannot be easily controlled and varies for each time-step. This makes

variable-step solvers unsuitable for hard real-time simulation requiring to maintain a fixed

processing time. To solve mathematical functions and equations at a given time-step, each

variable or system state is solved successively as a function of variables and states at the end of

the proceeding time-step. During a discrete-time simulation, the amount of real time required to

compute all equations and functions representing a system during a given time-step may be

shorter or longer than the duration of the simulation time-step. This technique is implemented

using two methods: one in which the computing time is shorter than the fixed time-step (also

referred to as accelerated simulation) and one in which the computing time is longer. These two

situations are referred to as off-line simulation. In both cases, the moment at which a result

becomes available is irrelevant. Typically, when performing off-line simulation, the objective is

to obtain results as fast as possible. The system solving speed depends on available

computational power and the system’s mathematical model complexity.

20

2.2.2 Components of smart grid co-simulator

Figure 2-2: Diagram showing the interaction of the components of the smart grid co-simulator

2.2.2.1 Power Simulator

A smart grid consists of energy sources, transmission networks and distribution components, all

containing of both physical components and ICT. The power generators are comprised of a

combination of thermal power plants, hydro power plants, solar power plants, wind power plants,

biogas power plants, etc. The transmission system consists of the towers, underground or

overhead cables, and the substation step up or step down transformers, etc. And the distribution

system has the feeders, step down transformers, cables, insulators as well as some form of

distributed energy sources as its components. In modeling smart grid systems, these components

need to be captured in the simulation. Therefore, in our federation of simulators, we include a

simulator capable of handling all the power modeling, and simulation needs of the multi-domain

21

system with regard to the power system off-line simulation in time-domain. In our case, we

incorporate PSCAD/EMTDC, which is where the design and implementation of the physical

power components of the smart grid is done. It has a graphical user interface which makes

human interaction with the simulator easier, and enables the design, implementation and

visualization of the results of the simulation. EMTDC is the simulation engine of the simulator,

and it is responsible for translating the models designed in PSCAD into equations and carry out

the computation. Finally, results are transmitted back to PSCAD for visualization. The choice of

PSCAD/EMTDC is mainly because our interest is in transient simulation studies and

PSCAD/EMTDC is among the few power simulators with the capabilities to carry out such

simulation. It can solve power systems with time ranging from nanoseconds to seconds but our

interest is in the transient studies which falls under the milliseconds range.

2.2.2.2 Communication Simulator

Different distributed resources in smart grid are able to coordinate their actions in controlling the

grid frequency and meeting power demand need because of the communication network overlay

in the grid. Usually, communication networks of different protocols are used to regulate the

transmission of information in smart grid. It could include wireless, wired, power line

communication, etc., It could also be characterized as a high or low bandwidth network

depending on the specification of the system under study. Now, modeling and studying the

statistical nature of communication network requires a communication simulator incorporated

into the multi-domain simulation. Therefore, to solve this need, we include a communication

network into the multi-domain simulator so as to handle all the modeling, design and

implementation of communication networks in the smart grid. In Figure 2-2, the communication

simulator is OMNET++ and it is this simulator that makes the grid more intelligent. This is

because, a traditionally centralized grid can be converted to a distributed grid with multiple

control structures to enhance efficiency, scalability and security of the power system. The

communication simulator provides basic graphical interfaces like start, stop and pause. The

INET is a framework built above OMNeT++ that provides Internet specific support for both

wired and wireless communication simulations. The IED (Intelligent Energy devices) as well as

the network topology is built into the communication simulator using models derived from INET

framework.

22

2.2.2.3 Control Simulator

Managing a power system involves some decision-making. Some are very basic decisions that

involve using a digital or logical controller. Others are complex, requiring more flexible

computations. Because of the distributed nature of the smart grid, some rigorous energy

management systems are installed to regulate and control the numerous distributed energy

resources attached to the grid. This is necessary as all of the energy sources participate in

regulating the frequency of the grid, and meeting the energy demand on the grid. Hence, in our

co-simulation platform, we incorporate a computational software which is capable of handling

all the specialized control features which ordinary power or communication simulator cannot

handle. In smart grid, a hierarchical control architecture is a common feature in which the

highest level of control participates in power management optimization and sends reference

signals to lower level controllers that implement the decisions. Modeling this top level controller

requires flexible computational software such as MATLAB/Simulink. Therefore, in a bid to

make our co-simulator versatile, we include MATLAB/Simulink in our federation of simulators

to solve the control engineering related problems.

2.2.2.4 Scheduler

In our federation of simulators, the power simulator, the communication simulator and the

control engineering simulator have been identified, but how they work together to make the

simulation realizable requires another piece of software which we call a scheduler. This software

is responsible for regulating the interaction amongst the federating simulators by determining

when each of them runs, imports data from each of them, processes the data, stores the data when

necessary, and exports the data to any of the simulator that needs it. It is also in this software that

the configuration of the sensor rates of the power system measurement is done. It is independent

of any of the simulators and can be built in any language. However, we program the software in

C language and embed it into the federation. The scheduler has an open source database called

SQLite, which is used to provide short history for the system. Providing the history of the system

is necessary in our design in order to enable us to eliminate errors which usually occur in many

synchronization processes highlighted in the previous chapter. The choice of employing the C

language for programming the software is for ease of integration with the rest of the simulators

as each of them has Application Programmable Interface (API) that can interact with C/C++

source files.

23

2.2.3 Methods of co-simulation

When we carry out distributed simulation, there is typically a defined way for data exchange

amongst the various components. This can be categorized as interactive and non-interactive co-

simulation.

2.2.3.1 Interactive co-simulation

Otherwise known as full co-simulation, this is a type of co-simulation in which the federating

simulators dynamically exchange data amongst themselves as the simulation progresses. Here,

there is full to and fro movement of information among the simulators based on design

instructions. Hence, dynamic activities in the systems involved in the federation can be modelled

and captured in the simulation [9].

2.2.3.2 Non-interactive co-simulation

As the name suggests, this type of simulation captures only dynamic activities in one of the

simulators while the other simulator is modelled as a static system. Usually, the data in the

simulator, which models the static system is extracted and stored before the dynamic simulator is

run. The data is then applied to the simulator, which models the dynamic system as the

simulation progresses according to the design instructions. Example of this scenario is a co-

simulation involving a communication network and power system in which delay values are

extracted from the communication simulator, saved and applied to the power simulator by hard

coding the values or using buffer files and the power simulator picks the delay values as the

simulation progresses. Here, disruptions, failures, etc., in the power system model can easily be

captured, but that is not the case with the communication network model which maps the power

system under study.

2.3 Smart grid co-simulator modeling approaches In order to carry out studies on smart grid, we first need to develop a model, solve or implement

the model and check whether we have an acceptable result depending on our interest of study. In

doing this, so many approaches are adopted in developing the co-simulator that will help us solve

the smart grid problem. Some of the methods are used alone, and some in conjunction with one

another. But the two major approaches adopted in developing a co-simulator are: mathematical

models and empirical models.

24

2.3.1 Mathematical models

The general procedure for predicting the behavior of a system is modeling. In this method, we

build models that predict a system's behavior. A model of a system represents a surrogate whose

behavior mimics that of the system. Models are classified into two: physically built and

mathematical [25]. One of the most commonly used models today is a mathematical model, since

they are the most adaptable and inexpensive. In specific applications, mathematical models

consist of computer codes that are easily modified to accommodate fluctuations in system

properties. Mathematical models are a blend of scientific and technological knowledge with the

aim of predicting system performance. Such knowledge is, thereby, incorporated into the

computational codes that the computers execute in model applications. At present, mathematical

models founded on numerical simulation make it easy to study complex systems and natural

phenomena that otherwise would be very expensive, harmful, or even impossible to study by

direct experimentation [25]. From this perspective the significance of mathematical and

computational modeling is clear as it presents the most efficient and effective means of

predicting the behavior of both natural and artificial systems of human interest. This approach is

adopted in studying complex systems like weather, power systems and even used in the oil and

gas industry. Mathematical models are made up of relationships and variables. Relationships can

be described by operators, such as algebraic operators, functions, differential operators, etc.,

while variables are abstractions of system parameters of interest, which can be quantified.

Mathematical models can be classified in the following ways:

2.3.1.1 Linear and nonlinear models

A system is said to be linear if the mathematical model of the system is based on the use of a

linear operator. Hence, if all the operators in a mathematical model exhibit linearity, the resulting

mathematical model is defined as linear. A model is considered to be nonlinear otherwise. The

definition of linearity and nonlinearity is dependent on context, and linear models may have

nonlinear expressions in them. For example, in a statistical, linear model, it is assumed that a

relationship is linear in the parameters, but it may be nonlinear in the predictor variables.

Similarly, a differential equation is said to be linear if it can be written with linear differential

operators, but it can still have nonlinear expressions in it. In a mathematical programming model,

if the objective functions and constraints are represented entirely by linear equations, then the

25

model is regarded as a linear model. And if one or more of the objective functions or constraints

are represented by a nonlinear equation, then the model is known as a nonlinear model [26].

The presence of nonlinearity in system model always complicates the study of the system and

makes it more difficult to study than linear systems. Usually, the principle of linearization is used

to simplify the system and solve it using linear methods. Although, it should be noted that

linearization does not hold the solution to every nonlinear model, especially the ones related to

irreversibility.

2.3.1.2 Static and dynamic models

This is also called steady state (static) and dynamic system. While dynamic model accounts for

time-dependent changes in the state of the system, a static model calculates the system in

equilibrium, and thus is time-invariant. Common representation of dynamic models is via

differential equations. These kind of models are used to model system stability, system

optimization, etc.

2.3.1.3 Deterministic and stochastic models

In a deterministic model, every set of variable states is uniquely determined by the parameters in

the model and by sets of previous states of these variables; therefore, the output of the model is

always consistent given a set of initial conditions. Conversely, in a stochastic model, randomness

is always present, and variable states are not described by unique values, but rather by

probability distributions [27]. In summary, given initial conditions, output of deterministic model

can be predicted with a high degree of accuracy, but that is difficult for the stochastic model.

Example of a stochastic process is the delay experienced in a communication network while

Newton’s law of physics is a case of deterministic systems.

2.3.1.4 Discrete and continuous models

A discrete model treats objects as isolated sets, such as the particles in a molecular model or the

states in a statistical model; while a continuous model represents the objects in a continuum of

possible states, such as the velocity field of fluid in pipe flows, temperatures and stresses in a

solid, and electric field that applies continuously over the entire model due to a point charge.

Discrete models represent an instant in time when an event occurs; in between events, the state

26

of the system is assumed constant. Whereas in continuous models, the events are from a

continuum and thus are dependent.

2.3.2 Simulation modeling

A simulation is an applied methodology that can describe the behavior of a system using either a

mathematical model or a symbolic model. It is also defined as the imitation of the operation of a

real-world process or system over a period of time [28]. When the real system cannot be

involved, simulation becomes the obvious choice. And the real system may not be involved

because of the following reasons:

• It may not be accessible,

• It may be dangerous to engage the system,

• It may be unacceptable to engage the system, or

• The system may simply not exist.

So to counter these objections, a computer will "imitate" operations of these various real-world

facilities or processes. In the field of distributed or federated simulation, two major types of

simulation are identified: interactive and non-interactive co-simulation.

2.3.2.1 Interactive simulation

Interactive simulation is a special kind of physical simulation, often referred to as a human in the

loop simulation, in which physical simulations include human operators, such as in a flight

simulator or a driving simulator. However, if human operators are replaced by another simulator,

the system is termed a distributed simulation or a co-simulation. Here, the dynamics inside any

of the simulators is shared amongst the federating simulators according to a defined data

exchange policy during the course of the simulation. This could be real-time or off-line, but the

most important feature is that each simulator reacts to both its own dynamics and that of other

federating simulators. Hence, there is a bi-directional flow of information during the simulation.

This is illustrated in Figure 2-3 below.

27

Figure 2-3: Interactive co-simulation depicting the bi-directional information exchange.

2.3.2.2 Non-interactive simulation

When there is only one direction of information flow, the co-simulation is said to be non-

interactive. Under this condition, simulator A transfers data to simulator B when it needs it, but

the converse is not true. Hence, while the output of simulator B is a product of its own dynamics

and that of the external simulator, the output of simulator A entirely depends on its own

dynamics. Figure 2-4 demonstrates this type of relationship.

Figure 2-4: Non-interactive co-simulation showing unidirectional flow of information.

2.4 Synchronization in smart grid co-simulation When more than one simulator is involved in a simulation, a problem of synchronizing the

simulation time of the simulators arises. This is necessary as we model practical scenarios in

which we cannot process an event in which its time has passed nor can we process an event in

which its time has not reached as this would result in incorrect modeling of the scenarios

involved or binging the simulators to an unexpected halt. Therefore, the challenge in using off-

line simulator is to process each event at exactly the timestamp it bears. When real-time

simulators are used, this challenge is integrally taken care of as the simulation time is same as

wall clock time. Hence, no need for synchronization. However, off-line simulators offer more

scalability than real-time simulators. Hence, the need for a rigorous process of synchronizing the

28

simulation time of the simulators. To solve this problem, many ideas are canvassed as

enumerated in the Table 2-1 below.

Table 2-1 Synchronization methods in some reviewed works.

Components Synchronization

EPOCHS[9] PSCAD, PSLF, NS2 Periodic

ADEVS[16] Adevs,NS2 Event-based

[20] Simulink, OPNET Not addressed

VPNET[6] Virtual Test Bed, OPNET Periodic

PowerNet [14] Modelica, NS2 Periodic

[21] OPNET only, Virtual Power System Delay estimation

SCADA CST [29] PowerWorld, RINSE Non-interactive

TASSCS [30] PowerWorld, OPNET Non-interactive

GECO [7] PSLF, NS2 Event-based

As can be seen from Table 2-1, the synchronization method can broadly be classified into two

types: periodic and event-based synchronization.

2.4.1 Periodic Synchronization

A situation in which there is an explicit synchronization time defined before the simulation starts

is called periodic synchronization. Here, the federated simulators are allowed to run

simultaneously, and periodically communicates with one another at every synchronization step.

Usually, a run time infrastructure which acts as a coordinator is embedded into the federation to

coordinate this data exchange and ensure that the simulation time of each of the simulators is

synchronized with the global synchronization time [9].

29

2.4.2 Event-based Synchronization

In some cases, there is no explicit defined synchronization point and the simulation time of the

simulators must not be synchronized to any reference. Rather, an event queue is developed which

stores events according to their timestamps. Here, the queue is made up of events from different

simulators and a scheduler determines when each event is processed and by which simulator.

This way, the simulators take time to run according to which event is next on the queue. This has

so many advantages as it eliminates accumulated errors inherent in periodic synchronization and

also removes the expensive process of building a runtime infrastructure that synchronies the

simulation time of each of the simulators involved in the simulation. Reference [7] uses this

method and it proved very effective in eliminating the most demerits of the periodic

synchronization experienced in [9].

2.5 Data Exchange in Smart Grid Simulation In distributed computing, simulation, system integration, etc., the communicating devices usually

have a means of getting access to data held by other devices in the federation. This could be in

the form of file system access, shared memory, windows pipes or sockets, etc. In our model for

instance, the devices in PSCAD/EMTDC will generate data that would be passed to OMNET++

or MATLAB/Simulink for processing. And when these simulators are done with processing of

the data, a set of command or control instructions are sent back to the devices in

PSCAD/EMTDC for necessary action. This task can be accomplished in many ways, but

prominent among them is the method used by [9] in which a Runtime Infrastructure is built to

manage the data exchange between different simulators involved in the simulation. Also, Globe

Scheduler is implemented by [7] by creating a discrete time-event based schedule that uses

timestamps to sort the events and processes them accordingly. While these methods are efficient,

their efficiency is low [4]. To increase the efficiency of this process, it would be better to employ

the use of Inter-Process Communication (IPC) to manage data exchange between these

simulators as it has been the default way of making stand-alone process communicate with one

another. It is fast, it is efficient and it can support real time transmission. Next, we discuss

different implementations of IPC. It simplifies the division of labor among processes, simulators

or computers in a network.

30

2.5.1 Pipes

In computer networking, a pipe is conceived of as an information conduit. It is assumed to have

two ends-one where the information is read, and one where information is written to. It is also

called a special FIFO8 file because of how it saves information in memory. It could also be

simplex or bi-directional. When it is a simplex pipe, only one end allows reading of data while

the other end allows writing of data. On the other hand, a bi-directional pipe allows reading and

writing of data from both ends. Two major classifications of pipes are: anonymous pipe and

named pipe [31].

• Anonymous pipe: A traditional Unix-like pipe that is used for transfer of information

from parent to child processes. Hence, it is only process persistent and dies when the

process dies and is not used for communication across a network. Also, it is a simplex

pipe, i.e. it only allows a unidirectional flow of information.

• Named pipe: It could be simplex or duplex and is used for communication between a pipe

server and client(s). It is system persistent and can be used for communication over the

network and locally. The term pipe server refers to the creator of the named pipe and pipe

client refers to processes that connect to the instance of the named server. In order to

allow multiple clients to use the named pipes simultaneously, instances of the named

pipes are created which share the same name, but have different buffers and handles,

which are used to provide communication between client/server separately.

2.5.2 Shared Memory

This uses a file mapping scheme which allows multiple processes to see the contents of a file as

if they were a block of memory in the process's address space. To accomplish this, each process

that wishes to access the same file mapping is given a pointer to memory in its own address

space which can be used to access the contents of the file. And to avoid concurrent access to the

file by multiple processes, a file locking system like semaphore is always implemented which

provides synchronization and prevents data corruption. In order to make the file mapping file to

8 FIFO is a special access technique which ensures First In, First Out for data in memory.

31

be seen as a shared memory block, a system swapping file is named during the file-mapping

object creation. This enables other processes to access the same block of memory by opening

same file-mapping object. While this is an efficient IPC scheme, its scope is limited only to a

local computer and cannot be used over the network.

2.5.3 Sockets

A protocol independent interface that is used both in windows and UNIX based systems.

Although the implementation details for each case is different, the underlying principle is the

same as both make use of the communication capabilities of the underlying protocols. A socket

is one endpoint of a two-way communication link between two programs running on the

network. A socket is bound to a port number so that the TCP layer can identify the application

which is the data sink. This process usually involves one or more clients and a server. The

development of most socket Application Programmable Interfaces (APIs) is based on the

standard popularized by Berkeley Software Distribution (BSD). It has become the most popular

means of inter-process communication, especially in a large networked system because of its

flexibility and efficiency. It handles both local and over the network communication efficiently

and this is the scheme we implement in our work.

2.6 Tool Sets Discussion In a distributed simulation, more than one simulator is usually involved in accomplishing the

task. In our own case, we federate three simulators with the PSCAD/EMTDC serving as a power

simulator, the OMNeT++ serving as a communication simulator, and the MATLAB/Simulink to

be used as a control engineering simulator. Below is a brief description of each.

2.6.1 PSCAD/EMTDC

PSCAD is the graphical user interface to the simulation engine called EMTDC. EMTDC is

developed to handle electromagnetic transient simulation, hence can capture dynamics in the

power systems from microseconds to seconds. PSCAD being the graphical interface, is where

the schematic construction of a circuit, running of a simulation, analysis of results and exporting

of the data to third party software for further processing is handled. It has online plotting

functions as well as meters and controls which could be used to modify the simulation

constraints during the run and observe the effects of such changes. The simulator has a pre-

32

programmed and tested simulation models, comprising of simple control functions, passive

elements like resistors, as well as complicated models like transmission lines and cables. To

extend the flexibility of using the simulator, it allows users the freedom to build their own

models in either FORTRAN or C programming language and incorporate it in the model. And it

is this capability that we will utilize in the development of our federated simulator. The choice of

PSCAD is based on its capability to capture transient dynamics which is our target area.

2.6.2 OMNeT++ /INET

OMNeT++ is an object-oriented, discrete event simulation framework. Its modular architecture

has been exploited in developing and simulating models in various disciplines outside

communication engineering. Some of the areas in which it has been used includes but not limited

to validating hardware architecture, protocol modeling, wired and wireless network

communication modeling, and other general modeling and simulation architectures that are

modular and discrete event driven. OMNeT++ provides the infrastructure for writing the

simulation, but not a simulator on its own. It only provides reusable modules which are

hierarchically designed for the user to assemble as desired to facilitate the simulation. INET

framework is an example of one of the reusable modules that is specifically designed for TCP/IP

and UDP simulation in OMNeT++ and serves as the underlying framework in our models.

2.6.3 MATLAB/Simulink

MATLAB is a high-level language that handles visualization, programming and numerical

computation. It is used to develop algorithms, create applications, do data analysis, etc. It has in

built functions and numerous tools that aid in designing models, running simulations and

implementing wide range of ideas in many disciplines such as communication and signal

processing, control engineering, finance, image and video processing, etc. MATLAB equally has

a rich Application Programmable Interface for integrating models developed in other languages

like C/C++ or Java, and this flexibility has made MATLAB one of the most popular

computational software. In our case, it is used in the energy management system to optimize the

set point of the distributed generators.

33

2.7 Applications of Smart Grid multi-domain simulator Based on the survey made by [11] using the classification of North American Synchrophasors

Initiative, the latency requirements of smart grid applications are listed in Table 2-2, and these

applications are briefly explained below.

Table 2-2: Smart Grid Applications Classified According to Latency Requirement.

Applicatio

n

Actions Data Source/Sink Data Latency

Requiremen

t

Time

Windo

w

State

Estimation

Power

flow,

AGC,

AVC, etc.

All substations/

Control center

Phasor,

Network

Model

1 second Instant

Small

Signal

Stability

Modes,

Damping,

etc

Key

Locations/Application

Servers

Voltage

Phasor

1 second 10-15

minutes

Voltage

Stability

Load

Shedding,

Capacitor

Switching,

etc.

Key

Locations/Application

Servers

Voltage

Phasor

1-5 seconds Minutes

Postmortem

analysis

Model

Validation

All PMU and DFR

data/Historian

All

measurement

s

NA Instant

and

Event

files

Transient

Stability

Load Trip,

Islanding,

Generating

substations/Applicatio

n Servers

Phasor 100

milliseconds

10-50

cycles

34

Generatio

n Trip

2.7.1 State Estimation

This provides an internal state of the given power system using measurements from the PMUs. It

tries to minimize the effects of corrupt or bad data on the computation of the system state.

2.7.2 Small Signal Stability

The ability of the power system to maintain synchronism when subjected to small disturbance

refers to small signal stability. It’s an event that occurs over time and collecting signals from key

locations, and observing mode damping, preventive measures can be taken to forestall the small

signal instability.

2.7.3 Voltage Stability Analysis

Deficiency in reactive power in an area can easily cascade and lead to system failure. Therefore,

voltage stability studies tries minimize the occurrence of this by taking voltage measurement in

an area and using VAR balancing or Islanding as a stabilizing measure. The distance that usually

exists between the source of the measured data and where it is processed implies that an effective

and reliable communication system is vital in this study. Therefore, this forms a core part of

smart grid applications that should benefit from multi-domain simulators like the one proposed

in this thesis.

2.7.4 Postmortem Analysis

Latency requirement is not applicable to this application, and it uses PMU data as well as data

from digital fault recorders (DFR) to update the engineering setting of the system.

2.7.5 Transient Analysis

This relates to the speed and internal angles of the generators in a power system but for

microgrid that is not using synchronous machines, it refers to the behavior of the system when it

is subjected to heavy transient disturbances such as loss of large loads or loss of generation. The

35

latency is small and the system requires fast control scheme in order to prevent it from becoming

transiently unstable.

2.8 Summary In this chapter, we described various aspects of smart grid co-simulation ranging from its

individual components to a variety of classifications. We provided a detailed discussion of

several modeling approaches in smart grid co-simulation. Then, the synchronization techniques

that have been used in smart grid studies were discussed as well as the data exchange options

that are most prevalent. Finally, we discussed the tool sets used in this work as well, and

provided the classification of smart grid application based on latency requirements. The table

below provides the key points of the two simulation methods.

Table 2-3: Smart grid co-simulation approaches with the pros and cons of each method

Simulation approaches Merits Demerits

Real-time co-simulation 1. No synchronization

required.

2. Could be faster for

systems with less

switching and non-

idealities.

1. Does not solve

algebraic loops.

2. Uses fixed time-steps,

hence knowledge of

appropriate time-step

is required.

Off-line co-simulation 1. Uses variable-step to

speed up the

simulation.

2. Better accuracy.

3. Solves algebraic

loops.

4. Higher numerical

stability.

1. Tolerance range and

type of solver must be

provided.

2. Synchronization is

required.

3. Uses fixed time-steps

for some systems.

37

Chapter 3

Software Implementation In this chapter, we give a detailed discussion of the software implementation of our multi –

domain simulator. In order to ease the understanding of the various components that make up the

simulator, we have dedicated a section to discussing each component. Section 3.1 discusses the

power system module and the realization, Section 3.2 details the energy management system, the

scheduler is discussed in Section 3.3 while we conclude the chapter with the explanation of the

realization of the communication module.

Figure 3-1: System architecture showing the interconnection of the different components that

make up the smart grid.

3.1 The Power System Module Figure 3-1 represents our system architecture and is made up of the power system, the

communication system and the energy management system which makes the top hierarchy

control decision. The power system is implemented in PSCAD/EMTDC and comprised mostly

of embedded applications which makes it easy for any user to test any system implemented in

38

PSCAD/EMTDC in the co-simulator with minimal work. In Figure 3-1, DERs, the distribution

grid and the connecting lines make up the power system module. Below is the figure which

describes the software architecture of the whole system.

Figure 3-2: Software Architecture of the Co-simulator with all the key components

We implement a system that could function like a plug and play device. So the role of each block

is explained below.

3.1.1 Power System Design in PSCAD

As described in previous sections, PSCAD is the graphical user interface for the EMTDC

simulation engine and this is where each user of the co-simulator can design any conceived

power system. With our software, our idea is that each person that wants to use the co-simulator

can design his/her own power system in PSCAD and use it in our system for transient studies in

the time domain that involves co-simulation without doing much work in modifying our system.

Hence, the type and scale of what is obtainable in this platform is only a decision made by the

39

user of the system. The pre-programmed models in PSCAD as well as the user defined models

can be used in this section for designing the desired system. Also, PSCAD is rich with many

control and input functions which are used to adjust the parameters of the system as the

simulation is running. PSCAD became a natural choice for us because it can simulate short

transient disturbances with high fidelity, and that is our area of interest. Finally, the results can

be analyzed in the PSCAD using various plotting tools or could be exported to third party

software like excel for further analysis and processing.

3.1.2 EMTDC and C Interface

EMTDC is the simulation engine of the PSCAD software and one vital feature of the software

which we exploit in the development of the multi-domain simulator is the freedom to build a

custom model and embed it into the simulation engine. FORTRAN source code, C source or

object files can be incorporated into the simulation engine using a special Application

Programmable Interface (API) provided by Manitoba Hydro who are the licensed owners of the

software. Therefore, pre-programmed models which are not in the supplied library that ships

with the software can be built using any of the mentioned programming languages above. With

this extension capability, we designed an interface using the C programming language to enable

the EMTDC to import and export data from the scheduler. For internal interactions (between

PSCAD and EMTDC custom built models), we develop a C subroutine which eases interaction

between the custom built models and the pre-programmed models in PSCAD. For external

interactions (between PSCAD/EMTDC and scheduler), we employ inter-process communication

using windows sockets programmed using the C programming language. The choice of windows

socket is obvious as we wanted something that is protocol independent and faster. And the ability

of the socket communication to make use of underlying operating system communication

capabilities has been proven to speed up communication based on the socket networking

communication. This interface contains two essential parameters which ought to be manually

configured by the user of the software and below is a description of both;

• Dynamic Buffer: A key design parameter and determines the quantity of data that could

be transferred out of and into EMTDC. To make the software scalable, we settled for a

dynamic buffer so that the volume of data that could be exchanged between EMTDC and

the rest of the system is only constrained by the available hardware storage in the

40

computer systems used in the simulation. This could be likened to statistical multiplexing

technique and allows freeing of the communication buffers and bandwidth for other

forms of traffic when the smart grid traffic is small in size. As expected, this feature is

present for both inbound and outbound traffics on both the EMS and the power simulator.

All that is required to activate the dynamic buffer is the knowledge of the number of

sensors that will be taking measurements in the system. This value would have to be

supplied by the user in the configuration file manually. The pseudo code for both the

outbound and inbound traffic are provided below.

Figure 3-3: Pseudo code for the dynamic buffer managing traffic from different sources going

into the communication network.

41

Figure 3-4: Pseudo code of the dynamic buffer managing traffic reception from the

communication network.

• Sampling rate: For synchronized measurement sensors like PMUs, this rate would have

to be uniform, hence, only one value is supplied. But where the design requires different

rates for different sensors, the user would supply this value manually in the configuration

file provided. This is a very important consideration as the sampling window affects the

utilization of the data as enumerated in Table 2-2. For instance, in time critical

applications like transient analysis, there is a higher sampling frequency per second while

in other applications, it could be lower. In order to accommodate whatever application

that this software could be used for, we have given the user the liberty to configure this as

it suits the system under study.

42

3.2 Energy Management System (EMS) In Figure 3-1, the EMS is the central controller. Again, this feature is incorporated in view of

different control and management structures available in smart grid. Generally, there are two

kinds of control systems structure: centralized and distributed control systems.

• Centralized control system: When the action performed by lower level controllers are

dictated by the information obtained from the central controller, then this arrangement is

called a centralized control system. Here, the central controller can instruct the lower

level controllers directly or if the system is so complex, a kind of hierarchy could be

created in which there are other controllers in between the top level and lower level

controllers. The communication between the top and lower level could be bidirectional

(feedback) or unidirectional (open loop) as the case may be but the lower level controllers

will always depend on instructions from the central controller.

• Distributed control system: As the name suggests, when the lower level controllers

function based on local information only, then the system is said to be a distributed or a

decentralized system. The major difference here is that the lower level controllers make

their decisions without depending on the instructions of any commanding influence.

Therefore, we integrate the EMS into our work so that any study system that requires a

centralized or hierarchical control system could easily be studied without many changes to the

underlying software. The system can equally handle a system that has a purely decentralized

control system. In order to accommodate both behaviors without degrading the results of an

experiment, we give the user the option of enabling or disabling the EMS during the simulation

configuration. And to ensure versatility in the control functions, we use the MATLAB/Simulink

computational software as the control/management system simulator. The choice of this is an

easy one considering the different field of study in which the software readily has the toolbox.

So, all that is required of the user is to design the control system in MATLAB/Simulink and the

rest of the system will be interacting with it as the need arises via the MATLAB/Simulink API

which is available in numerous programming languages. This way, data is fed into the

control/management system in MATLAB/Simulink and it returns the result to the sending

software after processing.

43

However, if the control system is a decentralized one and does not require heavy computational

decisions, then the EMS can be disabled, and the control and logical functions and tools inside

the power simulator is then utilized to design the controllers.

3.3 Scheduler As the name suggests, this is not a simulator, but a software component that is responsible for

coordinating the behavior of the simulators involved in the multi-domain simulation. It creates

the discrete-event queue of all events generated from the different simulators and sorts them

according to their timestamp, and then alternates the run of each simulator based on the next

event in the queue. The details of how this software is realized is explained in the sub-sections

below and the Figure 3-5 is the flowchart diagram of the interactions between different modules

in the software. It is worth mentioning that the choice of the simulator where this program is

embedded determines which programming language is used to implement it. In our case, we

embed it in the PSCAD/EMTDC simulator and because it allows only C or FORTRAN files, we

implement it using the C programming language.

44

Figure 3-5: Flowchart showing the working principle of the scheduler with Communication

Network and EMS as black boxes.

45

3.3.1 Create Table Module ()

As shown in the flowchart, when the simulation starts, as part of the initialization process, the

scheduler checks to see if that is the first time-step of the simulation and if so, it creates the

database tables that will be storing the history for each of the physical sensor devices in the

power system. Checking whether it is the first time-step was necessary to ensure conflict free

table creation process and prevents the simulation from coming to a halt.

3.3.2 Create Default Status ()

After the successful creation of the tables, then the scheduler creates default values to be applied

to the physical devices in the power system and choosing what this value should be is a matter of

knowledge of the nature of the physical devices and their expected input values. The import of

this is to enable PSCAD/EMTDC progress with the simulation at the initial stage. This is as a

result of the simulator being a single threaded program and requires all declared variables to

have a value before the next step of the simulation is executed.

3.3.3 Rate Module ()

In order to take measurements from the physical devices in the power system and transfer them

to the control center via the communication channels, the scheduler incorporates a rate module

which is responsible for figuring out when values from each physical device should be sent to the

equivalent control center device for processing. If PMUs are used as the sensing device, then

there is a synchronized rate for the devices and the dynamic buffer manages how the data from

each device is arranged in the buffer and transmitted. If rather the rates are not the same and

there is no need for synchronization, then rate module, using the rates supplied by the user of the

software for each physical sensing device, decides when the measurements are taken.

3.3.4 Search Send Flag Module ()

When any physical device needs to communicate with the control center, it sets a flag. So, the

search send flag module is responsible for continuously scanning the send flag to see whether

any of the physical devices in the power system needs to communicate with the control center. If

no flag is set, the simulation proceeds. However, if a flag is set, then the scheduler queues it in

the event list and initiates the communication with OMNeT++ using the windows socket

communication, which in turn will hand over the simulation time to the communication and

46

control simulators. Because the simulators are single threaded simulators, this will automatically

pause PSCAD/EMTDC until the scheduler has a power event in the queue that it wants to

execute.

3.3.5 Socket Module ()

An essential component of the scheduler, it facilitates the data exchange amongst the three

simulators. We make this a socket client, which binds to the socket server in the communication

simulator and transfers data to and fro from the communication simulator when the need arises.

The choice of windows socket was primarily due to its efficiency, protocol independency, and

ability to network multiple computers running these simulators with ease. This function is

activated by the search send flag module if there is any physical device that has set a flag for

transmission. It should be emphasized that there is no delay between setting the flag and

transmitting the measurement because the events are sorted according to the timestamps of their

occurrence in the queue and executed accordingly.

3.3.6 Write Database Module ()

If the socket module was activated and the communication was successful, then the next thing the

scheduler does is to store the returned control values from OMNET++ in the database of the

system which is built using the small, light, open source database called SQLite. One of the

benefits of using the SQLite database is the ease of embedding it in any application because of

its small size. Also, it has API for most popular programming languages and we make use of the

C API of the database to store the timestamp and the returned control value in the database.

These saved values are used to create an event queue and are sorted for execution according to

their timestamp. The control values could represent reference set points from top level

controllers to lower level controllers or simple close or open command from the actuators to

circuit breakers.

3.3.7 Read Database Module ()

Because the event list is in the database, for every time-step during the simulation, the read

database module checks the database to find out if there is any physical device that needs its

status updated by passing an instruction in the queue to it. Which device status gets updated is

solely dependent on the time stamp which was used to sort the events. Hence, if multiple devices

47

share same time stamp, they all get updated at the same time. This process is cardinal in

eliminating accumulated errors which occur by explicitly fixing when the synchronization should

occur, which is otherwise called periodic synchronization.

3.3.8 Update Output Module ()

Like we stated earlier, the PSCAD/EMTDC simulator requires that every variable has a value

before it proceeds to the next step. Hence, after reading the database and checking for

synchronization point, the output status module is updated. And then the update output module

updates the status of the physical devices connected to the scheduler and the simulation

proceeds. Output status module is an intermediate module which ensures that if there is no

synchronization point, values in the previous history of the output status module are applied to

the devices, and this corresponds to a scenario in which the system dynamics is assumed not to

have changed over time. This is just to ensure that devices (represented as variables) always have

values during the simulation and prevents the simulation from coming to a premature halt.

3.4 The Communication System For the energy management system at the control center and the local controllers to

communicate, a communication link has to exist to link the systems together and it is the

incorporation of this feature that is shaping the development of smart grid. To this end, features

which hitherto are not obtainable in the traditional power system as a result of limitation of

hardware or lack of communication between different parts of the power system are now being

studied and integrated. With the inclusion of communication overlay in the power system, the

power system is more responsive and dynamic to consumer behavior. Also, it is now possible for

so many decentralized controllers of power plants to participate in maintaining grid frequency

and voltage via efficient coordination engineered by the use of communication network

infrastructure as against the old situation of using one large power plant to manage voltage

and/or frequency. To facilitate the understanding of how we realize our communication system

for the smart grid, we break it into the sub-sections below.

48

3.4.1 OMNeT++

From Figure 3-2, the top of the communication system software is the OMNeT++ framework.

This is a collection of reusable objected-oriented modules that can be assembled to do network

simulation. The modules are hierarchically designed and ranges from simple to compound

modules. Also, the OMNeT++ comes with eclipse-like Integrated Development Environment

(IDE) which allows for design, simulation and analysis of the study system conveniently using

the rich graphical interface. For those that like UNIX-like command line method, it also allows

simulation using the command line. It is built using the C++ programming language and it is an

open source software. On its own, OMNeT++ is not a simulator, it is a collection of C++ objects

which have been arranged as modules and these modules can be run on any system that has a

C++ compiler installed on it. The graphical interface has functions like run, stop, debug, etc., and

equally has some simple online plotting tools to analyze results. In our model, this is where the

mapping of the power system into its equivalent communication system is done. Mapped

modules represent the physical devices in the power system like fuse, switch, substation, etc.

Also, the distance between physical devices are represented using channels in the

communication system. And the channel connection of the physical devices could be wired using

fiber or coaxial cables or it could be wireless using Wireless Local Area Network (WLAN),

Metropolitan Area Network (MAN), etc. In order to mimic the behavior of communication

network, the modules exchange information in a discrete manner.

3.4.2 INET Framework

Being an open-source software, the community of developers that use the OMNeT++ software

have been using the modules found in OMNeT++ to deepen special areas of network and related

simulation, and one of the areas that has received great attention is the Internet Protocol (IP)

communication. The IP suite is the set of computer networking models used on the Internet. To

this end, the INET framework for OMNeT++ has been evolving over time and solely serve as an

underlying framework for IP suite communication in the OMNeT++ simulation of systems that

use the packet switching protocol. Packet switching transmission can be classified into

Transmission Control Protocol/IP (TCP/IP) and User Datagram Packet (UDP). Below is a

description of each:

49

3.4.2.1 TCP/IP

A type of packet switched protocol that uses acknowledgement to ensure a reliable byte stream

transfer. If there is packet drop or non-delivery, a lack of acknowledgement will signal a

retransmit of the packet unless the number of retries is exceeded, which will force the sender to

return a transmission failure message. This type of protocol is employed in applications which

emphasizes reliability. It has the characteristics of client-server communication with sender

representing client9 and server10 representing receiver. TCP/IP is a connection-oriented protocol

and uses a virtual circuit connection method. Therefore, the resource is secured when it is

required and released after it is used. The connection process is made up of three phases

described below and shown in Figure 3-7:

9 Client is the computer that sends a connection request. The client has to know the IP address of the server and the port number in order to send the request for connection. 10 Server is the computer device which bind to a port to create a passive open connection and wait for connection requests from clients. When the clients establishes the connection, it becomes an active open connection.

50

1. Connection establishment: In order to create a connection between a client and a server, a

three way handshake11 to set the rules of the communication is established. This involves

the three steps below;

1. SYN: The client that wants to connect to the server sends the SYN to the server

and this turns a passive connection into an active connection. The client then sets

the segment number to random value A’

11 Handshake is a negotiation process in which computer devices that want to communicate dynamically set the conditions of the communication channel that links the two devices before the actual data transfer.

Figure 3-6 TCP Signaling diagram showing connection establishment, connection maintenance and connection

teardown

51

2. SYN-ACK: In reply to the client request, the server sends the SYN-ACK message

with SYN number incremented by 1 (i.e. A+1) and another random value chosen

by the server to serve as the packet sequence number.

3. ACK: In the last lap, the client sends an ACK to the server with the

acknowledgement number retained as A+1 but the sequence number modified to

B+1. In this state the connection is established and the next phase begins.

2. Connection maintenance: After the step one above, a full duplex communication is

established using the sequence numbers and the two devices can start transmitting and

receiving data to and from each other.

3. Connection teardown: Similar to the connection establishment, this uses a four way

handshake and the steps are outlined below and are shown in Figure 3-7.

1. FIN: The device that wants to close the connection initiates the active close by

sending a FIN message.

2. ACK: The remote device replies to FIN by sending an ACK to the remote device.

This is the last ACK from this device.

3. FIN: The remote device again sends its own FIN message to the initiator of the

close request.

4. Finally, the initiator device replies with its own last ACK message and the

connection is terminated.

Sometimes, this four way process is reduced to three thereby merging the two reply

messages from the remote device into one.

3.4.2.2 UDP

UDP is similar to the TCP/IP but does not use acknowledgement nor involve securing the

transmission channel for the entire packet before transmission. Rather, packets are sent out

individually and each packet is free to take any route to the destination. This presents the

challenge of packets arriving out of order and considerable resource is used in the receiver to

52

discern the order of the packets. Worse still, lack of congestion control in this protocol results in

some packets encountering a high delay and some packets even lost in transmission. Packet re-

sequencing is done at the receiver to reconstruct the original message unlike TCP in which all the

packets use the reserved channel and are usually serially transmitted, and where TCP packets

arrive out of order, the sequence number makes it easy to re-order the packets. This type of

packet switching is mostly used for applications that are tolerant of small packet loss, but

requires very low latency, and a typical example is voice communication. Also, it is deployed in

real time streaming service involving large clients as in IP television.

In OMNeT++, INET has been used to develop tested modules for TCP/IP and UDP network

simulations. Therefore, users can easily use the INET framework to design any network of their

choice using the INET framework as an underlying network module in OMNeT++. It is this

resource that we utilize in our system to build the communication protocol of any smart grid

system under study. The choice of UDP or TCP/IP in the protocol design of any system study

depends on which of the features of packet loss or latency is to be compromised. However, in all

of our test systems, we use the TCP/IP protocol.

3.4.3 C++ Interface

Because the OMNeT++ simulation framework is developed using the C++ programming

language, the interface to the scheduler is developed using the C++ programming language for

ease of integration. Our OMNeT++/INET has two interfaces to the two external simulators that it

is connected to and they are discussed below.

1. OMNET++ to PSCAD/EMTDC Interface: In order to be able to receive and send data to

the PSCAD/EMTDC using the socket communication discussed in the previous sections,

we integrate a socket which serves as a server and allows the PSCAD/EMTDC Interface

socket to make a connection request to it as a client. This establishes a full duplex

communication between the two simulators. And to ensure scalability, we use a dynamic

buffer in both ways to make sure there is no data corruption owing to buffer overflow.

2. OMNeT++ to MATLAB Interface: In reality, the communication network is in-between

the power system and the power management system (i.e. Control center). Hence, to

convey data from a power system to the control center during simulation, we develop a

53

bidirectional data transfer interface between the communication simulator and the control

simulator (MATLAB). To make its integration easy in a variety of programming

languages, MATLAB has an API which is easily used. However, the OMNeT++ is not

compatible with this API, so we use a socket communication between the OMNeT++ and

Visual Studio running a C program from which we make use of the MATLAB API to

start the MATLAB simulation engine. Therefore, using this software involves only

developing the control system in MATLAB and configuring the appropriate number of

variables in the C program running inside the Visual Studio.

3.5 Summary In this chapter, a software realization of the distributed simulation system is presented with

emphasis on the system level block components. The PSCAD/EMTDC houses the details of the

power system as well as the scheduler, software that is responsible for creating the event queue

by using the system history, and also regulates the alternating runs of the simulators. The

communication system is realized using the OMNeT++/INET framework and uses the TCP/IP

protocol to ensure reliability of data exchange, while the MATLAB/Simulink computational

software handles all the power management decisions required in the system. Also, the ability of

the system to accommodate the study of both a centralized and decentralized system is discussed.

With the scheduling system and the database, our system eliminates the use of fixed time-step

synchronization. Hence, there is no accumulated error during synchronization as each device

status is updated according to the timestamps of their events. Simulations in Chapter four will

confirm the effects of the scheduler in eliminating accumulated errors. Also, we developed a

variable buffering system in case the sensors have flexible sampling rates, and this has made the

system scalable and versatile.

54

Chapter 4

Test Cases: Microgrid Control and Communication A microgrid is a group of distributed energy generation and energy storage units, serviced by one

or more feeder downstream to a substation, and is capable of operating in main grid connected

mode, islanded mode and provide a two-way transition between the two modes [32]. The energy

generation units could be solar, wind turbine, thermal power plants, etc. The energy storage units

may be comprised of battery, flywheel, etc. The main feature of microgrid is its ability to

respond rapidly to energy fluctuations in the grid. To achieve this, an efficient control

mechanism has to be in place to regulate the operation of the grid. And suffice me to say that for

a hierarchical control structure, this is difficult to achieve without communication infrastructure.

Hence, the discussion of this chapter is about the operation of microgrid controllers and the role

of communication infrastructure in its operation. To demonstrate the operation, a time-domain

simulation of the power system in PSCAD and discrete simulation of the communication system

in OMNeT++ are performed and analyzed.

4.1 Study System: Microgrid In order to test every aspect of our system, we have chosen a microgrid because of the following

reasons:

• It is realistic due to the declining cost of energy storage and the increasing affordability

of renewable energy.

• It comprises all the key components that make up the smart grid:

o Control system

o Communication network

o Power system

55

• The coupling between the key components identified above presents a good opportunity

of studying the cyber-physical dependence of the components of the smart grid.

• The timescale for the reaction of controllers is within the range (milliseconds) we are

interested in.

According to Reference [11], latency defines the difference between the occurrence of a state

and when it was acted upon by an application, and this comprises communication delays and

others. Hence, from Table 2-2, transient stability is one of the smart grid applications and we

intend to do off-line time-domain simulation using the microgrid of Figure 4-1 to verify the

ability of the controller to stabilize the system after transient disturbance in the using

communication system vis-à-vis the results of [5]. The microgrid is a distribution system with a

large number of buses, feeder branches, and electric loads, thereby providing a more realistic

scenario to study the performance of the communication system and controllers. The microgrid

has 0.6 KV dispatchable electronically-interfaced Distributed Energy Resources (DER) units and

eleven loads which are connected to a 13.8 KV distribution system. A 1.5 KV DC voltage

source, a two-level voltage-sourced converter (VSC) and a series resistor-inductor (RL)

represents each of the DERs, which is connected to the main distribution feeder via a 0.6 KV/

13.8 KV step-up transformer at its point of common coupling (PCC). Below is the system data in

Table 4-1. All the testing is done with the system disconnected from the main grid by opening

the circuit breaker CBg.

56

Table 4-1 Microgrid Data showing the ratings of the DERs, load and line impedance.

Base Power Sbase 1.6 MVA

Base Voltage Vbase 13.8 KV

Size of DER units DER1:1.6 MVA DER2:1.2 MVA DER3: 0.8 MVA

Load Data L1 (0.08-0.94) L2 (0.16-0.98) L3 (0.17-0.90) L4 (0.17-0.94) L5 (0.25-0.96)

Line Impedance (0.35 + j0.31) ῼ/km

Transformer

Reactance

8%

RL Filter Z= 15% Quality Factor = 50

DER2

L2

2kmDER1

L1 CB1

2km

L5 2km2km

L4L2

7km

L1

CBg

Grid

DER3

2km

L22km

L1

L3 4km

3km4km

L3

2km 3km

Figure 4-1 Complete diagram of the study microgrid identifying the loads, the DERs, and the main grid.

57

From Figure 3-1, the study system is comprised of the Power management system, the DERs

with their local controllers, the distribution grid and the communication system. In the following

sections, these components are discussed.

4.1.1 Power Management

The power management system is adapted from [5] where DER1 and DER3 are designated as

power-controlled DER (PC-DER) units (their reactive and real power outputs are regulated)

while DER2 is designated as voltage-controlled DER (VC-DER) unit. This means that DER2

controls the voltage of its PCC and invariably the voltage profile of the whole grid. The choice of

DER2 as a VC-DER is because of its location within the studied system. Depending on the

distribution feeder length, the load information and configuration of the microgrid system,

several DERs can be designated as VC-DER to improve the load profile of the grid, while others

remain PC-DER. Using the communication link in Figure 3-1, the microgrid real/reactive power

is continually monitored, measured and transmitted to the PMS in Figure 3-1 so as to determine

the set points that are appropriate for the PC-DERs. The PMS uses the measured data and an

optimization scheme which ensures that load is shared among the generators according to their

capacity to determine the set points of the PC-DERs units, Pref,i and Qref,i, where i = 1,3. The

voltage set point of DER2 is also specified by the PMS by forcing it to maintain the voltage of its

PCC at a specified level like 1 per unit (p.u) with zero degree angle. The implication of the DER2

maintaining the voltage of its PCC is the result of Equation 4.1 below.

3,1,2,

3,1,2,

DERDERLOSSLOADDER

DERDERLOSSLOADDER

QQQQQPPPPP−−+=

−−+= (4.1)

Where PLoad/QLoad represents the total real/reactive power demand of the microgrid net load and

PLoss/QLoss is the total real/reactive power loss of the distribution feeder. Hence, the PMS controls

the power flow of the microgrid. The frequency at which the PMS performs its function is

dependent on the type of system under study. For microgrid, the measurement and control

functions by the PMS should be done often in order to ensure that the established power sharing

scheme among the DERs units is maintained. This is vital in ensuring the system handles

disturbances such as load disconnection or reconnection properly.

58

4.1.2 Frequency Control and Synchronization

Open-loop control is adopted for regulating the frequency of the microgrid [33]. An oscillator is

built inside the local controllers (LC) of each DERi and it generates a 60 Hz sawtooth waveform

with the following pattern ( ) ,ooi dt θτωθ += ∫ where oo fΠ= 2ω and of is the nominal power

frequency of the microgrid.

Figure 4-2 Sawtooth Waveform used to regulate the output of the DERs.

Figure 4-2 shows the sawtooth waveform which is obtained from the oscillator of the local

controllers and is used for the abc(dq) to dq(abc) transformation of the mathematical model of

the DERi. Based on the proposed control strategy in [33], a globally synchronized signal is

communicated to all the DER units via a global positioning system (GPS) that is described in

[34]. The importance of synchronizing the signal to all the DER is to;

• Check drift among the local controllers,

• Make ready the incoming DER units

Also, crystal oscillators with a high degree of accuracy are used in the local controllers so as to

be able to accurately regulate the controllers in the event of unavailability of the synchronizing

signal (communication network failure).

4.1.3 Local Controllers

The system uses a hierarchical control structure in which the PMS delivers power set points to

the local controllers, and the local controllers track the set points. The set points are transmitted

via the communication system overlay. Therefore, the LC of each DERi performs its control

59

function by rejecting disturbances and tracking the specified value from PMS. To achieve this,

the LCi takes the measured value of the three-phase voltage from its corresponding PCC,

transforms it into a dq frame signal using the phase-angle signal iθ that is generated by its

internal oscillators and synchronized with the global time-reference signal received from the

GPS. The details of the abc/dq and dq/abc transformation can be found in [35] and the main

reason for the transformation is to decouple the state variables for ease of control. Also, the

received voltage magnitude and angle set points [ ] iirefV δ∠, , are transformed to the dq frame to

produce dq-based reference signals iirefirefd VV δcos,,, = and iirefirefq VV δsin,,, = . The LCi then

uses the measured and reference values received to determine the dq voltage components, i.e.,

itdV , and itqV , of the terminal of the Voltage-Source Converters (VSC) in each DERi. This dq

terminal voltage is then used to generate the terminal voltage HViabctv ,,, at the high voltage side of

the transformer. In order to obtain the low voltage side of the transformer LViabctv ,,, , HViabctv ,,, is

divided by the turns-ratio of the transformer and shifted if need be. This low voltage value

LViabctv ,,, is then fed to a pulse width modulated (PWM) signal generator of the interface VSC of

the DERi. According to [33], the controller is capable of accounting for the mismatch between

realistic and theoretical turns-ratio and phase shift of the transformer.

4.2 Communication Architecture Mapping the equivalent communication of the power system under study can range from being a

simple task to a fairly complex one. This is because of the array of options available in the

design of the communication system. As outlined in Chapter 3, several considerations like the

volume of traffic, the response time, the type of transport channel (TCP, UDP), the geographical

area of the power system under study, wired or wireless channel, etc., form cardinal points in

making decisions about the structure and components of the communication system. In our case,

however, it is a fairly simple decision because the number of measurement points (3) is small,

the geographical spread of the grid is also small, and we know that the system requires high

reliability, hence the choice of TCP instead of UDP.

60

Figure 4-3: Communication system equivalent of the microgrid showing the interconnection

amongst the key components of the system.

4.2.1 Geni LAN

Geni LAN is the combination of the local controllers and the sensing devices which repeatedly

take the PCC readings of each DER, and it’s located within the LAN of each generator. In our

case, the real and reactive power components at the PCCs are continuously sensed and

transmitted to the power management system via the fiber optic cable. Also, the voltage profile

of the grid is monitored by the sensors in Gen2 to ensure it remains within the specified operating

range. Therefore, Geni Host is both a traffic source and traffic sink as sensed data emanate from

it and control data from the EMS is utilized by the local controllers inside it.

4.2.2 Energy Management System

As the name suggests, this houses the central controller, which manages the set points for each of

the local controllers. In our case, this is an interface to the MATLAB software and consists of

sockets for sending the sensed values to MATLAB and receiving the control decisions for

downward transmission to the local controllers.

61

4.2.3 Communication Channel

In Figure 4-3, all the vital components are connected by communication links or channels. The

channel could be optical fiber, coaxial cables, wireless radio or a combination. The choice of the

communication channel is influenced by the geographical spread of the grid, the bandwidth, and

the speed of transmission deemed appropriate for the power system. In a microgrid, the dynamics

of the system changes quickly and ought to be supported by a very fast communication channel.

Hence, in our model, we have made the connection using a fiber link. The capacity of the

channel is determined based on the required transmitted data. For standard sensing devices, the

transmitted data is always greater that the sensed data because of overhead, hence we made

provision to accommodate such discrepancy as evidenced in the table below, which is a

summary of the transmitted data as well as the channel capacity.

Table 4-2: Packet Structure of transmitted data, channel capacity, channel types and their characteristics

Timestamp Source

Address

Destination

Address

Measurement Total

Bits 64 32 32 64 192 Bits

Channel

Capacity

100 Mbps

Channel type Fiber Ethernet Real-time

WiMAX

Non-Real-Time

WiMAX

Channel

Delay

3.3μs 4.8-5.3μs 30-75ms 100ms

As can be seen from Table 4-2, the transmitted data is far below the channel capacity as we

imagine that in reality, it is not likely to be a homogenous traffic. For simulation purposes, the

capacity is easily adjusted to match any desired study by making minor changes in the

configuration file. However, in our simulation, we tested for many varying values of channel

delay just in case Ethernet, WiMAX is considered for use instead of fiber.

62

4.3 Simulation Results In order to demonstrate the usability of the multi-domain simulator developed, we consider the

off-line time domain simulation of the microgrid in Figure 4-1 under three different conditions.

First, we run the simulation as obtainable in [36] in which there is no communication medium,

but we incorporate the automated power management system instead of manually updating the

set points as done in the paper. This is to ensure that we are validating the behavior of the co-

simulator against a credible alternative. Second, we run a simulation in which there is no

interaction between the communication system and power system, but fixed average values of

the communication delays are extracted and hard coded into the power simulator to simulate the

impact of communication delays on the power system. Finally, we run a full multi-domain

simulation in which there is interaction among the simulators as the simulation progresses and

we discuss the performance of the controller as well as the observed similarities and differences

in the three scenarios. In all of the simulations, we start the simulation with the system loaded as

labeled in Figure 4-1. At 0.8s of the simulation time in PSCAD, the CB1 is disconnected and we

monitor how the controllers react to maintain the set objectives. When the system has settled, we

reconnect the load at 1.5s of the simulation time in PSCAD and we observe the system dynamics

again.

4.3.1 Performance metrics

At the beginning of this report, we identified different challenges that make co-simulation

difficult to include:

• Synchronizing the events from the two simulators to ensure that accumulated delays are

reduced or eliminated.

• Using effective and secure data exchange mechanism between the simulators.

• Scalability of the system developed

• And the ease of use of the co-simulator

63

In the simulation results that follow, we will measure the performance of the co-simulator in

solving the synchronization problem, discuss the scalability of the simulator and how user

friendly it is.

4.3.2 Single-domain simulation

Most studies of the microgrid typically focus their attention on investigating specific operational

aspects of the system like the control system, communication system, etc., while assuming

perfect conditions for the other areas of operation. As observed already, [36] developed a control

strategy for the microgrid and ran the off-line time-domain simulation as well as the real-time

simulation to validate the performance of the controller. The aim of the controller includes

regulating the power profile of the grid, regulating the grid voltage and sharing power output

among the generators according to their individual capacity. It equally recommended that

communication between the local controllers and the central power management system should

be done often to capture fast changing dynamics in the grid. However, the set point was

manually updated only twice during the course of the simulation. And the controller performed

as expected. So in our model, we have automated the power management system and enable it to

supply the set points more often as can be obtained realistically and see the impact of sampling

rate on the controller. It should however be noted that this work is not about optimizing the

controller designed in [33]. Rather, this model of the simulation will enable us to observe and

compare two things;

• The impact of sampling rate on the performance of the controller vis-à-vis the original

simulation done in [36]

• And also compare the effect of the communication delay as discussed in Sections 4.4.3

and 4.4.4.

4.3.3 Fixed-delay Single-domain simulation

We recognize that some study hard code the feature of the communication system into the power

system. This is usually done by extracting the average values of the communication parameter

and programming them into the behavior of the power system. So, in the second scenario of the

simulation, we independently run the communication system equivalent of our power system and

extract the average values of the communication delay and hard code them into the PSCAD

64

setup of the power system and repeat the simulation. The major difference between this and

Section 4.4.2 is the fixed communication delay which is absent in the latter.

4.3.4 Multi-domain simulation

In this section, we simulate the full system interaction between all the simulators used in the

design. This enables changes in any of the system to affect the behavior of other systems during

the simulation run. This scenario resembles Section 4.4.3 save for the fact that none of the

communication parameters are fixed. They are rather obtained as a response from the

communication system during the simulation.

4.4 Impact evaluation Having briefly discussed the simulation setup in Section 4.4, we discuss the results and make

deductions from the results.

4.4.1 Effect of sampling rate on controller

Reference [5] designed microgrid control and mentioned that the updating of the setup points

should be performed frequently, but it is silent on how often this should be done. So, we run a

series of simulation using the three scenarios discussed in Section 4.4 to determine at which rate

the controllers could be able to achieve the sharing scheme defined by the central controller that

provide the set points and equally maintain the power and voltage profile of the grid. In this

scheme, the EMS aims to ensure that DER’s power output is proportional to their capacity while

the local controllers aim to regulate the power and voltage profile of the grid. The knowledge of

this rate and the requirements of the system under study enables the designer to fine tune the

controller until the design aim is achieved. To demonstrate this, we run the islanded microgrid

and at 0.8s simulation time in the power simulator, we disconnect the CB1 shown in Figure 4-1

and we monitor the ability of the system local controllers in coordination with the EMS to

stabilize the system by reassigning the power output of the DERs and tracking the set points.

Also, at 1.5s simulation time in the power simulator, we reconnect the load and again monitor

the performance. The DER1 and DER3 regulate the power profile while DER2 regulates the

voltage profile of the grid. Below are snapshots from the simulation.

65

Figure 4-4 represents the real and reactive power components of the three generators for 1ms,

10ms and 25ms sampling rates. From the graph, it can be observed that as the sampling rate

decreased, the maximum deviation of the real and reactive power components of the power

controlled DER’s reduced. Generator three (Figure 4-4 c) has real power peak of 0.3 p.u. during

the load reconnection, and generator one dynamics was within the 1.0 p.u. throughout the

reconnection dynamics. However, a look at the generators for the 1ms (Figure 4-4 a) and 10ms

(Figure 4-4 b) shows that during the load reconnection, the spike of generator one real power is

above the rated 1 p.u. while generator three real power is equally higher than that of the 25ms

sampling rate. This demonstrates the effect of sampling rate on a controller action for this

scenario. The overall relevance is discussed at the end of the co-simulation example.

Figure 4-4: Single domain simulation showing the response of the DGs to transient instability

(load shedding)

66

Figure 4-5:Non-interactive simulation showing the response of the DGs to transient instability

(load shedding)

As with Figure 4-4, we have shown the results of increasing the sampling rates and it could be

seen that the Figures 4-4 and 4-5 are similar in behavior. This can be explained by the

geographical spread of the microgrid and the method of extracting the delay values. The

microgrid is usually a couple of kilometers in radius. Hence, the delay values are not expected to

be high, especially with no background traffic. However, while the Figure 4-5c behavior is

similar to Figure 4-4c which has reduced peak values for the power-controlled DER’s during the

load reconnection, the effect of the delay became apparent as the sampling rate reduced. From

Figure 4-5a to Figure 4-5c, it is obvious that the settling time increased as compared to Figure 4-

4. This is obviously as a result of the impact of the delay which affects the updating of the set

points. Hence, we could see that in Figure 4-5c that the reconnection dynamics settled after 1.8s

of the simulation time, which is farther than its counterpart in Figure 4-4c.

67

Figure 4-6: Full co-simulation showing the response of the DGs to transient instability (load

shedding)

In Figure 4-6, the results are similar to that of Figure 4-5 as the effect of the co-simulation delay

is manifest. However, as the sampling rate is reduced as evidenced in Figure 4-6c, the dynamics

of the power-controlled DER’s seemed smoother as in Figure 4-5c but the dynamics settled

faster than that of Figure 4-5c. As seen in the graph, the dynamics of all the generators settled

before 1.8s after load reconnection at 1.5s.

Also, we investigated the behavior of the controller using different communication network

protocols like WiMAX. To set this up, we fixed the sampling rate at 5ms and used the range of

the channel delay for real and non-real time communication via wireless channels like WiMAX

to simulate and see the effects of using such channels in delivering the set points as well as the

system state values to the energy management system. Below is the graph of this simulation for

30ms-75ms channel delay, which represents the real time communication on the wireless

channels.

68

Figure 4-7 is similar to both Figures 4-6 and 4-5 as the behavior is almost same in terms of the

generator dynamics during load disconnection and reconnection. Also, the settling time of the

dynamics is similar. Therefore, we argue that the effect of increasing the maximum channel

delay has minimal effect in this case because the transmitted bits is far smaller than the channel

capacity, there is no background traffic in the system and there are no intermediate processing

nodes as the system uses point-to-point communication. However, we are not ignorant of the fact

Figure 4-7: Real time simulation of the wireless channel and the power components of the DERs

69

that this simulation environment has made a simple assumption of no background traffic, but we

hope to incorporate this in future work.

Finally, we simulate the behavior of the system for other high latency applications in smart grid

and observe the behavior. We use 100ms maximum channel delay as that is the average delay for

wireless non-real time applications. Below is the graph.

As with Figure 4-5 and Figure 4-6, Figure 4-8 exhibited similar behavior because of the same

reasons we have adduced above.

4.4.2 Discussion of results

We discuss the performance of the co-simulator using the set objectives of Section 4.3.1.

Figure 4-8: Non-real time wireless channel simulation and the reaction of the DERs to load connection and

disconnection

70

4.4.2.1 Elimination of synchronization delay

A direct comparison between Section 4.4.2 and Section 4.4.4 immediately reveals the effects of

the co-simulator. The single domain simulation behaves like a time-stepped synchronization in

which there is a fixed time-step when the status of the IEDs’ in the power system are updated as

can be seen from Figure 4-4. Under this condition, the status of the generators are updated at

same time. However, in Figure 4-6, we succeeded in creating queued events made up of a mix of

the power and communication events using the scheduler. Hence, events are served according to

their time stamps and this is seen in the graph as each of the generators are updated according to

the timestamp of its event.

This technique ensures that no event waits for a fixed time-step but rather dynamically updates

its’ IED as at and when due. There is however a small error based on the fact that the timestamps

may not be equal to the PSCAD simulation time during synchronization. The source of this error

can be seen from the data structure below which is utilized for the synchronization;

71

Figure 4-9: Pseudocode showing the data structure used by the scheduler during

synchronization

The use of the open-source embedded database called SQLite facilitated this approach as it

helped us to have a temporary history of events from which the scheduler makes the decision of

which system or IED to execute its event. This error is however very small as the PSCAD time-

step is usually very small. Also, care is taken in choosing the time-step of the simulation to

ensure that two successive events in a particular IED does not fall in same synchronization

window. We used the knowledge of the round trip time (RTT) of the ping signal in the

equivalent communication network of the power grid to determine the accurate time-step. A

closer look at the graphs of Figure 4-4 to Figure 4-6 reveals two interesting scenarios discussed

below:

1. That sampling rate has effect on the behavior of the controller. In all, Figure 4-4c, Figure

4-5c, and Figure 4-6c have a more graceful reaction to the dynamics especially during the

transients experienced during load reconnection. The progressive reduction of the

sampling rates could represent a delay in delivery of information in the communication

72

channels or simply trying to determine the average set points per second which the

controller can track successfully. The delay in delivery of information can be attributed to

a node or communication failure in the communication network or a cyber-attack in the

communication network. So the system can be used to determine the limit at which the

controller can track the set points under different scenarios.

2. Also, the effect of the co-simulation is manifest in Figure 4-5c, Figure 4-6c as against

Figure 4-4c. It is evident that the delay experienced using the communication channel

does not portend a negative effect all the time as the updating of the generator controllers

according to their respective timestamps helped to allow for graceful reaction of the

system to the transient and reduced the peak of the spike experienced by the two power

controlled DERs. This delay arises in the communication channel because the generators

are not same distance apart, and contention for service in the EMS also introduces delay.

This spread in reaction of the generators to transient means that proximity of the

generators to the EMS has to be influenced by the capacity of the generators, and priority

routing at the EMS can also be employed to ensure that high capacity generators

experience less delay than the low capacity ones to avoid tripping or damaging of

generators during transient.

We limited the rate to 25ms because of the characteristics of the microgrid which requires fast

control operation. For other kinds of systems in the smart grid, the rate could be extended to

represent heavy traffic in the communication channel which could increase the delivery time of

information via the channel.

4.4.2.2 Reliable data exchange

Due to the high reliability required in the control and management of the microgrid, we used the

TCP/IP protocol as the protocol for the communication of information between devices in the

system. Although this tends to have higher latency than UDP, the reliability gained is worth it

because this is a time-critical application.

4.4.2.3 Scalability of the system

From the design point of view, our system is scalable. The only seen limitation to scalability is

the amount of sensor information which the buffer can handle at a time. But we used a dynamic

73

buffer to ensure that only system resources can limit the amount of information that can be sent

at same time. Hence, the only configuration information that is supplied in this regard is an

estimate of the number of sensors in the system, and the user can just supply a high value if this

system is developed by another person.

4.4.2.4 Ease of use

Our system is very easy to use and requires minimal interaction with the user. This simplification

is done to ensure that users see the system as a black box, and can effectively test their system on

the test bed with minimum information about the details of the co-simulator. However, the user

will have to supply the sampling rate as an array as well as a value greater than or equal to the

number of sensors in the system under consideration. For systems that use synchronized rates

like PMUs, only one of the rates is supplied and the system will populate the array accordingly.

It should be mentioned that while the power system interaction is as generic as possible, the

default communication mapping of the power system uses point-to-point communication and this

can be modified easily if the user deems it fit as the OMNeT++ modules already developed are

reusable components and can be used to effectively achieve the desired topology with minimal

difficulty.

4.5 Summary We focused on the controller behavior of the microgrid in this chapter using sensor sampling rate

and round trip time to study the behavior of the controller. And we demonstrated the impact of

doing full co-simulation as against the single domain simulation by showing and discussing the

effect of the communication network delay. Also, we showed different tests that can be done

using the co-simulation platform. Finally, the effect of ignoring the impact of the communication

network by doing only single domain simulation was discussed.

However, there is a limitation with our test bed which we believe gives room for improvement.

And that is the fact that most communication network engineers are not power system engineers.

Hence, knowledge of the two systems is required if one needs to map any power system under

study to its communication system equivalent and use the co-simulator as a test bed. Therefore,

developing an automated system that can map any given power system to its communication

74

network equivalent is one way of making the communication system more flexible as point-to-

point transmission may not be ideal for all scenarios.

Another obvious challenge is the geographical spread of microgrid which is small. Hence, the

difference in delay values are not that appreciable. But because our system can handle analysis of

power systems with the generators several kilometers away, we speculate that the co-simulation

effects will be more pronounced under such condition, and this will be captured in future works

in this area.

Despite the limitations above, our system showed consistency in its behavior and we believe it

can serve its’ purpose as co-simulator for power, control and communication networks.

75

Chapter 5

Conclusion

5.1 Summary and Conclusions This dissertation proposes a software-based co-simulator for smart grid systems. And the system

is comprised of:

1. A power system simulator called PSCAD/EMTDC which is utilized in designing the

power system component of the smart grid. It is also responsible for some logic control

action in the power system as it has IEDs.

2. A communication system simulator called OMNeT++ which is used to design the

equivalent topology of the power system in the communication domain for effective

communication between devices in the smart grid.

3. A high level computational software called MATLAB which is used by the PMS of the

smart grid to regulate the local controllers when hierarchical control scheme is employed

in the design.

4. A scheduling system which schedules the events generated by the power and

communication systems according to their timestamps and utilizes the SQLite database

for keeping history of the system events.

The elimination of accumulated delays resulting from synchronization of the simulators is

discussed. The design of the scheduler, which is used to eliminate the synchronization delays,

using C/C++ is explained. The scheduler has two interfaces:

1. The C interface which the scheduler uses to communicate with the PSCAD/EMTDC.

2. The C++ interface which the scheduler uses to interface to the OMNeT++.

76

The development of these interfaces is also discussed. To increase the scalability of the system,

we developed a dynamic buffer to ensure that the scale of system is only limited by hardware.

5.2 Challenges and Limitations In the course of the dissertation, we had some challenges both in design decisions and

functionality. Some of them are discussed below.

1. Our major challenge is the inclusion of background traffic in the communication protocol

developed in OMNeT++. This is because INET modules used to carry traffic has not yet

evolved to carry actual traffic. Rather, it uses a simple messaging system. Therefore,

analysis like congestions in the link, packet drop ratio were not tested at this time.

2. Also, we are concerned that mapping the power system requires considerable knowledge

of programming of the OMNeT++/INET modules by the user. For instance, the number

and position of IEDs in the power system must be captured in communication network

correctly and this requires some modifications in the existing communication network as

our default number of IEDs in the communication system is six. However, if the user has

IEDs less than or equal to six, and is comfortable using the point-to-point topology, then

all that is required of the user is modifying the distance between the IEDs according to

the power system topology.

3. Another obvious challenge is in choosing the time-step of the PSCAD to ensure that the

synchronization algorithm does not have successive IED events in same window. We are

able to overcome this by reducing the time-step to a small value using the RTT delay

value we got for the non-interactive simulation as a guide. So, our concern here is that the

user has to be aware of this and may need to get the average value of the RTT before

fixing the time-step of PSCAD and this again requires some experience using the

communication simulator.

77

5.3 Contributions The salient contributions of this thesis are as follows:

• It proposes and develops a co-simulator for smart grid system and it is made up of power

management system, communication system, and power system.

• It proposes a scheduling system that solves the synchronization delays which occurs

when fixed-time-step synchronization technique is used.

• It also uses the concept of dynamic buffer to ensure that a user of the system does not

have to modify the co-simulator to suit the system under test. Hence, results in a scalable

platform that is limited only by hardware.

• The system is used to demonstrate the effect of delay and sampling rate on a microgrid.

These features along with the secure data exchange between components make our system ease

to use, and scalable as demonstrated in Chapter four where the off-line simulations are carried

out.

5.4 Future Research Directions Our future work in this direction includes:

1. Incorporating background traffic in the communication channel to enable us do more

analysis. This may involve replacing the OMNeT++ communication network with a

commercial communication network simulator that carries actual traffic like OPNET.

This is necessary for us to study the effects of network congestion, link/node failures,

priority routing, cyber-attacks, etc. on the power system components and generators.

2. We also intend to move from off-line time domain simulation to real-time time domain

simulation. And with the capability of OPNET in this regard, coupled with a new real-

time power simulator we acquired recently, we believe this is realistic.

3. We intend to use software defined networks in the communication system to abate the

effect of cyber-attacks in smart grid.

78

References

[1] D. Karlsson, M. Hemmingsson and S. Lindahl, "Wide area system monitoring and control,"

Power and Energy Magazine, IEEE, vol. 2, pp. 68-76, 2004.

[2] M. Chenine, J. Ullberg, L. Nordstrom, Y. Wu and G. N. Ericsson, "A Framework for Wide-

Area Monitoring and Control Systems Interoperability and Cybersecurity Analysis," IEEE

Trans. Power Delivery, vol. 29, pp. 633-641, 2014 APRIL, 2014.

[3] J. Richards, Cyber-War: The Anatomy of the Global Security Threat. Gordonsville:

Macmillan [Distributor], .

[4] Mingkui Wei and Wenye Wang, "Greenbench: A benchmark for observing power grid

vulnerability under data-centric threats," in INFOCOM, 2014 Proceedings IEEE, 2014, pp.

2625-2633.

[5] A. H. Etemadi, E. J. Davison and R. Iravani, "A Generalized Decentralized Robust Control of

Islanded Microgrids," Power Systems, IEEE Transactions On, vol. 29, pp. 3102-3113, 2014.

[6] W. Li, A. Monti, M. Luo and R. A. Dougal, "VPNET: A co-simulation framework for

analyzing communication channel effects on power systems," in Electric Ship Technologies

Symposium (ESTS), 2011 IEEE, 2011, pp. 143-149.

[7] Hua Lin , S. S. Veda, S. S. Shukla, L. Mili and J. Thorp, "GECO: Global Event-Driven Co-

Simulation Framework for Interconnected Power System and Communication Network," IEEE

Trans. Smart Grid, vol. 3, pp. 1444-1456, 2012 SEPT., 2012.

[8] A. Hahn, A. Ashok, S. Sridhar and M. Govindarasu, "Cyber-Physical Security Testbeds:

Architecture, Application, and Evaluation for Smart Grid," Smart Grid, IEEE Transactions On,

vol. 4, pp. 847-855, 2013.

[9] K. Hopkinson, Xiaoru Wang, R. Giovanini, J. Thorp, K. Birman and D. Coury, "EPOCHS: a

platform for agent-based electric power and communication simulation built from commercial

off-the-shelf components," Power Systems, IEEE Transactions On, vol. 21, pp. 548-558, 2006.

79

[10] M. D. Ilic, "From Hierarchical to Open Access Electric Power Systems," Proceedings of the

IEEE, vol. 95, pp. 1060-1084, 2007.

[11] P. Kansal and A. Bose, "Bandwidth and latency requirements for smart transmission grid

applications," 2013 IEEE Power & Energy Society General Meeting, pp. 1-1, 2013 JULY, 2013.

[12] L. L. Lai, Chong Shum, Linyu Wang, W. H. Lau, N. Tse, H. Chung, K. F. Tsang and

Fangyan Xu, "Design a co-simulation platform for power system and communication network,"

in Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference On, 2014, pp.

3036-3041.

[13] M. Levesque, Da Qian Xu, G. Joos and M. Maier, "Co-simulation of PEV coordination

schemes over a FiWi smart grid communications infrastructure," in IECON 2012 - 38th Annual

Conference on IEEE Industrial Electronics Society, 2012, pp. 2901-2906.

[14] V. Liberatore and A. Al-Hammouri, "Smart grid communication and co-simulation," in

Energytech, 2011 IEEE, 2011, pp. 1-5.

[15] N. Moskalenko, Z. A. Styczynski, T. Sokolnikova and N. Voropai, "Smart grid — german

and russian perspectives in comparison," in Modern Electric Power Systems (MEPS), 2010

Proceedings of the International Symposium, 2010, pp. 1-7.

[16] J. Nutaro, P. T. Kuruganti, L. Miller, S. Mullen and M. Shankar, "Integrated hybrid-

simulation of electric power and communications systems," in Power Engineering Society

General Meeting, 2007. IEEE, 2007, pp. 1-8.

[17] C. Molitor, S. Gross, J. Zeitz and A. Monti, "MESCOS—A Multienergy System

Cosimulator for City District Energy Systems," IEEE Trans. Ind. Inf., vol. 10, pp. 2247-2256,

2014 NOV., 2014.

[18] Zhang Haiyang and Li ShanDe, "Research on microgrid," in Advanced Power System

Automation and Protection (APAP), 2011 International Conference On, 2011, pp. 595-598.

[19] Rukun Mao, Huijuan Li, Yan Xu and Husheng Li, "Wireless communication for controlling

microgrids: Co-simulation and performance evaluation," in Power and Energy Society General

Meeting (PES), 2013 IEEE, 2013, pp. 1-5.

80

[20] Kun Zhu, M. Chenine and L. Nordstrom, "ICT Architecture Impact on Wide Area

Monitoring and Control Systems' Reliability," Power Delivery, IEEE Transactions On, vol. 26,

pp. 2801-2808, 2011.

[21] Xiaoyang Tong, "The co-simulation extending for wide-area communication networks in

power system," in Power and Energy Engineering Conference (APPEEC), 2010 Asia-Pacific,

2010, pp. 1-4.

[22] D. Kundur, Xianyong Feng, Shan Liu, T. Zourntos and K. L. Butler-Purry, "Towards a

framework for cyber attack impact analysis of the electric smart grid," in Smart Grid

Communications (SmartGridComm), 2010 First IEEE International Conference On, 2010, pp.

244-249.

[23] A. AlMajali, A. Viswanathan and C. Neuman, "Analyzing resiliency of the smart grid

communication architectures under cyber attack." in CSET, 2012, .

[24] P. M. Menghal and A. J. Laxmi, "Real time simulation: A novel approach in engineering

education," in Electronics Computer Technology (ICECT), 2011 3rd International Conference

On, 2011, pp. 215-219.

[25] I. Herrera, Mathematical Modeling in Science and Engineering: An Axiomatic Approach.

Hoboken, N.J.: John Wiley, 2012.

[26] Q. Chen and V. Hans, "A new procedure for structural modeling of linear and non-linear

systems," European Journal of Mechanics - A/Solids, vol. 17, pp. 139-151, 1998.

[27] P. H. Sørensen, E. Hendricks and O. Jannerup, Linear Systems Control. Springer Berlin

Heidelberg, 2008.

[28] Anonymous Principles of Modeling and Simulation. Wiley, 2009.

[29] C. M. Davis, J. E. Tate, H. Okhravi, C. Grier, T. J. Overbye and D. Nicol, "SCADA cyber

security testbed development," in Power Symposium, 2006. NAPS 2006. 38th North American,

2006, pp. 483-488.

81

[30] M. Mallouhi, Y. Al-Nashif, D. Cox and T. Chadaga, "A testbed for analyzing security of

SCADA control systems (TASSCS)," in ISGT 2011, 2011, pp. 1-7.

[31] Yuan Hong and Gu Ping-ping, "Application of windows inter-process communication in

software system integration," in Intelligent System Design and Engineering Application

(ISDEA), 2010 International Conference On, 2010, pp. 375-378.

[32] Yi Luo, Longjun Wang, Gelan Zhu and Gang Wang, "Network analysis and algorithm of

microgrid reliability assessment," in Power and Energy Engineering Conference (APPEEC),

2010 Asia-Pacific, 2010, pp. 1-4.

[33] A. H. Etemadi, E. J. Davison and R. Iravani, "A Decentralized Robust Control Strategy for

Multi-DER Microgrids—Part I: Fundamental Concepts," Power Delivery, IEEE Transactions

On, vol. 27, pp. 1843-1853, 2012.

[34] Anonymous "IEEE Standard for Synchrophasors for Power Systems," IEEE Std C37. 118-

2005 (Revision of IEEE Std 1344-1995), pp. 0_1-57, 2006.

[35] A. Yazdani, Voltage-Sourced Converters in Power Systems : Modeling, Control, and

Applications. Hoboken, N.J.: Wiley, 2010.

[36] A. H. Etemadi, E. J. Davison and R. Iravani, "A Decentralized Robust Control Strategy for

Multi-DER Microgrids—Part II: Performance Evaluation," Power Delivery, IEEE Transactions

On, vol. 27, pp. 1854-1861, 2012.

A Multi-domain Co-Simulator for Smart Grid: Modeling Interactions in Power, Control ... ·...

Documents

Transcript of A Multi-domain Co-Simulator for Smart Grid: Modeling Interactions in Power, Control ... ·...