SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

194
SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED NETWORKED CONTROL SYSTEMS WITH MOBILE OBSERVERS USING HETEROGENEOUS CONTROLLERS BY JAVITT HIGMAR NAHITT PADILLA-FRANCO, B.S., M.S. A dissertation submitted to the Graduate School in partial fulfillment of the requirements for the degree Doctor of Philosophy, Engineering Specialization in Electrical Engineering New Mexico State University Las Cruces New Mexico December 2017 (c)2017 by Javitt Higmar Nahitt Padilla-Franco

Transcript of SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Page 1: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED

NETWORKED CONTROL SYSTEMS WITH MOBILE

OBSERVERS USING HETEROGENEOUS

CONTROLLERS

BY

JAVITT HIGMAR NAHITT PADILLA-FRANCO, B.S., M.S.

A dissertation submitted to the Graduate School

in partial fulfillment of the requirements

for the degree

Doctor of Philosophy, Engineering

Specialization in Electrical Engineering

New Mexico State University

Las Cruces New Mexico

December 2017

(c)2017 by Javitt Higmar Nahitt Padilla-Franco

Page 2: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

ProQuest Number: 10759028

All rights reserved

INFORMATION TO ALL USERS The quality of this reproduction is dependent upon the quality of the copy submitted.

In the unlikely event that the author did not send a complete manuscript and there are missing pages, these will be noted. Also, if material had to be removed,

a note will indicate the deletion.

ProQuest 10759028

ProQuestQue

Published by ProQuest LLC(2018). Copyright of the Dissertation is held by the Author.

All rights reserved.This work is protected against unauthorized copying under Title 17, United States Code.

Microform Edition © ProQuest LLC.

ProQuest LLC 789 East Eisenhower Parkway

P.O. Box 1346 Ann Arbor, Ml 48106-1346

Page 3: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

“Secure Framework for Implementing Distributed Networked Control Systems

With Mobile Observers Using Heterogeneous Controllers ” a dissertation prepared

by Javitt Higmar Nahitt Padilla-Franco in partial fulfillment of the requirements

for the degree, Doctor of Philosophy, Engineering, has been approved and accepted

by me following:y m e following: A

(ftvt I/Kts)r. Louf-Vicente Reyes QDr

Dean of the Graduate School

Dr. Nadipuram (Ram) Prasad Chair of the Examining Committee

Date

Committee in charge:

Dr. Nadipuram (Ram) Prasad, Chair

Dr. Jeffrey Beasley

Dr. Jaime Ramirez-Angulo

Dr. Satishkumar J. Ranade

Page 4: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

DEDICATION

I dedicate this work to my mother, Lucrecia.

Page 5: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

ACKNOWLEDGMENTS

I would like to thank my advisor, Dr. Nadipuram (Ram) Prasad, for giving

me the opportunity to work on his Energy Harvesting project, a true life-changing

experience. Personally, I would like to thank him for sharing his knowledge which

has enriched my study in philosophy. Additionally, I would like to give special

thanks to the next professors of the Klipsch School of Electrical and Computer

Engineering: Dr. Satish Randade, Charles (Chuck) Boehmer, Dr. Xenxin Liu

(Lehigh University), Dr. Paul Furth, Dr. Jaime Ramirez-Angulo, and Dr. Robert

Paz (RIP).

IV

Page 6: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

VITA

Jan. 21, 1977 Born in Chihuahua, Chihuahua, Mexico

1994-1999 B.S. Computer Systems EngineeringChihuahua’s Technological Institute II. Chihuahua, Mexico

2000-2001 M.S. Electronics EngineeringChihuahua’s Technological Institute. Chihuahua, Mexico

2002-2003 M.S. Computer EngineeringUniversity of Texas at El Paso. El Paso, Texas

2002-2003 Assistant Instructor and Research AssistantUniversity of Texas at El Paso. El Paso, Texas

2004-present Researcher and ProfessorAutonomous University of Ciudad Juarez. Cd. Juarez, Mexico

2008-2010 M.S. Computer ScienceNew Mexico State University. Las Cruces, New Mexico

2012-2015 Research Assistant and Teaching AssistantNew Mexico State University. Las Cruces, New Mexico

2010-2017 Ph.D. Electrical and Computer EngineeringNew Mexico State University. Las Cruces, New Mexico

PROFESSIONAL AND HONORARY SOCIETIES

Institute of Electrical and Electronics Engineers and Association for Computing Machinery

v

Page 7: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

PUBLICATIONS

- Javitt H.N. Padilla Franco, Jorge Perez et al. Control de altitud de un cuadri- coptero utilizando logica difusa. CULCyT/Logica Difusa. 2016.

- Sheret Tilvaldyev, Javitt H.N. Padilla Franco et al. Analysis of the aircraft coating innovations, improving aerodinamics parameters and reduciong sking friction. Academic Journal of Science. 2016.

- Onofre Amador Morfin Garduno, Javitt H.N. Padilla Franco et al. Diseno de un sistema de control descentralizado de un sistema eolico emulado con un grupo motor CD-Generador CA. Coleccion de reportes tecnicos de investigacion. 2015.

- Alejando Monroy Reyes, Javitt H.N. Padilla Franco et al. Sistema de moni- toreo remoto con detecion de movimiento basado en vision por computadora. CULCyT/Sistema de Vision. 2015.

- Octavio Rodriguez, Javitt H.N. Padilla Franco et al. Rediseno, fabricacion e integration de un sistema de vision para mediciones. CULCyT/Metrologia. 2015.

- Luis Alejandro Berumen Carlos, Javitt H.N. Padilla Franco et al. Diseno de her- ramienta para la fabricacion de terminales prototipo. CULCyT/Herramientas. 2015.

- Jose Barajas, Javitt H.N. Padilla Franco et al. Sustainability Via Energy Har­vesting and Scavenging. Journal of International Test and Evaluation Associ­ation. 2011.

- Rogelio Davila, Javitt H.N. Padilla Franco et a l, Constructive Analysis of In- tensional Phenomena in Natural Language. Journal of Engineering Letters of IAENG. 2007.

- Alberto Ochoa, Javitt H.N. Padilla Franco et al. Explain a Weblog Community. Advances in soft computing.2007

- Alberto Ochoa, Javitt H.N. Padilla Franco et al. Implementing Data Mining to Improve a Game Board Based Cultural Algorithms. Advances in soft comput­ing. 2007.

- Alma Cota, Javitt H. N. Padilla Franco et al. Diseno , Construction y Operation de un Secador Solar de Lodos Generados en Plantas Tratadoras de Aguas. X X X Semana de Energia Solar. 2006.

VI

Page 8: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Javitt H. N. Padilla Franco, Amado Lara, Pedro Marquez. Sistema Descubridor de Conocimiento en Bases de Datos Distribuidas. X X II Congreso Internacional de Ingenieria Electronica “Electro 2001”. 2001.

Javitt H. N. Padilla Franco, Amado Lara, Pedro Marquez. Sistema Inteligente para la Ensenanza de las Matematicas. X X II Congreso Internacional de Inge­nieria Electronica “Electro 2001”. 2001.

FIELD OF STUDY Major Field: Instrumentation and Intelligent Networked Control Systems

Page 9: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

ABSTRACT

SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED

NETWORKED CONTROL SYSTEMS WITH MOBILE

OBSERVERS USING HETEROGENEOUS

CONTROLLERS

BY

JAVITT HIGMAR NAHITT PADILLA-FRANCO, B.S., M.S.

Doctor of Philosophy, Engineering

Specialization in Electrical Engineering

New Mexico State University

Las Cruces, New Mexico, 2017

Dr. Nadipuram (Ram) Prasad, Chair

In this dissertation, a framework for the development and analysis of low-cost

monitoring and control systems for a set of distributed nodes comprising of dissim­

ilar controllers is presented. The ability to connect multiple disparate processors,

is possible due1 to the use of a secure minimalistic custom communication proto­

col. This architecture allows the use of mobile devices to interact with the main

Page 10: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

features of the system for monitoring and control. The server is at the core the

overall framework, which enables the transparent development of data acquisition,

processing and control models using a software interface. These models automat­

ically translate to functional systems executed by all microprocessors available to

the system. The framework contains software libraries that allows the integration

of neural networks and fuzzy logic that permit the implementation of intelligent

data acquisition and control systems.

The random behavior of the underlying communication network is modeled

in real-time to automatically compensate for the delays and lost packages in the

execution of control actions.

The framework enables a mapping of system inputs and outputs to state vari­

ables. These variables can then be used to form a system model to enable classical

control theory analysis, such as controllability, observability and stability test.

ix

Page 11: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Contents

LIST OF T A B L E S ..........................................................................................

LIST OF F IG U R E S .......................................................................................

1 INTRODUCTION .................................................................

1.1 Initial D e s ig n ..........................................................................................

1.2 System Requirem ents.............................................................................

1.3 M otivation.................................................................................................

1.4 A ntecedents.............................................................................................

1.4.1 Sensor Networks .......................................................................

1.4.2 Instrumentation of Generators, Hydro-machinery and Wa­

ter Flow M easurem ent.............................................................

1.4.3 Gibson’s M e th o d .......................................................................

1.4.4 Wireless Sensor Networks .......................................................

1.4.5 Security in Sensor N etw orks....................................................

1.4.6 Sensor Networks for C o n tro l....................................................

1.4.7 Frameworks for Sensor Networks ..........................................

1.4.8 Energy Consumption in Sensor N etw orks.............................

1.4.9 Stability of Distributed S ystem s.............................................

1.4.10 Mobile Applications for C o n tro l.............................................

1.4.11 Distributed Control Systems, Hybrid S y s te m s ...................

Page 12: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

1.4.12 Networked Control S y s te m s .................................................... 11

1.4.13 Stability Analysis and Control of Networked Control Systems 12

1.4.14 Wireless Networked Control S y s te m s .................................... 13

1.4.15 Intelligent Control .................................................................... 13

1.5 Problem D efin itio n ................................................................................ 13

1.6 Objective ................................................................................................ 14

1.7 Hypothesis................................................................................................ 15

1.8 Justifica tion ............................................................................................. 15

1.9 Related W o r k .......................................................................................... 15

2 THEORETICAL F R A M E ................................................................... 17

2.1 Sensor Networks....................................................................................... 18

2.1.1 Instrum entation.......................................................................... 19

2.1.1.1 Instrumentation of Energy Harvesters ................ 20

2.1.1.2 Flow M easurem ent.................................................... 22

2.1.1.3 Signal Conditioning and Sensors .......................... 23

2.1.2 Communication in Sensor N etw orks........................................ 29

2.1.3 Wireless Sensor Networks and Mobile C o m p u tin g 32

2.2 Networked Control S y s te m s ................................................................ 38

2.3 Intelligent Networked Control S y s te m s ............................................ 42

2.4 Mathematical Modeling of Networked Control S y s te m s ................ 47

2.5 Distributed Control in Networked Control Systems ...................... 51

XI

Page 13: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

3 DEVELOPMENT................................................................................... 54

3.1 System Design......................................................................................... 56

3.1.1 Use Case D iagram s.................................................................... 56

3.1.2 Sequence D iag ram s ! . . 60

3.1.3 Class D iag ram s.......................................................................... 65

3.2 Im plem entation...................................................................................... 70

3.2.1 S e r v e r .......................................................................................... 71

3.2.2 Mobile Observer A pplication.................................................... 83

3.2.3 Control N o d e ............................................................................. 88

3.2.3.1 A rd u in o ....................................................................... 92

3.2.3.2 Raspberry P i ...................................... 95

3.2.3.3 National Instruments H ardw are............................. 98

3.3 Deployment and T e s t in g ...................................................................... 102

4 R E SU LT S................................................................................................ 105

4.1 Deployment R esults................................................................................ 108

5 CONCLUSIONS ................................................................................... 109

5.1 Future W ork............................................................................................ I l l

A P P E N D IC E S ................................................................................................. 113

A WATER FLOW M EASUREM ENT................................................... 113

B INSTRUMENTATION IMPLEMENTATION ............................... 125

C EXTERNAL SOFTWARE L IB R A R IE S ......................................... 134

xii

Page 14: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

D SYSTEM PERFORMANCE COMPARISON

REFERENCES ..........................................................

Page 15: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

List o f Tables

1 Voltage Scale............................................................................................... 130

2 Simulation Results. [-5mm] .................................................................. 148

XIV

Page 16: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

List o f Figures

1 10 KW Hydro Generator........................................................................... 2

2 Monitoring of Multiple Generators......................................................... 3

3 Sensor Network Architecture.................................................................... 18

4 Knowledge Domain Areas Involved in the Project................................ 19

5 Instrumentation Architecture................................................................... 21

6 Sensors with DAQ...................................................................................... 23

7 Accelerometer.............................................................................................. 24

8 Temperature Signal Conditioning Components.................................... 24i

9 Wheatstone Bridge with Thermistor and Amplifier............................ 25

10 Water Proof Pressure Sensor.................................................................... 26

11 Hall Effect Sensor....................................................................................... 27

12 Current Sensor............................................................................................ 28

13 Resistive Divider for Voltage Measurement........................................... 28

14 Instrumentation GUI................................................................................. 29

15 Communication Sequence......................................................................... 30

16 Communication Protocol.......................................................................... 31

17 Distributed Monitoring.............................................................................. 33

18 Wireless Sensor Network Implementation............................................. 34

19 Security and Multiple Clients................................................................... 35

xv

Page 17: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

36

37

38

39

40

41

43

44

45

46

47

48

49

55

57

58

60

61

62

64

Communication Protocol with User Authentication..........................

Software Architecture...............................................................................

Mobile Application Graphical User Interface......................................

Elephant Butte Irrigation Channel Drop 8..........................................

Control Nodes with Communication.....................................................

Mathematical Model Incorporating Real Inputs and Outputs. . .

Simple Mathematical Model of a Neuron.............................................I

Simple Local Intelligent Control on Node a Based on Artificial Neu­

ral Networks...............................................................................................

Fuzzy Sets Representing a Particular Rule..........................................

Control Node Implementing a Fuzzy Controller.................................

Discrete and Continuous Elements of a Control Node......................

Hybrid System...........................................................................................

Simulation of the Hybrid system of Figure 31....................................

General Architecture................................................................................

UML Use Case Diagram for Control Node..........................................

UML, Use Case Diagram for the Math Module...................................

UML Use Case Diagram for the Server................................................

UML Use Case Diagram for the Mobile Client...................................

UML, Sequence Diagram for Reading Data from a Mobile Observer.

UML Sequence Diagram for Autonomous Control.............................

xvi

Page 18: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

66

67

69

70

71

72

74

76

77

78

79

80

81

82

84

85

86

87

88

89

91

UML Class Diagram for the Control Node.

UML Class Diagram for the Math Package.

UML Class Diagram for the Server Package

UML Class Diagram for the Client Package.

Server Login GUI............................................

Server System Preview Login GUI..............

Screen Detail of the System Preview GUI.

Server Data Layer Node Selection GUI. . .

Data of Selected Node GUI...........................

Link Selection GUI.........................................

Network Analysis GUI...................................

Control Design GUI........................................

Stability Analysis GUI...................................

Custom Node Specification GUI..................

Login Activity..................................................

Application Main Activity.............................

Server Access Services Activities.................

Activity Node Observation and Control. .

Instrumentation Activities............................

Manual Control Outputs. ..........................

Generic Control Node Algorithm.................

xvii

Page 19: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

93

94

95

96

97

98

99

100

101

102

103

105

107

119

122

126

127

128

129

130

131

Arduino

Arduino implementation of Open CACE Protocol. . . .

Raspberry Pi 3 model B.......................................................

Raspberry Pi Output Plate from PiPlates.......................

Raspberry Pi and Arduino..................................................

Wifi USB Dongle...................................................................

Python Implementation of Open CACE Protocol. . . .

NI USB Devices.....................................................................

NI CompactRIO....................................................................

Protocol Header in LabVIEW.............................................

Header TCP Transmission in LabVIEW..........................

Final Knowledge Domain Areas Involved in the Project

Gap Between Models and Implementations.....................

Contraction Coefficient.........................................................

Pipeline Cross Section..........................................................

Schematic Signal Conditioning PCB..................................

Final PCB for Signal Conditioning....................................

Linearized Pressure Sensor Response................................

Linearized Accelerometer Response...................................

Linearized Voltage Response Nom-High Swing...............

Linearized Voltage Response Nom-Low Swing................

xviii

Page 20: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

82

83

84

85

86

87

88

89

90

91

92

93

Linearized Current Response.................................................................. 132

LabVIEW Instrumentation Code........................................................... 133

Lighthouses in the East and West Coast.............................................. 137

Control of Lighthouse Mirror Motor..................................................... 138

Simple Distributed Control..................................................................... 140

LabVIEW DCAF Configuration............................................................ 142

DCAF Main Test Module....................... -............................................... 143

Mathematical Model of the Lighthouse Problem................................ 144

Open CACE implementation of the Lighthouse Problem................. 145

LabVIEW Simulation of the Lighthouse Motor Distributed Control. 146

Open CACE Simulation of the Lighthouse Motor Distributed Control. 147

Test Execution Comparison.................................................................... 149

xix

Page 21: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

1 IN TR O D U C TIO N

Energy storage and electricity production stand out as the fundamental build­

ing block of today’s digitally oriented world. Coal, gas, solar cells or water turbines

must be utilized to generate electrical power. During the spring of 2011 research

was conducted by Dr. Nadipuram (Ram) Prasad to harvest energy from water

flows. That work was funded by the Department of Energy[39] due to its origi­

nality and novel approach in using carbon composite materials such as Kevlar to

fabricate the turbine components.

The idea was to construct several low-cost, self-contained generators. An extra

feature of these energy harvesters was to provide real-time data on their behavior.

The research topic of this dissertation is the result of the author’s work on the

instrumentation and control of the hydro generators.

Figure l.a shows a diagram of the hydro generator’s modular architecture. Its

two main components can be observed: generator housing and discharge elbow.

The 10 KW generator resides inside a sealed submarine, which can maintain a

water-tight environment. As Figure l.b shows, the submarine is attached to the

venturi-shaped housing by the means of two hollow columns. The inside of the

columns is designed to hold the instrumentation circuits. Also, the power and

interface cables are directed to the exterior using the empty space in the columns.

The power cable transmits the generated power to the load, and the interface

Page 22: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

(b) Components.(a) Modular Design.

(c) Impeller. (d) Final Assembly.

Figure 1: 10 KW Hydro Generator.

cables carries all sensor signals to the monitoring console. Figure l.c shows how

the aluminum impeller is mounted inside the turbine. After assembling the turbine

on the top, and the discharge elbow on the bottom, the structure stands more

than two meters tall, as illustrated in Figure l.d for an scale idea. An advantage

of the modular design is the ease of deployment. The modular components can

be transported to the installation site and assembled quickly. Also, the additive

manufacturing process allows the construction, and installation of custom-made

2

Page 23: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

housing suitable for site specific requirements.

A team consisting of doctoral, master, and undergraduate students worked

diligently under Dr. Prasad’s direction to achieve the system of Figure 1. A group

of students worked on the development and integration of the technology required

to measure and display the generator’s physical variables such as speed, vibration,

temperature, water flow and generated power. One important characteristic of

the design is the ability to deploy many units to generate additional combined

power. Thereifore, the instrumentation must support the connection of multiple

generators, see Figure 2.

Monitoring and ■: Control: System

Generator Generator,

Figure 2: Monitoring of Multiple Generators.

1.1 Initial Design

The instrumentation and control of two 10 KW hydropower generators is

a challenging task. It involves: i) Establishing a communication network, ii)

Ensuring this network is secure, iii) Message coordination between components

to achieve a global state, iv) Display measured values to the user, v) Enable the

Page 24: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

easy adjustment of control outputs by the user.

It is common in a rapid prototyping environment to select National Instru­

ments (NI) components due to its availability and reliability. However, such a

decision makes it a high-cost implementation. The objective is to develop a low

cost and reliable instrumentation of the system. A low-cost structure can be

realized with the use of Arduino and Raspberry Pi micro-controllers.

Arduino and Raspberry Pi can interact with different sensors. The instrumen­

tation was designed to measure water flow passing through the turbine, as well as

the voltage and current generated. Those measurements could be used to monitor

the system behavior or to control system state variables such as the water inlet

gates or the angle position of the impeller.

The next step after the selection of the acquisition devices is the communica­

tion with LabVIEW software. Communication is required to transmit the mea­

surements from the sensor nodes to the instrumentation Graphical User Interface

(GUI) in order to be displayed. The first implementation of this GUI was pro­

grammed in the NI environment because of its multiple visualizing components.

LabVIEW works flawlessly with NI devices, but for Arduino, and Raspberry Pi

additional software must be used to enable interaction with the NI environment.

Some libraries exist to communicate between Arduino, and Raspberry Pi with

NI software. Unfortunately, they do not offer the simplicity required for efficient

communication between microcontrollers. Thus, regular serial or socket commu-

Page 25: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

nication was utilized to perform remote measurements and then send them to the

monitor GUI.

1.2 System Requirem ents

Several requirements were imposed on the design. The main requirement

was that the systems must be scalable. The plan was to have many generators

interconnected, such that every one of them can be monitored. In the first itera­

tion of the project, the measurement system was supposed to fulfill the following

requirements:

• It shall measure and record generator’s running variables, such as inlet water

pressure, outlet water pressure, generator vibration (in three axis), chassis

vibration (in three axis), water temperature, generator temperature, gener­

ated voltage and current.

• It shall compute the water flow using the calculated water velocity from the

differential water pressure measurements and the pipe diameter.

• It shall display the sensor readings on a monitor, in a user-friendly interface.

• It should be a low-cost reliable implementation.

I• It shall provide an infrastructure to monitor several generators concurrently.

5

Page 26: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

During the project evolution, additional requirements were added in response

to the research maturation. Those additional system qualifications are:

• It shall be able to provide control outputs in each sensor node.

• It shall support Wireless communication using standard Internet Network

Protocols.

• It shall allow the remote monitoring and control using mobile devices.

I

• It shall offer a GUI that allows the Control Engineer (CE), the user, toi

program and modify user-defined control mechanism easily.

• It shall be secure in its communications among components, i.e., no unau­

thorized people should monitor or control the system.

1.3 M otivation

During the initial research for the design, no single tool that combines hard­

ware and software able to cover the whole set of requirements was available. Such

a system could be implemented using Lab VIEW but with considerable amount of

effort. Even though this software tool in conjunction with its supporting hardware

provide many of the features required for the implementation, it is very expensive,

and it can not be called low-cost. Also, it lacks heterogeneous hardware support.

Additionally, Lab VIEW does not consider the delays in the network for the design

6

Page 27: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

of distributed control. This situation represents an excellent opportunity to fill

this market-gap by creating a reusable framework that allows the interaction of

many heterogeneous components to form a networked control system. This dis­

sertation in addition to describing the design and implementation process of such

system, also goes into the formal analysis of the networked control systems in

order to create a theoretical model with attributes, such a network statistics and

controller dynamics. This model is then implemented into a software tool which

translates and executes its representation into a distributed environment made of

heterogeneous components.

1.4 A ntecedents

Several research papers were analyzed during this project development, and

the main results are briefly presented and classified in the next section. Research

topics are chronologically ordered according to the time they appeared on the

project.

1.4.1 Sensor Networks

This project started as an instrumentation of hydro energy harvesters, in

which every harvester will contain a set of sensors. Those harvesters are inter­

connected using a computer network. In the electrical engineering field, the area

of “Sensor Networks” deals with the design and theoretical aspects of sensors in-

Page 28: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

terconnected by a network topology. Some topics were considered relevant during

the initial stages of the research like the ability of the sensing nodes to perform

some distributed computation [120]. The design of sensor networks often uses a

layered architecture [170] to achieve tasks such as data aggregation [76], real-time

communication [150, 183], better querying [33] and ease of management [40].

1.4.2 Instrum entation of Generators, Hydro-m achinery and W ater Flow

M easurem entiI

After defining the sensor network, it is necessary to establish how to measure

the physical variables. The principal elements to instrument in the case of the

hydro-generator are: water flow, voltage, generated current and vibration. Some

authors address measurement techniques in hydroelectric generators like [153, 144,

100, 104, 72]; other authors do the same for wind and steam turbines [142, 176].

Data obtained from the instrumentation is the used to mathematically model

behavior of generators as in [167, 66, 151, 108, 123]. This data could also be used

to present on-line monitored data to users [15]. Design Testing can be performed

using real-time instrumentation [64]. One example of such testing is blade pressure

analysis [8]. Some studies had been realized over the vibration of generators

[37, 32], One of the primary goals of generator instrumentation is to calculate its

power efficiency [166, 111].

8

Page 29: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

1.4.3 G ibson’s M ethod

Flow measurement is a key component in the instrumentation therefore it is

widely studied [177], especially in the case of flow inside pipelines [3, 69, 2]. An

integration technique called the Gibson’s Method is normally used [163, 70, 69, 2,

71] with differential pressure sensors on pipelines.

1.4.4 W ireless Sensor Networks

It is possible to extend the concept of a sensor network with the use of the

wireless technology. Some issues are treated in the literature [184, 141, 96] as

data aggregation of multiple nodes [1]. Several problems can be solved in wireless

sensor networks by developing a custom protocol [31, 55]. Some mathematical

models for the algorithms for wireless sensor networks [78] were studied. Multiple

areas could be involved in sensor networks as explained by [155]. Other problems

that will arise are coordination [24], coverage [63], load repartition [95]. Some

examples of other wireless sensor network projects were studied with application

in different domains and environments [139, 116].

1.4.5 Security in Sensor Networks

Sensor data travels from node to node inside a network of sensors. The

internetworking of its design, make sensor networks vulnerable to attacks. Thus,

9

Page 30: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

security is an important aspect of communication design in sensor networks [172,

112]. Wireless sensor networks [35, 113] present different challenges.

1.4.6 Sensor Networks for Control

Sensor networks could perform some control actions at the node level [148, 90].

Some extrapolation of the classical control theory to sensor networks [94] is desir­

able. Artificial intelligence and intelligent control [51, 147] is used in conjunction

with sensor networks, and even game theory [135] is applied.

1.4.7 Frameworks for Sensor Networks

One common design characteristic in sensor networks is the utilization of an

extra layer [98] to mask heterogeneity [115]. Other aspects such as communication

[159] or coordination aspects [20] inside sensor networks are implemented in that

additional layer. This extra layer is usually referred as “middle-ware” [58] or a

“framework” [61, 73]. Those frameworks are used with wireless technologies as

well [89, 117, 30],

1.4.8 Energy Consum ption in Sensor Networks

From the design phase, the energy consumption of the sensing nodes on the

sensor network is important. Protocols [121], frameworks [73], topologies [129, 88],

power-aware algorithms [185], game theory [135] and, other techniques [5] are used

10

Page 31: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

to achieve optimal power consumption.

1.4.9 Stability o f D istributed System s

Research was conducted on the stability of distributed systems such as a

Nyquist-type stability test for multi-variable distributed systems [92] and how to

maintain stability in a distributed generation system [25].

1.4.10 M obile Applications for Control

Mobile computing refers to the use of cellular telephones and the development

of their applications. The use of mobile computing in automatic control [6, 19, 105]

allows the ubiquitous access to system variables to change their values.i

1.4.11 D istributed Control System s, Hybrid System s

Distributed control is when components to be controlled reside in separate

geographic locations. Therefore this project perfectly fits into the definition of a

distributed control system [44, 161, 87, 9, 138]. A distributed control system can

be mathematically modeled by a hybrid system [21, 93, 22].

1.4.12 Networked Control System s

The proper subset of distributed control systems that represents the project

is the “Networked Control Systems” [60]. These systems have their mathemati­

11

Page 32: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

cal analysis and models [86], but are extremely dependent on the communication

[109]. Implementations of classic Proportional-Integral-Derivative (PID) in wire­

less environments [162] and other aspects such as behavior algorithms [106] are

examples of research in this area. Communication models [47] and communication

logistics [181] are also studied. Security remains a central topic due to the nature

of the system wireless construction [154, 190, 157, 156].

1.4.13 Stability Analysis and Control of Networked Control System s

Some papers go a little bit deeper in the stability analysis and control of Net­

worked Control Systems [168]. Others use Markov models to model the random

delays in the network [140]. State Feedback can be used [189, 186] to control the

system. Others do a characterization of input-output stability [102], There are

stochastic hybrid systems [10] also. Stabilization of a system when the packets

can be dropped randomly [188, 178], random delays [46, 28] or random sampling

periods [84] and their challenges were contemplated. Other stabilization topics

that were studied include adaptive stabilization [152], symbolic control design

[18], Communication Constraints [57] and, optimal control under communication

constraints [45]. Techniques such as event-triggering in Distributed Networked

Control Systems [174] and its impact on bandwidth in a wireless network [173]

were also examined.

12

Page 33: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

1.4.14 W ireless Networked Control System s

The logical research evolution is to include wireless into the game creating

a Wireless Control Network [110]. Other authors considered and studied the

implementation requirements for wireless networked control systems [99] and their

models [62].

1.4.15 Intelligent Control

Artificial Intelligence refers to the use of mathematical models that resemble

to some extent the abilities of the human mind. Computational intelligence could

be used to achieve intelligent control of normal [192] and distributed systems

[175, 179, 145, 191, 107],

1.5 Problem Definition

In the course of research conducted for the development of the instrumen­

tation system, numerous theoretical models were examined to describe the be­

havior of networked control systems. Also, various implementation frameworks

were considered to build the actual system. However, no single approach allows

the theoretical side advantages, such as the system’s proof of stability, and the

pragmatical side advantages, like the ease of deployment or security.

The central problem addressed by this dissertation is the necessity of merging

13

Page 34: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

theory and practice into one suitable framework for the design, analysis, develop­

ment, and. deployment of networked control systems.

1.6 O bjective

The main objective is to build a software infrastructure that allows the con­

trol and monitoring of multiple heterogeneous controllers distributed among the

Internet. This infrastructure must have the following specific objectives:

• It shall provide a secure schema to ensure that only authorized users will

monitor and control the nodes in the system. Also, the communication

between nodes must remain private at all times.

• It shall allow the user to perform a static analysis on the system such as

multi-variable Nyquist-type stability test.

• It shall allow dynamic analysis by recording activity of the system’s com­

munication. That information shall be used to model the random network

behavior using a hidden Markov chain. This model shall be utilized by the

framework to compensate for random delays on the communication.

• Given a 1 mathematical model to the framework, this shall automatically

generate a suitable mapping of the hardware components to implement the

given architecture using the available components efficiently.

14

Page 35: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

1.7 H ypothesis

The main hypothesis is that by the creation of a robust enough infrastructure

the implementation, and analysis of distributed control systems will become a

simple interaction with a programmers interface. Also, if the implementation

supports the interaction with heterogeneous components then its cost could be

reduced. Finally, if the infrastructure uses standard mathematical nomenclature

for describing its inputs and outputs, then common mathematical procedures for

control could be easily applied to the distributed system.

1.8 Justification

It is important to have a software infrastructure that flawlessly mixes the

world of control system synthesis with the real world implementation. The work

here presented fills this important gap in control engineering.

1.9 R elated Work

Many tools try to prove some mathematical properties out of hybrid systems.

Those conclusions mainly are if a particular state is reachable from an initial con­

dition. Some examples are mentioned next, but in no particular order. PH AVer

is a Hybrid systems verifier [42]. The tool named d\dt was designed to perform

reachability analysis of hybrid systems [11]. HyTech is used establish the condi­

15

Page 36: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

tions under a hybrid system satisfies a constraint [59]. CheckMate is a tool for

modeling, simulating and verifying properties of hybrid dynamic systems [143].

Matisse is a verification and reachable set computation of linear systems [49].

Ellipsoids is a library in MATLAB that can be used for reachability calculations

[80]. SpaceEx is a tool to verify, i.e., compute the sets of system’s reachable states

given a mathematical model of a hybrid system [43]. UPPAAL, It can model

and validate real-time systems that have been described as networks of timed au­

tomata [13]. HSolver is a verification tool for Nonlinear Hybrid Systems [124].

KeYmaera is a Hybrid verification and theorem Prover for Hybrid Systems [114].

PowerDEVS is a general-purpose software tool for Discrete Event System specifi­

cation modeling and simulation used to simulate hybrid systems [14]. HyEQ is a

Hybrid System Solver for MATLAB [128]. HyCreate is a tool for approximating

reachability of Hybrid Automata [12].

All these tools cover the mathematical modeling, simulation, and validation of

hybrid systems. They are all related to this dissertation in their effort to compute

trajectories of hybrid systems. Nevertheless, they do not allow the extrapolation

of the models to the real world application. For the real world construction of

networked control systems, LabVIEW reigns as one of the industry standards due

to the ease of its visual programming environment.

16

Page 37: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

2 THEORETICAL FR AM E

This chapter wili briefly cover the theoretical foundation studied to produce

this dissertation. Topics are discussed as they appeared on the project. The

approach of this section is to describe the application of the knowledge to the

project itself.

At first, when presented with the challenge of instrumenting and controlling

a set of hydro harvesters, the first thought was about the sensor data commu­

nication to the monitor station, i.e., how the sensor data flows from the node

to the terminal. Another crucial aspect to consider was that the system had to

be implemented in real life at a reasonable cost, i.e., research papers usually as­

sume theoretical situations that tend to be hard or expensive to integrate, in a

congruent manner and meeting the established requirements, using top-of-shelf

components. Lastly, it was imperative to create a mathematical model from the

deployed system. This model shall be used to characterize the system’s function­

ally, and then some stability could be proved. Additionally, the model shall allow

the development of a global optimum controller. First its optimality should be

theoretical established wit the help of the system’s mathematical libraries. This

model is then translated automatically to be executed on the array of nodes.

Communication seemed at the beginning to be the main issue, so we now

introduce some concepts in the sensor networks.

17

Page 38: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

2.1 Sensor Networks

A sensor network consists of a set of sensors interconnected to a main ob­

serving point. Figure 3 shows an architectonic view of a sensor network, many

crucial aspects of the practical implementation are omitted in this diagram. The

main topology could be observed, i.e., lines represent transmission media, but it

is not clear how communication is achieved. During the initial design phase, one

clear pattern emerged in sensor networks: the academic papers differ extensively

from the implemented technologies available commercially. Usually to construct

a sensor network, a company is hired to provide all the infrastructure tailoring

general products, like NI. On the other hand, research covers topics from physical

layer protocols, moving upwards in complexity all the way to the user application.

Research is seldom related to actual companies and its almost always independent

Sensor Node Gateway Sensor Node

Figure 3: Sensor Network Architecture.

18

Page 39: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

from particular devices. With this in consideration, the first design was exten­

sively based on commercial components. The knowledge domains involved in the

initial design put the project at the intersection of computer networks and instru­

mentation, as represented in Figure 4. Computer networks were introduced to

the game due to the fact that remote monitoring, security and mobile computing

were in the requirements. In a nutshell, the project was a sensor network that

consists of instrumentation nodes interconnected through a network.

U

Sensor Networks

Computer /J

Networks 1

Instru­mentation

Figure 4: Knowledge Domain Areas Involved in the Project.

2.1.1 Instrum entation

This section briefly introduces the theoretical concepts used in the instrumen­

tation of the hydro energy harvesters. This instrumentation consists of a series

of sensors monitoring the behavior of the different components installed on the

19

Page 40: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

generator. Those sensors are then connected to a virtual instrument console that

allows easy interaction with the user.

2.1.1.1 Instrum entation of Energy Harvesters

It is necessary to define which physical phenomena shall be measured, the

most important variables to monitor from a hydropower generating system are:

• The flow of water through the generating module. Q usually represents flow

3rate, and it is expressed in cubic meters per second

• Generated Voltage and Current (with their respective frequency). The volt­

age is proportional to the water flow passing through the generator.

• Water temperature. Different temperatures will affect the water density

thus changing the system behavior.

• Generator temperature. This measurement is important to ensure the safe

operation of the generator by avoiding overheating.

• Vibration. Unusual movement could be a sign of malfunction inside thei

generator.

• Shaft Speed. In combination with the water flow, this value is used to

calculate the generator’s efficiency.

20

Page 41: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

In Figure 5 the instrumentation architecture diagram shows that in order to

read the physical measurements, an interface is needed to acquire and communi­

cate data to the user. This real-world interface is called Data Acquisition (DAQ)

\= (£ M S S

/ — \ Wh

MonitoringConsole

DAQ Hydro Generator/ Sensors

Figure 5: Instrumentation Architecture.

system. One big drawback of such devices is their high price. Following the de­

sign requirement of constructing a low-price device, the Arduino was selected as

DAQ because; of its low price and twelve I/O analog channels, with 9615 Hz of

maximum sampling speed. The selection of the sampling speed obeys some tech­

nical reasons related to the rate of the acquired signals such as: i) The expected

maximum shaft speed is around 300 RPM. ii) The generated voltage shall have a

frequency in the range of 60 Hz. Other signals, even though time-dependent, are

not critical for the instrumentation; thus they can be monitored at any available

sampling-rate.

21

Page 42: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

2.1 .1 .2 Flow M easurem ent

For the proper generator characterization, it is important to estimate how

much water is flowing through the system and its response to the passing flow in

terms of electrical generation. See Appendix A. 1 for a brief introduction to fluid

dynamics.

The water discharge Q over a pipeline with cross-sections areas of A\ and A 2

and at different heights hi and h2 is:

Q = C o A ,A 2] f ^ A f (1)

where Cc represents the contraction coefficient.

Many standard methods are applied to measure the volume of water passing

a determined area. For this project, the differential-pressure flow meter was used,

i.e., Q must be estimated using knowledge about the dimensions of the pipeline

and the pressure the water is applying on the walls of the pipe. Consult Appendix

A.2 for a brief explication of differential pressure flow-meters.

Equation 1 can be computed using differential pressure as follows:

• 1 i 2 ■ s / m - P*)pi (2)P2 ^

with /3 = Appendices A.3 and A.4 present a detailed derivation of Equations

1 and 2.

The implication of Equation 2 for the instrumentation is that water discharge

22

Qm

Page 43: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

shall be computed using two pressure sensors placed on different segments of the

pipeline feeding the generator. Additionally, other sensors are required to instru­

ment the generator. In Figure 6 a general architecture of the sensor connection is

displayed.

5 1 E S B 8 2 1 1 1

©©(Effil)

<-------------Voltage ■ <-------------Current ■<— Generator Temperature —> < Water Temperature > <------- Vibration (x , y, z ) ------ > <----------- Shaft Speed----------- >

<---------- Inlet Pressure - <---------Outlet Pressure ■

Figure 6: Sensors with DAQ.

2.1.1.3 Signal Conditioning and Sensors

All sensors considered for this project react differently to the environment.

The acceleration sensor changes in voltage when moved, while the pressure one

will change its resistance. The DAQ internally has an analog-to-digital converter

with a specific voltage and current range.

Figure 7 shows the ADXL335 acceleration sensor from Analog Devices used

on this project. The acceleration sensor provides three outputs, one for each axis.

Each axis has an output voltage between -3.3 and 3.3 volts.

23

Page 44: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

mRID|B

SP

Its range of operation is ±5 gravi­

ties. Scaling was performed on soft­

ware with a linear function approxi­

mated by experimental data from the

initial testing of the chip. See Ap­

pendix B.2 for more details on the scal­

ing.

Sensing temperature requires additional steps. For this project, the thermistor

Figure 7: Accelerometer.

of Figure 8b was used. To map the change in resistance to change in voltage, the

circuit of Figure 9 was implemented. Scaling was partially achieved by using the

instrumentation amplifier of Figure 8a.

AD8221TOP VIEW

(a) AD8221 Precision Instrumen- (b) YSI-55008 Precision Thermis-

tation Amplifier, Analog Devices, tor 30K fh

Figure 8: Temperature Signal Conditioning Components.

24

Page 45: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

The final part of scaling was done by software in order to neglect the non-

linearities that the thermistor presents at its limits of operation, see Appendix

B.2 for exact scaling equations.

9Vcc

out

Figure 9: Wheatstone Bridge with Thermistor and Amplifier.

For the circuit in Figure 9, the input-output relation is the next. First, it was

established that the voltage at the non-inverting terminal is:

R tV+ = Vcc

R x + R T (3)

For the inverting terminal, KCL can be applied,

- V- Vcc - V_ V-R R'2 R ‘A (4)

Since for an ideal Op Amp, the voltages at the inverting and non-inverting ter­

minals are the same. VL = V+ can be plugged to Equation 4. After rearranging

25

Page 46: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

like terms:

Vout = RVcc Rr ( 1 1 1 \ 1+ ^ - + (5)Ri + Rt \R2 R$ R ) R2

The mapping described by Equation 5 will provide the input voltage for the scaling

process. That will convert with a linear approximation the provided voltage into

usable temperature. The scaled range of action is from 0 to 60 Celsius degrees.

To measure pressure, the water-resistant package shown in Figure 10 was used.

The unit contains a sensor, Wheat­

stone bridge and amplifier encapsu­

lated and hermetically sealed. For

this sensor, the scale is entirely im­

plemented in software. The package

needs a input voltage of 5 volts and its

ground. The sensor generates output Figure 10: Water Proof Pressure Sensor,

voltage proportional to the pressure on

the tip. Its range is from 0 to 5 volts scaling accordingly to 0 to 33 PSI. The sen­

sor model US333 could be bought from h ttp ://w w w .s tra in se n se . c o .uk/ with a

price of 165 USD. Its output final scaling was approximated using a linear function

also obtained from the initial device testing.

Temperature and pressure are required to compute flow as described before.

The instrumentation of this project also requires the ability to calculate efficiency,

both mechanical and electrical.

26

Page 47: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Using a Hall effect sensor US 1881 from Melexis, see Figure 11, in combination

with magnets attached to the shaft of the generator, it is possible to generate and

measure an analog signal with peaks on every magnet pass.

Once that signal is acquired, a software

processing is performed in order to

compute its frequency. That frequency

is relative to the number of magnets

present on the shaft and therefore the

5V+_________ 3 T I ________digital--------------1--------- I I OUT

G N D "------------ 1----------------1 *p J.,- rI Hff I

speed of the shaft. The software ap- Figure 11. Hall Effect Sensor,

proach was to use a digital counter over

a finite frame time, and since the number of magnets remains constant, it is just

a division to obtain the frequency, see Lab VIEW code on Figure 83 in Appendix

B.3.

For the current measurement, the Integrated Circuit ACS756 from Allegro,

see Figure 12, was used. This IC gives a voltage that is linearly proportional to

the current passing through the sensing input pins. Once the voltage reading is

acquired, scaling is performed with software by transforming the reading from a

range of 0 to 5 volts into 0 to 100 Ampere.

To measure generated voltage and since it is an AC generator, a transformer

was used to step down the generated voltage so it could be acquired by the DAQ.

27

Page 48: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

For an additional scaling, a voltage di­

vider was connected as shown in Figure

13. Several measurement types could

be implemented with this configura­

tion, such as phase-to-phase, or phase-

to-ground. Once the measured signal

was acquired, software scales back up

the reading into the actual voltage. This scale considers the transformer’s linear­

ity and the resistance values of the voltage divider. A printed circuit board was

Figure 12: Current Sensor.

Vin. Vi

V i n _

out

Figure 13: Resistive Divider for Voltage Measurement.

designed to hold all the signal conditioning, see Appendix B.l. In order to present

the measured values to the generator operator, a Graphical User Interface (GUI)

in Lab VIEW was designed. In Figure 14 the acquired variables are displayed using

graphical components. This GUI also allows the data logging of the generator’s

2 8

Page 49: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

operation. The program that controls the GUI is shown in Figure 83

EI3DO NMU.S. OF

ENERGYSTATE Energy Efficiency &

Renewable Energy

11*41

pm

_:'yz 1

**lr r w

Figure 14: Instrumentation GUI.

2.1.2 Com m unication in Sensor Networks

Communication inside the sensor network is the most important aspect of its

implementation. Communication could be expressed as the sequence of messages

between the participants and the structure or format these messages have. One

of the key requirements was to have a low-cost implementation, Arduinos were

used initially as the DAQ devices. Arduinos have a shield that implements the

Ethernet hardware, and thus the communication could be greatly simplified by

the use of standard transport protocols such as TCP/IP. Due to Arduino’s very

limited hardware, it was required to program the communication at the lowest

29

Page 50: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

level, i.e., using the socket interface.

The most basic form of interaction occurs when the server (Control GUI) re­

quests data to be displayed on the screen. Instead of the sensor node transmitting

every single change on the sensor, which will produce network congestion, the

server will request data from the sensor node every time an update event occurs.

In Figure 15 the messages required to perform this operation are shown. First, the

server, which was waiting for some event such as data refresh, send messages to

the read the x sensor. Upon arrival, the sensor node submits an acknowledgment

to the server and starts gathering the required data from the sensor. Once the

data is collected the sensor node transmits it to the server. If the server got the

requested data, it would send an acknowledgment to the sensor node.

Sensor Server Sensor NodeWAITING

display variable

SENSOR_X_READsensor data

_READ OATA nr.Y'n esw si REQ_RECEIVEDpattering sen sor data

SENSOR DATA.SENDREQ_ACCEPTED

READING .DATA

CLOSED

Figure 15: Communication Sequence.

30

Page 51: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

For the message, a custom protocol is defined for the transmission of acquired

data. Figure 16 shows the frame that travels inside a TCP frame. The fist 16 bits

represent the TYPE of the message. It is intended that the protocol will support

many operations, such as update variables or read values from the sensors. Such

different TYPEs of messages will be encoded on these bits. The next 4 bits will

describe the VERsion of the protocol used, and 12 bits of optional FLAGS. For

the source node 16 bits describe the requester numerically , and another 16 bits

describe the node that will provide the data. TYPE is the encoding requested.

NUMPACKS is the number of readings requested. The sequence number is con­

secutive integer assigned to the package. A time stamp will contain the operation

time. The payload will contain N sensor reads each one with an ID.

X

He

er

Pay

> load

Figure 16: Communication Protocol.

PACKID N SENSOR DATA N

DATA N cont’d . . . optional padding

0 7 11 15 19 31

TYPE VERS FLAGS

source node destination node

sensor number TYPE NUMPACKS sequence number

timestamp

PACKID 1 SENSOR DATA 1

DATA 1 cont’d • • •

31

Page 52: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Future implementations of the system will allow the porting of protocols used

in the electric industry such as the DNP3 (Distributed Network Protocol) and the

IEC 60870.5, a good reference source on these protocols is [27].

2.1.3 W ireless Sensor Networks and M obile Com puting

One disadvantage of the sensor network is the requirement of a guided medium

to establish the communication. The portable design of the energy harvester

allows to be deployed quickly throughout many different operation environments.

Such locations could not always provide wire connectivity to the Internet. Wireless

sensor networks provide a theoretical frame to expand on the sensor networks.

Also, they introduce the ability to create mobile interfaces to display sensor data,

since cellular telephones could be easily integrated. In Figure 17 a scenario is

presented in which many harvesters have been deployed on the irrigation channels

of the state of New Mexico; all of them are interconnected using wireless adapters

to the Internet. The system could then be monitored using a mobile device such

a tablet or cell phone.

To achieve such behavior the architecture of Figure 5 must include the WIFI

Arduino’s shield. Also, some access point must be considered. In Figure 18 a

router is used, to integrate the wireless sensor network to the Internet. Mobile

devices are used then to monitor the instrumentation variables in real time.

One big challenge that the wireless sensor networks confront is the security.

32

Page 53: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

MobileDevice

Internet

GeneratorArray

Figure 17: Distributed Monitoring.

On the architecture of Figure 18, there is no way to validate who is accessing the

wireless sensor network unless you hardwired them to the sensor nodes and the

mobile devices, i.e., the sensor network could only send data to a list of particular

preprogrammed mobile devices. Any change will require a recompilation and

redeployment of the sensor node code. Figure 19 adds an access server to deal

with the security and registration of mobile devices, and as well to receive and log

all the sensor activity.

By adding the access server, the operation of the wireless sensor network is

changed. In Figure 20 an extension of the original protocol is described in which

the Access server is included as well as the mobile device. The main change is that

33

Page 54: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

MobileDevice

WirelessRouter

Arduino + WiFly „ S!gnal. Hydro Generator/Conditioning Sensors

Figure 18: Wireless Sensor Network Implementation.

user must be authenticated to read the data acquired. After the user is validated,

the transmission of the sensor data occurs using the sequence of Figure 15. Now

the server will transmit the data to the mobile client. By having an access server

not only security is improved, but also this server could be permanently monitoring

and logging all sensor nodes, or even it could be programmed to perform some

automatic scripts affecting the whole sensor network.

The wireless sensor network with mobile observers is constituted by three parts:i

mobile device, access server, and the energy harvester. To be a useful monitoring

system, it must allow many mobile devices to access the network simultaneously;

also many harvesters are part of the network. To provide a reliable service, the

system must have many access servers to ensure the proper functionality on net­

works consisting of many clients. Even though there are many components in

the wireless sensor network, all mobile devices run the same program, the same

happens to the servers and harvesters. Figure 21 shows the software architecture

of all the components of the sensor networks.

Page 55: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

35

Page 56: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Mobile Device Server Sensor Node

Figure 20: Communication Protocol with User Authentication.

In the energy harvester module, all the software running on the sensor node

is contained. It has a submodule dedicated to data acquisition from the sensors.

Also, has a control I/O module in charge of sending control signals to the actu­

ators, this part will be covered in detail on the next subsection. The harvester

communicates all the sensor data using a specialized module, that could also en­

crypt and decrypt messages. The authentication server has modules to deal with

user login and data logging. One important feature of the server is its ability to

discover new harvesters that just connected to the system and incorporate them

as part of the sensing infrastructure. Also, the server has a multi-client support

communication module with encryption and decryption capabilities. The mobile

client will consist of the communication module plus a series of graphical interfaces

for displaying and controlling the sensor nodes.

36

Page 57: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Encryption/Decryption.

ModuleMonitoring

GUI Control

CommunicationModuleMobile. Observer MainGUI Harverster '

Locator

Mobile Monitoring Application

UserAuthentication

Data Logging

Transaction Loggin ;

Server.'MaimModule.:H a rv e rs te r

.- Discovery . <Services ■

Authentication Server

DAQ

Instrumentation ■. Main-Module

Communication ■ Module

Control 1/0

Encryption/'= Decryption,,

Module Energy Harvester

Figure 21: Software Architecture.

Figure 22 shows the storyboard of the user interaction with the mobile appli­

cation through its interfaces. First, the user will be presented with the login GUI.

After that, the GUI harvester locator enables the user to choose any harvester

registered on the access server. From there, the user could select the monitoring

GUI to observe the measurements of the sensors, or the control GUI to adjust the

variables of the system on the local nodes.

37

Page 58: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Control

NodeSelection

MainMenu

Monitor

Figure 22: Mobile Application Graphical User Interface.

2.2 Networked Control System s

Every harvester is deployed in a very particular environment, for example,

the first energy harvester installed from NMSU was situated on the Drop 8 of

Elephant Butte irrigation channel. Drop 8 has two inlet gates and two drop gates

that can be raised or lowered to control the flow of water passing through the

drop, see Figure 23. Those gates could be controlled automatically to adjust

the amount of water going into the generator, and thus controlling the generated

voltage. Such implementation requires that the DAQ module has the ability to

output signals in both analog and digital formats. Those output signals could be

38

Page 59: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

used to manipulate servo motors to raise or drop the gates in the Drop 8. Also,

these output signals could be used to control the voltage by means of additional

circuity.

(d) Drop 8 3D Model.

Figure 23: Elephant Butte Irrigation Channel Drop 8.

Initially, allowing the sensor nodes to provide outputs does not seem like a

big modification to the original architecture; but by definition, a sensor node will

not be a sensor node anymore, but an actuator node. When every sensor node

in the sensor network has the power to generate outputs based on its inputs, and

39

Page 60: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

on the inputs of all the other nodes, a set of different problems arise. In fact, the

architecture is no longer a sensor network but a Networked Control System.

For the architecture to control the system, additional dynamics based on the

input/output relation must be performed at the node level. This dynamics, usu­

ally in state-space representation, is not only affected by local variables, but also

from events happening in remote nodes. Figure 24 shows the mapping performed

on every control node a.

A ctuator,^ A ctuato r,^ A ctuator„iU

Control Node a

CommunicationModule

O ther Control Nodes

Figure 24: Control Nodes with Communication.

All the sensors of the node a, here named Sensor0>1. . . Sensorsa>n are mapped

directly into state variables I \ . . . /" and all actuators on the node, Actuatorail

. . . Actuator a_m, are mapped into state variables 0 * . . . 0™. One big challenge is

to map external state variables, for this purpose every control node has a secure

40

Page 61: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

communication module that allows the transmission of data from other control

nodes, this data then could be mapped into inputs state variables 1% or output

state variables Oyx.

This mapping make it possible to express the local controller dynamic as in

the example of Figure 25. The expression describes the dynamics of a control

node n with has two inputs and two outputs. Variables Oln and 0 \ represent the

output state variables one and two of the node n. Each of this variables will be

associated with an actuator. The input variables and I'2 are associated with

sensors in our system. Note that the expression can have feedback, transforming

the outputs into inputs of our system.

Figure 25: Mathematical Model Incorporating Real Inputs and Outputs.

In the case that the control dynamics uses a variable that is sensed in an ex­

ternal node data must be received from another node. On the other hand, when

41

Page 62: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

the output of the expression is an external node, actualization data is sent to

the required node. Lines in Figure 25 also represents a communication channel

between one or more external nodes. Every one of this channels has a set of prob­

abilistic distributions associated with it. Each distribution models randomness

in transmission time, packet loss, noise, etc. These models will be used by the

system to predict its running conditions.

The use of external variables inside the node is the main characteristic of

networked control systems. Some control applications only have a local effect, i.e.,

the only variables to control are the ones that reside on the same DAQ device.

This means that no communication or coordination is necessary. Other control

systems require a global effect on all the variables involved in the controller, for

example, the combined frequency of all generators must have minimum harmonics,

or the system must generate a combined constant power. This kind of operations

is called distributed control, but its primary execution relies on local controllers.

2.3 Intelligent Networked Control System s

To achieve the local desired behavior, standard control theory could be ap­

plied. Other approach is intelligent control. Intelligent control uses techniques

discovered in artificial intelligence to modify the state of a system using encoded

knowledge or experience. The idea is that the control node has the ability to deal1

with conventional control as in Figure 25 and additionally it is able to implement

Page 63: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

more sophisticated control such as fuzzy or neural controllers.

The neural controller is based on the artificial neuron model of Figure 26.

This model is an abstraction of the biological neuron, in which it is believed that

the input impulses of the neuron, in this case, must pass through

a resistance denoted by the weights wi,W2 ,W3 . The activation function / will

be trigger only if the sum of the input impulses is bigger than the bias 9. This

relation is shown in Equation 6 which assumes a direct output connected to an

actuator x of node a.

Bias6

n oActivatefunction Output

Ul2

H oWeights

Figure 26: Simple Mathematical Model of a Neuron.

n

°'i = f •“' < - * ) w

An artificial neural network is a collection of neurons interconnected in layers

like in Figure 27, The input layer will receive the signals to be processed. The

output layer will provide the desired response of the system. To achieve such

43

Page 64: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

behavior, the neural network must be trained using one of many algorithms such

as backpropagation [193, 65, 187].

Input Hidden Outputlayer layer layer

Figure 27: Simple Local Intelligent Control on Node a Based on Artificial Neural

Networks.

For its use in control, the neural network must take its role of a universal

function approximation to relate the local sensor inputs to its outputs. In Figure

27, for example, the local input variables I \ . . . 7® are used to generate the output

signals to the actuators 01,01, and 0^. The inputs and outputs of the neural

network could also include variables that are remote, using the mapping performed

44

Page 65: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

by the communication module of the control node.

A different approach in intelligent control is to use Fuzzy Logic. It is based on

the principle that linguistic variables could be modeled using fuzzy set. Such lin­

guistic variables then could be introduced in fuzzy rules to perform inference and

relate inputs with outputs. For example in Figure 28 two linguistic variables are

described using sets. The first one represents “Frequency” and has three possible

values: low, acceptable and high. The fuzzification logic could be interpreted as

one Hz is a 100% low frequency and, 100 Hz is 0 % low frequency. This model­

ing is shown On Figure 28 even though no scale was specified for the frequency.

Another possible linguistic variable is “Water Flow” with similar modeling but

different units and scale.

low acceptable high low normal high

0Frequency Hz Water Flow rrv

s

Figure 28: Fuzzy Sets Representing a Particular Rule.

Using these linguistic variables it is possible to specify control rules for the

generator, for example:

45

Page 66: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Rule 1:

IF frequency is low

THEN water flow is high

Rule 1 means that if the generator is producing a voltage at low frequency, the

system must increase the water flow to high to achieve the desired state. A

full fuzzy inference system must have many rules to relate the input variables to

the output variables of our system. In Figure 29 an example fuzzy controller is

presented that relates local inputs / * . . . I" and remote inputs and outputs, in this

case, 01 to the local outputs O*. . . 0 ™ and remote outputs 0®

Control Node a

FuzzyInference

Engine

Figure 29: Control Node Implementing a Fuzzy Controller.

46

Page 67: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Each node could also combine all technologies producing hybrid systems such

as neuro-fuzzy controllers.

2.4 M athem atical M odeling of Networked Control System s

The model of our networked control system must consider a peculiar charac­

teristic of the architecture: the communication section work with bursts of data,

i.e., data is not always available. On the other hand, the generator is constantly

working. In Figure 30 a general separation of the discrete/continuous components

is shown.

External

ControlNode

Discrete

/ Cont.

ControlServer

RegulationLayer

(Control)

CoordinationLayer

(Communication)

PhysicalLayer

(Generator)

Figure 30: Discrete and Continuous Elements of a Control Node.

47

Page 68: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

This behavior could be modeled using an automaton like in Figure 31. Every

transition of the automaton represents a discrete state of the system. Each discrete

state is governed by a different continuous dynamic. The example of Figure 31,

shows two possible states of the system O N and OFF. In a networked control

system, the state of the whole application depends on the discrete states of the

transmission. The continuous state is O* € M, and the discrete states are q e

{ON, OFF}. The change of 0* is described by the differential equation a — —aO}i

when in sta te1 q = OFF. The state q can not jump from O F F to O N unless

0} > 3. A simulation of this simple hybrid system can be seen in Figure 32.

O la < 91

O N, O F F

Figure 31: Hybrid System.

Formally a hybrid control system could be expressed using a Hybrid Automa­

ton which describes the evolution in time of the values of a discrete and continuous

state variables:

D efin ition 2.1. (H ybrid A u to m ato n ) A hybrid automaton H is an octuple

H = (Q , X, f , Init , Dom, E, G, R ), where:

• Q = {?i> 92, • ■ •} is a finite set of discrete states;

48

Page 69: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

• X = Rn is a set of continuous states;

• /( ' ) ■) ■ Q is a vector field;

• I nit C Q x X is a set of initial states;

• Dom(-) : Q —> 2X is a domain;

• E C Q x Q is a set of edges;

• G(-) : E -> 2X is a guard condition;

• R(-} •) : E x X -> 2X is a reset map.

Recall that 2X is the power set of X. On Definition 2.1, the function Dom

maps a set of continuous states Dom(q) C Rn to each discrete state q £ Q. The

100

O

0 0.2 0.4 0.6 0.8 1 1.2 1.4t

Figure 32: Simulation of the Hybrid system of Figure 31.

49

Page 70: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

relation (q,x) € Q x X is referred a the state of H.

The hybrid automata define the possible evolution of the state. In a nutshell,

starting from an initial value (q0,x o) G I nit, the continuous state x changes

according to the differential equation x — f(qo,x) with x(0) — x$, while the

discrete state q remains constant, i.e. , q(t) — qo■ Continuous change remains as

long as x remains in Dom(q0). If at a point the continuous state x reaches the

guard G(qo, q±) C ]Rn of some edge (qQj q\) € E, the discrete state may change to q\.

At the same time, the continuous state is reset to some value in R(q0i q\,x) C Rn.

After this discrete transition, continuous changes resumes, and the whole process

is repeated.

Hybrid automata involve both continuous changes determined by differential

equations and discrete transitions determined by a directed graph (like an au­

tomaton). The analysis of Hybrid automata implicates the computation of its

trajectories, i.e., to check if the system can reach a state given a set of inputs.

One big drawback of hybrid automata is the ability to produce models that are

unreasonable, either physically or mathematically.

One remark to make about the networked control systems is that the imple­

mentation is completely different from the theory, as a simple example, a state

variable could be used as an input or output, while the implementation requires

a distinction between inputs(sensors) and outputs(actuators).

50

Page 71: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

2.5 D istributed Control in Networked Control System s

The ability to achieve a global state using all the variables of the networked

control system employing autonomous controllers is called Distributed control.

Distributed Control Systems often have a central operator which also supervises

the control status of all combined nodes. This is because most implementations

of distributed control over networked control systems are in the manufacturing

industry, this scheme is called hierarchical control. In hierarchical control, there

is a bottom layer consisting of sensors and actuators. Then another layer with

the micro-controllers required to acquire and process the inputs. Then a super­

visory layer with computers that will condense all the information and present a

console to the user. This hierarchy could have more supervisory layers to achieve

different control projects. One drawback with the hierarchical control is that the

supervisory layers must receive the perceptions of the inferior layers and then

communicate a proper control action back to all inferior nodes.

In the case that the control must act fastly over the plant, there will not be time

to transmit the inputs and wait for the response. The control node must be able

to take some autonomous control decisions. Those decisions must be consistent

with the general objective of the distributed control, i.e., every autonomous part

must contribute to the general success of the control. One example of autonomous

control is the “Subsumption architecture” [23], this method was a prevalent way

51

Page 72: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

to implement behavior-based robotics in the eighties.

A very especial case of distributed control is when the optimal state must be

achieved by the combined dynamics of the nodes, i.e., optimal distributed control.

This task is way much harder that non-optimal control since the reachable state

of the system must be a maximum or minimum. Other example of optimization

is to minimize the response time of the system. In general, the system must

complete the optimization task by taking in consideration restrictions inherent to

the problem to solve.

Some methods exist to find an optimal solution, in particular, the greedy

algorithms have been used extensively in optimization, a pair of examples could

be found in [119] and [83]. In a nutshell, a greedy algorithm always makes the

“greedy choice” , i.e., it always selects the local minimum hoping to get the global

one. E.g., the knapsack problem states that if we have a set of elements with

value and weight, which elements should choose to maximize the value of items

that we can carry in a limited weight knapsack. The greedy choice is to pick

always the item that weights the less and has more value. If the greedy choice

is always applied, it is guaranteed that the knapsack will be optimized for value

and weight. The greedy approach works great in discrete cases, and the optimal

value is sure to be found if the problem to optimize is reduced to a Matroid [36]

or more general to a Greedoid [77]. The idea of the project is for each node to be

able to implement a greedy choice, i.e., each node will optimize locally. In some

Page 73: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

problems, this behavior will achieve the global optimum. Some other examples of

greedy algorithms in control could be consulted in [82] and [118].

Unfortunately, most of the control situations are not discrete in nature, like the

voltage. For these problems, Intelligent Optimization could be a viable alternative.

Particularly one bio-inspired algorithm called Particle Swarm Optimization (PSO)

can be used to solve more general optimization problems. The particle Swarm

optimization works emulating a bird flock, i.e., a population of candidate solutions

is given; these solutions then move into the search space according to a specific

set of social rules. W ith no specific way to measure the quality of a solution, the

optimum is obtained by the particles moving towards the solution. Some examples

of particle Swarm optimization applied to control could be found in [53] and [91].

53

Page 74: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

3 D EV ELO PM EN T

This chapter presents the analysis, design, and implementation of the system

proposed on previous sections. In a nutshell, the project represents a Computer-

Aided Control Engineering (CACE) tool. One early example of such tools is pre­

sented in [68]. Additionally, the project involves the implementation of a frame­

work [164], One main deviation is that the CACE tool is designed to handle

distributed control, and the framework is able to deal with security implementing

also wireless communication and mobile devices. Also, the tool allows the cre­

ation of intelligent control scheme based on artificial neural networks and fuzzy

logic. On top of all, the infrastructure implemented will provide an environment

to implement optimization schemes like PSO and greedy over interconnected het­

erogeneous components.

The system is divided into two sections: operation and analytics. The opera­

tion will deal with the functioning of the communication while the analytics part

will deal with the model extraction and stability test. Both parts are required to

work in harmony to achieve optimal distributed control. The programs will not

provide this control, but rather it is a tool that will provide an environment to

design, test, implement and operate automatic control models which goal is to

achieve a set of optimal conditions. All this functionally will be accomplished by

configuring an interface that will show different levels of granularity of the real

Page 75: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

implementation of the networked control system.

For this design, it is assumed that the electronics part will be readily avail­

able, i.e., sensors and signal conditioning. With this consideration the rest of the

components are software. Thus a software engineering methodology was used to

construct the tool.

The first step is to simplify the architecture of Figure 19 eliminating all the

electronics in the node. Figure 33 shows the main components of the system.

A set of wireless nodes connected to the server through the Internet. A set of

mobile clients or observers is connected to the server. The server has a central

monitoring system that allows a Control Engineer to operate and adjust globalI

system properties.

Server

(«.»)

Internet Internet

(«.»)

Mobile Observers Central Observer Control Nodes

Figure 33: General Architecture.

55

Page 76: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

3.1 System Design

The Unified Modeling language (UML) was used to define the layout of the

software. UML is widely used in computer science and even in control engineering,

see [134]. Also, has capabilities for describing distributed and real-time applica­

tions [50]. UML is furthermore suitable to design automatic control systems like

in [158]. An excellent source is [17] in which UML is explained by its creators.

For this project many UML models were produced, here a subset of the most

critical components and scenarios will be presented. The design aims to translate

the requirements established during the previous phases into a representation that

could be easily converted into a computer program.

The development order of UML models for this project is Use Case, Sequence

and Class diagrams. With this subset of models, many of this software featuresI

could be explained. Several characteristics of the diagrams were omitted such as

temporal variables, private internal methods, or nonrelevant messages in order to

maintain clear readable figures.

3.1.1 U se Case Diagrams

The Use Case Diagram shows the way the system is used by internal and

external entities. External entities are called Actors and are represented by stick

persons inside use case diagrams. Every oval represents a way in which the sys-

56

Page 77: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

tem is used. Use Case diagrams were established for each mayor architectonic

component.

Figure 34 shows the first Use Case of the system. This use case describe

the uses of the node control. For the node, the server, actuators, and sensors

all represent external components thus they are modeled as Actors. A verbal

description of the diagram is that the server uses a communication module to

send and receive data to the actuators and from the sensors.

A c t u a t o r

S e n s o r

Figure 34: UML Use Case Diagram for Control Node.

As well, the server uses the communication module to send update orders to

the autonomous control module. The Autonomous control logic uses the math

57

C o n t r o l N o d e

.Q u ltnta M a t h M o d u l e

[C o m m u n i c a t io n l

*S e r v e r

Page 78: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

module, which is described later, to perform the mathematical operation required

to adjust the system’s state. Sensors must use a DAQ module which has all scaling

and transformations needed for each of the appropriate sensors. The Actuator uses

the system by an output module which will have the necessary interaction logic

for each particular actuator.

The node itself becomes an external actor every time it uses the mathematical

library module. Figure 35 shows this scenario. This mathematical module can

solve symbolic math and linear algebra. The libraries must include functions to

solve systems of differential equations numerically.

M a t h M o d u l e

P .S O

E x t e r n a l C a l l e r

G r e e d y <

S y m b o l i c m a t h

Figure 35: UML Use Case Diagram for the Math Module.

These libraries must support the creation of more complex functions and even

58

Page 79: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

larger libraries, such as neural network, fuzzy logic, particle swarm optimization,

greedy optimization, deep learning, etc.

The server’s Use Case is shown in Figure 36. Here the mobile clients and the

control node are external actors. The mobile observers use the communication

module to send data request or control actions to the mobile client. The network

statistics module will keep information about the network traffic. This information

will be used by the control module to be considered in the actual control scheme.

The control module uses the communication module to send and receive state

updates to the control nodes. The control module uses the math library and the

optimization module to compute the new states. The data logging module will

store the transmitted information from the communication module. The location

services module will be in charge of keeping track of a list of control nodes andi

their geographical position. Finally, the Control Engineer (CE) has to use the

server through a GUI. This GUI uses the user authentication module to verify the

user credentials. Then the CE could be able to access all internal modules of the

system.

Lastly, the Mobile Client’s Use Case diagram is shown in Figure 37. Prom the

mobile client perspective, the server and the CE both are external actors. The

CE uses the mobile client to monitor and control state variables using respective

GUIs. These monitor and control GUIs use the communication module to contact

the server for an update or to read the general system defined variables.

59

Page 80: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

. O p t i m i z a ! io n j - * M o d u l e .

C o n t r o l N o d e

d M a t h M o d u l e

Figure 36: UML Use Case Diagram for the Server.

Diagrams in Figures 34, 35, 36 and 37 could be all put together into a big

System’s Use Case. The primary purpose of the Use Cases is to define the main

objects to be implemented. Three main classes are observed: the client, the node,

and the server. Each containing several subclasses to achieve their tasks.

3.1.2 Sequence Diagrams

The Use Case Diagram shows the way the system is used but it lacks the

concept of time, i.e., the Use Case does not show the sequence of messages in the

time required to achieve one task. The Sequence Diagram is introduced to solve

this problem. In a Sequence Diagram, a set of classes are displayed on top, and

Page 81: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

M o b i l e C l i e n t

M i W d U I l

Figure 37: UML Use Case Diagram for the Mobile Client.

each class has a line of life. Between two classes a synchronous or an asynchronous

message could be sent. The time is represented downwards in the object’s line

of life. There, must be a sequence of messages for every mayor scenario of the

software.

Figure 38 shows the scenario when a CE reads values from the control node.

The objects b and c belongs to the mobile client. Objects d, e, / , g, and h are part

of the server. The control node has objects i and j .

The first part of this diagram shows the login process. The CE sends a message

to the monitor GUI with his username and password. Then the communication

module of the mobile client sends this Authentication message to the server com­

munication module. Then the user authentication module verifies the presented

credentials. If the user is verified, a message is sent to the location services.

61

Page 82: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

62

Figu

re

38:

UML

Sequ

ence

Di

agra

m

for

Read

ing

Dat

a fro

m a

Mob

ile

Obs

erve

r.

Page 83: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

This module loops through all nodes and using the communication module

sends an alive message to every node. If the control node gets the alive message

from the server, it sends an acknowledgment to the server using its communication

module. The gathered list of alive nodes is then sent to the mobile client for the

CE to select a node to observe. Once the node was selected, a data message is

sent to the control node from the server using the communication modules of both

components. The data is then sent to the CE and also to the data logging module.

Statistics about the communication is stored by the network stats module to be

considered in the automatic control.

In Figure 39 the sequence of autonomous control is shown. Objects a, b, c, d

and e belong to the server while objects / , g , h, i, j are part of the control node. In

order to update the control state, the autonomous control module requests inputs

to the data acquisition module. Then it sends the variables to the server using the

communication module. The autonomous control module computes a new state

using the math lib module. The new state is then transmitted to the actuators

using the output module. This update is the sent using the communication module

to the server. The server gets this update using its communication module. The

server will update its log and it also updates the value of the reading to the control

module. A local manual update will have an update message sent from the node

to the server. The server will update ts logo and control states.

63

Page 84: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

64

Figu

re

39:

UML

Sequ

ence

Di

agra

m

for

Aut

onom

ous

Con

trol

.

Page 85: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

In the case the server has a global optimization scheme, the server will compute

the new state. A global update message will send the update values from the server

to the control node. This sequence will loop indefinitely until a manual control

action overrides the automatic scheme.

3.1.3 Class Diagrams

The last part of the design is to generate a hierarchy of classes that implement

all the messages established on the sequence diagrams. Related classes are then

put together in packages. The project has three main packages: the control node

package, the server package, and the client package.

In Figure 40 a simplified class diagram for the control node package is shown.

This package consists of five principal classes. The control Node is the main class.

It has methods to respond to global adjusts from the central server, as well as

methods to respond to the autonomous control. Its many attributes must store the

control parameters to drive the node itself. The control node has a communication

class. This class is in charge of dealing with the sending and receiving of messages

to the server. The output class encapsulates all the methods required to send

values to the actuators. The input class will have all methods necessary to read

values from the sensors. The control Node class has a dependency on the math lib

packages since the autonomous control is required to perform some mathematical

operation to update the node state variables.

Page 86: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

C ontro l N ode Package

M a th L ib

M a th

I n p u t s : v e c t o r S t a t e O u t p u t s : v e c t o r S t a t e

so lv e (p ro b le m ) : r e su lt

C o n tr o l N o d e

C o n t r o l P a r a m e t e r s : u i n t

a u t o n o m o u s C o n t r o l ( a c t u a l S t a t e ) : S t a t e g l o b a l C o n t r o l A d j u s t ( n e w S t a t e )

O u tp u t

n u m b e r O u t p u t s : u i n t

w r i t e O u t p u t s : S t a t e

SiMSS: ISSisS'XC o m m u n ic a t io nI n p u t s

p o r t N u m b e r : u i n tn u m b e r l n p u t s

s e n d L o c a l S t a t e ( ) g e t G l o b a l S t a t e ( )

r e a d i n p u t s : S t a t e

Figure 40: UML Class Diagram for the Control Node.

Figure 41 shows the Mathematical Library Package. Its main class, Math,

has methods to solve systems equations and to perform advanced mathematical

formulas on a set of stored variables. This main class uses a set of external

classes to solve different problems. The fact that the library is divided, not only

obey the reason of encapsulation but also has to be with the different computing

capabilities of the node, i.e., no all the nodes support all the external classes. One

Page 87: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

of the external libraries adds a class that implements artificial neural networks.

It has methods to train different architectures, and it can support many numbers

of layers. The math lib package also contains a fuzzy logic class with methods

able to fuzzify and to defuzzify system variables. The fuzzy class is capable of

inference over a stored set of rules.

M a th L ib

A N N

n u m L a y e r : u i n t n u m l n p u t s : u i n t n u m O u t p u t s : u i n t O D E S o l v e r ( )

F i n i t e E l e n i e n t A n a l y s i s ( )F i n i t e D i f e r e n c e s ( )L i n e a r A l g e b r a ( )

M a thF u z z y

m a t h V a r i a b l e s

c o m p u t e Qs o l v e ( )

G r e e d y

P S O

O p t i m i z e Qg r e e d y C h o i c e ( )

p o p u l a t i o n : u i n t S y m b o l ic M a th

O b j e t i v e F u n c t i o n ( )O p t i m i z e

S y n i b o h c V a n a b l e s

S o I v e Q

Figure 41: UML Class Diagram for the Math Package.

Other important features of the Math Lib package are the Particle Swarm

Page 88: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Optimization module and the discrete greedy class. Those two classes provide

methods to elaborate distributed optimal control.

The core of the Math Lib package is the numerical solvers and symbolic math

classes which both provide a robust set of functions to manipulate and solve

mathematical systems to control.

Figure 42 displays the Server package. The main class is the Server class that

has methods to listen for connections and accepts them once requested. Also, this

class has a queue to process the responses and queries to the server.

The server is in contact with the communication class, which is responsible

for implementing the protocol, i.e., sending and receiving data from the multiple

clients and nodes. This communication class is composed of a network statistic

class. The focus of this class is to predict the delay of the communication between

nodes. The communication class is also composed of a location services class which

will keep a list of all active nodes and observers in the network. The server has an

aggregate class to log all information pertaining the system, i.e., variable values,

control actions, etc. Another aggregate class of the server is the control class. This

class is resporisible for the distributed automatic control of the system. Also, it

is responsible for performing stability tests on the system’s model. The control

class uses the math library package in Figure 41 and an additional optimization

class with all methods required to implement optimal control. The server is used

by a CE through a graphical user interface, which it has its class. The server also

Page 89: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Server

G U I C o n tr o lM a th

a c t u a l U s e rs y s t e m M o d e l

v a r i a b l e sA n a l y t i c sC o n t r o lD e s ig nT e s t

S o lv e Q

O p t im iz a t io n

r e e s t r i c t i o n s

D a t a L o g g in g S e r v e rl o c a l O p t i m i z e ( )g l o b a l O p t i m i z e ( )D a t a a c t i v i t y Q u e u e

r e s p o n d ( )l i s t e n ( )a c c e p t ( ) U s e r A u th

u s e r L i s t

L o c a t io n S e r v ic e s

N e tw o r k S t a t sa c t i v e N o d e s : u i n t N o d e L o c a t i o n s

C o m m u n ic a t io n s

n o d e S t a t s e x p e c t e d D e la y s

c h e n t L i s tA l i v e ? ( n o d e )f i n d N o d e ( )

s e n d ( )r e c e i v e ( )

s t a t A n a l y s i s ( )d e l a y P r e d i c t i o n

Figure 42: UML Class Diagram for the Server Package.

has a class dedicated to authenticate users and assign their respective roles.

The last system package is the mobile client of Figure 43. These classes are

intended to run on mobile devices and therefore must be light. The client class is

in charge of executing orders received from the User GUI or the Server. To get

the updates from the servers an aggregate class of communication is introduced.

The GUI is divided into two classes: one for control the variables and other to

69

Page 90: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

read the sensors.

C lie n t ' :

Or e q u e s t Qu p d a t e Qc o n n e c t Q

a c t i v i t y Q u e u e

C lie n t

s e r v e r A d d r e s s

s e n d ( )r e c e i v e ( )

C o m m u n ic a t io n

r e a d V a r i a b l e Qc h a n g e N o d e ( )p l o t V a r i a b l e ( )

M o n ito r G U I

U s e r

v a r i a b l e sn o d e

m o d i f y V a r i a b l e ( )r e s t o r e V a r i a b l e ( )

C o n tr o l G U I

Figure 43: UML Class Diagram for the Client Package.

3.2 Im plem entation

This section will present some implementation details about the system. Di­

agrams of Subsection 3.1.3 give a brief insight of the system’s interior structure.

Those diagrams directly translate to code. The main language chosen to imple­

ment this project was Java. Java was selected not only because it is the number

one programming language as June 2017 (according to the TIOBE index [160]);

Java was chosen for the great number of available open source libraries and its

70

Page 91: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

portability. Also, Java is the main language on the Android platform, which was

chosen for this project. Many examples exist of Java used in control such as

[41, 16, 137]. The control node program was developed using a language accord­

ing to its computing capabilities. Three main applications were developed: the

server, the mobile application, and the control node.

3.2.1 Server

This section describes the Server of the Open CACE system. The server is the

system’s most important part. To explain its implementation a brief description

of its GUIs is presented in this subsection.

Figure 44 shows the login interface of the server. This user authentication GUI

is presented to the CE through the central observer console, see Figure 33, and it

runs directly on the server platform. The open CACE server could be executed re­

motely u§ing a terminal

Once the CE provides his username

and password, the user credentials

will be authenticated against the user

database. Different users have distinct

access levels: Administrative, control

engineer, and report generator. The

administrative user will be in charge

71

connected to the server.

12

Open CACES e r v e r E dition

U se r A u th e n tic a tio n : ..................................................

U sernam e: jnah itt

Passw ord:

Login

Figure 44: Server Login GUI.

Page 92: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

of creating users and assigning systems

roles. The report generator can access the data logs of the server, and its main

purpose is to analyze in real time the system’s behavior. Lastly, the control user

has access to modify the distributed control scheme.

Once the CE is authenticated the system preview GUI of Figure 45 is shown to

him. The system preview GUI is divided into three main sections. The Preview

map, the geographic information section and the system status section.

' C on tro l R unning :

j {N euro-Fuzzy

U se r A u th e

' Mobile C lien ts:

! R e ce iv e d P a c k e ts :

. S e n t P a c k e ts :

; 1 ,7 9 2 ,4 6 3

: A ctive N o d es :

1 T ota l S e n so rs :

T ota l A c tu a to rs :

P re v iew Z one

EXITL ong: 0 [ l a s C ru c e s , NM

Figure 45: Server System Preview Login GUI.

A screen detail of the preview map is shown in Figure 46. This map is divided

into three layers. The first layer shows the data nodes. In this section, a selected

72

I

Page 93: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

subset of the system control nodes is displayed in real time with an excerpt of

their sensor readings. The second layer shows the network links between each

control node. Every link displays information related to the connection such

as the number of lost packets, of average delay. Each node of the data section is

represented on the network layer. The preview map represents this by connections

(red lines) going from the first layer to the second. The last layer represents the

control logic that the whole system is running. It shows a directed graph relating

variables. These variables could be mapped to inputs or outputs of the data layer.

This mapping is represented on the map preview by connection (blue lines) from

the second layer to the control layer. It is important to remember that the preview

is an accurate’ depiction of the real system but hides many details. This layered

map preview is interactive, i.e., responds to the user’s mouse updating its content.

Through this interaction the CE is able to examine all system’s details.

Another component of the system preview GUI is the System Status section.

This part will provide a quick report on the main system characteristic. Each

layer properties are displayed at their level on the preview map. In the control

layer level, the status shows the actual control scheme running and the preferred

optimization method. At the network layer level, the status displays the number

of connections and several statistics about the transmission of packets. The data

layer level displays, on the status section, information about the active nodes and

the total number of sensors and actuators on the system.

Page 94: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

74

Figu

re

46:

Scre

en

Det

ail

of the

Sy

stem

Prev

iew

GU

I

Page 95: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

The last part of the system preview GUI is the geographical control. This

section allows the CE to select the preview section of the system. It also allows

the adjustment of the displayed features on the map and the zoom level.

The preview map interactivity is achieved when the CE clicks inside of any

of the layers. Each layer is defined by a rectangle as shown in Figure 46. Three

scenarios could happen here: i) If the CE clicks on the data layer then, the GUI

of Figure 47 is shown, ii) If the CE clicks on the network layer then the GUI of

Figure 49 is open, iii) Finally, if the CE clicks on the control layer, the GUI ofi

Figure 51 will be activated.

This layered preview map is generated using actual information from the sys­

tem. It consists of a bitmap that can be displayed on the server application or

transmitted to another program such as a web page or mobile application. The

open CACE server considers the most relevant information to discriminate dis­

played data and improve automatically readability. Also, it could be configured

to ensure the display of certain components that could be used during design or

synthesis of the system. The system preview map represents graphically all the

components and interactions of the system.

When the CE clicks click on the data layer, the GUI of Figure 47 is shown as it

was mentioned before. This GUI is composed of two main sections: the map and

the status section. The map shows a geographic rectangle using the Google Maps

Application Program Interface (API), see Appendix C for a complete description

75

Page 96: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

of the libraries used in this project. Over this map, the set of control nodes is

displayed using the JUNG library.

Q y 5yst«m Prewi

File Edit

Geographic infbn Geo C ook

P Q D ata Layer N o d e S e lec tion O’J!

H ap: .............................. *.......................

\

<§2$

Y ' %

. G eographic Coordinates:

: Upper Left Com er:

: Longitude:

: [-107(209SI _]i Lower Rigtht Com er:

; la ttitu d e :

; h u ™ 3.5 1

; Longitude:

;

-Node I n f o : ..................

Node Position:

Longitude:

[-106.7637 ]

Node Name:

[B ID Drop 8

, j . N ode D etafe \

» Refresh map

Figure 47: Server Data Layer Node Selection GUI.

The status section displays the geographic coordinates that enclose the selected

control nodes. Each control node in the map could be clicked to select a particular

node to inspect. When a node is selected its information is displayed on the status

section. If the CE double-clicks the node, then the Data GUI of Figure 48 is

displayed.

The data GUI is the main interface to see the instrumentation of each node.

This GUI can add visualization panes to display different type of node variables.

76

Page 97: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

fi>l System Prtv:

f-ite Edit

( j j j 0» i2 tsy-fcf Mc«l<» M:«<t:pri GUI

Map: ...............................

'S , , .

Geographic Coordnates:

Upper Left Comer:

Lattitude:

[& j In strum en ta tion GUI for N o d e : 1AC034

File Edit M easu rem en t Panel

3056 :.....

ktude:

T e m p e r a tu r e : ...................G enerator Temperature:

W ater T enperature:

Choose Graph:

[Genera torT errp

G e n T e m p . \ Inlet P resage :

t -e T " 11 ; , ( ib ] p s i

| VIH) j j O utlet Presure:

| I | 25

( Computed Discharge:

i M, i i ; ; [3 l m A3/s

l ^1 r! i • O w ose Graph:

1 i ■ ! [W ater Speed v j

W ater S p eed

Figure 48: Data of Selected Node GUI.

: St-2095793 !

f f R t f i t Corner:

I 387029

Power Variables:Current: !

Acceleration:

O iocse Sensor:voltage Acceleration! Amp.

{Generator v ;

Code:

Frequency:YaMS

Choose Graph Zaxis:| n -X jrir -+-T axil -ftrZ airii |

wg<mfresh map

These graphs are implemented using the JFreeChart library. In the example of

Figure 48, the GUI is configured to show generator temperature, water speed,

power generated and three-axial acceleration.

On the other hand, if the CE clicks on the network layer of the GUI of Figure

45 the Link selection GUI of Figure 49 is open, as it was also mentioned before.

The main emphasis of this GUI is to select the link edges between nodes. It

contains two main components: the map and the status panel. In the map section,

the selected set of nodes is shown. Additionally, link lines are displayed, each link

edge represents an active connection between nodes. These links are responsive

77

Page 98: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

jj£) lysttm Previe

File Edit

j f j Link Selection GUI

•Map: .....................

Geographic inform Geo Coord?

*

<

*

"lip

□ X

'N etw ork Status:

N urbe r o f I r t s :

I16 iA ctrvelhks:

Link In fo : ..................• Up Speed:

: [&4_! tops.■ Down Speed:

Mbps.

Figure 49: Link Selection GUI.

to the CE interaction. If one click is performed upon a link, its status is shown

in the status panel. With two clicks on the link, the Network Analysis GUI of

Figure 50 is presented.

The Network Analysis GUI is divided into two sections: global and local statis­

tics. The global section displays network information gathered during the system

operation such as average up/download speeds. Also with this information can

perform global network related operations like modeling with probabilistic dis-i

tributions the network behavior. Those distributions are then used to predictI

network contingencies and generate a more robust operation. The local section

78

Page 99: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

(2} lin k Selection GUI

d System Prfvie

File Edit

Ai

Geogr

foj N etw ork Analysis

r Global S tatis tic s :..............-.. ............. ..................

; A verage Speed {UP) [ l o t T l M b p s

; A verage Speed {DOWN) jlQO ] hfops

• Percentage o f Orop Packages [lOO ]

Network Degree o f Dstrtoution:

p .Compute-.-: j

Markov Random Reid (MRF) model:

[~ .C onV ite t~ |

KemeHearrBng regression Models:

Packet Lost Dist.0.020 1

0.015

0.010 i

0-005

0.000 JNum. ofPackets

Displaying graph oft [Packet l o s t Pistrhution

A v era g e C on n ection S p e e d

S tatx Markov Model

Compute

Between Node: J1ACD34

and Node: }2BAAFE

Oroped Packages: [lOO

Displaying Graph of: p i c k e t lost:.-;.. .; — y ]

Figure 50: Network Analysis GUI.

Network Status: - -

Number o f L H s:

; & _____________ I

• Active links:

: P 1. Link In fo :-- Up Speed:

: L«______ ] Mbps.

: Dawn Speed:

; § _________i Mbps.

displays information related to the network operation of the selected link between

nodes. The system uses this information to solve statistical models to forecast

future connectivity behavior.

The last option the CE has on the GUI of Figure 45 is to click on the con­

trol layer to activate the Control Design GUI of Figure 51. This control GUI is

composed of four sections: The control graph, the node specification, the control

scheme section and the system operation.

The most important section of the control GUI is the control graph. It rep-

79

Page 100: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

f#>| S y s te m P rev iew Wmc!o*v

Flie Edit

z

z

zj

G eograpbc

Geo|

lo n g : )

(751- m -m J7i jn .. „ n>n, Control Running:

a a L c s i t iJ L

C o n tro l D esign GUI

Control G raph • - -- N ode Specification:

■ S e lec ted N ode:

o!,

5 6 7 8 9

| . f - T y p e : - ■............

i (5) O u tp u t variable

i 1 ; O In teg ra to r

; ;' O0erivator; O Summation

O S u b t ra c t io n

| ? i O C o n s tan t

I O C u s to m

|I ! Control Schem as: - ....

| NwdO

, System O peration:

S tabS ty Analysis B 1 {~ Run : j O p tir r a

Figure 51: Control Design GUI.

resents graphically the relations established between the inputs and the outputs

of the system. This graph is the model of the automatic control scheme running

on the overall system. This control graph is fully editable. Details about the

selected node could be obtained by clicking once on each node; this information

is displayed on the node specification section. With a double-click on the node,

the GUI of Figure 53 is displayed. That GUI allows the custom specification of

each node mapping. This custom code could be Octave or Java based.

The control scheme section allows the CE to add global neuro, fuzzy and

hybrid control. Those control scheme could be combined with the control graph

80

Page 101: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

to generate intelligent control modes.

The system operation section is used to invocate the main operations on the

system: Stability analysis, simulation, execution, and optimization. In the case

the CE presses the stability analysis, the GUI of Figure 52 is shown.

OS

fl> l S y s tem P rev iew W in d o w

File Edit

A

A

A

G eographic

Geo

L o n g :!

j j>i C o n tro l D esign GUI

- Control G raph -.....

■O1

/'

, <

- S ystem O peration: ......

j StefcSty Analysis

§>j S tab ility A nalysis

•-Plot:-----------*--------

Nyquist Diagram

0 .4 0.0 Q.e 1.0 1.4 1.6 1 8 2.0 2 .2 2 .4 2 8 2 8

Analysts O ptions; -

G raph Type: m iyq iis t Contour] v j p Edit G raph O ptions

J

Figure 52: Stability Analysis GUI.

The Stability analysis GUI of Figure 52 has the preconfigured procedures to

compute simple or custom stability analysis on the system model. This GUI also

can display graphs such as the Nyquist Contour.

The CE could click on each graph’s node to enter the custom node specification

81

Page 102: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

GUI. This interface shown in Figure 53 allows adding a mapping between the

system variables programmatically. Using the JavaOctave library, any Octave

program could be inserted as node activity. Also, the vast amount of scientific

and mathematics Java libraries allows generating an endless amount of complex

expressions.

f i l System Preview Window

Fite E dit

- X

[13 Custom Node. Specification: — L3 X

....Control

C ontrol Gr;

n :

zO ctav e a id e :

a X

f u n c t i o n x d o t *■ f ( x , t ) ifca b o n : --

ffl : r “ 0 . 2 5 ; N ode:

I l 1 ;

: U ser A uthent

-___

___

\__

____

_ i 0 b * 0 . 1 6 ;

c - 0 . 9 ;

d » 0 . 8 ;

x d o t ( 1 ) = r * x ( 1 J * ( 1 - x ( l ) / 3 c ) - a * x ( 1 ) * x ( 2 ) / ( 1 + x d o t ( 2 ) - c * a . * x ( l ) * x ( 2 ) / U + b * x ( l ) J - d * x ( 2 ) ;

e n d f u n c c i o n

it Variable

u t variable

g ra te r

irator

matron

;tr action

: !

s ta n t 1 :

| T e s t E xternally j j A ccept j om

z \ ZS •O e ta te J

C ontro l S d iem a s : •

G eographic

Geo

Long;

; <

- S ys tem O pera tion : - -

V

>

| N eural |

| Fuzzy ]

! C ustom I

: [ S tabS ty Analysis ' j Simulate j ^ R un { O ptim iration j

' - J

Figure 53: Custom Node Specification GUI.

82

Page 103: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

3.2.2 M obile O bserver A pp lication

This section describes the mobile application used to interact with open CASE

system. This application was developed using the Java language, to capture the

Android market. An implementation in Swift Language is in progress to support

Apple devices.

The mobile client must support the main features of the main observer. Many

of the mobile application activities have a parallel GUI on the server. When the

application is started on a mobile device, the Login activity of Figure 54 is dis­

played. After the user authentication process, the main menu activity of Figure

55 is activated. From this menu, the CE has three options: System preview; auto­

matic control;'and node observation and control. If the CE clicks on the “System

Preview” button, then the activity of Figure 56a is displayed. If the “Automatic

Control” is clicked, then the activity of Figure 56b is activated. Finally, if the

“Node Observation and Control” button is clicked, then the activity of Figure 57

is shown.

The available actions from this main menu depend on the user authentication,

i.e., not all users have all buttons activated. The first two buttons correspond

to events that require Access to the Server services, see Figure 56 while the last

button is related to control node activities.

Figure 56a contacts the server to acquire the system preview map. This ac-

83

Page 104: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Open CADCE (c) 2 01 7 by.Nahitt Padilla

S y s te m . Login In fo rm atio n : E m ail. ' <

n a h it t@ h o tm a il.c o m

tivity is analogous the server GUI of Figure 45, although it is not yet interactive

as the server one. It has buttons to access the systems status dialog, which

presents information about the system operation and control scheme being used.

This activity also has a connection

overview button to display a dialog

with information about the underly­

ing network connections. Figure 56bi

also contacts the server to get the sys­

tem ’s automatic control graph. This

activity is analogous to the Server GUI

of Figure 51. 1 It displays the control1

graph but does not allow to modify it

yet. With its buttons allows the in­

spection of the control node properties,

control scheme and also allows to verify

in which operation mode is the system

working.

Passw ord (oplio'iel)

SIGN IN OR REGISTER

1 2 3 4 5 6 7 8 9 0

q w e r t y u i o p

a s d f g h j k I

k z x c v b n m Q

i ' . S ig n in7123

Figure 54: Login Activity.

If the CE clicks from the activity of Figure 55 the button of “node observation

and control” then the activity of Figure 57 is invoked. This mobile client activity

is a simplified version of the GUI of Figure 47. It uses the Google Maps API to

display the geographic position of every control node in the system. It has two

84

Page 105: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

main buttons: “View Node Variables

First, the CE has to click on the control

node over the map to select the node

to monitor or control. The information

of the selected node will be displayed

such as name and node code. The CE

has two options. If the first button is

clicked then the activity of Figure 47

is displayed, but if the second one is

clicked then the activity of Figure 59

is activated. This activity allows all

the Google Maps functionality such as

zooming and panning. Saved map sec­

tions could be selected and displayed

by using a drop-down menu. Activities

launched from' this screen could be cataloged into instrumentation such a the Fig­

ure 58 or control like in Figure 59. The first ones allow the CE to monitor node

variables while the second to modify their values.

After clicking on the “View Node Variables” button, the CE is presented with

the instrumentation activities of Figure 58. All plots in this activity are gener­

ated using AChartEngine library, which allows making charts on Android-based

85

and “Manual Control Node Variables” .

Open CADCE (c) 2017 by Nahitt Padilla

C h o o s e y o u r m o d e ;

SYSTEM PREVIEW

% Q 6:45

AUTOMATIC CONTROL

NODE OBSERVATION AND CONTROL

Figure 55: Application Main Activity.

Page 106: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

@pOD©a©i SMfife ©te;€

CONTROL STATUS

CONNECTIONS OVERVIEW CONTROL SCHEMAS

GEOGRAPHIC INFORMATION

(a) Activity System Preview. (b) Activity Automatic Control.

Figure 56: Server Access Services Activities.

devices. This interface is configurable to display multiple kinds of variables avail­

able through the system. For example, in Figure 58a the activity is displaying

information captured from the accelerometers. A line plot shows the history of

the readings in three-axis.

Another example is displayed in Figure 58b where the generated voltages are

shown. This instrumentation interface has four buttons. One is dedicated to

controlling the kind of chart used to display the sensor data. Other to change the

Page 107: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

displayed information about a specific variable. Other to switch between plot and

only numeric representations, and the last one to define data logging preferences.

$ S an Y sid ro

«v» \ A■? \ X \

I ' ' - X - ' - i ; ' _• \\ .. . y ■, ;L a s C ru c e s

clln A A " " \t .(342)- . „

University,'

\\T n r t im a e U@M) M esiila T o rtu g as

! Google \

On the other hand, if the CE clicks

on the “manual control node variable”

button of Figure 47 the activity of Fig­

ure 59 is displayed. This activity has a

scrollable list of node variables. These

variables could be discrete, painted in

yellow, or continuous, painted in blue.

Each variable is described by its system

name. For the discrete variables, the

option is to turn it on or off. The con­

tinuous variables could take any value

between zero and a hundred; this per-

centage is later translated by the seal- Figure 5y. Activity Node observation

ing module of the sensor node. Once an(j Control

the value is established the CE must

have to press the apply change button to transmit the value to the node. Vari-i

able details could be consulted such as symbolic names and scales using the “Var

1V *943 Logitude -106 834022'lode Cocje v e f ACD34 Node Nam e E 8!D D rop8

VIEW NODE VARIABLES

MANUAL CONTROL NODE VARIABLES

System.Section Map: Mesiila Valley

Details” button.

87

Page 108: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Acceleration, VoltageLOG OPTIONS LOG OPTIONS

EDIT VARIABLES • CHANGE CHART SHOW NUMERIC EDIT VARIABLES :; CHANGE CHART SHOW NUMERIC:

.V oltageNow Monitoring: Acceleration

(a) View Node Variable Acceleration. (b) View Node Variable Voltage.

Figure 58: Instrumentation Activities.

3.2.3 Control N ode

The control node software depends extensively on the platform chosen to

deploy the system. For the selection of such devices, one important restric­

tion imposed by the initial hydro generator design was that it must be low

cost. Initially, the Arduino was selected as the only source of data acquisition

and control. Arduino has some virtues such as good sampling rate and ease

8 8

Page 109: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

of programming in a quasi-C language, but it lacks of computing power and

speed. Also, the data acquisition sample rate is divided among all input chan­

nels. To overcome these problems several Arduinos could be used in parallel.

Arduinos are not suited to perform

complex calculations on short times,

i.e., the autonomous control must be

very limited and preferable imple­

mented using lookup tables.

Not all systems dynamics could

be captured Using static approaches

such as the look-up tables. For more

complex automatic control, new states

must be calculated in real time. One

solution is to increase the computer

power. A low-cost solution is the Rasp-

berry Pi board. Raspberry Pi has com- Figure 5g. Mamla| Control 0utputs

puter power measured in GHz, not in

MHz like the Arduino, but it lacks analog inputs or outputs. This issue is ad­

dressed in various ways, see Subsection 3.2.3.2.

The control node’s low-cost restriction must be eliminated to achieve full au­

tonomous control power. Using National Instruments DAQ devices with powerful

Sontinous Output Var Name:' VtIN . / - ;

1 .

0 .0 .1 '

. . . Ob

IOC

Numerical Value 8 2 /1 0 0

APPLY CHANGE . VAR DETAILS j

Discrete Output Var Name: ■

OFF0 .1 .1

ON

Actual Value: 1/1

APPLY CHANGE M VAR DETAILS

2ontinous\Output;Var Name:MIN

1

0_2_1 . | * : - ON

ioo'

Numerical Value:. - - . o ' - 7 /1 0 0

APPLY CHANGE

"...

, VAR DETAILS

Discrete Output Var Name: OFF

0_3_1ON

Actual Value:V

1/1

J r ^ APPLY CHANGE’. .. ; VAR DETAILS

Sontmous Output Var Name: o. Xi ' 1

Page 110: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

laptops or the CompactRIO System, maximum performance could be achieved,

but the price will grow considerably in comparison to the low-cost options.

Open CACE implements three possible scenarios: i) Low cost, low computer

power: Arduino. ii) Medium cost, medium computing power: Raspberry Pi. iii)

High cost, high computer power: NI CompactRIO.

This section describes the basic characteristics the software has to have for a

control node to communicate to the framework of the Open CACE tool. In fact,

the suffix name of this work “using heterogeneous controllers” is introduced due

to the frameworks’ ability to be implemented in different running environments.

Later, the implementation of the mentioned scenarios is introduced.

The main general algorithm every control node must run regarding of its com­

puter power or implementation details is shown in Figure 60

The first labor of a node it to initialize communication with the server. Then

the node could act into two modes: autonomous or manual; these modes are

mutually exclusive. If the autonomous control is activated, the sensor node will

compute the new state using information from the local and remote sensors. De­

pending on the computing power of the deployment platform, this control could

be simple, for the Arduino case, or very complex, using a platform such as Rasp­

berry Pi or the CompactRIO. After the estate actualization, the control node

must broadcast its changes to the affected nodes of the system.

For the manual mode, the node must be able to process data requests and

90

Page 111: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

i Communication Initialization ;2 w h i l e true d o

3 i f Autonomous Control t h e n

4 request data from other nodes ;5 compute new state ;6 apply state change ;7 broadcast changes to participant nodes ;8 e l s e

9 i f message is read t h e n

10 Parse the message ;11 s w i t c h message type d o

12 c a s e Data Request d o

13 get sensor data ;14 form response message ;15 send response message ;16 c a s e Output update d o

17 extract data from message ;18 send data to actuators ;19 send acknowledgement ;20 c a s e Change Control Mode d o

21 Autonmous Control Received Mode22 e n d

23 e n d

24 e n d ,

Figure 60: Generic Control Node Algorithm.

modification to values from different mobile clients through the centralized server.

The node must implement the protocol of Figure 16. This protocol has explicit

instructions for the node to update the outputs. The node is also responsible for

responding with a well-formed protocol frame to the server with variables values

after a request is received.

91

Page 112: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

3.2 .3 .1 A rduino

Arduino is a general purpose microcontroller equipped with analog-to-digital

converter. It has inputs to sense the environment and outputs to interact with

actuators. Arduino has to communicate with the server of Open CACE. Some

Libraries exist for Arduino to support Java such as the JArduino library, see

Appendix C, but it has the limitation of not executing code, just accessing inputs

and outputs from Java code. The node implementation was developed usingi

sockets and precompiled functions to allow runtime customization of the execution

control scheme.

For this project the Arduino Mega was used due to its large number of available

inputs: 16 analog and 54 digital I/O . The principal Arduino’s limiting aspect is

its speed of only 16Mhz. Another limitation is its low memory of only 256 KB. A

picture of the Arduino Mega is displayed in Figure 61a.

Another important impediment on the Arduino is its lack of network connec­

tivity. This can be solved by using an external shield like low-cost “WiFly” of

Figure 18 or the more reliable official Arduino WiFi Shield of Figure 61b. Those

shields can be easily mounted leaving many available connections. Figure 61c

shows an application of the WiFi shield and the Arduino Uno.

Since the Arduino’s processing restrictions the implementation must remain as

simple as possible. The protocol could be implemented using a C-like structure,

92

Page 113: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

(a) Arduino Mega 2560. (b) Arduino WiFi Shield, (c) Arduino Uno + WiFi.

Figure 61: Arduino.

see Figure 62. With the help of the WiFi library, the control node must open a

socket and then it must establish a connection with the server cluster. The control

node then actively interchange packages with the server. Every time a package

is received a simple decomposition is performed to identify the required action.

Similarly, every output must be structured with specific information about the

message.

A simple lexical analyzer is utilized to process the protocol string. It is im­

plemented using a loop with a conditional inside, and it is used to dispatch the

message to its specific module. For the sensor read message, the readinput func­

tion from the Arduino will be invoked, then the control node will assemble the

response package, and it will send it back to the server. For the update output

message, the package contains information that identifies the output to be changed

and the new value as well. The control node will execute an analog output on the

specified actuator. Also, the control node will send acknowledges to the involved

Page 114: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

components as indicated in protocol sequences of Figures 15 and 20.

The autonomous mode is designed to execute very light control, i.e., simple al­

gebraic expressions and user-defined procedures involving standard mathematical

functions relating the local input to the local outputs. This mode on the Arduino

is very limited but, in conjunction with the server’s control module, i t ’s able to

perform operations that will compensate the global behavior while waiting for the

optimal response.

# d e f in e MAXPACKAGES 16

ty p e d e f s tr u c t {

char pacID [ 8 ] ; char S en so rD a ta [ 2 4] ;

} p a c k a g e ;

ty p e d e f s tr u c t {/ / openCACE p r o t o c o l im p l e m e n t a t i o n

c h a r Type [1 6 ]; c h a r V ers [ 4 ] ; c h a r F l a g s [1 2 ]; c h a r SourceN ode [ 1 6 ]; c h a r D e s tin a tio n N o d e [ 1 6]; c h a r SensorN um ber [ 8 ] ; c h a r S ensorT ype [ 4 ]; c h a r NumPacks [ 8 ] ;

! char SequenceN um ber [ 1 2]; char TimeStamp [ 3 2 ]; package D ata [MAXPACKAGES];

} f ra m e , *aFram e;

Figure 62: Arduino implementation of Open CACE Protocol.

94

Page 115: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

3.2 .3 .2 R aspberry P i

The Raspberry Pi is a single-board, low-cost, high-performance computer

first developed in the UK by the Raspberry Pi Foundation. The Raspberry Pi 2

is constructed around a quad-core ARM cortex which runs at speeds of 900MHz.

It also has 17 GPIO for connecting with devices.

The Pi 2 has a 10/100 Mbps Ethernet RJ45 connector and an SD card adapter.

To plug in a mouse and a keyboard, the Pi 2 has 4 USB ports and a video output

with HDMI connector. It comes with 1 Gb of RAM.

The newest model is the Raspberry Pi 3, see Figure 63, which is faster. It

runs at a speed of 1.2 GHz, and it has an 802.l ln wireless LAN and Blue­

tooth connectivity. It has 40-pin GPIO for communication with external circuits.

One big problem with the two mod­

els of the Pi is their lack of ana­

log input/output ports. A circuit

could be designed to solve this prob­

lem, but many top-of-the-shelf solu­

tions are available. The I2C P i’s

port could be used to communicate

with the ADS1115 (16 bits of resolution) or the ADS1015 (12 bits) from

Adafruit. Those boards are available at a price of 15 USD in this site h t t p s :

SW4OC0S

\ f X* ‘

HDmi

Figure 63: Raspberry Pi 3 model B.

95

Page 116: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

/ / l e a r n .a d a f r u i t . co m /ad afru it-4 -ch an n e l-ad c -b reak o u ts / .

Another solution is to buy a full board with a set of input/output options like

the one from Pi Plates, see Figure 64. This board could be bought from the site

h t tp : / /p i-p la te s .c o m /d a q c r l / at a price of iust 34 dollars.

This plate has eight analog inputs

and two analog outputs with 2 PWM

channels. One advantage of us­

ing this board is that the sellers

offer a set of libraries to interact

with the module from Python eas­

ily. With instruction such as val =

round{DAQC,getADC(0,8),l) the Pi

could read a value from the sensor 1,

avoiding thus all the programming ef­

fort required to build a custom library.

Another alternative is to use an Ar­

duino for the analog input/output and then send this information through the

serial port to the Raspberry Pi, see Figure 65. With this configuration, the Ar­

duino plays the dedicated role of an Analog-to-Digital converter. A library of

functions must be implemented to deal with the serial communication between

the platforms.,

96

( f i » 1 ■*••».I » *'#' t 'i i

o f t? a tn

Figure 64: Raspberry Pi Output Plate

from PiPlates. Retrieved June 28, 2017

from web site: http://pi-plates.com/wp-

content/uploads/2014/11 /ppD AQC-

1024x795.jpg.

Page 117: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Additional power is required to supply both boards. Also, an additional USB

cable is required to achieve the serial interface.

The Arduino performs pooling on the

inputs, and then it sends the format­

ted data to the Raspberry buffer. The

Open CACE implementation supports

both the Pi-plate and the Arduino in

serial communication with the Pi by

providing entry points on the frame-Figure 65: Raspberry Pi and Arduino.

work to customize the input/output

function calls.:

When this project began, the Raspberry Pi 2 was the standard. To fix its main

deficiency, a WiFi dongle is required, see Figure 66a. With this device, the Pi 2 is

ready to communicate wirelessly with the system. Such a device could be acquired

from www.ebay.com at 9.00 dollars the piece. The Pi 3 has a WiFi included as

mentioned before, so it is ready to use. To write the communication protocol the

standard Python socket library was used. Python has a very expressive way to

deal with strings and protocols. The code of Figure 67 shows the Open CASE

protocol implementation used for the Raspberry Pi node implementation.

97

Page 118: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

(a) USB WiFi Card. (b) Raspberry Pi + USB WiFi Card.

Figure 66: Wifi USB Dongle.

I

3.2.3.3 N ational Instrum ents Hardware

National Instruments had invested millions of dollars and thousands of hours

into the development of an integrated system that allows the software realization

of industrial automation. NI has achieved that with a line of hardware products

that allows data acquisition and control. Additionally, NI has its development en­

vironment, LabVIEW, a de facto standard that allows the graphical programming

of such systems.

The only drawback of NI is the cost; the programming environment LabVIEW

oscillates between 4000 and 6000 USD. Add to that all the required hardware and

the price skyrockets.

The Open CACE environment is in direct competition with National Instru­

ments approach, both try to achieve similar results, but the Open CACE platform

is based on low-cost open components and free software packages.

98

Page 119: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Still, it is important for our system to be compatible with the industry stan­

dard. In this section, the LabVIEW Node Implementation is described.

from s t r u c t im p o rt * im p o rt d a t e t i m e

# a c u t a l s e n s o r datas i = ’TYPE’ . e n c o d e ( ’u t f - 8 ’ )s2 = ’VERS’ . e n c o d e ( ’u t f - 8 ’ )s3 = ’FLAGS ’ . encode ( ’ u t f —8 ’ )s4 = ’ SOURCE JNfODE’ . encode ( ’ u t f - 8 ’ )s5. = ’DESTINATIONJtfODE’ . encode ( ’ u t f - 8 ’ )s6 = ’SENNUM’ . e n c o d e ( ’u t f - 8 ’ )s 7 ' = ’STYPE’ .en co d e ( ’ u t f - 8 ’ )s8 = ’NPACKS’ . encode ( ’u t f —8 ’ )s9 = ’SEQNUM’ • encode ( ’ u t f —8 ’ )slO = d a t e t i m e . d a t e t i m e ,now() . s t r f t i m e ( ”%Y—%n-%L%

H:%M:%S” ) . e n c o d e ( ’ u t f - 8 ’ )

sphck = p a c k ( ” 16s 4 s 12 s 16 s 16 s 8 s 4 s 8 s 12 s 3 2 s ” , s i , s2 , s3 , s4 , s5 , s6 , s7 , s8 , s9 , s l O )

p r i n t ( s p a c k )

^ u n p a c k i n g example from c o l l e c t i o n s im p o r t n amedt up le P r o t o c o l F r a m e = na me dt up le ( ’P r o t o c o l F r a m e t y p e ,

t vers , f l ags , sou rcen ode , de s tno de , sensornum , s e n s o r t y p e , numpacks , seqnumber , t i m es ta mp ’ )

r e s u l t = P r o t o c o l F r a m e . _make( u n p a c k ( ’ 16 s 4 s l 2 s l 6 s l 6 s 8 s 4 s 8 s l 2 s 3 2 s ’ , s p a c k ) )

p r i n t ( ” P r o t o c o l F r a m e - = _ % r ” % ( r e s u l t , ) )

Figure 67: Python Implementation of Open CACE Protocol.

Open CACE supports the use of USB DAQ devices, see Figure 68. The main

characteristic of these devices is that they communicate their readings and receive

their outputs via the serial interface of the computer running LabVIEW. NI offers

99

Page 120: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

different devices with different resolutions and sampling rates. The higher the

specs the higher the cost, like the device of Figure 68a with a cost of 1200 USD

plus the 4750 USD for the multifunction I/O device. A much cheaper option is

to use low-cost NI devices such as the one of Figure 68b with a cost of a couple

of hundred dollars. Still the programming environment comes apart at a price of

4000 USD.

S3?

(a) NI USB DAQ. Retrieved

June 28, 2017, from NI web site:

http://ww w.ni.com /cm s/im ages /

devzone/tut/06171005.png.

(b) NI Low-Cost USB DAQ. Re­

trieved June 28, 2017, from NI

web site: http://www.ni.com /en-

us/support/m odel. usb-6003.html.

Figure 68: NI USB Devices.

One, even more, cheap alternative is to use an Arduino instead of a NI

DAQ with the use of the library Linx that can be found on the next site: p

https://www.labviewmakerhub.com/doku.php?id=dibraries:linx:start.

This library provides a standard interface to the Arduino, i.e., in the same way,

1 0 0

Page 121: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

that the NI instruments devices are connected. In fact, the code implemented for

the Open CACE framework is too general that it will work with the Arduino

instead of a NI DAQ Device with only minimum adjustment.

The use of USB DAQ requires a computer to run. Another alternative is to

use the CompcatRIO. Figure 69 show NI CompactRIOan embedded industrial

type controller, see h ttp ://w w w .ni.com /com pactrio /. CompactRIO is able to

interact with different modules that increase its capabilities.

At a price starting at 8000 dollars

plus 2000 for every additional mod­

ule, the CompacatRIO is a Linux run­

ning machine with different configura­

tions ranging from'the 667 MHz to 1.91

GHz and storage up to 16 GB. Labview

with its real-time modules is used to

program the controller and to interact

with sensors and actuators. The same

program used to USB could be config-I

ured differently to be used with the CompactRIO making it fully compatible with

the Open CACE framework.

The protocol’s implementation on Lab VIEW requires the definition of the

cluster of Figure 70. A cluster is a way to represent C-like structures in Lab VIEW.

1 0 1

Figure 69: NI CompactRIO. Retrieved

June 28, 2017, from NI web site: p

http://www.ni.com/cms/images/ p

devzone/pub/Ql09JNL_pagel8_-

figl-jpeg-

Page 122: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

S tcpSimpleClient.vi Front Panel on ProtocolLexer.lvproj/My Computer *

File Edit View Project Operate Tools Window Help

□ X

■>> # ® I I 15pt A pplication Font .pD-r M te,

^ . . + . ^ , - P ro tocg |P rgmJe^ j L l | i | i l l l . . , U 4 - , r 4 . X j - ^ - j . j- j . t . 1

4 t t FltAGS*8 VERS FLAGS

DE-STSn ODESQURgEmODJDESTINATIONNODESOURCENODE

[email protected] SEn SQRIA'HE NUMBAGK'S SEQKI0IS1

SENSORNUM SENSORTYPE NUMPACKS SEQNUM

amp

a 12:43:14.879 PM7/27/2017

■cr .> v * ■■“ " T f T '

- . ..

ProtocolLexer,lvproj/M y C o m p u te r | <

Figure 70: Protocol Header in Lab VIEW.

This cluster could be then dynamically casted to a string to be transmitted over a

socket. Figure 71 also the protocol header casting and its visual implementation.

The simple socket cycle of life is depicted in Figure 71 also. It opens the

communication channel. Then transmits the casted protocol header, then finally

closing the connection. Lab VIEW allows the dynamic casting from strings to

clusters making it relatively straightforward to receive a string with the protocol

header and using it later inside a case structure.

3.3 Deployinent and Testing

The Open, CACE system was implemented on two Linux based laptops with

i5 cores and 2 GHz each. These servers were connected to a single router with

Page 123: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

i js tcpSimpleClient.vi Block Diagram on ProtocolLexer.lvproj/My Computer *

File Edit View Project Operate Tools Window Help

□ X

< J > _ # 1 ® _I|J[@_ ? S Wq 15pt A pplication F ont |- r ga v .pa T < 0 ^ [ I------1

P ro toco lFram e

jloca lhostj

gTcP11

Protoco llexer.lvpro j/M y C o m p u te r <

Figure 71: Header TCP Transmission in Lab VIEW.

internet access. The intention was to deploy the servers in the same development

environment. Servers provide service to an intranet, i.e., all mobile devices, control

nodes and servers must be on the same network. For the control nodes, three

Arduinos Mega, two Raspberry Pis, one computer with USB NI DAQ and another

computer with NI DAQ internal card were employed for development and testing.

The mobile application was developed on two Mac Book Pros with i5 processors,

and it was deployed on a Nexus 6p Android phone and the Sony Xperia (two

models). The test consisted of using the whole system to simulate a typical

running environment. It was run on a server, a set of control nodes and a set of

mobile devices.

The most complex part of testing is the control node since it has to use different

inputs and outputs. The first setup was to test it on the Arduino platform, using

103

Page 124: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

a function generator and a relay board, inputs and outputs were simulated. The

Arduino connected properly through the WiFi with the server sending data from

the sensor. Then the server sent his data to the mobile devices. Also, the mobile

device was used to update the control node’s state. Also, the automatic control

feature was tested by running a simple script. For the next platform, a Raspberry

Pi with a Pi-plate module was used instead of the Arduino in the configuration.

The same signals and actuators were used. The system performed in the same

manner. Next, the Raspberry Pi with an Arduino for DAQ was tested, giving

similar results. Lastly, the NI devices were tested. A PC connected to the network

with a PCI DAQ card was used, providing an excellent result. Then the same PC

but with a USB NI DAQ also passed the test.

104

Page 125: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

4 RESULTS

The project suffered a transformation process from a sensor network to a whole

framework supporting intelligent control and distributed optimization. Figure 4

shows the initial areas concerning the system’s development. In contrast Figure

72 shows the latest areas incorporated into the project.

Artificial Intelligence

IntelligentInstrumentation

IntelligentControl

Instrumen­tation

Control

NetworkedControlSystems

SensorNetworks Networks

Figure 72: Final Knowledge Domain Areas Involved in the Project.

105

Page 126: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

The grey section of Figure 72 represents the scope of this project and it means

that the Open CACE framework is on the intersection of the control, networks,

instrumentation and artificial intelligence areas. This is the main result, the cre­

ation of an infrastructure that allows the simplification in the creation of complex

distributed optimal control over a range of heterogeneous platforms. Many con­

cepts were merged to create robust communication and software models. Those

models captured the essence of the areas displayed on the diagram of Figure 72.

Those models later became the blueprints of the underlying framework’s software

implementation. This development was possible due to an implicit area in the

project: software engineering which is a subset of systems engineering. Software

engineering allows the software development in a controlled manner.I

Another important result achieved by this works is the reduction of the gap

existent between the praxis and theory in the area of distributed control. In Figure

73 two main approaches to control are shown: the mathematical or theoretical

models and the practical implementation of systems. Those sets mean that there

are several mathematical models to explain same number of systems. We know

how to analyze certain type of these models, and to write beautiful equations with

the ability to control them. On the other hand, the practical implementation

of control systems differs a lot from their theoretical counterpart. Just to give

a trivial example of this difference, in theory, variables are refereed x , y , z , . . .

and it is assumed that they are bidirectional, i.e., They can be read or written.

Page 127: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

u

Optimal

PracticalImplementation

TheoreticalModel

Figure 73: Gap Between Models and Implementations.

This situation!just can never happen in the real implementation, since inputs are

connected to sensors and outputs to actuators. There is, however, a soft spot

where mathematics perfectly applies to the real world. The infrastructure here

presented aims the gray section of Figure 73 by allowing the user to visually

analyze and build complex mathematical models with almost no effort. Then the

system automatically perform the complex tasks of mapping the mathematical

entities to real devices and to dealing with the transmission and state updates to

the whole operation. Furthermore, optimality has to be proven theoretically, thus

existing only a few of such optimal implementations. Open CACE also allows

the construction pseudo-optimal or optimal-tending systems to achieve this even

107

Page 128: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

harder task of implementing an optimal distributed control system.

4.1 Deploym ent R esults

In the more practical sense, not only models and abstractions were created.

Millions of lines of code were developed and typed. The implementation of these

models that intersects knowledge areas are by themselves a big result. This in­

frastructure contains primarily the server code, which is the heart and brains of

the operation;. Also, one android version of the mobile observer is contained.

Lastly, the implementation has four different nodes to be deployed on the control

nodes, one for Arduino + WiFi, one for Raspberry Pi + WiFi + P-plate, one

for Raspberry Pi 4- Arduino as DAQ and one main code for the NI platform in

Lab VIEW. For comparison purposes a test was designed and executed between

the Open CACE and Lab VIEW, see Appendix D for results.

108

Page 129: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

5 CONCLUSIONS

the implementation of distributed control systems is a very complex task due

to the integration of disparate technologies. The work presented here achieves

this integration with the development of a common running framework installed

on every system’s component. The framework consists of a series of libraries and

a custom communication protocol. The libraries contain functions that provide a

standard Application Program Interface (API) for the creation of more complex

programs. The custom communication protocol implements the required sequence

of steps to transmit information through the control network. The libraries and

the protocol enables a real-time transparent mapping of inputs and outputs to

state variables.

The framework is used to develop three main software components: the server,

the control node, and the mobile client. The server allows the user to create com­

plex mathematical models using the mapped state variables. The server auto­

matically translates the model into a real world application utilizing the control

nodes. The models supported by the framework include neural networks, fuzzy

logic, greedy optimization, and particle swarm optimization.

The principal innovation of the framework is the ability to apply classic control

methods to the distributed control system. It is possible to compute observability,

controllability, and stability of a distributed networked control system because of

109

Page 130: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

the next three main factors: i) The existent mapping of state variables: Sensors

and actuators are used inside equations, automatically updating their values when

changed, ii) The transparent communication implemented by the framework:

Communication allows the use of remote variables, residing in other nodes, like if

they were local to the node, iii) A standard programming API that enables the use

of multiple solvers and functions: The mathematical library has classes to store

vector spaces.1 These classes are used as inputs and outputs of other functions

such as numerical solvers and optimizers. In a nutshell, using the framework’s

combined tools, it is possible to design and implement automatic control schemes

using the visual server interface. One extra feature of the framework is the ability

to model the communication randomness of the underlying network dynamically.

With this model, the framework compensates on the delivery time of control orders

sent to the nodes.1

When the server is running concurrently with a set of control nodes and a set

mobile clients, the framework’s full potential is achieved.

The server presents to the user a visual environment for the development of

automatic control models. Also the server is in charge of running all background

services required for framework to work. Such services include: state variable

mapping, secure user authentication, geographic node location, data logging, and

collection of operation statistics.

The control node software module implements the protocol and API on three

1 1 0

Page 131: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

different platforms: Arduino, Raspberry Pi, and NI Lab VIEW. The Arduino ap­

plication allows for a low-cost implementation but not a very sophisticated local

control logic, due to its limitation in computing power. The Raspberry Pi and the

CompcatRIO were utilized as control nodes, to increase its computing power. The

framework allows the communication with multiple nodes concurrently, executing

on different platforms.

The framework’s last part is the mobile client. Every component of the system

could be monitored and adjusted using the application running on a mobile device.

Smartphones allow the distributed real-time monitoring of all system inputs. Also,

the mobile client enables the manual adjustment of every system output.

5.1 Future Work

Expand the library of mathematical functions to achieve a more specific and

robust implementation such as the Kalman Estimator for the sensor input process­

ing. Another important aspect to improve is the Graphical User Interface(GUI)

to make it more functional. Further development must expand the Interface con­

cept to create a true visual language with a well-defined syntax and semantics.

Other functions should be developed on the intelligent control library including

the support of Pulse-Coupled Neural Networks to process sequence of pulses in

time instead of vectors like the traditional neural networks do. One important

aspect to work in the future is to establish the computability limits of the models

Page 132: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

that could be executed by the automatic system. The interface allows the cre­

ation of multiple models. However, the model size, affected primordially by the

number of variables, could make it uncomputable. Thus, it is crucial to establish

the limits of the software.

1 1 2

Page 133: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

APPENDICES

A W A T ER FLO W M E A S U R E M E N T

The generator must adjust its behavior with the change of the water flowing

into it. In order to perform this adjustment, an accurate water flow measurement

must be realized. This section explores the current technologies in the filed of

electronic water flow measurement.

Examples of flow measurement technologies are: coriolis, differential pressure,

magnetic, positive displacement, target, thermal, turbine, ultrasonic, variable

area, vortex.

The obtained measurement could be: Volume flow rate (Volume), Mass flow

rate (Mass), Flow velocity (Velocity).

Some information provided in Appendix A.l was obtained from on-line man­

uals of [177]. 1

I

A .l F lu id P re lim in aries

For the instrumentation, many hours of many weeks were dedicated to theI

study of fluids and fluid dynamics. The first concept is the definition of fluid that

is a material that if some stress is applied the fluid will deform creating a flow.

It is assumed some basic knowledge of physical variables such as: pressure p with

units N / m 2. velocity v expressed in m /s, density r in k g /m 3 and viscosity m in

113

Page 134: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

terms of kg/(m — s) or also Pascal ■ s, and the obvious time in s (seconds)

Same laws apply to gases and liquids, for mathematical purposes they are both

fluids. The main difference between them is that you can compress gases but not

liquids.

The flow can be classified in either laminar or turbulent depending on how

viscous it is and how they react to inertia. This relation is expressed in the

Reynolds number.

The fluid flow can be modeled using the Navier-Stoke equations which governi

all laminar flows. In the special case of fluid with no viscosity, the BernoulliI

equation could be used to describe the behavior of the flow.

Bernoulli equation

For the Bernoulli equation to work, several facts must be assumed: The flow is

non-turbulent1 also called laminar due to the fact that is modeled with streamlines,I

An streamline is the path of the fluid in a flow. Also this flow must be perfect

that means has zero viscosity. Also the flow must be compressible due to its

change in density, this effect is presented on high velocity flows with speeds of 0.3

Mach, low velocity flows are usually uncompressible. Further more the flow must

be baritropic,: which is the flow whose density and pressure are not temperature

114

Page 135: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Dependant. Then the Bernoulli equation describing a constant fluid flow is:

V 2 P—— I- z H— = C(streamline) (7)29 9

This equation states that the addition of fluid velocity V, pressure P and

head z must remain constant C in along all streamline in a particular flow. This

constant could be different from streamline to streamline. In the case of flows

that can not create loops, called irrotational, The constant C is the same for all

streamlines in the flow. Usually only flows with no viscosity can be irrotational.

For an incompressible fluid the Bernoulli Equation becomes:

V 2 p7)— V z + — — C (8)29 99

A .2 Differential Pressure Fiow-m eters

From the many flow measurement studied to develop the original instrumen­

tation it was concluded that the differential flow method was the most common

method to measure flow inside pipelines. Other factor that had influence on the

selection of the differential pressure flow meter is the Venturi shape of the gener­

ator which perfectly fits the requirements of that kind of measurement.

In a nutshell, the differential flow uses the fact that the summation of forces

inside the pipe must remain constant, fact derived by the Bernoulli relation in

Equation 8.

115

Page 136: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

The working principle of this flow measurement is reducing the value in one

variable of Equation 8, then inferring the flow velocity, e.g., the flows goes thought

a pipe of diameter D and then is forced to pass trough a section of the pipe with

diameter d and D > d but still the summation of its pressure, height and velocity

must remain constant. This algebra is on the Appendix A.3, Combining this

velocity with the pipe diameter the water flow, in m3/s could be then computed.

The main drawback of differential-flow meters is that they generate a change in

pressure, in this particular application , this change is not a problem but a design

feature used to increase the generator’s blade speed. Industrial applications reefer

to this kind of application as Venturi Tube flow measurement

A .3 Differential Flow M eter Formula Derivation

The first important relation to establish is the power generated given a flow

rate: ij

Definition A .l . (Generated Power) Generated Power P by a water flowing

Idown from height h is defined a s P = p- h- Q- g - k , where

• P is the power in watts,recall the definition of a watt W = j = =

= V ■ A;S'5 5

• p is the density of water (~ 1000^);

• h is the height in meters;

116

Page 137: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

• Q is the flow rate in cubic meters per second;

• g is the acceleration due to gravity of « 9.8^

• k is the coefficient of efficiency ranging from 0 to 1.

Other very important relation is the Pressure p defined as

P = l W

Prom Newton’s second law F = m • g and knowing that m = volumen • p

Equation 9 becomes

V — h ■ p ■ g (10)

Pressure unit is Pascal, Pa =7 m * n - s z

Water discahrge Q is defined in m3/s as

Q = A V (11)

where A is the cross-sectional area of the portion of the channel occupied by the

flow m 2 and V is the average flow velocity m /s

On on the basis that the water is incompressible, the contiuity of water flow

is stableshed as

Q = AiVy = A 2V2 = . . . = AnVn (12)

Prom classical mechanics, the kinetic energy of a non-rotating object of mass

m traveling at! a speed v is

K e = ^ m v 2 (13)

117

Page 138: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

And its unit is the Joule defined as J = = N ■m = Pa -m 3 = W ■ s = C -V

Also Potential energy is the energy stored in an object due to its position

U = m ■ g ■ h (14)

if a point of equilibrium is assumed, i.e. where the potential energy is equal

to the kinetic energy, from 13 and 14

and doing some manipulation

v = y/2 ■ g ■ h (15)

Then substitute 15 on 11 to obtain

Q = Ayj2 ■ g ■ h (16)

At the orifice Flow and due to the physical properties of water a reduction in

the flow is noticed, see Figure 74. Introducing that term to Equation 16

Q = CcA^J 2 - g - h (17)

where Cc ss 0.61

To establish the energy balance flow relationships, from 10 the pressure energy

head can written as

Ph = (18)

P ' 9

118

Page 139: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

0.61 A

(a) Orifice with a head of water1 (b) Maximu area of flow

Figure 74: Contraction Coefficient. Retrieved June 9, 2016, from USBR, web site:

h t t p : / /www. usbr.gov/pm ts/hydraulics_lab/pubs/w m m /

Similarly the potential energy head from 14 is!

Uh =

m - g(19)

and the velocity head from 15 is

h =2 g

(20)

All heads are in meters. The total energy head can be calculated as Hi

pressure head + potential energy head + velocity head

Hi — hi + - 2- + ZiY l2 g

(21 )

119

Page 140: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Similarly at another downstream location

H2 — h,2 + + Z2YL2 9

( 2 2 )

Then the total energy balance is then

hi + —-—b Z\ — h2 + ——b Z 2Y i 2 9

(23)

Application of energy equation to measure flow

from the continuity equation it is concluded that V\A\ — V2-A2, and solving

from V2

Substituting 24 in 23, if Z\ — Z2

, VI . V i ( Ai 1+ 2g ~ 2 + 2g \ A 2/

solving for the head difference

' A '

then we can express V2

(24)

(25)

(26)

V,2 _ 1 ~ *2)29

'4 i^2

(27)

1 2 0

Page 141: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Taking the square root of both sides and multiplying by A\

(28)

Rearranging and including the contraction coefficient

Q — CcA iA 2{hi - h2 )2 g

A f - A l(29)

A .4 ISO 5 l6 7 Derivation

The International Standard Organization standardized in 2003 in the ISO

5167 standard named “measurement of fluid flow by means of pressure differential

devices inserted in circular cross-section conduits running full” the equation used

in instrumenting venturi type tubes. Derivation of this formula is presented here

to compare with Equation 29 used for this project.

Bernoulli equation states:

The first term p ■ g ■ h is the potential energy, If a constant height is assumed

Const = p- g- h + - p - v 2 + P (30)

where p is the fluid density, v is the linear velocity of the fluid element and P is

pressure.

121

Page 142: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

this term can be discarded.

Cost = \ -p -v 2 + P (31)64

here the term | p-v 2 represents the kinetic energy with density replacing mass.

Applying this equation to a circular cross-section pipe that is reduced in di­

ameter as it goes downstream in horizontal direction, see Figure 75

Figure 75: Pipeline Cross Section.

-p i ■ v\ + Pi — - p 2 ■ vl + P2 (32)

Mass is conserved (nor created nor destroyed) as it flows through the pipe

Qm = P2 • v2 ■ A 2 = pi • vi ■ Ai (33)

Qm is the mass flow rate along pipe, units kg/sec. Squaring both sides of 33

122

Page 143: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

and solving for v%

2 I Pi ' ^12

* = (34)

plug it in 32

2 ■ (-Pi — -P2) = p2 - v l ~ Pi- v\ (35)

then substituting v2 from 34 into 35

o (p n \ _ 2 [ 2 / P l ' ^ l V 2 | ,2 P2 ' (Pi ’ A l f ~ Pi ‘ (P2 ‘ A?)2

(36)

from 36 iq, can be written as

«i = \ /2 • (P, - P2) ■ , j — (pl • A i * -------------------------(37)V P2 ■ (p i • ^ i ) - p i • (P2 • Mr

this value is substituted on Equation 33

Q m = P i ' - t U = P i ■ (Pi - P2) • I 2 . (P' AA ' f {P\ ' „ (38)V p2 • (p i • Mr - pi • (P2 ■ Mr

Since the pipes are circular with D upstream diameter and d downstream

diameter. Their circular cross areas are: A 2 — it ■ {d/2)2 and A x — n ■ {D/2)2.

123

Page 144: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

After substituting on Equation 38 and ordering terms:

« « = J n T Z p ' i ' d2 ' %/2 ' <p i - p » ) ' P' <39)

with j3 = jj. This equation was only obtained from Bernoulli and mass con­

servation. It is very similar to equation (1) on page 6 of ISO 5167-1:1991(E)

document. In fact for uncompressible fluid, & = 1, it gets even more similar.

For more information the reader can consult the standard itself at:

ISO 5167-4:2003(en) Measurement of fluid flow by means of pressure dif­

ferential deviqes inserted in circular cross-section conduits running full h t tp s :

/ / www.iso. o rg /o b p /u i/# is o : s t d : i s o :5 1 6 7 :-4 :e d -1 :v l :en

124

Page 145: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

B INSTR U M EN TA TIO N IM PLEM ENTATION

This section presents design components generated during the original hydro­

generator instrumentation. Such components are the signal conditioning circuit,

the scale formulas sued for the inputs and the labview code for the Virtual Instru­

ment.

B .l Printed Circuit Board

The signal conditioning board encloses circuits from Figures 9, 11 and 12

into an single 'component with an interface which is easily connected to the DAQ

selected in this project. The schematic of the signal conditioning bard is shown

in Figure 76. This board has four I/O ports J 1, J 2, J 4 and J 5 which powers

the broad and send the amplified signals to the data acquisition segment. Thisi

port has two three-axial acceleration outputs. Also has two bipolar temperature

inputs connected to a Wheatstone bridge then to the instrumentation amplifier

providing temperature outputs in the range of the DAQ component.

It has the 'hall effect connector with it respective output. Also provides the[

current and voltage inputs for measuring generated power.

125

Page 146: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

OD53U f .

T E M P-U flTER ^]

126

I

Figu

re

76:

Sche

mat

ic

Sign

al C

ondi

tioni

ng

PCB

.

Page 147: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Figure 77 shows the final board before they were sent to the manufacturer. A

batch of five such tables were produced.

3 2

R18•{JTb}* ••O*•GD*n * ♦ f c l iU

•OU**{cTT}* •{H}*♦ }

,ni nc s Nm i *

•{cT|* ♦■Ic? >♦

R 1 9•O-*

J* *•

llllIlll

•fcTSl*• • • • • •

000000® l~pJL_pJ U p J l_ p j l_ p j

♦ ♦ ♦ ♦

o

$

3 5

•-EKJ-* •-EE}-*•-Ea*# ♦-E2>*

w

HyPER I n s t r u m e n t a t io n

HMSU

(a) PCB Silkscreen. (b) PCB Tablet.

Figure 77: Final PCB for Signal Conditioning.

B.2 Sensor Scales

After the signal conditioning phase, all outputs range from 0 to 5 volts. I t ’s

then required to scale the output to properly fit the sensor response. Every sensor

has a response table, provided by the constructor, that can be used to scale.

Unfortunately most of them the are not lineal at the extremes of operation. In

127

Page 148: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

order to simplify the scale calculations a lineal scale is preferred. All sensors were

selected with a good lineal response on the region of interest, so no data is lost

during the scaling. All scales are done in software.

Pressure Sensor

A linear response is assumed in the next interval: 0.5 Volt corresponds

to 0 PSI and 4.5 Volts to 30 PSI. In Figure 78 a plot of the sensor response

to the change in pressure is presented. So the line response is given by the

points (0,0.5); and (30,4.5). The line equation is Y = m x + b with m = (y2 —

yl) / (x2 — x l ) = (4.50.5)/(300) = 2/15. Therefore the offset value is b = 0.5.

The scale equation is Volts =

(2/15) * P S I + 0.5 If Volts

are given the scale can be

expressed as (Volts — 0.5) *

(15/2) = PSI , |

£

Acceleration Sensor

For the accelerometer the

operation ranges is: 1.5 VoltsI

— 0 G and the change rate in pjgure yg- Linearized Pressure Sensor Response,

gravities is of 300 mV — 1 G.

128

Pressure Scale

4

3

2

1

0 5 10 15 20 25 30

Page 149: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Figure 79 shows the sensor response to the change in acceleration. So the line

equation using the points (-5,0) and (5,3) is Y = m x + b with m = (y2 — y l ) / (x2 —

x l ) = (30)/(5 — (—5)) = 3/10. So b = 1.5 The scale equation is Volts = (3/10) *

Gs + 1.5 And given the volts the scale becomes (Volts — 1.5) * (10/3) = Gs.

Voltage M easurem ent

This section presents data on voltage monitoring output. Table 1 shows the

values obtained after a high-swing test.

Figure 80 plots the sensor

response to the change in volt­

age. So the line equation using

the points (—240,0.281) and

(240,4.719) is Y = mx+b with

m = (y2 — y l ) / (x2 — x l) =

(4.7190.281)/(240 — (—240)) =

0.009245833. 1 Therefore b =

2.5. The scale equation is

Vout = (0.009245833) * Vin +

2.5. And if given the Vout the

scale becomes (Vout — 2.5) * (1/0.009245833) = Vin

Table 1 also shows the values obtained after a low-swing test. In figure 81 the

129

Acceleration Scale

3

2

1

0

6 2 0 2 64 4Gravities

Figure 79: Linearized Accelerometer Response.

Page 150: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Table 1: Voltage Scale.

Nom-High swing Nom-Low swing

V,n (V) 240 -240 24 -42

V o u t swing(V) 4.719 0.281 2.888 2.112

Generator Speed Test for Voltage Sensor Calibration.

sensor response to the change in voltage is plotted.

So the line equation us­

ing the points1 (-42,2.112) and

(42,2.888) is Y = m x + b

with m = (y2 — y l ) / (x2 —

xl ) = (2.8882.112)/(42 -

(-42)) = 0.009238095. So

b — 2.5. The scale equation is

Vout = (0.009238095) * Vin +

2.5. And given the Vout the

scale becomes (Vout - 2.5) *

(1/0.009238095) - Vin.

Voltage Scale

5

4

3

2

1

0300 -200 -100 0 100 200 300

Vin

Figure 80: Linearized Voltage Response Nom-

High Swing.

130

Page 151: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Current Sensor

For AC current the ratio is divided as 0-5V corresponding to 0-150A of current.

Figure 82 graphically shows

the sensor response to the

change in current. So the line

equation using the points (0,0)

and (5,150) is Y = m x + 6

with m = (y2 — y l ) / (x2 —

x l ) = (1500)1/(50) = 30, so

6 = 0. The scale equation is

Current = (30) * V in

B.3 Lab V IEW Code

Voltage Scale

5

4

3

2

1

0 1- ...................................................

-300 -200 -100 0 100 200 300Vin

Figure 81: Linearized Voltage Response Nom-

Low Swing.

The original stand-alone

implementation of the hydro­

generator instrumentation is described here. This virtual instrument is displayedi

in Figure 83. Basically it is in infinite loop that takes both pressure measurements

and the it computes the water velocity to calculate next the water discharge. This

basic GUI has no external communication with any other program and it is de­

signed to monitor the energy harvester while connected to a USB NI-DAQ or an

131

Page 152: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

arduino working as DAQ. This Lab VIEW program allows the data logging and

in conjunction with the interface of Figure 14 is used to monitor all the system

response.

Current Scale

150

100

0 2 3 41 5Vin

Figure 82: Linearized Current Response.

132

Page 153: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

133

Figu

re

83:

Lab

VIEW

In

stru

men

tatio

n C

ode.

Page 154: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

C EX TER N A L SOFTW ARE LIBRARIES

This part a list of libraries developed by other companies or programmers

is presented. These libraries influenced the development of this project either by

using some code or borrowing some implementation aspects. When code was used,

proper care was taken in not violating any copyright. The name of the package is

provided next to a link to its web page.

• Open Source Fuzzy Logic Library and FCL language implementation: Used

a the base of the fuzzy logic package of this project.

h t t p :/ / j fu z z y lo g ic . so u rc e fo rg e .n e t/h tm l/jav a .h tm l

• Neuroph, java neural network framework: Used to implement the neural

network training algorithms.

http://neuroph.sourceforge.net/

• Java genetic Algorithms Package: Used to implement the genetic and evo­

lutive optimization section of this project

h ttp : / / j gap. sourcef orge.net/I

• Open Source Swarm Optimization Package: The PSO code of this project

is based on this library

h ttp : / / j swarm-pso. sourceforge.net/

134

Page 155: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

• JavaOctave: This library allows to run Octave scripts from java, it was used

to enable the matlab based control definition in the software.

h t t p s : / / k e n a i . com/proj e c t s / j avaoctave /pages /H om e

• Java Universal Network/graph Framework: This library is the base of all

graphical graph/network representations used in the GUIs. The node selec­

tion interface, and the control design relay heavily in this package.

h t t p : / / j u n g . s o u r c e f o r g e . n e t /

• JFreeChart: A library to plot graphics, heavily used in the node instrumen­

tation visualization on the server.

h t t p : / / www. j f r e e . o r g / j f r e e c h a r t /

• GoogleMaps API: This library was used in the server and mobile client, and

allows the creation of interactive maps in real time.

h t t p s : / / d e v e l o p e r s . g o o g l e . com/maps/

• JScience, java tools and libraries for the advancement of sciences: It contains

a set of mappings to mathematical structures such as rings and fields. Also

has a matrix template to solve linear algebra.

h t t p : / / j s c i e n c e . o r g /

• Apache Math: This library contains many useful classes to solve problems

that range from ODE to machine learning.

135

Page 156: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

h t t p : / / commons. a p a c h e . org/p roper /commons-m ath/

• JavaCalculus: Allows the program to parse algebraic expression from strings

h t t p s : / / c o d e . g o o g l e . c o m / a r c h i v e / p / j a v a c a l c u i u s /

• Michael Thomas Flanagan’s Java Scientific Library: Some of its functions

to solve control systems are used in the system’s math library.

h t t p s : / /www. e e . u c l . a c . u k / ~ m f l a n a g a / j a v a /

• AChartEngine: It is a library to make charts on Android based devices. It

is heavily used on the mobile client in the instrumentation section.

h t t p :/ /www. a c h a r t e n g i n e . o r g /

• JArduino: Allows the reading and writing of inputs to a serial communicated

arduino from a java program, it has the limitation of not executing code,

just accessing inputs and outputs.

h t t p s : / /g i th u b .c o m /S I N T E F - 9 0 1 2 /J A r d u in o /w ik i

I

• Mikhail Gorodetsky Python’s packing and unpacking library: Used in the

protocol production for the raaspberry pi pythno client.i

h t t p s : / / p a s t e b i n . com/XJyZMyHX

136

Page 157: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

D SYSTEM PER FO R M A N C E COM PARISON

To compare the system developed in this dissertation with the commercial

top-of-the-shelf software the next hypothetical scenario is supposed. There are

two lighthouses, one in the New York coast and other in Los Angeles. Figure 84

depicts the distance of about four thousand Km between the two points. Each

Lighthouse has a turning mirror that rotates at variables speeds over a bright

incandescent light. The mirror turns thanks to a motor that has hall-effect speed

encoder.

Lighthouse

Lighthouse

Figure 84: Lighthouses in the East and West Coast.

The motor on NY is running at full speed while the one in LA is stalled. The

idea is to make them spin at the same rate automatically. This coordination is

137

Page 158: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

achieved because each lighthouse has an identical node comprised of: i) A motor,

ii) A tachometer attached to the motor shaft, iii) A Micro-controller with the next

abilities: it can acquire the speed sensor data, and it can adjust the actual motor

speed to achieve the desired speed reference. Figure 85 depicts the lighthouse’s

control node.

SpeedAdjustment

Speed.ererenceDAQ and Control

Figure 85: Control of Lighthouse Mirror Motor.

i

D .l Control A rchitecture

To achieve coordination among the lighthouses’ motor speeds the next se­

quence of events must occur. Each control node must send its current speed to

the other node and receive the other node’s speed, then use the received infor­

mation to adjust its own. The other control node must transmit and receive the

138

Page 159: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

same information simultaneously. This configuration is depicted in Figure 86.

This architecture shows the main problem in distributed control: communication

between nodes. Communication is problematic for the next reason: if the LA

node is running a zero velocity, this is the information that will be sent to the

NY node. NY node also will send the full speed to the LA node. These mes­

sages could reach their destination at different times, but once they have arrived,

they affect the speed of its receiving node. Every node is supposed to send its

speed to the other at a constant rate no matter if it has received a new update

or not. T h is1 phenomenon implies that for several moments during the control

process, the node will send a state and immediately it will modify its state with

newly received information, thus making the sent information inaccurate. These

discrepancies make harder to coordinate the lighthouses’ motor speeds accurately.

D.2 Test Description

A test to compare Lab VIEW with Open CACE was designed in the next

form: two simulated nodes running a motor model with its speed control are

connected through a local network with added random latency. To simulate the

latency from LA to NY information provided from the next AT&T site was used:

h t t p : / / ipnetw ork . bgtmo. i p . a t t . net/pw s/ netw ork_delay. h tm l. A simulated

normal distributed latency of 60 ms average with a standard deviation of 20 ms

was used in the test.

139

Page 160: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

SpeedAdjustment

SpeedReference

DAQ and Control

Los Angeles

l auJsVrSpeed

Adjustment

reed

Speed DAQ and Control k J

Reference

New York

Figure 86: Simple Distributed Control.

D .3 Lab VIEW Im plem entation

Lab VIEW offers at least three methods for implementing distributed con­

trol systems: i) TCP/U D P Sockets, ii) The DataSocket Transfer Protocol, iii)

The Distributed Control and Automation framework. The TCP/U D P Socket is

the most low-level form of communication implemented by NI. It consists of Vir­

tual Instruments (VI) that visually implement the socket API in LabVIEW. This

Imethod is the fastest in LabVIEW, but it is also the most complex to implement[

since all the bookkeeping effort is delegated to the programmer.

The DataSocket Transfer Protocol (DSTP) is an encapsulation of the simple

socket. It allows a simpler communication between VI by hiding from the pro­

grammer all variable casting details. This method puts an additional execution

140

Page 161: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

overhead in the program, making it slower than the simple Socket approach.

These first two methods are too complex to implement leaving all the ab­

straction of the controller tasks, variable mapping, and updating to the program­

mer. To reduce this complexity, NI has The Distributed Control and Automation

Framework (DCAF). Is against the DCAF that the comparative test is performed

due to its similar approach to variable mapping, and variable auto update.

To set up a distributed control using DCAF in LabVIEW the user must create

a configuration file describing the system using the interface of Figure 87. This

interface allows the mapping creation of measured variables. Also, this interface!

sets up the control logic used by every node. Every control node, or in the DCAF

lingo: standard engine, could be configured with a specific IP address to access

distant nodes through the Internet.

Figure 87 shows the DCAD configuration of a simulation consisted of three

engines, one for each lighthouse motor, and a principal monitoring station to

display the results to the user. Running a distributed control has to implement

all the engines and to put them in a loop, this is depicted in Figure 88. Additional

VI are required for every engine implementing the control of the motor, also a VI

for the visual user interface. Each engine must be in different VI files.

The programmer must define a VI that allows the graphical display of the

system properties. Then the monitor interface must be hooked up to the stan­

dard engine (UI) inside the DCAF configuration interface. After that, the User

141

Page 162: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

{ 3 SimulatedSystem.pcfg

File Tools About

B t l3 System Configuration 3 © PC

3 0 Standard Engine (UI) ; -j:::Tags : Mappings• |Q Ul Reference ' 9.M UOP

S 0 Standard Engine (Simulation) iS iiiT ags

: &S&K99*• (35 Motor Controller Logic : - J Motor Model; US UOP

: & 0 Standard Engine Tags

■ Mappings i IjScRIO

Manual Mapping ; Visualization ; Automated Interface Matching

Channel Module a Tags | aTemperature Motor Controller li output range high Motor Controller Li output range low Motor Controller Ic

FanLampSetpointP

Thermocouple Rea/ Motor Model Temperature i UDP

1

| .

V !

Tag Dir Channel Module a

Setpoint >> setpoint Motor ControlP » Kc Motor Control1 »> Ti Motor Control0 . » Td Motor ControlFan on? ; > > Fan on? Motor ControlFan << fan Motor ControlLamp << . lamp Motor ControlFan » Fan PWM Motor ModelLamp . » Lamp PWM Motor Model0 . «< . D ' UDPFan on? < < Fan on? UOP1 i << : 1 UOPlam p ' >> lam p UOPP : « p UOPSetpoint << Setpoint • UDP

Figure 87: LabVIEW DCAF Configuration.I

Interface (UI) must be placed in the view section of the program of Figure 88

allowing the centralized monitoring of the distributed control system. This UI is

shown in Figure 91.

D .4 Open CACE Im plem entation

A simple' mathematical model for the lighthouse motor control is shown in

Figure 89. The transparent mapping of inputs and outputs implemented by the

open CACE framework allows the simple creation of a graph that represents the

system. Figure 90 shows the control model for the lighthouse problem. Every

square on the model is directly implemented as a custom function executed by

Ithe node.

The other node in LA will run exactly the same program only switching NY to

142

Page 163: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Pams imi Liy'iUiouJC Mtrts;S I SimulwcrfSw

Fite Edit VimFite Edit View Project Operate Toob Window Help

;#[ii| i TSpt Application font

DCAF Editor

30 System Co£r Ci Stem

s Host Matn.vi Stock Diagram on Lighthouse Motor Controller Testtvproj/My Computer

File Edit View Project Operate Toob Window Help

: ^ ] M ] ( ^ ] Uu 'q* XJ r. I5pt Application font - J ' ? a ’"](-Cn*"j

: ’ K JU 1 0 Ci Stemt : j r T,

’ : ‘ r-fit hr

N

: K JU j 6 G Stem

L j r T.1 ; Shi- 1 3 cRJO

Open end initialize the onftguration for the

specified target alias. [Monitor engine state}

[Check for errors! f* Stopped"

|Close everything!

n the engines I

Thu VI automatically opens when run and the Ul Reference module running mithin the engine will interact with H.

E l i ---------

file nameBuSer

target alias

ED—

This VI contains all of the engine's dependencies. This VI can be updated automatically in the 'Target'

of the Configuration Editor.

Figure 88: DCAF Main Test Module.

LA on the variable description. Both programs are assumed to be inside an infinite

control loop. To achieve this two nodes are activated inside the data layer, and a

random latency is specified. Each node automatically maps its inputs and outputs

to state variables. Custom names could be specified into an XML configuration

file. Then on the control layer, the logic is specified. With a click on the screen,

the framework presents a list of available variables. After placing the variable

in the model space, the user could also select a custom bitmap to represent the

variable. With another click on the screen, the user places variables and custom

functions to operate the controller. Finally, variable flows must be specified by

connecting elements on the screen. Figure 90 shows the implementation of the

143

Page 164: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Error

LA(t)

NY(t ) Control Dynamics

Figure 89: Mathematical Model of the Lighthouse Problem.

lighthouse problem in the Open CACE framework.

After defining the data layer and the control layer, it is necessary to specify

how the node will visualize the system behavior. To do this, a double-click on the

node over the map allows the adjustment of visualization panes. For this test only

one line plot was wired to the system variable of the local motor speed, this can

be seen in Figure 92. The speed behavior tends to the half velocity point stably.

D.5 Test Results

The first test was done using LabVIEW and DCAF. The three engines were

started: two control nodes and the main monitoring UI. The execution result is

shown in Figure 91, and the data log is presented in Table 2.

The second part of the test consisted of the execution of the same control in

the Open CACE environment. The instrumentation GUI of the test is shown in

Figure 92. Also, its log is included in Table 2.

144

I

Page 165: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[j> | Control Design GUI, - □ X

Control Graph . .................. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........................................ . . . . . . . . . . . . . . .A ; : Node Specification:

; j Selected Node:

I L A

i ! I ° ..................................... I

: ; ; Type: . . . . . . . . . . . . . . . . -- • :

i O In P u t Variable |

( 3 I il ® Output variable

i j ; ; 0 Integrator

Error -— __^ControlDynamics \ I\ 0 Derivator

: ! Q Summation

' 1 2 ! * ; 0 Substraction !

I j ; O Constant

'■ \\ O Custom

Q 1N Y

i , ------------------------j j j E d it Details |

0 \ - Control Schemas: •

| ; | Neural |

<... ............. ............. ....................................... .................. . ......... . ......_________ | | | Fuzzy |

System Operation: --------- - .............. - .......... ............ - .......... ............... --------------------- !| ! | Custom | i

1 Stability Analysis Simulate 1 .. Run - ,1 v Optimization. ; :•

— .... . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ........................................................................................ *

Figure 90: Open CACE implementation of the Lighthouse Problem.

Data logging was specified to occur every one-tenth of a second. The test run

for five seconds providing fifty samples. A toe-to-toe plot of both execution results

is presented in Figure 93. It is not clear from the simple visual plot inspection

what system is performing better than the other. Nevertheless, smaller and more

compact oscillations could be observed in the OpenCACE execution trace. To

establish if there is any statistical difference between the two signals a student

t-test was performed.

145

Page 166: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

a lig h th o u s e M o to r Speed GUI Vt.vi

File Edit V iew Project O perate Tools W in d o w Help

" • H * i ® i n

w v "f*» | 5 ,^ )

IlkihtHouseiMotoftStiTiuiatjon

\swmmTimeTSfo'flftw**m

Figure 91: LabVIEW Simulation of the Lighthouse Motor Distributed Control.

D .6 Student’s t-test

This section presents the procedure of the Student’s t-test. This test is used

for comparing the means of two samples (see Table 2), in simple terms, the t-

test compares the actual difference between two means with the data variation,

represented by the standard deviation of the difference between the means.

The first sample is the time, in seconds, of the data acquisition. The second

146

Page 167: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Instrum entation GUI for N o d e : 1ACD34

File Edit M easurem ent Panel

Motor Speed: • -

Motor Speed

Sec.

Speed:

.55

3 .0 3 .5 5 .0

Motor Speed]

Figure 92: Open CACE Simulation of the Lighthouse Motor Distributed Control.

sample is the motor velocity of the LabVIEW implementation, and the third one

is the Open CACE motor speed. The experiments are paired, i.e., both executions

were realized at the same time interval.

Procedure

1. Construct a null hypothesis: A suitable null hypothesis is that there is no

significative difference between the two execution speeds.

2. List the data: The data obtained after the paired experiment is shown in

Table 2.

3. Record the number of replicates: rij = 51 and n2 = 51. Note that the

147

Page 168: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Table 2: Simulation Results.

X y l y2 X yl y2 X yi y2

0 1 1 1.7 0.3 .27 3.4 0.63 .490.1 0.66 0.56 1.8 0.26 .3 3.5 0.6 .430.2 0.18 .17 1.9 0.33 .4 3.6 0.53 .370.3 0.48 .47 2 0.51 .56 3.7 0.49 .440.4 0.6 .59 2.1 0.57 .64 3.8 0.45 .470.5 0.84 .87 2.2 0.6 .7 3.9 0.4 .510.6 0.49 .53 2.3 0.75 .65 4 0.37 .540.7 0.37 .35 2.4 0.69 .60 4.1 0.38 .570.8 0.21 .23 2.5 0.63 .53 4.2 0.42 .530.9 0.33 .35 2.6 0.57 .46 4.3 0.44 .471 0.48 .56 2.7 0.5 .37 4.4 0.48 .44

1.1 0.54 .60 2.8 0.48 .40 4.5 0.52 .401.2 0.72 .63 2.9 0.36 .43 4.6 0.56 .361.3 0.84 .83 3 0.42 .50 4.7 0.59 .411.4 0.72 .70 3.1 0.5 .53 4.8 0.57 .461.5 0.51 .49 3.2 0.53 .57 4.9 0.54 .511.6 0.45 .42 3.3 0.63 .54 5 0.49 .55

Motor Speed Results for the Lighthouse Problem Simulation. The column x is the time in seconds, y l is the motor velocity in LabVIEW and y2 is the motor velocity in Open CACE.

Student’s f-test works even with a different number of replicates.

4. Calculate the mean of each sample: for the LabVIEW it is y\ = 0.5198 and

for the Open CACE y2 — 0.5049 respectively.

5. Estimate the Variance of the populations using the sample: for the sample

one it is a\ = 0.0249 and for sample two a \ = 0.0226. This must be

148

Page 169: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

- LabVIEW- Open CACE

0.8

'■ da>cpaco

0.4

0 2 3 41 5Time in seconds

Figure 93: Test Execution Comparison,

represented as S f and Sf respectively since they are samples.

6. Calculate the variance of the difference between two means: This step is

achieved by ad 2 = & + = M2M + o^e, = 0.000931373J a n \ r i2 51 51

7. Calculate ad- Simply take the square root of the previous result ad =

0.030518397

8. Calculate the Lvalue: this vales is obtained as follows t — when doing

this transpose the means so a positive value is always obtained. In this case

0 .5 1 9 8 -0 .5 0 4 90 .0 3 05 1 8 39 7

0.488230101

9. Get the Lvalue: This step is performed by entering the Ltable at (n i+ n 2—2)

degrees of freedom; and choose a significance required (normally p = 0.05)

149

Page 170: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

and read the tabulated table. In this case the degrees of freedom are 98 and

the t-value at 0.95 is 1.660

10. Analyze the result: If the calculated t value exceeds the tabulated value it is

said that the null hypothesis was rejected , i.e., the means are significantly

different at that level of probability

This test is enough to ensure that the speed obtained from LabVIEW program

is not statistically different from the speed that takes the Open CACE version at

95 percent leyel of confidence.

D .7 Test Conclusions

The result of the two test not being statistically different indicates a hugeI

bias in the simulation. The pseudo-random number generator initializes the seed

at the beginning of the execution providing the same amount of pseudo-randomI

delay. This determinism in conjunction with the sampling rate creates very similar

signals that display the nature of the simulation. More experiment replicas could

be considered to mitigate a little the effect of pseudo-randomness. The definitive

test is a race between LabVIEW and Open CACE using real motors and real

distances, until the day of more resources the simulated results must serve as a

baseline.

The intuition affirms that Open CACE should be faster than LabVIEW. The

150

Page 171: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

main reason why Open CACE should outperform LabVIEW is the fact that Lab­

VIEW is an interpreted language heavily based in graphical interfaces while the

framework developed in this dissertation focuses on the minimalistic implementa­

tion of the communication, increasing thus its performance. Just the DACF adds

more than five hundred additional modules, and it bases most of its communica­

tion on a third party app called Putty.

The point that Open CACE easily won is the ease of use compared to a regular

programming language and even the Visual language of LabVIEW.

151

Page 172: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

References

[1] T. Abdelzaher, Tian He, and J. Stankovic. Feedback control of data aggregation in sensor networks. In Decision and Control, 2004■ CDC. 43rd IEEE Conference on, volume 2, pages 1490-1495 Vol.2, Dec 2004. d o i :1 0 .1109/CDC.2004.1430254.

[2] Adam Adamkowski, Waldemar Janicki, Janusz Kubiak, Gustavo Urquiza, Fernando Sierra, and Jose M. Fernandez D. Water turbine efficiency measurements using the gibson method based on special instrumenta­tion installed inside pipelines. 6th International Conference on In­novation in Hydraulic Efficiency Measurements, 2006. URL: h t tp : / / www.ighem .org/Paper2006/dl5.pdf.

[3] Anish Aggarwal. Overview of instrumentation, scada & telecommunica­tion systems of pipelines. Indian Oil Pipelines Division, 2009. [Online; accessed 5-July-2015]. URL: h t tp : / /p e tro fe d .w in w in h o s t in g .n e t/ ;> upload/1AI/T0&G/II_Agarwal_10CL.p d f.

[4] P. Albertos. Fuzzy controllers. In L. Boullart, A. Krijgsman, and R. Vingerhoeds, editors, Application of Artificial Intelligence in Pro­cess Control, pages 343 - 367. Pergamon, Oxford, 1992. URL: h t tp : / / w w w .sciencedirect. com /science/artic le /p ii/B 9780080420165500243, d o i :h t t p s : / / d o i . o rg /1 0 . 1016/B978-0-08-042016-5.50024-3.

[5] C. Alippi and C. Galperti. An adaptive system for optimal solar en­ergy harvesting in wireless sensor network nodes. Circuits and Systems I: Regular Papers, IEEE Transactions on, 55(6): 1742-1750, July 2008. d o i :1 0 .1109/TCSI.2008.922023.

[6] A. Allan. iOS Sensor Apps with Arduino: Wiring the iPhone andiPad into the Internet of Things. O’Reilly Media, 2011. URL: h t tp s :/ /b o o k s.goog le . com/books?id=ej5HrzWgDJgC.

[7] Frank Allgower, Vincent Blondel, and Uwe Helmke. Control theory: ■ Mathematical perspectives on complex networked sys­tems. Oberwolfach Reports, 9(1):661—732, 2012. URL: h t tp s :/ / www. mf o . de/document /1209/0WR_2012_12. p d f.

[8] Marco Amorim and Kurt Veggeberg. On-board data logger for . Hy­drovision Brazil, National Instruments, 2012. [Online: accessed 6-July- 2015]. URL: f tp ://f tp .n i.c o m /p u b /b ra n c h e s /la ta m /b ra z il/2 6 3 _ o n _ p board_datalogger_f or_dynamic_pressure_and_strain_f or_f in a l .p d f .

152

Page 173: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[9] M. Andreasson, H. Sandberg, D.V. Dimarogonas, and K.H. Johansson. Dis­tributed integral action: Stability analysis and frequency control of power systems. In Decision and Control (CDC), 2012 IEEE 51st Annual Confer­ence on, pages 2077-2083, Dec 2012. do i: 1 0 .1109/CDC.2012.6426463.

[10] Jo ao P. Hespanha. Modeling and analysis of networked control systems using stochastic hybrid systems. AnnualReviews in Control, 38(2): 155-170, 2014. URL: h t tp :/ / www.s c ie n c e d ire c t . com /science/article/p ii/S1367578814000352, d o i :h t t p : / /d x .d o i .o rg /10.1016/j . a r c o n tro l .2014.09.001.

[11] Eugene Asarin, Thao Dang, and Oded Maler. The d /d t tool for verifica­tion of hybrid systems. In Proceedings of the 1 4 th International Conference on Computer Aided Verification, CAV ’02, pages 365-370, London, UK, UK, 2002. Springer-Verlag. URL: h ttp :/ /d l.a c m .o rg /c ita t io n .c fm ? id = 647771j757171.

[12] Stanley Bak and Marco Caccamo. Computing reachabil­ity for nonlinear systems with hycreate, 2013. URL: h t tp : / / s t a n l e y b a k .c o m / p r o j e c t s / h y c r e a t e / p o s t e r .p n g .

[13] Gerd Behrmann, Alexandre David, and Kim G. Larsen. A tuto­rial on i u p p a a l . In Marco Bernardo and Flavio Corradini, editors, Formal Methods for the Design of Real-Time Systems: 4th Interna­tional School on Formal Methods for the Design of Computer, Com­munication, and Software Systems, SFM-RT 2004, number 3185 in LNCS, pages 200-236. Springer-Verlag, September 2004. URL: h t tp : / /p e o p le .c s .a a u .d k /~ a d a v id /p u b lic a tio n s /2 1 - tu to r ia l .p d f.

[14] Federico Bergero and Ernesto Kofman. Powerdevs: A tool for hybrid system modeling and real-time simulation. Simulation, 87(1-2): 113—132, January 2011. URL: h ttp ://dx .do i.o rg /10 .1177/0037549710368029 , doi : 10.; 1177/0037549710368029.

[15] Marc R. Bissonnette, Amy Stevenson, and Randy Wallman. Case studies of problems diagnosed using on-line machine monitoring on hydro-generating machines. HydroVision, 2006. URL: h t t p : / / w w w . r e s e a r c h g a t e . n e t / p p u b lic a t io n /2 6 5 2 4 6 2 4 8 _ C a se _ S tu d ie s_ o f _Problems_Diagnosed_^ Using_On-Line_M achine_M onitoring_on_Hydro-Generating_M achines.

[16] A. Blum, V. Cechticky, A. Pasetti, and W. Schaufelberger. A java-based framework for real-time control systems. In EFTA 2003. 2003 IEEE Con­ference on Emerging Technologies and Factory Automation. Proceedings

153

Page 174: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

(Cat. No.03TH8696), volume 2, pages 447-453 vol.2, Sept 2003. d o i: 1 0 .1 109/ETFA.2 0 0 3 .1248733 .

[17] Grady Booch, James Rumbaugh, and Ivar Jacobson. Unified, Model­ing Language User Guide, The (2nd Edition) (Addison-Wesley Object Technology Series). Addison-Wesley Professional, 2005. URL: h t tp : / / d l . acm. o r g / c i t a t i o n . cfm?id=1088874.

[18] Alessandro Borri, Giordano Pola, and Maria Domenica Di Benedetto. Symbolic control design of nonlinear networked control systems. CoRR, ab s /1404.0237, 2014. URL: h ttp :/ /a rx iv .o rg /a b s /l4 0 4 .0 2 3 7 .

[19] Azzedine Boukerche. Handbook of Algorithms for Wireless Network­ing and Mobile Computing (Chapman & Hall/Crc Computer & In­formation Science). Chapman & Hall/CRC Computer and Informa­tion Science Series. Chapman & Hall/CRC, 2005. URL: h t t p s : / / b o o k s . g o o g l e . com/books?id=CR7NBQAAQBAJ.

[20] Athanassios Boulis, Chih-Chieh Han, and Mani B. Srivastava. De­sign and implementation of a framework for efficient and pro­grammable sensor networks. In Proceedings of the 1st Inter­national Conference on Mobile Systems, Applications and Ser­vices, MobiSys ’03, pages 187-200, New York, NY, USA, 2003. ACM. URL: h ttp ://do i.acm .o rg /10 .1145 /l066116 .1066121 ,d o i :10.1145/1066116.1066121.

[21] MichaelS. Branicky. General hybrid dynamical systems: Modeling, anal­ysis, and control. In Rajeev Alur, ThomasA. Henzinger, and EduardoD. Sontag, editors, Hybrid Systems III, volume 1066 of Lecture Notes in Com­puter Science, pages 186-200. Springer Berlin Heidelberg, 1996. URL: h t t p : / / d x . d o i . o r g / 1 0 . 1007/BFb0020945, d o i : 1 0 . 1007/BFb0020945.

[22] M.S. Branicky, V.S. Borkar, and S.K. Mitter. A unified framework for hy­brid control: model and optimal control theory. Automatic Control, IEEE Transactions on, 43(1):31—45, Jan 1998. doi: 10.1109/9.654885.

[23] R. Brooks. A robust layered control system for a mobile robot. IEEE Journal on Robotics and Automation, 2(1):14—23, Mar 1986. do i: 1 0 .1 1 0 9 / JRA.1 9 8 6 .1087032 .

[24] Nirupama Bulusu, Deborah Estrin, Lewis Girod, and John Heidemann. Scalable coordination for wireless sensor networks: self-configuring local­ization systems. In in Proc. 6th International Symposium on Commu­nication Theory and Applications (ISCTA ’ 01),Ambleside, Lake District,

154

Page 175: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

2001. URL: h ttp ://c ite see rx .is t.p su .ed u /v iew d o c /su m m a ry ?d o i= 10.1.1.183.5854.

[25] J. Cardell and M. Ilic. Maintaining stability with distributed generation in a restructured industry. In Power Engineering Society General Meeting, 2004- IEEE, pages 2142-2149 Vol.2, June 2004. d o i : 1 0 .1109/PES. 2004.1373260.

[26] Gabriela Cembrano and Gordon Wells. Neural networks for control. In L. Boullart, A. Krijgsman, and R. Vingerhoeds, editors, Application of Artificial Intelligence in Process Con­trol, pages 388 - 402. Pergamon, Oxford, 1992. URL: h t tp :/ / w w w .sciencedirect. com /science/article/pii/B 9780080420165500267, d o i :h t t p s : / / d o i . o rg /1 0 . 1016/B978-0-08-042016-5.50026-7.

[27] G.R. Clarke, D. Reynders, and E. Wright. Practical Modern SCADA Pro­tocols: DNP3, 60870.5 and Related Systems. Engineering : instrumentation & control. Elsevier, 2004. URL: h ttp s://books.goog le .com /books?id= Y_dTAAAAMAAJ.

[28] M.B.G. Cloosterman, N. van de Wouw, W.P.M.H. Heemels, and H. Ni- jmeijer. Stability of networked control systems with uncertain time-varying delays. ,Automatic Control, IEEE Transactions on, 54(7): 1575-1580, July 2009. d p i :1 0 .1109/TAC.2009.2015543.

[29] R. D’Andrea. Linear matrix inequalities, multidimensional system opti­mization, and control of spatially distributed systems: an example. In Pro­ceedings of the 1999 American Control Conference (Cat. No. 99CH36251), volume 4, pages 2713-2717 vol.4, 1999. d o i : 1 0 .1109/ACC. 1999.786563.

[30] Flavia Coimbra Delicato, Paulo F. Pires, Luci Pirmez, and Luiz Fer­nando Rust da Costa Carmo. A flexible web service based architecture for wireless sensor networks. In Proceedings of the 23rd International Conference on Distributed Computing Systems, ICDCSW ’03, pages 730- , Washington, DC, USA, 2003. IEEE Computer Society. URL: h t tp : / / d l . acm. o r g /c i t a t io n . cfm?id=839280.840681.

[31] Piergiuseppe Di Marco. Protocol design and implementation for wireless sen­sor networks. Technical report, KTH, Automatic Control, 2008. URL: h ttp : / / www.d iv a - p o r ta l . o rg /sm ash /g e t/d iv a2 :572787/FULLTEXT01.p d f.

[32] Lam Dorjee. Analysing Inherent Vibration Problems of a Vertical Hydro­electric Turbine Machine. University of New Brunswick, Department of Me­chanical Engineering, 2003. URL: h t t p s : //b o o k s .g o o g le . com/books?id= ZQncPgAACAAJ.

155

Page 176: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[33] H. Dubois-Ferriere, D. Estrin, and T. Stathopoulos. Efficient and practical query scoping in sensor networks. In Mobile Ad-hoc and Sensor Systems, 2004 IEEE International Conference on, pages 564-566, Oct 2004. do i: 10 .1109/MAHSS.2004.1392209.

[34] G. E. Dullerud and R. D’Andrea. Distributed control of heterogeneous systems. IEEE Transactions on Automatic Control, 49(12):2113—2128, Dec2004. d o i :10.1109 /TAC.2004.838499.

[35] Bruno Dutertre, Steven Cheung, and Joshua Levy. Lightweight key man­agement in wireless sensor networks by leveraging initial trust. Technical report, Technical Report SRI-SDL-04-02, SRI International, 2004. URL: h t t p : / /www.csl. s r i . c o m /u se rs /b ru n o /p u b lis /s ri-sd l-0 4 -0 2 .p d f.

[36] Jack Edmonds. Matroids and the greedy algorithm. Math­ematical Programming, 1(1):127—136, Dec 1971. URL: h t t p s : / / d o i . 6 r g / 1 0 .1 0 0 7 /BF01584082, d o i : 1 0 . 1007/BF01584082.

[37] M. El-hami, P. Glynne-Jones, N.M. White, M. Hill, S. Beeby, E. James, A.D. Brown, and J.N. Ross. Design and fabrication of a new vibration-based electromechanical power generator. Sensors and Ac­tuators A: Physical, 92(l-3):335 - 342, 2001. XIV. URL: h t tp : / / w w w .sciencedirect. com /science/article/p ii/S0924424701005696, d o i :h ttp ://d x .d o i.o rg /1 0 .1 0 1 6 /S 0 9 2 4 -4 2 4 7 (01)00569-6.

[38] Hilding, Elmqvist and Sven Erik Mattsson. Simulator for dynami­cal systems using graphics and equations for modeling. Third Sympo­sium on Computer-Aided Control System Design, 1989. URL: h t tp : / / www.ie e e c s s .o r g /C S M /l ib r a L r y /1 9 8 9 /j a n l9 8 9 /w 5 3 - 5 8 .p d f .

[39] Department of Energy. 2014 water power program peer review, compiled presentations. Technical report, Hydropower Technolo­gies, February 2014. [Online; accessed 7-July-2015]. URL: h t tp : / /e n e rg y .g o v /s i te s /p ro d /f i le s /2 0 1 4 /0 5 /f15/61853.pdf.

[40] Laurent Eschenauer and Virgil D. Gligor. A key-management scheme for distributed sensor networks. In Proceedings of the 9th ACM Conference on Computer and Communications Security, CCS ’02, pages 41-47, New York, NY, USA, 2002. ACM. URL: h t tp : / /d o i . a;cm. o rg /1 0 .1145/586110.586117, doi : 10.1145/586110.586117.

[41] Gonzalo Farias, Anton Cervin, Karl-Erik Arzen, Sebastian Dormido, and Francisco Esquembre. Java simulations of embedded con-

156

Page 177: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

trol systems. Sensors, 10(9):8585-8603, 2010. URL: h t tp :/ / www.mdpi. com/1424-8220/10/9/8585, d o i : 10.3390/sl00908585.

[42] Goran Frehse. Phaver: algorithmic verification of hybrid sys­tems past hytech. STTT, 10(3):263—279, 2008. URL: h t tp s : / / u s e r s . e c e . emu. edu/~krogh/otherN SFpapers/Fre08.pdf.

[43] Goran Frehse, Colas Le Guernic, Alexandre Donze, Scott Cotton, Ra- jarshi Ray, Olivier Lebeltel, Rodolfo Ripado, Antoine Girard, Thao Dang, and Oded Maler. Spaceex: Scalable verification of hybrid systems. In Shaz Qadeer Ganesh Gopalakrishnan, editor, Proc. 23rd International Con­ference on Computer Aided Verification (CAV), LNCS. Springer, 2011. URL: h t t p : / / s p a c e e x . i m a g . f r / s i t e s / d e f a u l t / f i l e s / p a p e r _ 5 5 . p d f .

[44] A.V. Fursikov. Optimal Control of Distributed Systems. Theory and Applications. American Mathematical Soc., 1999. URL: h t tp s : / / b o o k s . g o o g l e . com/books?id=hN7XTRTasZcC.

[45] M.E.M.B. Gaid, A. Cela, and Y. Hamam. Optimal integrated con­trol and scheduling of networked control systems with communication constraints: application to a car suspension system. Control SystemsTechnology, IEEE Transactions on, 14(4):776-787, July 2006. do i:10.1109/TCST.2006.872504.

[46] Huijun Gao, Xiangyu Meng, and Tongwen Chen. Stabilization ofnetworked control systems with a new delay characterization. Auto­matic Control, IEEE Transactions on, 53(9):2142—2148, Oct 2008. doi: 1 0 .1109/TAC.2008.930190.

[47] E. Garcia, P.J. Antsaklis, and L.A. Montestruque. Model-Based Con­trol of Networked Systems. Systems & Control: Foundations & Applications. Springer International Publishing, 2014. URL: h t t p s : / / b o o k s . g o o g l e . com/books?id=JL0_BAAAQBAJ.

[48] Xiaohua Ge, Fuwen Yang, and Qing-Long Han. Distributednetworked control systems: A brief overview. InformationSciences, 380(Supplement C):117 - 131, 2017. URL: h t tp :/ / www.s c ie n c e d ire c t . com /science/artic le/p ii/S0020025515005551 , d o i :h t t p s : / / d o i . o rg /1 0 . 1 0 1 6 /j. i n s .2015.07.047.

A. Girard and G. J. Pappas. Approximate bisimulations for constrained linear systems. In Proceedings of the 44th IEEE Conference on Decision and Control, pages 4700-4705, Dec 2005. do i: 1 0 .1109/CDC.2005.1582904.

157

Page 178: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[50] Hassan Gomaa. Designing concurrent, distributed, and real-time applica­tions with uml. In Proceedings of the 23rd International Conference on Soft­ware Engineering, ICSE ’01, pages 737-738, Washington, DC, USA, 2001. IEEE Computer Society. URL: h t tp :/ /d l.a c m .o rg /c ita t io n .c fm ? id = 381473.381619.

[51] Himani Goyal, M. Hanm, and D. P. Kothari. An artificial intelligence based approach for control of small hydro power plants. In Centre for Energy Studies, Indian Institute of Technology, Hauz Khas, New Delhi-110016,2001. URL: h t t p : / / c i t e s e e r x . i s t . psu.edu/viewdoc/summary?doi= 10.1.1.492.1623.

[52] J. M. Guerrero, M. Chandorkar, T. L. Lee, and P. C. Loh. Advanced control architectures for intelligent microgrids 2014;part i: Decentralized and hier­archical control. IEEE Transactions on Industrial Electronics, 60(4):1254- 1262, April 2013. d o i : 1 0 .1109/TIE.2012.2194969.

[53] C. Guo, Y. Zhang, X. You, X. Chen, and Y. Zhang. Optimal con­trol of continuous annealing process using pso. In 2009 IEEE Interna­tional Conference on Automation and Logistics, pages 602-605, Aug 2009. d o i :1 0 .1109/ICAL.2009.5262851.

[54] R. Hampel, M. Wagenknecht, and N. Chaker. Fuzzy Control: Theory and Practice. Advances in Intelligent and Soft Computing. Physica-Verlag HD, 2013. URL: https://books.google.com/books?id=HW hEAAAAQBAJ.

[55] M.T. Hansen and Edoardo Biagioni. Btp: A block transfer protocol for delay tolerant wireless sensor networks. In Local Computer Networks (LCN), 2010 IEEE 35th Conference on, pages 897-904, Oct 2010. doi: 1 0 .1109/LCN.2010.5735830.

[56] S.S. Haykin. Neural Networks: A Comprehensive Foundation. International edition. Prentice Hall, 1999. URL: h t tp s : / /b o o k s .goog le . com/books?id= bX4pAQAAMAAJ.

[57] W.P.M.H. Heemels, A.R. Teel, N. van de Wouw, and D. Nesic. Networked control systems with communication constraints: Tradeoffs between trans­mission intervals, delays and performance. Automatic Control, IEEE Trans­actions on, 55(8): 1781—1796, Aug 2010. doi : 1 0 .1109/TAC.2010.2042352.

[58] W.B. Heinzelman, A.L. Murphy, H.S. Carvalho, and M.A. Perillo. Middle­ware to support sensor network applications. Network, IEEE, 18(1):6—14, Jan 2004. d o i :1 0 .1109/MNET.2004.1265828.

158

Page 179: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[59] Thomas A. Henzinger, Pei-Hsin Ho, and Howard Wong- toi. Hytech: A model checker for hybrid systems. Soft­ware Tools for Technology Transfer, 1:460-463, 1997. URL:h t t p : / / w w w .c i s .u p e n n .e d u / ~ l e e /9 9 c i s 6 4 2 / p a p e r s /h y t e c h .p s .

[60] J.P. Hespanha, P. Naghshtabrizi, and Yonggang Xu. A survey of recent results in networked control systems. Proceedings of the IEEE, 95(1): 138— 162, Jan 2007. d o i : 10. U09/JPR0C. 2006.887288.

[61] Ivan Howitt, Wayne W. Manges, Phani Teja Kuruganti, Glenn All­good, Jose A. Gutierrez, and James M. Conrad. Wireless industrial sensor networks: Framework for qos assessment and qos manage­ment. {L5M} Transactions, 45(3):347 - 359, 2006. URL: h t tp : / / w w w .s c ie n c e d ir e c t . c o m / s c i e n c e /a r t i c l e / p i i / S 0 0 1 9 0 5 7 S 0 7 6 0 2 1 7 1 , d o i :h t t p :/ / d x . d o i . o r g / 1 0 . 1 0 1 6 /S 0 0 1 9 -0 5 7 8 ( 0 7 ) 6 0 2 1 7 - 1 .

[62] Chang-Chun Hua, Ying Zheng, and Xin-Ping Guan. Model­ing and control for wireless networked control system. Inter­national Journal of Automation and Computing, 8(3):357-363, 2011. URL: h ttp ://d x .d o i.o rg /l0 .1 0 0 7 /s ll6 3 3 -0 1 1 -0 5 9 2 -2 ,d o i :1 0 .1007/sll633-011-0592-2.

[63] Chi-Fu Huang and Yu-Chee Tseng. The coverage problem in a wire­less sensor network. In Proceedings of the 2nd ACM International Conference on Wireless Sensor Networks and Applications, WSNA ’03, pages 115-121, New York, NY, USA, 2003. ACM. URL: h t tp : / / d o i . acm . o rg /1 0 .1145/941350.941367, d o i :10.1145/941350.941367.

[64] L.E. Humbard, E.J. Rogers, and D. A. Gillies. Instrumentation tech­niques for low voltage impulse testing of power transformers. Power Ap­paratus and Systems, IEEE Transactions on, PAS-91(3):1281-1293, 1972. d o i : 1 0 .1109/TPA S.197 2 .2 9 3 4 8 7 .

[65] S. i. Hprikawa, T. Furuhashi, and Y. Uchikawa. On fuzzy model­ing using fuzzy neural networks with the back-propagation algorithm. IEEE Transactions on Neural Networks, 3(5):801—806, Sep 1992. doi: 1 0 .1 1 0 9 /7 2 .1 5 9 0 6 9 .

[66] B.A.T. lamamura, Y. Le Menach, A. Tounzi, N. Sadowski, and E. Guillot. Study of static and dynamic eccentricities of a synchronous generator using 3-d fern. Magnetics, IEEE Transactions on, 46(8):3516—3519, Aug 2010. d o i : 1 0 .1109/TMAG.2010.2043347.

159

Page 180: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[67] Ren Jager. Adaptive fuzzy control. In L . Boullart, A. Krijgsman,and R. Vingerhoeds, editors, Application of Artificial Intelligence inProcess Control, pages 368 - 387. Pergamon, Oxford, 1992. URL: h t tp :/ / www. s c i e n c e d i r e c t . c o m /s c ie n c e /a r t i c l e /p i i /B 9 7 8 0 0 8 0 4 2 0 1 6 5 5 0 0 2 5 5 , d o i :h t t p s :/ / d o i . o r g / 1 0 . 1 0 1 6 /B 9 7 8 - 0 -0 8 - 0 4 2 0 1 6 -5 .5 0 0 2 5 - 5 .

[68] Chris P. Jobling. User interface issues and the role of artificial intelligence in computer-aided control engineering. In L. Boullart, A. Krijgsman,and R. Vingerhoeds, editors, Application of Artificial Intelligence inProcess Control, pages 303 - 338. Pergamon, Oxford, 1992. URL: h t tp :/ / w w w .s c ie n c e d ir e c t . c o m /s c ie n c e /a r t ic le /p i i /B 9 7 8 0 0 8 0 4 2 0 1 6 5 5 0 0 2 2 X , d o i :h t t p s : / / d o i . o r g / 1 0 . 1 0 1 6 /B 9 7 8 - 0 - 0 8 - 0 4 2 0 1 6 - 5 . 50022-X .

[69] Pontus P. Jonsson, Michel J. Cervantes, and Marie Finnstrom. Numer­ical investigation of the gibson’ s method - effects of connecting tubing -. 2nd IAHR International Meeting of the Workgroup on Cavitation and Dynamic Problems in Hydraulic Machinery and Systems, 2007. URL: h t t p s : / / p u r e . l t u . s e / p o r t a l / f i l e s / 1218600/38_J o n sso n C e r F in .p d f .

[70] Pontus P. Jonsson, Jorgen Ramdal, Michel J. Cervantes, Ole G. Dahlhaug, and Torbjorn K. Nielsen. The pressure-time measurements project at ltu and ntnu. Proceedings, IGHEM, 2010 : 8th International Conference on Hydraulic Efficiency Measurement, 2010. URL: h t tp :/ /a h e c .o r g . in / l in k s /IG H E M _ 2 0 1 0 /P a p e r s /T S B -0 1 .p d f .

[71] P.P. Jonsson, J. Ramdal, and M.J. Cervantes. Experimental in­vestigation of the gibson's method outside standards. 2fth Sym­posium on Hydraulic Machinery and Systems, 2008. URL: h t tp :/ / p u r e . l t u . s e / p o r t a l / f i l e s / 2 2 7 4 8 5 1 / I A H R 2 0 5 0 .p d f .

[72] Anestis I. Kalfas, Torsten Franson, Howard P. Hodson, Howard P. Hodson, Tony Arts, Reza S. Abhari, Pietro Zuninio, and Jan Lepicovsky. The 18th symposium on measuring techniques in turbo-machinery. [Online; accessed 6-July-2015]. URL: h t t p : / / t u r b o . w e b . a u t h . g r / .

[73] A. Kansal and M.B. Srivastava. An environmental energy harvesting frame­work for sensor networks. In Low Power Electronics and Design, 2003. ISLPED '03. Proceedings of the 2003 International Symposium on, pages 481-486, Aug 2003. d o i :1 0 .1109/LPE.2003.1231958.

[74] E.D. Kolaczyk and G. Csardi. Statistical Analysis of Network Data with R. Use R! Springer New York, 2014. URL: h t t p s :/ / b o o k s . g o o g l e . com/books?id=cNMhBAAAQBAJ.

160

Page 181: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[75] Eric D. Kolaczyk. Statistical Analysis of Network Da,ta: Methods and Mod­els. Springer Publishing Company, Incorporated, 1st edition, 2009. URL: h t t p : / /w w w .s p r in g e r . com /us /b ook /9780387881454 .

[76] George Kollios, John W Byers, Jeffrey Considine, Marios Hadjieleftheriou, and Feifei Li. Robust aggregation in sensor networks. IEEE Data Eng. Bull, 28(1):26—32, 2005. URL: h ttp ://w w w .c s .a r iz o n a .e d u /c la s s e s /p cs645 /fa ll05 /cs645-papers/D ataA ggregation /dataC .pdf.

[77] B. Korte, L. Lovasz, and R. Schrader. Greedoids. Algorithms and Combinatorics. Springer Berlin Heidelberg, 2012. URL: h t t p s : / / b o o k s . g o o g l e . com/books?id=IeDrCAAAQBAJ.

[78] Bhaskar Krishnamachari. Mathematical modeling and algorithms for wire­less sensor networks, 2005. [Online; accessed 8-July-2015]. URL: h t tp : / / i l a b . u s e . e d u /c la s se s /2 0 0 5 c s5 9 7 f /M o d e lin g A lg o r ith m sF o rW S N . ppt.

[79] B.C. Kuo. Digital Control Systems. Oxford series in electrical and computer engineering. Oxford University Press, 1992. URL: h t tp s : / /books.goog le . com/books?id=WMB4wRjT3oIC.

[80] A. Kurzhanskiy and Pravin Varaiya. Ellipsoidal toolbox. Technical Re­port No, UCB/EECS-2006-46, Electrical Engineering and Computer Sci­ences, University of California at Berkeley, May 2006. URL: h t t p s : / /www2. e e c s . b e r k e l e y . ed u /P u b s/T ech R p ts /2 0 0 6 /E E C S -2 0 0 6 -4 6 . pdf.

[81] C. Langbort and R. D’Andrea. Distributed control of heterogeneous sys­tems interconnected over an arbitrary graph. In f2nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH31fl5), volume 3, pages 2835-2840 Vol.3, Dec 2003. do i: 1 0 .1109/CDC.2003.1273055.

[82] E. Lavretsky. Greedy control revisited. In Proceedings of the 2002 American Control Conference (IEEE Cat. No.CH37301), volume 5, pages 4173-4176 vol.5, May 2002. doi : 1 0 .1109/ACC.2002.1024585.

[83] M. Leon and N. Xiong. Greedy adaptation of control parameters in differential evolution for global optimization problems. In 2015 IEEE Congress on Evolutionary Computation (CEC), pages 385-392, May 2015. d o i :1 0 .1109/CEC.2015.7256916.

[84] Yuan Li, Qingling Zhang, and Chong Jing. Stochastic stability of networked control systems with time-varying sampling periods. J. Inf. Syst. Sci, 5:494- 502, 2009. URL: h t t p : / / w w w .m a t h .u a l b e r t a . c a / i j i s s / S S - V o l u m e - 5 - p 2 0 0 9 /N O -3 - 0 9 /S S -0 9 - 0 3 - 2 3 . p d f .

161

Page 182: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[85] Z. Li, Y. Xia, and C.Y. Su. Intelligent Networked Teleopera­tion Control. Springer Berlin Heidelberg, 2015. URL: h t tp s : / /books.goog le . com/books?id=7WZlCQAAQBAJ.

[86] Feng-Li Lian, J. Moyne, and D. Tilbury. Analysis and modeling of networked control Systems: Mimo case with multiple time delays. In American Control Conference, 2001. Proceedings of the 2001, volume 6, pages 4306-4312 vol.6, June 2001. d o i :1 0 .1109/ACC.2001.945654.

[87] Feng-Li Lian, J. Moyne, and D. Tilbury. Network design consideration for distributed control systems. Control Systems Technology, IEEE Transac­tions on, 10(2):297-307, March 2002. d o i : 10.1109/87.987076.

[88] J. Liu and Baochun Li. Distributed topology control in wireless sensor networks with asymmetric links. In Global Telecommunications Conference, 2003. GLOBECOM ’03. IEEE, volume 3, pages 1257-1262 vol.3, Dec 2003. d o i :1 0 .1109/GL0C0M.2003.1258440.I

[89] Jianming Liu and Tong Lee. A framework for performance modeling of wireless sensor networks. In Communications, 2005. ICC 2005. 2005 IEEE International Conference on, volume 2, pages 1075-1081 Vol. 2, May 2005. d o i :1 0 .1109/ICC.2005.1494513.

[90] Arthur Low. Evolution of wireless sensor networks for industrial con­trol. Technology Innovation Management Review, 3(5), 2013. URL: h t tp : / /www. n i v i s . com/resources/Low_TIMReviewJlay2013. pdf.

[91] A. Ltaief, A. Taieb, and A. Chari. Pid-pso control for takagi-sugeno fuzzy model. In 2013 International Conference on Control, Decision and Information Technologies (CoDIT), pages 086-091, May 2013. do i: 1 0 .1109/CoDIT.2013.6689525.

[92] D.W. Luse. A nyquist-type stability test for multivariable distributed sys­tems. Automatic Control, IEEE Transactions on, 33(6):563—566, Jun 1988. d o i : 1 0 .1 1 0 9 /9 .1 2 5 2 .

[93] John Lygeros, Claire Tomlin, and Shankar Sastry. Hybrid systems: Mod­eling, analysis and control. Technical report, University of California at Berkeley, December 2008. [Online; accessed 7-July-2015]. URL: h t tp : / / i n s t . c s .berkeley.edu/~ee291e/sp09/handouts/book.pdf.

[94] J. Lynch, Y. Wang, R. Swartz, K.-C. Lu, and C.-H. Loh. Implementation of a closed-loop structural control system using wireless sensor networks. Structural Control and Health Monitoring, 15(4):518—539, 2011. [Online;

162

Page 183: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

accessed 7-July-2015]. URL: h t t p : / / e i l . S ta n fo rd .e d u /p u b lic a tio n s /p j e rry _ lynch/C ontrolPaperSingleSpace.p d f.

[95] Moufida Maimour, Congduc Pham, and Julien Amelot. Load repartition for congestion control in multimedia wireless sensor networks with mul­tipath routing. In Wireless Pervasive Computing, 2008. ISW PC 2008. 3rd International Symposium on, pages 11-15. IEEE, 2008. URL: h t tp : / /a rx iv .o rg /p d f /0 8 1 0 .1139.pdf.

[96] S. Misra, S.C. Misra, and I. Woungang. Guide to Wireless Mesh Networks. Computer Communications and Networks. Springer, 2009. URL: h t t p s : / / b o o k s . g o o g l e . com/books?id=k8y54FSMLMEC.

[97] F. Morant, M. Martnez, and J. Pic. Supervised adaptive con­trol. In L. Boullart, A. Krijgsman, and R. Vingerhoeds, ed­itors, Application of Artificial Intelligence in Process Con­trol, pages 427 - 455. Pergamon, Oxford, 1992. URL: h t tp :/ / w w w .sciencedirect. com /science/article/pii/B 9780080420165500292, doi :h t t p s : / /d o i . o rg /1 0 . 1016/B978-0-08-042016-5.50029-2.

[98] M.M.R. Mozumdar, F. Gregoretti, L. Lavagno, L. Vanzago, and S. Olivieri.A framework for modeling, simulation and automatic code generation of sen­sor network application. In Sensor, Mesh and Ad Hoc Communications and Networks, 2008. SECON ’08. 5th Annual IEEE Communications Society Conference on, pages 515-522, June 2008. do i: 1 0 .1109/SAHCN.2008.68.

I[99] Payam Naghshtabrizi and JoaoP. Hespanha. Implementation considera­

tions for wireless networked control systems. In Sudip K. Mazumder, editor, Wireless Networking Based Control, pages 1-27. Springer New York, 2011. URL: h ttp ://d x .d o i.o rg /l0 .1 0 0 7 /9 7 8 - l-4 4 1 9 -7 3 9 3 -l_ l, d o i :10.1007/978-1-4419-7393-1_1.

[100] Valentin Nedelea, Ladislau Augustinov, and Gheorghe Liuba. Online moni­toring of the air gap for large synchronous hydrogenerators with the purpose to prevent some operating troubles. Analele Universitalii “Eftimie Murgu” Resila A'nul XIX, 2012. URL: h ttp ://a n a le - in g .u e m .ro /2 0 1 2 /2 2 .p d f .

[101] M. Negnevitsky. Artificial Intelligence: A Guide to Intelli­gent Systems. Addison Wesley/Pearson, 2011. URL: h t tp s : //books', goog le . com/books?id=toIFQgAACAAJ.

[102] D. Nesic and A.R. Teel. Input-output stability properties of networked control systems. Automatic Control, IEEE Transactions on, 49(10): 1650- 1667, Oct 2004. d o i :1 0 .1109/TAC.2004.835360.

163

Page 184: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[103] H.T. Nguyen, N.R. Prasad, C.L. Walker, and E.A. Walker. A First Course in Fuzzy and Neural Control. CRC Press, 2002. URL: h t tp s : //b o o k s .g o o g le . com/books?id=wvPKBQAAQBAJ.

[104] Ryszard Nowicki and Raegan Macvaugh. Xy measurements for radial position and dynamic motion in hydro turbine generators. ORBIT Vol .30 N o .l, 2010. URL: http://w w w .ge-m cs.com /dow nload/m onitoring/p 0RBIT_v30nl_2010_rl57.2032-39.pdf.

[105] P. Ogren, E. Fiorelli, and N.E. Leonard. Cooperative control of mobile sensor networks:adaptive gradient climbing in a distributed environment. Automatic Control, IEEE Transactions on, 49(8): 1292—1302, Aug 2004. d o i :1 0 .1109/TAC.2004.832203.

[106] K. Olejnlchenko. Algorithms of networked control system design. In Modem Problems of Radio Engineering, Telecommunications and Computer Science (TCSET), 2010 International Conference on, pages 201-201, Feb 2010.

i

[107] R. Olfati-Saber, J.A. Fax, and R.M. Murray. Consensus and cooperation in networked multi-agent systems. Proceedings of the IEEE, 95(1):215—233, Jan 2007. d o i :10 .1109/JPR0C.2006.887293.

[108] Ozren Oreskovic and Earl Goodeve. Identification of geometry and dynamic properties of hydro generators based on signal analy­sis of air gap. Iris Rotating Machine Conference, 2013. URL: h t t p : / /www.irispow er.com /U pload /P apers/lden tifica tion% 20 p of7o20Geometry°/o20and%20Dynamic7o20Properties7.20of7020Hydro7o20p Generators7o20Based7o20on7o20Signal7o20Analysis7»20of7o20Air7o20 p Gap7.20-7.20IRMC7o202013-7,20Goodeve. pdf.

[109] P.G. Otanez, James R. Moyne, and D.M. Tilbury. Using deadbands to reduce communication in networked control systems. In American Control Conference, 2002. Proceedings of the 2002, volume 4, pages 3015-3020 vol.4,2002. d o i :1 0 .1109/ACC.2002.1025251.

[110] M. Pajic, S. Sundaram, G.J. Pappas, and R. Mangharam. The wire­less control network: A new approach for control over networks. Auto­matic Control, IEEE Transactions on, 56( 10):2305—2318, Oct 2011. do i: 1 0 .1109/TAC.2011.2163864.

[111] Shantaram.S. Patil, H.K. Verma, and Arun Kumar. Efficiency measurement of hydro machine by thermodynamic method. Proceedings, IGHEM, 2010 : 8th International Conference on Hydraulic Efficiency Measurement, 2010. URL: http://w ww .ighem .org/Paper2010/TSD 05.pdf.

164

Page 185: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[112] Adrian Perrig, Robert Szewczyk, Justin Douglas Tygar, Victor Wen, and David E Culler. Spins: Security protocols for sensor networks. Wireless net­works, 8(5):521—534, 2002. URL: h t t p : / /w w w .c s e e .u m b c .e d u /c o u r s e s /p graduate/CMSC691A / S p r i n g 0 4 / p a p e r s / s p i n s - w in e - j o u r n a l .p d f .

[113] Roberto Di Pietro, Luigi V Mancini, Alessandro Mei, Alessandro Pan- conesi, and Jaikumar Radhakrishnan. Connectivity properties of se­cure wireless sensor networks. In SA SN ’Of Proceedings of the 2nd ACM workshop on Security of Ad hoc and Sensor Networks, pages 53-58. Association for Computing Machinery, 2004. URL: h t tp : / / r e p o s i t o r y . i a s . a c . i n / 8 9 5 0 3 / l / l 6 - a . p d f .

[114] Andre Platzer and Jan-David Quesel. KeYmaera: A Hy­brid Theorem Prover for Hybrid Systems (System Description), pages 171-178. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. URL: h ttp s://do i.o rg /1 0 .1 0 0 7 /9 7 8 -3 -5 4 0 -7 1 0 7 0 -7 _ 1 5 ,d o i :1 0 .1007/978-3-540-71070-7_15.

[115] Sameera Poduri, Sundeep Pattern, Bhaskar Krishnamachari, and Gau- rav Sukhatme. A unifying framework for tunable topology con­trol in sensor networks. Technical report, University of South­ern California, 2005. [Online; accessed 7-July-2015]. URL: h t tp : / / c r e s . u s e . e d u /R e s e a r c h / f i l e s /N e t G r a p h s .p d f .

[116] Dario Pompili, Tommaso Melodia, and Ian F. Akyildiz. Deploy­ment analysis in underwater acoustic wireless sensor networks. In Proceedings of the 1st ACM International Workshop on Underwater Networks, WUWNet ’06, pages 48-55, New York, NY, USA, 2006. ACM. ’ URL: h ttp ://do i.acm .o rg /l0 .1145 /1161039 .1161050 , doi: 10.1145/1161039.1161050.

[117] Amro Qandour. Application framework for wireless sensor networks. Tech­nical report, Edith Cowan University, 2012. [Online; accessed 7-July-2015]. URL: h t t p : / / r o . e c u . e d u . a u / t h e s e s / 4 7 2 / .

[118] F. Y. Qian, R. K. Mai, Y. S. Zhu, and S. H. Leung. Greedy algo­rithm based power control technique for mimo systems with maximum likelihood receiver. Electronics Letters, 48(15):926-927, July 2012. doi: 1 0 .1 0 4 9 /e l.2012.0543.

[119] J. Qin, Y. Chow, J. Yang, and R. Rajagopal. Distributed online modi­fied greedy algorithm for networked storage operation under uncertainty. IEEE Transactions on Smart Grid, 7(2 ) : 1106—1118, March 2016. doi: 1 0 . 1109/TSG .2015 .2422780 .

165

1

Page 186: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[120] M. Rabbat and R. Nowak. Distributed optimization in sensor net­works. In Information Processing in Sensor Networks, 2004■ IPSN 2004- Third International Symposium on, pages 20-27, April 2004. doi: 10.1109/IPSN.2004.1307319.

[121] Venkatesh Rajendran, Katia Obraczka, and J. J. Garcia-Luna- Aceves. Energy-efficient, collision-free medium access control for wireless sensor networks. Wirel, Netw., 12(1) :63—78, February 2006: URL: h ttp ://d x .d o i.o rg /1 0 .1 0 0 7 /s ll2 7 6 -0 0 6 -6 1 5 1 -z , do i:1 0 .1007/sll276-006-6151-z.

[122] A. Rantzer. Distributed control of positive systems. In 2011 50th IEEE Con­ference on Decision and Control and European Control Conference, pages 6608-6611, Dec 2011. d o i :1 0 .1109/CDC.2011.6161293.

[123] G. Chella Rao, K., and N.. Design and analysis of hydro gen­erator ventilation system using experimental and cfd. World Academy of Science, Engineering and Technology, 54:277, june 2011. URL: h t t p : / / c o n n e c t i o n . e b s c o h o s t . e o m / c / a r t i c l e s / 6 0 8 0 1 4 8 6 / p d e s i g n - a n a l y s i s - h y d r o - g e n e r a t o r - v e n t i l a t i o n - s y s t e m - u s i n g - p e x p e r im e n ta l - c f d .

[124] Stefan Ratschan and Zhikun She. Safety verification of hy­brid systems by constraint propagation-based abstraction re­finement. ACM Trans. Embed. Comput. Syst., 6(1), February 2007. ' URL: h ttp ://do i.acm .org /10 .1145/1210268 .1210276 ,d o i :10.1145/1210268.1210276.

[125] L. Reznik. Fuzzy Controllers Handbook: How to Design Them,How They Work. Elsevier Science, 1997. URL: h t t p s :/ /books.goog le . com/books?id=arO-84SktfQC.

[126] D. Russell. Introduction to Embedded Systems: Using A N SI C and the Arduino Development Environment. Synthesis lectures on digi­tal circuits and systems. Morgan & Claypool, 2010. URL: h t t p s : / / b o o k s :g o o g l e . com/books?id=VZgYNDqWuRYC.

[127] S.J. Russell and P. Norvig. Artificial Intelligence: A Mod­ern Approach. Always learning. Pearson, 2013. URL: h t t p s : / / b o o k s . g o o g l e . com/books?id=DFJtngEACAAJ.

[128] Ricardo Sanfelice, David Copp, and Pablo Nanez. A toolbox for simula­tion of hybrid systems in matlab/simulink: Hybrid equations (hyeq) tool­box. In Proceedings of the 16th International Conference on Hybrid Systems:

166

Page 187: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

Computation and Control, HSCC ’13, pages 101-106, New York, NY, USA,2013. ACM. URL: h ttp ://do i.acm .o rg /l0 .1145 /2461328 .2461346 , doi: 10.1145/2461328.2461346.

[129] Paolo Santi. Topology control in wireless ad hoc and sen­sor networks. ACM Comput. Surv., 37(2): 164-194, June 2005.URL: h ttp ://do i.acm .o rg /l0 .1145 /1089733 .1089736 , doi:10.1145/1089733.1089736.

[130] Chris Savarese, Jan M. Rabaey, and Koen Langendoen. Robust position­ing algorithms for distributed ad-hoc wireless sensor networks. In Pro­ceedings of the General Track of the Annual Conference on USENIX An­nual Technical Conference, ATEC ’02, pages 317-327, Berkeley, CA, USA,2002. USENIX Association. URL: h t tp :/ /d l.a c m .o rg /c ita t io n .c fm ? id = 647057.713854.

[131] A.K. Sawhney and P. Sawhney. A Course in Electrical and Elec­tronic Measurements and Instrumentation. Rai, 1996. URL: h t tp s : / /books.goog le . com/books?id=_bb4HgAACAAJ.

[132] K. Seaman, A. Kalogeratos, and N. Vayatis. A greedy approach for dynamic control of diffusion processes in networks. In 2015 IEEE 27th International Conference on Tools with Artificial Intelligence (ICTAI), pages 652-659, Nov 2015. d o i :1 0 .1109/ICTAI.2015.99.

[133] K. W. Schmidt and E. G. Schmidt. Distributed real-time protocols for industrial control systems: Framework and examples. IEEE Transactions on Parallel and Distributed Systems, 23(10): 1856—1866, Oct 2012. do i: 1 0 .1109/TPDS.2011.300.

[134] C. Secchi, C. Fantuzzi, and M. Bonfe. On the use of uml for mod­eling physical systems. In Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pages 3990-3995, April 2005. d o i :1 0 .1109/R0B0T.2005.1570731.

[135] S. Sengupta, M. Chatterjee, and K.A. Kwiat. A game theoretic framework for power control in wireless sensor networks. Computers, IEEE Transac­tions on, 59(2):231—242, Feb 2010. d o i : 1 0 .1109/TC.2009.82.

[136] T. Senjyu, Y. Miyazato, A. Yona, N. Urasaki, and T. Funabashi. Optimal distribution voltage control and coordination with distributed generation. IEEE Transactions on Power Delivery, 23(2): 1236—1242, April 2008. doi: 1 0 .1109/TPWRD.2007.908816.

167

Page 188: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[137] C. M. J. A. Serodio, P. M. M. A. Silva, C. A. C. Couto, and J. L. Monteiro; Embedded java in agricultural control systems. In Indus­trial Electronics Society, 1999. IECON ’99 Proceedings. The 25th An­nual Conference of the IEEE, volume 2, pages 716-721 vol.2, 1999. d o i : 1 0 .1109/IEC0N.1999.816489.

[138] Iman Shames, Andre M.H. Teixeira, Henrik Sandberg, and Karl H. Johansson. Distributed fault detection for interconnected second- order systems. Automatica, 47(12):2757 - 2764, 2011. URL: h t tp : / / w w w .sciencedirect. com /science/article/p ii/S0005109811004511, d o i :h t t p : / /d x .d o i .o rg /1 0 .1016/j . au to m atica .2011.09 .O il.

[139] C. Sharp, S. Schaffert, A. Woo, N. Sastry, S. Sastry, and D. Culler. De­sign and implementation of a sensor network system for vehicle tracking and autonomous interception. In Wireless Sensor Networks, 2005. Pro- ceeedings of the Second European Workshop on, pages 93-107, Jan 2005. d o i :10.1109/EWSN.2005.1462002.

[140] Yang Shi and Bo Yu. Output feedback stabilization of networked con­trol systems with random delays modeled by markov chains. Auto­matic Control, IEEE Transactions on, 54(7):1668-1674, July 2009. do i: 10 .1109/TAC.2009.2020638.

[141] Rajeev Shorey and Chan Mun Choon. Mobile, Wireless and Sensor Net­works: Technology, Applications and Future Directions. Wiley-Interscience, 2005. URL: h t t p : / / c i t e s e e r x . ist.psu.edu/view doc/dow nload?doi= 1 0 .1 .1 .117.420&rep=repl&type=pdf.

[142] U. Sill and W. Zorner. Steam Turbine Generators Process Control and Diag­nostics: Modern Instrumentation for the Greatest Economy of Power Plants. Wiley, 1996. URL: http://books.google.com/books?id=hMIeAQAAIAAJ.

[143] B. Izaias Silva, Keith Richeson, Bruce Krogh, and Alongkrit Chutinan. Mod­eling and verifying hybrid dynamic systems using checkmate. In In AD PM, 2000. URL: h t t p : //c ite see rx .is t.p su .ed u /v iew d o c /d o w n lo ad ?d o i= 1 0 .1 .1 .4 7 4 .2084&rep=repl&type=pdf.

[144] D. Silva, F. Carazas, and G. Souza. Method to select instrumentation for hydraulic turbines in retrofitting process. In Computers Industrial Engineer­ing, 2009. CIE 2009. International Conference on, pages 1192-1197, 2009. d o i :1 0 .1109/ICCIE.2009.5223837.

[145] S.N. Simic. A learning-theory approach to sensor networks. Pervasive Com­puting, IEEE, 2(4):44-49, Oct 2003. doi: 1 0 .1109/MPRV.2003.1251168.

168

Page 189: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[146] A. K. Singh, R. Singh, and B. C. Pal. Stability analysis of networked control in smart grids. IEEE Transactions on Smart Grid, 6(1):381—390, Jan 2015. d o i :1 0 .1109/TSG.2014.2314494.

[147] Vipul Singhvi, Andreas Krause, Carlos Guestrin, James H. Garrett, Jr., and H. Scott Matthews. Intelligent light control using sensor networks. In Proceedings of the 3rd International Conference on Embedded Net­worked Sensor Systems, SenSys ’05, pages 218-229, New York, NY, USA,2005. ACM. URL: h ttp ://do i.acm .o rg /l0 .1145 /1098918 .1098942 , doi: 10.1145/1098918.1098942.

[148] B. Sinopoli, C. Sharp, L. Schenato, S. Schaffert, and S.S. Sastry. Dis­tributed, control applications within sensor networks. Proceedings of the IEEE , 91(8):1235—1246, Aug 2003. d o i : 10.1109/JPR0C.2003.814926.

[149] A. Sridharan, C. Joo, and C. E. Koksal. Energy efficient greedy link schedul­ing and power control in wireless networks. In 2012 IEEE International Symposium on Information Theory Proceedings, pages 811-815, July 2012. d o i :1 0 .1109/ISIT.2012.6284672.

[150] J.A. Stankovic, T.F. Abdelzaher, Chenyang Lu, Lui Sha, and J.C. Hou. Real-time communication and coordination in embedded sensor networks. Proceedings of the IEEE, 91 (7): 1002—1022, July 2003. do i: 10.1109/JPR0C.2003.814620.

[151] Flavius Dan Surianu and Constantin Barbulescu. Using hydro mathe­matical model in simulating dynamic behaviour of hydromechanical equip­ment of hydro power plant raul mare-retezat, romania. In Proceed­ings of the 8th Conference on Simulation, Modelling and Optimization, SMO’08, pages 27-32, Stevens Point, Wisconsin, USA, 2008. World Sci­entific and Engineering Academy and Society (WSEAS). URL: h t tp : / / d l . acin. o r g /c i ta t io n . cfm?id=1503955.1503964.

[152] A. H. Tahoun and Fang Hua-Jing. Adaptive stabilization of networked control systems. Journal of applied sciences, Nov 2006. URL: h t tp : / /d o c s d r iv e . co m /p d fs /an sin e t/jas /2 0 0 7 /3 5 4 7 -3551.pdf.

[153] P. Talas and P. Toom. Dynamic measurement and analysis of air gap variations in large hydroelectric generators. Power Apparatus and Systems, IEEE Transactions on, PAS-102(9):3098-3106, 1983. do i: 10.1109/TPAS.1983.318116.

169

Page 190: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[154] S. Tatikonda and S. Mitter. Control under communication constraints. Automatic Control, IEEE Transactions on, 49(7): 1056-1068, July 2004. d o i :1 0 .1109/TAC.2004.831187.

[155] Y.C. Tay. Mathematics for networking. [Online; accessed 8-July-2015]. URL: h t t p : / / www. m ath. n u s . edu . sg/~m attyc/N etM aths. p p t.

[156] A. Teixeira, H. Sandberg, and K.H. Johansson. Networked control sys­tems under cyber attacks with applications to power networks. In Amer­ican Control Conference (ACC), 2010, pages 3690-3696, June 2010. do i: 1 0 .1109/ACC.2010.5530638.

[157] Andre Teixeira. Toward secure and reliable networked control sys­tems. Technical Report 2011:075, KTH, Automatic Control, 2011. QC 20111124. URL: h t tp : / /k th .d iv a -p o r ta l .o rg /s m a s h /g e t /d iv a 2 : 457661/FULLTEXT01.pdf.

[158] K. C. Thramboulidis. Using uml in control and automation: a model driven approach. In Industrial Informatics, 2004. INDIN ’Of- 2004 2nd IEEE International Conference on, pages 587-593, June 2004. doi: 1 0 .1109/INDIN.2004.1417414.

[159] Bin TIAN, Yi xian YANG, Dong LI, Qi LI, and Yang XIN. A security framework for wireless sensor networks. TheJournal of China Universities of Posts and Telecommunica­tions, 17, Supplement 2(0):118 - 122, 2010. URL: h t tp :/ / w w w .sciencedirect. com /science/article/p ii/S1005888509605719, d o i :h t t p : / / d x .d o i . o rg /1 0 .1016/S1005-8885(09)60571-9.

[160] TIOBE. Software index, 2017. [On­line; , accessed 7-October-2017]. URL: h t tp :/ / www. t i obe. com /index.php/ cont e n t/ p ap erin f o /tp c i / in d e x .htm l.

[161] H.S. Tzou and L.A. Bergman. Dynamics and Control of Dis­tributed Systems. Cambridge University Press, 1998. URL: h t tp s : //b o o k s :goog le . com/books?id=3Q3LygAACAAJ.

[162] Volkan Ungan. Networked pid controllers for wireless sys­tems. Technical report, KTH, Reglerteknik, Nov 2009.URL: h t tp : / /k th .d iv a -p o r ta l .o rg /s m a s h /re c o rd .js f?p id =diva2%3A570067&dswid=5334.

[163] Gustavo Urquiza, Miguel A. Basurto, Laura Castro, Adam Adamkowski, and Waldemar Janicki. Flow measurement methods applied to hydro power

170

Page 191: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

plants. Flow Measurement, 2012. URL: h t t p : / /w w w .in t e c h o p e n .c o m /p b o o k s / f lo w -m e a s u r e m e n t / f lo w -m e a s u r e m e n t -m e th o d s -a p p l ie d - to -p h y d r o -p o w e r -p la n ts , d o i :10.5772/36508.

[164] Mishiga Vallabhan, Seshadhri Seshadhri, S. Ashok, S. Ramaswmay, and R. Ayyagari. An analytical framework for analysis and design of net­worked control systems with random delays and packet losses. CoRR, abs/1506.06261, 2015. URL: h t tp :/ /a rx iv .o rg /a b s /l5 0 6 .0 6 2 6 1 .

[165] J. C. Vasquez, J. M. Guerrero, J. Miret, M. Castilla, and L. Garcia de Vi­cuna. Hierarchical control of intelligent microgrids. IEEE Industrial Elec­tronics Magazine, 4(4):23-29, Dec 2010. doi : 1 0 .1109/MIE.2010.938720.

[166] H.K. Verma and Arun Kumar. Instrument networking for efficiency mea­surement in small hydro power stations, the International Group for Hy­draulic Efficiency Measurement, 2004. [Online; accessed 5-July-2015]. URL: h t t p : / /www .ighem .org/Paper2004/lV erm aK um ar.pdf.

[167] Viet-Hung Vu, Marc Thomas, and Francois Lafleur. Operational modal monitoring of high power hydro-electrical generators. 4th In­ternational Operational Modal Analysis Conference, 2011. URL: h t tp : / / j e a n . f a b r i . p e r s o . s f r . f r / s f m / p a p i e r s _ s u r v e i l la n c e / 5 _ V u .p d f .

[168] G.C. Walsh, Hong Ye, and L.G. Bushnell. Stability analysis of net­worked control systems. Control Systems Technology, IEEE Transactions on, 10(3):438—446, May 2002. do i: 10.1109/87.998034.

[169] Y. Wan, .J. Cao, G. Chen, and W. Huang. Distributed observer-based cyber­security control of complex dynamical networks. IEEE Transactions on Circuits and Systems I: Regular Papers, 64(11):2966—2975, Nov 2017. do i: 1 0 .1109/TCSI.2017.2708113.

[170] Dan Wang, Yan Long, and F. Ergun. A layered architecture for de­lay sensitive sensor networks. In Sensor and Ad Hoc Communications and Networks, 2005. IEEE SECON 2005. 2005 Second Annual IEEE Communications Society Conference on, pages 24-34, Sept 2005. d o i : 1 0 .1109/SAHCN.2005.1556861.

[171] R. Wang, X. Dong, Q. Li, and Z. Ren. Distributed time-varying formation control for linear swarm systems with switching topologies using an adaptive output-feedback approach. IEEE Transactions on Systems, Man, and Cy­bernetics: Systems, PP(99):1-12, 2017. doi : 1 0 .1109/TSMC. 2017.2765203.

171

Page 192: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[172] Weichao Wang and Bharat Bhargava. Visualization of worm­holes in sensor networks. In Proceedings of the 3rd ACM Work­shop on , WiSe ’04, pages 51-60, New York, NY, USA, 2004. ACM. URL: h ttp ://do i.acm .o rg /10 .1145 /l023646 .1023657 , doi: 10.1145/1023646.1023657.

[173] Xiaofeng Wang and M.D. Lemmon. Event-triggered broadcasting across distributed networked control systems. In American Control Conference, 2008, pages 3139-3144, June 2008. d o i :1 0 .1109/ACC.2008.4586975.

[174] Xiaofeng Wang and M.D. Lemmon. Event-triggering in distributed net­worked control systems. Automatic Control, IEEE Transactions on, 56(3):586-601, March 2011. d o i : 1 0 .1109/TAC.2010.2057951.

[175] Zhuwei Wang and Xiaodong Wang. Optimal distributed control for net­worked dontrol systems with delays. CoRR, abs/1312.3543, 2013. URL: h ttp :/ /a rx iv .o rg /a b s /l3 1 2 .3 5 4 3 .

[176] M.R. Wilkinson, F. Spinato, and P.J. Tavner. Condition monitoring of generators & other subassemblies in wind turbine drive trains. In Diagnostics for Electric Machines, Power Electronics and Drives, 2007. SDEMPED 2007. IEEE International Symposium on, pages 388-392, 2007. d o i :1 0 .1 109/DEMPED.2007.4393125.

[177] Cheng Wu, Vijay Kumar, Jiunjie Liau, Francis L'Esperance, and Gary Baker. , Efunda online manuals on flowmeters. [Online; accessed 6- July-2015]. URL: h ttp ://w w w .efunda.com /designstaj1d a rd s /s e n so rs /j5 flow m eters/flow m eter_ in tro . cfm.

[178] Jing Wu and Tongwen Chen. Design of networked control systems with packet dropouts. Automatic Control, IEEE Transactions on, 52(7): 1314— 1319, July 2007. d o i :1 0 .1109/TAC.2007.900839.

[179] Qinglong Wu, Fei-Yue Wang, and Y. Lin. A mobile-agent based distributed intelligent control system architecture for home automation. In Systems, Man, and Cybernetics, 2001 IEEE International Conference on, volume 3, pages 1599-1605 vol.3, 2001. do i: 1 0 .1109/ICSMC.2001.973513.

[180] Y. Xia, M. Fu, and G.P. Liu. Analysis and Synthesis of Networked Con­trol Systems. Lecture Notes in Control and Information Sciences. Springer Berlin Heidelberg, 2011. URL: h t t p s : //b o o k s .g o o g le . com/books?id= 3CUVe9LkGPQC.

172

Page 193: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[181] Yonggang Xu and. J.P. Hespanha. Optimal communication logics in net­worked control systems. In Decision and Control, 2004■ CDC. 43rd IEEE Conference on, volume 4, pages 3527-3532 Vol.4, Dec 2004. doi: 10 .1109/CDC.2004.1429259.

[182] M. Yazdanian and A. Mehrizi-Sani. Distributed control techniques in mi­crogrids. IEEE Transactions on Smart Grid, 5(6):2901—2909, Nov 2014. d o i :1 0 .1109/TSG.2014.2337838.

[183] Gary V Yee, Brian Shucker, Joe Dunn, Anmol Sheth, and Richard Han. Just-in-time sensor networks. In Proc. 3rd Workshop EmNets. Citeseer,2006. URL: h t t p : / / c i t e s e e r x . i s t . psu.edu/viewdoc/download?doi= 1 0 .1 .1 .114.8133&rep=repl&type=pdf.

[184] Jennifer'Yick, Biswanath Mukherjee, and Dipak Ghosal. Wireless sensor network survey. Computer Networks, 52(12):2292 - 2330, 2008. URL: h t tp : / / w w w .sciencedirect. com /science/article/p ii/S1389128608001254, d o i :h t tp : / /d x .d o i .o r g /1 0 .1016/j.co m n e t.2008.04.002.

[185] Suyoung Yoon, R. Dutta, and M.L. Sichitiu. Power aware routing algorithms for wireless sensor networks. In Wireless and Mobile Communications, 2007. ICWMC ’07. Third International Conference on, pages 15-15, March 2007. d o i :1 0 .1 109/ICWMC.2007.69.

[186] K. You, N. Xiao, and L. Xie. Analysis and Design of Networked Control Systems. Communications and Control Engineering. Springer London, 2015. URL: h t t p s : / /b o o k s.goog le . com/books?id=tFsMBgAAQBAJ.

[187] Chien-Cheng Yu and Bin-Da Liu. A backpropagation algorithm with adap­tive learning rate and momentum coefficient. In Neural Networks, 2002. IJCNN ’02. Proceedings of the 2002 International Joint Conference on, vol­ume 2, pages 1218-1223, 2002. d o i :10.1109/1JCNN.2002.1007668.

[188] Mei Yu, Long Wang, Tianguang Chu, and Guangming Xie. Stabilization of networked control systems with data packet dropout and network delays via switching system approach. In Decision and Control, 2004■ CDC. 43rd IEEE Conference on, volume 4, pages 3539-3544 Vol.4, Dec 2004. do i: 1 0 .1109/CDC.2004.1429261.

[189] Dong Yue, Qing-Long Han, and Chen Peng. State feedback controller design of networked control systems. Circuits and Systems II: Ex­press Briefs, IEEE Transactions on, 51(11):640—644, Nov 2004. doi: 1 0 .1109/TCSII.2004.836043.

173

Page 194: SECURE FRAMEWORK FOR IMPLEMENTING DISTRIBUTED …

[190] Liying Zhang, Lun Xie, Weize Li, and Zhiliang Wang. Security solutions for networked control systems based on des algorithm and improved grey pre­diction model. I.J. Computer Network and Information Security,, 1:78-85,2014. [Online; accessed 9-July-2015], d o i : 1 0 .5 8 1 5 /ijc n is .2 0 1 4 .0 1 :10.

[191] Ying Zheng, Huajing Fang, and H.O. Wang. Takagi-sugeno fuzzy-model- based fault detection for networked control systems with markov delays. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE Transactions on, 36(4):924—929, Aug 2006. d o i : 1 0 . 1109/TSMCB.2 0 05 .861879 .

[192] Jaroslava Zilkova, Jaroslav Timko, and Peter Girovsky. Nonlinear system control using neural networks. Acta Polytechnica Hungarica, Aug 2006. URL: h t t p : / /u n i-o b u d a .h u /jo u r n a l/Z i lk o v a _ T im k o _ G iro v sk y _ 8 .p d f .

[193] Y.H. Zweiri, J.F. Whidborne, and L.D. Seneviratne. Athree-term backpropagation algorithm. Neurocomput­ing, 50(Supplement C):305 - 318, 2003. URL: h t tp :/ / w w w .scienced irect.com /science/article/p ii/S0925231202005696, d o i :h t t p s : //doi.org/10 .1016/S0925-2312(02)00569-6 .

174