Technology Management 873 Individual Assignment 2015

Compton Saunders compton.saunders@gmail.com

Name: Compton Saunders

Student Number: 13718436 Degree: MEng- Engineering Management

Lecturers: Prof Tinus Pretorius Due Date: February 20th, 2015

ii

Table of Contents

List of Abbreviations and Acronyms ......................................................................................... iii List of Figures ............................................................................................................................ iii List of Tables ............................................................................................................................. iv

List of Equations ........................................................................................................................ iv

Individual Assignment: The Application of Technology Readiness Levels and Integration Readiness Levels in order to assess a Maximum Demand Control system utilising ZigBee Wireless Mesh Technology ........................................................................................................ 1

1. Introduction ........................................................................................................................... 1

2. Technology Readiness Assessment (TRA) .............................................................................. 2

3. Technology Readiness Assessment (TRA) Submission Document ......................................... 4

3.1 Purpose of This Document ............................................................................................... 4

3.2 Programme Objective ...................................................................................................... 5

3.3 Programme Description ................................................................................................... 6

3.4 System Description ........................................................................................................... 7

3.5 Critical Technology Elements (CTEs) .............................................................................. 11

3.6 Review of TRL Findings ................................................................................................... 14

4. Review of Demand Control System Using Qualitative Maturity Multi Metric Technique .. 17

4.1 SRL Calculation for the Demand Control System ........................................................... 21

5. Conclusion ............................................................................................................................ 23

Appendix A ............................................................................................................................... 26

Appendix B ............................................................................................................................... 30

Bibliography ............................................................................................................................. 31

iii

List of Abbreviations and Acronyms

AD Advancement Degree of Difficulty ADMD After Diversity Maximum Demand CTEs Critical Technology Elements DOD Department of Defense EU European Union GAO General Accountability Office IRL Integration Readiness Level IRT Independent Review Team NASA National Aeronautics and Space Administration NATO North Atlantic Treaty Organization OECD Organisation for Economic Co-operation and Development PC Personal Computer PLC Programmable Logic Controller SCADA Supervisory Control and Data Acquisition SRL System Readiness Level TRA Technology Readiness Assessment TRL Technology Readiness Level TCP/IP Transmission Control Protocol and Internet Protocol VA Volt-Ampere

List of Figures

Figure 1: Technology Readiness Levels. Source: (DoD 2011) ................................................... 3

Figure 2: Requirements of a Technology Readiness Assessment Document. Source (DoD 2011).................................................................................................................................................... 4

Figure 3: Basis of Technology Maturity Assessments throughout Acquisition. Source (DoD 2009) .......................................................................................................................................... 5

Figure 4: Arial view of property where ZigBee Wireless Mesh Technology load switches are installed. Large 200m radius area. ............................................................................................. 8

Figure 5: Load management load shape objectives. Source (Malik and AL Mata’ni 2007) ...... 9

Figure 6: Demand of 175 KVA was reached prior to MDC install – monthly cost R15 118 ..... 10

Figure 7: Demand of 135 KVA was reached after MDC install – monthly cost R11 626 ......... 10

Figure 8: Online Web portal to via historical and quasi real-time data .................................. 10

Figure 9: SCADA interface which allowing users direct monitoring and control capability .... 11

Figure 10: Real World Ecosystem: Smart Metering with energy efficient heat pumps and Zigbee based maximum demand control ................................................................................ 11

Figure 11: Technology Assessment process proposed by (Bilbro 2007) ................................. 18

Figure 12: Relationship between TRL, IRL and SRL .................................................................. 18

Figure 13: System Readiness Level Calculation. Source (SIT 2010) ........................................ 22

Figure 14: Different Engineering Lifecycles and how the System Readiness Level (SRL) is mapped. Source (Sauser and Ramirez-Marquez 2007) ........................................................... 23

Figure 15: Descriptive Requirements for Technology Readiness Assessment Document. Source (DoD 2009) ............................................................................................................................... 30

iv

List of Tables

Table 1: Technology Readiness Levels for Demand Control System ........................................ 14

Table 2: Summary of CTE Technology Readiness Level ............................................................ 14

Table 3: Techniques for Assessing Qualitative Maturity. Source (Azizian et al. 2009) ............ 17

Table 4: Integration Readiness Levels. Source (Sauser and Ramirez-Marquez 2007)............. 19

Table 5: Hardware Technology Readiness Definitions, Descriptions, and Supporting Information. Source (DoD 2009) .............................................................................................. 26

Table 6: Hardware Technology Readiness Definitions, Descriptions, and Supporting Information. Source (DoD 2009) .............................................................................................. 27

Table 7: Additional Definitions of TRL Descriptive Terms. Source (DoD 2009) ....................... 29

List of Equations Equation 1 ................................................................................................................................ 19

Equation 2 ................................................................................................................................ 20

Equation 3 ................................................................................................................................ 20

Equation 4 ................................................................................................................................ 20

Equation 5 ................................................................................................................................ 21

Individual Assignment: The Application of Technology

Readiness Levels and Integration Readiness Levels in

order to assess a ZigBee Wireless Mesh Technology

based Demand Control system.

1. Introduction Globally, engineers are faced with the development of technology and the integration of

these technologies within larger systems. System integration is defined by Buede (2000) as

“the process of assembling the system from its components, which must be assembled from

their configuration items.” The impression one gets from this definition is that putting

together a system is a relative simplistic task, however Buede (2000) elaborates to say that

the process of integration is very complex and contains numerous tasks that overlap and are

iterative in order to create a system which meets the original requirements as well as

successfully operate in the intended environment. Many challenges exist within the

integration process. Some of these challenges are technology specific while others are related

to the integration and emergence observed due to integration; and dependant on how

mature the technology or the integration of technology is.

In order asses the maturity of technology and its integration, numerous metrics have been

developed in order to assist decision making. Some of these assessment metrics include the

Technology Readiness Level (TRL) (DoD 2009); Integration Readiness Level (IRL) and the

System Readiness Level (SRL) (Sauser, Gove, Forbes and Ramirez-Marquez 2010, Sauser and

Ramirez-Marquez 2007). These metrics attempt to provide a consistent manner in which

different technologies can be compared in terms of its maturity.

This paper will use some of these technology maturity assessment techniques in order to gain

an understanding of how they are applied and used in practise. A case study will then discuss

2

an energy demand control system which was developed and commercially deployed in South

Africa, but experienced issues due to technology and integration challenges.

2. Technology Readiness Assessment (TRA)

During the 1970s the National Aeronautics and Space Administration (NASA) developed the

Technology Readiness Assessment (TRA) as a tool to manage risk within its research and

technology development programmes (Mankins 2009). The purpose of the TRA is to gain an

understanding of the TRL of all the various technologies being used within a greater system.

The TRL is essentially a metric to evaluate the risk related to technology development. The

first comprehensive definitions of each of TRLs were released in 1995 and have since been

adopted by the U.S. Congress’ General Accountability Office (GAO); U.S. Department of

Defense (DOD); Organisation for Economic Co-operation and Development (OECD); European

Union (EU); North Atlantic Treaty Organization (NATO) and other countries such as Australia,

Canada and the United Kingdom (Mankins 2009, Bolat 2014).

According to the U.S. DOD TRA Deskbook (DoD 2009:6), the definition of a TRA is:

“A TRA is a formal, systematic, metrics-based process and accompanying report that assesses

the maturity of technologies called Critical Technology Elements (CTEs) to be used in systems.

CTEs can be hardware or software.”

The definition of a CTE is: “A technology element is “critical” if the system being acquired

depends on this technology element to meet operational requirements (within acceptable cost

and schedule limits) and if the technology element or its application is either new or novel or

in an area that poses major techno- logical risk during detailed design or demonstration.”

(DoD 2009:6)

A technology can be classified as a CTE when it poses a significant risk and in such a case the

TRA should include technical information that can be used to reduce risk. TRLs are used as a

metric by an Independent Review Team (IRT) consisting of subject matter experts (SMEs)(DoD

2009).

3

The TRL scale ranges from one through nine, as seen in Figure 1, where (Technology Readiness

Level) TRL1 would consider early stages of scientific investigation and TRL9 will indicate that

a technology was successfully used within a system.

Figure 1: Technology Readiness Levels. Source: (DoD 2011)

In order to determine the maturity of a particular technology, the programme ideas,

technology requirements as well as the proven technology capabilities are evaluated by a

TRA. Typically a CTE will be assigned a readiness level based on the TRA. TRLs are indicative

of a reached level of maturity at the time that the CTE was measured and does not indicate

how valid a design is and also does not provide an indication of the challenges involved with

progressing to the next level. When CTEs are identified they should be assessed from a

systems engineering perspective and the assumption should be made that the relevant CTE

is capable of performing its required function. CTEs needs to be evaluated while considering

how it will be integrated into a system as the CTE may appear as mature on its own but could

prove to be immature due to other system effects. CTEs can also be classified as hardware or

software and depending on the classification it will have different evaluation criteria which

can be seen in Appendix A (DoD 2011).

4

3. Technology Readiness Assessment (TRA) Submission

Document

The U.S. DOD TRA Deskbook (DoD 2009) provides an outline for a TRA Submission as seen in

Figure 2.

Figure 2: Requirements of a Technology Readiness Assessment Document. Source (DoD 2011)

This study will only consider key areas of the document requirements in order to introduce

and assess the case study from a TSA and TRL perspective. A more detailed requirement of

the TRA document can be found in Appendix B. The proceeding sections will attempt to

develop the TRA document based on the demand control system case study. The following

sections will follow the guidelines provided in Figure 2 and an attempt to compile a TRA

document.

3.1 Purpose of This Document

This document is representative of a TRA, performed independently, for the demand control

programme in support of the Milestone B decision. The TRA was performed at the request of

Company X Technology Director.

There are three major Milestones which indicate a stage within the acquisition cycle which

are Milestone A, B and C. The reason Milestone B was selected is due to the programme

already being in the Engineering and Manufacturing Development phase of the Acquisition

System and the need to identify which technologies are not mature and would result in

additional costs and delays within development schedules (DoD 2009).

5

Figure 3: Basis of Technology Maturity Assessments throughout Acquisition. Source (DoD 2009)

3.2 Programme Objective

The demand pattern at which electricity is utilised by a system can be managed by suing

demand control or load management systems. Load management is regarded as being

integrated with demand side management and involves the decrease and alteration of the

demand required by the system over time with the goal of improving the balance observed

between the energy requirements of the customer or consumer and the current generation

capacity of the supplier, future generation capacity, future generation capacity, transmission

and distribution resources (Malik and AL Mata’ni 2007).

There are a number of ways which user energy demand can be managed by altering

consumption patterns. However, long term sustainability of demand management system

largely depends on the behavioural response of users and the how the design of a load

management system influences users and, in addition, the success or failure of such a system

is often determined by the attitude of the user (ABU-Zeid and AL-Shakarchi 2002). ABU-Zeid

and AL-Shakarchi (2002) also state that the load management goal is to flatten the load curve

by influencing the behavioural consumption of energy.

The purpose of the programme is to develop a modular control system capable of dynamically

switching non-essential electrical loads such as geysers, boilers, air conditioners, heat pumps,

chillers, freezers and lights in order to manage the electrical load required by a system.

6

Commercial consumers are normally billed for their energy consumption based on tariffs

which have components such as levies and consumed energy in kilowatt-hours (kWh). Tariffs

can also include a charge for demand or notified demand1 which is dependent on the

maximum demand.

Once the load can be managed, it will become possible to manipulate consumption or

demand profiles which can reduce the costs of electricity. The cost saving can be realised by

reducing notified demand, shifting consumption to off peak periods, reducing overall energy

consumption and provide flexibility to adjust billing tariff to exploit adjustable demand

profile.

3.3 Programme Description

The demand control system is an incremental improvement on existing systems with similar

capability. The proposed programme will however not build on any existing system but

develop a new system with new technology in order to achieve demand control objectives.

Although the basic methods for demand control exist, the proposed programme will develop

its own demand control philosophy and methodology and algorithms on a processing

platform, the Remote Data Acquisition and Control (RDAc) platform, which has never been

used in such a deployment.

In addition to the processing platform, ZigBee Wireless Mesh Technology (ZigBee) will be used

in order to provide a two way communication highway to send and receive data across the

system. Historically, large scale demand control systems used radio frequency ripple control

systems which only had one way communication and the load switches used to control

devices could only receive commands but not provide any feedback regarding their status.

The utilisation of ZigBee Technology stems from requirements set out by Eskom which

dictates that demand-side management (EEDSM) projects use two way communications

systems within its system architecture in order to send and receive data. The ripple control

systems thus had to rely on statistical methods using after diversity maximum demand

1 Maximum demand notified in writing by the customer and accepted by the utility (Eskom).

7

(ADMD)2 in order to make decisions regarding the shedding of load clusters without having

real-time decision making data. The use of ZigBee will enable load switching devices to

communicate measured values and status in real-time, drastically improving demand

management capabilities.

3.4 System Description

The goal of a load management or demand control system is to manage the power

requirements of an energy consuming system. Power (watts) is measured instantaneously

and is the rate at which work is performed while energy (kwh) is the integral of power over

time. As an example, If a 100 watt light bulb requires 100 watts of power and is switched on

for one hour, that light bulb will use 100 watt-hours of energy. The maximum demand of the

light bulb will be 100 watt. Maximum demand can be seen as the maximum instantaneous

power required from the main power grid by a system over a specific timeframe. Demand is

measured in volt-ampere (VA).

The assessed system uses ZigBee load switching and measurement devices that have the

capability of real-time measurement of the instantaneous power (in watt), energy it

consumption (in watt-hours) and status (on/off) of an industrial electrical appliance. The

ZigBee device can then in real-time send the measured information back to a central

processing unit or platform via the ZigBee Wireless Mesh network. This capability enables the

demand control system to dynamically determine the loads that are consuming power at that

specific instance in time (switched on)3. The demand control system can calculate what the

reduction in demand will be when the load is switched off.

Another clear benefit of using the ZigBee Technology is that it can cover large areas. The

control and monitoring coverage of large distributed electrical loads is not possible and too

2 Simultaneous maximum demand of a group of consumers divided by the number of consumers, expressed in kilovolt amperes. 3 It is important to know whether a load is consuming energy as a geyser could be switched on at the control point but not consuming energy as the thermostat switch can be off due to the water in the geyser being at its desired temperature.

8

costly using standard cables or conventional Wi-Fi technology. ZigBee Technology forms a

self-healing mesh where each node can act as a relay hop to form the mesh network and so

enables the communication coverage of large areas in a cost effective manner. Figure 4 shows

a large school property that has distributed geyser loads which is controlled via ZigBee based

load switches.

Figure 4: Arial view of property where ZigBee Wireless Mesh Technology load switches are installed. Large 200m radius

area.

The demand of the entire system and all the loads in Figure 4 are then sent to the central

processing platform every 10 seconds via the ZigBee Wireless Mesh Network.

The processing platform will be the Remote Data Acquisition and control or RDAc technology

that provides hardware intelligence for control, data logging and storage while integrating

multiple inputs and outputs with various communication possibilities. The RDAc platform is

regarded as a hybrid between cell phone, programmable logic controller (PLC) and Personal

Computer (PC) technology.

As the RDAc receives the data in real-time, it uses control algorithms specifically developed

for the system, to predict what the system demand would be within a 30 minute integration

period. The algorithm then calculates how much load (devices/appliances) it needs to shed

(switch off) or it restore (switch on) while remaining within the constraints of the demand set

9

point provided to the system. The switching of devices does not just consider the on/off status

of a device but considers other parameters such as priority and cool down periods depending

on the device it is controlling. The demand control algorithm and system can very accurately

determine what to control due to its two-way communication capability compared to older

systems that use statistical techniques.

There are a range of load management techniques such as peak clipping, valley filling,

strategic load conservation and load shifting as seen in Figure 5.

Figure 5: Load management load shape objectives. Source (Malik and AL Mata’ni 2007)

The assessed system predominantly uses the load shifting techniques which still utilises the

same total energy but results in lower electricity bills by reducing maximum demand peaks as

well as moving energy consumption into cheaper off-peak periods.

Figure 6 shows a scenario prior to the deployment of the assessed demand control system.

The required demand is more than 175 kilo volt ampere (KVA). When considering Figure 7,

the demand is managed to remain below 140 KVA via the demand control system which

equals a monthly saving of about R3500.

10

Figure 6: Demand of 175 KVA was reached prior to MDC install – monthly cost R15 118

Figure 7: Demand of 135 KVA was reached after MDC install – monthly cost R11 626

The system also uses smart meters to measure the load at the grid connection point. This

information is also sent back to the RDAc via the ZigBee network. All system data is also logged

in the RDAc in 1 minute and 30 minute log intervals and read back to a central long term data

storage and analytics server using automated meter reading via GPRS/3G on the Vodacom

network.

The users have two main ways of interacting with the system. The first is via an online web

portal which graphically displays the logged system data and enables the analysis of demand

trends and tariff studies. This can be seen in Figure 8 below.

Figure 8: Online Web portal to via historical and quasi real-time data

Users are also capable of monitoring and controlling the system via

supervisory control and data acquisition (SCADA) software, as seen in Figure 9, developed

11

specifically for the demand control system. The SCADA system enables users to monitor,

interact and control the physical system, in real-time, via a computer user interface.

Figure 9: SCADA interface which allowing users direct monitoring and control capability

3.5 Critical Technology Elements (CTEs)

Figure 10 shows the entire demand control ecosystem or architecture of the system under

assessment.

Figure 10: Real World Ecosystem: Smart Metering with energy efficient heat pumps and Zigbee based maximum demand

control

12

Table 1 outlines the identified CTEs within the demand control system which will be assessed

according to the hardware and software readiness definitions, descriptions and supporting

information set out in Appendix A (DoD 2011).

Technology Hardware/

Software

Function in Relation to System TRL Reason for TRL level

CTE1 ZigBee

Technology

HW Suite of high-level communication protocols

that allow the creating of a local mesh

network. Mesh network is critical for

deployment in a large decentralised

environment and enables the relay of two

way communication and data transfer. Load

switches are ZigBee enabled and capable of

measuring and controlling the connected

load. Measurement data is then relayed back

to the central control platform via the ZigBee

mesh network. Load switching devices as well

as all smart meters have ZigBee capabilities.

TRL9 The Development of ZigBee based

systems first emerged around 2005

(Eady 2005) and has now already

successfully been implemented and

operated in a range of applications.

Examples of where ZigBee is used

within a similar data acquisition

scenario can be found in Calmeyer

(2012) and Shariff, Rahim and Hew

(2015).

CTE2 3G

Technology

HW 3G (third generation) is regarded as the third

generation of mobile telecommunications

technology. 3G was used to send logged data

as well as real-time data back to the central

database server over the mobile operator

network. The main processing platform, the

RDAc, has 3G capabilities as well smart

meters.

TRL9 3G technology was introduced to the

market in 1998. The technology has

already progressed to 4G and 5G

technologies. Within the deployment

of the system the 3G technology

worked within an operational

environment.

CTE3 Linux

Technology

SW The RDAc platform uses Linux technology, an

open source platform, as its operating

system. An operating system enables the

software running on the device to access

hardware functions available on the device.

The RDAc platform hosts a range of hardware

capabilities such as digital inputs, analogue

inputs, digital outputs, Ethernet Interface, 3G

communications, RS232 interface, RS485

interface, storage media, LC and C-

programming language. The Linux operating

system acts as an interface to all the

hardware and software functions.

Linux is, in simplest terms, an operating

system. It is the software on a computer that

enables applications and the computer

TRL9 Linux was first introduced in 1991 and

has since become a stable platform

used in many types of applications.

Linux is known for its stability and

flexibility (Proffitt 2009). The Linux

operating system proved itself to be

stable within the operating

environment.

13

Technology Hardware/

Software

Function in Relation to System TRL Reason for TRL level

operator to access the devices on the

computer to perform desired functions.

CTE4 Control

Algorithm

SW The control algorithm which is developed

using the C programming language is at the

heart of the system. The algorithm collects all

the data via the Ethernet and ZigBee

interfaces. The algorithm then interprets the

information and makes load shedding or

restore decisions based on programmed

parameters.

TRL7 Demonstrated feasibility within

operational prototype scenarios.

Software fully integrated with

operational hardware and software

systems. The technological

capabilities of the software have been

measured against its required

capabilities.

Although some documentation has

been started not all documentation

has been completed.

CTE5 RDAc

Platform

HW The RDAc provides the platform the hardware

platform and computing power to collect

data, process data, execute programmes and

commands, log data, and communicate using

various technologies. It is central to the

system.

TRL8 The RDAc platform has proven to

operate successfully under expected

conditions providing satisfactory

results. The platform is not

experiencing any more base

development but slight incremental

improvements are made to

embedded code. The RDAc platform

meets its design specifications.

Although the platform operates as

required there are still some problems

encountered which are mainly related

to the embedded operating system

and software drivers.

CTE6 Smart

Metering

HW Smart meters are devices that can record the

consumption of electricity and capable of

two-way communications. Within the system

smart meters are deployed at select locations

such as the main point of supply from the

grid. This is the point where energy is billed

and also the main measurement point for the

system demand. The smart meters are

equipped with ZigBee communications for

local area network communication and the

relay of information back to the central

processing platform. The smart meters also

gave 3G modems which enable the

communication of data back to the central

data collection server.

TRL9 The smart meter technology has

proven to operate successfully under

operational scenarios. The platform is

not experiencing any more

development. Smart meter

technology has also been used

successfully within the metering

industry for a number of years (Aslam,

Soban, Akhtar and Zaffar 2015, Bago

and Campos 2015).

14

Technology Hardware/

Software

Function in Relation to System TRL Reason for TRL level

CTE7 TCP/IP SW Transmission Control protocol and Internet

Protocol (TCP/IP) is the basic communication

language used by private networks as well as

the Internet. TCP/IP offers what is called end-

to-end communications and specifies how

data is packaged, addressed, sent, routed and

the received at the desired destination.

Within the system TCP/IP is used between the

RDAc platform and the ZigBee devices. The

ZigBee network coordinator collects data

from the ZigBee Mesh Network and then

encapsulated the data and transports it

within the TCP protocol over an Ethernet

connection to the RDAc platform. The RDAc

also uses 3G technology which used TCP/IP.

TRL9 TCP/IP technology has been available

for many years and has been

integrated into everyday life. The

TCP/IP software is readily repeatable

and usable and completely integrated

into the operational hardware,

software and environment. TCP/IP has

been documented and verified.

TCP/IP has had successful operational

experience with sustainable

engineering support.

Table 1: Technology Readiness Levels for Demand Control System

3.6 Review of TRL Findings

The summary of the technology readiness assessment can be seen in Table 2. A general

observation is that most technologies are relatively mature with the selected technology

readiness levels ranging from TRL7 to TRL9. All technologies have proven to work within an

isolated environment as well as within the system environment. The conclusion drawn from

Table 2 is that there are no technologies within the system which falls below TRL7 and the

project can thus be approved for Milestone C, instead if Milestone B, which is the approval to

enter into low production.

Identified CTE CTE Technology Hardware/Software Technology Readiness Level

CTE1 ZigBee Technology HW TRL8

CTE2 RDAc Platform HW TRL8

CTE3 3G Technology HW TRL9

CTE4 Linux Technology SW TRL9

CTE5 Control Algorithm SW TRL7

CTE6 Smart Metering HW TRL9

CTE7 TCP/IP SW TRL9

Table 2: Summary of CTE Technology Readiness Level

According to TRL Desk book (DoD 2009), Milestone C marks the point at which low rate

production can be imitated with limited deployment of in order to test operational readiness.

15

Milestone C is important as it should show how technology deficiencies have been resolved

during the Engineering and Manufacturing Development (EMD) phase. It also indicates that

all CTEs are maturing as anticipated and will continue to mature through the use of

Technology Maturation Plans. Considering software, TRL7 indicates that all the source code

has been developed and has been tested to ensure that it integrates into the entire system

and that it can successfully be ported to a different host platform (DoD 2009).

How does this assessment compare to a real-world situation? The demand control system

described in previous sections for the document was deployed at three commercial

businesses in Mpumalanga after it went through the initial development process. The initial

development process took substantially longer than anticipated as there were two major

issues. The first challenge was that the ZigBee product platform developed by a Danish

company was still undergoing changes. The current products they had available worked as

required but after introducing the changes, which were required to operate on the ZigBee

Smart Energy Profile, their platform experienced some problems. This was mainly due to the

added security on the Smart Energy profile which required that the hardware had more

processing and storage capacity. This delayed the initial deployment schedule by 6 months as

they were upgrading and refining their solution. The second challenge was that the initial

selected central processing platform was not flexible and capable enough of handling all the

required system functionality which evolved with the project. The RDAc was then selected as

the best available platform but was never deployed in this particular solution or system. The

combined delay in delivering the first system was 12 months. All the technologies were tested

individually and tested as an integrated system and deployed to commercial customers. After

1 month of successful operation the ZigBee platform, more specifically the software code on

the ZigBee hardware, crashed leaving the entire system non-operational and after another 6

months of effort the system was upgraded in order to offer more stability which incremental

changes to other parts of the system as well.

The TRL alone does not seem adequate enough to ensure that a technology within an

integrated system can be successfully deployed. The literature finds many cases where TRL as

16

a metric is criticised and according to Cornford and Sarsfield (2004) it is not accurate,

subjective with no clear definition and does not have much value in supporting certain

decisions. Tetlay and John (2010) also debate that that the term “System Readiness” and

“Maturity” should not be used in an interchangeable manner. Other concerns are that it does

not offer an encompassing way of understanding the difficulty of integrating individual

technologies or subsystems into a complete operational system (Mankins 2002, Valerdi and

Kohl 2004) and that it does not provide a way of comparing alternate TRLs (Valerdi and Kohl

2004, Smith 2005, Mankins 2002). The inference made in the literature is that when a system

is considered instead of a single technology there are a more comprehensive set of metrics

required to assess system readiness. Sauser, Verma, Ramirez-Marquez and Gove (2006)

propose that an additional readiness metric called the Integration Readiness Level (IRL) needs

to be used on conjunction with the Technology Readiness Level (TRL) metric in order to

determine the system readiness which is then in turn measured by a System Readiness Level

(SRL). Bilbro (2007) also proposes using two different metrics when assessing technologies;

the first is TRL scale developed by NASA as well as another scale consisting of nine levels which

is called the Advancement Degree of Difficulty.

Azizian, Sarkani and Mazzuchi (2009) provide a summary of the numerous readiness levels and maturity assessments and

can be seen in

Table 3 below.

17

Table 3: Techniques for Assessing Qualitative Maturity. Source (Azizian et al. 2009)

4. Review of Demand Control System Using Qualitative

Maturity Multi Metric Technique

Bilbro (2007) regards the technology assessment as a two tiered process which first considers

the current technology maturity though assessing TRLs and then secondly; follows another

process by which it is determined how challenging it will be to take move a technology from

its current TRL to the next TRL by using Advancement Degree of Difficulty (AD2). The iterative

process can be seen in Figure 11 below.

18

Figure 11: Technology Assessment process proposed by Bilbro (2007).

Although a Microsoft Excel tool does exist to perform an assessment using the method

proposed by Bilbro (2007), it was found that the tool is very complex and does not offer real

guidance in terms of interpreting the results. For the purpose of this paper the method

proposed by Sauser and Ramirez-Marquez (2007) will be assessed from a practical

perspective. According to Sauser and Ramirez-Marquez (2007), the inefficiencies of TRLs, as

discussed in the previous section, can be addressed by a composite method, which they have

termed the System Readiness Level (SRL), to act as a quantifier in assessing system maturity.

The SRL is a function of the existing TRL of each technology within a system and IRL which is

a metric to assess the complexity involved with integrating the relevant technologies. The

definition of each IRL can be seen in Table 4 with Figure 12 graphically representing the

rational by which SRL is developed.

Figure 12: Relationship between TRL, IRL and SRL. Source (Sauser and Ramirez-Marquez 2007)

19

Table 4: Integration Readiness Levels. Source (Sauser and Ramirez-Marquez 2007)

The SRL metric can thus be calculated as a TRL and IRL pair-wise comparison matrix that is

normalised and then interpreted as index of maturity between 0 and 1 (Sauser and Ramirez-

Marquez 2007). The single column [TRL] matrix is defined in Equation 1 where a system of n

technologies.

Equation 1

20

The possible integration of all technologies are then represented in a symmetric (n x n) square

matrix. In a system which has n technologies the [IRL] matrix is defined in Equation 2 with the

IRL between technologies i and j is indicated by IRLij. In the event that no integration is

planned between two technologies it is given a IRL of 9 (Sauser and Ramirez-Marquez 2007).

Equation 2

The values that were originally obtained from the TRL and IRL can be used but the normalising

the values provided a more accurate comparison when the use of competing technologies are

considered. The original [TRL] and [IRL] matrix values are thus generally normalised from their

1 to 9 levels to 0 to 1. The [SRL] matrix is then calculated by multiplying [TRL] and [IRL] as seen

in Equation 3.

Equation 3

The calculated [SRL] matrix contains a single element for every fundamental technology

regards to integration. The readiness of level of each specific technology in relation to another

technology is quantified while simultaneously accounting for the state of development of

each respective technology through its TRL. The [SRL] calculation for a system with n

technologies can be seen in Equation 4.

Equation 4

The resulting SRL values that are calculated via Equation 4 will reside in the interval 0 to n but

from a consistency perspective should be normalised to the interval 0 to 1 by dividing by n.

21

Interestingly the [SRL] matrix can also be used as a tool to assess which elements should be

prioritised in terms of system technology integration and expose paucities within the maturity

process.

Lastly the SRL for the entire system is calculated as the average of all the [SRL] values that

were normalised and the calculation can be seen in Equation 5.

Equation 5

4.1 SRL Calculation for the Demand Control System

The demand dontrol system that was assessed in previous sections will now be assed

according to the multi metric techniques proposed by (Sauser and Ramirez-Marquez 2007).

After selecting the appropriate Integration Readiness levels according to Table 4, the [SRL]

matrix and SRL of the demand control System is calculated using an online tool developed by

the Stevens Institute of Technology (SIT 2010). The results can be seen in Figure 13. Firstly,

the [SRL] results for each individual technology which is indicated as the Integrated

Technology Readiness Level for each technology in Figure 13 is considered. The SRL for the

RDAc has the lowest score at 0.83 indicating that the technology has deficiencies within the

maturity process and needs to be prioritised. Interestingly there are other technologies such

as the control algorithm which and TCP/IP with a SRL of 0.85 which is lower than the ZigBee

SRL of 0.86. Intuitively it would make sense that the RDAc, which is at the heart of the system,

would have the lowest SRL score. However the scores for the TCP/IP and ZigBee technologies

were not was expected. TCP/IP is a very mature technology and is standard almost equipment

for most technology based projects while the most demand control system failures were due

to the ZigBee platform.

22

Figure 13: System Readiness Level Calculation. Source (SIT 2010)

The system SRL can be mapped against the engineering lifecycles shown in Figure 14. The SRL

score of 0.86 would indicate that the system is ready for use according to all of the

engineering lifecycles shown in in Figure 14.

23

Figure 14: Different Engineering Lifecycles and how the System Readiness Level (SRL) is mapped. Source (Sauser and

Ramirez-Marquez 2007)

This would verify the decision to have had deployed the demand control in an operational

environment. However, the system would often fail and remain offline for extended periods

(2 – 4 weeks) due to problems with the ZigBee technology. Once the ZigBee component was

restored due to upgrades, the system would function for extended periods of time (6 – 9

months). However Sauser and Ramirez-Marquez (2007) does state that a system will hardly

ever achieve an SRL of greater than 0.9 as systems are generally deployed with technology

and integration which is not completely mature.

5. Conclusion

The demand control System was first tested using the TRL method developed by the U.S DoD

(DoD 2009). The obtained results indicate that all the technology was mature enough to reach

Milestone C which dictated limited system deployed in order to test whether the system was

ready for operations and does not mean that the system is completely ready for in-field

deployment.

24

In reality, the demand control system was deployed on a limited scale with three installations

but as a commercially ready solution and with no Technology Maturation Plans, which goes

against the recommendation of the book (DoD 2009). The demand control system, initially

operated according to the initial specifications but after a few weeks of operating,

experienced a major failure due to the ZigBee technology. The problems with the ZigBee

technology were mainly due to the ZigBee equipment not being stringently tested in a similar

condition with a weak ZigBee network and large volumes of data constantly having to retry

and find alternate routes. When considering the TRL of the ZigBee technology alone, it would

seem that is has matured enough to be deployed as a technology which was not the case. The

TRA where only the TRL is used as a metric thus seems to have limitations when integration

between technologies is required.

Literature revealed that TRL metric alone does not seem adequate to ensure that a

technology within an integrated system can be successfully deployed and that there is a

requirement for a more comprehensive set of metrics required to assess system readiness

(Cornford and Sarsfield 2004, Tetlay and John 2010, Valerdi and Kohl 2004, Smith 2005,

Mankins 2002). In order to address the deficiencies in the sole use of the TRL, the SRL multi

metric assessment method, proposed by Sauser et al. (2006), was used to assess the demand

control system. The SRL which uses the conventional TRL metric in conjunction with another

metric called the IRL to derive a SRL providing a slightly more comprehensive result which

indicated where priority should be given to increasing integration maturity. However

although the SRL provided a more comprehensive analysis of where the integration focus

should be some of the results were slightly counter intuitive leaving some doubt around the

accuracy or the interpretation of the TRL and IRL level definitions. The overall System

Readiness calculated at 0.86 indicates that the system is ready for operational use and

reinforces the decision to have deployed the demand control system. However, due to

practical challenges faced after deploying the demand control system into its intended

operational environment it is questionable how useful these metrics are and it is clear that

there are gaps within the methodology.

25

Integration is a complex subject and interpreting the TRL and IRL levels are not always easy.

Although the case study of the demand control system indicates that the IRL is capable of

revealing integration maturity apprehensions regardless of the high TRL levels of the

integrating technologies. There are however still a few uncertainties surrounding the IRL such

as during which level of integration should the IRL be applied, how IRL handles emergent

system behaviour, its inability to assess R&D effort or costs and schedules (Sauser et al. 2010).

It would seem reasonable to conclude that the metrics are not able to guarantee the complete

success of an integrated system.

26

Appendix A

Table 5: Hardware Technology Readiness Definitions, Descriptions, and Supporting Information. Source (DoD 2009)

27

Table 6: Hardware Technology Readiness Definitions, Descriptions, and Supporting Information. Source (DoD 2009)

28

`

29

Table 7: Additional Definitions of TRL Descriptive Terms. Source (DoD 2009)

30

Appendix B

Figure 15: Descriptive Requirements for Technology Readiness Assessment Document. Source (DoD 2009)

31

Bibliography

ABU-ZEID, N. S. & AL-SHAKARCHI, M. R. G. 2002. A Study of Load Management by Direct Control for Jordan's Electrical Power System. Science & Technology, 7.

ASLAM, W., SOBAN, M., AKHTAR, F. & ZAFFAR, N. A. 2015. Smart meters for industrial energy conservation and efficiency optimization in Pakistan: Scope, technology and applications. Renewable and Sustainable Energy Reviews, 44, 933-943.

AZIZIAN, N., SARKANI, S. & MAZZUCHI, T. A comprehensive review and analysis of maturity assessment approaches for improved decision support to achieve efficient defense acquisition. Proceedings of the World Congress on Engineering and Computer Science, 2009. 20-22.

BAGO, R. & CAMPOS, M. 2015. 16 - Smart meters for improved energy demand management: The Nordic experience. In: BESSÈDE, J.-L. (ed.) Eco-Friendly Innovation in Electricity Transmission and Distribution Networks. Oxford: Woodhead Publishing.

BILBRO, J. W. 2007. Systematic Assessment of the Program/Project Impacts of Technological Advancement and Insertion. George C. Marshall Space Flight Center.

BOLAT, S. 2014. Technology Readiness Level (TRL) math for innovative SMEs [Online]. Available: http://serkanbolat.com/2014/11/03/technology-readiness-level-trl-math-for-innovative-smes/ [Accessed 14 February 2015].

BUEDE, D. M. 2000. The engineering design of systems: models and methods, Wiley. CALMEYER, J. 2012. ZigBee as a technology platform for advanced metering infrastructure (AMI).

63rd AMEU Convention. CORNFORD, S. L. & SARSFIELD, L. Quantitative methods for maturing and infusing advanced

spacecraft technology. Aerospace Conference, 2004. Proceedings. 2004 IEEE, 6-13 March 2004 2004. 681 Vol.1.

DOD 2009. Technology Readiness. Assessment (TRA)Deskbook. DOD 2011. Department of Defense Technology Readiness Assessment (TRA) Guidance. EADY, F. 2005. Chapter 17 - A New Kid in Town Who Calls Himself ZigBee. In: EADY, F. (ed.)

Implementing 802.11 with Microcontrollers. Burlington: Newnes. MALIK, A. S. & AL MATA’NI, F. S. 2007. Load Management In Industrial Sector And Its Impact On

Power Planning – A Case Study Of Oman. International Conference on Communication, Computer & Power. Muscat.

MANKINS, J. C. 2002. Approaches to strategic research and technology (R&T) analysis and road mapping. Acta Astronautica, 51, 3-21.

MANKINS, J. C. 2009. Technology readiness assessments: A retrospective. Acta Astronautica, 65, 1216-1223.

PROFFITT, B. 2009. What Is Linux: An Overview of the Linux Operating System [Online]. Available: https://www.linux.com/learn/new-user-guides/376-linux-is-everywhere-an-overview-of-the-linux-operating-system [Accessed 15 February 2015].

SAUSER, B., GOVE, R., FORBES, E. & RAMIREZ-MARQUEZ, J. E. 2010. Integration maturity metrics: Development of an integration readiness level. Information, Knowledge, Systems Management, 9, 17-46.

SAUSER, B. & RAMIREZ-MARQUEZ, J. E. 2007. A System Maturity Index for Decision Support in Life Cycle Acquisition. DTIC Document.

SAUSER, B., VERMA, D., RAMIREZ-MARQUEZ, J. & GOVE, R. 2006. From TRL to SRL: The concept of systems readiness levels. Conference on Systems Engineering Research, Los Angeles, CA.

SHARIFF, F., RAHIM, N. A. & HEW, W. P. 2015. Zigbee-based data acquisition system for online monitoring of grid-connected photovoltaic system. Expert Systems with Applications, 42, 1730-1742.

32

SIT. 2010. System Readiness Level Calculator [Online]. Available: http://personal.stevens.edu/~bsauser/SysDML/Methods,_Processes,_Tools_files/index.htm.

SMITH, J. D. An Alternative to Technology Readiness Levels for Non-Developmental Item (NDI) Software. System Sciences, 2005. HICSS '05. Proceedings of the 38th Annual Hawaii International Conference on, 03-06 Jan. 2005 2005. 315a-315a.

TETLAY, A. & JOHN, P. 2010. Clarifying the Concepts of System Maturity, System Readiness and Capability Readiness through Case Studies.

VALERDI, R. & KOHL, R. J. 2004. An approach to technology risk management. Engineering Systems Division Symposium, 2004 Cambridge. MIT.