DELIVERABLE D2.6 Evaluation Report of the bIoTope...
Transcript of DELIVERABLE D2.6 Evaluation Report of the bIoTope...
DELIVERABLE
This project has received financial support from the European Union Horizon 2020 Programme under grant agreement no. 688203.
D2.6 Evaluation Report of the bIoTope Pilots
Project Acronym: bIoTope
Project title: Building an IoT Open Innovation Ecosystem for Connected Smart Objects
Grant Agreement No. 688203
Website: www.bIoTope-project.org
Version: 1.0
Date: 30 June 2017
Responsible Partner: BIBA – Bremer Institut für Produktion und Logistik
Contributing Partners: BMW, CIRB, ControlThings, CSIRO, Enervent, FVH, Greater Lyon, IRISNET
Dissemination Level: Public X
Confidential – only consortium members and European Commission Services
Ref. Ares(2017)3317345 - 03/07/2017
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 2 30th of June 2017
Revision History
Revision Date Author Organization Description
0.1 29/05/2017 Dirk Werthmann BIBA Initial draft
0.2 12/06/2017 Dirk Werthmann BIBA Final draft for reviewing
0.3 20/06/2017 Robert Hellbach BIBA Inserting remarks and comments from
reviewers
1.0 28/06/2017 Robert Hellbach BIBA Final Version
Every effort has been made to ensure that all statements and information contained herein are accurate, however the bIoTope Project Partners accept no liability for any error or omission in the same.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 3 30th of June 2017
Table of Contents
1. Introduction ..................................................................................................................................... 8
1.1. Objective of the Deliverable ........................................................................................................................ 8
1.2. Structure of the Deliverable ......................................................................................................................... 9
1.3. Evaluated subjects within bIoTope .............................................................................................................. 9
1.3.1. bIoTopes’ Core Components .................................................................................................................... 10
1.3.2. bIoTopes’ Pilots ........................................................................................................................................ 10
2. Progress of development within bIoTope ........................................................................................ 12
2.1. Evaluation by assessing the Technology Readiness Level as well as the Integration Readiness Level within
bIoTope .................................................................................................................................................................. 12
2.2. Evaluation of Core components ................................................................................................................. 18
2.3. Evaluation of Technologies developed within the Cross-domain Smart City Pilots..................................... 19
2.3.1. Pilot: Brussels – Capital Region ................................................................................................................. 19
2.3.2. Pilot: FVH – Forum Virum Helsinki ............................................................................................................ 19
2.3.3. Pilot: Greater Lyon .................................................................................................................................... 19
2.4. Evaluation of Technologies developed within the Domain-specific Pilots .................................................. 20
2.4.1. Pilot: Smart Mobility ................................................................................................................................. 20
2.4.2. Pilot: Smart Building and Equipment ........................................................................................................ 20
2.4.3. Pilot: Smart Air Quality ............................................................................................................................. 20
3. Matching of KPIs and Requirements ................................................................................................ 21
3.1. Evaluation Methodology for Matching of KPIs and Requirements ............................................................. 21
3.2. Evaluation of Cross-domain Smart City Pilots ............................................................................................ 23
3.2.1. Pilot: Brussels – Capital Region ................................................................................................................. 23
3.2.2. Pilot: FVH – Forum Virum Helsinki ............................................................................................................ 25
3.2.2.1. Use Case: Electrical Car charging service using IoT BnB concept ......................................................... 25
3.2.3. Pilot: Greater Lyon .................................................................................................................................... 27
3.2.3.1. Use Case: Bottle Banks ......................................................................................................................... 27
3.2.3.2. Use Case: Heat Wave ........................................................................................................................... 29
3.3. Evaluation of Domain-specific Pilots .......................................................................................................... 31
3.3.1. Pilot: Smart Mobility ................................................................................................................................. 31
3.3.2. Pilot: Smart Building and Equipment ........................................................................................................ 35
3.3.3. Pilot: Smart Air Quality ............................................................................................................................. 39
4. Preparation of User Acceptance Testing for ex-post Evaluation within bIoTope Pilots ....................... 42
4.1. Evaluation of bIoTope Pilots based on User Acceptance Testing ................................................................ 42
4.2. Evaluation of Cross-domain Smart City Pilots ............................................................................................ 49
4.2.1. Pilot: Brussels – Capital Region ................................................................................................................. 49
4.2.2. Pilot: FVH – Forum Virum Helsinki ............................................................................................................ 51
4.2.2.1. Use Case: EV Charging & Parking App .................................................................................................. 51
4.2.3. Pilot: Greater Lyon .................................................................................................................................... 53
4.2.3.1. Use Case: Bottle Banks ......................................................................................................................... 53
4.2.3.2. Use Case: Heat Wave ........................................................................................................................... 55
4.3. Evaluation of Domain-specific Pilots .......................................................................................................... 57
4.3.1. Pilot: Smart Mobility ................................................................................................................................. 57
4.3.2. Pilot: Smart Building and Equipment ........................................................................................................ 59
4.3.3. Pilot: Smart Air Quality ............................................................................................................................. 61
4.3.4. Preparation of the KPI Test Scenarios for ex-post Evaluation of the bIoTope Pilots ................................ 63
5. Conclusion ...................................................................................................................................... 66
6. References ..................................................................................................................................... 67
7. Annex 1 .......................................................................................................................................... 68
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 4 30th of June 2017
List of Tables
Table 1: Technology Readiness Levels (TRL) definition [U.S. Government Accountability Office] ................... 13
Table 2: Integration Readiness Level (IRL) [Sauser] .......................................................................................... 14
Table 3: Core Components assessed by using TRL ............................................................................................ 15
Table 4: Technologies/ Software Components assessed by using TRL ............................................................. 15
Table 5: TRL and IRL of bIoTopes’ Core Components ........................................................................................ 18
Table 6: TRL and IRL of Pilot: Helsinki ................................................................................................................ 19
Table 7: TRL and IRL of Pilot: Greater Lyon ....................................................................................................... 19
Table 8: TRL and IRL of Pilot: Smart Mobility .................................................................................................... 20
Table 9: TRL and IRL of Pilot: Smart Building and Equipment ........................................................................... 20
Table 10: TRL and IRL of Pilot: Smart Air Quality............................................................................................... 20
Table 11: Structure of the spreadsheet for matching KPIs and Requirements ................................................. 22
Table 12: Matching of KPIs and Requirements for Pilot: Brussels – Capital Region ......................................... 23
Table 13: Matching of KPIs and Requirements for Pilot: Forum Virium Helsinki - Electrical Car charging service
using IoT BnB concept ............................................................................................................................... 25
Table 14: Matching of KPIs and Requirements for Pilot: Greater Lyon - Use Case: Bottle Banks ..................... 27
Table 15: Matching of KPIs and Requirements for Pilot: Greater Lyon - Use Case: Heat Wave ....................... 29
Table 16: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Charging Station Selection
Service ....................................................................................................................................................... 31
Table 17: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Route Planning Service . 33
Table 18: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Electric Car Gear Service34
Table 19: Matching of KPIs and Requirements for Pilot: Smart Building and Equipment - Use Case: Smart
Building Interaction with Smart Mobility .................................................................................................. 35
Table 20: Matching of KPIs and Requirements for Pilot: Smart Building and Equipment - Use Case: Smart
Equipment ................................................................................................................................................. 38
Table 21:Matching of KPIs and Requirements for Pilot: Smart Air Quality – Identify polluted areas .............. 39
Table 22: Matching of KPIs and Requirements for Pilot: Smart Air Quality – Scenario Provide councils view of
their city ..................................................................................................................................................... 41
Table 23: Example of a Test Condition Matrix .................................................................................................. 46
Table 24: Example of a Test Schedule ............................................................................................................... 46
Table 25: Start dates and end dates for User Acceptance Testing within bIoTopes’ Pilots .............................. 48
Table 26: Test Condition Matrix for Pilot: Brussels – Use Case: Safety around School .................................... 49
Table 27: Test Schedule for Pilot: Brussels – Use Case: Safety around School ................................................. 50
Table 28: Test Condition Matrix for Pilot: Forum Virium Helsinki - Use Case: EV Charging & Parking App...... 51
Table 29: Test Schedule for Pilot: Forum Virium Helsinki - Use Case: EV Charging & Parking App .................. 52
Table 30: Test Condition Matrix for Pilot: Greater Lyon - Use Case: Bottle Banks ........................................... 53
Table 31: Test Condition Matrix for Pilot: Greater Lyon - Use Case: Heat Wave .............................................. 55
Table 32: Test Condition Matrix for Pilot: Smart Mobility ................................................................................ 57
Table 33: Test Schedule for Pilot: Smart Mobility ............................................................................................. 58
Table 34: Test Condition Matrix for Pilot: Smart Building and Equipment ....................................................... 59
Table 35: Test Schedule for Pilot: Smart Building and Equipment .................................................................... 60
Table 36: Test Condition Matrix for Pilot: Smart Air Quality ............................................................................. 61
Table 37: Test Schedule for Pilot: Smart Air Quality ......................................................................................... 62
Table 38: Progress of the preparation for the KPI Test Scenarios for ex-post Evaluation ................................ 63
Table 39: Start dates and end dates for executing the KPI Test Scenarios for ex-post Evaluation within bIoTopes’
Pilots .......................................................................................................................................................... 64
Table 40: Template for the KPI-based ex-post Evaluation of the bIoTope Pilots .............................................. 65
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 5 30th of June 2017
List of Figures
Figure 1: Initial Evaluation Procedure of the Business Services within bIoTope according to D2.2 “Evaluation
Methodology for Pilots Validation” ........................................................................................................... 43
Figure 2: Status of User Acceptance Testing at month 18 of bIoTope [Hambling and von Goethem] ............. 44
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 6 30th of June 2017
Executive Summary
This deliverable (D2.6) reports the evaluation of technologies developed within bIoTope, especially the domain-specific pilots,the cross-domain smart city pilots and furthermore used technologies developed in WP3-5 and used in the pilots. D2.6 keeps track of the development of the project by assessing the Technology Readiness Level (TRL) as well as the Integration Readiness Level (IRL) for the core components, the cross-domain city pilots and the domain-specific pilots. The deliverable also evaluates the matching of Key Performance Indicators (KPIs) and requirements for the cross-domain smart city pilots and the domain-specific pilots. Finally, D2.6 prepares the ex-post evaluation of the bIoTope pilots based on user acceptance testing and for the cross-domain smart city pilots.
Based on the previous deliverable D2.2 “Evaluation Methodology for Pilots Validation” from the track of Task T.2.E “Evaluation Methodology and Pilot Validation” the evaluation methodology is applied amongst the project results, reflecting an intermediate status of the bIoTope project due to month 18. Within the evaluation, the project partners evaluated each technology according to the status of the ongoing development as well as defined a target level (based on TRL and IRL), which they want to reach until the end of the project. Pilots as well as technologies that are developed in WP3, 4 and 5, which are fundamental for an interoperation in an IoT ecosystem, are considered. Those technologies (see Table 5 and regarding deliverables for further details) rely to the Everything-as-a-Service (XaaS) approach, support the interconnection / orchestration of heterogeneous services and establish business value inside the bIoTope ecosystem.
After a short introduction to this deliverable in the first chapter, each pilot is quantitative evaluated based on the TRL and IRL description. Summarized, current cross-pilot TRL assessment turned out legitimately high, while IRL provides more innovation potential. This is actually the representation of a problem statement, bIoTope address in the current landscape of IoT, whereas already existing smart objects (products with high TRL) need to be orchestrated in a way that seamless information flow between vertical silos can be achieved. The problem representation consequently is directly characterized in the IRL indications of the pilots and XaaS connected technologies.
Facing previous results, the following chapter documents the mapping of recorded requirements from deliverable D2.1 with KPIs stated in deliverable D2.2. This shall ensure an alignment of the evaluation procedure to the intended project results according to the pilots. The relationship of requirements to KPIs is n to 1, which addresses a measurability in the evaluation process. Then, the already prepared evaluation procedure starts with the preparation of User Acceptance Testing. This considers test conditions and schedule for each pilot and gives the pilot case owners a guideline for evaluating their developments. Future work is and conclusions are stated in the last chapter, while changes that will occur during the agile development will be documented in the following deliverable D2.9.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 7 30th of June 2017
Terms
Big Data as a Service (BDaaS) A core component of bIoTopes’ ecosystem to handle the amount of
services and respectively data in the ecosystem.
Billing as a Service (BiaaS) A core component of bIoTopes’ ecosystem to buy and pay for services.
Business Service (BS) A digital service helping end users to perform a task in a better way.
Context as a Service (CoaaS) A core component of bIoTopes’ ecosystem to respond intelligently to
context-related events.
Development Test & Evaluation (DT&E) A testing procedure, which is conducted during the engineering and
development process to ensure that the specifications have been met.
Everything-as-a-service (XaaS) Approach used in bIoTope to define functionalities as services used in
a system of systems landscape.
Information as a Service (IaaS) A core component of bIoTopes’ ecosystem to discover services within
the ecosystem.
Integration Readiness Level (IRL) A metric of the integration maturity between two or more
components.
Internet of Things (IoT) Interconnection and -action (collecting and exchange of data) of things
(physical devices) embedded with electronics, software, sensors,
actuators and connectivity.
Internet of Things service PuBlication
and Billing (IoTBnB Marketplace/ Service
Catalogue)
A core component of bIoTopes’ ecosystem that acts as a service
catalogue for discovering and consuming data, connected to a billing
framework that allows micropayments for the value of data and / or
services.
Knowledge as a Service (KaaS) A core component of bIoTopes’ ecosystem to interpret and analyse a
huge amount of data.
Key Performance Indicator (KPI) An indicator to evaluate the success of organisations or activities.
National Aeronautics and Space
Administration (NASA)
A governmental organisation of the United States of America, being
responsible for aeronautics research, explores the earth and the whole
universe, development and operation of space technology as well as
commercial spaceflights.
Open Source Software (OSS) Licensed source code that is available for studying, change and
distribution to anyone for any purpose.
Operational Test & Evaluation (OT&E) A field test, which is conducted under realistic conditions.
Security as a Service (SeaaS) A core component of bIoTopes’ ecosystem to ensure a trust-based
communication of new business models to protect its stakeholders
and their data / services..
System of systems (SoS) Collection of different systems, which can co-exist as stand--alone
version, but are able to pool resources and capabilities.
Technology Maturation Plan (TMP)
A plan, which is aiming at bringing immature critical technology up to a desired TRL.
Technology Readiness Assessment (TRA) A systematic that evaluates the maturity of technologies.
Technology Readiness Level (TRL) A measurement scale used to rate technologies according to their maturity.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 8 30th of June 2017
1. Introduction
bIoTope aims to improve economic, ecological and societal sustainability by supporting innovation and cocreation of services in the Internet of Things (IoT). To achieve this, bIoTope develops an open, interoperable, secure and highly context-sensitive Systems-of-Systems (SoS) platform for IoT that will enable developers to publish, consume and compose services without any programming.
Within the project, work package WP2 “Requirements, Methods and Pilot Validation” is responsible for producing the technical and Open Source Software (OSS) specifications of the bIoTope IoT based SoS platform collected and processed from all bIoTope stakeholders. Stakeholders can be cities, application integrators, developers, end-users, and others. The structured and methodical collection and analysis of requirements allows not only the definition and subsequent development of the City Pilots in the involved cities, but also the definition of a proper methodology for pilot evaluation and validation throughout the project duration. Thus, WP2 also supports the software development process and ensures that coding procedures and conventions are respected throughout the project. The overall goal of WP2 is to ensure, that the bIoTope project will reach the objectives presented in the Part B of the proposal.
To summarize,WP2 has to achieve the following tasks:
Elicit and analyse requirements from all the stakeholders involved in the bIoTope ecosystem, including the involved cities, infrastructure providers, application integrators, platform developers, end-users, open source communities.
Define the most relevant City Pilots according to the requirement analysis outcomes.
Produce technical specifications of the bIoTope SoS platform for the IoT in terms of functional and non-functional features. The structuring specification also addresses the open source project.
Specify an evaluation methodology and validation plan of the bIoTope pilots.
1.1. Objective of the Deliverable
In the context of WP2, deliverable D2.6 reports part of the results of Task 2.E “Evaluation Methodology and Pilot Validation”. Task 2.E develops a framework for the evaluation of bIoTopes’ Core Components and Pilots. The periodic evaluation in this task enables agile adjustment of the validation arrangements where needed, peer learning and knowledge transfer, as well as best practice sharing across the Use Cases. The aim of the evaluation framework is to assess the Use Case internal arrangements and validation processes, as well as their impact against the KPIs assigned for the validation outcome.
The evaluation includes ex-ante-evaluation and ex-post-evaluation. The ex-ante evaluation focuses on actors, processes and interfaces in the Use Cases. The ex-post evaluation will focus on policy, economic, societal, technical and environmental impact of the Use Cases. Such evaluation methodology allows refining RTD specifications all along the project. This version of deliverable D2.6 is mainly concerned with ex-ante-evaluation, while ex-post-evaluation will be performed at a later stage. However, proper preparation of the ex-post-evaluation is an important topic in this version as well.
The evaluation work reported in deliverable D2.6 is based on the evaluation methodology developed in deliverable D2.2 “Evaluation Methodology for Pilots Validation” and the general description of the quality methodology within deliverable D1.2 “Quality plan”. Deliverable D1.2 specifies the objectives and initial set of Key Performance Indicators (KPIs) in general. Thereupon, Deliverable D2.2 describes evaluation methodologies for assessing bIoTopes’ Core Components and Pilots. Prominent among the system assessment methodologies are the Technical Readiness Level (TRL) and the Integration Readiness Level (IRL) assessment. Moreover, User Acceptance Testing is part of the evaluation as well. These methodologies have been used in order to prepare the evaluation reported in this deliverable.
Deliverable D2.2 also describes the state-of-the-art in the Cross-domain Smart City Pilots and the Domain-specific Pilots, as well as the benefits of bIoTopes’ implementations within these pilots.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 9 30th of June 2017
Deliverable D2.6 also considers the results of deliverable D2.1 “Ecosystem Stakeholder Requirements Report and Pilots Definition” and deliverable D2.4 “bIoTope SoS Reference Platform Specification” (version 1.1 from 20 January 2017).
Deliverable D2.1 reports the requirements of the various bIoTope stakeholders, such as infrastructure providers, application integrators, and application service providers, according to the envisaged value chains. Deliverable D2.4 transforms the fundamental requirements into a specification of the overall architectural framework and the specifically detailed Pilots. It describes the development of the SoS platforms and service components, as well as the implementation strategies of the Pilots and their interrelations. Thus, the requirements of each Use Case described in deliverable D2.1 have been detailed in user stories and use cases descriptions.
What is evaluated in deliverable D2.6 are Core Components and Pilots. On the one hand D2.6 evaluates technologies, which means the critical services and Core Components that bIoTope develops. On the other hand, it evaluates real world Use Cases within the Cross-domain Smart City Pilots and the Domain-specific Pilots, in which bIoTope implements the developed technologies.
1.2. Structure of the Deliverable
The deliverable consists of five main chapters, each comprising several sections.
Chapter 1 describes the objectives of the deliverable within the context of work package 2 (section 1.1) and the structure of the deliverable (section 1.2)
Chapter 2 keeps track of the development progress within bIoTope by assessing the Technology Readiness Level (TRL) and the Integration Readiness Level (IRL), the core components (services), the cross-domain city pilots and the domain-specific pilots. Section 2.1 assesses the Technology Readiness Level (TRL) and the Integration Readiness Level (IRL); section 2.2 assesses the core components. Section 2.3 evaluates the technologies developed within the cross-domain smart city pilots and section 2.4 evaluates the technologies developed within the domain-specific pilots.
Chapter 3 evaluates the matching of Key Performance Indicators (KPIs) and requirements. Section 3.1 assigning the defined KPIs to the functional requirements. Based on this assignment, section 3.2 evaluates the matching of KPIs and requirements for the cross-domain smart city pilots, while section 3.3 evaluates it for the domain-specific pilots.
Chapter 4 preparation of the ex-post Evaluation of the bIoTope Pilots. Section 4.1 prepares the evaluation by assessing the improvements based on user acceptance testing, whereas section 4.2 prepares the evaluation of cross-domain smart city pilots.
Chapter 5 is concluding the presented results. Further actions are described and explicit procedure documented in the previous evaluation deliverable D2.2.
1.3. Evaluated subjects within bIoTope
As mentioned before deliverable D2.6 is about the evaluation of technologies developed within bIoTope, mainly the Core Components. The Use Cases make use of the Core Components, in terms of combining them and adding individual services in order to realize Business Services being of value for the end-users. bIoTopes’ Core Components are presented in chapter 1.3.1 to provide sufficient information for the reader of deliverable D2.6. In addition, deliverable D2.6 is about the evaluation of bIoTopes’ Use Cases. Those Use Cases are based on the technologies developed within bIoTope and will demonstrate the improvements which can be achieved by bIoTope. Chapter 1.3.2 explains the Use Cases as being part of different Pilots in which the Test Scenarios will make the improvements assessable.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 10 30th of June 2017
1.3.1. bIoTopes’ Core Components
The overall architectural framework of bIoTopes’ SoS is based on an “Everything-as-a-Service” (XaaS) paradigm, where six key XaaS topics, considered as essential for any Software-as-a-Service (SoS) Platforms are created and tested:
IaaS - information-as-a-Service: This service includes a standards-based framework, around Open API standards, for the publication of information provided and consumable by heterogeneous Smart Objects.
KaaS - knowledge-as-a-Service: This service creates knowledge from data and information, making use of semantic web, data analytics and machine learning tools.
CoaaS - context-as-a-Service: This service, also referred to as “knowledge broker” invokes software components to discover, predict, validate and supply relevant ‘context(s)’ to applications and/or entities requesting it.
SeaaS - Security-as-a-Service: This includes offering of security related applications, such as e.g. anti-virus software, delivered externally. It includes development of new systems that are “context-sensitive”, i.e. able to self-adapt protection and privacy policies based on one or more “Context dimensions”.
BiaaS - Billing-as-a-Service: This service includes provision of core billing operations over a cloud infrastructure.
BDaaS - BigData-as-a-Service: Big data is an increasingly important paradigm that is driven by the pervasive diffusion and adoption of smart connected objects, mobile devices, social media tools, and other forms of information systems. The owner of big data may conduct data storage, management, and analysis and provide Web APIs for users to access the service-generated big data or the analyzed results; or may outsource the big data processing (or part of it) to a third party.
The IoTBnB - Marketplace and service catalogue: This service acts as a catalogue for discovering and consuming data. Inherent of this component is a discovery engine and a directory service to link several services to a new business process.
Each XaaS will result in one or more standards-based software core components, forming in sum the bIoTope XaaS Suite.
1.3.2. bIoTopes’ Pilots
bIoTope develops seven pilots that ensure innovation. Four pilots are so called Domain-specific Pilots to ensure innovation, dissemination and exploitation impact through the involved companies. One pilot (Smart Waste Management, see following bullet points) is provided by an associated partner of the consortium, which leads to special constellation regarding reporting and timing. In addition to these Domain-specific Pilots, bIoTope has established three Cross-domain Smart City Pilots. These Cross-domain Smart City Pilots serve as key proofs-of-concept of horizontal system interoperability across multiple application domains. Moreover, three cross-domain Smart City pilots in bIoTope offer the possibility to compare and evaluate variances in requirements and capabilities, as different legislation. The following paragraphs describe each pilot briefly.
Domain-specific Pilots
Smart Mobility: This Domain-specific Pilot involves consists of three Use Cases such as a Charging Station Selection Service, a Route Planning Service and an Electric Car Gear Service. Every Use Cases wants to make the use of electric vehicles more convenient. Therefore, the involved project partners will develop new Business Services based on cloud and IoT technologies.
Smart Building and Equipment: This Domain-specific Pilot consists of two Use Cases such as Smart Building Interaction with Smart Mobility (in particular, e-cars), and Smart Equipment (in particular, air handling units). The main objectives of the Pilot are to make home automation solutions more
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 11 30th of June 2017
common by developing solutions, which are based on a reduced complexity and can be implemented easily in order to reduce costs. Moreover, the Pilot wants to come up with open interfaces to avoid lock-in effects to just one manufacturer and to allow interaction with other smart devices.
Smart Air Quality: The smart air quality Domain-specific Pilot involves Use Cases such as identification of polluted areas, an overview about the pollution across the whole city for the city council, and a better experience to push bike riders. Crowd-sensing by citizens carrying mobile sensors can achieve the necessary granularity for realizing the planned Business Services.
Smart Waste Management*: This Domain-specific Pilot serves as test environment for a waste management system. Smart garbage bins (SGBs) around the urban area of St. Petersburg are equipped with sensors and actuators. Using micro-billing mechanisms, citizens are motivated to deliver waste to those SGBs. The SGBs help to monitor their status and offering a broad overview on the waste situation in the area of St. Petersburg that helps to properly plan waste management (truck routing, capacities, maintenance, etc.)
* The Smart Waste Management pilot is provided by the ITMO University from St. Petersburg, whom is an associated partner of the bIoTope project. Their access to the project started in month 11 and the Russian ministry realizes the founding for this partner. Therefore, a synchronization / alignment of created results is not feasible. A proper consideration of the pilot regarding the evaluation might be possible for the upcoming version of this deliverable planned for month 36 at the end of bIoTope. Nevertheless synergies between the pilots (especially similarities to Lyons bottle bank use case) are valuable, even though reporting on evaluation is not synced.
Cross-domain Smart City Pilots
Brussels – Capital Region: This Cross-domain Smart City Pilot serves as test environment for the Use Case for establishing safe roads for children and bikes’ green light. The Pilot will make use of already existing sensor networks and connect the data to realize the necessary business services.
FVH – Forum Virum Helsinki: This Cross-domain Smart City Pilot serves as test environment for the Use Case for an electrical car charging service using the IoT BnB concept. (private charging stations appended to public systems).
Greater Lyon: This Cross-domain Smart City Pilot serves as test environment for the Use Cases for bottle bank services and heat wave services. Improving the services for bottle bank emptying will need to implement new sensors into the bottles banks and make the data available to the city council and the citizens. Services based on hyperlocal weather data make use of already available sensor data for triggering actions like watering of plants.
For the pilot cases, deliverable D2.1 has described basic requirements, whereas deliverable D2.4 has further refined them in order to derive specifications. Each pilot has been specified by user stories, descriptive specifications, enumerated data sources and ArchiMate models, which serve as a formal notation of the intended functionalities that are revealed to the bIoTope ecosystem.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 12 30th of June 2017
2. Progress of development within bIoTope
Based on the approach described in chapter 2.1. of deliverable D2.2 “Evaluation Methodology for Pilots Validation”, the assessment of bIoTopes’ Core Components as fundamentals for an ecosystem and the technologies/ software components developed within bIoTopes’ pilots is described in the following chapters. Before presenting details of the evaluation of each Core Component and the technologies/ software components, the progress of development is described in section 2.1, in general. Section 2.2 is considering bIoTopes’ Core Components, whereas section 2.3 is looking on the technologies/ software components developed within bIoTopes’ cross-domain smart city pilots and chapter 2.4 is doing the same considering the domain-specific pilots. The assessment is done by calculating the Technology Readiness Level (TRL) according to the U.S. Government Accountability Office [U.S. Government Accountability Office] and the Integration Readiness Level (IRL) according to Sauser [Sauser].
2.1. Evaluation by assessing the Technology Readiness Level as well as the Integration
Readiness Level within bIoTope
The project management members uses the assessment based on TRL and IRL to monitor the progress of the development of bIoTopes’ Core Components as well as the different technologies/ software components developed within bIoTopes’ pilots. Based on the results of the evaluation the project management members take necessary actions, in order to achieve the objectives planned. Furthermore, the project management members will use the evaluation to review the technical approaches of bIoTope and refine bIoTopes’ Research and Technological Development (RTD) specifications, if needed.
Status of the development within bIoTope
During the intermediate evaluation of all bIoTope technologies, which are critical for bIoTopes’ success, all projects partners being responsible for the development of a critical technology executed a self-evaluation. Within the evaluation, the project partners evaluated each technology according to the status of the ongoing development as well as defined a target level, which they want to reach until the end of the project. The results of this evaluation process are presented in the subsequent chapters.
Before presenting the results, the following paragraphs describe the steps and actions of the evaluation process. The TRL process, as defined by the Government Accountability Office’s (GAO) [U.S. Government Accountability Office] was taken as reference, when assessing bIoTopes’ Core Components and the technologies/ software components. Along with the TRL assessment the IRL assessment was executed. Section 2.1.1 of deliverable D2.2 describes this process, which the whole consortium has approved. The description of the TRL and the IRL assessment in the following sections is based on this approach.
Step 1 Design the overall technology and integration maturity assessment strategy for the programme or project.
As already mentioned, section 2.1.1 of deliverable D2.2 describes the TRL assessment strategy, which has been chosen as a process template for bIoTope. These include the definitions of the TRLs according to the U.S. Government Accountability Office [U.S. Government Accountability Office], the TRL questionnaire according to the Air Force Research Laboratory [Air Force Research Laboratory] and the assessment process according to the Government Accountability Office’s (GAO) [U.S. Government Accountability Office]. Table 1 lists the TRL definitions once again. Based on the TRL questionnaire according to the Air Force Research Laboratory [Air Force Research Laboratory], an excel spreadsheet was designed to support the project partners being responsible for the assessment selecting the appropriate TRLs.
Regarding the IRL assessment strategy the assessment team has chosen the IRL definitions and their detailed descriptions according to Sauser [Sauser]. Table 2 gives an overview about these IRL descriptions. More information about the different IRL levels can be found in section 2.1.2 of deliverable D2.2. Based on the information provided by deliverable D2.2 and a spreadsheet designed by the assessment coordinator BIBA, the project partners selected suitable IRLs for each critical technology.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 13 30th of June 2017
Table 1: Technology Readiness Levels (TRL) definition [U.S. Government Accountability Office]
TRL – Technology Readiness
Level
Description
1 Basic principles observed
and reported
Lowest level of technology readiness. Scientific research begins to be
translated into applied research and development (R&D). Examples
might include paper studies of a technology's basic properties.
2 Technology concept and/or
application formulated
Invention begins. Once basic principles are observed, practical
applications can be invented. Applications are speculative, and there
may be no proof or detailed analysis to support the assumptions.
Examples are limited to analytic studies.
3
Analytical and experimental
critical function and/or
characteristic proof of
concept
Active R&D is initiated. This includes analytical studies and laboratory
studies to physically validate the analytical predictions of separate
elements of the technology. Examples include components that are not
yet integrated or representative.
4
Component and/or
breadboard validation in
laboratory environment
Basic technological components are integrated to establish that they
will work together. This is relatively “low fidelity” compared with the
eventual system. Examples include integration of “ad hoc” hardware in
the laboratory.
5
Component and/or
breadboard validation in
relevant environment
Fidelity of breadboard technology increases significantly. The basic
technological components are integrated with reasonably realistic
supporting elements so they can be tested in a simulated environment.
Examples include “high-fidelity” laboratory integration of components.
6
System/subsystem model or
prototype demonstration in
a relevant environment
Representative model or prototype system, which is well beyond that
of TRL 5, is tested in a relevant environment. Represents a major step
up in a technology's demonstrated readiness. Examples include testing
a prototype in a high-fidelity laboratory environment or in a simulated
operational environment.
7
System prototype
demonstration in an
operational environment.
Prototype near or at planned operational system. Represents a major
step up from TRL 6 by requiring demonstration of an actual system
prototype in an operational environment.
8
Actual system completed
and qualified through test
and demonstration.
Technology has been proven to work in its final form and under
expected conditions. In almost all cases, this TRL represents the end of
true system development. Examples include developmental test and
evaluation (DT&E) of the system in its intended operational
environment to determine if it meets design specifications.
9
Actual system proven
through successful
operations in an operational
environment.
Actual application of the technology in its final form and under
operational conditions, such as those encountered in operational test
and evaluation (OT&E). Examples include using the system under
operational conditions.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 14 30th of June 2017
Table 2: Integration Readiness Level (IRL) [Sauser]
IRL – IntegrationReadiness
Level Description
1
Semantic
An interface between technologies has been identified with sufficient
detail to allow characterization of the relationship.
2 There is some level of specificity to characterize the Interaction (i.e.
ability to influence) between technologies through their interface.
3 There is compatibility (i.e. common language) between technologies to
orderly and efficiently integrate and interact.
4
Syntactic
There is sufficient detail in the quality and assurance of the integration
between technologies.
5 There is sufficient control between technologies necessary to establish,
manage, and terminate the integration.
6 The integrating technologies can accept, translate, and structure
information for its intended application.
7 The integration of technologies has been verified and validated and an
acquisition/insertion decision can be made.
8
Pragmatic
Actual integration completed and qualified through test and
demonstration, in the system environment.
9 Integration is proven through successful operations in the system
environment.
Step 2 Define the individual TRL’s and IRL’s purpose; develop a TRL and IRL plan and assemble the assessment team.
The purpose of using the TRL assessment was to monitor the progress of the development of bIoTopes’ Core Components and the technologies/ software components, as described in chapter 3 of D2.2. In addition, the IRL assessment is aiming at measuring the integration of bIoTopes’ Core Components and the technologies/ software components into the SoS developed within bIoTope. The TRL and IRL assessment was used to evaluate the status of the TRL at the moment the Intermediate Evaluation was executed and to define the intended TRL and IRL at the end of the bIoTope project. According to Part B of the proposal, the assessment was coordinated by BIBA, being the leader of Task 2.E, and executed by the project partners, being responsible for the WPs 3 – 5 and the six Pilots.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 15 30th of June 2017
Step 3 Select critical technologies
The selection of critical technologies was done in two way. Selecting the Core Components has already been done within deliverable D2.4., Table 3 shows the selected Core Components. The leader of WPs 3, 4 and 5 are responsible for the evaluation of those technologies. Within the six pilots, use case specific technologies/ software components are developed. Within Table 4 the pilots and the technologies/ software components developed especially for the different use cases are displayed. The Pilot leaders selected the technologies/ software components, which are of great importance for demonstrating the benefits of bIoTope within the use cases.
Table 3: Core Components assessed by using TRL
Core Components
KaaS (WP4)
BiaaS (WP3)
IaaS (WP3)
SeaaS (WP3)
Open API for O-DF (WP5)
Dashboard Framework – UiaaS (WP5)
CoaaS (WP4)
BDaaS (WP4)
Table 4: Technologies/ Software Components assessed by using TRL
Pilots Technologies/ Software Components
Cross-domain Smart City Pilots
Brussels – Capital Region Android & IoS APP
Back end (O-MI Node)
API Manager (http://API.Brussels)
BigData Platform (Warp10)
FVH – Forum Virum Helsinki Back end (O-MI Node) for management
Charging pole, physical installation with software
Android app
Greater Lyon Data Platform
Lora Sensors
Domain-specific Pilots
Smart Mobility BMW Connected Drive
Smart Building and Equipment Aalto app to Mist
IoTBnB to Mist
Smart Air Quality Air Quality Monitoring and Prediction
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 16 30th of June 2017
Step 4 Evaluate critical technologies
The project partners, being responsible for the WPs 3, 4 and 5 and the six Pilots 1 , also executed the assessment. They used the definitions listed in Table 1 and the provided excel-based questionnaire to select the appropriate TRL of each critical component. In accordance with the assessment team at BIBA, the questionnaire was not used in a dogmatic way. The decision to interpret the TRL definitions in a pragmatic way was made by the people involved in the assessment team. The decision was motivated by the fact, that bIoTope is a research project. Moreover, the software developed within bIoTope is not as critical as technologies developed for rockets or other flying equipment, for which the NASA TRL assessments originally introduced. Instead, the use of the TRL assessment within bIoTope should guide the developers to improve the software tools used within bIoTope. The superior objective using TRL assessments was to help the companies being involved in bIoTope to offer innovative software solutions based on the innovations made within bIoTope. In order to sum it, the questionnaire was used in a pragmatic way by the assessment team and not every statement within the questionnaire had to be answered with “YES” to reach a specific TRL level.
The same project partners responsible for the TRL assessment executed the IRL assessment. They used the information provided in chapter 2.1.2 of deliverable D2.2 and the questionnaires provided for the documentation of the selected IRLs. Due to the fact, that not such a strict questionnaire was provided, the project partners had some more leeway for choosing the appropriate IRL. However, the basic idea using IRL for the Evaluation of the technologies is about the same as for using TRL. The assessment should guide the developers towards the objective of bIoTope to create a SoS, which enables the exchange of information across different domain-specific platforms. By designing bIoTopes’ SoS in the way, companies can integrate new Business Services into the SoS in a very efficient way, the system will enable those companies to offer new Business Services for Users.
Step 5 Prepare, coordinate and submit TRA reports
After filling the current and the intended TRL and IRL questionnaires for each critical technology, the partners, being responsible for the WPs 3 – 5 and the six Pilots, sent the questionnaires to BIBA being the coordinator of the whole assessment. BIBA generated tables, summarizing the results of the Evaluation. The tables show the current and the intended TRL and IRL Core Components and the technologies/ software components. Chapter 2.2 presents the results of the evaluation of the Core Components. The results of the evaluation of the technologies developed within the Cross-domain Smart City pilots can be found in chapter 2.3 and the results of the evaluation of the technologies developed within the Domain-specific Pilots can be found in chapter 2.4.
Step 6 Using TRA results and developing a Technology Maturation Plan (TMP)
By having identified the current TRL and the intended TRL at the end of bIoTope, the partners being responsible for the development of the different Core Components and the technologies/ software components as well as the project management knows, which steps need to be taken to reach bIoTopes’ objectives. Especially the questionnaires lead the path for further development by listing relevant steps to achieve the intended TRL and IRL. Aligned with the planned development documented and measured objectively by using the TRL as well as the IRL approach a detailed plan for the ongoing development of the Core Component and the technologies/ software components is documented in the deliverables of WPs 3 – 5 and the deliverables of the six Pilots. The documentation of the paths of future development within these deliverables is within the meaning of a Technology Maturation Plan (TMP). All relevant deliverables explaining the status of the development and the plan for future development and are presented in the tables showing the evaluation results in sections 2.2 - 2.4.
1 Seventh pilot (Smart Waste Management) is excluded in this report and reasoned in the previous chapter (section 1.3.2)
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 17 30th of June 2017
Summary of the development within bIoTope
As already mentioned before, the responsible project partners within the WPs 3 – 5 and the six Pilots filled out the spreadsheets to evaluate the status of each critical technology and to define the level they want to reach until the end of bIoTope. So the Intermediate Evaluation of bIoTopes’ technologies was completed in time and the results of the evaluation considering the Core Components are presented in 2.2. Section 2.3 and 2.4 present the results of the evaluation of technologies/ software components developed within bIoTopes’ Cross-domain Smart City Pilots and Domain-specific Pilots.
The following paragraphs present a summary of the evaluated TRL and IRL levels for bIoTopes’ Core Components and technologies/ software components. Based on the determined levels conclusions considering the status and the planned developed are presented.
As Table 5 shows, most of the current TRLs for bIoTopes’ Core Components are medium level values ranging from level 4 to level 7. The project partners intend to raise these values by roughly two levels on average, resulting in levels of 6 to 9. The TRL evaluation of the Billing-as-a-Service and the BigData-as-a-Service are outliers to the side of low levels, both in its current and its intended levels. These results are reflecting the generally lower technical maturity of Micro-Billing Services as well as Big Data Technologies. Within both topics, a lot of research is still ongoing, because they are of big importance for an economically successful IoT. Billing-as-a-Service is needed to monetize the Business Services and BigData-as-a-Service is of great importance to analyse the huge amount of data generated and made accessible by the IoT. As also shown in Table 5, the current IRLs assigned to bIoTopes’ core components are somewhat lower than the TRLs, ranging from levels 2 to 6. Due to the fact that bIoTopes’ main objective is to interconnect different technologies within the IoT, the IRLs rise slightly more than the TRLs. Most project partners plan to raise the IRLs of the technology they are responsible for by 2 or 3 levels. As a result, the most IRLs should reach the level 8, just two Core Components will probably be below, reaching an IRL of 4 and 6. In order to sum it up, the Core Components are on a good way to reach the necessary maturity levels in order to support the Use Case specific Business Service with the required IoT infrastructure.
As Table 6 up to Table 10 show, the current TRL of the technologies/ software components developed especially for bIoTopes’ Pilots are widely spread, reaching from level 2 to level 5. Even the intended levels at the end of bIoTope differ a lot. The reason for the wide spread TRLs are, that the technologies developed within the Use Cases are very specific developments; even their maturity differs a lot from each other. However, this is not causing issues for the whole project, because they are just of importance for the Use Cases. In the end, this will result in different time to market for the developed Business Services. The main objective of the Pilots to proof the applicability of bIoTopes’ approach is not at risk, this is also possible with prototypes having a very high maturity level. Regarding the IRLs, the situation is about the same. Across all technologies/ software components developed within bIoTopes’ Pilots the IRLs are widely spread. The current IRLs reach from level 1 to level 4 and the intended IRLs at the end of bIoTope reach from level 3 to level 7. Due to the fact, that the worst IRL will be level 3, it is ensured, that each Use Case is able to demonstrate in a prototypical way, how the interconnection of different technologies within bIoTope will work. According to the IRL definitions applied within the Intermediate Evaluation Level 3 represents the minimum required level to realize a successful integration.
The Intermediate Evaluation based on the TRL and IRL methodology, which quantified the maturity levels of the different technologies used within bIoTope, not just generated a report for the project management team, it also provided support to the project partners being responsible for technology development. By going through the TRL and IRL evaluation process, they were able to reflect the status of the work on the technologies at the end of the first half of the project and come up with ideas how they need to proceed to reach the intended status of their technology. By using TRL and IRL as Evaluation methodologies, the development of each technology was assessed on the one hand. On the other hand, interfaces across the different technologies to realize cross-domain Business Services were assessed as well. Especially the development of those interfaces in the context of the IoT across different ecosystems is in the focus of bIoTope. In order to reach bIoTopes’ objectives the different development teams within bIoTope do have now
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 18 30th of June 2017
a detailed overview about the status and have a kind of guideline, based on the TRL and IRL definitions, how to proceed to reach the intended levels.
In respect of the results of the TRL and IRL evaluation, the deliverables of WPs 3 – 5 considering the Core Components describe actions how to they will proceed will the further development. Considering the technologies/ software components developed by the Pilots, the actions for making each Use Case a successful demonstrator of bIoTopes’ objectives are described in the deliverables D6.1 – D6.14.
Decisions deduced from status of the development within bIoTope
Based on the approach deliverable D2.2 “Evaluation Methodology for Pilots validation” presents in section 2.1 the evaluation team and the project management office came up with the following decisions:
Monitoring of the IRLs should be in the focus of the evaluation team and the project management team in order to guarantee bIoTopes’ success. This is due to the fact, that bIoTopes’ main objectives are inspired by a cross-ecosystem data exchange and the number of levels between current IRLs and intended IRLs at the end of bIoTope are quite challenging.
TRL and IRL definitions should be in mind of the development teams to guide them during the development process to the intended maturity levels.
TRL and IRL progresss of each critical bIoTope technology need to be presented in front of the consortium members at each meeting of the technical team (initiated by WP 4 and accompanied by WP3 and 5) by using the provided spreadsheets to keep track of the development process and to compare the current maturity levels with the intended levels at the end of bIoTope.
2.2. Evaluation of Core components
The overall architectural framework of bIoTopes’ SoS is based on an “Everything-as-a-Service” (XaaS) paradigm, where six key XaaS topics, considered as essential for any SoS Platforms are created and tested. Each XaaS results in one or more standards-based software core components.
Table 5 lists the Technology Readiness Levels as well as the Integration Readiness Levels of the bIoTope core components, which have resulted from the evaluation of the stakeholders:
Table 5: TRL and IRL of bIoTopes’ Core Components
Core Components
Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress
KaaS (WP4) 7 8 6 8 D4.2 & D4.7
BiaaS (WP3) 2 4
3 6
(7 in best case)
D3.4
IaaS (WP3) 7 9 D3.2
SeaaS (WP3) 7 8 D3.1
Open API for O-DF (WP5)
4 6 5 8
UIaaS - Dashboard Framework
(WP5)
4 6 5 8 D5.3 & D5.6
CoaaS (WP4) 2 4 2 4 D4.5 & D4.9
BDaaS 9 9 6 8 D4.1
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 19 30th of June 2017
2.3. Evaluation of Technologies developed within the Cross-domain Smart City Pilots
In preparation of this deliverable, the technologies used within the cross-domain smart city pilots developed in bIoTope have been evaluated from user perspective. As part of the state of the art evaluation, stakeholders influencing and influenced by the pilot have been identified. The objectives of each pilot were named by the “Stakeholder Viewpoints” of each pilot within D2.1 “Ecosystem Stakeholder Requirements Report and Pilot Definition”. The identified stakeholders include both, stakeholders in general as well as potential end users. The evaluation itself was mainly prepared by members of each Pilot as well as the project partners being responsible for WP 3 - 5. The results are reported in the following sections 2.3.1-2.3.3.
2.3.1. Pilot: Brussels – Capital Region
Technology Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress
Android & IoS APP 9 9 2 7
D6.5 & D6.11
Back end (O-MI Node)
3 6 2 6
API Manager (http://API.Brussels)
8 9 8 9
BigData Platform (Warp10)
8 9 2 7
2.3.2. Pilot: FVH – Forum Virum Helsinki
Table 6: TRL and IRL of Pilot: Helsinki
Technology Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress
Back end (O-MI Node) for management
5 6 4 6
D6.6 & D6.12 Charging pole, physical installation with software
2 4 2 4
Android app 3 4 3 4
2.3.3. Pilot: Greater Lyon
Table 7: TRL and IRL of Pilot: Greater Lyon
Technology Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress
Data Platform 2 7 1 4 D6.4 & D6.10
Lora Sensors 1 5 1 3
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 20 30th of June 2017
2.4. Evaluation of Technologies developed within the Domain-specific Pilots
The technologies used within the domain-specific smart city pilots have been evaluated from user perspective as well. The identified stakeholders again include both, stakeholders in general as well as potential end users. The evaluation itself was again mainly prepared by project management members. The results are reported in the following sections 2.4.1-2.4.3.
2.4.1. Pilot: Smart Mobility
Table 8: TRL and IRL of Pilot: Smart Mobility
Technology Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress
Pilot as a whole
(BMW Connected Drive – MobiVoc – O-MI node, see D2.1 and D2.4)
2 4
(5 in best case)
2 6 D6.3 & D6.9
2.4.2. Pilot: Smart Building and Equipment
Table 9: TRL and IRL of Pilot: Smart Building and Equipment
Technology Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress
Aalto app to Mist (WP3)
Not applicable (just integration)
Not applicable (just integration)
1 7
D6.2 & D6.8 IoTBnB to Mist (WP3)
Not applicable (just integration)
Not applicable (just integration)
3 7
2.4.3. Pilot: Smart Air Quality
Table 10: TRL and IRL of Pilot: Smart Air Quality
Technology Current TRL Intended TRL Current IRL Intended IRL Detailed information about progress
Air Quality Monitoring and Prediction
2 3 2 3 D6.1 & D6.7
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 21 30th of June 2017
3. Matching of KPIs and Requirements
In order to ensure, that the Business Services developed within bIoTope address the needs of the intended users within each Use Case the project partners execute different assessments. These assessments are based on different criteria, motivated by the users’ needs as well as technical requirements. Finally, bIoTopes’ Business Services have to meet both kinds of criteria. On the one hand, they have to fulfil the users’ needs, motivating the customer to pay for the service. On the other hand, the services have to work reliably, otherwise the customer will not buy it neither. Therefore, the services have to meet the technological requirements.
3.1. Evaluation Methodology for Matching of KPIs and Requirements
During the preparation of deliverable D2.2, every Pilot defined KPIs for its Use Cases. Those KPIs are based on user stories defined for each bIoTope Use Case. Each KPI illustrates how the developed Business Services improve the daily life of Europe’s citizens. The KPIs detail bIoTopes’ superordinate objectives, defined by the consortium partners within bIoTopes’ proposal Part B of the Description of Actions. This kind of perspective Based on the proposal the KPIs were clustered as follows:
Planned Economical Improvements
Planned Societal Improvements
Planned Technical Improvements
Planned Environmental Improvements
The defined KPIs are a kind of non-functional requirements for the Business Services developed within bIoTope. They are expressing the planned improvements of the Use Case’s processes. Finally, the project partners will evaluate the KPIs in Test Scenarios after the implementation of the needed IoT infrastructure. Based on the results of the Test Scenarios, it can be said, whether bIoTope is a success from user perspective. Chapter 5 of deliverable D2.2. described the first version of the test scenarios. Within chapter 5 of the deliverable being on hand updates triggered by the Use Case Owners as well as bIoTopes’ development teams are shown. In order to make the KPIs tangible they should the SMART criteria listed in deliverable D2.2 and repeated here:
S – Specific: The criterion should make the need for an explicit goal so that there is no room for misinterpretation. A specific goal will usually answer the six 'W' questions: Who, What, Where, When, Which and Why.
M – Measurable: The next criterion means that goals must be measurable based on concrete, operational criteria. Usually, the entire goal statement is a measure for the project, but there are several short-term or smaller measurements built into the goal.
A – Achievable: This criterion should make sure that the goals are realistic and attainable. That means it must be possible to achieve the goal. However, they need to be stretching, ambitious and not too easy, because it will not be motivating.
R – Relevant: The fourth criterion Relevant means to choose goals that matter. Goals that are relevant for the employees, the team or the organization will receive needed support and drive the team forward.
T – Time-bound: The last criterion Time-bound is about setting the timeframe within which any identified objectives should be achieved. A commitment to a deadline helps a team focus their efforts towards completion of the goal.
In contrast to the approach for defining non-functional requirements based on KPIs, bIoTopes’ deliverable D2.1 defines formal requirements. bIoTopes’ software and hardware developer use those formal requirements to design each component according to the needs of the users as well as according to the needs
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 22 30th of June 2017
resulting from bIoTopes’ SoS approach. The Pilot owners developed those requirements based on ArchiMate models, being a technical standard from The Open Group based on the IEEE 1471 standard. In general, the standard provides methodologies to model business processes and IT infrastructures. Based on the available set of methodologies within ArchiMate bIoTopes’ project partners had decided on a number of so-called viewpoints. Each viewpoint is a selection of a relevant subset of ArchiMate’s methodologies offering the possibility to describe, analyse and visualise business processes and IT infrastructures from a certain perspective. For bIoTopes’ purposes, the project partners decided to use the following Viewpoints give an overview of high-level requirements regarding architectural aspects of each Pilot.
Organization Viewpoint
Layered Viewpoint
Stakeholder Viewpoint
Goal Realization Viewpoint
In order to ensure, that the KPIs, representing non-formal requirements are aligned with the formal requirements defined by the use of the ArchiMate modelling approach, the following evaluation procedure was executed. By using the structure shown in Table 11, all Pilots assigned the requirements presented originally in chapter 3 and 4 of deliverable D2.1 to the KPIs defined originally in chapter 5 of deliverable D2.2. Therefore, BIBA, as task leader of Task 2.E prefilled Excel spreadsheets with all KPIs defined. Afterwards, the responsible project partners had to go through the document and select the suitable requirements being needed to realize the KPI from an already prepared dropdown menu. It was possible to assign more than one requirement to a KPI. This is due to the fact, that even more features of the developed Business Service are needed to fulfil the KPI. At minimum one requirement needs to be assigned to a KPI. If no requirement is assigned to a particular KPI, the KPI is not addressed by the Business Service. This means the Business Service will not be able to achieve this KPI. If such a circumstance occurs, the project partners have to think about, whether the additional requirement is needed on order to be realised by the Business Service, or whether the KPI is not relevant any more. Therefore, the project partner executing this evaluation had to go through the KPIs and the requirements as well and update them if necessary. Chapter 5 shows the updates made to the KPIs during the Intermediate Evaluation.
Table 11: Structure of the spreadsheet for matching KPIs and Requirements
Summary of the Matching of KPIs and Requirements within bIoTope
As the following sections show, the software and hardware development process within bIoTopes’ is addressing the needs of the Use Case’s processes. Therefore, all KPIs are addressed by a minimum of one formal requirement. Hence, it will be possible to evaluate all KPIs within the Test Scenarios during bIoTopes’ final Evaluation in order to assess, the improvements which can be realized by bIoTope in Use Cases inspired by real world processes.
Pilot's Name Use Case's Name
Nr. KPI Group KPI Requirements
1 Requirement 1
Requirement 2
Requirement 3
Requirement 4
Requirement 5
Requirement 6
Requirement 7
Requirement 8
Requirement 9
Requirement 10
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 23 30th of June 2017
3.2. Evaluation of Cross-domain Smart City Pilots
In order to compare the KPIs and requirements of each Cross-domain Smart City Pilots developed in bIoTope have assigned to each other. Sections 3.2.1 - 3.2.3 present the evaluation results in detail.
3.2.1. Pilot: Brussels – Capital Region
Table 12: Matching of KPIs and Requirements for Pilot: Brussels – Capital Region
Nr. KPI Group KPI Requirements
1 Planned Economical Improvements Lowering the amount of accidents in school zones and on the itinerary of the children Req_Brussel_2_1 Predict peak hours Requirement 1
Req_Brussel_2_2 Predict traffic Requirement 2
Req_Brussel_2_3 Identi fy Safer Routes Requirement 3
Req_Brussel_3 Local ize s tudents Requirement 4
Req_Brussel_4 Reduce cars around schools Requirement 5
Req_Brussel_5 Reduce traffic speed Requirement 6
Req_Brussel_7 Promote safer & greener mobi l i ty Requirement 7
2 Shortening the average daily commute time of children and their parents Req_Brussel_2_1 Predict peak hours Requirement 1
Req_Brussel_2_2 Predict traffic Requirement 2
Req_Brussel_4 Reduce cars around schools Requirement 3
Req_Brussel_4_2 Inform drivers about school zone Requirement 4
Req_Brussel_6 Organize co-mobi l i ty Requirement 5
Req_Brussel_5_2 Alert drivers (via navigation systems)[1] Requirement 6
Req_Brussel_5_3 Alert drivers (via dynamic message panels ) Requirement 7
3 Lowering traffic jams in around schools Req_Brussel_2_1 Predict peak hours Requirement 1
Req_Brussel_2_2 Predict traffic Requirement 2
Req_Brussel_4 Reduce cars around schools Requirement 3
Req_Brussel_4_2 Inform drivers about school zone Requirement 4
Req_Brussel_6 Organize co-mobi l i ty Requirement 5
Req_Brussel_5_2 Alert drivers (via navigation systems)[1] Requirement 6
Req_Brussel_5_3 Alert drivers (via dynamic message panels ) Requirement 7
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 24 30th of June 2017
4 Planned Societal ImprovementsIncrease the effective security of school children, to be measured via perceived sense of
securityReq_Brussel_1 Col lect rea l -time data Requirement 1
Req_Brussel_2_3 Identi fy Safer Routes Requirement 2
Req_Brussel_1 Col lect data Requirement 3
Req_Brussel_1_2 Col lect data about s tudents Requirement 4
Req_Brussel_1_3 Col lect perceived sense of securi ty Requirement 5
5 Increase usage of other transport methods than the personal carReq_Brussel_6_2 Organize co-mobi l i ty for pedestrians (APP-
based)Requirement 1
Req_Brussel_6_3 Organize co-mobi l i ty for bikes (APP-based) Requirement 2
Req_Brussel_6_4 Organize co-mobi l i ty for publ ic transport (APP-
based)Requirement 3
Req_Brussel_7 Promote safer & greener mobi l i ty Requirement 4
6 Lowering the amount of accidents in school zones Req_Brussel_2_3 Identi fy Safer Routes Requirement 1
Req_Brussel_4 Reduce cars around schools Requirement 2
Req_Brussel_4_1 Deviate traffic Requirement 3
Req_Brussel_4_2 Inform drivers about school zone Requirement 4
Req_Brussel_5 Reduce traffic speed Requirement 5
Req_Brussel_5_1 Adapt dynamical ly speed traffic s igns Requirement 6
Req_Brussel_5_2 Alert drivers (via navigation systems)[1] Requirement 7
Req_Brussel_5_3 Alert drivers (via dynamic message panels ) Requirement 8
7 Planned Technical Improvements Bundling different sources (sensors) via O-DF/O-MI Req_Brussel_1_1 Col lect data from Telecom Operator Requirement 1
Req_Brussel_1_2 Col lect data about s tudents Requirement 2
Req_Brussel_1_3 Col lect perceived sense of securi ty Requirement 3
8Planned Environmental
ImprovementsRoute planning of children keeps into account overall pollution along possible itineraries Req_Brussel_7 Promote safer & greener mobi l i ty Requirement 1
Req_Brussel_7_1 Reward green behaviour Requirement 2
Req_Brussel_7_2 Inform about green mobi l i ty modes Requirement 3
Req_Brussel_7_3 Inform about safer and greener paths Requirement 4
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 25 30th of June 2017
3.2.2. Pilot: FVH – Forum Virum Helsinki
3.2.2.1. Use Case: Electrical Car charging service using IoT BnB concept
Table 13: Matching of KPIs and Requirements for Pilot: Forum Virium Helsinki - Electrical Car charging service using IoT BnB concept
Nr. KPI Group KPI Requirements
1 Planned Economical ImprovementsProvide equal possibilities to the companies, individuals, and organizations to participate in
creation of ecosystems for electrical carsReq_FVH_1 Publ ishing charging s tation Requirement 1
Req_FVH _1_1 Workflow to integrate new charging systems Requirement 2
Req_FVH_1_1_1 Deploying service information and metadata in
easy and s tandardized wayRequirement 3
Req_FVH_1_1_2 Connecting to the supplementary services Requirement 4
2 Reduce risks of monopolization of charging services by utilities companiesReq_FVH_1_1_1 Deploying service information and metadata in
easy and s tandardized wayRequirement 1
Req_FVH _1_1 Workflow to integrate new charging systems Requirement 2
3Tremendous increase in the amount of charging stations – less distances to charge, less
energy consumption Req_FVH_2 Providing Charging Service Requirement 1
Req_FVH_2_1 Integration to car navigation system or mobi le
devicesRequirement 2
Req_FVH_2_1_1 Integration to car navigation system Requirement 3
Req_FVH_2_1_2 Providing fi l tering poss ibi l i ties Requirement 4
4 Increase sales of electrical cars Req_FVH_1 Publ ishing charging s tation Requirement 1
Req_FVH_2 Providing Charging Service Requirement 2
5 Planned Societal ImprovementsEnables mass roll out of electrical cars in Northern countries that was somewhat slow due
to a lack of suitable charging stationsReq_FVH _1_1 Workflow to integrate new charging systems Requirement 1
Req_FVH_1_1_2 Connecting to the supplementary services Requirement 2
6 Less traffic due to better navigation and improved serviceReq_FVH_2_1 Integration to car navigation system or mobi le
devicesRequirement 1
Req_FVH_2_1_2 Providing fi l tering poss ibi l i ties Requirement 2
Req_FVH_2_1_3 User mobi le or web access Requirement 3
7 Public and private assets optimization Req_FVH _1_1 Workflow to integrate new charging systems Requirement 1
Req_FVH_1_1_2 Connecting to the supplementary services Requirement 2
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 26 30th of June 2017
8 Planned Technical Improvements Single entry point for all charging stations, easy and fast to searchReq_FVH_1_1_1 Deploying service information and metadata in
easy and s tandardized wayRequirement 1
Req_FVH_1_1_3 Service provis ioning Requirement 2
Req_FVH_2_1 Integration to car navigation system or mobi le
devicesRequirement 3
9 Easy to pay Req_FVH_2_2 Integration of payment system (optional ) Requirement 1
Req_FVH_2_3 Authentication methods Requirement 2
10 Seamless adding of new stations and suppliers without programming and integration effort Req_FVH _1_1 Workflow to integrate new charging systems Requirement 1
Req_FVH_1_1_1 Deploying service information and metadata in
easy and s tandardized wayRequirement 2
11Planned Environmental
ImprovementsLess CO2 emission due to increased amount of electrical vehicles Req_FVH_1 Publ ishing charging s tation Requirement 1
Req_FVH_2 Providing Charging Service Requirement 2
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 27 30th of June 2017
3.2.3. Pilot: Greater Lyon
3.2.3.1. Use Case: Bottle Banks
Table 14: Matching of KPIs and Requirements for Pilot: Greater Lyon - Use Case: Bottle Banks
Nr. KPI Group KPI Requirements
1 Planned Economical Improvements Collection costs lowered, due to less use of trucks Req_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1
Req_Lyon_1_3 Cons ider banks ful lness rate and other cri teria
in col lection schedul ingRequirement 2
Req_Lyon_1_4 Cons ider traffic predictions and real time traffic
in col lection schedul ing and road mapsRequirement 3
Req_Lyon_1_5 Providing generic API for bottle bank Information
exchangeRequirement 4
2Bottle bank costs lowered, due to less emptying operations on the banks and control of
bottle banks handlingReq_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1
Req_Lyon_1_3 Cons ider banks ful lness rate and other cri teria
in col lection schedul ingRequirement 2
Req_Lyon_1_5 Providing generic API for bottle bank Information
exchangeRequirement 3
Req_Lyon_1_6 Dai ly bottle banks dashboard Requirement 4
3Better equality between operators in tender procedures regarding the discarded glass
collectionReq_Lyon_1_6 Dai ly bottle banks dashboard Requirement 1
4 Planned Societal ImprovementsCitizens satisfied because they are informed of bottle bank availability, they can report
problems and most of the time bottle banks are not full
Req_Lyon_1_3 Cons ider banks ful lness rate and other cri teria
in col lection schedul ingRequirement 1
Req_Lyon_1_6 Dai ly bottle banks dashboard Requirement 2
5 Less trucks in the streets, less traffic Req_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1
Req_Lyon_1_4 Cons ider traffic predictions and real time traffic
in col lection schedul ing and road mapsRequirement 2
6 The metropolis has a better knowledge of the waste glass collection activity Req_Lyon_1_6 Dai ly bottle banks dashboard Requirement 1
7 Planned Technical Improvements
Interaction between a digital system owned and managed by the metropolis (the bottle
banks’ sensors network) and a digital system owned and managed by a subcontractor (the
trucks navigation system)
Req_Lyon_1_5 Providing generic API for bottle bank Information
exchangeRequirement 1
Req_Lyon_1_8 Data integration from datalyon.com Requirement 2
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 28 30th of June 2017
8Planned Environmental
ImprovementsLess noise in the streets Req_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1
Req_Lyon_1_4 Cons ider traffic predictions and real time traffic
in col lection schedul ing and road mapsRequirement 2
9 Less noise in residential areas, respect of “quiet” time slotsReq_Lyon_1_2 Cons ider res identia l area in col lection
schedul ingRequirement 1
10 Less pollution due to the reduction of the use of trucks through collections optimization Req_Lyon_1_1 Data for col lection schedul ing for subcontractor Requirement 1
Req_Lyon_1_3 Cons ider banks ful lness rate and other cri teria
in col lection schedul ingRequirement 2
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 29 30th of June 2017
3.2.3.2. Use Case: Heat Wave
Table 15: Matching of KPIs and Requirements for Pilot: Greater Lyon - Use Case: Heat Wave
Nr. KPI Group KPI Requirements
1 Planned Economical Improvements Less water consumption, use of rainwater Req_Lyon_2_10 Watering scheduler Requirement 1
Req_Lyon_2_9 Rainwater tank level and water debit
meterRequirement 2
Req_Lyon_2_6 Soi l humidity sensors Requirement 3
Req_Lyon_2_7 Evapotranspiration sensors Requirement 4
Req_Lyon_2_8 Weather forecast Requirement 5
Req_Lyon_2_4 Humidity and temperature sensors Requirement 6
2 Less trees in bad health to replace Req_Lyon_2_7 Evapotranspiration sensors Requirement 1
Req_Lyon_2_10 Watering scheduler Requirement 2
3 Planned Societal Improvements Increased citizen’s well-being Req_Lyon_2_10 Watering scheduler Requirement 1
Req_Lyon_2_3 Temperature information Requirement 2
Req_Lyon_2_1 Citizen porta l Requirement 3
4 A better life quality due to green spaces in good health Req_Lyon_2_10 Watering scheduler Requirement 1
5 The metropolis demonstrates its commitment in climate change adaptation Req_Lyon_2_1 Citizen porta l Requirement 1
Req_Lyon_2_10 Watering scheduler Requirement 2
6 Planned Technical ImprovementsSmart watering solution involving various sensors and actuators and external
data/informationReq_Lyon_2_10 Watering scheduler Requirement 1
7 The solution could be extended easily to others streets Req_Lyon_2_10 Watering scheduler Requirement 1
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 30 30th of June 2017
8Planned Environmental
ImprovementsHeat wave mitigation based on a natural solution Req_Lyon_2_6 Soi l humidity sensors Requirement 1
Req_Lyon_2_7 Evapotranspiration sensors Requirement 2
Req_Lyon_2_4 Humidity and temperature sensors Requirement 3
Req_Lyon_2_9 Rainwater tank level and water debit
meterRequirement 4
Req_Lyon_2_10 Watering scheduler Requirement 5
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 31 30th of June 2017
3.3. Evaluation of Domain-specific Pilots
Sections 3.3.1 - 3 show the results of the evaluation in order to ensure, that all KPIs are addressed by formal requirements leading to the development of appropriate features of the Business Services developed by each Pilot.
3.3.1. Pilot: Smart Mobility
Table 16: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Charging Station Selection Service
Nr. KPI Group KPI Requirements
1 Planned Economical ImprovementsLess energy consumption, resulting in less costs due to the reduction of saved kilometres
searching for appropriate charging stationsReq_SmartM_1 Parking Spot Discovery Requirement 1
Req_SmartM_3 Charging Station Database Requirement 2
Req_SmartM_5 Pre-Conditioning System Requirement 3
2Increased number of sold electric vehicles due to a more convenient search for appropriate
charging stationsReq_SmartM_3 Charging Station Database Requirement 1
Req_SmartM_3_1 Charging Stations Requirement 2
Req_SmartM_4 Uniform Payment System Requirement 3
3 Increased number of sold charging stations due to the increased number of electric vehicles Req_SmartM_3 Charging Station Database Requirement 1
Req_SmartM_3_1 Charging Stations Requirement 2
Req_SmartM_4 Uniform Payment System Requirement 3
4 Single payment method for at least 3 different charging station providers Req_SmartM_4 Uniform Payment System Requirement 1
5 Planned Societal Improvements Less traffic caused by searching for appropriate charging stations Req_SmartM_1 Parking Spot Discovery Requirement 1
Req_SmartM_2 Predict Traffic Si tuation Requirement 2
Req_SmartM_3 Charging Station Database Requirement 3
Req_SmartM_3_1 Charging Stations Requirement 4
Req_SmartM_6 Driver Identi fication Requirement 5
Req_SmartM_6_1 Driver Data Requirement 6
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 32 30th of June 2017
6 Better usage of space and existing charging stations Req_SmartM_1 Parking Spot Discovery Requirement 1
Req_SmartM_3 Charging Station Database Requirement 2
Req_SmartM_3 Charging Station Database Requirement 3
7 Planned Technical ImprovementsReduced complexity to integrate a charging station selection service into navigation
systems due to standardized data interfacesReq_SmartM_3 Charging Station Database Requirement 1
Req_SmartM_4 Uniform Payment System Requirement 2
Req_SmartM_6 Driver Identi fication Requirement 3
8 Reduced complexity to integrate different payment solutions into the charging stations Req_SmartM_4 Uniform Payment System Requirement 1
9Charging stations taken into account are from different providers (min 3; max (amount of
providers in area))Req_SmartM_3 Charging Station Database Requirement 1
Req_SmartM_3_1 Charging Stations Requirement 2
10Planned Environmental
ImprovementsLess carbon dioxide emissions due to the increased number of electric vehicles Req_SmartM_3 Charging Station Database Requirement 1
Req_SmartM_4 Uniform Payment System Requirement 2
Req_SmartM_5 Pre-Conditioning System Requirement 3
Req_SmartM_5_1 Cars & Bui ldings Requirement 4
11 Better air quality in the city due to the increased number of electric vehicles Req_SmartM_3 Charging Station Database Requirement 1
Req_SmartM_4 Uniform Payment System Requirement 2
Req_SmartM_5 Pre-Conditioning System Requirement 3
12Less resource consumption due to less traffic caused by the search for appropriate charging
stationsReq_SmartM_1 Parking Spot Discovery Requirement 1
Req_SmartM_2 Predict Traffic Si tuation Requirement 2
Req_SmartM_3 Charging Station Database Requirement 3
Req_SmartM_4 Uniform Payment System Requirement 4
Req_SmartM_5 Pre-Conditioning System Requirement 5
Req_SmartM_6 Driver Identi fication Requirement 6
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 33 30th of June 2017
Table 17: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Route Planning Service
Nr. KPI Group KPI Requirements
1 Planned Economical Improvements Better use of existing infrastructure Req_SmartM_1 Parking Spot Discovery Requirement 1
Req_SmartM_2 Predict Traffic Si tuation Requirement 2
Req_SmartM_3 Charging Station Database Requirement 3
2Increased number of sold vehicles / navigation systems due to more convenient route
planning services
Req_SmartM_1_1 Real Time Parking Information
SystemRequirement 1
Req_SmartM_2_1 Real Time System Requirement 2
Req_SmartM_6_1 Driver Data Requirement 3
3 Planned Societal Improvements Less unnecessary traffic in specific areas; e.g. school zones Req_SmartM_2 Predict Traffic Si tuation Requirement 1
4 Less stress impact on drivers due to unnecessary hazards Req_SmartM_2_1 Real Time System Requirement 1
5 Planned Technical Improvements Reduced effort to integrate new route-relevant data sources into the service Req_SmartM_1 Parking Spot Discovery Requirement 1
Req_SmartM_2 Predict Traffic Si tuation Requirement 2
Req_SmartM_3 Charging Station Database Requirement 3
Req_SmartM_6 Driver Identi fication Requirement 4
6 Higher accuracy of data can be used for additional service generationReq_SmartM_1_1 Real Time Parking Information
SystemRequirement 1
Req_SmartM_2_1 Real Time System Requirement 2
Req_SmartM_3_1 Charging Stations Requirement 3
Req_SmartM_6_1 Driver Data Requirement 4
7Planned Environmental
ImprovementsLess pollution due to reduced unnecessary driving and traffic jams Req_SmartM_1 Parking Spot Discovery Requirement 1
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 34 30th of June 2017
Table 18: Matching of KPIs and Requirements for Pilot: Smart Mobility - Use Case: Electric Car Gear Service
Nr. KPI Group KPI Requirements
1 Planned Economical ImprovementsEfficient use of energy, as the preconditioning is triggered based on the actual events and
no fixed time scheduleReq_SmartM_5 Pre-Conditioning System Requirement 1
Req_SmartM_5_1 Cars & Bui ldings Requirement 2
2 Reduced overall costs for batteries in particular hot or cold countries and urban mobility Req_SmartM_5 Pre-Conditioning System Requirement 1
3 Planned Societal ImprovementsElectric Vehicles have a bigger effective range, reducing the range anxiety and increasing
the attractiveness of electric vehicles in urban areasReq_SmartM_3 Charging Station Database Requirement 1
Req_SmartM_3_1 Charging Stations Requirement 2
Req_SmartM_5 Pre-Conditioning System Requirement 3
4 Future combination of Smart Home and Smart Mobility Req_SmartM_5_1 Cars & Bui ldings Requirement 1
5 Planned Technical Improvements Reducing the cost for creating cross-domain based services Req_SmartM_4 Uniform Payment System Requirement 1
Req_SmartM_6_1 Driver Data Requirement 2
6Planned Environmental
ImprovementsBetter usage of space and existing charging stations Req_SmartM_1 Parking Spot Discovery Requirement 1
Req_SmartM_1_1 Real Time Parking Information
SystemRequirement 2
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 35 30th of June 2017
3.3.2. Pilot: Smart Building and Equipment
Table 19: Matching of KPIs and Requirements for Pilot: Smart Building and Equipment - Use Case: Smart Building Interaction with Smart Mobility
Nr. KPI Group KPI Requirements
1 Planned Economical ImprovementsTremendous increase in the amount of parking (and power outlet) alternatives – less
distances to parking, less energy consumptionReq_SmartBE_1_1 A private parking space which can be shared Requirement 1
2Provide equal possibilities to the companies, individual, and organizations to participate in
creation of ecosystems for parking
Req_SmartBE_1 Parking spot as a shared space of a smart
bui ldingRequirement 1
Req_SmartBE_1_1 A private parking space which can be shared Requirement 2
Req_SmartBE_1_1_1 A service description publ icly access ible to
anyone within des ignated groupRequirement 3
Req_SmartBE_1_2 A communication infrastructure between
parking space provider and parking space consumer (car driver)Requirement 4
3 Improved competition between parking alternatives Req_SmartBE_1_1 A private parking space which can be shared Requirement 1
Req_SmartBE_1 Parking spot as a shared space of a smart
bui ldingRequirement 2
Req_SmartBE_1_1_1 A service description publ icly access ible to
anyone within des ignated groupRequirement 3
Req_SmartBE_1_2 A communication infrastructure between
parking space provider and parking space consumer (car driver)Requirement 4
4 Planned Societal Improvements Increased parking alternatives, more likely to find close to destinationReq_SmartBE_1 Parking spot as a shared space of a smart
bui ldingRequirement 1
Req_SmartBE_1_1 A private parking space which can be shared Requirement 2
Req_SmartBE_1_1_1 A service description publ icly access ible to
anyone within des ignated groupRequirement 3
Req_SmartBE_1_2 A communication infrastructure between
parking space provider and parking space consumer (car driver)Requirement 4
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 36 30th of June 2017
5 Less traffic due to more parking alternatives and less parking challengesReq_SmartBE_1 Parking spot as a shared space of a smart
bui ldingRequirement 1
Req_SmartBE_1_1 A private parking space which can be shared Requirement 2
Req_SmartBE_1_1_1 A service description publ icly access ible to
anyone within des ignated groupRequirement 3
Req_SmartBE_1_2 A communication infrastructure between
parking space provider and parking space consumer (car driver)Requirement 4
6 Public and private assets optimizationReq_SmartBE_1_2_1 Open standardized communication
channel between des ignated groupsRequirement 1
Req_SmartBE_1_2 A communication infrastructure between
parking space provider and parking space consumer (car driver)Requirement 2
7 Planned Technical Improvements Single entry point for all charging stations, easy and fast to searchReq_SmartBE_1_2_1 Open standardized communication
channel between des ignated groupsRequirement 1
Req_SmartBE_1_3_2 UI for discovering services Requirement 2
Req_SmartBE_1_3_4 Feature for selecting des ired service Requirement 3
8 Easy to pay Requirement 1
9 Seamless adding of new parking suppliers without programming and integration effortReq_SmartBE_1_5_1 Feature for providing a name for the
parking s lotRequirement 1
Req_SmartBE_1_5 Providing attributes for a parking spot Requirement 2
Req_SmartBE_1_5_4 Optional feature for describing additional
equipmentRequirement 3
Req_SmartBE_1_3_1 UI for describing a service Requirement 4
Req_SmartBE_1_2_1 Open standardized communication
channel between des ignated groupsRequirement 5
10Planned Environmental
ImprovementsImproved utilisation of parking lots
Req_SmartBE_1 Parking spot as a shared space of a smart
bui ldingRequirement 1
Req_SmartBE_1_1_1 A service description publ icly access ible to
anyone within des ignated groupRequirement 2
Req_SmartBE_1_2 A communication infrastructure between
parking space provider and parking space consumer (car driver)Requirement 3
Req_SmartBE_1_3_2 UI for discovering services Requirement 4
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 37 30th of June 2017
11Less CO2 emission due to reduced kilometres resulting originally from driving round and
searching for available parking lots
Req_SmartBE_1 Parking spot as a shared space of a smart
bui ldingRequirement 1
Req_SmartBE_1_1 A private parking space which can be shared Requirement 2
Req_SmartBE_1_1_1 A service description publ icly access ible to
anyone within des ignated groupRequirement 3
Req_SmartBE_1_2 A communication infrastructure between
parking space provider and parking space consumer (car driver)Requirement 4
Req_SmartBE_1_3_2 UI for discovering services Requirement 5
12Less resource consumption due to less traffic caused by the search for appropriate charging
stationsReq_SmartBE_1_3_4 Feature for selecting des ired service Requirement 1
Req_SmartBE_1_3_2 UI for discovering services Requirement 2
Req_SmartBE_1_3_3 Feature for fi l tering services (optional ) Requirement 3
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 38 30th of June 2017
Table 20: Matching of KPIs and Requirements for Pilot: Smart Building and Equipment - Use Case: Smart Equipment
Nr. KPI Group KPI Requirements
1 Planned Economical Improvements Reduced service costs Req_SmartBE_2_7_1 Abi l i ty to ad hoc connect new stakeholders Requirement 1
Req_SmartBE_2_7 Remote monitoring and control of a i r
handl ing unitRequirement 2
Req_SmartBE_2_6_2 Fi rmware release hosting service Requirement 3
Req_SmartBE_2_6_1 Fi rmware compatibi l i ty fi l tering Requirement 4
Req_SmartBE_2_6_6 Feature for flashing an a i r handl ing unit
with a new fi rmwareRequirement 5
2 Reduced energy costsReq_SmartBE_2 Air handl ing unit control in the field of smart
equipmentRequirement 1
3 Planned Societal Improvements Increased human productivity due to better indoor air qualityReq_SmartBE_2 Air handl ing unit control in the field of smart
equipmentRequirement 1
4 Healthier buildingsReq_SmartBE_2 Air handl ing unit control in the field of smart
equipmentRequirement 1
5 Planned Technical Improvements Smooth connectivity due to Mist/WiFi technology (O-MI/O-DF) Req_SmartBE_2_5 Ad hoc IoT connectivi ty Requirement 1
Req_SmartBE_2_1 Generic connectivi ty Requirement 2
Req_SmartBE_2_2_3 Flexible support for new sensor types ,
without hardware modificationsRequirement 3
Req_SmartBE_2_3_1 Support for remote (internet relayed), as
wel l as loca l (peer-to-peer) communicationRequirement 4
Req_SmartBE_2_3 Smartphone UI for interaction with a i r
handl ing unitRequirement 5
6Planned Environmental
ImprovementsLess energy consumption, less CO2 emissions
Req_SmartBE_2 Air handl ing unit control in the field of smart
equipmentRequirement 1
Req_SmartBE_2_3_2 Feature for visual is ing / communicating a i r
qual i ty information to end userRequirement 2
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 39 30th of June 2017
3.3.3. Pilot: Smart Air Quality
Table 21:Matching of KPIs and Requirements for Pilot: Smart Air Quality – Identify polluted areas
Smart Air Quality Identify polluted areas
Nr. KPI Group KPI Requirements
1 Planned Economical Improvements Complementary AQMP measurements Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_6 Compute Air Qual i ty Requirement 4
2 Revenue stream for local councils aware of and finding air polluters Req_SmartAQ_3 Uti l i ty Computation Requirement 1
Req_SmartAQ_1 Air Qual i ty Requirement 2
3 Planned Societal Improvements Increased trust of air quality measurements by EPA and local councils Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_4 Provide heat maps Requirement 2
Req_SmartAQ_2 Push Noti fications Requirement 3
Req_SmartAQ_6 Compute Air Qual i ty Requirement 4
4 Personalised heat maps for healthier environments Req_SmartAQ_2 Push Noti fications Requirement 1
Req_SmartAQ_4 Provide heat maps Requirement 2
5 Cleaner air for communities Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_2 Push Noti fications Requirement 2
Req_SmartAQ_5 Efficiency Requirement 3
6 Planned Technical Improvements Crowd-sourced complementary measurements of air quality Req_SmartAQ_1_2 pocket level Requirement 1
Req_SmartAQ_3 Uti l i ty Computation Requirement 2
Req_SmartAQ_4_2 Web Vers ion Requirement 3
7 Personalised awareness of air quality through a smartphone app Req_SmartAQ_2 Push Noti fications Requirement 1
Req_SmartAQ_3 Uti l i ty Computation Requirement 2
Req_SmartAQ_5 Efficiency Requirement 3
8 Evolutionary learning and improvement of air quality predictions Req_SmartAQ_6 Compute Air Qual i ty Requirement 1
9Planned Environmental
ImprovementsIncreased density of air quality measurements Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_3 Uti l i ty Computation Requirement 2
Req_SmartAQ_5 Efficiency Requirement 3
10 Cleaner air as the result of continuous monitoring and awareness of air quality Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_2 Push Noti fications Requirement 2
Req_SmartAQ_3 Uti l i ty Computation Requirement 3
Req_SmartAQ_4 Provide heat maps Requirement 4
Req_SmartAQ_5 Efficiency Requirement 5
Req_SmartAQ_6 Compute Air Qual i ty Requirement 10
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 40 30th of June 2017
Smart Air Quality Identify polluted areas
Nr. KPI Group KPI Requirements
1 Planned Economical Improvements Complementary AQMP measurements Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_6 Compute Air Qual i ty Requirement 4
2 Revenue stream for local councils aware of and finding air polluters Req_SmartAQ_3 Uti l i ty Computation Requirement 1
Req_SmartAQ_1 Air Qual i ty Requirement 2
3 Planned Societal Improvements Increased trust of air quality measurements by EPA and local councils Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_4 Provide heat maps Requirement 2
Req_SmartAQ_2 Push Noti fications Requirement 3
Req_SmartAQ_6 Compute Air Qual i ty Requirement 4
4 Personalised heat maps for healthier environments Req_SmartAQ_2 Push Noti fications Requirement 1
Req_SmartAQ_4 Provide heat maps Requirement 2
5 Cleaner air for communities Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_2 Push Noti fications Requirement 2
Req_SmartAQ_5 Efficiency Requirement 3
6 Planned Technical Improvements Crowd-sourced complementary measurements of air quality Req_SmartAQ_1_2 pocket level Requirement 1
Req_SmartAQ_3 Uti l i ty Computation Requirement 2
Req_SmartAQ_4_2 Web Vers ion Requirement 3
7 Personalised awareness of air quality through a smartphone app Req_SmartAQ_2 Push Noti fications Requirement 1
Req_SmartAQ_3 Uti l i ty Computation Requirement 2
Req_SmartAQ_5 Efficiency Requirement 3
8 Evolutionary learning and improvement of air quality predictions Req_SmartAQ_6 Compute Air Qual i ty Requirement 1
9Planned Environmental
ImprovementsIncreased density of air quality measurements Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_3 Uti l i ty Computation Requirement 2
Req_SmartAQ_5 Efficiency Requirement 3
10 Cleaner air as the result of continuous monitoring and awareness of air quality Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_2 Push Noti fications Requirement 2
Req_SmartAQ_3 Uti l i ty Computation Requirement 3
Req_SmartAQ_4 Provide heat maps Requirement 4
Req_SmartAQ_5 Efficiency Requirement 5
Req_SmartAQ_6 Compute Air Qual i ty Requirement 10
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 41 30th of June 2017
Table 22: Matching of KPIs and Requirements for Pilot: Smart Air Quality – Scenario Provide councils view of their city
Smart Air Quality Provide councils a view of their city
Nr. KPI Group KPI Requirements
1 Planned Economical Improvements Complementary AQMP measurements Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_6 Compute Air Qual i ty Requirement 2
2 Revenue stream for local councils aware of and fining air polluters Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_3 Uti l i ty Computation Requirement 2
3 Planned Societal Improvements Personalised heat maps for healthier environment Req_SmartAQ_2 Push Noti fications Requirement 1
Req_SmartAQ_4 Provide heat maps Requirement 2
4 Cleaner air for communities Req_SmartAQ_1 Air Qual i ty Requirement 1
Req_SmartAQ_2 Push Noti fications Requirement 2
Req_SmartAQ_5 Efficiency Requirement 3
5 Aware and informed about air quality city authorities Req_SmartAQ_2 Push Noti fications Requirement 1
Req_SmartAQ_4 Provide heat maps Requirement 2
6 Planned Technical Improvements Personalised dashboards Req_SmartAQ_4 Provide heat maps Requirement 1
7 Near real-time air quality monitoring alerting and predicting Req_SmartAQ_4 Provide heat maps Requirement 1
Req_SmartAQ_6 Compute Air Qual i ty Requirement 2
8Near real-time air quality monitoring
alerting and predictingIncreased density of air quality measurements Req_SmartAQ_6 Compute Air Qual i ty Requirement 1
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 42 30th of June 2017
4. Preparation of User Acceptance Testing for ex-post Evaluation within
bIoTope Pilots
Based on the approach described in section 2.2. of deliverable D2.1 “Evaluation Methodology for Pilots Validation”, the evaluation of the Business Services developed within bIoTope is described in the following sections. Before presenting details of the Evaluation of each Use Case, the preparation status of the ex-post Evaluation of bIoTopes’ Pilots is described in general in section 4.1. Section 4.2 addresses bIoTopes’ Cross-domain Smart City Pilots, whereas section 4.3 addresses bIoTopes’ Domain-specific Pilots. The methodology used to execute the ex-post Evaluation of bIoTopes’ Pilots is “User Acceptance Testing” according to Hambling and von Goethem [Hambling and von Goethem], as described in section 2.2 of deliverable D2.2.
4.1. Evaluation of bIoTope Pilots based on User Acceptance Testing
In order to ensure that the Business Services developed within bIoTope are addressing user’s KPIs, described in chapter 5 of deliverable D2.2 “Evaluation Methodology for Pilots Validation”, User Acceptance Tests are executed within bIoTope. The definition of the KPIs is based on user stories defined for each bIoTope Use Case. Those Use Cases exemplify how bIoTope can improve the daily life of Europe’s citizens. Improved process flows for each bIoTope Use Case are described in chapter 5 of deliverable D2.1. In addition, a number of different Key Performance Indicators (KPIs) expresses the improvements bIoTope wants achieve through the realization of several Business Services. Each KPI is contributing to the planned achievements described in bIoTopes’ Part B of the Description of Actions. For clustering the KPIs the following criteria were used:
Planned Economical Improvements
Planned Societal Improvements
Planned Technical Improvements
Planned Environmental Improvements
In fact, the improvements are not assessed directly through the KPIs within User Acceptance Tests. In order to evaluate non-functional requirements represented by the KPIs, based on the process perspective and functional requirements, based on the technical perspective, also the KPIs were translated into requirements. The technical perspective is needed as well to make sure, that the Business Services are running reliably. Nevertheless, it needs to be ensured, that all KPIs are represented by at least one requirement. For this reason, the Evaluation presented in chapter 3 of this deliverable was executed to show how the KPIs are represented by the requirements. In order to sum it up, the User Acceptance Tests are evaluating the requirements listed within D2.1 “Ecosystem Stakeholder Requirements Report and Pilot Definition”.
Regarding the procedure to evaluate the Business Services developed within bIoTope, the plan illustrated in Figure 1 was set up. The plan was presented in section 2.2 and 3 of deliverable D2.2 and approved by the whole consortium. As visualized, it was planned to finish the first User Acceptance Tests of already running Business Services until the Intermediate Evaluation, which is documented in this deliverable and illustrates the status of bIoTopes’ Pilots at month 18 of the project.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 43 30th of June 2017
Figure 1: Initial Evaluation Procedure of the Business Services within bIoTope according to D2.2 “Evaluation Methodology for
Pilots Validation”
User Acceptance Testing is possible only when the software development is almost completed and the software is running reliably. Until the Intermediate Evaluation bIoTopes’ Business Services are still under development. An insight into the progress within the deployment of the Business Services is provided by deliverable D6.1 up to D6.14. The deliverables dealing with the Domain-specific Pilots “Smart Mobility”, “Smart Building and Equipment” and “Smart Air Quality” will be submitted at month 18 of bIoTope. Deliverables presenting the current and future work within Cross-domain Smart City Pilots “Brussels – Capital Region”, “FVH – Forum Virum Helsinki” and “Greater Lyon” will be submitted at month 25 of bIoTope. Nevertheless, the developers of each Business Service have already executed some tests based on the User Stories described in chapter 5 of D2.2. Those tests are a regular part of agile software development, which is used for developing the Business Services within bIoTope.
Status of User Acceptance Testing for ex-post Evaluation within bIoTope Pilots
In the following, the status of User Acceptance Testing is described. Therefore, the process of User Acceptance Testing within bIoTope and the documents generated during this process are shown in Figure 2. Due to the fact, that all bIoTopes’ Business Services are still under development, no User Acceptance Tests could be implemented and reported until month 18 of bIoTope, when this deliverable is submitted. Nevertheless, the preparation of the test has already been going on. Therefore, a “Test Condition Matrix” and a “Test Schedule” have already been designed for every bIoTope Use Case. In the following the status of the preparation is described in detail.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 44 30th of June 2017
As shown in Figure 2 the “Set Up/ Plan of the Testing” as well as the “Design of the Tests” has been finished successfully. So almost all processes for preparing the initial tests have been terminated. Shaping, planning and preparing of the User Acceptance Tests was already finished and described in section 2.2.1 of D2.2 in order to finish the first step “Set Up/ Plan of the Testing”. During the first step, a rough schedule, presented in section 3.1 of D2.2 was set up, the procedure was defined and the associated tables were prepared and presented in section 2.2 of D2.2.
Figure 2: Status of User Acceptance Testing at month 18 of bIoTope [Hambling and von Goethem]
In addition, the “Design of the Tests” is finished and presented in this deliverable. Section 4.2 is presenting the documents regarding the Cross-domain Smart City Pilots, whereas section 4.3 is presenting the documents regarding the Domain-specific Pilots.
During the “Design of the Tests”, the test conditions were identified for each Use Case and documented in the
spreadsheet named “Test Condition Matrix”. Test conditions are the first step in the creation of tests. Test
conditions are logical statements that represent some aspects of a requirement. In other words, a single test
condition means "there is something that can be tested". All test conditions are addressing the requirements
described in deliverable D2.1. Each test condition represents a single feature of a Business Service that can be
assessed as either true or false. Consequently, the feature is correctly implemented if and only if all conditions
are true. Within the evaluation procedure, every requirement has to be tested at least once. For the
preparation of User Acceptance Testing, every bIoTope Use Case has filled out the “Test Condition Matrix”
illustrated in
Table 23.
In order to ensure that the test cases are of relevance for using the developed Business Services in a real world scenario they have been designed based on the real world processes of bIoTopes’ Use Cases described in chapter 4 and 5 of deliverable D2.2. The project partners being responsible for the evaluation of bIoTopes’
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 45 30th of June 2017
Pilots will use the state of the art processes as well as the test processes described chapter 4 and 5 of deliverable D2.2 to evaluate the KPIs. In order to execute bIoTopes’ evaluation procedures in the most efficient way, the User Acceptance Tests and the KPI assessment will be executed in the same Tests Scenario in parallel, if appropriate.
In order to execute a meaningful evaluation procedure of all test cases, it is necessary to set up a test schedule and a test logging mechanism. In Table 24 the structure of the “Test Schedule” for evaluating a Business Service is presented. The order of the tests should be according to the importance of the requirements, to make sure that the most important tests are examined during the evaluation procedure.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 46 30th of June 2017
Table 23: Example of a Test Condition Matrix
Table 24: Example of a Test Schedule
Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4 Condition 5
1. Creating
contactsXXX
Create a contact
category1.1
Create a contact
category on excels ior-
contacts
Create a contact
category page
Create a contact
category webs iteAdd a l ink to a contact
Check category status is
new
Create contacts
sub-category1.2
check main
contacts1.3
Sequence no. Requirement no. Service Test case no.Test case description Input data Expected output Test script ID Tester ProcessDate completed
[dd.mm.yy]
1. XXXGeneral
Functional i ty SecurityGF-S1.54
Veri fy that the team
member can log on
to excels ior
Team
member 1
logon
Excels ior home
page appearsSY1.5 Doe John
New
contact
approval
07.03.17
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 47 30th of June 2017
Summary of the status of User Acceptance Testing for ex-post Evaluation within bIoTope Pilots
As already mentioned before, the responsible project partners within the Pilots filled out the spreadsheets “Test Condition Matrix” and “Test Schedule” in the preparation of User Acceptance Testing within the Use Cases. Section 4.2 shows the spreadsheets considering the Cross-domain Smart City Pilots. Spreadsheets considering the Domain-specific Pilots are part of section 4.3. Because no Use Case is running at the time the Intermediate Evaluation is due, the Use Case Owners could not fill the spreadsheets for documentation and reporting of the User Acceptance Tests, yet. Nevertheless, working on the spreadsheets “Test Condition Matrix” and “Test Schedule” was most helpful for the Use Case Owners. That is why; the spreadsheets forced them to deliberate about the implementation of the features of the Business Services. Therefore, the spreadsheets supported the developers to consider all requirements belonging to the Use Case. The Use Case Owners did this by describing the variations of the Conditions, each one representing a feature of a Business Service, belonging to the different requirements.
For guaranteeing a smooth execution of the User Acceptance Tests, the Use Case Owners need to update the “Test Condition Matrix” and “Test Schedule” along with the ongoing development of the Business Services. The reason for this is that the spreadsheets are just representing the status of software development. Especially within a research and innovation project like bIoTope, software development is even more agile than during common software development processes. Therefore, updates of the “Test Condition Matrix” and the “Test Schedule” are necessary to guarantee that the tests cases fit to the developed software prototypes. This enables the Use Cases to adjust themselves according to the findings of bIoTope.
Based on the documents, listed at the bottom of Figure 2, it is obvious that work on the last four documents has not started, when bIoTopes’ Intermediate Evaluation was executed. The reason for that is: it only makes sense to proceed with the preparation of the User Acceptance Tests, when the software development is almost done and the “Test Condition Matrix” and “Test Schedule” are almost finished. When this moment will be reached, the Use Case Owners have to fill the “Test Script”. The “Test Script” is a corner stone for the execution of the User Acceptance Test within each Use Case.
Along with the execution of the User Acceptance Tests the rough “Test Log” and the detailed “Incident Report” can be filled out. Finally, a summary of each User Acceptance Test can be documented in a “Completion Report”.
Decisions deduced from status of User Acceptance Testing for ex-post Evaluation within bIoTope Pilots
Based on the approach deliverable D2.2 “Evaluation Methodology for Pilots validation” presents in section 2.2 and the delays regarding User Acceptance Testing for ex-post Evaluation within bIoTope Pilots the evaluation team and the project management office came up with the following decisions:
Timing of User Acceptance Testing in WP2 and Pilot Deployment in WP6 has to be harmonized more
intensively by defining the start dates and the end dates of User Acceptance testing in accordance with
the Pilots. Table 25 presents the dates agreed.
“Test Condition Matrix” und “Test Schedule” need to be detailed on time, based on the dates
presented in Table 25, in order to define one test case for each test condition within the “Test Script”
as well right before starting User Acceptance Testing.
Tester need to be selected and a detailed timetable needs to be agreed with the tester according to
the high-level schedule of each Pilot.
Documentation of User Acceptance Tests need to be send over to BIBA, being the task leader of Task
2.E, in order to ensure a proper evaluation of the project and for including the results into the final
deliverable D2.6 at month 36 of bIoTope.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 48 30th of June 2017
Completion reports need to be presented in front of the consortium members at the meeting
following each User Acceptance Test. At the end of the presentation the consortium has to discuss
about the incidents occurred and whether some additional actions are needed across the whole
consortium to solve them. Moreover, ongoing developments within bIoTope can consider the
evaluation results, to improve the impact of bIoTope.
Start and End dates of User Acceptance Tests have to be monitored with emphasis by BIBA as leader
of Task 2.E and by the project management office.
Table 25: Start dates and end dates for User Acceptance Testing within bIoTopes’ Pilots
Pilot Start of User Acceptance Testing End of User Acceptance Testing
Cross-domain Smart City Pilots
Brussels – Capital Region M23 M27
FVH – Forum Virum Helsinki M22 M27
Greater Lyon - Bottle Banks M23 / M25 (staged) M26 / M28 (staged)
Greater Lyon - Heat Wave Mitig. M21 / M28 (staged) M24 /M31 (staged)
Domain-specific Pilots
Smart Mobility M26 M33
Smart Building and Equipment M20 M23
Smart Air Quality M24 M28
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 49 30th of June 2017
4.2. Evaluation of Cross-domain Smart City Pilots
The following sections 4.2.1 – 3 show the “Test Condition Matrix” and the “Test Script” for the preparation of the User Acceptance Tests within each use cases of the cross-domain Smart City pilots.
4.2.1. Pilot: Brussels – Capital Region
Table 26: Test Condition Matrix for Pilot: Brussels – Use Case: Safety around School
Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3
1. Manage issuesReq_Brussel_2_
3Identi fy Safer Routes 1.1 Dashboard enables consultation of i ssues
2. Organise co-mobi l i ty
sess ionReq_Brussel_6 Organize co-mobi l i ty 2.1 Co-mobi l i ty groups are created
Co-mobi l i ty i tinerary are
created
Req_Brussel_6_
5_1Start a co-mobi l i ty sess ion 2.2
Other members of the group receives a
noti fication
Co-mobi l i ty sess ion is
vis ible on a map
Req_Brussel_6_
5_2Join a co-mobi l i ty sess ion 2.3 Co-mobi l i ty sess ion is vis ible on a map
Req_Brussel_6_
5_3Stop a co-mobi l i ty sess ion 2.4 Parents receives a noti fication (optional )
3. Inform students about
safety
Req_Brussel_1_
2Col lect data about s tudents 3.1
Req_Brussel_2_
3Identi fy Safer Routes 3.2
Waze information are col lected and
analysed
beMobi le information
are col lected and
Safety routes are
analysed
Req_Brussel_7_
3
Inform about safer and
greener paths3.3
Waze information are col lected and
analysed
beMobi le information
are col lected and
Safety routes are
analysed
4. Ca lculate safer
i tinerariesReq_Brussel_1 Col lect data 4.1
Waze information are col lected and
analysed
beMobi le information
are col lected and
Safety routes are
analysed
Req_Brussel_2_
2Predict traffic 4.2 Traffic data are analysed
5. Send/Receive
noti ficationsReq_Brussel_9 Manage noti fications 5.1
Noti fication are exchanged between
students , school management, parents
6. Manage Gamification
points
Req_Brussel_7_
1Reward green behaviour 6.1 Gamification points are ca lculated
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 50 30th of June 2017
Table 27: Test Schedule for Pilot: Brussels – Use Case: Safety around School
Sequence
no.Requirement no. Service Test case no. Test case description Input data Expected output Test script ID Tester Process
Date completed
[dd.mm.yy]
1Req_Brussel_8 -
Manage IssuesIssues Bxl -001
Create an issue via
mobi le app
Student
geolocal isationIssue is created
2Req_Brussel_6 -
Organize co-mobi l i tyIssues Bxl -002
Issues are vis ible
for s tudentsIssues
A comobi l i ty group is
created
3Req_Brussel_6 -
Organize co-mobi l i tyCo-mobi l i ty Bxl -002 Plan co-mobi l i ty
A comobi l i ty group and de
predefined i tinerary are
created
4Req_Brussel_6 -
Organize co-mobi l i tyCo-mobi l i ty Bxl -004
Start de co-mobi l i ty
sess ionGroup & i tinerary
Other group members
receives a noti fication
5Req_Brussel_6 -
Organize co-mobi l i tyCo-mobi l i ty Bxl -005
Accept invi tation to
join the sess ion
Co-mobi l i ty
Sess ion
Leader recieves a
noti fication
6Req_Brussel_6 -
Organize co-mobi l i tyCo-mobi l i ty Bxl -006
Decl ine invi tation
to join the sess ion
Co-mobi l i ty
Sess ion
Leader recieves a
noti fication
7Req_Brussel_6 -
Organize co-mobi l i tyCo-mobi l i ty Bxl -007
Start joining the
group
Co-mobi l i ty
Sess ion
8Req_Brussel_6 -
Organize co-mobi l i tyCo-mobi l i ty Bxl -008
Stop de co-mobi l i ty
sess ion
Co-mobi l i ty
Sess ion
Sess ion is s topped &
(optional ) parent receives a
noti fication
9
Req_Brussel_7 -
Promote safer &
greener mobi l i ty
Itineraries Bxl -009Calculate Safer
i tinerary
Issues , analysed
data
Influence i tineraries based
on data col lected on
Brussels Region
10
Req_Brussel_7 -
Promote safer &
greener mobi l i ty
Safety Informations Bxl -010Show Safety
information
Issues , analysed
data
Issues and safer routes are
vis ible on a map
11Req_Brussel_9 -
Manage noti ficationsNoti fication Bxl -011 Send a noti fication
Recipients receives
noti fication
12
Req_Brussel_7_1 -
Reward green
behaviour
Gamification Bxl -012
Receive
gamification points
for i ssue creation,
green mobiti ly
usage, …
Is sues ,
geolocal isationAdded Gamification points
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 51 30th of June 2017
4.2.2. Pilot: FVH – Forum Virum Helsinki
4.2.2.1. Use Case: EV Charging & Parking App
Table 28: Test Condition Matrix for Pilot: Forum Virium Helsinki - Use Case: EV Charging & Parking App
Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4 Condition 5
1. 1 Insta l l & run app 1.1Download the app from
web page
Insta l l the App on
AndroidRun the app View main page
2. 2 Login 1.2 Select login page View proper login page Type login credentia ls Veri fy successful login
3. 3 Map interface 1.3 View a map Pan & zoom mapChoose location on
map
2. 4View parking
spots2.1
Find parking spots on
mapChoose parking spot
View
avai lable/reserved
parking spots on map
View info on parking
spot
3. 5 Configure profi le 3.1 Choose profi le View profi le pageEnter EV socket
parametersView updated profi le
4. 6Find suitable
charging spots4.1
View map with profi le
compatible charging
spots
Choose target parking
spot
View map with target
selected
5. 7 Routing 5.1 Have target selected Cl ick on routing
View a route from
current pos i tion to
target on map
6. 8Approaching
target6.1 View a route to target
View updates on route
whi le moving
View stop routing,
target found message
Acknowledge parking
s tarted
View updated parking
spot s tatus on map
7. 9 Access charger 7.1View instructions on
access ing chargerOpen charger l id Connect charger to car Observe charging s tatus
8. 10 Stop charging 8.1 Select end charging Detach charger cable Close charger l id Observe charging s tatus
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 52 30th of June 2017
Table 29: Test Schedule for Pilot: Forum Virium Helsinki - Use Case: EV Charging & Parking App
Sequence no. Requirement no. Service Test case no.Test case description Input data Expected output Test script ID Tester ProcessDate completed
[dd.mm.yy]
1. App insta l l
Insta l lation of EV
Charging & parking
app
App-1Can user insta l l the
app succesfuly.apk fi le
App can be
s tarted on
device
30.06.17
2. Map view
Show avai lable
parking & charging
spots
Map-1
Are avai lable
parking & charging
spots vis ible and
understandable on
map to user
App
interaction
Map shows
spots30.06.17
3. Info viewShow info on
selected spotMap-2
Test that app can
show data on target
App
interaction
Info view on
target spot30.06.17
4. Target selection Define target spot Map-3
Test that target
selection on map
works
App
interaction
Map shows a
route to target30.06.17
5. Route fol lowing Routing Map-4Show advancement
on mapGPS Updated route 30.06.17
6. Charger permiss ionRelease charger for
usePole-1
User can open
charging pole with
reservation
credentia ls
App
interactionCharger opens 30.06.17
7. Charger charges Actual charging of EV Pole-2User connects cable
to car for charging
Measured
current on
pole
Charging s tatus
view30.06.17
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 53 30th of June 2017
4.2.3. Pilot: Greater Lyon
4.2.3.1. Use Case: Bottle Banks
Table 30: Test Condition Matrix for Pilot: Greater Lyon - Use Case: Bottle Banks
Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4 Condition 5
1.Col lection
schedul ing and
road maps
optimization
Req_Lyon_1_1
Generate the l i s t
of bottle banks to
empty
1.1
Bottle banks under the
ful lness l imit are
miss ing
Bottle banks above the
ful lness l imit are
present
Establ ish the
road maps1.2
Road map is relevant
regarding bottle banks
to empty
Road map cons ider the
truck capacity
Road map
adjusted during
col lection
1.3
New road map
proposed when a bottle
bank next to the truck i s
ful l
2.Cons ider
res identia l
area
Req_Lyon_1_2
Road map and
tour schedul ing
cons ider
forbidden time
s lots in
res identia l areas
2.1
Define a new
res identia l area ->
impact on the tour
schedul ing/road map
Modifiy a time-s lot for
a res ientia l area ->
impact on the tour
schedul ing/road map
3.Banks
ful lness rate,
weather
forecast,
coming events
Req_Lyon_1_3
The proposal of
l i s t of banks to
empty learn from
his torica l data on
ful lness rate
3.1
Fulness prediction
improved after 2 weeks
of operations
Fulness prediction
improved after 4 weeks
of operations
Fulness prediction
improved after 6 weeks
of operations
Weather forecast
i s cons idered in
the prediction of
ful lness
3.2
Fulness prediction
modified i f warm
temperature forecasted
Fulness prediction
modified i f ra inee
week-end coming
Coming events
are cons idered in
the prediction of
ful lness
3.3
Fulness prediction
modified i f an event i s
coming
Fulness prediction
modified i f an event i s
cancel led
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 54 30th of June 2017
4.Traffic
predictions
and real -time
traffic
Req_Lyon_1_4
Road maps are
modified in “real -
time” cons idering
the real - time
road events
4.1
Road map modified
depending on trafic
jams
One-hour traffic
prediction is used
to optimize the
road maps
4.2
Road map modified
depending on trafic
conditions in the next
hour
5.Optimization
in terms of
dis tances , time
Req_Lyon_1_5
Road maps are
ca lculated in
order to minimize
the dis tance
traveled by the
trucks and the
duration of each
road map
5.1
6. Information
exchange with
ci tizen porta l
Req_Lyon_1_7
Status of each
bottle bank is
ava i lable
6.1Consultation of ful l
bottle banks
Consultation of
ava i lable bottle banks
Consultation of oi t of
order bottle banks
Citizens can
parameter his
favouri te porta l
and a lerts
6.2Reception of an a lert i f
"my" bottle bank is ful l
Ci tizens can
report problems
on bottle banks
6.3Reporting of a ful l
bottle bank
Reporting of an out of
order bottle bank
7. Dai ly bottle
banks
dashboard and
a lerts
Req_Lyon_1_8Data for each
bottle bank7.1
Dashboard enables
consultation of
detai les and his torica l
data for each bottle
bank
Consol idated
data7.2
Dashboard enables
consultation of dayly,
monthly, yearly
consol idated data
Alerts 7.3Bottle banks about to
be ful l are vis ible
8. IoT BnB
compl ianceReq_Lyon_1_9
Bottle bank
sensors avai lable
in IoT BnB
8.1Access to data sensors
via IoT BnB / O-MI node
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 55 30th of June 2017
4.2.3.2. Use Case: Heat Wave
Table 31: Test Condition Matrix for Pilot: Greater Lyon - Use Case: Heat Wave
Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4 Condition 5
1.Citizen porta l Req_Lyon_2_1
Various cl imatic
informations are
avai lable on the
porta l
1.1
Access to the cl imatic
conditions at pi lot
location on the ci tizen
porta l
Ci tizen can post
on the porta l his
perception about
temperatures/hu
midity conditions
1.2
Creation of a
perception on the
porta l
2. Temperature
profi le Req_Lyon_2_2
Sensors data are
processed to
produce various
indicators
2.1
Temparature profi le i s
ava i lable in the pi lot
Dashboard and
relevant
3. Temperature
information Req_Lyon_2_3
Local current
temperature is
ava i lable at any
time publ ished
on O-MI node
3.1
Temperature
information is
ava i lable in the pi lot
Dashboard and
relevant
Sensors present in IoT
BnB
4. Sensors
network - Ai r
temperature
and humidity
sensors
network
Req_Lyon_2_4
Data avai lable
form
crowdsources and
metropol i tan
sensors
publ ished on O-
MI node
4.1Sensors present in IoT
BnB
5. Soi l humidity
sensorsReq_Lyon_2_6
Tens iomanager
data publ ished in
O-MI node
5.1Sensors present in IoT
BnB
6. Trees
evapotranspira
tion sensors
Req_Lyon_2_7
Pepipiaf data
publ ished in O-MI
node
6.1Sensors present in IoT
BnB
7. Weather
forecast Req_Lyon_2_8
Weather forecast
avai lable7.1
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 56 30th of June 2017
8. Rainwater
tank level Req_Lyon_2_9
Tank data
publ ished in O-MI
node
8.1Information present in
IoT BnB
9. Watering
scheduler Req_Lyon_2_10
Watering is
triggered
according to the
operating rules
9.1
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 57 30th of June 2017
4.3. Evaluation of Domain-specific Pilots
The “Test Condition Matrix” and the “Test Script” prepared for the Use Cases of the Domain-specific Pilots in order to execute the User Acceptance Tests are shown in the sections 4.3.1 - 3.
4.3.1. Pilot: Smart Mobility
Table 32: Test Condition Matrix for Pilot: Smart Mobility
Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4 Condition 5
Find suitable parking
locationsReq_SmartM_1
Connect to discovery
service
Lis t a l l parking locations in
area
Retrieve detai led information
about avai lable parking
poss ibi l i ties
Req_SmartM_1_1Connect to discovery
service
Lis t a l l parking locations in
area
Query additional
information for the parking
s tations
Information is retrieved
as l inked data
Information contains
payment options
Find information sources for
traffic dataReq_SmartM_2
Connect to discovery
service
Lis t traffic data sources in
areaRetrieve s tatic data
Get rea l -time information
about trafficReq_SmartM_2_1
Connect to discovery
service
Lis t traffic data sources in
area
Retrieve l ive data
described as l inked data
data is geo-referenced
us ing a known and
suitable format
Find suitable charging
s tationsReq_SmartM_3
Connect to discovery
service
Lis t a l l charging s tations in
area
Retrieve s tatic data about
charging s tations
Charging s tation avai labi l i ty Req_SmartM_3_1Connect to discovery
service
Lis t a l l charging s tations in
area
Retrieve s tatic data about
charging s tations
Retrieve avai labi l i ty of
charging s tations
Uniform Payment system
avai lableReq_SmartM_4
Identi fy poss ible
payment methods for
enti ties
Perform a secure transaction
(payment process)
Pre-Conditioning of vehicle Req_SmartM_5
Secured service for pre-
conditioning offered by
vehicle
Pre-Conditioning service can
be triggered by authorized
person
Combine Smart Home and
VehicleReq_SmartM_5_1
Pre-Conditioning
service can be triggered
by smart home (m2m)
Driver identi fication Req_SmartM_6
Secure identi ty
management in
ecosystem avai lable
Make use of driver related
dataReq_SmartM_6_1
Secure identi ty
management in
ecosystem avai lable
Relevant mobi l i ty data can
be used for driver-related
services
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 58 30th of June 2017
Table 33: Test Schedule for Pilot: Smart Mobility
Sequence no. Requirement no. Service Test case no.Test case description Input data Expected output Test script ID Tester ProcessDate completed
[dd.mm.yy]
1. Req_SmartM_1Find suitable
parking locations
Lis ted Parking spots can be
used for parking services
2. Req_SmartM_2
Find information
sources for traffic
data
Traffic sources can be used
for traffic and routing
systems
3. Req_SmartM_3Find suitable
charging s tations
Retrieved charging s tations
can be used for charging
the electric car
4. Req_SmartM_4Uniform Payment
system avai lablePayment poss ible (/done)
5. Req_SmartM_5Pre-Conditioning of
vehicle
Car s tarts preconditioning
on-demand
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 59 30th of June 2017
4.3.2. Pilot: Smart Building and Equipment
Table 34: Test Condition Matrix for Pilot: Smart Building and Equipment
Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4 Condition 5
Device commiss ioning to identi ty-
based network
Req_SmartBE_2_3
_1
Commiss ion Mist device to
identi ty-based network
Connect to device us ing identi ty-
based communication over mobi le
network
Publ ish expert to directoryReq_SmartBE_1_3
_2
Create identi ty for the
expert
Publ ish expertise identi ty (with
description) to a cata logue service
Expert discovery from directoryReq_SmartBE_2_7
_3
Lis t experts (discover their
identi ties )
Connect to expert (us ing identi ty-
based communication)
Expert access to device Req_SmartBE_2_7 Send request to expertExpert achieves connection for
monitoring and control
Publ ish parking service to directoryReq_SmartBE_1_3
_2
Publ ish Mist device to
directory
Discover parking service from
directory
Req_SmartBE_1_3
_2
Lis t publ ished services ,
discover their identi ties
Connect to parking service Req_SmartBE_1_2
Connect to discovered
parking service provider
(us ing identi ty-based
communication)
Send activate/deactive commands
to parking serviceReq_SmartBE_2_7 Start parking meter Stop parking meter
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 60 30th of June 2017
Table 35: Test Schedule for Pilot: Smart Building and Equipment
Sequence no. Requirement no. Service Test case no. Test case description Input data Expected output Test script ID Tester Process
Date
completed
[dd.mm.yy]
1.Req_SmartBE_2_3
_1
Device commiss ioning
to identi ty-based
network
Device has generated i tsel f an
identi ty, i s commiss ioned to the
network, and the device admin role
i s cla imed by another identi ty
01.06.17
2.Req_SmartBE_1_3
_2
Publ ish expert to
directory
An expert description a long with
his identi ty i s included in a service
catalogue
01.08.17
3.Req_SmartBE_2_7
_3
Expert discovery from
directory
An app is able to l i s t experts (and
their identi ties ) from a service
catalogue
01.09.17
4. Req_SmartBE_2_7 Expert access to device
Expert i s able to establ ish identi ty-
based communication to the
device
01.10.17
5.Req_SmartBE_1_3
_2
Publ ish parking service
to directory
A parking service description a long
with the service provider identi ty i s
included in a service catalogue
01.08.17
6.Req_SmartBE_1_3
_2
Discover parking
service from directory
An app is able to l i s t parking
services from a service catalogue01.09.17
7. Req_SmartBE_1_2Connect to parking
service
An identi ty-based communication
to the parking service provider i s
achieved
01.10.17
8. Req_SmartBE_2_7
Send activate/deactive
commands to parking
service
The customer has an abi l i ty to
activate / deactive the parking
service
01.11.17
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 61 30th of June 2017
4.3.3. Pilot: Smart Air Quality
Table 36: Test Condition Matrix for Pilot: Smart Air Quality
Spec/ Design ref Requirement no. Requirement name Reference Condition 1 Condition 2 Condition 3 Condition 4
1. Access ing Sensor Network Req_SmartAQ_1 Air Qual i ty 1.1
Req_SmartAQ_1_1 Macro Level 1.2 Lis t ava i lable Sensorsidenti fication and
loca l isation of sensorsclass i fy sensors by region
Req_SmartAQ_1_2 pocket level 1.3 Lis t ava i lable Sensorsidenti fication and
loca l isation of sensors
class i fy sensors by type (mobi le
objects )
Update rate and
connectivi ty of sensors
2. Push Noti fications Req_SmartAQ_2 Push Noti fications 2.1Val idate constra ints (IFTTT-l ike
express ions)
identi fy (user) location /
condition
Receive noti fication
Req_SmartAQ_2_1 O-MI/O-DF 2.2val idate compl iance to O-MI / O-
DFIdenti fy users current context
3. User integration to
publ ish dataReq_SmartAQ_3 Uti l i ty Computation 3.1
Enable users to publ ish data
(crowd sourcing)
Integration of micro-
payments
4. Create heat maps Req_SmartAQ_4 Provide heat maps 4.1
Heat maps with di fferent degree
of detai l (zooming between
macro and micro view)
Req_SmartAQ_4_1 Mobi le vers ion 4.2
communicating with the sensor to
capture and contribute a i r qual i ty
data (see ref. 1.3)
a l lows user to personal ise
the a i r qual i ty subscriptions
(see ref. 2.1)
track user's context and match
subscriptions on the move (see
2.2)
provide visual isation
components to present
a i r qual i ty data to the
user
Req_SmartAQ_4_2 Web Vers ion 4.3 Real -time a ir qual i ty provis ioning
5. Ensure efficency Req_SmartAQ_5 Efficiency
Req_SmartAQ_5_1 Energy 5.1control l the duty cycle of the
sensor
use a publ ish subscribe
approach where data i s only
contributed i f there is a need
use mobi le device onboard
analytics and s torage
Req_SmartAQ_5_2 Communication 5.2us ing rea l -time data reduction on
the mobi le device
provide loca l s torage and
query process ing instead of
sending a l l the data
reduce the communication
between the external sensor and
the mobi le device by restricting
i t to certa in times when data i s
required
Req_SmartAQ_5_3 Duty Cycle 5.3 us ing duty cycle control
use context via mobi le
devicce to determine when to
request data from the sensor
Req_SmartAQ_5_4 Cost 5.4accuracy of the data provided by
the service
determine cost of hosting the
AQMP service (cost of external
sensor and uti l i ty)
6. Compute Air Qual i ty Req_SmartAQ_6 Compute Air Qual i ty 6.1use of models to estimate a i r
qual i ty
use of predictive models to
determine a i r qual i ty
Req_SmartAQ_6_1 Data Fus ion 6.2
efficiently and effectively fuse
data to del iver greater accuracy to
users
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 62 30th of June 2017
Table 37: Test Schedule for Pilot: Smart Air Quality
2
2 AQMP – Air Quality Monitor Provisioning
Sequence
no.Requirement no. Service Test case no.Test case description Input data Expected output Test script ID Tester Process
Date
completed
[dd.mm.yy]
1. Req_SmartAQ_11. Acces Sensor
NetworkAQMP.1
Air qual i ty data i s gathered from
external sensors in the field
Triggering sensors (e.g.
method ca l l )
Sensor va lues from static and
mobi le enti ties (might be
synthes ised fi rs t)
AQMP.script.1
2. Req_SmartAQ_2 2. Push Noti fications AQMP.2Noti fications are released /
triggeredn by constra ints
IFTTT-l ike constra int as
user input
O-MI / O-DF compl iant
noti ficationsAQMP.script.2
3. Req_SmartAQ_33. User integration to
publ ish dataAQMP.3
Pipe external sensor data to the
bIoTope ecosystem
user enables data
release
publ ication of (data) service on
IoTBnB marketplace (including
micropayment)
AQMP.script.3
4. Req_SmartAQ_4 4. Create heat maps AQMP.4Data shown in heatmaps with zoom
functional i ty
data from sensors (macro
and micro sca le)
visual ize heatmap on mobi le and
web vers ion of AQMPAQMP.script.4
5. Req_SmartAQ_5 5. Ensure efficency AQMP.5
Monitor and log (or show) duty cycle
of external sensor and mobi le
device, communication properties
and accuracy
device s tatus
monitoring of devices ' s tatus
(mobi le device and external
sensors )
AQMP.script.5
6. Req_SmartAQ_66. Compute Air
Qual i tyAQMP.6
Stress -test computation of models to
estimate and predict a i r qual i ty and
enhancement in accuracy
various (heterogenous)
sensor data
enriched sensor data with better
accuracyAQMP.script.6
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 63 30th of June 2017
4.3.4. Preparation of the KPI Test Scenarios for ex-post Evaluation of the bIoTope Pilots
Regular evaluation of a complex international research project is of great importance for the success of the project itself and the research and innovation programme as a whole. Based on the evaluation results, the responsible people can adjust or trigger actions to achieve the planned objectives. Of great importance for the evaluation of projects are well defined KPIS. Especially for research projects, considering the IoT, like bIoTope, demonstrating the success of the developed hardware and software solutions is of great importance. Therefore, Part B of bIoTopes’ proposal already defined the following KPIs:
30% increased length of battery operation of electric car on a daily basis – Smart Mobility Pilot
20% increased electric car battery life time – Smart Mobility Pilot
20% energy-reduction in smart building pilot – Smart Building and Equipment Pilot
25% enhanced predicted failure rate regarding HVAC equipment – Smart Building Pilot
50% enhanced early pollution detection – Smart Air Quality Pilot
80% time reduction for creating new services based on available information sources coming from different application domains – Cross-Domain Smart City Pilots
≥50% acceptance of services by citizens (collecting user feedback with a specific UIaaS widget)
Based on these superordinate KPIs the bIoTopes’ Pilots defined individual KPIs for each Use Case. Chapter 5 of deliverable D2.2 presents all KPIs being of importance for the bIoTope Use Cases. For evaluating the KPIs the project partners being responsible for each Pilot defined Test Scenarios. Those Test Scenarios are also described within chapter 5 of deliverable D2.2 along with the KPIs.
Due to the fact, that deliverable D2.2 was already prepared at M8 of the project, some necessary adjustments concerning the whole project and some Pilots have become necessary, since. Such adjustments are obviously when executing complex international projects such as bIoTope. Based on those adjustments it is necessary to update KPIs and the Test Scenarios as well. In the majority of cases, the reason for necessary updates are findings made during the progress of the project, which resulted in changed requirements.
Also within bIoTope the KPIs and the associated Test Scenarios need regular updates. This needs to be done by each project partner being responsible for a Pilot. The responsible project partners have to ensure, that the Test Scenario and the KPIs for the ex-post Evaluation are still in line with the Use Case’s requirements as well as the findings from software and hardware developments. In order to enable the evaluation team and the project management office to monitor the preparation of the ex-post Evaluation of bIoTopes’ KPIs, Table 38 summarizes the confirmations from the Pilots regarding the updates of the KPIs and the Test Scenarios during the Intermediate Evaluation.
Table 38: Progress of the preparation for the KPI Test Scenarios for ex-post Evaluation
Pilot KPIs confirmed [Y/N] Test Scenarios confirmed [Y/N]
Cross-domain Smart City Pilots
Brussels – Capital Region N Y
FVH – Forum Virum Helsinki N Y
Greater Lyon N Y
Domain-specific Pilots
Smart Mobility N Y
Smart Building and Equipment Y - updated Y - updated
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 64 30th of June 2017
Smart Air Quality N Y (to be updated)
Relevant updates of the KPIs and the Test Scenarios will be send to the evaluation team and the project management office during adaptations on use cases and demonstrators caused by on-going development and final documented in the evaluation deliverable D2.9.
In order to ensure that the projects partners being responsible for the evaluation of the project can execute the assessment in an efficient way and to have meaningful evaluation results, the Evaluation needs to be prepared. Therefore, the Evaluation hast to be planned and coordinated across the whole project. Considering the KPI Test Scenarios, being part of the project’s evaluation, Table 39 shows the start dates and the end dates of the execution of the tests. As already expressed in chapter 4, the User Acceptance Tests and the KPI-based ex-post Evaluation of the bIoTope Pilots can be executed at the same time. Hence, the project partners being responsible for the execution of the evaluation should align the start and end dates of the User Acceptance Tests, shown in Table 25, and the KPI Tests Scenarios shown inTable 39.
Table 39: Start dates and end dates for executing the KPI Test Scenarios for ex-post Evaluation within bIoTopes’ Pilots
Pilot Start of KPI Tests Scenarios End of KPI Tests Scenarios
Cross-domain Smart City Pilots
Brussels – Capital Region M27 M31
FVH – Forum Virum Helsinki
Greater Lyon – Bottle Banks M27 M 31
Greater Lyon – Heat Wave Mitig. M29 M33
Domain-specific Pilots
Smart Mobility
Smart Building and Equipment
Smart Air Quality
Smart Waste Management
In order to ensure an efficient evaluation of all bIoTope Use Cases, the assessment team developed a template for the KPI-based ex-post Evaluation of each Use Case within the Pilots. When executing the Test Scenarios the project partners should use the EXCEL-based template for documentation and reporting. The developers will also use the template to adapt the Business Services, if the objectives are not fully met during the first test run. Based on the template the evaluation results will be included in deliverable D2.9 in M36 of bIoTope.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 65 30th of June 2017
Table 40: Template for the KPI-based ex-post Evaluation of the bIoTope Pilots
Use
Case no.Use Case KPI Tester [name]
Date raised
[dd.mm.yy]
KPI Test
Scenario ID
[A,B,C]
KPI Objective archieved [Y, N, %] Issue resolution status Assigned to [name]
1 Safe Roads for ChildrenLowering the amount of accidents in school zones and on
the i tinerary of the chi ldren
Shortening the average dai ly commute time of chi ldren and
their parents
Lowering traffic jams in around schools
Increase the effective securi ty of school chi ldren, to be
measured via perceived sense of securi ty
Increase usage of other transport methods than the
personal car
Lowering the amount of accidents in school zones
Bundl ing di fferent sources (sensors ) via O-DF/O-MI
Route planning of chi ldren keeps into account overa l l
pol lution a long poss ible i tineraries
KPI Test
Scenario
ID
A
B
C
Test Scenario Description
A survey with chi ldren and their parents i s foreseen to determine the s i tuation prior to the project and 3 months after the project has been deployed. This survey wi l l evaluate chi ldren ́s and their parents ’
perception on the fol lowing:
Sense of (in)securi ty of the chi ldren and their parents
Commute time of chi ldren separately
The need of parents to accompany chi ldren on chi ldren 's commute
Mode of transport (including chi ldren separately and/or together with the parents )
Shi fts in any of the above
KPI Test Log
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 66 30th of June 2017
5. Conclusion
Initially the development progress within bIoTope was made by assessing the Technology Readiness Level (TRL) and the Integration Readiness Level (IRL), of the intended technologies in the ecosystem, namely the core components (services), the cross-domain city pilots and the domain-specific pilots.
Summarized, current cross-pilot TRL assessment turned out legitimately high, while IRL provides more innovation potential. This is actually the representation of a problem statement, bIoTope addresses in the current landscape of IoT, whereas already existing smart objects (products with high TRL) need to be orchestrated in a way that seamless information flow between vertical silos can be achieved for seamless data exchange and creation of new services. The problem representation consequently is directly characterized in the IRL indications of the pilots and core components (XaaS-connected technologies).
Previously recorded KPIs and requirements are mapped to each other, which shows the alignment of planned use cases and evaluation strategy. This results in a stable preparation of user acceptance testing for a ex-post evaluation later in the project. Anyhow minor adaptations are necessarily not excluded due to changes in the pilots. The evaluation methodology is used to fundamentally reason such decisions and re-arrangements to ensure a positive project result in a holistic manner.
Future work will cover the evaluation of existing prototypes, which are initially provided by the domain-specific pilots and later in M25 by the cross-domain city pilots, as well as finalized demonstrators in M33 and M36. Introduced test schedules will be applied and further test scripts will created and executed.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 67 30th of June 2017
6. References
Air Force Research Laboratory (AFRL). AFRL Hardware and Software Transition Readiness Level Calculator,
Version 2.2. [Online] [Cited: 16 01 2017] https://acc.dau.mil/CommunityBrowser.aspx?id=741047.
Hambling, Brian and von Goethem, Pauline. User Acceptance Testing: A Setp-By-Step Guide. Swindon: BCS,
2013.
Sauser, Brian; Gove, Ryan; Forbes, Eric and Ramirez-Marquez, Jose Emmanuel. Integration maturity metrics:
Development of an integration readiness level. In: Information Knowledge Systems Management, vol.9, pp 17-
46, 2010.
U.S. Government Accountability Office. Technology Readiness Assessment Guide: Best Practices for Evaluating
the Readiness of Technology for Use in Acquisition Programs and Projects. [Online] [Cited: 16 01 2017]
http://www.gao.gov/assets/680/679006.pdf.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 68 30th of June 2017
7. Annex 1
Template - IRL (Integration Readiness Level) Questionnaire (p. 69)
Template - TRL (Technology Readiness Level) Questionnaire (p. 70)
Template - UAT (User Acceptance Test) Worksheet (several sheets for different steps, pp.71)
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 69 30th of June 2017
Project Acronym: bIoTope
Project title: Building an IoT Open Innovation Ecosystem for Connected Smart Objects
IRL Questionnaire
IRL Definiton Description (Start at the top and first Definition answered with yes leads to potential rigth IRL)Checkbox
(y/n)potential IRL
IRL 9Integration is proven through successful
operations in the system environment.
IRL 9 represents the integrated technologies being used in the system environment successfully. In order for
a technology to move to TRL 9 it must first be integrated into the system, and then proven in the relevant
environment, so attempting to move to IRL 9 also implies maturing the component technology to TRL 9.
IRL 9
IRL 8
Actual integration completed and qualified through
test and demonstration, in the system
environment.
IRL 8 represents not only the integration meeting requirements, but also a system-level demonstration in the
relevant environment. This will reveal any unknown bugs/defect that could not be discovered until the
interaction of the two integrating technologies was observed in the system environment.
IRL 8
IRL 7
The integration of technologies has been verified
and validated and an acquisition/insertion
decision can be made.
IRL 7 represents a significant step beyond IRL 6; the integration has to work from a technical perspective,
but also from a requirements perspective. IRL 7 represents the integration meeting requirements such as
performance, throughput, and reliability.
IRL 7
IRL 6
The integrating technologies can accept,
translate, and structure information for its
intended application.
IRL 6 is the highest technical level to be achieved, it includes the ability to not only control integration, but
specify what information to exchange, unit labels to specify what the information is, and the ability to translate
from a foreign data structure to a local one.
IRL 6
IRL 5
There is sufficient control between technologies
necessary to establish, manage, and terminate
the integration.
IRL 5 simply denotes the ability of one or more of the integrating technologies to control the integration itself;
this includes establishing, maintaining, and terminating.IRL 5
IRL 4
There is sufficient detail in the quality and
assurance of the integration between
technologies.
Many technology integration failures never progress past IRL 3, due to the assumption that if two
technologies can exchange information successfully, then they are fully integrated. IRL 4 goes beyond
simple data exchange and requires that the data sent is the data received and there exists a mechanism for
IRL 4
IRL 3
There is compatibility (i.e. common language)
between technologies to orderly and efficiently
integrate and interact.
IRL 3 represents the minimum required level to provide successful integration. This means that the two
technologies are able to not only influence each other, but also communicate interpretable data. IRL 3
represents the first tangible step in the maturity process.
IRL 3
IRL 2
There is some level of specificity to characterize
the interaction (i.e. ability to influence) between
technologies through their interface.
Once a medium has been defined, a “signaling” method must be selected such that two integrating
technologies are able to influence each other over that medium. Since IRL 2 represents the ability of two
technologies to influence each other over a given medium, this represents integration proof-of-concept.
IRL 2
IRL 1
An interface between technologies has been
identified with sufficient detail to allow
characterization of the relationship.
This is the lowest level of integration readiness and describes the selection of a medium for integration. IRL 1
In the following Questionnaire, a framework is provided for calculating IRLs within bIoTope.
How to use: To calculate a IRL for a relevant technology, the answers and definitions within the following TOP LEVEL VIEW Table have to be answered, starting from the top of the table. The first question or
definition answered with YES leads to the potentially right IRL.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 70 30th of June 2017
Project Acronym: bIoTope
Project title: Building an IoT Open Innovation Ecosystem for Connected Smart Objects
TRL Questionnaire
Nr. TOP LEVEL VIEW (Start at the top and first Question answered with yes leads to potential rigth TRL)Checkbox
(y/n)Continue with
1Has an identical technology component been successfully used in an operational scenario or in an identical
configuration?TRL 9
2Has an identical technology component been demonstrated in an operational scenario, but in a different
configuration/system architecture?TRL 8
3Has an identical technology component been qualified for an operational scenario but not operationally
demonstrated?TRL 8
4 Has a prototype technology component been demonstrated in an operational scenario? TRL 7
5 Has a prototype been demonstrated in a relevant scenario, on the target or surrogate platform? TRL 6
6Has a breadboard technology component been demonstrated in a relevant (typical; not necessarily
stressing) scenario?TRL 5
7 Has a breadboard technology component been demonstrated in a laboratory (controlled) scenario? TRL 4
8 Has analytical and experimental proof-of-concept been demonstrated? TRL 3
9 Has a concept or application been formulated? TRL 2
10 Have basic principles been observed and reported? TRL 1
11 None of the above TRL 0
In the following Questionnaire, a framework is provided for calculating TRLs within bIoTope. The framework is based on the AFRL Hardware and Software
Transition Readiness Level Calculator, Version 2.2 developed by the Air Force Research Laboratory (AFRL) [Air Force Research Laboratory]. This tool is
aligned with the TRL elements defined by the U.S. DoD.
How to use: To calculate a TRL for a relevant technology, the answers within the following TOP LEVEL VIEW Table have to be answered, starting from the top
of the table. The first question answered with YES leads to the potentially right TRL. In order to check whether this first estimation is right, more detailed
facts are going to be analysed. Those checks are provided in Sheet TRL 9 to TRL 0. All facts listed within each of the tables have to be answered with YES to
achieve the corresponding TRL.
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 71 30th of June 2017
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 72 30th of June 2017
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 73 30th of June 2017
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 74 30th of June 2017
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 75 30th of June 2017
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 76 30th of June 2017
D2.6 Evaluation Report of the bIoTope Pilots
© 688203 bIoTope Project Partners 77 30th of June 2017