IES Cities D3.3: Second Phase Execution and Monitoring · 2017-04-25 · IES Cities D3.3: Second...

IES Cities Internet-Enabled Services for the Cities across Eur ope

Grant Agreement: 325097

D3.3: Second Phase Execution and

Monitoring Doc. Reference: IES CITIES-WP3-D33-REP-160308-v10 Responsible: TOSHIBA Author(s): TECNALIA, UD, KWMC, TOSHIBA, ROVERETO, FBK, ZARAGOZA,

AYTOMAJ, GEKO, EUROHELP and ZIB Date of issue: 08/03/2016 Status: Accepted Dissemination Level: Public

Version Date Description

v0.1 18/11/2015 T.O.C. creation

v0.2 04/12/2015 T.O.C. agreed

v0.3 25/01/2016 ROVERETO contribution

v0.4 04/02/2016 TOSHIBA, MAJADAHONDA contribution

v0.5 19/02/2016 BRISTOL, KWMC, ZARAGOZA, TECNALIA contributions

v0.6 23/02/2016 ROVERETO, TOSHIBA contribution

v0.7 24/02/2016 Final contributions received – ready for external review

v0.8 02/03/2016 External reviews complete and recommendations implemented

v0.9 07/03/2016 Minor improvements

v1.0 08/03/2016 Final version – ready to be submitted

IES Cities D3.3: Second Phase Execution and Monitoring IES CITIES-WP3-D33-REP-160308-v10

Table of contents 1. Executive Summary ............................................................................................................................................. 5

2. Introduction .......................................................................................................................................................... 6

2.1. Description and Objectives ........................................................................................................................... 6

2.2. Objective ....................................................................................................................................................... 6

2.3. Approach ....................................................................................................................................................... 7

3. Methodology for evidence collection .................................................................................................................... 8

3.1. Lessons learned from First Phase ................................................................................................................ 8

3.2. Changes from First Phase ............................................................................................................................ 8

3.3. Sub-phases ................................................................................................................................................... 8

3.4. Logging ......................................................................................................................................................... 9

3.4.1 Events ................................................................................................................................................... 9

3.4.2 Querying ............................................................................................................................................. 10

3.4.3 Swagger .............................................................................................................................................. 11

3.4.4 Log Dashboard ................................................................................................................................... 11

3.5. Questionnaires ............................................................................................................................................ 11

3.5.1 Alpha ................................................................................................................................................... 12

3.5.2 In-app .................................................................................................................................................. 12

3.5.3 On-line ................................................................................................................................................ 14

3.5.4 Developer and Local Authorities Questionnaires ............................................................................... 15

3.5.5 In-depth Interviews .............................................................................................................................. 16

3.6. Support ....................................................................................................................................................... 16

3.6.1 Zaragoza ............................................................................................................................................. 17

3.6.2 Bristol .................................................................................................................................................. 17

3.6.3 Rovereto ............................................................................................................................................. 18

3.6.4 Majadahonda ...................................................................................................................................... 19

4. Feedback on evidence collection ....................................................................................................................... 20

4.1. Logging ....................................................................................................................................................... 20

4.2. Software ...................................................................................................................................................... 20

4.2.1 Zaragoza ............................................................................................................................................. 21


4.2.2 Bristol .................................................................................................................................................. 21

4.2.3 Rovereto ............................................................................................................................................. 23

4.2.4 Majadahonda ...................................................................................................................................... 24

4.3. Activities ...................................................................................................................................................... 24

4.3.1 Zaragoza ............................................................................................................................................. 24

4.3.2 Bristol .................................................................................................................................................. 27

4.3.3 Rovereto ............................................................................................................................................. 29

4.3.4 Majadahonda ...................................................................................................................................... 32

4.4. Questionnaires ............................................................................................................................................ 33

4.4.1 Zaragoza ............................................................................................................................................. 33

4.4.2 Bristol .................................................................................................................................................. 34

4.4.3 Rovereto ............................................................................................................................................. 35

4.4.4 Majadahonda ...................................................................................................................................... 35

4.5. Support ....................................................................................................................................................... 36

4.6. Objectives ................................................................................................................................................... 37

4.6.1 Zaragoza ............................................................................................................................................. 37

4.6.2 Bristol .................................................................................................................................................. 38

4.6.3 Rovereto ............................................................................................................................................. 38

4.6.4 Majadahonda ...................................................................................................................................... 39

4.7. Service Exchange ....................................................................................................................................... 40

4.7.1 Zaragoza ............................................................................................................................................. 40

4.7.2 Bristol .................................................................................................................................................. 40

4.7.3 Rovereto ............................................................................................................................................. 41

4.7.4 Majadahonda ...................................................................................................................................... 41

4.8. Feedback on data sharing between apps & cities ...................................................................................... 42

5. Conclusions ........................................................................................................................................................ 43

6. Comments from External Reviewers ................................................................................................................. 44

6.1. Reviewer 1 – UD ......................................................................................................................................... 44

6.2. Reviewer 2 – ZIB ........................................................................................................................................ 45

7. Abbreviations ..................................................................................................................................................... 46


8. References ......................................................................................................................................................... 47

9. Appendix - Monthly engagement tables............................................................................................................. 48

9.1. Zaragoza ..................................................................................................................................................... 48

9.2. Bristol .......................................................................................................................................................... 57

9.3. Rovereto ..................................................................................................................................................... 64

9.4. Majadahonda .............................................................................................................................................. 69


1. Executive Summary

IES Cities is the last iteration in a chain of inter-related projects promoting user-centric and user-provided mobile services that exploit open data and user-supplied data. The technical components and achievements of several former European projects along with newly developed components were integrated to form an open Linked Data apps-enabling technological platform. The resulting IES Cities platform was deployed in different cities across Europe, allowing the citizens to produce and consume internet-based services (apps) based on their own and external open data related to the cities.

IES Cities main objective was to provide a user-centric urban apps enabling platform for users in four cities across Europe. Each city initially tested the functionalities of the platform with completely independent services, different at each location, in order to allow the cities and their citizens covering different needs under a common platform. By means of the tools created within the IES Cities framework, users were able to continuously enrich the consumed services with their own data and to have a voice in the proposal and selection of new services to be developed. After the first phase of the city pilots a 2nd phase followed, where a further set of services was deployed and tested in the participating cities. In this 2nd phase of the pilots, at least one service per city was exchanged with another city.

The present Deliverable D3.3 – Second Phase Execution and Monitoring analyses and discusses the rich volume of evidence collected during the 2nd pilot phase of the IES Cities project and how this was recorded. It looks at both quantitative and qualitative data following the evaluation framework and plan defined in D4.1 – Evaluation Plan [1] and the questionnaires described in D5.2.X – Zaragoza, Bristol, Rovereto and Majadahonda Questionnaires [2]. It then reflects on several questions: a) what has worked, b) what has not worked, and c) how does the 2nd phase compare against the 1st phase.


2. Introduction

2.1. Description and Objectives

The objective of this deliverable, which is the realization of the IES Cities tasks T3.3 is to create a report describing the different problems and solutions faced during the second phase of the IES Cities trial. As stated in Annex I – Description of Work [3] and in D3.1.2 – Pilots Planning v2 [4], trial phase II focuses on the provision of specific city services (i.e. mobile apps) where the following scenario is considered:

• Four services per participating city are tested during this second phase trial; two services that were developed for the first phase and two new services developed for the second phase of the IES Cities pilot. Pilot cities will choose services from each domain to assure that a high variation of city services become evaluated.

• Out of the two newly developed services, at least one will be a service exchanged with one of the other participating cities. The exchange of the service can either mean that one service will re-use existing code from another city and then adapt it accordingly, or alternatively that one service is built from the ground up but based on the idea of another city service.

• In addition to the service exchange, participating cities will examine whether it is possible to also share service data between cities.

• The infrastructure assures that all services are accessible anywhere in the city. Hereby, the IES Cities project will cooperate with local governments to provide adequate access to required data.

• As cities from various countries are involved in the trials the service acceptance evaluation has to consider diverse cultural backgrounds of the people and differences among the cities.

• The pilot phase II trial will be open to all citizens living in a city. To satisfy the large scale conditions of the trial, up to five hundred participants are envisioned per city. Users will be able to download the services through Google Play directly or through the IES Cities service marketplace, i.e. the IES Cities Player [5].

• The duration of the trial was originally planned to be up to 7 months, including the pre-pilot sub-phase. With the required refinements of the execution plans and timings the 2nd phase pilot ran for 8 months in total; 2 months of pre-pilot running, followed by 2 months of beta testing and then with the final pilot running for 4 months.

2.2. Objective

T3.3 is one of the tasks within WP3 whose main objective is to execute the trials in each city as planned and under the designed and analysed parameters from earlier WPs. Specifically, the main purpose of T3.3 is to execute, monitor and log the results of each pilot in order to provide inputs for the assessments in the WP4.


2.3. Approach

Deliverable D3.3 discusses the rich volume of evidence collected during the 2nd trial phase of the IES Cities project. Rather than reproducing overlapping information contained within other deliverables such as [1], [2], [4] and [6], this deliverable records feedback from the IES Cities consortium partners. Herein partners consider the collection of both the quantitative and qualitative data and then reflect on:

• What has worked well?

• What has not worked well?

• How does the 2nd phase execution compare against the 1st phase execution?

This document is divided into seven areas of interest: logging, software, activities, questionnaires, support, objectives and service exchange.

Logging

Feedback from technical partners is recorded as to how well the logging systems [4] have worked. It includes the sections: apps KPIs and logs, and Swagger KPIs and logs.

Software

This includes feedback from the cities as to how well the IES Cities software [4] has worked. It includes the sections: IES Cities platform, apps and Swagger.

Activities

This includes feedback from the cities as to how well the users’ engagement activities have worked. This is structured using the new 2nd phase engagement methodology [6]. It includes the sections; awareness, downloads, active use and users feedback.

Questionnaires

This includes feedback from the cities as to how well the questionnaires [2] have worked. It includes the sections; alpha, in-app, on-line and in-depth and developer questionnaires.

Support

This includes feedback from the cities as to how well their support systems [6] have worked.

Objectives

This includes feedback from the cities as to how their 2nd phase objectives [4] have been met.

Service Exchange

This includes feedback from the cities as to how the service exchange worked for them.


3. Methodology for evidence collection

Various methods were used to collect evidence of the technical performance of apps and platform, and also of the users’ perceived experience of the apps.

3.1. Lessons learned from First Phase

Users found the in-app and on-line questionnaires to be overly long and cumbersome to use. They were also unattractive and repetetive. Very few in-depth questionnaires were completed, And where they were, these were made in the form of case study videos.

3.2. Changes from First Phase

A simplified and reduced list of questions has been created so that the whole in-app questionnaire fits elegantly onto a single page. Ratings used a star based system that was familiar for anyone using apps and voting within app stores. A further improvement was enabling the consortium to be able to look up results directly from the IES platform during the 2nd phase.

For the on-line questionnaires, the number of questions was halved. All duplicate questions between in-app and on-line were removed. This defined the role of the questionnaires more clearly; the in-app questionnaires asked about app design and usability, while the on-line questionnaires asked about the citizen’s experience of the city through the IES Cities services and also about the IES platform business model. The Google Forms back-end spread sheet was also rebuilt.

3.3. Sub-phases

As described in D3.1.2 – Pilots Planning v2 [2] the 2nd phase of the IES Cities experiment was split into two further sub-phases: a) pre-pilot phase and b) pilot execution phase. This enabled a short internal 1st subphase for alpha testing, before the public release 2nd sub phase, initially as beta and then a fully functioning and supported release:

Figure 1 – The original 2nd phase pilot execution p lan


The first sub-phase was used to make sure that the new features of the IES Cities infrastructure (back-end and player), but particularly the 4 services (2 old and 2 new ones), i.e. IES Cities-compliant apps, running on top of it are fully tested and mostly bug-free. In this first sub-phase, a controlled and selected reduced group of alpha testers external to the project reported about their experience using the services and infrastructure. Their feedback, gathered through interviews and alpha test questionnaires designed in T5.2 [2], was used for several purposes:

• apply necessary corrective actions to ensure a smooth execution of the second pilot phase, and

• make sure that all the services to be tested are properly instrumented with execution logs and in-app questionnaires so that the right details about them can be collected to be used then by WP4 in the pilot analysis.

New apps were instrumented with newly designed in-app questionnaires adapted to address the usability problems identified in phase 1. Additionally, the 2 old apps per city were re-instrumented with the new in-app questionnaires as well, so as to have a consistent feedback collection across all apps active in the 2nd phase pilots.

Additionally, trial site and app specific KPIs, as well as new ones were defined during this sub-phase. The testing session involving external users was complemented by a crosstesting session involving IES Cities consortium members. In this three days cross-testing session, each member of the IES Cities consortium was assigned two specific services to test. After this session, every member reported technical bugs by using the technical tool deployed for that purpose (Redmine BUGS tracking system).

The second sub-phase corresponded to the actual launching of the Pilot Phase II. This subphase consisted of a further two subphases; beta and public release. First, an open set of beta testers were given access to the apps published in beta mode on Google Play. In this subphase, it is sufficient to supply users with a link to the app to be able to install it but the engagement activities also focused on fostering the IES Cities web interface usage. Finally in the last subphase, the public release of the apps made all the apps available to anyone through the Google Play store or, alternatively, the IES Cities player.

The final schedule of the 2nd phase pilots, as refined throughout its execution to accommodate all the pilot requirements and to help achieve the objectives was the following:

• Alpha testing: June 2015 – July 2015 (original: June 2015)

• Beta testing: August 2015 – September 2015 (original: July 2015 – September 2015)

• Public release: October 2015 – January 2016 (original: October 2015 – December 2015)

3.4. Logging

Logging assessments were quantitative .

3.4.1 Events


The logging mechanism is documented more completely in D4.1 – Evaluation Plan [3], but the basic mechanism is that mobile apps can use the IES Cities Platform REST API to log events that occur within the app as it executes. Each event consists of the application name, an event type describing the occurrence and a text field to give further information. It is marked with meta-data that additionally records the time, an identifier that uniquely identified the installed device, and optionally some timing data.

If the JavaScript logging API is used correctly by the app, some events (the ones that record when an app is actually running in the foreground on the device) are generated automatically; other types of events require developers to directly call the API to record the event. These events aim to capture specific types of interaction with the app, such as data consumption or the production of user supplied content.

In addition to the application logging, all accesses to the data sources that are registered in the platform are logged, using a similar mechanism. These events are logged automatically by the platform and require no developer effort.

A related mechanism captures the in-app questionnaire responses and stores them in the platform database.

3.4.2 Querying

Each app KPI was implemented as a PostgreSQL query for running directly on the logs database table on the platform. An interface to run these queries was made available over the IES Platform rest interface. This was used mainly directly through the swagger interface. Individual city taskforces took responsibility for running these queries and tabulating the data over the months of the pilot. An example query to count the number of photos added in the MyBristol app would look like:

select COUNT("MESSAGE") from "APPLOG" where ("APPID"='MyBristol' OR

"APPID"='MyKW') AND "TYPE"='AppProsume' AND "MESSAGE"='PhotoAdd'

In addition, some top level general KPIs that were appropriate for all apps were designed and collected. These capture the following metrics on a monthly basis for each app:

• Unique devices: Number of distinct devices (session ids)

• Active Users: Number of distinct devices that generate AppConsume events. This is designed to strip out trivial app usage, but this statistic may be misleading for apps that generate these events without direct user intervention.

• User Sessions: Number of Appstart events generated.

• In-app Responses: Number of AppQuestionnaire events generated.

This was implemented via a JavaScript component that could be launched at any point during the pilot. This was useful for quick live testing of the logging mechanism and apps, these metrics are also reported on in D4.2 – Report on the Monitoring of Results [8] in more detail where some post-processing of the logs has been done to clean the data.


In the second phase each city taskforce took the responsibiltiy of collating and updating a shared spreadsheet of KPIs. Thus all partners could view KPIs across all cities.

3.4.3 Swagger

Swagger UI is part of the Swagger project [7]. The Swagger project allows us to produce, visualise and consume the IES platform RESTful services. Swagger UI is a dependency-free collection of HTML, JavaScript, and CSS assets that dynamically generate documentation and sandbox from the Swagger-compliant API.

3.4.4 Log Dashboard

For the phase two pilot an app dashboard web page1 was set up that displayed standardised usage levels of each app through a set of charts. Figure 2 shows a session browsing the logs from the “Rovereto Segnala” app.

Figure 2 – The Log Dashboard web page showing usage statistics of the "Rovereto Segnala" app

3.5. Questionnaires

Questionnaire assessments were qualitative and have allowed collecting feedback from the users´ experience throughout the IES Cities platform and services. The approach followed during the Pilot Phase II with regards to the questionnaires is mostly the same than in phase I since during this stage the questionnaires used to collect information are: alpha, in-app, on-line questionnaires and the in-depth interviews. The main differences are related to the content and questions included in each questionnaire. Taking into account the outcomes from pilot phase I,

1 This is available at http://iescities.com:3838/sample-apps/dashboard


it was decided to reduce and simplify the on-line questionnaire as well as to improve the usability of the in-app questionnaire. Finally, in order to collect feedback from developers, a specific online questionnaire was created for that purpose.

3.5.1 Alpha

Taking into account the results gathered during the pilot phase I through the Alpha-script, this questionnaire has been also used during the phase II maintaining the same structure and questions.

This questionnaire, as it will be detailed in WP4, will be mainly used and completed by IES Cities consortium members who are expected to participate in a cross-testing session planned during the launching of pilot phase II.

The Alpha script uses Google Forms and is available on-line through a web browser: https://docs.google.com/forms/d/1bCGPKiLibsMpSXCudW47-FOpSqXkDsXRymzYhM7hsb0/viewform

3.5.2 In-app

As for the in-app questionnaire, the analysis of the feedback collected during pilot phase I resulted in the decision to improve the in-app questionnaire by reducing the number of questions, simplifying those questions and enhancing the user interface.

The new version of the in-app questionnaire consists of a reduced set of questions in such a way that all these questions fits onto a single page. The purpose is to avoid the user jumping from one screen to another screen to answer all the questions. With this new version the user will be able to rate several aspects of the application such as: design, performance, ease of use, usefulness and IES Cities service content by means of a star based system. Finally, the

Figure 3 – Alpha script questionnaire


user will be given the chance to provide any other feedback through a free text box and to be informed and kept in touch with IES Cities members by giving her/his email address. The new version of the in-app questionnaire is shown below:

Figure 4 – In-app single screen questionnaire

In-app questionnaires use a JavaScript library, which was easy to integrate with Apache Cordova/PhoneGap hybrid app development approach. Questions were presented as an overlay over the app, and were hard-coded into a JSON file. In-app questionnaires were triggered manually and also after a programmed number of app launches.

User responses were stored in the IES Cities back-end and are accessible through the IES Cities Web Interface where a new section has been added to simplify the analysis of the

feedback collected. This page2 collects and displays the responses from an app in a form that can be viewed and downloaded. Figure 5 shows a typical session.

2 Available at http://iescities.com:3838/sample-apps/questionnaires


Figure 5 – In-App Questionnaire browser viewing the responses of the “MyBristol” app.

3.5.3 On-line

Similar to the in-app questionnaire, during the pilot 1st phase the on-line questionnaire platform was far less successful, being deemed by different users as being too complex and sometimes irrelevant. Therefore, it was decided that the on-line questionnaire has to be simplified for the 2nd phase.

The 2nd phase on-line questionnaires contained half the number of questions than that of the 1st phase. All duplicate questions that were in the an-app questionnaire were removed. That is, questions about design and useability were restricted to the in-app questionnaire. Questions in the on-line questionnaire were based around the city and the citizen, and the potential business models for the platform.

Figure 6 – On-line questionnaires

The on-line questionnaire uses Google Forms and is available through a web browser: https://docs.google.com/forms/d/1r4JvkaH6jsHI8uIqNXkfQHfpa62_IpjVlii2YXyH3Yk/viewform


3.5.4 Developer and Local Authorities Questionnaire s

After the pilot 1st phase, it was observed that feedback from two important stakeholders was missing. These stakeholders were: 1) developers and 2) local authorities. While local authorities were part of the IES Cities consortium and therefore easy to reach and collect feedback through the in-depth interviews methodology, this was not the same for the developers. Therefore it was decided to create a specific questionnaire for that kind of stakeholder.

This questionnaire was mainly designed to collect feedback from developers while they were using the IES Cities’ platform and services during planned hackathons. The questions included in the questionnaire were designed to evaluate the REST API’s methods and the functionality and features added in the platform to simplify the process of creating and publishing IES Cities services and applications.

The developer questionnaire also uses Google Forms and is available through a web browser: https://docs.google.com/forms/d/1xQYEeVmdfi3YZ0Lw6R4_kSDDXmGHZWyvqu9OKM2qfbo/viewform

Figure 7 – Developer Questionnaire interface

To collect feedback from the local authorities that are part of IES Cities project; i.e. Bristol, Majadahonda, Rovereto and Zaragoza; a new questionnaire has been designed and published for pilot 2nd phase. This questionnaire consists of a series of questions to evaluate not only the specific functionality provided by the IES Cities platform for the local authorities but also the IES Cities solution as the means and a tool to promote an urban app ecosystem prosuming Open Data and user-generated data in the city. This questionnaire is also available through a web browser using Google Forms:

https://docs.google.com/forms/d/1rHAS6QhiHocY8iwizOHWTWOpkTqGm2pFN8O3eb5wyH4/vi


ewform

Figure 8 – Local Authority Questionnaire interface

3.5.5 In-depth Interviews

To increase the understanding of the users’ perspective of the 2nd phase apps, optional in-depth interviews were encouraged.

These could take various forms: an interview, a video recording, an audio recording, text notes, case studies or any additional feedback from the users’ that would enrich the responses from the other types of questionnaires.

Qualitative case studies could be documented using a variety of different media - text, photo, audio and video. This media could form the basis of short documentary style films to be made about the apps of each city. Unfortunately a lack of film making skills across the task forces often limited this appraoch.

When preparing the questions for the case study, consideration was given to the original objectives of the apps, as reported in section 4.6 - Objectives of the present deliverable. At least one question per objective was to be included, ensuring that the interviewee could directly acknowledge the objective in their answer and evaluate the app’s success in meeting it. This process was descibed in the deliverable D5.2.X “Bristol/Zaragoza/Rovereto/Majadahonda questionnaires” [2].

3.6. Support

This section outlines the support mechanisms that were available for users during the second phase of the pilots in the four cities.


3.6.1 Zaragoza

Zaragoza City Council provides technical and functional support for the four IES Cities apps through “Your Opinion Matters”, both web version and smartphone version. Additionally, the 010 number is also available for this second phase pilot execution just like it was during the first one.

When a user/citizen uses 010 an operator takes care of the needs of him/her and tries to solve the request. In order to make the task easier, a list of Frequently Asked Questions (FAQ) is available, with the most common errors that could happen as well as how to solve them.

If the error/issue does not appear in the list of FAQ or if it does not work properly, the operator will transfer the call to the Smart City Department where one of the technicians will listen to the citizen. If he/she fixes the problem then he/she will update the list of Frequently Asked Questions for similar future issues. If the technician cannot deal with the issue because it is a software problem, then he/she will divert the issue to our task force team, EUROHELP, who will be in charge of solving the problem and reporting to the Zaragoza City Council when the bug has been solved.

When a user makes his request through “Your Opinion Matters”, personnel from the city council proceeds to evaluate it and transfers it to the adequate department. The user can always check the status and progress of the issue via web or through the smartphone app.

3.6.2 Bristol

Official support for the Bristol applications during the 2nd phase used the same system as was used during the 1st phase. Bristol’s support process is:

• User sends email to [email protected], which is monitored by the Bristol City Council (BCC) team.

• An automated response is generated, which acknowledges receipt and provides links to further information, including IES Cities web site and introductory video of the project concepts.

• BCC triage all emails at regular intervals and log calls for audit purposes.

• Depending on the type of call, these are passed to Knowle West Media Centre (KWMC) and\or Toshiba support, as appropriate.

• If appropriate, calls are added to the tracking system for further action.

• Once the call has been evaluated, it may trigger an email response to the user (assuming they have provided contact details).

• Common issues may be added to FAQ’s, which will also be available from the initial auto response.


Figure 9 – Bristol app support model

3.6.3 Rovereto

During the second pilot phase the support procedures available for the users in Rovereto were mainly the same as those used during the first phase.

• First support procedure steps 1. Users who need support write an email to the address: [email protected] 2. The Rovereto Municipality team involved in this project will read those emails. The

members of the team will be on hand during the whole office hours. 3. The team will evaluate all the questions asked by users and try to answer most

completely as possible in a very short time. 4. In case of technical or very specific questions the team will forward those queries to FBK

researchers, who will provide a more precise support. • Second support procedure steps

1. Users who want to directly reach FBK scientific researchers and app developers and get support from them have to write an email to the address: [email protected]

Rovereto pilot support team

o Rovereto Municipality team, represented by the staff of the General Direction Office.


o As already specified the team was on hand during the whole working hours to try to answer all the questions as soon as possible.

3.6.4 Majadahonda

The Majadahonda City Council uses the already implemented Complaints and Suggestion service to provide support for IES Cities apps and users. It consists of a web-based interface available to citizens called “Majadahonda Helpdesk” and an issues tracking system, called “Majadahonda XPERTA”.

Once a request is submitted through Majadahonda Helpdesk, personnel from the city council proceed to evaluate it. If the user is reporting an issue, a new ticket will be generated into XPERTA to allocate the resources needed and fix it, if possible. The user can always check the status and progress of the issue via the Helpdesk interface.

Internally, XPERTA assigns the issues to different City Council departments. In particular, the department of New Technology has designated someone responsible within its staff to deal with the IES Cities issues. If technical assistance is required, this person will dump the issue into IES Cities Redmine, where the technical IES Cities partners will provide their assistance.

Once the issue has been solved, the IES Cities responsible marks it as resolved in XPERTA, automatically notifying the user who contacted the City Council.

The following figure illustrates all this process:

Figure 10 – Majadahonda support process


4. Feedback on evidence collection

This section of the document outlines the feedback that is relevant to the logging mechanisms that were used during the second phase of the pilots.

4.1. Logging

Herein is feedback from the technical partner (Toshiba) who developed the logging system, as to how well the project’s 2nd phase logging - as described in D3.1.2 “Pilots Planning v2” [4] - has worked. It includes the sections: apps KPIs and logs, and Swagger KPIs and logs. The questions answered by each city are: a) what has worked, b) what has not worked, and c) what are the lessons learnt for the 2nd phase?

Questionnaire types

What has worked well? What has not worked well? Comparison with 1 st phase

App Logging

• As in the first phase, generally the logging mechanism has resulted in detailed and useful logging data from all apps.

There were still some issues with logging: • One app disabled the logging

due to server downtime and did not re-enable this for an extended period, missing many logs.

• Another app misconfigured the logger, causing logs to be generated with a default name. This was discovered and some statistics could be recovered.

• Testing usage still has the potential to pollute the user logs dataset

• A number of suggested improvements were not made due to the potential of breaking existing apps that were logging.

• Some changes were made to reduce erroneous “AppStop” event generation.

• Mechanisms were adopted to remove some of the testing usage in the log analysis.

Rating Logging

• The second phase saw a much better user response rate for in-app ratings mechanism.

The ratings mechanism also suffers from the same testing usage pollution issue.

• Much simplified in-app questionnaire to increase response rate.

• Questionnaire web page to allow viewing/download of responses

App KPIs

• Logging mechanism provided sufficient data to feed the KPI calculation process.

• Getting KPI data was still a messy process. The dashboard was a partial solution, but was perhaps made available too late to maximise its impact.

• Log dashboard to improve feedback of logging statistics to app developers and cities.

Table 1 – Feedback on logging by technical partners

4.2. Software

Herein is feedback from the technical partners as to how well the project’s 2nd phase software - as described in D3.1.2 “Pilots Planning v2” [4] - has worked. It includes the sections: IES Cities platform, apps and Swagger. The questions answered by each city are: a) what has worked, b) what has not worked, and c) how did the 2nd phase software compare against the 1st phase.


4.2.1 Zaragoza

Software What has worked well? What has not worked well? Comparison with 1 st phase

Platform

• The platform has been tested and used by developers who took part in the hackathons organised by the ZGZ city council. These developers created hybrid mobile applications using the services offered by the platform. They found that these services were useful in order to reduce the development time of an app.

• Non experienced users found it difficult to know how to integrate platform services in their apps

• Fewer bugs were detected and the platform was use more intensively.

• More apps were created using the platform.

Apps

• Apps for second phase have had good feedback from users.

• In the case of “Zaragoza Participa” app, the idea that it is a tool that enhances citizen participation has been very well received.

• Users have actively participated in the design of “Zaragoza Participa” and also this app has been shared with Bristol.

• Some delays in the expected date for publishing the final apps, mainly due to:

• The number of different stakeholders involved in the process

• Some errors were not detected until December when applications began to be used by those attending the activities and, on the other hand, some suggestions for improving the app came up too late to be added to the application.

• We have less downloads than in the previous phase due to the delay in publishing the app. However, during February and March we hope to meet the download targets thanks to the planned activities. In any case, both the feedback obtained and the involvement of citizens in the creation of apps have been satisfactory.

Swagger

• Users in general and developers in particular found the swagger interface very useful in order to know the methods offered by the platform’s APIs. It was considered a very intuitive and visual way to document an API.

• During this phase the swagger interface has been used more intensively and thus it has been tested more in depth.

Table 2 – Feedback on software by Zaragoza

4.2.2 Bristol


Platform

• Good performance and reliability

• Lots of features and functionality

• Rich documentation • Makes app development

easier and quicker • Integration of logging and

rating

• Some reliability issues still exist • Uptime could be still improved • Some functionality still lacking (e.g.

user management) • Platform deployment is not always

straightforward • Some undocumented or

unannounced improvements can break apps

• Improved reliability • Increased uptime • Improved

responsiveness • Additional

functionality • Improved

documentation


Apps

• App development was accelerated by using the platform

• Shared app (YouDecide) had a quick development time as there was significant code reuse

• Both 2nd phase apps have been reliable and well received

• Technical glitches and bugs were addressed speedily

HealthyOffice • Having the target audience

as IES partner organisations meant there were fewer barriers to engagement.

• Longer lead-in of early testing

• Collaborative approach to app design with Toshiba and KWMC

• Fast interactive process in development

• Partners sharing equipment • Positive approach to co-

working • Engaged partner working • App working well across all

partner organisations • Responsive changes to

sensor availability • Possible future development

collaboration

YouDecide : • People liked that their

questions were going somewhere, they found that because the council were going to see their views/polls it seemed to make it more real.

• It was sometimes easy to use, there were issues depending what device they were using but generally it was quite quick and simple which definitely encouraged others to use it once they had seen someone else complete the polls.

• It was interesting to see users adding their own polls and getting quite passionate about something they would change about their community if they

• App reliability was sometimes plagued by platform downtime

• Network delays can make the apps slow to respond – ideally apps need offline versions as well

• Users often complained about signing into their accounts – tried to use email instead of username (i.e. platform does not offer sign-in with email)

• Shared app (YouDecide) was delivered to Bristol team with significant delay and non-agreed changes, thus hindering further development and deployment.

• Additional requirements for apps came too late into the development cycle, thus no allowing them to be accommodated.

HealthyOffice • Limited by Android only • Having to add another tablet to

someone’s kit made app less integrated with people’s daily lives

• Using cross-app IES authentication actually caused problems with forgotten passwords and non-email logins

• Manager could only review one days data

• Not enough time to make key changes fed back from test groups

• Removal of sensor integration due to lack of hardware

• Queries about confidentiality • Getting people to do anything

outside their daily routine is Challenging

YouDecide : • If it were to be used long-term user

friendliness would need to be improved. It isn’t always easy. There are lots of issues in terms of moving around the site and simple things such as not having a back button.

• There were issues with users that felt the app needed developing before being used as a tool by authorities to officially collect public opinion.

• Because of the fact the app was co-developed, it meant delays in

• Increased platform reliability meant improved app reliability

• Faster development due to more experience with the platform

• Better designed engagement campaigns for apps

• Improved coordination between development and engagement teams


could. • Common themes came up in

the People’s Polls which was interesting.

• Overall good feedback from users.

• A few users said it would be good to have something like YouDecide in different community centres so they could regularly see the issues coming up from the authorities and they would get to have their say.

developing and general engagement because essentially we didn’t have a finished/refined app and therefore when we piloted the app we couldn’t make changes based on user feedback.

• If users had previously been a user of any other IES apps they could not log in if they couldn’t remember a part of their log in details which became an issue because it relied on username rather than email.

Swagger

• Swagger has proven to be a very useful tool for quick testing and also very useful as a source of documentation for the platform API

• While good, documentation should be even richer with examples of usage for the various features

• Improved documentation

Table 3 – Feedback on software by Bristol

4.2.3 Rovereto


Platform

• The platform has been tested and used by high school students. They have been able to produce easily 3rd part applications integrating IES Cities Platform

• Developing hybrid mobile applications the integration was easier than for native applications

• A few bugs in the IES Cities API were discovered

• In some cases the

documentation was hard to understand for external developers

• We had a deeper test using students of high school. This let us analyse better the platform

Apps • Big interest raised in users: high downloads obtained

• The combination of the two apps (Rovereto Percorsi and Tesori Nascosti) with the same focus in two different realities: Rovereto and the Alto Garda area (including more than a single municipality) has been a success

• Not as many data have been inserted by users as expected

• Increased number of filled in-app questionnaires with the new form

• Developing hybrid mobile applications made easier to generate android and iOS versions

Swagger • Swagger has been tested by students and they have found it really useful. A perfect tool to help their applications’ development

• We found some exceptions not managed

• The swagger has been tested better by many developers and this allowed us to find more bugs but in general it is the favourite tool for developers

Table 4 – Feedback on software by Rovereto


4.2.4 Majadahonda

Table 5 – Feedback on software by Majadahonda

4.3. Activities

Herein is feedback from the cities as to how well the project’s 2nd phase users engagement activities have worked. This is structured using the 2nd phase engagement methodology [6]. It includes the sections: awareness, downloads, active use and users feedback. The questions answered by each city are: a) what has worked, b) what has not worked, and c) how did this stage compare to the relevant 1st phase of engagement activities.

4.3.1 Zaragoza

Pilot execution phase Purpose of activity What has worked

well? What has not worked well?

Comparison with 1 st phase

Support

• To bring awareness • To collect feedback

• We have gathered valuable info about bugs or minor issues directly from users in the engagement activities.

• Not a single error has been reported through official channels available through the whole pilot execution. (“Your Opinion Matters” app -web

• At any stage of the project user have used the official ways to report issues but also informal ones.


Platform

• The platform software allowed us to establish a Collaboration framework with Francisco de Vitoria University.

• IES Cities platform software has raised the interest of third parties during the hackathon.

• Activities aimed at developers have been insufficient and were carried out very late in the project schedule, which led to very little usage of the platform.

• Open data inclusion into the platform has been difficult due to unavailability of formats (such as PDFs and images)

• More Majadahonda open datasets than in the 1st phase were made available through the platform

• An IES Cities hackathon was held in Francisco de Vitoria University; no such event happened with the 1st phase software

Apps

• The second phase apps are more intuitive, and their graphics design is more attractive. On average, they have more features and usability.

• The second phase apps logging and the KPIs were designed together, so the apps activity is more easily measured.

• The sensors providing data to Majadahonda Healthy City app have had some issues that are demanding time-consuming maintenance tasks.

• The second phase apps had a minor issue related to the app starts and in-app questionnaires logging that was identified and quickly fixed in January 2016.

• The alpha testing procedure was very similar. Perhaps in the second phase has been reinforced and has involved more testers. Many minor fixes have been done to improve the apps, as in the first phase.

Swagger

• It’s a great tool for monitoring the apps’ logs and the KPIs computation.

• The integration of the online questionnaire responses, if possible, has not been carried out.

• The features seem to be the same than in the first phase, but now dealing with all the pilot apps.


o smart phone version- and the 010 number).

Training

• To bring awareness • To make first

contact • To promote active

use of services

• In-service training session with personal staff provided a first test about the app also as a few alpha questionnaires filled out. On the other hand it was useful for gathering the first impressions about the apps.

• All testing team members had a technical background so we only gathered a partial insight.

• As in the previous phase, the staff showed interest in collaborating in the testing process applications.

Communication

• To bring awareness • To promote active

users of platform services.

• To make first contact.

• We have used different channels to let citizens know that services were ready for downloading (www.zaragoza.es, newsletters, Open Urban Lab blog, local media, social networks, and press media).

• The promotion succeeded with actors such as citizenship, apps developers and students but not so much with other stakeholders such as public managers or decision makers.

• The promotional campaign has been more intense than in the first phase and the results have been quite satisfactory.

Release

• To bring awareness • To make first

contact

• Getting a good and stable working version, suitable for testing by interested parties.

• Coordination between task force members.

• Usability issues were detected by users after the alpha and the internal tests.

Monitoring

• To collect feedback • Use of on-line, paper and in-app questionnaires to measure user perception of the services deployed.

• Cross-testing arranged among different partners: Testing session turned out to be a valuable experience in the previous phase.

• Check whether pilot targets are met.

• Trying to get as much information as we can from different stakeholders in training activities, workshops etc.

• People do not like filling out questionnaires. In most cases we have to offer material incentives (gadgets, merchandising gifts, etc.) to get their feedback.

• There are some cases where obtaining feedback is more difficult. A representative example is the case of students who are not allowed to use the phone during school hours nor while visiting our facilities. In this

• We have used the info provided by KPIs, log rating… better therefore we have been able to adapt the engagement activities to suit project needs in quasi real time.


case, we have tried collecting their views through a survey on paper.

Execution

• To collect feedback • To promote active

use of platform and services.

• The contributions of Citizens and other stakeholders who attended the training and engagement sessions on how to use the apps have been very useful.

• Open innovation sessions.

• School visits: students are trained in the use of the four apps.

• II Hackathon Zaragoza App Store with the aim of promoting the IES Players, the IES Cities Web Interface and the development of new apps by using available city council datasets.

• * More info at Zaragoza Engagement Plan.

• Some minor problems in the integration between Bristol and Zaragoza and mainly some problems regarding Zaragoza internal API caused a delay in the release of ”Zaragoza Participa” app and consequently in the active usage of the app.

• In the first phase, since all engagement activities took place in our premises, we were very constrained by the calendar of school visits to our facilities, in the second one on the contrary we planned some of the engagement activities in the target group facilities what resulted in much better engagement outcomes.

Evaluation

• To bring awareness • To collect feedback.

• Questionnaires, in-depth interviews, rating KPIs and feedback in general provided us the chance to discover whether the requirements in design sessions were consistent with the app users had in their hands.

• With the information

provided through all these means we have been able to adjust most of issues arisen in the whole project span.

• We found an error in the API council which complicated the task of tuning the application.

Reaction

• To promote active use of platform and services.

• To collect feedback.

• Correction of the issues detected as a consequence of the monitoring and evaluation processes.

• Some errors were detected when the implementation phase of the pilot was in an advanced stage.

• As in the previous phase, our task force (EUROHELP) has successfully


• Every single error or malfunction identified were immediately reported to the technical team and has been quickly sorted out.

solved the technical problems that have emerged over the implementation phase.

Table 6 – Feedback on activities by Zaragoza

*Engagement monthly templates for Zaragoza are presented in the Appendix.

Figure 11 – Zaragoza actual schedule followed

4.3.2 Bristol

Pilot execution phase

Purpose of activity

What has worked well?

What has not worked well?

Comparison with 1st phase

Support

• To bring awareness

• Most user queries were resolved quickly and successfully

• Users typically bypassed official support channel and contacted developers directly

• Increased ability to quickly and easily make changes, updates and development in order to maintain user engagement

Training

• To promote active use of platform and services

• Users were able to learn to use the apps quite quickly

HealthyOffice: • As it requires daily

use to get better results, it is harder to get users to achieve daily use.

• People without Android had to be given a tablet

• Managers could only view daily use.


Communication


YouDecide: • Fulfilled a need

that people were keen on.

• Easy to introduce to relevant community meetings.

• Met an important need of being able to communicate with your authority

• Being able to reach a particular area that needed their voices heard.

• The 2nd phase apps were developed around community needs possibly more so than 1st phase considering the difference between need and want.

Release


• Timely release for HealthyOffice

• Delayed release for YouDecide

• Less time to engage and develop the apps. Longevity of engagement with local people.

Monitoring

• To collect feedback

HealthyOffice: • Easy to use. • Doesn’t take long.

HealthyOffice: • Can only put in one

mood at a time. • Overall pie chart

diagram comparing others moods never changes.

• Not able to see the cube sensor data within the app (we know this is planned for the future).

Execution

• To collect feedback

YouDecide: • Quick to use • Good to be able to

have a say.

YouDecide: • Could be a lot more

simplified considering it is a simple idea.

Evaluation


HealthyOffice: • Really active and

engaged partners and already a wellbeing charter in place and a wellbeing charter being pushed nationally.

• Work stress being a national focus.

HealthyOffice: • It was android only

which created a barrier in terms of getting people that don’t have an android device involved and interested.

• The 1st phase apps could be used on cross platforms websites/web apps. This made user engagement a lot broader and accessible.

Reaction

• To promote active use of platform and services

HealthyOffice: • Some users

engaged regularly and were obviously finding it useful

• Ease of use was very different in comparison to 1st phase apps.


• It was a quick task.

Table 7 – Feedback on activities by Bristol

*Engagement monthly templates for Bristol are presented in the Appendix.

Figure 12 – Bristol actual schedule followed

4.3.3 Rovereto

Pilot execution phase

Purpose of activity What has worked well? What has not

worked well? Comparison

with 1 st phase

Support

• to promote active use of platform and services

• to collect feedback

• Very few support requests, mainly thanks to the apps simplicity

Training

• to bring awareness

• to make first contact


• to collect

• A lot of activities have been done with developers during the summer period at FBK, guaranteeing the active usage of the platform and the apps and helping a continuous improvement of them

• In a first preliminary test phase users were informed

• A deeper contact with interested users has been used, through single face to face meeting and live demos of the apps


feedback by single face to face meeting, during which a deep presentation of the apps and their main functionalities was done

Communication

• to bring awareness



• Rovereto Percorsi: official release announced through a public press conference which raised a big interest

• Didn't succeed in organising many public events

• The public press conference attracted many people and local newspapers, helping for a wider knowledge spread about Rovereto Percorsi

Release


• Rovereto Percorsi: after a test period the app was widely released

• Rovereto Segnala: the

traffic wardens were identified as ambassadors for this app

• Some problems in the integration between Rovereto Segnala and the Rovereto Council’s actual software for internal management caused a delay in its release

• Rovereto Segnala: The identification of the ambassadors will guarantee the “official” structure of the complaints received by the Council

Monitoring



• The most of KPIs planned have been reached

The face to face live demos of the apps done in the awareness stage encouraged people in downloading and using the apps

Execution



• Rovereto Percorsi: this app showed a good active usage, with some contents added by users, in particular about the Christmas Market path

• Rovereto Segnala: traffic wardens daily use the app to send complaints to the Council

Evaluation • to collect feedback

• Rovereto Percorsi: All the activities done by the

• It’s important to have dynamic


working team in order to guarantee a continuous development and improvement of the app, with the addition of “fresh” contents etc., let the app to be intriguing since its release

• Rovereto Segnala: the identification of the ambassadors guarantees a continuous active use of the app

apps, continuously updated and enriched

Reaction


• In this second phase both paper questionnaires and in-depth interviews have been done, giving some interesting feedback and comments which helped the improvement of the apps

• In-app questionnaire: the new layout of this questionnaire was well accepted by users, who fill it easily

• The in-depth interviews done were very useful to get deep comments about the apps, leading to their better improvements

Table 8 – Feedback on activities by Rovereto

*Engagement monthly templates for Rovereto are presented in the Appendix.

Figure 13 – Rovereto actual schedule followed


4.3.4 Majadahonda

Pilot execution phase Purpose of activity What has worked

well? What has not worked well?

Comparison with 1st phase

Support



• Redmine has proven to be a good tool for internal issues, such as cross testing of the apps.

• Most users lack awareness of XPERTA. They would probably report apps malfunctioning through Google Play.

• No issues reported through XPERTA as in the 1st phase.

Training

• To promote active use of platform and services.

• School visits has resulted in many downloads and in-app questionnaires.

• Users have to be encouraged by means of prize draws to boost their engagement.

• They are giving more ideas for improving the apps in comparison with the 1st phase.

Communication

• To bring awareness.


• Sport centres are willing to participate and to help us during the dissemination of the apps

• Difficult to measure the outcomes of these activities. City council departments’ collaboration is not as fluid as expected.

• Communication campaigns have had similar results to the first phase.

Release


• To promote active use of the platform and services.

• Apps are more appealing to the users.

• Layout/User interface has been improved.

• We have had Sports4U app with the last new features ready very late.

• There has not been a big difference in comparison with the first phase.

Monitoring


• Short term monitoring has helped in raising concerns on the aspects with poorer results, enabling reaction.

• City Council agenda sometimes is very inflexible to quickly adapt to the project needs.

• KPIs have been designed according to the app features making it easier to track the activity of the apps.

Execution

• To bring awareness.




• In-app questionnaires feedback is better due to its new design.

• New engagement methodology.

• Developers and other stakeholders’ engagement has been reinforced.

Evaluation


• Engagement levels segmentation and new charts and graphics allow a better understanding of

• An issue with the app logs was identified at the middle of pilot phase and corrected immediately.

• Evaluation of individual engagement campaigns rather than overall engagement


the partial results and the efforts made up to the present.

Reaction




• Quick identification of issues.

• Lack of coordinated procedures for the implementation of new features and issues solving.

• In general, better coordination within the Majadahonda task force and a more proactive attitude.

Table 9 – Feedback on activities by Majadahonda

*Engagement monthly templates for Majadahonda are presented in the Appendix.

Figure 14 – Majadahonda actual schedule followed

4.4. Questionnaires

Herein is feedback from the cities as to how well the project’s 2nd phase questionnaires [5] have worked. It includes the sections: alpha, in-app, on-line and in-depth questionnaires. The questions answered by each city are: a) what has worked, b) what has not worked, and c) how did the 2nd phase compare to the 1st.

4.4.1 Zaragoza

Questionnaire What has worked well? What has not worked well? Comparison with 1 st phase

Alpha

• A small group of the Zaragoza City Council staff collaborate actively for testing the app.

• Technical and functional requirements and also usability were deeply tested.

• The group was not heterogeneous as all players were people with technical background what could have affected the results of the test.

• As “Zaragoza Participa” app was released in November the Alpha test focused on the basic

• As in the previous phase, this activity brought up some valuable suggestions as well as some minor bugs were detected.


app functions.

In-app

• The new layout is shorter and users tend to fill it rather than skip it. Moreover, the ability to participate in a raffle acts as an incentive for filling out the questionnaire.

• There is not a way of launching manually the questionnaire in Zaragoza apps. Probably it may have helped in getting more questionnaires.

• So far the number of in-app questionnaires filled is higher than in the previous phase.

On-line

• Some citizens who joined the engagement activities filled a paper questionnaire, based on the on-line one. All of these forms have already been introduced in the on-line form.

• The number of completed surveys is rather low because in many of the places where engagement activities are conducted, there is a lack of available computers for completing the forms and users do not spontaneously fill out the forms when they return home.

• Many of the users who filled out the questionnaire did not provide their email address for further collaboration.

• Quality of answers is not always satisfactory.

• The vocabulary and orientation of some of the questions is not always suitable for all user profiles.

• Although the number of questions in the on-line form has been reduced, users still consider the process of completing it long and tedious.

• To outbalance the former inhibitor, we have campaigned more intensively for the questionnaires to be filled-in. This has led to an increased in the number of questionnaires gathered, exceeding expectations.

In-depth

• We have replaced, in this phase, the in-depth interviews by informal talks, chats and mini stories, which have been possible thanks to the engagement activities.

• Through informal discussions with people previously selected we collected some relevant testimonies in terms of value of the developed apps and IES Player. Overall quality of information matches what was intended to be collected through the in-depth interviews.

• We have focused on specific target groups

Table 10 – Feedback on questionnaires by Zaragoza

4.4.2 Bristol


Alpha

• Easy and quick to use. Generic questions and response boxes allowed for fairly broad responses.

In-app

• New design, reduced questions and single page hugely improved this experience. Consequently, it was easier to collect Reponses.

• Users still do not like random pop-up questionnaire.

• Great example of less is more. Much more effective than 1st phase.


On-line

• Number of questions was halved. No duplicate questions between in-app and on-line.

• Still laborious to find on-line. Most questions considered irrelevant by local community.

• Improved by 50% reduction in questions.

In-depth • Lack of time in 2nd phase

has limited the possibility of useful case studies.

Table 11 – Feedback on questionnaires by Bristol

4.4.3 Rovereto


Alpha • In this second phase Alpha questionnaires were not used, while the focus was on the in-depth interviews

In-app • Its new layout has been very well accepted by users, now it’s easier and requires less time to fill

• The need of a shorter questionnaire is successfully fulfilled

On-line • All the users of the beta test phase filled a paper questionnaire, based on the on-line one, and will be soon converted in the on-line form

• In the open phase, from the release on the market for the general public it is difficult to know who has the app installed and to send him to the on-line questionnaire page

In-depth

• The in-depth interviews collected gave the development team a lot of deep comments and feedback which really help in improving the apps

• Doing an in-depth interview requires a lot of time, and it is not always possible to spend so much time doing interviews

• During the first phase no interviews were done, so the improvement of the “quality” of comments obtained from this type of feedback is undeniable

Table 12 – Feedback on questionnaires by Rovereto

4.4.4 Majadahonda


Alpha • Different types of users

were involved. • There wasn’t any issue

identified. • The users were mainly

people with technical background.

In-app

• The new in-app questionnaire design has helped in boosting this kind of feedback, as it’s shorter and visually more appealing.

• The existence of a way of triggering the in-app questionnaire in Majadahonda apps may have helped in retrieving more questionnaires.

• There was a problem logging in app questionnaires in the second phase apps that has made it difficult to analyse the responses.

• The users, generally

speaking, are still reluctant to give their email for further collaboration.

• The new in-app has been a significant improvement to the one used in the first phase thanks to its interactive and shorter design. Its use among all project apps reinforces the IES Cities project image and allows comparison between the pilots.

• The pop-up rate has also been adjusted.


On-line

• Users are too lazy to fill in the online questionnaire, so the incentives have eased the way for them, as their collaboration can be rewarded somehow.

• The activities done have not led the users to the web interface enough, probably a link in the apps would have helped.

• The delay in the apps release has made it difficult to engage active users and get them to give this kind of feedback.

• The online questionnaire has been shortened in order to be quickly filled in by the interviewees.

In-depth • No in-depth interview/study

case has been carried out yet.

• The identification of active users through engagement campaigns is pretty difficult.

• As in the 1st phase, no in-depth interview/study case was carried out.

Table 13 – Feedback on questionnaires by Majadahonda

4.5. Support

This section reports on feedback relevant to how well the support mechanisms across the four cities worked during the second phase of the pilots.

City What has worked well? What has not worked well? Comparison with 1 st phase

Zaragoza

• Any reported issue has been solved.

• Users do not seem to like provided procedures for solving issues.

• As in the first phase, no issues were reported through the official channels.

Bristol

• The agreement between Bristol’s task force as to how we triaged the support request.

• No citizens appeared to use the official support system. Instead citizens would contact KWMC directly, as they knew the faces.

• Citizens circumventing the official support system. KWMC would then triage tech off to Toshiba.

Rovereto

• Thanks to the apps’ simplicity no support requests asked by users

• No comparison is possible, since the support mechanisms were not used during the 2nd phase

Majadahonda

• The use of the XPERTA system for the user support seems to have been a good decision due to its maturity, and also because allow the citizens to report any issue through it, including those related to the IES Cities project.

• Having the City Council department involved in the project in charge of dumping the issues into the IES Cities’ tracking system has probably been the most cost-effective solution.

• The users have not received enough information about the possibility of reporting IES Cities issues through XPERTA. The engagement campaigns have placed their emphasis in other topics.

• In addition, the users will be probably pointing out apps malfunctioning in Google Play as it is much faster, and thus, a “surveillance” of the apps rating and comments (and other KPIs) is still needed.

• There haven’t been users reporting problems related to IES Cities using XPERTA up to now.

• The approach followed has been practically identical to the one done during the first phase.


Table 14 – Feedback on support by consortium partners

4.6. Objectives

This section reports feedback relevant to the objectives of the four cities during the second phase of the pilots.

4.6.1 Zaragoza

Objectives Activities accomplished Main outcomes Lessons learned

• Creating an online community of like-minded individuals, which could eventually transpose into business.

• Involve developers within the platform and the services developed.

• II Hackathon Zaragoza App Store.

• Awareness of developers in the use of the platform, web interface, development tools and open data available.

• New apps developed and uploaded at IES Cities Platform.

• New datasets uploaded in the IES Cities platform.

• New users registered. • Several forms filled out.

• The IES Cities platform provides a simple and flexible set of tools for launching and organizing hackathons.

• Helping developers to promote their apps and reinforcing local community of developers.

• Open innovation sessions with developers and computing science students.

• Fostering the culture of open data and smartphone apps development.

• The swagger interface is highly appreciated by students and teachers in the field of apps development.

• Encouraging active participation of the citizens in local decision-making through the Zaragoza Participa App.

• School Visits. • Co-Creation sessions. • Open innovation

sessions. • Face-to-Face meetings

with decision makers.

• Interesting debates about city making issues have taken place.

• The knowledge about the existence of new channels has been spread out through some of the communities.

• When users are engaged since the beginning of the pilot (co-design phase), they tend to stay involved over the whole project cycle.

• Getting involvement of the citizens in their city’s life, through the four Zaragoza apps that could give them a central role in the enhancement of the services offered by their town.

• Training sessions. • The groups to which activities were addressed learnt to use the apps and in some cases gave us valuable feedback.

• “Learning by doing” has revealed as an effective way to get users involved.

• Tightening the links with project partners for expanding in time the results of the IES Cities project along future collaboration

• Multiple Project meetings and calls

• New collaboration prospects have been identified between Zaragoza and Bristol

Table 15 – Feedback on objectives by Zaragoza


4.6.2 Bristol


Healthy Office: • To allow office workers

to monitor the well-being parameters of their working environment

• Intro workshops with 3 organisations.

• 1-month pilot with staff. • Promotion on social

media

• Learning to develop the app further ready for further funding and citywide rollout.

• Android only limits the audience.

• Extra training for staff unused to apps.

• Desire to see sensor data in the app.

• Need buy-in from right manager to access and take action on staff input

YouDecide: • We are looking to

provide citizens with a tool for casting their votes in response to different issues or questions within the city: e.g. the best place for a new children’s play area, local projects to receive funding, key priorities for police in the community.

• Meeting with NP coordinators, NP managers and communicating with the e-consultation team to gather content for app.

• General outreach. • Community meeting

and events including the Love Filwood event

• Promotion on social media

• Lots of citizens interested in light tough approach to decision making in an app.

• Building relationships with relevant council departments that could use a digital tool for citizen involvement.

• Lot more development time needed with council with gathering content.

• Building understanding of potential use.

• Need longer development time for iterative process to respond and develop based on feedback to create better app for buy-in and better usability.

• Need to test app robustness for if used for citizen voting

Table 16 – Feedback on objectives by Bristol

4.6.3 Rovereto

In this second phase some objectives still remain mainly the same as those of the first one. In particular the first and the last of them, which deal with an increase of the interest in the ICT services and to promote a culture of active participation in the town’s growth.


• Increase the interest of citizens the ICT services

• Face to face meetings with interested people

• Public press conference to promote the app Rovereto Percorsi

• Many people interested in discovering the potential of these technologies and the services offered by their town’s Administration

• Well informed people could be very interested in discovering these kind of new services

• Involve developers within the platform and the services developed

• Stages and internships organized for high school students, during which they learnt to use and manage data and platforms and to create new apps

• Good response from some students who attended those stages during the summer period and realised a new app based on the data coming from their town’s territory

• Young people and students are very interested in discover and manage the new technologies, so it could be useful both for them and their town to involve them in


collaborative works and projects

• Rovereto Percorsi: Make people aware about routes and paths offered by the town and its territory, improving citizens and visitors’ knowledge about the beauties of nature and all the culture and sport sites of the town by the use of a smartphone app

• Public press conference to promote the app Rovereto Percorsi

• Some meeting with Rovereto’s tourism bodies have been done in order to propose a collaborative network within the town

• App users are allowed to co-produce information, enriching each path with comments and sharing their pictures

• Warm response from the tourism bodies, who share some information about the project and the app on their websites

• Some contents have been posted in-app by users, in particular during the Christmas Market period, when the app has been enriched with a special path about the Market

• It could be very useful to realise a network of services thanks to the joint work of many bodies of the town. In this way it could be really possible to push strongly on the town’s life quality itself

• A dynamic app, always in evolution is very well seen by users, who are encouraged to share their pictures and comments fostering the app’s improvement

• Rovereto Segnala: Identify some “ambassadors” who daily use the app to highlight in real time the town’s problem to the Council, strongly helping the town’s quality improvement

• Meeting with the traffic wardens to present them the app showing its functionalities

• Traffic wardens accepted to adopt the app as one of their daily working tools

• Some public authorities and bodies could adopt this app as a useful tool for their work, fostering at the same time an active collaboration with the Council

• Promote a culture of active participation in the improvement of the services offered

• Users were asked to freely report each kind of comment and suggestion to improve the apps

• Some very interesting and helpful feedback collected, in particular through in-depth interviews

• If not forced to do it, people often spontaneously help the improvement giving their opinion

Table 17 – Feedback on objectives by Rovereto

4.6.4 Majadahonda


• To stimulate and promote the commercial activity of the city allowing traders to reach a higher audience based on the dissemination of specific events organized by them.

• Meeting with traders

• Many shopping centres have shared their space for promoting the second phase IES Cities apps for free.

• New communication channel.

• New data uploaded into the platform.

• Those meetings are the best way to engage partners in the project

• To promote sports and the sport facilities. To engage in the project the sport centres of Majadahonda.

• Meeting with the sport centres managers.

• Stand in the Monte del Pilar Running Race

• App downloads. • Alpha testing. • New data uploaded into

the platform.

• The collaboration of the communication department is crucial for a successful promotion campaign.

• The data upload should be automated.

• To inform students about the different apps

• Talks in technical seminar Urban Tech.

• We engaged many students.

• Take this opportunity to establish long-term


and possibilities of the platform.

• University/High school visits.

• Coder Dojo/ Hackathon

• Downloads. • Feedback retrieval.

relationships and foster further collaboration.

• To promote the project and the apps.

• Social networks activity, press releases.

• IES Cities Merchandising (power banks, memory USB keys...)

• Stands in Shopping Centres.

• RECI Video Conference

• Awareness of IES Cities project.

• App downloads. • Partners for

collaborating in open data projects in the future?

• App-specific engagement campaigns are more effective.

Table 18 – Feedback on objectives by Majadahonda

4.7. Service Exchange

In this section each city presents the feedback relevant to the service exchange that took place for the second phase of the pilots.

4.7.1 Zaragoza

Service name: Zaragoza Participa

Shared with: Bristol (YouDecide)

Development process:

App interface was developed by Zaragoza and then Bristol re-used the code produced by Zaragoza and also adapted the back-end to the Bristol’s specific requirements. Also made some changes to the look and feel of the app and added specific functionalities for Bristol


Cooperation between city councils has been fluent and overall development time has been reduced thanks to this collaboration

What hasn’t worked? Some problems aroused when adapting the back-end of the app to Bristol’s needs. Collaboration between cities involves a lot of interaction between cities and this induced some delays to the development process

Lessons learned It is hard to work with code created by other developers and it takes time to understand it. Collaboration between cities in the development of an app slows down the development process

Table 19 – Feedback on service exchange by Zaragoza

4.7.2 Bristol

Service name: YouDecide

Shared with: Zaragoza (ZGZ Participa)



Zaragoza liked the idea of the 1st phase Bristol app Democratree, where citizen have the power to make suggestions for new tree locations. They decided to create a new app that extends this idea and allows Zaragoza citizens to voice their opinion on various issues of local interest. Zaragoza and Bristol worked together on the requirements for the new app and, then, Zaragoza developers took the lead in developing the app. Bristol developers received the first version of the new app and adapted it for Bristol, providing visual, operational changes and of course Bristol specific content


• Development time required for Bristol was minimised, as significant part of the code was reused through the Zaragoza version of the app.

• The visual design of the app and its performance were of a very high standard.

What hasn’t worked?

• There was significant delay in receiving the first stable version of the app, so available time for further development by Bristol was reduced as a consequence.

• Some agreed features were missing or were different than anticipated, thus requiring additional and unplanned development time by Bristol

Lessons learned

• A concrete design and feature agreement needs to be implemented as early as possible and then followed as closely as possible, unless otherwise agree by both parties at the earliest possible opportunity.

• Delays on one side adversely affect development time and opportunities on the other end.

Table 20 – Feedback on service exchange by Bristol

4.7.3 Rovereto

Service name: Zaragoza, tu opinión importa

Shared with: Rovereto (Rovereto Segnala)


We took the design of issues visualization, insertion and navigation but we redeveloped the code using a different framework (Ionic). We also maintained the Open311 standards for civic issue tracking


Keeping the same graphic design allowed to start the development as soon as possible saving a significant amount of time.

What hasn’t worked? Rovereto development team had little knowledge of hybrid applications (HTML), so it was a steep learning curve to develop this new app using HTML, JS and CSS.

Lessons learned Trying to understand the code from other developers is always a hard challenge

Table 21 – Feedback on service exchange by Rovereto

4.7.4 Majadahonda

Service name: Majadahonda In-Route

Shared with: Zaragoza Mapas (Zaragoza Collaborative Maps)



Majadahonda liked the urban service implemented in Zaragoza during the first pilot phase and named “Zaragoza Mapas”. Therefore, the starting point was the “Zaragoza Mapas” application. In the case of Majadahonda, this application was designed by using as basis the cartographic maps available In Majadahonda and Google Maps. The functionality of this urban service is similar to the one provided in Zaragoza Maps focusing on creating a very friendly and intuitive application.


Taking as basis a previously implemented urban application (“Zaragoza Maps”) allowed to speed up the development process and put the focus on the creation of maps and markers new dataset including private, public and collaborative maps. By analysing the gaps and feedback from phase I, focus was also put on the authorization and authentication issues within the urban service. Collaboration between EUROHELP and TECNALIA has resulted in a friendly urban application.

What hasn’t worked?

Collaboration between development teams of TECNALIA and EUROHELP has worked pretty well. The only problem to highlight is related to the dataset that Majadahonda Maps application uses as basis. In order to create a successful application it is needed that the Maps dataset contains high value maps. TECNALIA has created a semi-automatic process to allow Majadahonda City council to feed this application with new maps. So far the number of maps created by Majadahonda City council is too low.

Lessons learned

Sometimes is difficult to start the development by using SW code created by other developers. A close communication among development teams is really necessary but also a close collaboration with the city council to give city council the possibility to provide new datasets.

Table 22 – Feedback on service exchange by Majadaho nda

4.8. Feedback on data sharing between apps & cities

Due to the different nature of apps deployed across the four participating cities, as well as due to the different issues that are relevant to each city, there was no data sharing between app and cities. What was shared was a number of apps among cities, such as the voting app that was shared between Bristol and Zaragoza.


5. Conclusions

Deliverable D3.3 ‘Second Phase Execution and Monitoring’ has discussed the rich volume of evidence collected during the 2nd phase of the IES Cities project. D3.3 looked at both the quantitative and qualitative data collection processes. It then reflected on what has worked well, what has not worked well and what was different compared to the 1st phase of the project. D3.3 focused around the specific areas of user experience, logging, software, activities, questionnaires, support, objectives and service sharing

It is not the place for this deliverable to analyse in detail the results obtained during the 2nd phase. Rather it is a place to collect feedback from the cities and their technical partners about all aspects of the project particularly related to the citizen experience when using the IES Cities platform and its apps.

It is apparent that all the cities have been very active when working with their citizens. This report has a great deal of feedback from the City Councils and indirectly from citizens. There is also feedback from the technical partners and, in contrast to the first phase, there is also feedback from developers; in the 1st phase there was not appropriate scope for gathering developer feedback. This omission was removed in the 2nd phase, as the online questionnaires were adapted to gather developer feedback as well.

The level of activity in the second phase has clearly brought rewards as to how engaged the citizens have been and ultimately how active they have been in terms of app usage. Additionally, the quantitative measurement logging system appears to have worked well, as have the qualitative questionnaires; the first phase questionnaires had been deemed too complex and sometimes irrelevant by the users and to address this the questionnaires were simplified for the 2nd phase. Overall the whole IES cities platform and the apps appear to have worked well, especially after several tweaks and improvements were applied as a result of feedback collected during the first phase of the project.


6. Comments from External Reviewers

6.1. Reviewer 1 – UD

February 26th, 2016

Issue Yes No

Score

(1=low to 5=high)

Comments

Is the architecture of the document correct?

X 5

Does the architecture of the document meet the objectives of the work done?

X 5

Does the index of the document collect precisely the tasks and issues that need to be reported?

X 5

Is the content of the document clear and well described?

X 5

Does the content of each section describe the advance done during the task development?

X 5

Does the content have sufficient technical description to make clear the research and development performed?

X 5

Are all the figures and tables numerated and described?

X 4 Appendix tables do not have caption nor are numerated

Are the indexes correct? X 5

Is the written English correct? X 5

Main technical terms are correctly referenced?

X 5

Glossary present in the document?

X 5

Unai Aguilera

[email protected]

University of Deusto


6.2. Reviewer 2 – ZIB

March 1st, 2016

Issue Yes No

Score

(1=low to 5=high)

Comments

Is the architecture of the document correct?

X 5

Does the architecture of the document meet the objectives of the work done?

X 5

Does the index of the document collect precisely the tasks and issues that need to be reported?

X 5

Is the content of the document clear and well described?

X 4 Minor issues, see comments

Does the content of each section describe the advance done during the task development?

X 4 Minor issues, see comments

Does the content have sufficient technical description to make clear the research and development performed?

This is not a technical deliverable about research and development

Are all the figures and tables numerated and described?

X 4 Some page breaks need adjustments so captions do not fall off to the next page; appendix tables are not numbered but described and therefore ok

Are the indexes correct? X 5 References should be linked from the text though

Is the written English correct? X 3,5 See comments, some fixes inline

Main technical terms are correctly referenced?

X 4 Abbreviations could be at least once fully explained before being used (despite the list below)

Glossary present in the document?

X 5

Nico Kruber

[email protected]

ZIB


7. Abbreviations

IES Cities Internet Enabled Services for the Cities across Europe

KWMC Knowle West Media Centre

BCC Bristol City Council

FBK Fondazione Bruno Kessler

API Application Programming Interface

App Application

CSS Cascading Style Sheet

FAQ Frequently Asked Questions

HTML Hypertext Markup Language

JSON JavaScript Object Notation

KPI Key Performance Indicator

REST Representational State Transfer

WPX Work Package X


8. References

[1] CIP ICT-PSP IES CITIES project deliverable 4.1: “Evaluation Plan” accepted version 1.0 IES Cities consortium, October 2014

[2] CIP ICT-PSP IES CITIES project deliverables 5.2.x: “Zaragoza, Bristol, Rovereto and Majadahonda Questionnaires” accepted version 1.0 IES Cities consortium, October 2014

[3] CIP ICT-PSP IES Cities project “Description of Work”. IES Cities – Internet Enabled Services for the Cities across Europe, March 2014 (New accepted version after amendment submission)

[4] CIP ICT-PSP IES CITIES project deliverable 3.1.2: “Pilots Planning v2” accepted version 2.0 IES Cities consortium, July 2015

[5] IES Cities Player app. Last visit: February 2016 https://play.google.com/store/apps/details?id=eu.iescities.player

[6] CIP ICT-PSP IES CITIES project deliverables 5.1.x v2: “Zaragoza, Bristol, Rovereto and Majadahonda Engagement Plan” accepted version 2.0 IES Cities consortium, March 2015

[7] IES Cities swagger. Last visit: February 2016. http://iescities.com/IESCities/swagger/

[8] CIP ICT-PSP IES CITIES project deliverable 4.2: “Report on the monitoring of results”, IES Cities consortium, February 2016


9. Appendix - Monthly engagement tables

9.1. Zaragoza

• July 2015:

MONTH PLAN (BEFORE EXECUTION) EVALUATION (AFTER EXECUTION)

Engagement activity

Engagement Level App Target

audience Drivers and Incentives Description Expected

results Quantitative

Outputs Comments

II Hackathon Zaragoza App

Store

Active Use

IES Player

Registered Users

Catering, economic

prizes: 1.250€

Electronic devices:

Raspberry Pi and Arduino

kit

II Hackathon Zaragoza APP

Store.

http://zaragozappstore.exitae.es

• 20 attendees • 2 registered

users in the Web Interface

• 2 apps uploaded in IES Player

• 3 forms filled

• 28 attendees • 8 registered

users in the Web interface

• 8 apps uploaded in IES Player

• 1 new dataset • 8 forms filled

The meeting was a success. The teams worked collaboratively and new applications were created. Some attendees demanded more appealing datasets. Developers told us a very positive opinion about IES Player and Web interface for developers Attendees filled out the form for developers to provide feedback on the platform.

Feedback

First Contact Web Interface

• August 2015:

Holiday period: no activity conducted.

• September 2015:


Engagement activity


audience Drivers and Incentives Description Expected results Quantitative

Outputs Comments

Social Media Awareness IES Player Active Users N/A Dissemination activities through

social media, newsletter and openurbanlab blog

Awareness of the new apps developed for the citizens

No, at the moment.

Meeting with Delicias Association (Steering

Committee) First Contact Delicias 2.0 Public

Authorities N/A To show how it works the new

app and its feedback for improvement

Feedback of the use of the new app from the Steering

Committee and its commitment to collaborate in

7 attendees, from

Ebropolis, Delicias

The meeting was successful. They show a really interest to

use the apps and to participate in the awareness campaign in


the engagement campaign and in the workshops

Association and the City

Council

its neighbourhood and to contribute with its feedback with the improvement of the

app

• October 2015:


Engagement activity



Outputs Comments

Social Media Awareness All Zaragoza's Others N/A

Dissemination activities through social media, newsletter and

Open Urban Lab blog. Post in open urban lab

communicating the upcoming release of the apps

(http://openurbanlab.es/2015/11/05/apps-para-una-ciudad-abierta/)

Awareness of the new apps developed for the citizens N/A

First workshop Delicias 2.0 Active Use Delicias 2.0 Public

Authorities N/A These activity was address to civil

servants from different areas of city hall. The aim was to show internally how the apps work.

Later, it is intended to extend to

the whole city.

5-10 attendees. 5

Second workshop

Delicias 2.0 Active Use Zaragoza App

Store Public

Authorities N/A 5-10 attendees. 7

Informal meeting First Contact Zaragoza App Store Developers N/A

Informal meeting with the managers of two business

accelerators to show them the new app and ask for their

cooperation to promote it within their community.

Placing "Zaragoza App Store" as an essential developer’s repository.

The application aims to be an important element to be up-to-date with available apps related to the city and also to add any new app developed by the entrepreneurial

community of Zaragoza.

9

• November 2015:



Engagement activity



Outputs Comments

Social Media Awareness All Zaragoza's Others N/A Dissemination activities through social media,

newsletter and openurbanlab blog

Awareness of the new apps developed

for the citizens 700 views

Feedback retrieval from the beta test friendly users

First Contact Delicias 2.0

Public Authorities N/A

Meeting with the heads of the neighbourhood association and university teachers involved

in the application. The aim is to deeply test the app and set the

roadmap for the workshops to be held in December with the different target groups of

the application.

4 attendees 7 attendees

• December 2015:


Engagement activity Engagement Level App Target


Outputs Comments

Promoting and raising

awareness of the apps

Awareness

All

Zar

agoz

a's

Others N/A

Two press notes in one of the most relevant

newspaper of Zaragoza. Dissemination activities through twitter and three

new posts at openurbanlab blog: http://goo.gl/0YGxFd,

http://goo.gl/zi6AeJ and http://goo.gl/HP3Wqp

We intend to attract citizens to

scheduled activities and get their interest to download the apps. We use

social media and mainly press

notes to disseminate

training activities are planned for

December.

By Dec, 8th already 70 people have downloaded

the apps

http://www.elperiodicodearagon.com/noticias/aragon/capital-tiene-ya-propia-central-

aplicaciones-moviles_1072603.html


First workshop Delicias 2.0 Feedback

Del

icia

s 2.

0

Active Users

Merchandising gifts handed out to the students who participated in the

workshop. Sense of

belonging for having participated in the design of a

specific application to their

neighbourhood.

This first workshop will be directed mainly to the neighbours but also

students of the department of geography at the

university will come to participate in the activity. The workshop focuses on these groups since were

the ones who were involved as co-designers

from the early stages of the app.

The idea is that after participating in the

workshop, attendees act as driving forces of the app in their social circles in order to spread their use among

other citizens.

We expect at least 10

attendees. During the

session we will record a video

that will be public as soon properly

edited. The purpose of the video is to show the high

degree of participation and motivation of the

participants.

We had 17 students who were actively involved in

the activity. Some attendees filled out a survey on the app. There

were several proposals to

develop new urban services in the

fields of parking, recycling, public transportation...

During the workshop, a video was recorded in order

to document the activity. Besides, we wrote two posts

in openurbanlab blog: http://goo.gl/HP3Wqp,

http://goo.gl/zi6AeJ

Second workshop Delicias 2.0 First Contact Delicias

2.0 Active Users

Change a routine day of school for

something different and interesting.

Some USBs with the project logo will

be distributed.

The workshop is aimed at two groups of students

belonging to the neighbourhood "Delicias". The feeling of belonging to

the community and the possibility to make (or support) proposals for

improving their neighbourhood will be the

main axis for its commitment to using the

app. In addition to using and

carefully explain the application, we will take

advantage of the session to discuss the project IES

Cities as a whole.

15 attendees (expected). Awakening

students' interest in such topics as: What is IES Cities

project? How is a

participatory project?

Testing the app

About 100 students attended the workshop

Even though the activity lasted more than 4 hours,

the workshop dynamics was agile and students actively

participated, made interesting suggestions and made constructive feedback

about the app. We all learned a lot and

certainly it was an enriching experience.


Active Use Zaragoza App Store Developers

See first-hand how the apps was developed.

The workshop will take place in a school in the

neighbourhood of Delicias. It will be targeted to

computer science students and it will be conducted in

collaboration with EUROHELP.

We want to show students the methodology used to

develop the application, the use of open data, the IES

Player etc., in an entertaining way and with a

real case.

15 attendees (expected) Awakening

students' interest in such topics as:

What is a European project?

developing an application IES

Cities Application

testing

About 25 students attended the workshop.

A number of improvement were

brought up by future developers. EUROHELP has turned some of

those suggestions into a to-do list to

implement in future releases. Active Use IES Player Developers N/A

Placing IES Player as an essential repository where apps related to the city and any new app developed by

the entrepreneurial community of Zaragoza

could be uploaded.

Active Use Web Interface Developers N/A

Placing Web Interface as a tool for developer

community.

• January 2016:


Engagement activity


audience Drivers and Incentives Description Expected results Quantitative Outputs Comments

Promoting the IES Player Active Use IES Player

Developers N/A We will start to promote the III

Hackathon Zaragoza App Store (April-May 2016).

We will meet with the developers of the two local incubators, as well as

Increasing the degree of developers

participation for the next hackathon as well

as the number of

N/A

New datasets, ideas, prizes and themes

were defined in order to increase the impact of the hackathon and

Active Users N/A


Registered Users N/A

with developers linked to eTOPIA_, to inform them of the upcoming

hackathon.

dataset used. Promoting the use of

the platform IES Cities.

the use of IES Cities Platform.

Setting strategy for school visits

Feedback

Complaints &

Suggestions Public Authorities N/A

Our "traditional" school visits begin. (From April to May)

As the official visits are scheduled

between April and May we are planned a special edition in order to promote the apps

among students.

N/A

We set up new contents for increasing

awareness from the users. We expect 10

schools to be visit next month.

Collaborative Maps

Strengthening the promotion

of apps First Contact All

Zaragoza's General Public

N/A

We keep on promoting the apps through social media tools to raise the interest of citizens to download

the applications.

A continuous increase in downloads / users

2 new posts were wrote in the

openurbanlab.blog

• February 2016:

For when we are filling this document we do not have yet the quantitative results of foreseen activities for February and March.


Engagement activity



Improve your neighbourhood and your app (Feb, 11th -

IES Tiempos modernos)

Active Use Collaborative Maps

Active Users

Participate in contest to win 10 DIY Hardware kits

as a prize at a special event in

Etopia Center for Arts and

Technology

Educational activity with high school students to promote digital

engagement through participatory city making using the tools

"Zaragoza Maps" and Zaragoza Participa and to identify usability

issues

70 attendees, creation of new maps and new

proposals. Identification of usability issues

Feedback Delicias 2.0 Registered Users


IES Tiempos modernos)


Active Users




Educational activity with "vocational training" students to promote digital engagement through participatory

city making using the tools "Zaragoza Maps" and Zaragoza Participa and to identify usability

issues.





Technology


IES Avempace)


Active Users




Technology

Educational activity with ICT high school students to promote

"Zaragoza Maps" and Zaragoza Participa and to identify technical

issues


proposals. Identification of technical issues



IES Santiago Hernández)


Active Users




Technology



issues




Improve your neighbourhood

(Feb, 17th - Santa Ana

School)

First Contact Collaborative Maps

Active Users




Technology



issues



First Contact Delicias 2.0 Registered Users


(Feb, 18th - IES El Portillo)


Active Users




Technology



"Zaragoza Maps" and Zaragoza Participa



First Contact Delicias 2.0 Active Users


(Feb, 19th - IES El Portillo)


Active Users





"Zaragoza Maps" and Zaragoza



Active Use Delicias 2.0 Active Users



Technology

Participa


(Feb, 24th - IES Antonio Machado)


Active Users




Technology





proposals



(Feb, 25th - Santo

Domingo de Silos School)


Active Users




Technology





proposals


Improve your app (Feb, 26th

- IES Angel Sanz Briz)

Active Use Zaragoza App Store

Active Users




Technology

Educational activity with high school students to promote the use and

feedback about the new Zaragoza App Store app. The feedback is

retrieved through the Complaints and Suggestion app developed in the 1st

phase

75 attendees, identification of issues

Feedback Zaragoza App Store

Active Users

Active Use Complaints & Suggestions

Active Users


IES Miguel de Molinos)


Active Users




Technology



issues




• March 2016:

For when we are filling this document we do not have yet the quantitative results of foreseen activities for February and March.



Engagement activity



Improve your neighbourhood and your app (Mar, 2nd -

Corazonistas School)


Active Users

Participate in contest to win 10

DIY Hardware kits as a prize at a special event in Etopia Center for

Arts and Technology




issues




Improve your neighbourhood and your app

(Mar, 3rd - Corazonistas

School)


Active Users

Participate in contest to win 10

DIY Hardware kits as a prize at a special event in Etopia Center for

Arts and Technology

Educational activity with "vocational training" students to promote digital engagement through participatory

city making using the tools "Zaragoza Maps" and Zaragoza Participa and to identify usability

issues.




Promoting the apps Active Use Web

Interface

Developers N/A Second meeting intended to continue the preparations for the

upcoming III Hackathon Zaragoza App Store

Defining in detail the program of the III Hackathon with

stakeholder involved.

Public

Authorities N/A

Final Event High Schools Awareness

Collaborative Maps

General Public Event and social

network campaign

Final event and contest to win 10 DIY Hardware kits to the "best digitally engaged" High School.

Event held in Etopia Center for Arts and Technology

150 attendees, creation of new maps and new proposals.

Identification of usability issues

Delicias 2.0 General Public


9.2. Bristol

• July 2015:


Engagement activity



Outputs Comments

Alpha testing Feedback Bristol Voting Public

Authorities

Stakeholders who want to engage

citizens in decision-making

Send Alpha test to Neighbourhood Manager,

Neighbourhood Partnerships and council’s engagement

team and arrange to meet or receive feedback by email

5 downloads 0 No app yet created

Alpha testing Feedback Bristol HealthyOffice Others

People want to improve their

wellbeing, organisations

commitment to wellbeing and

improved productivity

Send Alpha test to key BCC, KWMC, Toshiba staff already

aware 5 downloads 5 apk installations

/ no downloads No downloads as direct apk

Workshops First Contact Democratree Others

Interactive way to decide where to

plant a tree digitally, and get votes to make it reality

Workshops with One Tree Per Child and Greenfields School

25 new users registered and planted

a tree 12 new users

One tree workshop happened. Greenfields delayed.

Outreach & workshops First Contact MyBristol General

Public Usefulness of the

apps to the audience

Various workshops and events including one day at @Bristol, and community

groups

25 new users (each month, will include

downloads and registered users, as many people have

iPhones and will use the web apps), 25 new

posts / active use

8 new users (RK check)

College workshop went ahead but at Bristol moved to August.

General promotion

Planning First Contact Bristol Voting General Public

Usefulness of the apps to the audience

Plan events and workshops for voting app in September

launch and for key stakeholders testing

4 key advocates 0 No app yet created


Communication / Newsletters Awareness All Bristol's General

Public

Relevant to their interests for which newsletters they

read

Begin promoting the apps to related newsletters, e.g. The Wellbeing Newsletter, The

Knowledge, Our City, Happy City, Good Food Network, so they can be included when

the apps are out

Agreed 4 newsletter entries for future, 1 per

app

9 feature about MyKW in the

knowledge from Feb 2015. Also

MyBristol.

MyKW in every issue of The Knowledge (photo page). Nothing on other apps as are not working. HealthyOffice press release being released for upcoming pledges.

Xserveprojects/IES Cities.

• August 2015:


Engagement activity



Workshops First Contact MyBristol Others

Interactive way to decide where to plant a tree

digitally, and get votes to make it reality

Engage with young people at STEAM event

10 new users

Ad hoc engagement with young people during

STEAM week.

Outreach & workshops First Contact MyBristol

General Public


Outreach event @Bristol science

centre

25 new users

One-day outreach @Bristol, engaging approx. 70

people. 3 new posts and numerous signs ups.

Outreach & workshops First Contact Democratree General

Public Usefulness of the apps

to the audience

Outreach event @Bristol science

centre

25 new users

One-day outreach @Bristol, engaging approx. 70 people

Feedback from @Bristol: "Thank you so much for joining us yesterday. John was telling me

how much visitors were enjoying your activities, and Jen from our marketing team

loves Democratree!"

Alice Barber, Live Lab Officer

Workshops First Contact MyBristol General Public


Mini Maker Fair at @Bristol

New content

Exposure of MyBristol brand to a city wide

audience.

• September 2015:



Engagement activity



Alpha testing Feedback Bristol

YouDecide Public

Authorities

Stakeholders who want to engage

citizens in decision-making

Send Alpha test to Neighbourhood Manager,

Neighbourhood Partnerships and council’s engagement

team and arrange to meet or receive feedback by email

5 downloads Delayed until Nov.

Alpha testing Feedback Bristol HealthyOffice Others

People want to improve their

wellbeing, organisations

commitment to wellbeing and

improved productivity

Send Alpha test to key BCC, KWMC, Toshiba staff already

aware 5 downloads Delayed until Nov.


Public Usefulness of the apps

to the audience

Various workshops including Celebrating Age and Open

Doors and Beginners Computer class

25 new users (each month, will include

downloads and registered users, as many people have iPhones and will use the web apps), 25 new posts / active use

4 sessions introducing to computer beginners classes - 30 people. 5

new users.

Planning First Contact Bristol YouDecide

General Public


Plan events and workshops for voting app in

September/October launch and for key stakeholders

testing

4 key advocates Delayed until Nov.

Communication / Newsletters Awareness All Bristol's General

Public

Relevant to their interests for which

newsletters they read

Begin promoting the apps to related newsletters, e.g. The Wellbeing Newsletter, The

Knowledge, Happy CityGood Food Network, and Smart

Circle website.

Agreed 4 newsletter entries for future, 1 per

app

MyKW continues to be promoted in The Knowledge. Smart Circle has

received an update on all IES apps across the cities. We will make

contact with the Happy City good Food Network and the Wellbeing Newsletter - 2 Newsletter entries

have been confirmed already.


Public

Sharing local news. Celebrating local

successes Knowle West Fest

Have initial conversations with 30 people, give

booklets so they can sign up in their own time and

see the event being

8 posts about the festival either the day before of the day of. Spoke

with approximately 25 people and handed out booklets.


documented in real time

Outreach & workshops

Active Use Democratree General Public

Introducing people to tree planting in Bristol and trying to inform council decisions on

this subject.

One-to-one sessions with community members, including people from Bristol Tree Forum

and the Kebele Centre.

5 new users

Blocked from Bristol Tree Forum due to One Tree running out of

money. However there is interest on using in the future, but not with

One Tree project.

Outreach & workshops First Contact Democratree General

Public

Introducing people to tree planting in Bristol and trying to inform council decisions on

this subject.

Workshop at Celebrating Age Festival

10 new users from the event, hand out booklets for people to sign up in

their own time

Cancelled due to One Tree running out of money. However there

interest of using it in the future, but not with One Tree project.

planning First Contact Bristol HealthyOffice

Public Authorities

Introduce app to group for testing and

feedback

Plan workshop to introduce app to members of Toshiba,

KWMC, BCC and the Wellbeing Charter for feedback

Decisions to be made on development of the app. Delayed until Nov.

outreach & workshops First Contact Democratree Others

Introduce Bristol Tree Forum to

Democratree, dispel myths about the use of the app, discuss ways

of moving forward together

Presentation at Bristol Tree Forum

10 new users, potential partnerships and ideas

for ways forward generated.

Blocked from Bristol Tree Forum due to One Tree running out of

money. However there interest of using it in the future, but not with

One Tree project.

• October 2015:


Engagement activity



Workshops First Contact Democratree Others Interactive way to decide where to

plant a tree digitally, and get votes to make it reality

Workshops with schools e.g. Greenfields School

25 new users registered and planted

a tree

23 new users. 0 trees planted.

Meeting First Contact Democratree Others Interested in taking over the app after IES.

Meeting with Tree Forum to discuss potential partnerships.

Sustainable future for app

Meeting First Contact Democratree Others Interested in taking over the app after IES.

Meeting with University of Bristol to take over the app.

Sustainable future for app


Workshops First Contact MyBristol General Public Local app 3 workshops with Computer

Beginners 29 aware of app

Workshops Active Use MyBristol General Public Local app

Workshop with Knowle West Health Association (cooking

attendees) 3 new users 3 new users

• November 2015:


Engagement activity



Workshops Active Use MyBristol General Public Local app 3 workshops with

Computer Beginners 5 new users 3 new users

Workshops Active Use MyBristol General Public

Documenting their project

Charity challenge team training 5 new users 4 new users

Workshops First Contact Bristol HealthyOffice Others Healthy work life

Beta testing with Toshiba, KWMC and

BCC staff

Feedback to develop the app further and

agree trial for Dec/Jan 8 people at workshop

Workshops First Contact MyBristol General Public

Documenting their project

Junior Digital Producers shown app 4 new users 5 new users

Workshops First Contact Bristol YouDecide

Public Authorities Local services use Beta testing with BCC

staff

Feedback for further app development and trial neighbourhood and voting options

4 people attended

Most people had iPhones! Also hard to find app by name on

Google Play Store due to different results for

different Android versions.

Workshops First Contact Bristol YouDecide

General Public

Moving local plans forward

Beta testing with community - public and

organisations

Feedback for further app development and interest in further use

when ready

6 people attended Most people had iPhones!

Event First Contact MyBristol General Public

Photo competition to win Christmas

decorations

Circulating the KWMC AGM with Android device introducing

people to or encouraging existing

5 new users 3 new users. 3 new people with

awareness. 17 new posts (active use)

Poor internet at Green Business Park limited

update


people to post photos.

Event First Contact MyBristol Others

Finding out more about user

engagement and citizen-centred design

- and how it could impact their work

Attending NextGen conference 2015 and

awards ceremony at the House of Lords (where MyKW West had been

shortlisted)

At least 20 new people aware of the app, development process and IES Cities platform

10 organisations / entrepreneurs engaged in conversations about the app and development process; all

delegates at awards ceremony (approx. 80) aware of the app due to

nominee announcement and inclusion in an information pack

• December 2015:


Engagement activity



Outputs Comments

Meeting First Contact Bristol YouDecide

Public Authorities

Reach wider audiences in

Neighbourhood Partnerships.

Meeting with Neighbourhood Coordinators about digital services and

using our apps.

2 of the team working with us on introducing the apps to their neighbourhoods. Also begin discussions for post-project

exploitation.

Workshop Active Use Bristol HealthyOffice Others Healthier work life Workshop with BCC staff 10 new users

Workshop Active Use Bristol

HealthyOffice Others Healthier work life Workshop with Toshiba staff 20 new users

Workshop Active Use Bristol HealthyOffice Others Healthier work life Workshop with KWMC staff 20 new users

Meeting First Contact Bristol YouDecide Others


regeneration planning.

Introducing the voting app to Knowle West Futures

Group are interested in using it to engage the public in the New Year

Meeting First Contact

Bristol YouDecide

Others


regeneration planning.

Introducing the voting app to organisations involved in the multi-

agency group Knowle West Together, with a view to using it at an event in

2016

Group are interested in using it to engage the public in the New Year

• January 2016:



Engagement activity Engagement Level App Target

audience Drivers and Incentives Description Expected results

Quantitative Outputs Comments

Office pilot Active Use Bristol HealthyOffice

Registered Users Healthier work life Pilot running in Toshiba, KWMC and

BCC 50 active

users 99 new users Regularly checking use and offering refreshers

Event Active Use MyBristol General Public

To share pictures of the glow event

General outreach at the Glow event (Thursday 20th Jan) 2 new users 1 new user

Outside, so device needs good mobile internet

connections

Workshop X4 Active Use Bristol

HealthyOffice Registered

Users Healthier work life Introductory workshop for the

Healthy Office app for KWMC staff 10 active

users 17 active users

One-on-one training Active Use Bristol

HealthyOffice Public

Authorities Improving wellbeing at

work for staff team

Introductory workshop for Bristol City Council team leader for the

managerial role for Healthy Office

1 active manager

1 active manager

Manager to check daily stats and to be sent monthly report from

Toshiba

Office Pilot Active Use Bristol HealthyOffice

Registered Users

Healthier work life Pilot running in Toshiba office 5 - 10 active users

7 active users

In app and online evaluation questions

answered Feedback Bristol

HealthyOffice Active Users

To get feedback from users to improve the

app

Ask users to complete in-app and online IES Cities questionnaires 5 completed 17 completed

Outreach First Contact Bristol YouDecide

General Public

Suggesting ideas and voting on the choices in

your community

(27th Jan) - Would Like To Meet event. Pilot test YouDecide with

young people

20 new users 36 new users Splitting 6 month new user

target over 2 months.

• February 2016:


Engagement activity




Outputs Comments


General Public

Suggesting ideas and voting on the choices in your

community

Outreach to general local community discussing what residents think of the official statistics relating to KW and

Filwood - Love Filwood event (4th Feb)

20 new users 32 new users Splitting 6 month new user target over 2 months.

Office pilot Active Use Bristol HealthyOffice

Registered Users Healthier work life Pilot running in Toshiba, KWMC and BCC 50 active users 127 active

users Regularly checking use and

offering refreshers


Film Feedback Bristol

HealthyOffice Active Users

To create a film about the Healthy

Office story

To make a case study film covering all three office pilots for Healthy Office

1 completed film for Healthy

Office

1 completed film

Film to be uploaded on YouTube, made available on

IES Cities website, and will be used for dissemination.


General Public

Suggesting ideas and voting on the choices in your

community

General outreach to start once Neighbourhood Coordinators add the Neighbourhood choices to vote on.

50 active users 65 active users

Splitting 6 month new user target over 2 months.

9.3. Rovereto

• July 2015:


Engagement activity


audience

Drivers and

Incentives Description Expected results Quantitative Outputs Comments

Rovereto's new apps

presentation Awareness

Rovereto Segnala

General Public

Personal contact with interested people presenting the new apps developed in Rovereto. Informative will be given with

a deep guide to download and install the apps and a description of their main

functionalities

20 people involved, 20 downloads

20 testers engaged, 20 downloads

In depth interviews with testers planned in October after two

months of test Rovereto Percosi

Developers engagement

and internship at FBK

Active Use Web Interface

Developers

2 high school students attended an internship at FBK, working in the

SmartCommunity Lab in close contact with lab developers (other 2 students

have been involved in June). They test the API platform using swagger and

creating the app Tesori Nascosti

bugs and issues discovering, 1 test

app, 1 dataset and 1 new application

created

1 test app, 1 new dataset and 1 new application

created. Issues discovered using

swagger (same results obtained in June)

Students actively used the IES Cities platform

and the swagger, moreover they strongly

worked with the API through the making of

an application

Feedback retrieval from developers

Feedback Web Interface

Developers Students filled a special questionnaire for developers. Done also by students

involved in June

2 "developer questionnaires" filled

(plus 2 in June)

2 "developer questionnaires" filled

(plus 2 in June)

Obtain a more detailed feedback about the

apps, in particular about the technical aspects,

thanks to the opinion of developers


• August 2015:


Engagement activity



Developers engagement and internship at FBK

Active Use Web

Interface Developers

4 high school students attended an internship at FBK, working in the

SmartCommunity Lab in close contact with lab developers. They test the API platform using swagger and creating

the app Tesori Nascosti

bugs and issues discovering, 1 test

app, 1 dataset and 1 new application

created

1 test app, 1 new dataset and 1 new application created.

Issues discovered using swagger

Students actively used the IES Cities platform and the

swagger, moreover they strongly worked with the

API through the making of an application

Feedback retrieval from developers

Feedback Web

Interface Developers Students filled a special questionnaire

for developers 4 "developer

questionnaires" filled 4 "developer

questionnaires" filled

Obtain a more detailed feedback about the apps,

in particular about the technical aspects, thanks

to the opinion of developers

• September 2015:


Engagement activity



Rovereto's new apps release on

google play First Contact

Rovereto Segnala General

Public Release of the apps on google

play

At least 20 downloads for Rovereto Percorsi and 12 downloads for

Rovereto Segnala

78 downloads for Rovereto Percorsi and

9 downloads for Rovereto Segnala

Very good impact for Rovereto Percorsi, while Rovereto Segnala is a little below the expected results, this could be due some little problems in the communication between the app's software

and the Council's software Rovereto Percosi

• October 2015:



Engagement activity



Outputs Comments

Individuation of ambassadors Awareness Rovereto

Segnala Public

Authorities

The app is a useful and easy

tool to monitor the situation

Local police and traffic wardens have been identified as possible

"ambassadors" for Rovereto Segnala. Introductive meeting will

be held in order to present the app's potential and its main functionalities

5 ambassadors founded

5 ambassadors founded

A meeting with more people will be held in November in order to

increase the number of ambassadors

Meeting with local bodies

operating in the field of tourism

Awareness Rovereto Percosi

Local Traders

The app could be a strong tool to

increase the tourists' interest in visiting Rovereto and its territory

During the meeting the potential of the app will be described, and a

request of collaboration in increasing the quality of the app's

contents will be asked

20 downloads 20 download reached

These local bodies will accept to collaborate with the Municipality in order to

improve the app and planning a collaborative

work of publicization among tourists and visitors

Tourism local bodies

engagement First Contact Rovereto

Percosi Local

Traders

The collaboration between the Municipality and the tourism local bodies would be a great way to continuously improve the app's quality, enriching contents and

adding new routes. This could boost the appreciation of the territory by

tourists and visitors

20 downloads 20 download reached Boost in the downloads and users number

Public press conference Awareness

Rovereto Percosi

General Public

On the 13th of October there will be a press conference during which

Rovereto present the app Rovereto Percorsi

80 downloads (40 downloads for

Rovereto Percosi, 40 downloads for Tesori Nascosti)

More than 65 downloads for

Rovereto Percorsi and more than 75

downloads for Tesori Nascosti

Raising of interest for the apps and publicity on local newspapers and networks

Feedback retrieval from the beta test friendly

users

Feedback Rovereto Percosi

Active Users

Some of the 20 friendly users of the beta test done during the summer

period answered an in-depth interview. The others filled a

questionnaire

At least 5 in-depth interviews

14 questionnaires filled + 6 in-depth

interviews collected

Interesting hints obtained which could help the apps'

improvement

Feedback retrieval from the beta test friendly

users

Feedback Rovereto Segnala

Active Users

Some of the 20 friendly users of the beta test done during the summer

period answered an in-depth interview. The others filled a

questionnaire

At least 5 in-depth interviews

14 questionnaires filled + 5 in depth

interviews collected

Interesting hints obtained which could help the apps'

improvement


• November 2015:


Engagement activity




Outputs Comments

Individuation of ambassadors Awareness

Rovereto Segnala

Public Authorities

The app is a useful and easy tool to monitor the situation

Local police and traffic wardens have been identified as possible "ambassadors" for

Rovereto Segnala. After the first contact with those figures made in October, a meeting will be held in order to present the app's potential

and its main functionalities

Other 5 ambassadors

founded

7 ambassadors founded among local wardens

Christmas Market app

release First Contact

Rovereto Percosi

Others

Engagement material, such as banners, flyers,

stickers, etc.

A new version of the apps Rovereto Percorsi and Viaggia Rovereto have been released on

google play and apple store. Rovereto Percorsi includes a special route which is the Christmas market path; Viaggia Rovereto has a special section which invites users to park

outside the centre and reach the markets using the free shuttle bus service.

At least 20 downloads for

each app


each app

Viaggia Rovereto

Christmas Market

feedback retrieval

Feedback Viaggia Rovereto Others Christmas

gadget

People who use the shuttle bus service and deliver the questionnaire about the Christmas Market and Viaggia Rovereto filled are given

a Christmas gadget

At least 50 questionnaires

filled

25 questionnaire filled with a complete

evaluation (star system) of the app

The questionnaire has been given only the last two

weekends of November, and the majority of people

coming to visit the Market didn't know the app, so they

can't evaluate it properly

• December 2015:


Engagement activity



Christmas Market app First Contact Rovereto

Percosi Others Engagement material, such as banners, flyers,

A new version of the apps Rovereto Percorsi and Viaggia Rovereto have been released on google

At least 20 downloads for



release

Viaggia Rovereto

stickers, etc. play and apple store. Rovereto Percorsi includes a special route which is the Christmas market path; Viaggia Rovereto has a special section

which invites users to park outside the centre and reach the markets using the free shuttle bus

service.

each app Rovereto Percorsi and more than 80

downloads for Viaggia Rovereto

Christmas Market

feedback retrieval

Feedback Viaggia Rovereto Others Christmas gadget

People who use the shuttle bus service and deliver the questionnaire about the Christmas Market and Viaggia Rovereto filled are given a

Christmas gadget

At least 80 questionnaires

filled

More than 100 questionnaires filled

with a complete evaluation (star

system) of the app

Ambassadors engagement First Contact Rovereto

Segnala Public

Authorities

During the introductive meeting they have been

made aware of the usefulness of the app,

which could help them in their work

Traffic wardens will start a continuous and deep usage of the app during the last part of

December and the whole January

At least 5 downloads

7 downloads by traffic wardens

• January 2016:


Engagement activity



Outputs Comments

Rovereto Segnala active

use Active Use

Rovereto Segnala

Public Authorities

The app used as useful daily work tool

Traffic wardens adopt the app Rovereto Segnala as a daily

work tool

At least 5 complaints sent by traffic wardens

0 complaints collected

Due to the nature of the app it should be used only in case of problems encountered. In this month no problems found out

by the traffic wardens

Rovereto Segnala feedback collection

Feedback Rovereto Segnala

Public Authorities

Thanks to traffic wardens' feedback

the app will be improved following

their needs

Traffic wardens will be asked to answer a questionnaire, or an in-depth interview, in order to collect their feedback after a

period of active use

At least 3 questionnaires/in-depth interviews

collected

4 in-depth interviews collected

Public bodies collaboration Awareness Viaggia

Rovereto Local

Traders

The Rovereto Multiservice Enterprise posted the link of the

app Viaggia Rovereto on its website

At least 20 downloads 22 downloads


Rovereto Segnala

presentation First Contact Rovereto

Segnala Public

Authorities Local Policemen were asked to actively use the app

At least 10 downloads and 5 active users

13 downloads and 9 active

users

• February 2016:


Engagement activity



Outputs Comments

Rovereto Segnala active use Active Use Rovereto

Segnala Public

Authorities

The app used as useful daily work

tool

Local Policemen were asked to actively use the app during the last days of

January and the first half of February

At least 3 complaints sent by local policemen

5 complaints collected

Rovereto Segnala feedback collection Feedback Rovereto

Segnala Public

Authorities Collection of local policemen's feedback At least 1 questionnaire/in-depth interview collected

1 in-depth interview collected

9.4. Majadahonda

• July 2015:


Engagement activity



results Quantitative Outputs Comments

Meeting sport centres Awareness Sports4U Local

Traders To promote the sport

centres Meeting with the directors of the

sport centres Awareness of

S4U app

The attendance of the majority of directors of

sport centres

They were also interested in contribute with content to the

Vive Majadahonda app

Social Networks Awareness All

Majadahonda General Public

Being updated with innovative projects

news

Social media content and articles about apps and latest

developments in Majadahonda

General Knowledge of the

new apps

Readings, retweets, likes (FB)

Tester Training Session Feedback All

Majadahonda Public

Authorities

The role of alpha or beta tester is a driver incentive

We will be posting the development of apps and

encouraging feedback

Feedback of the tester.

• August 2015:


No engagement activities during August.

• September 2015:


Engagement activity



Social Media, digital newspaper

and radio Awareness All Majadahonda General

Public Being updated with

innovative projects news

Social media content and articles about apps in digital

newspapers General Knowledge readings, retweets, likes,

downloads

University visits Active use All Majadahonda Developers Merchandising material

Lectures in the University Francisco de Vitoria

Promotion of the IES Cities platform and creation of new

apps

New registered users in IES-CITIES platform

Sport Centre Meeting First contact Sports4U General

Public

To participate and to promote their centres.

Be the first to know about the apps

Meeting with seven Sport Centres and with the

employees of the Sport Department

The attendance of the directors of all sport centres

and their willingness to contribute

new active users, registered users in the IES Cities platform

High Scholl Visits Active use Sports4U Active Users Merchandising

material

500 children will receive hands-on training on IES

Cities apps

General knowledge of the app and of the IES Cities

platform

Number of downloads. Number of new apps

• October 2015:

No engagement activities during October.

• November 2015:


Engagement activity



results Quantitative Outputs Comments

Meeting with the sports department First Contact Sports4U Public

Authorities

To promote sport in the

town

During the meeting, the sport department gave its opinion about the

app and proposed several changes

Improvement of the app

A person of the sport department is going to upload

content, data, in the application.

There are many changes that has to be

made in the app.

• December 2015:



Engagement activity



Workshop Majadahonda Lab Active Use IES Player Developers

To have the possibility to

programme with real data

We will show the participants the apps and the features of

the platform

Downloads of the apps and awareness of the

platform

New registered users in IES Cities. Possible new apps

Stand at the Race Majadahonda

Monte del Pilar Awareness Majadahonda

In Route General Public

To have all the information about

the race

We will give information about the new app in-route and show

the public map of the Majadahonda cross

Downloads of the apps Downloads, Registered users

Meeting with Centro Oeste, Equinoccio Active Use

All Majadahonda

Local Traders

To promote their activities

During the meeting, we show them the apps and we arrange different promotional activities

Downloads and data loading

We encouraged then to load data in the platform. They are going to help

in the promotional activities

Social Media Awareness All Majadahonda

General Public NA Social media content about

apps in digital newspapers.

General knowledge about the project and Majadahonda apps.

Unknown/Untracked number of readings, retweets, likes.

Increasing trend on downloads.

• January 2016:


Engagement activity



University visits Active Use All Majadahonda Developers

To have the possibility to


Talks in different engineering classes about the project, the

apps and the platform

Knowledge of the IES CITIES Platform

New registered users in IES Cities. Possible new apps

Stand in Centro Oeste (Shopping Centre) January

18th 2016

First Contact All Majadahonda

General Public

Users who download the 4

apps will be given a power bank gift and will enter a

draw for an Android Tablet

A stand will be set in the Shopping Centre for the

awareness of general public about the IES Cities project

and to increase App downloaded. Marketing

material will be used as a gift

50 downloads of each app, 10 online questionnaires

This activity has resulted in an increase of 15-20

downloads for each of the four pilot apps.


and in a prize draw.

High School Visit, January 26th 2016

Awareness

Web Interface

General Public

The students will participate in a

power bank draw and, if they

introduce their mail in the in-app questionnaires,

they will enter in a draw for an

Android Tablet

Short presentation of IES Cities project, showing the web and

giving general information.

Awareness of IES Cities Project, open

data paradigm and IES Player.

3 downloads for 1st phase apps, 74 and 16 downloads for Sports4U and In-Route

respectively, 47 online questionnaires (5 for 1st phase, 42 for 2nd phase

apps), 2 in-app (Sports4U).

All

Majadahonda

Demonstration of Majadahonda Apps, especially focused on the second phase apps. The

audience is encourage to download the apps, use them

and fill in an online questionnaire.

10-25 downloads of first phase apps.

25 - 50 downloads of second phase apps. 10 online questionnaires.

5 in-app questionnaires

Stand in Equinoccio (Shopping Centre) January 30th 2016

First Contact All Majadahonda

General Public

Users who download the 4

apps will be given a power bank gift and will enter a

draw for an Android Tablet

A stand will be set in the Shopping Centre for increasing

App and IES Player downloaded and to get

feedback from users. Marketing material will be used as a gift

and in a prize draw.

50 downloads of each app, 10 online questionnaires

Very few downloads, 2 online questionnaire for first

phase apps. 5 in-app questionnaire, mainly from

Sports4U

Article about the second phase apps in the City Council

web

Awareness

Sports4U General Public

Being update about new

projects carry out in the city

A press release in the webpage of Majadahonda City Council to inform about the second phase

apps.

General Knowledge about IES Cities

project. Awareness of the second phase apps

been released.

Unknown/Untracked number of readings and

views.

Link to the article

Majadahonda in Route

Social Media Awareness All Majadahonda

General Public

NA Social media content about apps in digital newspapers.


Unknown/Untracked number of readings,

retweets, likes. Increasing trend on downloads.

• February 2016:


Engagement activity



High School Visits (5 high schools, 14

Awareness IES Player General Public

The students will participate in a power bank draw and, if they

Short presentation of IES Cities project, showing the

web and giving general

Awareness of IES Cities Project, open

data paradigm and IES

First phase apps: 10 downloads; Sports4U: 53

downloads, 6 In-app


presentations) introduce their mail in the in-app questionnaires,

they will enter in a draw for an Android Tablet

information. Player. questionnaires; Majadahonda In-Route: 21 downloads, 7 In-app

questionnaires

All Majadahonda

Demonstration of Majadahonda Apps, especially focused on the second phase

apps. The audience is encourage to download the apps, use them and fill in an

online questionnaire.

10-25 downloads of first phase apps. 25 -

50 downloads of second phase apps. 10 online questionnaires.

5 in-app questionnaires

Social Media Awareness

Sports4U General Public NA Social media content about

apps in digital newspapers.


Unknown/Untracked number of readings,

retweets, likes. Increasing trend on

downloads. Majadahonda in

Route

Coder Dojo First Contact Web Interface Developers To have the possibility to


We will show the apps and the features of the platform to the

participants

Downloads of the apps and awareness of the

platform

New registered users in IES Cities. Possible new

apps

Majadahack 2k16 Active Use Web Interface Developers

To have the possibility to programme with real

data

Developers' meeting in which the goal is to develop collaborative software.

Knowledge of the IES CITIES Platform

New registered users in IES Cities. Possible new

apps

Link Majadahack

2k16

IES Cities D3.3: Second Phase Execution and Monitoring · 2017-04-25 · IES Cities D3.3: Second...

Documents

Transcript of IES Cities D3.3: Second Phase Execution and Monitoring · 2017-04-25 · IES Cities D3.3: Second...