Probes and Big Data: Opportunities and Challenges · New data sources, probes and big data have the...
Transcript of Probes and Big Data: Opportunities and Challenges · New data sources, probes and big data have the...
Probes and Big Data:
Opportunities and
Challenges
Fiona Calvert, Director Information Services and Mapping, Department
of Transport Planning and Local Infrastructure
New data sources, probes and big data have
the potential to revolutionise how we
manage and plan the transport system
2 Source: Danny Woolard, INRIX – Big Data in Transportation, BITRE Workshop Sydney May 2014
INRIX GPS data points – monthly, worldwide
Probes and sensors are sources of big data, but not the only ones
Probes allow the movement of people and things to be tracked over the transport network
Key examples are GPS enabled devices and mobile phones (or cellular data
3
4
There are many types of probes and sensors
Active
Passive
Intrusive Non-intrusive
Point sensors
Point to point
Origin/ Destina-
tion
Big data is big because of its volume, variety and velocity
Traditional data is structured
New data sources include a wide range of unstructured data
‘Golden nuggets’ come from the insights provided by combining structured and unstructured data
5
Business Intelligence
Reporting and visualisation
tools
Analytics such as data
mining, event processing, predictive analytics
Insights from analytics
Structured
Unstructured
Data sources
Tradi-tional data
sources
Social media
Probe data
Inter-net of Things
Image data
Voice record-
ings
There are a range of uses of big data, particularly in measuring and understanding congestion
Source: Danny Woolard, INRIX – Big Data in Transportation, BITRE Workshop Sydney May 2014
Other uses include
Understanding demand
• Demand forecasting
• Marketing public transport
• Managing demand for road use
Operational performance
• Real time management including optimisation and incident response
• Network planning
Customer service
• Sentiment mapping and crowd sourcing
• Customer service delivery planning and management (eg call centre resource planning)
Asset management
• Condition tracking (eg bridge and pavement cracking)
• Asset location and registers (eg speed signs)
Mobile phone data is beginning to be used overseas to understand transport activity
Source: FHWA Cell Phone Data and Travel Behaviour Research, Symposium Summary Report, Exploratory Advanced Research Program, February 2014
This example shows changes in taxi use in different traffic and weather conditions
Source: FHWA Cell Phone Data and Travel Behaviour Research, Symposium Summary Report, Exploratory Advanced Research Program, February 2014
Mobile phone data is also being used to understand origins and destinations of trips
Source: FHWA Cell Phone Data and Travel Behaviour Research, Symposium Summary Report, Exploratory Advanced Research Program, February 2014
Bluetooth and WiFi data is successfully being used to monitor and manage traffic, as in Portsmouth, UK
This made the installation a very easy process, with all work undertaken on the pavement so no traffic
management was required and the equipment was installed at a height which minimised working at height
issues.
The only equipment needed at Portsmouth’s traffic management centre is a PC with an internet browser installed
(Internet Explorer, Chrome, Safari etc.) This is used to connect to the cloud server via a secure web page and to
display the traffic status dashboard.
Portsmouth City Council, as the local highway authority, is using the system both tactically and strategically. On a
day to day basis they can use the system to monitor the actual road conditions in real time and manually control
the phasing of traffic signals when congestion is identified.
On a strategic level they are able to measure the impact of major road works, maximise the effectiveness of
diversion routes and improve their traffic models. They can also use the data to plan for major events in the city
centre, for example the BUPA Great South Run which is held on a Sunday in October and attracts around 25,000
athletes along with their families and supporters to watch the event. The journey time graph for the 27th
October
2013 (the day of the run) is shown below along with the data from the previous Sunday for one of the main routes
into the city. (Please note that the vertical scales on the two graphs are very different).
Source: SmartCCTV Ltd, Portsmouth Bluetooth and WiFi journey time and congestion monitoring system
Presentation Sourcing New Data7
BLUESCAN
7Source: Luis Willumsen, Opportunities for sourcing new data, BITRE Workshop, Sydney May 2014
A system approach combining multiple approaches shows much promise
Part of the power of big data comes from its volume, which overcomes sampling issues
Source: Ben Pierce, Batelle, FHWA Cell Phone Data and Travel Behaviour Research, Symposium Summary Report, Exploratory Advanced Research Program, February 2014
Victoria is already involved
SCATS loops and traffic controllers
Bluetooth sensors in metro Melbourne (around 30 installed) and trialing analytical systems
Strava cycling and running data
Splunk trials on predictive maintenance for IT assets
SUNA data on traffic
Includes probe data from fleets using SUNA navigation systems
Image data analysis for speed sign locations, asset condition monitoring
Call centre quality control using voice recordings
Customer service centre resource planning
Sentiment mapping
Real time location information for buses and trams feeding passenger information displays and PT apps
800 000 hits per day for TramTracker
Key challenges are that big data changes the role of government and requires new capabilities and resources
Government goes from a data provider to a data customer
What is the role of government in providing a single platform and open data?
A fundamental change is needed in the way data is regarded
• Asset to be harvested
New capabilities
• Data scientists
• IT specialists and systems (eg storage, servers, software, skills)
• eg Hadoop, Splunk etc
Questions we need to be thinking about include:
• What is needed in-house?
• What is needed for Government to be an educated purchaser of big data?
• How do we design our approach to get the most out of big data?
• Avoid silos and duplication
• Establish interoperability from the start
Privacy and security issues are a major challenge that must be addressed up front
Legislation differs state to state adding to the challenges
• NTC 2013 paper on Cooperative ICT recommends national principles/policy
• Victoria has just put in place a new Privacy and Data Protection Act, establishing a new Commissioner for Privacy and Data Protection
Issues include opt out clauses, user understanding of data use, potential for misuse….
16
Governments find it hard to take risks and care is needed so that in building an appetite for big data it is not seen as a sole answer to all information needs
We will need to take risks
• not every algorithm will work
• not every correlation will be meaningful
• not everything we think will be related will be found to have any link
Big data is not a panacea
• It is good at showing ‘what’ but not ‘why’
• It will save costs sometimes but does not come for free
Why Google Flu Is
A Failure
It seemed like such a
good idea at the
time.
Source: Forbes, http://www.forbes.com/sites/stevensalzberg/2014/03/23/why-google-flu-is-a-failure/
The way forward involves pilots and proofs of concept, developing strong cases and partnerships
Partnerships
• InfraHack is a good example of the potential for partnerships
• Victoria has also started working with NICTA and the IBM Research Lab and is developing relationships with Google and HERE.com for GTFS feeds
Pilots
• Opportunities to build capabilities and establish proofs of concept
• Build business cases to change mindsets and secure resources
Partnerships across the transport portfolio can also be built to minimise costs and maximise opportunities
InfraHack 1st Prize: BetterWay Product: ‘road load balancing’ tool to predict congestion and readjust traffic lights in real-time
Approach: combination of ‘deep learning’ (fast machine learning), predictive modelling and a graph-theoretic approach using datasets from VicRoads and TomTom
InfraHack 2nd Prize: LoZo Product: ‘loading zone finder’ that alerts drivers to available loading zone bays as they approach the area of their destination
Approach: routing algorithms, PHP and Javascript using data from parking events from bays with sensors and Here.com API