1

The Shift to {Open|Big|Linked} DataOr, how I learned to stop worrying and love the data

Pia WaughDirector of Gov 2.0 and Data

Technology and Procurement DivisionDepartment of Finance

22

Great expectationsPublic expectations and problem solving

http://www.flickr.com/photos/zebble/8212264/

33

[Open|Big|Government] Data

Credit: Joel Gurin (with permission)

44

Key Benefits to Community/Industry in Opening Data

Economic• Creates opportunities for industry to value-add to government data• New services, systems and industries• New opportunities and innovation in industry, research, civil societyAccountability• Visibility to government spending, projects, effectiveness, etc• Increases incentive to follow evidence based approachBetter policy and programs• Enables greater participation in policy planning and implementation• More informed public → better decision making• Improvements to data → better policy and decisions

55

Key Benefits to the Public Service in Opening Data

• Efficiencies from proactively publishing common requests• Cheaper and more modular services delivery• Reduced regulatory burden through machine readable data supporting compliance and automated reporting• Better policy outcomes by leveraging cross-agency data• More consistency & less duplication across government• Improved opportunities to leverage innovation and collaboration (citizens, industry, other depts)• Opportunities to improve data quality through verifiable public contributions

66

Tips for ensuring benefits realisation of open data

• Adopt an approach of “data user and developer empathy”• Data publishing built into your BAU• Initial focus on data that supports you build capability• Consume your own data APIs (apps, datavis, BI, etc)• Ensure you consider:

• Quality – no one can use bad data, but perfect is enemy of the good• Currency – is it up to date? How often is it updated?• APIs – is it programmatically available?• Publishing – have you provided supporting materials (taxonomies)?• Discoverability – is it hosted or linked on data.gov.au?• Reusability – have you tested it with data users?• Licensing – Creative Commons By Attribution the default

• Automation wherever possible!

77

Australia is doing pretty well

88

99

Open Data Portals

Council Portals:• City of Melbourne• City of Brisbane

1010

Policies Components

APS policies in aggregate:• Permissive copyright – CC-BY as the default• Open by default, machine readable accessible data• Support reuse and innovation• More public engagement • Better use of data for government policy and service development

States/Territories add:• Procurement – open by design• Reporting – dashboards• Departmental strategies• Declaration of Open Data

1111

Open Data Discovery Model

National Map Datavis Application

developmentAnalysis& Policy

Value Creation

Discovery

Data

Full Discovery

New Services

FINDNational

spatial index (gov, private,

research)

1212

data.gov.au

Free, cloud, scalable API enabled platform for hosting government data.

Staged approach1. Publishing (2013 – mid 2014)

Improving the functionality and ease of

publishing for agencies with training and

documentation

2. Value realisation (2014-2015)Providing useful front end tools for data.gov.au

including data visualisation and analysis tools.

Publishing quality data a pre-requisite.

3. Data quality (2014-2015)Looking at ways to provide agencies the ability

to accept iterative data improvements in a

verifiable way

Features•Manual and automated publishing options• API access to government data• Easy to publish, download & interact• Use cases and site|data|org analytics• Data Request Site• Metadata harvesting from gov data gateways• National Map integration• Data model registry• Federated search for discoverability

In Planning• 5 star quality plugin• Selective crowdsourcing for updates• League Table

1313

NationalMap

1414

1515

DSS and ATO Statistics

1616

Spatial Case Study

http://www.finance.gov.au/blog/2014/02/24/guest-post-on-the-value-of-open-roof-prints/

1) Reduced work2) Value adding3) Quality

improvements4) Zero cost API

access to their own data

1717

The Government Data Landscape (latest version online)

1818

Other Data Projects

• Administrative• Spatial• Research data• Imagery• Sensor• Realtime (eg Transport)• Census/Statistics• Cultural• Data about government• Fiscal

1919

Data Integration

• Challenging but great potential for improved policy/services.• Unit record sharing is complex, raises privacy concerns.• Unit record data is mostly useful to researchers, who have appropriate mechanisms with legal, technical, ethical constraints to access such data.• Data aggregated by common spatial boundaries is comparative across datasets and over time.• Unfortunately, data owners traditionally aggregate to boundaries that constantly change (electorates, postcodes, etc).• The Australian Statistical Geography Standard (ASGS) provides a consistent set of spatial boundaries that can be mapped to other needs.• Anonymisation on the fly APIs also provide mechanism for appropriate public/agency access to unit record level data (e.g. ABS.Stat)

http://statistical-data-integration.govspace.gov.au/

https://toolkit.data.gov.au/index.php?title=Definitions#Types_of_data

2020

Open by Design – drawing a line in the sand

Building proactive publishing into:• Systems• Processes• Procurement• Planning• Records management

Leveraging open data through:• Public APIs• Analysis tools and datavis• Internal processes looking for external sources• Existing databases!

2121

Some Challenges

• Education• Legislative• Culture• Systems• Privacy and anonymisation• Reactive vs proactive• Metadata/semantic context• Too much data• Real time vs historic• Definitions and common references• Limited skills and over specialisation

2222

New and Old Skills Required

• Publishing and Automation• Project management, reporting• Metadata/linked data• API development and serving• Plumbing between systems• Analysis and statistics• Policy development• Data, info and policy visualisation• Public consultation and engagement• Online community management

2323

Broader goal: Government as an APIMaking all gov data, content, services API enabled

Care of fedAPI.gov

2424

Support

• http://toolkit.data.gov.au is updated regularly. Recent updates include:• How to automate data updates to data.gov.au with FME• Improved information on how to clean data• How to manage your own catalogue harvesting• Government data landscape to identify projects of use

• Open Data Community Forum – soon to be moved to analyticsspace• Talk to your colleagues across government(s)

• Other sources• Communities of interest: Data Science Meetup groups, Data

Analytics Centre of Excellence, Linked Data Working Group, National Statistical Service, etc

• GovHack Developers Kit: Become a data scientist in an hour, data tools, APIs, datavis, spatial, mashup techniques, statistical

2525

Simplified Guidance, Policy and Case Studies

http://toolkit.data.gov.au

2626

Key Case Studies

• Publishing Budget 2014 Data Report• Open data – Transforming the Provider / Stakeholder Paradigm• On the Value of Open Roof Prints• 100 years of patent and IP data released on data.gov.au

More available soon at http://toolkit.data.gov.au

Other Australian case studies/documentation• SA Open Data Toolkit• QLD Government Case Studies• Victorian Government Showcase• NSW Apps Showcase• ACT examples

2727

The future is here....And it is already widely distributed

http://www.flickr.com/photos/mr_matt/3568892622/

Challenge #1: CollaborateChallenge #2: ShareChallenge #3: MeasureChallenge #4: Play

Questions?@piawaugh

@datagovaudata.gov.au

toolkit.data.gov.au