Discussion Paper Title Goes Here - Store & Retrieve Data ......programming interface), this...

Discussion paper: ePlanning data warehouse

ePlanning program

ePlanning program | Discussion paper: ePlanning data warehouse 2

December 2018 © Crown Copyright, State of NSW through its Department of Planning and Environment 2018

Disclaimer

While every reasonable effort has been made to ensure this document is correct at time of printing, the State of NSW, its agents and employees, disclaim any and all liability to any person in respect of anything or the consequences of anything done or omitted to be done in reliance or upon the whole or any part of this document.

Copyright notice

In keeping with the NSW Government’s commitment to encourage the availability of information, you are welcome to reproduce the material that appears in ePlanning program: Discussion paper: ePlanning data warehouse. This material is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0). You are required to comply with the terms of CC BY 4.0 and the requirements of the Department of Planning and Environment. More information can be found at: http://www.planning.nsw.gov.au/Copyright-and-Disclaimer.

https://creativecommons.org/licenses/by/4.0/deed.en

http://www.planning.nsw.gov.au/Copyright-and-Disclaimer


Contents

Executive Summary 4

Part 1 – Building digital transformation capability 8

New direction for digital analytics in ePlanning 9

Objectives of the project 9

ePlanning data warehouse 10

Standard reporting 10

Benefits of near real-time data reporting 10

How the project will be developed 11

Part 2 – ePlanning digital data services 13

Overview 14

Existing ePlanning data sources 14

Local Development Performance Monitoring (LDPM) 14

Application Tracking Data Interchange Specification (ATDIS) 15

Have your say 16

ePlanning digital services 17

DDA data sources 17

Technical Data Specification for DDA 17

Have your say 18

Part 3 – DDA performance measures and data provisioning 19

Key performance indicators to be derived from DDA – data warehouse 20

Have your say 21

Data provisioning for DDA platform 21

Conclusion: collecting, reporting, and presenting of planning information 23

Have your say 23

Appendices 24

Appendix A: Roadmap for the DDA project 25

Pre-project 25

Project design 25

MVP 1 26

Public exhibition 26

MVP 2 27

Appendix B: Core fields within Local Development Performance Monitoring (LDPM) 28

Executive Summary


Overview

The ePlanning branch at the Department of Planning and Environment (DPE) is currently implementing a digital

framework to support the planning system across NSW. Actively delivering against its digital transformation

strategy, ePlanning are deploying digital services to stakeholders. Embedding new and existing data sources via

an ecosystem of integrated workflows, the strategy is enriching not only data coordination but providing new

opportunities in how development transactions and submissions are carried out.

ePlanning strategy and digital development data services

A key driver of ePlanning’s digital enrichment strategy is creating better data coordination between State and

local government. In this instance, an opportunity to systematically capture development applications (DAs) and

complying development certificates (CDCs) data via a technical data specification exists. Using APIs (application

programming interface), this automated process will extract and deposit data from council and private certifier

systems according to ePlanning’s technical data specification into a centralised data warehouse. Handling

the demand side of the data equation, following cleansing and any necessary transformation, a near real-time

supply of development data will be deployed to policy / decision makers.

To introduce the reader to all data sources feeding into the DDA – data warehouse, figure (i) presents existing sources such as LDPM (Local Development Performance Monitoring) and ATDIS (Application Tracking Data Interchange Specification). The launch of DA / CDC online represents ePlanning’s new digital services which the DDA – data warehouse will draw on as part of its data capture and ingestion in 2019 and beyond. The third component is DA / CDC data from councils, and private certifier data from the BPB (Building Professionals Board).

Figure (i): ePlanning digital shop front for DDA – data warehouse

Since 2007-08, LDPM has collected detailed information on every determined DA and CDC across NSW which

DPE publish in an annual report https://datasets.seed.nsw.gov.au/dataset?q=ldpm. This report monitors key

aspects of the planning and development assessment system, benchmarks council performance in the

development assessment, supports decision making on planning system reform, and underpins State and

National (COAG) reporting on planning system performance.

The annual issuance of the LDPM is a useful resource, but its data capture is limited and cumbersome. On this

basis, the decision has been made to establish the DDA - data warehouse. LDPM will continue to function in its

existing capacity as the DDA - data warehouse is designed. At this stage, it is expected that LDPM will perform its

operational activities until the end of financial year 2021. By that point ePlanning anticipate having a full uptake of

digital services such as DA / CDC Online, and / or a sufficient volume of data coming through the DDA - API

supported data warehouse.

https://datasets.seed.nsw.gov.au/dataset?q=ldpm


As DPE upgrade and enhance their online digital services, ePlanning will transition from their current data

reporting services, ATDIS and LDPM to a digital service called DDA (Development Data Analytics). In DDA,

customers will experience a more comprehensive data service via the design and implementation of a DDA -

data warehouse.

ePlanning’s data warehouse

In the first half of 2018, ePlanning re-engaged with council property-information system vendors to better

understand the challenges and opportunities associated with automatically extracting data from DA’s and

CDC’s. Drawing on the ATDIS data specification, which was designed and subsequently developed into

production by 27 councils in 2014 (see map 1, p.16), dialogue with the vendors highlighted how it could be

improved. ePlanning consequently drafted the technical data specification for DDA and extended their

consultation network to include councils and private certifiers (see figure ii).

Figure (ii): Roadmap for the DDA project

The project team will advance the project design, MVP 1 (minimum viable product), using ATDIS and LDPM as we await feedback from the public exhibition. MVP 2 meanwhile will represent the core part of the project. MVP 2 has three deliverables.

(i). Data providers building the necessary APIs,

(ii). ePlanning designing and implementing the data warehouse,

(iii). ePlanning developing data outputs into various analytical platforms.

The public exhibition and MVP 2 will be divided into four key stages (see figure iii).

• Stage 1 is the design and socialisation of the draft technical data specification which is being showcased as part of the public exhibition

• Stage 2 is the data capture mechanism (API). Councils will develop these to enable the data artefacts detailed in the specification to be integrated into the data warehouse

• Stage 3 represents the necessary data transformation and storage related procedures in the data warehouse

• Stage 4 is the delivery of cleansed data from the data warehouse into an API for analysis, reporting and curation across various technologies including BI tools.


Figure (iii): DDA – data warehouse, project deliverable life-cycle

Scaling up digital transformation to generate the systematic capture and reporting of development data across NSW is placing new demands on ePlanning and its stakeholders. Defining a new technology roadmap around the DDA’s technical data specification will support the strategic evolution of all development analytics and the long-term proposition of the DDA – data warehouse.

It is these changes which are critical in seizing today’s technological opportunities.

Part 1 – Building digital transformation capability


New direction for digital analytics in ePlanning

The ePlanning branch at DPE is currently implementing a digital framework to support the planning system across

the State of NSW. Designed in accordance with planning policy directives and legislative requirements, it is

helping accelerate the capture and dissemination of planning-specific information with the aim of improving

digital services.

Actively delivering against its digital transformation strategy, ePlanning has deployed digital services, such as

Concurrence and referral, DA / CDC Online, to stakeholders within and beyond DPE. Capturing and integrating

data into an ecosystem of digital workflows, these data provisions have extended digital capabilities into the

hands of end users.

A key driver of ePlanning’s digital enrichment strategy is creating better data coordination between State and

local government. In this instance, an opportunity to systematically capture DA / CDC data via a technical data

specification readily exists. Using APIs (application programming interface), a daily capture mechanism will

extract specified data from each council and private certifier systems and deposit into a centralised data

warehouse. Following the necessary cleansing and transformation, a near real-time supply of data will be

deployed to policy / decision makers.

Objectives of the project

The aim of the DDA – data warehouse project is to capture near real-time development data and ingest into a data repository (data warehouse) for storage, analysis and reporting.

The objectives of the DDA – data warehouse project is two-fold. Technical and non-technical:

Technical:

1. Produce a technical data specification to councils and private certifiers

2. Guide councils in creating an API to enable the capture and ingestion of data artefacts detailed in the technical data specification

3. Design and development a data warehouse to transform and store the DDA data

4. ePlanning to build a delivery-based API to enable self-service reporting for the curation and analysis of data

Non-technical:

1. Develop a communication strategy to assess the data and technical readiness of councils and private certifiers

2. Assess the change strategies required to ensure the technical data specification and time-based requirements can be realised

3. Identify technical and non-technical channels to support stakeholder engagement and their commitment to delivering the technical data specification and the API mechanism to capture and deliver the necessary data


ePlanning data warehouse

As ePlanning’s shopfront of digital services broadens to include online DA / CDC applications and API

mechanisms to secure the DDA data specification from councils and private certifiers, the number of digital data

points are set to increase exponentially. To host these anticipated data volumes, ePlanning require a data

warehouse.

Functioning as a centralised repository, the DDA – data warehouse will support the capture, ingestion,

transformation, storage, and delivery of data from the noted digital services. Supporting greater data velocity

and volume, the data warehouse will enrich end user knowledge and understanding of development activity

across NSW.

To ensure data coordination, alignment, and governance across all ePlanning services, a draft version of the

technical data specification was circulated to a working group in October 2018 and now to everyone via the

public exhibition process. Feedback will be integrated accordingly, and a final version of the technical data

specification shall be presented to the DPE Secretary for sign off. It is at this stage that the project will transition

to the technical development stages.

Following the stage of data capture, all data will be ingested into ePlanning’s technical environment where it will

be transported to the DDA - data warehouse for cleansing, transformation, and storage. The completion of these

business process and quality assurance procedures will ensure the data is fit for purpose and in alignment with

the finalised technical data specification. It is at this stage that the data will represent the single source of truth

for all development assessments in NSW and become readily available for publication and any subsequent

analysis.

Standard reporting

As the delivery of the DDA – data warehouse disrupts how development data is captured and managed, it will provide new insights for end users via the provision of data reporting in a near real-time capacity.

A key priority of the DDA – data warehouse is the utility and accessibility of the data for end users. To enable this,

ePlanning intend to make this process as simple as possible by providng end-users with standard reports.

Advanced and customised reporting options are also being considered. For example, ePlanning will deliver an

API which end users will be able to integrate into their native environment and utilise as a SaaS (software as a

service) to help formulate data analytical outputs specific to their needs.

Benefits of near real-time data reporting

The purpose of the DDA – data warehouse is to capture DA and CDC data from councils and private certifiers.

Improving service delivery, policy design, and decision making, the provision of near real-time development

data will not only become more readily available, but will be supplied in greater volumes, at a higher frequency,

and across multiple sources. Summarised as the 3V’s, volume, velocity, and variety; in aggregate these

dimensions produce a 4th V, value, in the form of providing many data-driven insights and supporting decision

making processes. See figure 1.


Figure 1: DDA 4 V’s benefit statement

How the project will be developed

The DDA – data warehouse will be delivered between now and 2021. MVP 1 is currently being built and will

include the integration of existing ePlanning data sources (LDPM, ATDIS). Over the next 2 years, ePlanning will

roll out four stages of design and development as detailed in figure (ii).

Stage 1 is the design and socialisation of a draft technical data specification for DDA. Designed with the objective

of superseding the LDPM survey, the specification is currently in draft format and to date has only been shared

with the project working group. To advance the technical development stages (stages 2, 3, and 4), ePlanning

are seeking acknowledgement that the draft technical data specification is an appropriate data collection

protocol. As part of this public exhibition, ePlanning are accepting feedback and submissions from stakeholders.

Stage 2 is the design of the data capture mechanism. To enable data alignment, ePlanning will request that each

council, in partnership with their vendor (Tech One, Infor-Pathway, Sol Orient, Civica), develop an API to ensure

the capture of the data artefacts detailed in the technical data specification.

Stage 3 caters for data transformations and storage where all necessary business rules will be applied. The

purpose is to measure data quality tolerances. If standards are not met, the data is automatically returned for

amendment and updating. Once qualified as fit for purpose, the cleansed information will be stored in the data

warehouse.

Stage 4 involves rolling up some of the stored data into an API and delivering to end users for extrapolation into

various technologies for reporting purposes. These include dashboarding in Power BI, but also web-based

applications like the NSW Planning Portal Spatial Viewer. These will be complemented with a rest service which

will be made available to support the curation of spatial data analytics in ArcGIS.


The DDA – data warehouse will become operational towards the end of 2018-2019. As a growing number of

councils and private certifiers are onboarded, the reporting capabilities of the platform will be enrichened over

the course of 2019-20.

Have your say

This discussion paper, which is part of the DDA – data warehouse public exhibition, is looking for feedback to

the questions contained in this document but also comments relating to the accompanying draft technical data

specification. As part of our efforts to create good data governance via synchronised ingestion of data from

stakeholders, ePlanning are actively interested in receiving responses specific to data capture and provisioning.

Feedback on the proposed technical data specification can be submitted via the NSW Planning Portal

(https://www.planningportal.nsw.gov.au) where you can also complete a short online survey. The exhibition

will open on January 22nd 2019.

Following the close of the public exhibition on February 22nd 2019, ePlanning will release a report detailing the

feedback we have received in response to the specified questions and general feedback relating to the draft

technical data specification. With DPE Secretary approval, a final version of the technical data

specification shall be published, which will become the DA / CDC data governance standard instrument for

the collection and administration of the DDA - data warehouse moving forward. The DDA – data warehouse and

its associated stages will subsequently be developed, approved, and made available through the ePlanning

Planning Portal.

https://www.planningportal.nsw.gov.au/

Part 2 – ePlanning digital data services

ePlanning program | Discussion paper: ePlanning data warehouse

14

Overview

The ePlanning shop front has broadened over the course of 2018 with the GO LIVE of numerous digital services including DA / CDC and C&R online (see figure 3). As part of the DDA – data warehouse design and implementation, the data model will capture the necessary fields from these new digital services and integrate them into the DDA – data warehouse. To minimise duplication and the multi-handling of data by stakeholders, the data warehouse will exercise versioning authority and change-control processes. Functioning as the centrally governed source of truth, handling data in this way will ensure that master data management is upheld, and all data outputs are aligned with the artefacts in the technical data specification.

To introduce the reader to all data sources feeding into the DDA – data warehouse, figure 3 presents existing sources such as LDPM and ATDIS. The launch of DA / CDC online represents ePlanning’s new digital services which the DDA – data warehouse will draw on as part of its data capture and ingestion in 2019 and beyond. The third component is DA / CDC data from councils and private certifier CDC data from the BPB (building professionals board). These sources will capture and ingest data into the DDA – data warehouse via a suite of API services.

Figure 3: ePlanning digital shop front for DDA – data warehouse

Existing ePlanning data sources

Local Development Performance Monitoring (LDPM)

Since 2007-08 the Department has collected detailed information on every determined DA and CDC across NSW using the LDPM (Local Development Performance Monitoring) survey. Published as an annual report, https://datasets.seed.nsw.gov.au/dataset?q=ldpm it monitors key aspects of the planning and development assessment system, benchmarks council performance in the development assessment, supports decision making on planning system reform and underpins State and National (COAG) reporting on planning system performance.

Data fields

In the 2017-18 edition of the LDPM survey there were 42 fields of data collected, see Appendix B, for each DA,

DA modification and CDC. Councils uploaded this data from their property-information system, via Excel or SFTP

(secure file transfer protocol), to an online database administered by ePlanning for validation and reporting. The

clear majority of DAs and DA modifications in NSW were determined by local councils (60,700 DA’s; 14,500

https://datasets.seed.nsw.gov.au/dataset?q=ldpm


15

modifications to DA’s; 30,000 CDC’s in 2017-18), but approximately 90% of CDC’s were determined by private

accredited certifiers (see figure 4).

Figure 4: NSW development determinations, 2009-10 to 2017-18

How the data is used

Summary data is published annually, and analysed across all councils.

Key indicators include:

• Number of developments determined each year • Percentage of developments determined as complying developments (max. 10-day approval) • Total value of development • Average and median determination (overall and by development type) • Number of developments by development type • Average and median time for State agency referrals • Average and median time for stop-the-clock • Percentage of developments determined by Councilors or council staff

The annual issuance of the LDPM is a useful source, but its data capture is limited and cumbersome. On this basis,

the decision has been made to establish the DDA data warehouse. LDPM will continue to function in its existing

capacity as the DDA - data warehouse is designed. At this stage, it is expected that LDPM will perform its

operational activities until the end of financial year 2021. By this point ePlanning anticipate having a full uptake of

digital services such as DA / CDC Online, and / or a sufficient volume of data coming through the DDA - data

warehouse.

Application Tracking Data Interchange Specification (ATDIS)

The main objective of the ATDIS project was to help every council in NSW work with a vendor of their choice to

implement an electronic application tracking tool that could extract application information from their property

0

20,000

40,000

60,000

80,000

100,000

120,000

20

09

-10

20

10-1

1

20

11-1

2

20

12-1

3

20

13-1

4

20

14-1

15

20

15-1

6

20

16-1

7

20

17-1

8

DA s.96 CDC


16

information systems via an API. Making this information available online required a common data standard for

reporting and extracting application-tracking information from councils.

This data standard was implemented by 27 councils across NSW (see map 1). In late 2018, each of these councils

continue to supply live ATDIS data feeds to ePlanning, but these are not controlled or coordinated in alignment

with LDPM or any other reporting source used elsewhere in DPE. Moreover, given that the ATDIS was never

properly implemented across all NSW councils, its data administration was not properly maintained by DPE

despite value in its underlying frequency and provisioning. Mindful of its limitations, the ATDIS API’s have a

broader purpose in that they can be tailored and extended to cater for the data requirements contained in the

technical data specification. A piggy-back strategy, it will allow the project team to fast track the DDA

implementation with these participating councils.

Map 1: ATDIS councils, NSW, 2018

Have your say

Are there any components or features of the LDPM survey / ATDIS feed which you would like to be included in the DDA – technical data specification?


17

ePlanning digital services

CDC and DA online services

Designed with the objective of supporting the Premier’s Priority of achieving faster housing approvals, CDC volumes have doubled over the past decade. Representing approximately one-third of total determinations, this trend is likely to continue through existing data channels and via CDC online. DA online will also function as a self-service submission by the applicant, and like CDC is expected to enhance the digital capture of data and accelerate business processes including pre-lodgment, lodgment, assessment, and determination status.

Facilitating automated data capture, CDC and DA online will help ensure an improvement in data quality and accuracy, which when integrated into the DDA – data warehouse will broaden CDC and DA reporting insights and capabilities.

DDA data sources

Council DA / CDC data

As noted throughout the discussion paper, the aim of the DDA – data warehouse is to capture council-based DA’s and CDC’s. Using the technical data specification as the data blueprint, ePlanning will work with councils in establishing the design of the API’s to ensure the capture and delivery of the necessary data provision.

Building Professionals Board (BPB) – CDC data

The BPB released a data specification in early 2018 and in the interim have been capturing private certifier and council private certifier data across multiple channels. As part of the DDA – data warehouse design and implementation, ePlanning intend to integrate a retro and current perspective of the BPB’s CDC activity into the data warehouse (see figure 3, p.14). While CDC online will start capturing a growing volume of transactions moving forward, most data will continue to be captured through the BPB and select private certifiers aggregators.

Technical Data Specification for DDA

The technical data specification is the key driver in moving the project forward. Instrumental to the design

and success of the project, the reader is advised to review the specification in detail to get a firmer understanding

of our business objectives.

The objectives of the technical data specification are:

1. To provide accurate and timely access to DA data, which will enable the analysis of processes and determination times

2. To ingest DA information from stakeholder electronic management systems in a structured format

3. To store and curate data from multiple sources and provide a comprehensive data service.

The technical data specification schema defines the data elements that are published as a compliant feed.

The schema has a defined structure which is broken down into 11 blocks which are categorised into fields. The

block types presented in figure 5 can be mandatory or conditional. Any data-capture feed must include all


18

mandatory block types and should include any of the conditional block types unless the record type is not

applicable.

Figure 5: Technical data specification schema, key blocks of DDA – data warehouse

For more information, the reader is advised to review the technical data specification.

Have your say

What aspects of the proposed design and technical data specification are satisfactory and which parts could be improved?

Is there a simpler way of capturing the data? What challenges does the technical data specification pose?

How long would it take to ensure data alignment with the technical data specification?

To what data quality standard could the data artefacts / fields in the technical data specification be provided?

How quickly could you build an API to enable connectivity and ingestion into the ePlanning technical

environment?

Part 3 – DDA performance measures and data provisioning


Key performance indicators to be derived from DDA – data warehouse

With data capture and technical delivery options being advanced, consideration must be given to how DA /

CDC data might be analysed to better understand trends or support policy issues that require revision. There are

many possible combinations which could be analysed, but some likely inclusions are listed in figure 7.

Figure 7: Standard data measures to be included in high level DDA reporting

DA specific

• Number of development applications lodged / determined • Proportion of development applications approved or refused development consent • Number of approvals by development type • Proportion of development applications processed by different consent authorities (e.g. regional

planning panel, local planning panel, council staff etc) • Timeframes taken by consent authority to determine development applications • Timeframes taken by State / external agencies to respond to concurrence and referrals • Proportion of development applications subject to public notification processes • Number of deemed refusals • Number of applicant appeals • Number of third party appeals

• Proportion of decisions subject to appeal which are dismissed via the Land and Environment Court (LEC) • Proportion of decisions subject to appeal which are upheld by the LEC • Proportion of decisions subject to appeal which are upheld with amended plans by the LEC • Proportion of decisions subject to appeal which are withdrawn • Total value of development applications approved • Total value of development applications approved by development type • Number of development applications that impact or affect a heritage item

CDC specific

• Number of complying development certificate (CDC) applications lodged • Proportion of CDCs issued or refused • Number of CDCs issued by development type • Proportion of CDCs determined by accredited private certifiers or council certifiers • Timeframes taken by certifier to determine CDC applications • Proportion of CDC applications subject to public notification • Number/proportion of complying development certificate applications issued or refused consent • Total value of complying development certificates issued • Total value of complying development certificates issued by development type • Number of site / building inspections undertaken in accordance with the requirements of the

Environmental Planning and Assessment Act 1979 and Regulations 2000.

Other – Building and subdivision certificates (e.g. Subdivision certificates, Construction Certificates, Occupation Certificates, Compliance Certificates)

• Number of subdivision applications approved • Number of construction certifications / occupation certificates (final and interim) issued • Number of subdivision works certificates / subdivision certificates issued • Number of site / building inspections undertaken in accordance with the requirements of the

Environmental Planning and Assessment Act 1979 and Regulations 2000 • Number of enforcement notices issued by different types


Have your say

Do you agree with the proposed standard metrics?

What are some important data metrics that you would like to be included as standard measures?

Data provisioning for DDA platform

Enabling the monitoring of priorities – staged release as per system build

The capture of these standard statistics and other yet to be determined non-standard statistics will be catalogued and

representative of standard reporting and analysis of development assessment-based metrics for NSW. Providing several

evidence-based outcomes, the DDA – data warehouse will highlight where planning progress is being made or where policy

and legislative reform is necessary.

Targets: 60% of DAs onboarded by mid-2019

As ePlanning roll out the DDA – data warehouse strategy and successfully onboard and align councils and private certifiers

with the technical data specification, a milestone of 60% of new DA / CDC assessments are expected to flow into the data

warehouse by mid-2019. Using the 2017-18 LDPM survey as an estimate, 60% of determined DA’s and CDC’s were derived

from 26 councils; an 80% threshold was realised with 47 councils (see map 2 and figure 8).

Map 2: Councils within 60% and 80% thresholds using 2017-18 LDPM survey


Figure 8: Determined DA’s and CDC’s, 2017-18

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

Central Darling Shire CouncilWarren Shire Council

Walgett Shire CouncilHay Shire Council

Walcha CouncilCarrathool Shire Council

Lachlan Shire CouncilBalranald Shire Council

Gwydir Shire CouncilWeddin Shire Council

Narromine Shire CouncilJunee Shire Council

Kyogle CouncilMoree Plains Shire Council

Liverpool Plains Shire CouncilNarrabri Shire Council

Muswellbrook Shire CouncilForbes Shire Council

Oberon CouncilLeeton Shire Council

Upper Lachlan Shire CouncilBroken Hill City Council

Gunnedah Shire CouncilUpper Hunter Shire Council

Inverell Shire CouncilHunters Hill Municipal Council

Greater Hume Shire CouncilFederation Council

Lithgow City CouncilBurwood CouncilSingleton Council

Mosman Municipal CouncilAshfield Municipal Council

Yass Valley CouncilLane Cove Council

Kempsey Shire CouncilBotany Bay City Council

Goulburn Mulwaree CouncilOrange City Council

Rockdale City CouncilBega Valley Shire Council

City of Canada Bay CouncilShellharbour City Council

Clarence Valley CouncilNorth Sydney Council

Tamworth Regional CouncilQueanbeyan-Palerang Regional Council

Byron Shire CouncilCanterbury City Council

Georges River CouncilWagga Wagga City Council

Port Stephens CouncilRandwick City Council

Ku-ring-gai CouncilWollondilly Shire Council

Ryde City CouncilPort Macquarie-Hastings Council

City of ParramattaBankstown City CouncilNewcastle City Council

Sutherland Shire CouncilPenrith City Council

The Hills Shire CouncilCamden Council

Central Coast Council

60% of determined DA’s / CDC’s 2017-18

80% of determined DA’s / CDC’s 2017-18


Conclusion: collecting, reporting, and presenting of planning information

The DDA – data warehouse will function as the repository of captured data from council and private certifiers.

Disrupting how DA and CDC data is managed, the digitisation of all development data will support change in

data analytics and reporting. Unearthing new value pools, the DDA - data warehouse will enhance and evolve

customer decision journeys via new data processes and capabilities such as near real-time data provisioning.

Moreover, as greater volumes of data are captured, DDA’s digital strategy will make access to any number of

touch points better, faster, and more efficient. Helping assemble an integrated story-board of data reporting

touchpoints for each stakeholder group, the building of analytical capabilities will harness new mindsets.

Challenging ideas and uncovering problems, better data coordination will bring data hotspots to the attention of

end users, which will impact operational and strategic priorities amongst decision / policy makers.

As the DDA – data warehouse is integrated into the broader ePlanning technical infrastructure, it will become

ePlanning’s data engine room. Representing the single source of truth for DA’s and CDC’s, its role as a digital

factory will be best realised via the delivery of its data servicing platform. In this capacity, several digital data

services will be made available to end users. These include the provision of the data via an API where the data

can be exported into any number of output formats. There are also web-interfacing tools such as Power BI or

Tableau, where standard or customised reports can be assembled using the principles of SaaS (software as a

service). The data will also be made available as an ArcGIS rest service for integration into spatial technologies.

Have your say

Do you agree with the delivery and reporting tools for the DDA - data warehouse?

How would you like to access / be provided with these data / standard metrics?

Appendices


Appendix A: Roadmap for the DDA project

Pre-project

Consultation with procurement vendors – design of new technical data specification (Jan – June 2018)

In the first half of 2018, ePlanning re-engaged with council property-information system vendors to better

understand the challenges and opportunities associated with automatically extracting DA and CDC data.

Drawing on a data specification that was designed and subsequently developed into production by 27 councils

in 2014, dialogue with the vendors highlighted how ATDIS could be improved to deliver a more comprehensive

technical data specification and upgraded data extraction mechanism.

This extensive engagement process with the vendors helped ePlanning design a draft specification, business

requirements, and implementation plan for the DDA – data warehouse.

Project design

Consultation with working group - feedback to technical data specification, data capture methodology (Oct – Nov 2018)

In October 2018, a working group of councils was selected using an ePlanning criteria methodology. It included

a mix of metropolitan and regional councils according to high and low DA / CDC application volumes, and

representation across each of the vendors.

A consultation session was held on 22.10.2018 in Sydney. The working group comprised 20 councils (see map

3, p.26) whom were complimented with attendees from the four key vendors and major private certifier groups,

including the BPB (Builders Professional Board). The consultation was interactive allowing participants to express

and annotate their views on the draft technical data specification, its gaps, the challenges they would face in

meeting the requirements, and the resources necessary to build the API to extract the specified data fields into

the data warehouse.


Feedback and follow up sessions in Parramatta (12.11.2018) and Newcastle (15.11.2018) delved into more field

specific details, such as development types and the handling of certificate and inspection data.

Map 3: DDA working group councils

MVP 1

Build of data warehouse to host captured data (Dec 2018)

As noted in the executive summary, MVP 1 is currently being built using LDPM and ATDIS. These will act as the

test and validation sources for DDA’s technical workflow.

Public exhibition

Public exhibition (Jan – Feb 2019)

The public exhibition will provide the opportunity to express views and provide submissions on the technical

data specification, challenges meeting the requirements, and resourcing / scheduling issues associated with

the design and delivery of the necessary API’s.

Feedback from working group and public consultation (Jan – Feb 2019)

The feedback from the working group has already been embedded into the draft technical data

specification, and if applicable, any feedback from the public exhibition will be considered and integrated

accordingly.


MVP 2

Defining the build of the specification and the data capture methodology (Dec – Jan 2019)

These communications represent the opportunity to express views and provide feedback on the draft technical

data specification. In response, ePlanning expect councils to be aligned with the final technical data

specification and the technical requirements in terms of data fields, data types etc.

In preparation, ePlanning will carry out the necessary data mapping with each council to ensure an appropriate

level of alignment exists with ePlanning’s data governance framework for the DDA – data warehouse. Data

alignment is critical if good quality data is going to be captured.

Once data alignment is established, MVP 2 will focus on the capture, transformation, and storage of data relating

to the technical data specification. ePlanning will work in an advisory capacity with councils to support them in

the design, delivery and testing of their data capture mechanisms (API’s).

Integration into BI tools and broader ePlanning digital data services (Dec 2018 and into 2019)

As part of MVP 2, a delivery mechanism will be made available to end users to test the data using BI tools and

web-based mapping tools. In due course, a delivery-based API will be designed and developed which

registered users will be able to integrate into their own technical environments.


Appendix B: Core fields within Local Development Performance Monitoring (LDPM)

• Application type

• Council reference no.

• Address fields (lot no. / block, DP no., section no., longitude / latitude, apartment / unit no., house

no., street name, suburb, postcode

• Estimated construction value

• Date of lodgment

• Category of development

• Concurrence (Y / N)

• Designated development (Y / N)

• Integrated development (Y / N)

• Stop clock days

• Referral days

• Date of determination

• Level of determination

• Type of determination

• New dwellings no.

• Pre-existing dwellings no.

• New dwellings to be demolished no.

• Zoning

• Lot size

• SEPP control

• s96 category (Y / N)

• Private assessment (Y / N)

• Use of IHAP (Y / N)

• Submission no.

• Pre-DA meeting held (Y / N)

• Change of use (Y / N)

• Advertised (Y / N)

• Description

• Political donation by developer disclosed (Y / N)

• Political donation by person making submission disclosed (Y / N)

• Affordable rental housing SEPP category of development (Y / N)

• New affordable rental housing dwellings no.

• Site compatibility certificate issued (Y / N)

Discussion Paper Title Goes Here - Store & Retrieve Data ......programming interface), this...

Documents

Transcript of Discussion Paper Title Goes Here - Store & Retrieve Data ......programming interface), this...