Post on 26-Jan-2017
Real Time Data Strategy And Architecture
Alan McSweeney
http://ie.linkedin.com/in/alanmcsweeney
Real Time Data Collection Strategy And Architecture Approach
• These notes are concerned with describing a generalised approach to defining a strategy for collecting (near or actual) real time, high volume data
• This is data is generated by sensors that is transmitted to a central location for processing, reporting, analysis and ultimately action
• Sensors can be regarded as logical or physical sources of streams of measurement data
• (Near) real time data can be termed Telemetry
March 8, 2016 2
Real Time Data Collection Strategy And Architecture Approach
• Approach adopted from TMForum Resource Domain Frameworx eTOM (Enhanced Telecoms Operating Model) Business Process Framework
• Approach has been generalised for real-time data and telemetry
March 8, 2016 3
Real Time Data Collection Strategy And Architecture
• Collect data from range of data sources across the organisation’s (internal and external) operating landscape
• Approach can be applied to collection of measurement data from multiple sources and of multiple types through “sensors”: −Different entities interacting with the organisation and the
gathering of data on different actions and events
• Approach can be applied to areas such as Telemetry, Big Data, Smart Metering and Internet of Things implementations and operations
March 8, 2016 4
Real Time Data Collection Strategy And Architecture
March 8, 2016 5
Real Time Data Strategy
Telemetry Strategy Big Data
Strategy
Internet Of Things
Strategy SmartX Strategy
Digital Strategy
Why Have A Real Time Data Collection Strategy And Architecture?
• Real time situational data gives rise to situational awareness and understanding which in turn presents opportunities for effective and rapid situational decisions −What is happening – usage, performance
−What can be improved
−What are the optimisation and productivity opportunities
• Real time situational data enables greater situational visibility which means increased operational intelligence
March 8, 2016 6
Organisation Operating Landscape
March 8, 2016 7
Organisation Operating Landscape
• The operating landscape of the organisation defines the number and type of interactions outside the organisation
• This operating landscape affects the decision on what and how to measure
• Not all interactions with all entities are measured
• Need to be realistic about what can be collected and processed
• Need to understand the need for sensors to collect data
March 8, 2016 8
Data Sensors – Combinations Of Options
• Sensor Type – Physical sensors are actual units such as RTUs (Remote Telemetry Units) that measure and generate data while Logical sensors are representations of data sources
• Sensor Ownership – Direct sensors are those installed and maintained by the collecting organisation while Indirect sensors are installed by a third-party
March 8, 2016 9
Sensor Type
Logical Physical
Sensor Ownership
Direct • Web Site and App Activity
and Usage Data • Internet of Things Devices • Remote Real Time Unit • Smart Devices
Indirect • Third-Party Web Site and
App Activity and Usage Data
• Third-Party Devices
Measurement Data Sensors
March 8, 2016 10
Measurement Data Sensors
• These can gather data for different measures using multiple measurement techniques
• These can be regarded as collectors of any data – usage, activity, performance, throughput
• Each measurement type will have a unit or dimension
• Logical representation of data collectors
• Need to decide on what can be measured, what to measure and how to measure it
March 8, 2016 11
Measurement Data Sensors – Decide On What And How To Measure
March 8, 2016 12
Real Time Data Architecture Complexity
March 8, 2016 13
Real Time Data Architectures Tend To Focus On The Simplicity
Of A Possible Real Time Data Collection Architecture …
… And Ignore The Complexity And Difficulties Of Sensor Installation, Operation, Maintenance, Errors,
Rework, Logistics, Service Management, Data Volumes
Real Time Data Architecture Complexity
• Structured approach is intended to ensure that complexity is understood and can be appropriately addressed at an early stage before problems become to embedded to be solved
March 8, 2016 14
Real Time Data Architecture Complexity
March 8, 2016 15
Don’t Let The Ignored But Knowable And Addressable Complexity Sink Your Real
Time Data Programme/Initiative
Real Time Data Strategy And Architecture Issues
• What business benefits will a real time data strategy yield?
• How can the benefits be realised?
• What is the business case for investment in a real time data strategy?
• What infrastructure, communications/connectivity, data and application architectures are needed to support a real time data strategy?
• What integration is required with existing applications?
• What skills, capabilities and changes does the organisation need to adopt and exploit a real time data strategy?
• How is the real time data strategy managed and serviced?
• What solution and service providers and tools/platforms and sourcing strategy should be selected?
• What are the privacy and security issues and requirements?
March 8, 2016 16
Evolution Of Real Time Data Architecture
March 8, 2016 17
Create Awareness for Real Time Data
Investment
Undefined/unarticulated/uncertain real time data strategy. Real time data processes are ad hoc, focussed on individual solution and
outcomes vary widely.
Cra
wl
W
alk
Ru
n
F
ly
Building Real Time Data Investment
Foundation
Implement investment controls and develop key foundational capabilities
Developing Complete Real
Time Data Portfolio
Comprehensive selection and control processes with benefit and risk criteria
linked to strategy requirements
Improving Real Time Data Processes
Process evaluation techniques focus on improvement of performance and
management
Leveraging for Strategic Outcomes
Real time data management, reporting and analysis techniques are deployed for strategic mission/business outcomes
Admitting
There must be a better way
Communicating
Establishing and communicating business case
Governing
Making and implementing effective investment
decisions
Managing
Processes, mechanisms and metrics
Optimising
Sense and respond
Developing A Real Time Data Strategy – Generalised High Level Steps
March 8, 2016 18
Real Time Strategy And
Planning
Real Time Capability Delivery
Real Time Development
And Retirement
Real Time Management
and Operations
Support And Readiness
Real Time Provisioning
Real Time Data
Collection And
Distribution
Real Time Problem
Management
Real Time Performance Management
Workforce Management
Real Time Data
Aggregation and
Reporting
4 1 2 3 5
10 9 8 7 6
Developing A Real Time Data Strategy – Generalised High Level Steps
• Comprehensive set of steps from definition of what is required from real time data to commissioning of real time data collection facilities to effective use of collected data
• Not all steps are relevant to all real time data initiatives − For example, if installation, commissioning, operation and maintenance
of physical sensors is not applicable then related steps will not be required
• Represents an idealised organisation and process breakdown across entire spectrum of real time data from strategy to workforce management to device installation, data collection and data usage and actioning
• Provides a basis for developing a work breakdown and an implementation plan
• Represents a comprehensive structure that can be adapt to meet its long-term real time data needs
March 8, 2016 19
Developing A Real Time Data Strategy – Generalised High Level Steps 1 – 2
Step Scope 1. Real Time Strategy And Planning
• Develop real time data strategy, policies and plans for the organisation governed by long-term business, market, product and service needs directions
• Perform research and analysis to determine real time targets and strategies to reach the defined targets
• Understand the real time data capabilities of the existing infrastructure • Build real time data model • Define approaches to real time data quality and real time data governance • Define and agree the infrastructure needs based on market, product and service strategies of the
organisation • Manage the capabilities of the suppliers and partners to develop and deliver new real time data
capabilities and detail the approach to the deployment of new and enhanced infrastructure • Define the real time data implementation standards sought, key real time data capabilities
required, real time data support levels and approaches required, real time data design elements to be developed, and real time data cost parameters and targets.
• Define the policies relating to technical real time data sensors and their implementation
2. Real Time Capability Delivery
• Ensure that network, application and computing real time data facilities are deployed • Provide the physical real time data capabilities necessary for the ongoing operations and long-
term well-being of the organisation and ensure the basis on which all real time data capabilities and services will be constructed
• Plan real time data resource supply logistics • Plan real time data sensor installation • Verify the real time data sensor installation • Handover real time data capabilities to operations
March 8, 2016 20
Developing A Real Time Data Strategy – Generalised High Level Steps 3 – 6
Step Scope 3. Real Time Development And Retirement
• Develop new or enhance existing technologies and associated real time data types applying the capability definition or requirements defined by the Real Time Strategy And Planning step
• Decide on acquisition of real time data resources from third parties • Retire or remove technology and associated real time data resource types that are no longer
needed by the organisation
4. Real Time Management and Operations Support And Readiness
• Manage the types of real time data resources and ensure that necessary application, computing and network facilities are available and ready to implement and manage resource instances
5. Workforce Management
• Manage the direct and indirect personnel who perform work assignments or work orders relating to real time data resources installation, commissioning and maintenance as well as managing the actual activity being performed
• Report and monitor activities • Establish, manage and allocate work assignments to direct and indirect personnel • Establish and manage priority and urgent assignment capabilities to allow for modification of
work assignments as required to meet urgent and high priority conditions
6. Real Time Provisioning • Allocate, install, configure, activate and test of real time data resources to meet the defined requirements
• Resolve real time resource capacity issues, availability issues or failure conditions • Configure and activate physical and/or logical real time resources • Update of real time resource register database to reflect that the specific real time resource has
been allocated, modified or recovered
March 8, 2016 21
Developing A Real Time Data Strategy – Generalised High Level Steps 7 – 8
Step Scope 7. Real Time Data Collection And Distribution
• Collect and distribute management information and real time data between data sources and service instances and other organisation functions and processes
• Work with the real time data resource and service instances to collect usage, network and technology events and other management information for distribution to other processes within the organisation
• Handle and process command, query and other management information for distribution to resource and service instances
• Process the data and management information through actions such as filtering, aggregation, formatting, transformation and correlation of the information before presentation to other processes, real time data instances or service instances
• Perform usage reporting, fault and performance analysis, service quality management analysis, resource performance analysis of resources and services
8. Real Time Problem Management
• Manage real time data resource problems including security events • Detect, analyse, manage and report on resource alarm event notifications • Initiate and manage real time data resource problems reports • Perform real time data resource problem localization analysis and resolve problems • Reporting progress on resource trouble reports to other processes • Assign and track real time data resource problem testing and resolution activities • Managing real time data resource problem urgent conditions
March 8, 2016 22
Developing A Real Time Data Strategy – Generalised High Level Steps 9 – 10
Step Scope 9. Real Time Performance Management
• Manage, track, monitor, analyse and report on the performance of real time data resources • Identify real time data resource performance disruptions or a service performance disruptions
10. Real Time Data Aggregation and Reporting
• Manage real time data resource events by correlating and formatting them into a usable format • Report of real time data resource data
March 8, 2016 23
Sample Expansion – Step 1 – Real Time Strategy And Planning – Activities 1.1 – 1.7
March 8, 2016 24
Step Scope 1.1 Gather And Analyse Real Time Data Information
Research and analyse customer, technology, competitor and marketing information to identify new real time data requirements and industry real time data capabilities and availability.
1.2 Manage Real Time Data Research
Manage internally driven research investigations and activities which are used to provide detailed technical assessment or investigation of new and emerging real time data capabilities.
1.3 Establish Real Time Data Strategy And Architecture
Establish the real time data strategies based on market trends, future needs, technical capabilities and addressing shortcomings in existing real time data support.
1.4 Define Real Time Data Support Strategies
Define the principles, policies and performance standards for the operational organisation providing real time data support.
1.5 Produce Real Time Data Business Plans
• Develop and deliver annual and multi-year real time data plans in support of services, products and offers that include volume forecasts, negotiation for required levels of resources and budgets.
• Obtain real time data development and management as well as supply chain commitment and executive approval for the plans.
• Identify the impacts that new or modified real time data infrastructure will cause on the installed infrastructure and workforce and establish the functions and benefits that new or modified real time data will provide to users.
1.6 Develop Real Time Data Partnership Requirements
Identify the requirements for real time data capabilities to be sourced from partners or suppliers, and any real time data capabilities to be delivered internally to the organisation.
1.7 Gain Enterprise Commitment To Real Time Data Plans
Obtain organisation commitment to the resource strategy and business plans including all aspects of identification of stakeholders and negotiation to gain stakeholder approval.
Sample Expansion – Step 1 - Real Time Strategy And Planning – Activity 1.5 – Tasks 1.5.1 – 1.5.5
March 8, 2016 25
Step Scope 1.5.1 Develop And Deliver Annual/Multi Year Real Time Data Business Plans
Develop and deliver annual/multi year real time data business plans focus on developing and delivering annual and multi-year real time data in support of services, products and offers that include volume forecasts, negotiation for required levels of resources and budgets, gaining real time data development and management as well as supply chain commitment and executive approval for the plans.
1.5.2 Forecast High Level Real Time Data Demand And Capture New Opportunities
Forecast real time data demand and capture new opportunities processes ensures that budgets are assigned which allow the organisation to implement the real time data capabilities and capacity necessary for the future needs of their customers and potential customers.
1.5.3 Assess Impact Of Real Time Data Business Plans
Asses impact of real time data business plan processes assess the impacts that new or modified real time data infrastructure will cause on the installed infrastructure and workforce, and establish the functions and benefits that new or modified real time data will provide to users
1.5.4 Identify Timetables For New Real Time Data Capability Introduction
Identify timetables for new real time data capability introduction
1.5.5 Identify Logistics For New Real Time Data Capability Introduction
Identify logistics for new real time data capability introduction
Real Time Data Quality And Data Governance
• Real-time data is inherently: − High Volume – lots of sensors generating lots of data − Noisy – lots of variation, statistical noise, inaccuracies, sensor drift, errors,
incorrect calibration − Changing – data landscape subject to substantial change along the dimensions
of data sources, volumes, types − Inconsistent – different sensor types measuring different values with different
units of measure and at different intervals − Heterogeneous – very mixed data sources − Non-standardised – multiple, emerging, overlapping standards and
approaches
• Sophisticated approach to data quality and data governance must be embedded in any real-time architecture
• Traditional approach of data collection storage and analysis may need to change to handle data volumes and quality – data filtering, quality, summarisation and transformation component
March 8, 2016 26
Real Time Data Principles
March 8, 2016 27
To manage and utilise real time information as a strategic asset
To implement processes, policies, infrastructure and solutions to govern, protect, maintain and use real time information
To make relevant and correct real time information available in all business processes and IT systems for the right people in the right context at the right time
with the appropriate security and with the right quality
To exploit real time information in business decisions, processes and relations
Real Time Data And Data Governance And Data Quality
• Data Quality - measure, assess, improve, and ensure the fitness of data for use
• Data Governance - authority and control over the management of data assets
March 8, 2016 28
Real Time Data Governance
• Core function of real time data management
• Interacts with and influences each of the surrounding ten data management functions
• Data governance is the exercise of authority and control (planning, monitoring, and enforcement) over the management of data assets
• Data governance function guides how all other data management functions are performed
• High-level, executive data stewardship
• Data governance is not the same thing as IT governance
• Data governance is focused exclusively on the management of data assets
March 8, 2016 29
Real Time Data Governance – Definition and Goals
• Definition − The exercise of authority and control (planning, monitoring, and
enforcement) over the management of data assets
• Goals − To define, approve, and communicate data strategies, policies,
standards, architecture, procedures, and metrics
− To track and enforce regulatory compliance and conformance to data policies, standards, architecture, and procedures
− To sponsor, track, and oversee the delivery of data management projects and services
− To manage and resolve data related issues
− To understand and promote the value of data assets
March 8, 2016 30
Real Time Data Governance Structure
March 8, 2016 31
Real Time Data Governance Framework
Real Time Data Architecture to Implement
Data Governance
Real Time Data Infrastructure to Implement Data
Architecture
Real Time Data Operations to Manage
Data Infrastructure
Real Time Data Governance Activities
March 8, 2016 32
Real Time Data Governance
Real Time Data Management Planning
Understand Strategic Enterprise Real Time Data Needs
Develop and Maintain the Real Time Data Strategy
Establish Real Time Data Professional Roles and Organisations
Identify and Appoint Real Time Data Stewards
Establish Real Time Data Governance and Stewardship Organisations
Develop and Approve Real Time Data Policies, Standards, and Procedures
Review and Approve Real Time Data Architecture
Plan and Sponsor Real Time Data Management Projects and Services
Estimate Real Time Data Asset Value and Associated Costs
Real Time Data Management Control
Supervise Real Time Data Professional Organisations and Staff
Coordinate Real Time Data Governance Activities
Manage and Resolve Real Time Data Related Issues
Monitor and Ensure Regulatory Compliance
Monitor and Enforce Conformance withReal Time Data Policies, Standards and Architecture
Oversee Real Time Data Management Projects and Services
Communicate and Promote the Value of Real Time Data Assets
Real Time Data Governance Inputs And Outputs
March 8, 2016 33
•Business Goals •Business Strategies •IT Objectives •IT Strategies •Data Needs •Data Issues •Regulatory Requirements
Inputs
•Business Executives •IT Executives •Data Stewards •Regulatory Bodies
Suppliers
•Intranet Website •E-Mail •Metadata Tools •Metadata Repository •Issue Management Tools •Data Governance KPI •Dashboard
Tools
•Executive Data Stewards •Coordinating Data Stewards •Business Data Stewards •Data Professionals •DM Executive •CIO
Participants
•Data Policies •Data Standards •Resolved Issues •Data Management Projects and Services •Quality Data and Information •Recognised Data Value
Primary Deliverables
•Data Producers •Knowledge Workers •Managers and Executives •Data Professionals •Customers
Consumers
•Data Value •Data Management Cost •Achievement of Objectives •# of Decisions Made •Steward Representation / Coverage •Data Professional Headcount •Data Management Process Maturity
Metrics
Real Time Data Governance
Real Time Data Quality Management
• Critical support process in organisational change management • Data quality is synonymous with information quality since poor
data quality results in inaccurate information and poor business performance
• Data cleansing may result in short-term and costly improvements that do not address the root causes of data defects
• More rigorous data quality program is necessary to provide an economic solution to improved data quality and integrity
• Institutionalising and operationalising processes for data quality oversight, management, and improvement hinges on identifying the business needs for quality data and determining the best ways to measure, monitor, control, and report on the quality of data
• Continuous process for defining the parameters for specifying acceptable levels of data quality to meet business needs, and for ensuring that data quality meets these levels
March 8, 2016 34
Real Time Data Quality Management – Definition and Goals
• Definition − Planning, implementation, and control activities that apply quality
management techniques to measure, assess, improve, and ensure the fitness of data for use
• Goals − To measurably improve the quality of data in relation to defined
business expectations
− To define requirements and specifications for integrating data quality control into the system development lifecycle
− To provide defined processes for measuring, monitoring, and reporting conformance to acceptable levels of data quality
March 8, 2016 35
Real Time Data Quality Management Inputs And Outputs
March 8, 2016 36
•Business Requirements •Data Requirements •Data Quality Expectations •Data Policies and Standards •Business metadata •Technical metadata •Data Sources and Data Stores
Inputs
•External Sources •Regulatory Bodies •Business Subject Matter Experts •Information Consumers •Data Producers •Data Architects •Data Modelers
Suppliers
•Data Profiling Tools •Statistical Analysis Tools •Data Cleansing Tools •Data Integration Tools •Issue and Event Management Tools
Tools
•Data Quality Analysts •Data Analysts •Database Administrators •Data Stewards •Other Data Professionals •DRM Director •Data Stewardship Council
Participants
•Improved Quality Data •Data Management •Operational Analysis •Data Profiles •Data Quality Certification Reports •Data Quality Service Level Agreements
Primary Deliverables
•Data Value Statistics •Errors / Requirement Violations •Conformance to Expectations •Conformance to Service Levels
Metrics
Real Time Data Quality
Management •Data Stewards •Data Professionals •Other IT Professionals •Knowledge Workers •Managers and Executives Customers
Consumers
Real Time Data Quality Plan Definition Activities
March 8, 2016 37
Real Time Data Quality Plan Definition
1. Develop and Promote Data Quality Awareness
2. Define Data Quality Requirements
3. Profile, Analyse and Assess Data Quality
4. Define Data Quality Metrics
5. Define Data Quality Business Rules 6. Test and Validate Data Quality
Requirements
7. Set and Evaluate Data Quality Service Levels
8. Continuously Measure and Monitor Data Quality
9. Manage Data Quality Issues 10. Clean and Correct Data Quality
Defects
11. Design and Implement Operational Data Quality Management Procedures
12. Monitor Operational Data Quality Management Procedures and
Performance
Developing A Real Time Data Strategy – Generalised High Level Steps
• 10 high level steps with activities and tasks
• Over 180 detailed tasks for a complete view of work required
• Comprehensive set of steps from definition of what is required from real time data to commissioning of real time data collection facilities to effective use of collected data
March 8, 2016 38
Real Time Data Strategy and
Implementation
1 Real Time Data Strategy And Planning
Activities
Tasks
2 Real Time Data
Capability Delivery
3 Real Time Data
Development And
Retirement
4 Real Time Data
Management and
Operations Support And
Readiness
5 Workforce Management
6 Real Time Data
Provisioning
7 Real Time Data
Collection And
Distribution
8 Real Time Data Trouble Management
9 Real Time Data
Performance Management
10 Real Time Data
Aggregation and
Reporting
Real Time Data Strategy and Implementation – Organisation, Function And Process Structure – Steps 1-10
March 8, 2016 39
Real Time Data Strategy and
Implementation
1 Real Time Data Strategy And
Planning
1.1 Gather And Analyse Real Time Data Information
1.2 Manage Real Time Data Research
1.3 Establish Real Time Data
Strategy And Architecture
1.4 Define Real Time Data
Support Strategies
1.5 Produce Real Time Data
Business Plans
1.6 Develop Real Time Data
Partnership Requirements
1.7 Gain Enterprise
Commitment To Real Time Data
Plans
2 Real Time Data Capability Delivery
2.1 Map And Analyse Real Time
Data Requirements
2.2 Capture Real Time Data Capability Shortfalls
2.3 Gain Real Time Data Capability
Investment Approval
2.4 Design Real Time Data
Capabilities
2.5 Enable Real Time Data
Support And Operations
2.6 Manage Real Time Data
Capability Delivery
2.7 Manage Handover To Real
Time Data Operations
3 Real Time Data Development And
Retirement
3.1 Gather And Analyse New Real Time Data Ideas
3.2 Assess Performance Of
Existing Real Time Data
3.3 Develop New Real Time Data
Business Proposal
3.4 Develop Detailed Real
Time Data Specifications
3.5 Manage Real Time Data
Development
3.6 Manage Real Time Data
Deployment
3.7 Manage Real Time Data Exit
4 Real Time Data Management and
Operations Support And
Readiness
4.1 Enable Real Time Data
Provisioning
4.2 Enable Real Time Data
Performance Management
4.3 Support Real Time Data Trouble
Management
4.4 Enable Real Time Data
Collection And Distribution
4.5 Manage Real Time Data Inventory
4.6 Manage Logistics
5 Workforce Management
5.1 Manage Schedules and Appointments
5.2 Plan and Forecast
Workforce Management
5.3 Administer and Configure
Workforce Management
5.4 Report Workforce
Management
5.5 Manage Work Order Lifecycle
6 Real Time Data Provisioning
6.1 Allocate And Install Real Time
Data
6.2 Configure And Activate Real Time
Data
6.3 Test Real Time Data
6.4 Track And Manage Real Time Data Provisioning
6.5 Report Real Time Data
Provisioning
6.6 Close Real Time Data Order
6.7 Issue Real Time Data Orders
6.8 Recover Real Time Data
7 Real Time Data Collection And
Distribution
7.1 Collect Management
Information And Data
7.2 Process Management
Information And Data
7.3 Distribute Management
Information And Data
7.4 Audit Management And
Security Data Collection And
Distribution
8 Real Time Data Trouble
Management
8.1 Survey And Analyse Real Time
Data Trouble
8.2 Localise Real Time Data Trouble
8.3 Correct And Resolve Real Time
Data Trouble
8.4 Track And Manage Real Time
Data Trouble
8.5 Report Real Time Data Trouble
8.6 Close Real Time Data Trouble
Report Flow
8.7 Create Real Time Data Trouble
Report
9 Real Time Data Performance Management
9.1 Monitor Real Time Data
Performance
9.2 Analyse Real Time Data
Performance
9.3 Control Real Time Data
Performance
9.4 Report Real Time Data
Performance
9.5 Create Real Time Data
Performance Degradation
Report
9.6 Track And Manage Real Time Data Performance
Resolution
9.7 Close Real Time Data
Performance Degradation
Report
10 Real Time Data Aggregation and
Reporting
10.1 Aggregate Real Time Data
Records
10.2 Report Real Time Data Records
Step 1 – Real Time Data Strategy And Planning – Processes And Functions Details
March 8, 2016 40
1 Real Time Data Strategy And
Planning
1.1 Gather And Analyse Real Time Data Information
1.1.1 Gather Real Time Data
Information
1.1.2 Analyse New Real Time Data Requirements
1.1.3 Analyse To Develop
New/Enhance Real Time Data
Requirements
1.2 Manage Real Time Data Research
1.2.1 Manage Real Time Data Research
Investigations
1.2.2 Manage Administration Of
Real Time Data Research
1.2.3 Define Real Time Data Research
Assessment Methodologies
1.3 Establish Real Time Data Strategy And Architecture
1.3.1 Establish Real Time Data Strategy
1.3.2 Develop Real Time Data Strategy
1.3.3 Establish Real Time Data Delivery
Goals
1.3.4 Establish Real Time Data
Implementation Policies
1.4 Define Real Time Data Support
Strategies
1.4.1 Define Real Time Data Support
Principles
1.4.2 Define Real Time Data Support
Policies
1.4.3 Define Real Time Data Support
Performance Standards
1.5 Produce Real Time Data Business
Plans
1.5.1 Develop And Deliver
Annual/Multi Year Real Time Data Business Plans
1.5.2 Forecast High Level Real Time
Data Demand And Capture New Opportunities
1.5.3 Assess Impact Of Real Time Data
Business Plans
1.5.4 Identify Timetables For New
Real Time Data Capability
Introduction
1.5.5 Identify Logistics For New
Real Time Data Capability
Introduction
1.6 Develop Real Time Data
Partnership Requirements
1.6.1 Identify The Requirements For
Real Time Data Capabilities
1.6.2 Recommend Real Time Data
Partnership
1.6.3 Determine Extent Of Real Time
Data Capabilities Sourcing
1.7 Gain Enterprise Commitment To Real Time Data
Plans
1.7.1 Identify Stakeholders To Real Time Data
Strategy And Real Time Data Plans
1.7.2 Gain Real Time Data Strategy And
Real Time Data Plans Stakeholders
Approval
1.7.3 Gain Enterprise
Commitment To Real Time Data
Strategy And Real Time Data Plans
Step 2 - Real Time Data Capability Delivery – Processes And Functions Details
March 8, 2016 41
2 Real Time Data Capability Delivery
2.1 Map And Analyse Real Time
Data Requirements
2.1.1 Capture Real Time Data Demand And Performance
Requirements
2.1.2 Agree Real Time Data
Infrastructure Requirements
2.2 Capture Real Time Data Capability
Shortfalls
2.2.1 Capture Real Time Data Capacity
Shortfalls
2.2.2 Capture Real Time Data
Performance Shortfalls
2.2.3 Capture Real Time Data
Operational Support Shortfalls
2.3 Gain Real Time Data Capability
Investment Approval
2.3.1 Develop Real Time Data Capability
Investment Proposals
2.3.2 Approve Real Time Data Capability
Investment
2.4 Design Real Time Data Capabilities
2.4.1 Define Real Time Data Capability
Requirements
2.4.2 Specify Real Time Data Capability
Infrastructure
2.4.3 Select Real Time Data Capability
At Other Parties
2.5 Enable Real Time Data Support And
Operations
2.5.1 Design Real Time Data
Operational Support Process
Improvements
2.5.2 Identify Real Time Data Support Groups, Skills And
Training
2.5.3 Identify Real Time Data Support
Requirements
2.6 Manage Real Time Data Capability
Delivery
2.6.1 Co-Ordinate Real Time Data
Capability Delivery
2.6.2 Ensure Real Time Data Capability
Quality
2.6.3 Manage Commissioning Of
New Real Time Data Infrastructure
2.6.4 Establish Real Time Data Capability
Sourcing
2.7 Manage Handover To Real
Time Data Operations
2.7.1 Co-Ordinate Real Time Data
Operational Handover
2.7.2 Validate Real Time Data
Infrastructure Design
2.7.3 Ensure Real Time Data Handover
Support
Step 3 - Real Time Data Development And Retirement – Processes And Functions Details
March 8, 2016 42
3 Real Time Data Development And
Retirement
3.1 Gather And Analyse New Real Time Data Ideas
3.1.1 Gather Real Time Data
Information
3.1.2 Analyse Real Time Data Classes
3.1.3 Develop Real Time Data Classes
3.2 Assess Performance Of
Existing Real Time Data
3.3 Develop New Real Time Data
Business Proposal
3.3.1 Develop Real Time Data Business
Proposal
3.3.2 Gain Real Time Data Business
Proposal Approval
3.4 Develop Detailed Real Time Data Specifications
3.4.1 Develop Detailed Real Time
Data Technical Specifications
3.4.2 Develop Detailed Real Time
Data Support Specifications
3.4.3 Develop Detailed Real Time Data Operational
Specifications
3.4.4 Develop Detailed Real Time
Data Manuals
3.5 Manage Real Time Data
Development
3.5.1 Identify Required Processes And Procedures For
Real Time Data
3.5.2 Develop Required Processes And Procedures For
Real Time Data
3.5.3 Develop Service And Operational
Agreements For Real Time Data
3.5.4 Gain Service And Operational
Agreements Approval For Real
Time Data
3.5.5 Produce Supporting
Documentation And Training Packages
For Real Time Data
3.6 Manage Real Time Data
Deployment
3.6.1 Manage Real Time Data Process
And Procedure Implementation
3.6.2 Manage Real Time Data
Operational Staff Training
3.6.3 Develop Real Time Data
Supplier/Partner Operational Support
3.6.4 Manage Real Time Data
Acceptance Testing
3.7 Manage Real Time Data Exit
3.7.1 Identify Unviable Real Time
Data
3.7.2 Identify Impacted Real Time
Data Customers
3.7.3 Develop Real Time Data Transition
Strategies
3.7.4 Manage Real Time Data Exit
Process
Step 4 - Real Time Data Management and Operations Support And Readiness – Processes And Functions Details
March 8, 2016 43
4 Real Time Data Management and Operations Support And
Readiness
4.1 Enable Real Time Data Provisioning
4.1.1 Plan And Forecast Real Time Data Infrastructure
Requirements And Manage Capacity Planning
4.1.2 Establish, Manage, And Develop Organisetion, Tools And
Processes
4.1.3 Develop And Implement Capacity And Operational Rules
And Procedures
4.1.4 Perform Acceptance Test And Address And Monitor The
Change
4.1.5 Track And Supervise The Rollout Of New And/Or Modified Infrastructure
4.1.6 Monitor, Report And Release Mgmt. Of Real Time
Data Infrastructure And Capacity Utilisation
4.1.7 Optimise Existing Real Time Data Infrastructure
Utilisation
4.1.8 Track, Monitor And Report Real Time Data Provisioning
4.1.9 Update Inventory Record
4.2 Enable Real Time Data Performance Management
4.2.1 Monitor And Manage Regulatory Issues
4.2.2 Establish And Maintain Performance Threshold
Standards
4.2.3 Undertake Performance Trend Analysis
4.2.4 Monitor And Analyse Real Time Data Performance Reports,
And Identify Issues
4.2.5 Correlate The Performance Problem Reports And Manage
Inventory Repository
4.2.6 Manage Real Time Data Performance Data Collection
4.2.7 Establish, Maintain And Manage The Support Plans
4.2.8 Assess And Report Real Time Data Performance Management Processes
4.2.9 Provide Supporting Procedures And Quality Management Support
4.3 Support Real Time Data Trouble Management
4.3.1 Manage Real Time Data Trouble And Performance Data
Collection
4.3.2 Manage Real Time Data Infrastructure, Provisioning And
Preventive Maintenance Schedules
4.3.3 Create Report
4.3.4 Establish Warehouse And Manage Spares Including Other
Parties
4.3.5 Track, Monitor And Manage Real Time Data Trouble
Management Processes Including Other Parties
4.3.6 Provide Support For Real Time Data Trouble Management
And Support Service Problem Management Processes
4.4 Enable Real Time Data Collection And Distribution
4.4.1 Manage And Administer Real Time Data Collection And
Distribution
4.4.2 Manage Real Time Data Storage Facilities And Associated
Processes
4.4.3 Track, Monitor And Report Real Time Data Collection Processes And Capabilities
4.4.4 Identify Data Collection Issues And Report
4.5 Manage Real Time Data Inventory
4.5.1 Manage Real Time Data Inventory Database And
Processes
4.5.2 Track And Monitor Real Time Data Repository
Capabilities
4.5.3 Identify Repository Issues And Provide Reports And
Warnings
4.6 Manage Logistics
4.6.1 Manage Warehousing
4.6.2 Manage Orders
4.6.3 Track And Monitor Logistics And Manage Real Time
Data Inventory
4.6.4 Identify Logistic Issues And Provide Reports
Step 5 - Workforce Management – Processes And Functions Details
March 8, 2016 44
5 Workforce Management
5.1 Manage Schedules And Appointments
5.1.1 Workforce Management Schedule
5.1.2 Determine Work Schedule
5.1.3 Manage Reservations
5.1.4 Manage Appointments
5.2 Plan And Forecast Workforce Management
5.2.1 Forecast Demand
5.2.2 Forecast Workforce Availability
5.2.3 Adjust Durations
5.3 Administer And Configure Workforce
Management
5.3.1 Configure Work Catalog
5.3.2 Administer Human Real Time Data Catalog
5.3.3 Administer Organisation's Catalog
5.3.4 Administer Tools And Materials Catalog
5.3.5 Configure Skill Catalog
5.3.6 Configure Schedules
5.3.7 Administer Registration And Access
5.3.8 Configure Logging And Audit
5.4 Report Workforce Management
5.5 Manage Work Order Lifecycle
5.5.1 Issue Work Order
5.5.2 Analyse And Decompose Work Order
5.5.3 Assign Task
5.5.4 Dispatch Task
5.5.5 Track And Manage Work Order
5.5.6 Close Work Order
5.5.7 Report On Work Order
Step 6 - Real Time Data Provisioning – Processes And Functions Details
March 8, 2016 45
6 Real Time Data Provisioning
6.1 Allocate And Install Real Time
Data
6.1.1 Determine Real Time Data
Availability
6.1.2 Reserve Real Time Data
6.1.3 Release Real Time Data
6.1.4 Allocate Real Time Data
6.1.5 Install And Commission
Real Time Data
6.2 Configure And Activate
Real Time Data
6.2.1 Configure Real Time Data
6.2.2 Implement Real Time Data
6.2.3 Activate Real Time Data
6.3 Test Real Time Data
6.3.1 Test Specific Real
Time Data
6.3.2 Develop Test Plans
6.3.3 Capture Test Results
6.4 Track And Manage Real
Time Data Provisioning
6.4.1 Coordinate Real Time Data
Provisioning Activity
6.4.2 Track Real Time Data
Provisioning Activity
6.4.3 Manage Real Time Data
Provisioning Activity
6.4.4 Update Real Time Data
Repository
6.5 Report Real Time Data
Provisioning
6.5.1 Monitor Real Time Data
Order Status
6.5.2 Distribute Real Time Data
Order Notification
6.5.3 Distribute Real Time Data
Provisioning Reports
6.6 Close Real Time Data Order
6.7 Issue Real Time Data
Orders
6.7.1 Assess Real Time Data
Request
6.7.2 Create Real Time Data
Orders
6.7.3 Mark Real Time Data Order
For Special Handling
6.8 Recover Real Time Data
6.8.1 Develop Real Time Data Recovery Plan
6.8.2 Provide Real Time Data
Recovery Proposal
Notification
6.8.3 Request Real Time Data
Recovery Authorisation
6.8.4 Commence Real Time Data
Recovery
6.8.5 Complete Real Time Data
Recovery
6.8.6 Recover Specific Real
Time Data
Step 7 - Real Time Data Collection And Distribution – Processes And Functions Details
March 8, 2016 46
7 Real Time Data Collection And
Distribution
7.1 Collect Management
Information And Data
7.1.1 Intercept Events/Information
7.1.2 Deliver Management Information
7.2 Process Management
Information And Data
7.2.1 Determine Recipients For
Information/Data
7.2.2 Filter Information/Data
7.2.3 Aggregate Information/Data
7.2.4 Format Information/Data
7.3 Distribute Management
Information And Data
7.3.1 Distribute Information/Data
7.3.2 Manage Distribution
7.3.3 Confirm Distribution And Clean-
Up
7.4 Audit Management And Security Data
Collection And Distribution
Step 8 - Real Time Data Trouble Management – Processes And Functions Details
March 8, 2016 47
8 Real Time Data Trouble
Management
8.1 Survey And Analyse Real
Time Data Trouble
8.2 Localise Real Time Data
Trouble
8.3 Correct And Resolve Real Time
Data Trouble
8.4 Track And Manage Real
Time Data Trouble
8.5 Report Real Time Data
Trouble
8.6 Close Real Time Data
Trouble Report Flow
8.7 Create Real Time Data
Trouble Report
Step 9 - Real Time Data Performance Management – Processes And Functions Details
March 8, 2016 48
9 Real Time Data Performance Management
9.1 Monitor Real Time Data Performance
9.1.1 Manage Real Time Data Performance Data
9.1.2 Record Real Time Data Performance Data
9.1.3 Correlate Real Time Data Performance Event
Notifications
9.2 Analyse Real Time Data Performance
9.2.1 Perform Specific Real Time Data
Performance Diagnostics
9.2.2 Manage Real Time Data Performance Data
Collection Schedules
9.3 Control Real Time Data Performance
9.3.1 Instantiate Real Time Data Performance
Controls
9.3.2 Instantiate Real Time Data Trouble
Controls
9.4 Report Real Time Data Performance
9.4.1 Monitor Real Time Data Performance
Degradation Report
9.4.2 Distribute Real Time Data Quality
Management Reports And Summaries
9.5 Create Real Time Data Performance
Degradation Report
9.5.1 Generate Real Time Data Performance
Degradation Problem
9.5.2 Convert Report To Real Time Data Performance
Degradation Report Format
9.6 Track And Manage Real Time Data
Performance Resolution
9.6.1 Coordinate Real Time Data Performance
9.6.2 Request Other Parties Performance Degradation Report
Creation And Update
9.6.3 Update First In Real Time Data Testing
Results
9.6.4 Cancel Real Time Data Performance
Degradation Report
9.6.5 Escalate/End Real Time Data Performance
Degradation Report
9.6.6 Clear Real Time Data Performance
Degradation Report Status
9.6.7 Engage External Party Real Time Data
9.7 Close Real Time Data Performance
Degradation Report
Step 10 - Real Time Data Aggregation And Reporting – Processes And Functions Details
March 8, 2016 49
10 Real Time Data Aggregation And
Reporting
10.1 Aggregation Real Time Data
Records
10.1.1 Validate Real Time Data
Records
10.1.2 Normalise Real Time Data
Records
10.1.3 Convert Real Time Data
Records
10.1.4 Correlate Real Time Data
Records
10.1.5 Remove Duplicate Real
Time Data Records
10.2 Report Real Time Data Records
Using Real Time Data Strategy and Implementation Approach
March 8, 2016 50
Activity Timeline Real Time Data Strategy and Implementation Plan
1 Real Time Data Strategy And Planning
1.1 Gather And Analyse Real Time Data Information
1.1.1 Gather Real Time Data Information
1.1.2 Analyse New Real Time Data Requirements
1.1.3 Analyse To Develop New/Enhance Real Time Data Requirements
1.2 Manage Real Time Data Research
1.2.1 Manage Real Time Data Research Investigations
1.2.2 Manage Administration Of Real Time Data Research
1.2.3 Define Real Time Data Research Assessment Methodologies
1.3 Establish Real Time Data Strategy And Architecture
1.3.1 Establish Real Time Data Strategy
1.3.2 Develop Real Time Data Strategy
1.3.3 Establish Real Time Data Delivery Goals
1.3.4 Establish Real Time Data Implementation Policies
1.4 Define Real Time Data Support Strategies
1.4.1 Define Real Time Data Support Principles
1.4.2 Define Real Time Data Support Policies
1.4.3 Define Real Time Data Support Performance Standards
…
Using Real Time Data Strategy and Implementation Approach
• Use the proposed work breakdown to produce a detailed plan
March 8, 2016 51
Real Time Architecture High-Level Components
March 8, 2016 52
Data Sources Data Collection
And Data Source Management
Communications And Security
Data Integration
Data Quality/ Summary/
Filter/ Transformation
Data Storage Data Storage Infrastructure
Data Reporting and Analysis
System Management, Administration
and Control
External Systems (Asset Management,
Workforce Management)
Real Time Architecture High-Level Components
March 8, 2016 53
Component Description Data Sources These are the data collection/generation sources/sensors installed in the data collection landscape and new
sensor devices and signal data sources. Over time the approach to data collection may be standardised and old equipment replaced.
Data Collection And Data Source Management
This logical component acts as a local front-end to existing signal data collection/generation sources/sensors. It eliminates the need to replace existing devices. It offers a standard interface. It manages the sensor infrastructure and landscape
Communications And Security
This is the communications infrastructure used to securely transmit remotely collected data from local data sources/sensors and data collection units to the central facility.
Data Integration This component manages the receipt of multiple data types in multiple formats from multiple data sources.
Data Quality/ Summary/ Filter/ Transformation
This component applies data quality algorithms to intrinsically noisy real time data to make it more usable. Data may be filtered, summarised and transformed prior to storage
Data Storage This is the software component for storing in a structured manner and providing access to real time data.
Data Storage Infrastructure
This is the underlying data storage infrastructure. The volumes of real time data are potentially very substantial – hundreds of millions of data points per day. The data storage requirements could amount to thousands of Terabytes over several years. This components includes facilities for backup, recovery, archiving and deletion.
System Management, Administration and Control
This component provides facilities to manage, administer and control the overall real time system.
Data Reporting and Analysis
This provides reporting and analysis facilities to meet a wide variety of business requirements.
External Systems Real time data can be merged with other data such as asset to provide a usage dimension to static asset data and to integrate with workforce management to manage sensor infrastructure.
Possible Approaches To Real Time Data Architecture
• You can expend a great deal of time, resources and money on defining the requirements of an idealised real time data architecture before embarking on any procurement and implementation
OR
• You can research the viable products and their functionality and what other water utilities have implemented and define requirements in terms of what is realistically achievable, reducing costs and time and delivering results and learning more quickly
March 8, 2016 54
Getting Real Time Data Right
• Avoid false starts. Balance implementation urgency and pace with organisational readiness and maturity. Success requires change management
• Cannot do it all at once
• Real time data is an enabler and not an end in itself
• Embed use of real time data data in the organisation
• Don't turn it into a "finding the right tool" decision
• Recognise the interaction of governance, processes and tools that enables the organisation optimise its real time data investment
• Focus on value, risks and prioritisation with active engagement of key stakeholders
• Avoid using it solely for one-time annual budget decisions. It is about continuous alignment, tracking and benefits realisation of existing real time data systems and new projects
• Clarify governance and put a process in place that uses portfolio management approach to consider, make and enforce decisions
March 8, 2016 55
Summary
• Real time data includes Telemetry, Big Data, Smart Metering and Internet of Things
• Represents an idealised organisation and process breakdown across entire spectrum of real time data from strategy to workforce management to device installation, data collection and data usage and actioning
• Provides a basis for developing a work breakdown and an implementation plan
• Real time data infrastructure and systems without organisation and processes will yield few benefits
• Represents a comprehensive structure that an organisation can adapt to meet its long-term real time data needs
• Enables application and use of real time data to be embedded in the organisation
March 8, 2016 56