White Paper on Key Performance Indicators (KPI) for … · which initial Key Performance Indicators...

White Paper on Key Performance Indicators (KPI) for FiberNet

Prepared for Montgomery County, Maryland by CTC Technology & Energy

March 2018

FiberNet KPI Whitepaper | March 2018

ii

Contents 1 Executive Summary ................................................................................................................. 1

1.1 Reflecting the Master Plan and Organizational Study ..................................................... 1

1.2 Internal, Customer Satisfaction, and Financial KPIs ......................................................... 1

1.3 Strategic Questions .......................................................................................................... 1

1.4 Potential KPIs to Guide FiberNet ...................................................................................... 2

1.5 Process and Roadmap for Selecting KPIs ......................................................................... 5

2 Introduction ............................................................................................................................ 6

2.1 Background....................................................................................................................... 6

2.2 Objective, Intended Audience, and Approach ................................................................. 6

3 General Characteristics of KPIs ............................................................................................... 7

3.1 Why KPIs Are Needed....................................................................................................... 7

3.2 Requirements for KPIs – Defined Business Processes ..................................................... 7

3.3 Characteristics of Good KPIs ............................................................................................ 8

3.4 Standards and Best Practice Frameworks ...................................................................... 10

3.5 Metrics Measured and Tracked by Other Government Networks ................................ 18

4 FiberNet Key Goals and Objectives ....................................................................................... 22

5 Mapping KPIs to Goals and Objectives ................................................................................. 24

6 Available Data Sources to Support KPIs ................................................................................ 29

7 Readiness Assessment and Proposed Phasing of KPIs ......................................................... 33

8 Proposed Roadmap/Next Steps ............................................................................................ 39

Appendix A: Master Plan Goals and Objectives ............................................................................ 41

Appendix B: Full COBIT Enterprise Goals and Sample Metrics ..................................................... 44

Appendix C: Full COBIT IT Goals and Sample Metrics ................................................................... 49


iii

Figures Figure 1: Arlington Comcast-based APS to Own Fiber Migration ................................................. 20

Figure 2: APS Bandwidth Utilization ............................................................................................. 21

Tables Table 1: Selected COBIT Enterprise Goals and Sample Metrics ................................................... 11

Table 2: Selected COBIT IT Goals and Sample Metrics ................................................................. 13

Table 3: Enterprise and IT Goals Mapping (Primary/Secondary) ................................................. 16

Table 4: Example Enabling Processes for Cost Transparency ....................................................... 17

Table 5: KPIs of FiberNet Objectives ............................................................................................. 24

Table 6: KPI Data Sources ............................................................................................................. 29

Table 7: KPI Phasing and Process Dependencies .......................................................................... 33

Table 8: Roadmap ......................................................................................................................... 39


1

1 Executive Summary This paper presents a framework and introduces three distinct management domains within

which initial Key Performance Indicators (KPI) may be formulated, calculated, and tracked for

FiberNet by the Configuration Control Board (CCB) and Interagency Technology Policy and

Coordination Committee (ITPCC). These metrics should have strategic value, be aligned with

major objectives, be measurable, and be implementable in the near future.1

1.1 Reflecting the Master Plan and Organizational Study

The initiatives outlined in the FiberNet Master Plan of 2017 provide an excellent backdrop for a

discussion of which KPIs to track. In addition to priorities stemming from ITPCC direction,

continuation of efforts from previous years, and regular operational requirements, the Master

Plan incorporated a number of initiatives and recommendations that grew out of the

Organizational Study of 2017.

1.2 Internal, Customer Satisfaction, and Financial KPIs

The Master Plan initiatives fall mainly into three FiberNet domains: Internal (Operations),

Customer Satisfaction, and Financial (Transparency and Value Proposition). In addition, the

Organizational Study recommended taking steps to address issues related to governance, policy,

and strategy more broadly by clarifying and streamlining executive oversight and governance.

The Internal domain tracks and answers questions related to the performance of routine

operations (including traditional measures related to network performance and availability)—as

well as service delivery more generally, in terms of delivering new sites and services. It also

addresses users’ satisfaction with the services and support offered.

The Customer domain addresses the Customer view, and including capturing customer needs,

delivering services the customer wants, and doing it at high levels of satisfaction.

The Financial domain addresses questions related to financial transparency and value

propositions. Metrics in this domain focus on what is allocated and spent, and the value and

benefits achieved.

1.3 Strategic Questions

Strategic goals drive decisions about which KPI metrics should be developed, and KPI metrics also

provide useful information to drive strategic decision-making. For example:

1 Eventually all initiatives should have associated metrics, but especially for a governance view, it is important to focus on a few, strategic measures. Arlington and Fairfax counties, for example, have metrics associated with different operational processes—but for the executive pictures, they generally focus on a few key metrics such as finances, sites added/decommissioned, circuits/services offered, and capacity and performance of services.


2

• What should be the limits and scope of FiberNet?

• Should FiberNet operate the networks of smaller agencies?

• Should FiberNet be given a Letter of Agency (LOA)2 by individual ITPCC members for

acquisition of WAN circuits or internet access from third parties?

• What are the advantages of such service offerings?

• What are the risks to the FiberNet program, the County Government, and the agencies

involved?

• What are the costs and who pays?

Answers to such questions are not directly measurable in metrics. Yet, a cost-benefit KPI that

includes a risk analysis, such as relative resource strains or reliability risks versus savings in

economies of scale, can provide a very useful analysis to assist ITPCC in answering these

questions and to guide overall strategic decision-making.

Using or developing metrics that can inform strategic questions requires a clear strategy, clear

objectives, and clear processes. In our example of strategic questions above, relevant

information that would inform discussions would include current and projected capacity, desired

service offerings, ability to manage operations and configuration complexity, ability to segment

traffic for security and demarcation purposes, ability to balance needs against training and

staffing requirements, and ability to provide high-speed circuits that agencies can manage and

monitor themselves. Several of these issues can be collected as metrics—which would factor into

such strategic decision-making.

1.4 Potential KPIs to Guide FiberNet

Focusing on the Internal, Customer, and Financial domains, we would be able to address the

following types of questions:

1. Internal: Performance & Capacity Planning

KPIs in this domain could address questions such as:

o Is FiberNet meeting its service-level agreement (SLA)?

2 An LOA is a document authorizing a telecommunications provider to act on a consumer's behalf.


3

o Are any parts of the network experiencing congestion that adversely impacts

performance, or may do so in the future?

o Is the FiberNet design something that needs to be improved to deliver needed

capacity or performance?

o How much is FiberNet used by ITPCC?

o How much spare capacity does FiberNet have?

The types of measures that address these questions are available through FiberNet’s

network monitoring system. From a strategic perspective, they work hand in glove with

questions that address requirements and need—for example, from public safety or

specific agencies that require quality of service (QoS) and service-continuity guarantees.

They also work with internal processes that project future uses and needs based on

current uses and industry trends.

2. Customer: Service Delivery and Fulfillment and Satisfaction


o Does each agency consider the performance it receives from FiberNet as

comparable to service from commercial carriers/providers?

o If not, or if an agency desires better performance, how does FiberNet show that it

needs more resources to achieve higher levels of satisfaction?

Service delivery quality is generally assessed via satisfaction surveys—whether an annual

survey of administrators, clients, and users, or an automated satisfaction poll generated

as part of a help-desk process. These are subjective measures, and therefore depend in

part on users’ expectations of service—which are also reflected in SLAs, as well as being

inherent in selecting differently priced options in commercial SLAs. Formalizing SLAs and

other performance and service expectations are therefore closely related to service

delivery satisfaction.

There are also objective measures of service delivery—which require well-defined service

catalogs, processes, and service desks procedures, and which can be extracted from

service desks.

3. Financial: Financial Transparency



4

o Is it clear where funds are coming from, where they are allocated, and how they

are spent?

o Is it clear who is accountable for which areas of financial decision-making and

spending?

o How should fund expenditures be prioritized to achieve maximum effect?

Measures in this area are most often derived from official budgets (operating and capital),

but also involve more dynamic real-time tracking of spending against defined categories

of projects and initiatives. While the former measures are more available, the latter

require strong enterprise accounting systems capable of tracking such costs and can take

some time to set up. In the interim, tracking of spending can be presented at routine

intervals to analyze trends and outcomes.

At the most granular level, such tracking also enables answers to broader strategic

questions related to governance, such as: Do we have the correct sourcing strategy? Are

we using the most efficient contractual vehicles? Do we accurately capture project costs?

4. Financial: Value Proposition

KPIs in this domain would address questions such as:

o How do we perform at a high level and use KPIs to demonstrate that we are

delivering exceptional value for the County?

o How do we show that FiberNet is delivering for each agency at an exceptional

level?

o How is dense wavelength division multiplexing (DWDM) allowing FiberNet to

better provide an exceptional service at a reasonable cost point?

Measures in this area vary. For example, showing avoided costs against expended costs

is relatively simple at the aggregate level to demonstrate cost-effectiveness and the

financial value of FiberNet. At the level of an individual link, more granular sets of

measures and calculations might be necessary to capture real FiberNet costs for a

particular circuit to be activated and operated. Likewise, for specific uses/departments

with higher needs for security and operational continuity, formal requirements would

need to be defined to allow for cost comparisons.

In addition, value propositions extend far outside pure costs (such as saved lives, flexibility

to rapidly expand or change services to meet public safety needs, etc.). Likewise, specific


5

capital improvements/upgrades such as FiberNet III delivering DWDM may allow agencies

and FiberNet to achieve certain objectives that otherwise would not be possible. If these

are defined formally as benefits/objectives, it is relatively easy to set up KPIs to measure

whether those benefits have actually been achieved or not (e.g., are they activated? Are

they used? By how many users?).

1.5 Process and Roadmap for Selecting KPIs

The KPI project has established a functioning database framework for incorporating some of the

available data that exists in Excel spreadsheets maintained by senior FiberNet staff. A preliminary

proof of concept that allows reporting and dashboard integration has also been accomplished,

utilizing Microsoft Power BI for dashboarding.

This paper provides the necessary background and guidance for the CCB to adopt a limited

number of KPIs for implementation this fiscal year. A work session with CCB will be scheduled to

facilitate this process; the KPI project will incorporate the CCB-recommended KPIs and present

its implementation to the CCB and ITPCC for feedback and review.


6

2 Introduction

2.1 Background

This White Paper on Key Performance Indicators (KPI) for FiberNet grew out of an Organizational

Study commissioned by ITPCC in 2016 and concluded in April 2017. That study explored how

FiberNet could be more effectively organized to meet its current and future needs. Among the

recommendations adopted and highlighted by the ITPCC was the conclusion that FiberNet should

adopt KPIs aligned with its strategic and operational objectives. The study emphasized that such

KPIs could support major areas of focus for ITPCC and FiberNet leadership to:

• Address transparency in funding allocations and spending (capital and operational),

including spend on agency-related projects

• Track the progress of major ITPCC-endorsed initiatives

• Better track and manage internal resources from DTS and DOT

• Better track and manage vendor and consultant support to help manage performance

and costs, allocate resources, and source needed support

• Better track and align service management to a new portfolio of services associated with

FiberNet II.v and FiberNet III

2.2 Objective, Intended Audience, and Approach

This paper is intended to facilitate discussion and prioritization of KPIs by the governing

stakeholders and managers of FiberNet. The CIO Subcommittee decided that the CCB would be

the forum best suited to developing and selecting the KPIs.

To prepare the CCB in advance of a discussion, this paper first reviews common characteristics of

effective KPIs, and best practices and KPIs often used for government networks. It provides

examples from similar regional networks, and generic KPIs from best-practice frameworks such

as ITIL, COBIT, and COSO. The paper then reviews findings and identified gaps from the

Organizational Study and associated suggested KPIs to capture performance or risk areas. These

reviews help identify candidate KPIs—which are then subjected to a high-level analysis, assessing

the current state for each KPI in terms of how ready it is for implementation in the short, middle,

or long term. Based on the readiness analysis, the paper then recommends KPIs for a pilot to

build out a database and a presentation and dashboard system.

Finally, the paper proposes a roadmap for developing and implementing KPIs based on a

roadmap that was included in the 2017 Master Plan, which was endorsed by ITPCC CIOs and

recommended for adoption to the ITPCC Principals.


7

3 General Characteristics of KPIs Key Performance Indicators, when well designed, tell a story about whether an important

objective has been met or not, and enable the tracking of trends and progress. They are also

quantitative measures with consistent calculation methodologies—which allow for comparisons

over time, and against industry benchmarks or set targets.

KPIs oriented to specific project initiatives allow for answering questions such as: Did we fulfill

our desired objectives? When can we expect to get there? What is our current progress? They

allow decision-makers and managers to intervene when performance diverges from expectations

by calling attention to both the goals and progress toward it.

KPIs that are oriented toward general performance capabilities measure processes rather than

discrete project initiatives. They answer questions such as: How well are we doing financially?

What is our performance compared to the rest of the industry? How well do we manage our

resources? How well are we performing against customer expectations?

3.1 Why KPIs Are Needed

KPIs allow organizations to learn and improve because they link comparative metrics to

strategically important objectives. They are useful as devices to communicate to and educate

both internal and external stakeholders about things that matter to the organization; similarly,

they can be used to demonstrate compliance with regulations or laws. KPIs can also serve as

mechanisms for controlling and monitoring resources and performance—alerting organizations

to the need to intervene for more efficient management.

3.2 Requirements for KPIs – Defined Business Processes

KPIs require a defined organizational process and clear objectives. For example, to understand

whether a service-level agreement (SLA) has been met, KPIs must be developed and a service

level management process needs to support the implementation, operations, and measurement

of the service.

To enable the creation of KPIs, a service catalogue must clearly define different types of services,

and the services’ performance, commissioning, and decommissioning must be tracked. (The

development of performance KPIs generally must wait until new processes or services have been

developed and have matured enough to be stable and consistent.)

To measure the KPIs, there must be processes to ensure that a monitoring system captures the

service performance for that specific site, and that the monitoring tracks activities, downtime,

restoration, and ticket closing.


8

3.3 Characteristics of Good KPIs

According to Keebler et al (1999), 3 key metrics, or what we today call KPIs, have several

characteristics:

• KPIs are quantitative. Quantitative metrics allow for comparisons and consistency in

methodology. They also allow for subsequent automation; ideally at some point KPIs can

be extracted from monitoring systems that automatically keep track of progress.

• They are easy to understand. Ideally, the measure conveys not only what it is measured,

but also how it is derived. Obscure metrics are not helpful for ensuring everyone is on the

same page regarding what is important, how the organization is doing in terms of

performance against the important objective, and how it should be measured.

• They encourage appropriate behavior. A good KPI is aligned with strategic objectives and

signals productive action toward the objectives being measured.

• They are visible. If used to encourage productive activities, KPIs need to be visible to those

whose activities are being measured. At the same time, this does not mean they need to

be universally shared. KPIs should not be used to create unnecessary conflicts (e.g., by

sharing them with people who may misappropriate them).

• This means that KPIs also need to be appropriate for their audience. Organizations often

publish different sets of KPIs for different audiences–including KPIs for public

consumption, for executives (to enable them to make strategic decisions and manage

risks and opportunities), for managers (to help them identify performance issues, trouble

spots, and misalignments with strategic objectives), and for employees (to track

performance against group goals).

• KPIs are well defined and mutually understood. Ideally, all key process participants or

owners have active input in defining them and/or agree to their definition and

methodology for measurement.

• KPIs measure only what is important. This is not always self-evident, and many so-called

KPIs published for public consumption say little of value. A KPI should say something of

value about a process or objective.

3 Keebler, J., Manrodt, K., Durtsche, D., and Ledyard, D. (1999), “Keeping Score: Measuring the value of logistics in the supply chain”. Council of Logistics Management, Chicago.


9

• KPIs should be timely. They should allow for historical comparison and trendspotting. For

unique project-based objectives, that means progress toward an end-state based on

defined criteria such as budget, timelines, and functionality achieved. For process-based

KPIs, such as “SLA performance” or “uptime,” they should be comparable within and

sometimes even across organizations for industry benchmarks.

• They should be cost-effective. The benefits of the KPI should outweigh the costs of

collection and analysis. Cost of collection and analysis can also be an indicator of the

degree of maturity of underlying processes that facilitate tracking and management of

metrics and indicators supporting the KPIs: If processes are well defined and automated,

KPIs are typically easy to generate as well.

In addition to these general characteristics, it is also important to manage expectations regarding

KPIs:

• No KPI is perfect. Collection methods, methodologies, and objectives change and become

refined, often in tandem with the maturity of processes. Transparent definitions and

methodologies allow stakeholders to get a better understanding of what the KPI can tell

and what it cannot.

• No KPI stands on its own. A single KPI cannot to tell the whole story. Organizations often

use balanced scorecards (BSC) and dashboards with aligned KPIs and metrics to allow for

different aspects and nuances of performance to be analyzed.

• Premature KPI implementation serves nobody. In theory, developing a KPI is a simple

matter of cost and resources. In other words, if something is important enough to

measure, it is almost always possible to piece together some kind of metric to capture it

(especially if enough resources are thrown at the effort). This would not only violate the

condition of cost-effectiveness, however, it would also cause stakeholders to lose trust in

a not-ready-for-primetime measure that has contestable assumptions and

approximations.

A better approach is to define simpler and rougher measures as an interim step while efforts are

made to improve processes to support better measurements down the road. For example,

getting a good KPI on project completion requires the ability to track by defined project, with

defined budgets, milestones, owners, authority and resources. But in the interim, it would be

relatively painless to focus on measures for total projects above a certain budget slated for a

fiscal year that were completed within the budget year.


10

3.4 Standards and Best Practice Frameworks

While KPIs depends on strategic objectives and therefore vary with different organizations, there

are IT governance frameworks that propose KPIs for generic processes for any organization with

formalized IT governance oriented toward continual service improvement and value

optimization. The two major best practice frameworks for IT are ITIL and COBIT.

The tables below present sample metrics for two COBIT-based generic process goals. (Full tables

are included in the appendices to this report.) In IT Governance, especially in COBIT, the foremost

concern is to ensure alignment between the overall business goals of the enterprise (in this case,

the County) and the IT serving it; in the case of FiberNet, goals and measures straddle both the

enterprise and IT since FiberNet answers to its own governance board (ITPCC) and essentially

delivers IT as its product as well. Both views are organized in a balanced scorecard (BSC)

delineation of Financial, Customer, Internal, and Learning & Growth domains.


11

Table 1: Selected COBIT Enterprise Goals and Sample Metrics

BSC

Dimensions

No. Enterprise Goals Metrics

Financial EG02 Managed business risks (safeguarding of assets)

- Percent of critical business objectives and services covered by risk assessment

- Ratio of significant incidents that were not identified in risk assessments vs. total incidents

- Update frequency of risk profile

EG04 Stakeholder value of business investments

- Percent of investments where value delivered meets stakeholder expectations

- Percent of products, and services where expected benefits realized

- Percent of investments where claimed benefits are met or exceeded

EG05 Financial transparency

- Percent of investment business cases with clearly defined, and approved expected costs, and benefits

- Percent of products and services with defined, and approved operational costs, and expected benefits

- Satisfaction survey of key stakeholders regarding the transparency, understanding, and accuracy of enterprise financial information

EG06 Customer-oriented service culture

- Number of customer service disruptions due to IT service-related incidents (reliability)

- Percent of business stakeholders satisfied that customer service delivery meets agreed-upon levels

- Number of customer complaints

- Trend of customer satisfaction survey results

EG07 Business service continuity and availability

- Number of customer service interruptions causing significant incidents- Percent of complaints as a function of committed service availability targets

Customer EG09 Information-based strategic decision-making

- Degree of board and executive management satisfaction with decision-making


12

Many of these goals and measures are less important for a government enterprise delivering

infrastructure services primarily to other government agencies and partners, in part because the

enterprise does not depend on bringing in revenue to support itself. Likewise, while conditions,

needs, and technologies do change, as they have for FiberNet, the changes are less dramatic and

frequent than in other industries and markets, where rapid change is a necessity for

organizational survival; government organizations’ need for agility therefore tends to be much

less important than cautious risk management, prudent stewardship of public funds, service

reliability, and predictable business processes.

EG11 Optimization of business process functionality

- Frequency of business process capability maturity assessments

- Trend of assessment results

- Satisfaction levels of board, and executives with business process capabilities

EG13 Managed business change programs

- Number of programs on time and within budget

- Percent of stakeholders satisfied with program delivery

- Level of awareness of business change induced by IT-enabled business initiatives

EG14 Operational and staff productivity

- Number of programs/projects on time and within budget

- Cost and staffing levels compared to benchmarks

EG15 Compliance with internal policies

- Number of incidents related to non-compliance to policy

- Percent of stakeholders who understand policies

- Percent of policies supported by effective standards, and working practices

Internal

EG16 Skilled and motivated people

- Level of stakeholder satisfaction with staff expertise and skills

- Percent of staff whose skills are insufficient for the competency required for their role

- Percent of satisfied staff


13

Table 2: Selected COBIT IT Goals and Sample Metrics

BSC

Dimensions

No. IT Goals Metrics

Financial ITG01 Alignment of IT and business strategy

- Percent of enterprise strategic goals, and requirements supported by IT strategic goals

- Stakeholder satisfaction with scope of the planned portfolio of programs, and services

- Percent of IT value drivers mapped to business value drivers

ITG03 Commitment of executive management for making IT-related decisions

- Percent of executive management roles with clearly defined accountabilities for IT decisions

- Number of times IT is on the board agenda in a proactive manner

- Frequency of IT strategy (executive) committee meetings

- Rate of execution of executive IT-related decisions

ITG04 Managed IT-related business risks

- Percent of critical business processes, IT services, and IT-enabled business programs covered by risk assessment

- Number of significant IT-related incidents that were not identified in risk assessment

- Percent of enterprise risk assessments including IT-related risks


ITG05 Realized benefits from IT-enabled investments, and services portfolio

- Percent of IT-enabled investments where benefit realization monitored through full economic life cycle

- Percent of IT services where expected benefits realized

- Percent of IT-enabled investments where claimed benefits met or exceeded


14

BSC

Dimensions


ITG06 Transparency of IT costs, benefits, and risk

- Percent of investment business cases with clearly defined, and approved expected IT-related costs, and benefits

- Percent of IT services with clearly defined, and approved operational costs, and expected benefits

- Satisfaction survey of key stakeholders regarding the transparency, understanding, and accuracy of IT financial information

Customer ITG07 Delivery of IT services in line with business requirements

- Number of business disruptions due to IT service incidents

- Percent of business stakeholders satisfied that IT service delivery meets agreed-upon service levels

- Percent of users satisfied with quality of IT service delivery

- Adequate usage of applications, information, and technology solutions

ITG08 Percent of business process owners satisfied with supporting IT products, and services

- Level of business user understanding of how technology solutions support their processes

- Satisfaction level of business users with training, and user manuals

ITG11 Optimization of IT assets, resources, and capabilities

- Frequency of capability maturity and cost optimization assessments


- Satisfaction levels of business and IT executives with IT-related costs and capabilities

Internal ITG13 Delivery of programs on time, on budget, and meeting requirements, and quality standards

- Number of programs/projects on time, and within budget

- Percent of stakeholders satisfied with program/project quality

- Number of programs needing significant rework due to quality defects


15

BSC

Dimensions


- Cost of application maintenance vs. overall IT cost

ITG15 IT compliance with internal policies



- Percent of policies supported by effective standards and working practices

- Frequency of policy review and update

Learning & Growth

ITG16 Competent and motivated IT personnel

- Percent of staff whose IT-related skills are sufficient for the competency required for their role

- Percent of staff satisfied with their IT-related roles

- Number of learning/training hours per staff

The two “views” of measuring the organization, IT and Enterprise, map onto each other. That

means any one measure does in fact often capture several primary and secondary goals and

objectives—as can be seen below:


16

Table 3: Enterprise and IT Goals Mapping (Primary/Secondary)

Source: ISACA, COBIT 5, USA, 2012

http://www.isaca.org/COBIT/Pages/default.aspx


17

The various goals and associated processes in COBIT provide a best practice governance

framework for which enabling processes, policies, and resources may have to be further

developed. For example, the enterprise goal of Financial Transparency cascades primarily to the

IT goal of Manage IT costs, benefits and risks. These in turn cascade to COBIT primary processes

(EDM02 Ensure Benefits Delivery, EDM03 Ensure Risk Optimization, EDM05 Ensure Stakeholder

Transparency, APO06 Manage Budget and Costs, APO12 Manage Risk, APO13 Manage Security,

and BAI09 Manage Assets) and secondary processes (EDM01 Ensure Governance Framework

Setting and Maintenance, EDM04 Ensure Resource Optimization, APO03 Manage Enterprise,

APO05 Manage Portfolio, APO08 Manage Relationships, APO09 Manage Service Agreements,

APO10 Manage Suppliers, BAI01 Manage Programs and Projects, BAI10 Manage Configuration,

MEA01 Monitor, Evaluate and Assess Performance and Conformance, MEA02 Monitor, Evaluate

and Assess the System of Internal Control).4

These processes in turn are supported by their own supporting process practices and activities—

so, for example, one of the supporting process practices for APO06 Manage Budget and Costs is

APO06.04 Model and Allocate Costs, which outlines the following best practice activities:

Table 4: Example Enabling Processes for Cost Transparency

Management Practice: APO06.04 Model and Allocate Costs

Establish and use an IT costing model based on the service definition, ensuring that allocation

of costs for services is identifiable, measurable and predictable, to encourage the responsible

use of resources including those provided by service providers. Regularly review and

benchmark the appropriateness of the cost/chargeback model to maintain its relevance and

appropriateness to the evolving business and IT activities.

Activities

1. Categorize all IT costs appropriately, including those relating to service providers,

according to the enterprise management accounting framework.

2. Inspect service definition catalogues to identify services subject to user chargeback and

those that are shared services.

3. Define and agree on a model that:

• Defines how IT costs will be calculated

• Is differentiated, where and when appropriate

• Is aligned with the IT budget

4 These and other COBIT processes are defined and explained in COBIT 5: Enabling Processes. ISACA.

Additional information is available from http://www.isaca.org/cobit.


18

4. Design the cost model to be transparent enough to allow users to identify their actual

usage and charges, and to better enable predictability of IT costs and efficient and effective

utilization of IT resources.

5. After review with user departments, obtain approval and communicate the IT costing

model inputs and outputs to the management of user departments.

6. Communicate changes in the cost model with enterprise process owners.

Outputs of Practice

Categorized IT costs

Cost allocation model

Cost allocation communications

Operational procedures

Internal

Internal

Internal

Internal

3.5 Metrics Measured and Tracked by Other Government Networks

Government networks typically capture and publish standard network performance metrics such

as uptime/availability. This gives stakeholders a sense of how the network compares against

commercial alternatives and an inkling of how well the network is managing capacity and

availability. In addition, government networks also tend to publish metrics that give users a sense

of the scope of the networks footprint, such as fiber miles and/or number of sites served. Many

networks also publish high-level financial metrics such as total budget and comparisons against

previous years’ budgets. Commercial operators tend to publish more detailed financial

information that relates to profitability, revenue, and market share, as well as specific measures

that are public-friendly—such as diversity hiring and number of connections for low-income

residents.

These metrics are, however, more for public and political consumption and do not provide a

meaningful picture of progress against specific objectives. Where other more meaningful KPIs

are used, they are not typically published outside their immediate stakeholder groups and even

their existence is often not shared with groups outside the organizations. Examples of metrics

that are shared more extensively—at least internally—relate to service improvement.

Operational support metrics linked to network management and service desk software is a

relatively easy target for collection and analysis if business processes have been sufficiently

developed and defined against expected service levels.

Thus, tracking service requests and ticket closure within a defined service level is a fairly typical

and meaningful metric if implemented correctly, and can be used to identify bottlenecks in

processes or resources. For internal consumption, project management tracking is standard in IT

departments with a formalized PMO. Meaningful implementation of such metrics, however,

depends heavily on two things: First, the availability of mature project management capabilities,


19

adequate middle management resources, and skills in business process analysis and project

management; and second, well-defined business processes implemented in operational and

capital projects that have clear budgets, timelines, assigned resources, and role assignments and

owners.

As an example, Arlington County, Virginia, has implemented a number of IT governance initiatives

that support its own generation of KPIs. It reported that for high-level metrics it tends only to

publish measures such as sites connected, bandwidth, types and number of services, utilization,

and cost. But they also reported that they have many more KPI as they drill down further—

following IT governance best practices. For example, Arlington developed a separate engineering

and design standards document. This in turn allows the County to track the number of contracts

and projects that explicitly refer to these standards, ensuring the County can both incentivize

that contracts and internal engineering support use up-to-date and consistent standards, but also

locate instances where such inclusions did not occur. This helps the County to identify

opportunities for better business processes or the need for better coordination with other

departments.

Another example of objective-linked KPIs relate comes from Fairfax County, Virginia, which

provides its lit service over Cox franchise fiber the county does not own. Like Montgomery

County, Fairfax needs to ensure service continuity to critical public safety and government

functions, some of which do not have on-site 24-hour technical staff. Accordingly, Fairfax’

mitigation efforts target resiliency both in the design phase and in a reworking of its monitoring

system.

For such efforts a KPI of the percentage of County CIP projects where DIT engineers reviewed

and advised the sponsor of site selection, utility design and diverse building entry is meaningful

for upfront resilience as a leading indicator. To improve proactive monitoring, the County

installed remote sensing OTDR units in its hubs on a pair of unused strands. The continuous OTDR

testing allow the County to identify potential issues in the physical cable. The County’s software

for monitoring electronics and circuits, its documentation software, GIS platform and the remote

OTDR system are all integrated into one view. This allows the County to produce KPIs related to

performance over time at each site to identify patterns that might indicate problems of a

particular nature such as power, misconfiguration, or partially crushed conduits due to

construction (and actual incident the County was able to identify before either Cox or the builder

knew that anything was amiss).

For status reporting to executives and managers, KPIs are often rolled up from more granular

automatically generated metrics or manually copied from other status tracking software or tools.

An example of what that may look like is shown below in Figure 1 and Figure 2. Note that this is


20

an example of a specialized report and not a dashboard, which would enable managers and

executives to interact with the data and drill down for specific views (operational performance,

financial management, customer relations, etc.).

Figure 1: Arlington Comcast-based APS to Own Fiber Migration


21

Figure 2: APS Bandwidth Utilization


22

4 FiberNet Key Goals and Objectives While FiberNet does have a charter for its governance, it does not spell out FiberNet’s mission or

purpose. However, the website aptly describes FiberNet as “the backbone infrastructure to

enable delivery of communications services for all County agencies,” which effectively means

that it should provide cost-effective connectivity services to the County’s traffic signaling

network, to public safety, and to partner agencies at availability and capacity that meet their

needs.

The main source for FiberNet strategic goals and objectives is the FiberNet Master Plan of 2017,

which seeks to capture and operationalize key objectives as articulated through the County and

ITPCC participation process. This Master Plan recognizes and adopts many of the

recommendations outlined in the Organizational Study; a summary list is in Appendix A.

The Master Plan initiatives fall chiefly into goal areas that strengthen particular domains of

FiberNet. The BSC framework adopted widely as a standard divides the enterprise into four main

domains, and seeks to articulate metrics that taps each of those domains. (Finance, Customer,

Internal, and Learning and Growth). The Master Plan initiatives fall mainly into three of the BSC

domains: Internal (Operations), Customer (Customer Management and Satisfaction), and

Financial (Value Proposition). Included in the Financial aspect are broader areas of policy, not

strictly financial, relating to value. In addition, the Organizational Study also recommended taking

steps to address issues related to governance, policy, and strategy more broadly by clarifying and

streamlining executive oversight and governance.

Each of the three BSC domains—Internal, Customer, Financial – and the added area of Strategy

answer different types of questions. The Internal Domain tracks and answers questions related

to performance of routine operations as well as service delivery more generally in in delivering

new sites and services. Traditional measure related to network performance and availability and

similar performance tracking is associated with this area. Tracking the addition of new sites and

circuits are likewise to be found in this domain, as are traditional service desk metrics that

capture service delivery performance.

Questions related to strategy are not typically captured in metrics, although the processes

supporting the articulation of strategy can captured in metrics—for example the number of

sessions of the ITPCC devoted to strategy discussions, the number of times strategies are formally

reviewed and revisited, and so on. Strategy questions are essential and shape the articulation of

metrics and can also be informed by them. What should be the limits and scope of FiberNet?

Should it offer to operate the networks of smaller agencies such as HOC? What are the

advantages of such service offerings? What are the risks? The answers are no measurable in

metrics, but if benefits and risks are well articulated, it is also possible to develop metrics that


23

can inform decision-making by estimating costs, capturing some of the risks such as relative

resource strains vs savings in economies of scale. But the nature of exploring strategic

considerations are such that metrics are rarely much assistance. In the end, metrics require

strong clear strategy, clear objectives, and strong clear processes.


24

5 Mapping KPIs to Goals and Objectives The initiatives identified above reflect priorities adopted by FiberNet and its executive and

management stakeholders. Below is a list of proposed KPIs that map to some of these initiatives

and in turn to broader IT goals as listed in Table 2.

Table 5: KPIs of FiberNet Objectives

KPI Area Major KPI Related KPIs/

Elaboration

Supported

IT Goals

Related Master Plan

Initiatives

Finance Maintenance and Operations Spend Against Budget

Breakdown of NOC (tiers), dispatch, and fiber maintenance, DOT vs DIT vs contractor hours per site

ITG06, ITG13

Align DOT OSP engineers to FiberNet

Improve consistency in resource assignments and tracking

Disaggregate current and future contracts and purchase orders (POs) against financial categories to better track spending in real time against specific initiatives and budget categories

Total CIP Spend Against Budget

Breakdown by type of project

ITG06, ITG13

Create FiberNet cost transparency

Develop and add project-specific sub-categorization to contracted and in-house detailed reporting

Completed CIP costs against Budget projections

ITG06 Create FiberNet cost transparency

Actual spend on CIP POs against PO size/budget



25


Elaboration

Supported

IT Goals

Related Master Plan

Initiatives

Avoided Cost Total (based on mean market pricing per mile x miles – actual total construction costs)

ITG06, ITG05, ITG01

Per site avoided cost (based on mean market pricing – actual construction cost + prorated maintenance and operations costs) -> projected payback period

ITG01, ITG06, ITG05

Per site avoided cost (based on estimated pricing for specific site – actual construction cost) + prorated maintenance and operations costs -> projected payback period

ITG06, ITG05, ITG01

Enterprise Costs Capturing real hours/allocations of DTS/DOT staff in FiberNet support

ITG06, ITG11, ITG13

Create FiberNet cost transparency

Align DOT OSP engineers to FiberNet

Improve consistency in resource assignments and tracking

Disaggregate current and future contracts and purchase orders (POs) against financial categories to better track spending in real time against specific initiatives and budget categories


26


Elaboration

Supported

IT Goals

Related Master Plan

Initiatives

Network/Site

Expansion

Total Number of Sites on Network

Number of added/decommissioned sites

ITG01

Provide connectivity services to ITPCC agencies

Add sites for the next 3-4 fiscal years

Adopt Project Management templates and formalize project tracking

Number of sites in each phase of a pipeline (candidate, requested, funded, procured, designed, in construction, await agency, on net)

ITG13

Sites on track for completion

ITG13

Number of Sites Added Out of Total

Sites funded through facility CIP

ITG06

Sites funded with FiberNet CIP

ITG06

Customer

Outreach

Consultation with/by Agency

Total hours of consultation

ITG07, ITG08

Number of meetings/satisfaction surveys regarding needs or existing services

ITG07, ITG08

Training Training/ Professional Development

Costs of Training programs

ITG16

Training slated as initiative in future Master Plan update

Number of training hours - total, per FTE

ITG16

Project

Tracking

Project Progress and Definition

Number of project types (improvement/enhancement of sites, core technology rollout, upgrade/patching etc.)


Develop and add project-specific sub-categorization to


27


Elaboration

Supported

IT Goals

Related Master Plan

Initiatives

contracted and in-house detailed reporting

Number of projects with key project metrics

ITG13 Evaluate the effectiveness of tools used for OSP projects and explore alternative management tools

Explore project management templates for O&M and CIP projects

Percent projects completed on time and budget.

ITG13 Explore project management templates for O&M and CIP projects


Number of Projects requiring rescope or reconsideration

ITG13, ITG04


Service

Management

& SLA

performance

Service Catalog Use

Number of sites subscribed to x service

ITG07 Develop a formal service catalog

Sites/Services with Uptime SLA

Number of sites with SLAs for monthly uptime not met

ITG07, ITG11, ITG15

Update SLAs for FiberNet II PON, and II.v DWDM service

Service Desk SLAs Other SLAs if defined (e.g., service desk handling in incident management process etc.)

ITG07 Develop and track KPIs

Develop and attain targets for operation and site activation

Service/Change Request Time

Time it takes from request of new service to turn on. Can be broken

ITG07


28


Elaboration

Supported

IT Goals

Related Master Plan

Initiatives

down to funded/unfunded

Develop and attain targets for operation and site activation

Develop a formal service catalog

Length of time for new site activation

ITG07

Time for change config / add new service from service catalog on existing connections

ITG07, ITG11

Provision new services, esp. DWDM links (staff resource problems) Project backlog, breaking inter-dep of projects

ITG07

Circuit Performance – Backbone links only

ITG07, ITG11

Uptime, Time to Repair – Gross uptime per site; TTR comes out of the trouble ticket system,

ITG07, ITG11

Trouble Tickets Resolution

Total volume ITG07

Mean time of resolution ITG07, ITG11


29

6 Available Data Sources to Support KPIs While some KPIs may be of stronger strategic value to FiberNet stakeholders, the readiness of

KPIs to be implemented depends on mature existing business processes to support their

articulation and tracking, and data sources to track or calculate them. The table below

summarizes preliminary research regarding data source availability.

Table 6: KPI Data Sources

KPI Area Major KPI Related KPIs/Elaboration Data Sources

Finance Maintenance and

Operations

Spend Against

Budget

Breakdown of NOC (tiers),

dispatch, and fiber

maintenance, DOT vs DIT vs

contractor hours per site

Not yet available

Total CIP Spend

Against Budget

Breakdown by type of

project Data should be available

soon in a separate

spreadsheet ledger Completed CIP costs against

Budget projections

Actual spend on CIP POs

against PO size/budget

Not yet available

Avoided Cost Total (based on mean

market pricing per mile x

miles – actual total

construction costs)

Not yet available

Per site avoided cost (based

on mean market pricing –

actual construction cost +

prorated maintenance and

operations costs) ->

projected payback period

Not yet available

Per site avoided cost (based

on estimated pricing for

specific site – actual

construction cost) +

prorated maintenance and

Not yet available


30


operations costs ->

projected payback period

Enterprise Costs Capturing real

hours/allocations of

DTS/DOT staff in FiberNet

support

Not yet available

Network/Site

Expansion

Total Number of

Sites on Network

Number of

added/decommissioned

sites

In spreadsheet database

Number of sites in each

phase of a pipeline

(candidate, requested,

funded, procured,

designed, in construction,

await agency, on net)

In spreadsheet database

Sites on track for

completion

In spreadsheet database;

will need “approved to

build” date for each project

filled in

Number of Sites

Added Out of

Total

Sites funded through facility

CIP


historical data only available

as fiscal years

Sites funded with FiberNet

CIP


historical data available only

as fiscal years

Customer

Outreach

Consultation

with/by Agency

Total hours of consultation Not yet available

Number of

meetings/satisfaction

Not yet available


31


surveys regarding needs or

existing services

Training Training/

Professional

Development

Costs of Training programs Not yet available

Number of training hours -

total, per FTE

Not yet available

Project

Tracking

Project Progress

and Definition

Number of project types

(improvement/enhanceme

nt of sites, core technology

rollout, upgrade/patching

etc.)

A basic classification with

numbers may be close to

being available in

spreadsheet database

Number of projects with

key project metrics

Not yet available

Percent projects completed

on time and budget.

Not yet available

Number of Projects

requiring rescope or

reconsideration

Not yet available.

Service

Management

& SLA

performance

Service Catalog

Use

Number of sites subscribed

to x service

Not yet available

Sites/Services

with Uptime SLA

Number of sites with SLAs

for monthly uptime not met

Not yet available

Service Desk SLAs Other SLAs if defined (e.g.

service desk handling in

incident management

process etc.)

Not yet available


32


Service/Change

Request Time

Time it takes from request

of new service to turn on;

can be broken down to

funded/unfunded

Should be able to capture it

in database/spreadsheet by

capturing request date

Length of time for new site

activation

Will be able to track in

database since it keeps start

dates

Time for change config /

add new svc from svc

catalog on existing

connections

Where are these requests

captured? Through change

management in ticket

system?

Provision new services, esp.

DWDM links (staff resource

problems) Project backlog,

breaking inter-dep of

projects

Not yet available

Circuit Performance –

Backbone links only

NMS system Uptime, Time to Repair –

Gross uptime per site; TTR

comes out of the trouble

ticket system,

Trouble Tickets

Resolution

Total volume Available? Are trouble

tickets coded for FiberNet?

For agency? Mean time of resolution


33

7 Readiness Assessment and Proposed Phasing of KPIs As mentioned in the previous section, readiness of a KPI is a function of business process maturity

and data source availability. The table below proposes when a KPI could be available for

implementation. It does not imply that these are the recommended KPIs. It is meant only as

additional guidance for FiberNet stakeholders to select the KPIs on which to focus in initial and

subsequent stages.

Phase 1 is considered to be shortly available (i.e., something that could be implemented shortly

or within 12 months). Phase 2 KPIs require definition and/or maturation of business processes or

data source creation that would require 12 to 24 months. Phase 3 KPIs require significant

business process re-engineering and possible organizational adjustments supported by the

governance board, and therefore require longer timelines.

Table 7: KPI Phasing and Process Dependencies

KPI Area KPIs/Elaboration Process Dependencies Proposed

Phase

Finance Breakdown of NOC (tiers),

dispatch, and fiber

maintenance, DOT vs DIT vs

contractor hours per site

Requires timesheet and

contractor support coding for

hours and/or categories in POs.

May require updates to

existing POs for invoicing that

allows tracking. Will require

coordination with DOT to

capture hours. May require

coding and capturing of

support tasks such as tier 1, 2,

and 3.

2

Breakdown by type of project Architect is reconstructing

numbers from past years

spends. But may not be able to

get down to project level at the

type of projects currently

defined. Going forward, a

process for capturing spends

via invoicing and recording on

ledger will need to get

formalized.

1


34


Phase

Completed CIP costs against

Budget projections

1

Actual spend on CIP POs

against PO size/budget

For real time tracking, this will

require recoding of

invoices/instructions to vendor

on how to invoice. In an

enterprise accounting system,

this would be a requirement in

the Finance and procurement

system of invoicing against

defined line items.

2

Total (based on mean market

pricing per mile x miles – actual

total construction costs)

Will require tracking of

construction costs per project.

Estimates and fiber lengths

currently captured in work files

for each project. Need a central

spreadsheet or database to

capture lifecycle of estimates

and costs, including fiber

mileage. For cost and fiber

length three estimates:

Desktop estimate (FiberNet),

Contractor walkout estimate

(currently Baldwin), actual (as-

built/power meter).

2

Per site avoided cost (based on

mean market pricing – actual

construction cost + prorated

maintenance and operations

costs) -> projected payback

period

As above, but will also require

capturing other "self-managed"

cost projections.

2

Per-site avoided cost (based on

estimated pricing for specific

site – actual construction cost)

As above 2


35


Phase

+ prorated maintenance and

operations costs -> projected

payback period

Capturing real

hours/allocations of DTS/DOT

staff in FiberNet support

May require moving FiberNet

to an NDA and assigning

relative percentage of

participation of DTS resources

(including CIO) as cost. Will

require developing imputed

revenue, and potentially sunk

cap ex and op ex for assets to

calculate actual value, and

incorporate asset depreciation.

3

Network/Site

Expansion

Number of

added/decommissioned sites

1

Number of sites in each phase

of a pipeline (candidate,

requested, funded, procured,

designed, in construction,

await agency, on net)

May need to adjust phases to

match actual lifecycle of

approvals. May need further

decomposition when

procurement process is aligned

to specific project build task.

1

Sites on track for completion Anticipated completion date

based on an assumption that

the average time to complete

from Approved to Build to

commissioning is 18 months.

This estimate would improve

as progress is made getting to

the Under-Construction state.

Need to reconstruct/fill start

dates.

1-2

Sites funded through facility

CIP

1


36


Phase

Sites funded with FiberNet CIP 1

Customer

Outreach

Total hours of consultation Will need process for capturing

such events.

2

Number of

meetings/satisfaction surveys

regarding needs or existing

services

Will need process for capturing

such events.

2

Training Costs of Training programs Requires internal identification

of needs and opportunities,

and potential knowledge gaps.

Will require separate budge

line item, and a document for

"training plan".

2

Number of training hours

(total, per FTE)

In conjunctions with planned

initiatives developing project

management methodology,

outside plant engineering and

technology to streamline

business processes in

construction. Training may be

in Project Management,

technology, and/or use of

templates/checklists.

2

Project

Tracking

Number of project types

(improvement/enhancement

of sites, core technology

rollout, upgrade/patching etc.)

A basic interim classification

may be possible, but to fully

flesh out, requires project

management methodologies

and templates to be adopted.

1

Number of projects with key

project metrics

Requires definition of Budget,

timeline, milestones,

success/testing criteria,

assigned resources

2


37


Phase

Percent projects completed on

time and budget.

As above, but also requires

spend tracking.

2

Number of projects requiring

rescope or reconsideration

As above, but also requires

processes for handling project

change management.

2

Service

Management

& SLA

Performance

Number of sites subscribed to x

service

Will require at least initial

definition and tracking of

services.

2

Number of sites with SLAs for

monthly uptime not met

Will require definition and

capture of both business hours

and 24/7 SLA’s within NOC and

service desk

2

Other SLAs if defined (e.g.,

service desk handling in

incident management process,

etc.)

2

Time it takes from request of

new service to turn on. Can be

broken down to

funded/unfunded

May need more formalized

change request process with a

service request document (like

networkMaryland/NCRnet for

example)

2

Length of time for new site

activation

May need archival research if

historical data for past couple

of years is desired.

1

Time for change config / add

new service from service

catalog on existing connections

Will require definition of

service catalog and capturing

such services in service desk

incident and change and

configuration management

processes.

2


38


Phase

Provision new services, esp.

DWDM links (staff resource

problems), Project backlog,

breaking inter-dep of projects

Data will be available once

project tracking and template

is adopted.

2

Circuit Performance –

Backbone links only

1

Uptime, Time to Repair – Gross

uptime per site; TTR comes out

of the trouble ticket system,

1

Total volume Need process for tagging

tickets in system as FiberNet-

related.

2

Mean time of resolution 2


39

8 Proposed Roadmap/Next Steps The roadmap below presents a rough timeline and series of major tasks. A more detailed project

plan is available. The deliverable at the end of the project would be a handful of implemented

KPIs in a database along with a dashboard solution for viewing them, as well as the updated KPI

catalog based on this white paper with associated assessment of maturity level/gap analysis for

required processes to support such KPIs in the future.

Table 8: Roadmap

No. Milestone Involvement/ Resources

Target Completion

Status

1 Identify candidate KPIs CTC/DTS 10/6/2017 Completed

2 Kickoff with ITPCC CIOs

ITPCC CIOs, Principals, DTS

10/2/2017 Completed

3 Identify candidate KPIs with FiberNet Architect

FiberNet Architect 10/17/2017 Completed

4 Prepare White Paper / KPI analysis for CCB

FiberNet Architect 3/14/2018 Close to Completion

5 Implement Pilot Database FiberNet Architect 12/15/2017 Completed

6 Facilitate KPI Working group brainstorming session

CCB 3/28/2018 Not Started

7

Identify resources/sources of data for KPIs

CTC/DTS for FiberNet related data, individual agencies for any agency-specific data

4/15/2018 In Progress

8 Score KPIs on maturity CTC 5/15/2018 In Progress

9 Select high priority KPIs for inclusion in dashboard

CTC, CCB 5/25/2018 Not Started

10 Identify County KPI platforms

CTC, DTS (CountyStat, Tivoli, Netcool, PowerBI, Excel)

5/20/2018 In Progress

11 Evaluate suitability of solutions for KPI dashboard

CTC, DTS, Vendor support

6/1/2018 Not Started

12 Map KPIs against Business Processes

CTC, DTS 6/15/2018 In Progress

13 Implement select KPIs for pilot DTS 6/1/2018 In Progress

14 Feedback/Validate with CCB CCB 6/10/2018 Not Started

15 Implement remaining selected KPIs

CTC, DTS 6/20/2018 Not Started

16 Feedback from CCB and rest of ITPCC

CCB, ITPCC CIOs, Principals


17 Implement feedback changes to dashboard



40

No. Milestone Involvement/ Resources

Target Completion

Status

18 Document generic business processes

CTC 7/10/2018 Not Started

19 Develop ConOps for KPI management and dashboard solution


20 Migrate Platform to County

CTC, Architect, NOC Manager


21 Generate KPI catalog CTC 7/1/2018 Not Started

22 Generate Solution Presentation and Training Presentation

CTC 7/31/2018 Not Started


41

Appendix A: Master Plan Goals and Objectives The Master Plan incorporated several initiatives targeting areas for improvement that should be

incorporated into a KPI program. The initiatives and priorities adopted in the 2017 Master Plan

include:

• Provide connectivity services to ITPCC agencies including dark fiber, DWDM, PON,

internet, WAN, telephony transport, Wi-Fi, custom internetworking, and public safety

communications backhaul

• Add sites for the next three to four fiscal years (54 sites are “in the pipeline” with 26 sites

currently “approved”)

• Continue incremental rollout of FiberNet II and II.v to sites

o Continue FiberNet II PON deployment at selected sites

▪ Update SLA for FiberNet II PON service

o Continue deployment of FiberNet II.v DWDM at selected sites for optical

segmentation or point-to-point agency connectivity needs, and evaluate with

agencies other potential sites for inclusion

▪ Update SLA for FiberNet II.v DWDM service

• Phase I core upgrade of four to six sites for FiberNet III 10 and 100 Gbps services

• Continue hub remediation activities

o Expand capability of hubs to terminate large numbers of fiber optic strands

o Improve resiliency with hardening of environmentals and electronic redundancy

o Evaluate need and feasibility for hardening resilience of hub sites for a future

remediation initiative

• Feeder fiber expansion/overbuilds to close gaps between ICBN fiber segments and hubs

• Explore new more affordable fiber deployment technologies to increase strand capacity

• Continue OSP digitization, as-built access, and procedures for as-built capture in earlier

phases of deployment, remediation, or change pf physical plant

• Adopt several initiatives deriving from the Organizational Study:


42

o Work with the County Office of Management and Budget (OMB) to develop

roadmap for migrating FiberNet to an NDA; if an NDA is not feasible, an enterprise

accounting system can be developed with an as-if-NDA approach

o Develop financial categories—some of which track against funding source—for

reporting

▪ The categories should be able to disaggregate contracts, spending, and

resource times against capital improvement/expansion vs maintenance

and operations

▪ The categories should be broad enough to allow outsourced contracts to

be comparable to in-sourced staff time to facilitate sourcing decisions, and

to allow FiberNet to identify where augmentation contract vehicles may

be desired

o Disaggregate current and future contracts and purchase orders (POs) against

financial categories to better track spending in real time against specific initiatives

and budget categories

o Analyze timesheets to enable similar disaggregation and tracking of resource time

o Develop and add project-specific sub-categorization to contracted and in-house

detailed reporting so costs can be tracked by project; FiberNet projects specifically

identified by ITPCC as supporting particular strategic objectives should, to the

extent possible, adopt naming that reflects such objectives

o Develop coordination between DOT and FiberNet at the management level to

incorporate supervision and FiberNet-related job evaluation metrics into DOT’s

formal evaluation process for shared resources

o Develop proposed financial reporting templates/dashboard—also known as key

performance indicators, or KPI—that could serve as a starting point for ITPCC,

Department of General Services (DGS), DTS, and DOT feedback

o Undertake organizational re-alignment to streamline accountability and reporting

o Develop a formal service catalog

o Review available department tools for efficiencies relating to outside plant (OSP)

project management and reporting


43

o Evaluate the effectiveness of tools used for OSP projects and explore alternative

management tools

o Explore project management templates for O&M and CIP projects

Develop FiberNet III incremental approach for modernization


44

Appendix B: Full COBIT Enterprise Goals and Sample Metrics

Intentionally Left Blank


45

SC

Dimensions

No. Enterprise Goals Metrics

Financial EG01 Compliance with external laws and regulations

- Cost of regulatory non-compliance, including settlements, and fines

- Number of regulatory non-compliance issues causing public comment or negative publicity

- Number of regulatory non-compliance issues relating to contractual agreements with business partners

EG02 Managed business risks (safeguarding of assets)

- Percent of critical business objectives and services covered by risk assessment

- Ratio of significant incidents that were not identified in risk assessments vs. total incidents


EG03 Portfolio of competitive products and services

- Percent of products, and services that meet or exceed targets in revenues, and/or market share

- Ratio of products, and services per lifecycle phase

- Percent of products, and services that meet or exceed customer satisfaction targets

- Percent of products, and services that provide competitive advantage

EG04 Stakeholder value of business investments

- Percent of investments where value delivered meets stakeholder expectations

- Percent of products, and services where expected benefits realized

- Percent of investments where claimed benefits are met or exceeded


46

EG05 Financial transparency

- Percent of investment business cases with clearly defined, and approved expected costs, and benefits

- Percent of products and services with defined, and approved operational costs, and expected benefits

- Satisfaction survey of key stakeholders regarding the transparency, understanding, and accuracy of enterprise financial information

- Percent of service cost that can be allocated to users

Customer EG06 Customer-oriented service culture

- Number of customer service disruptions due to IT service-related incidents (reliability)

- Percent of business stakeholders satisfied that customer service delivery meets agreed-upon levels

- Number of customer complaints

- Trend of customer satisfaction survey results

EG07 Business service continuity and availability

- Number of customer service interruptions causing significant incidents

- Business cost of incidents

- Number of business processing hours lost due to unplanned service interruptions

- Percent of complaints as a function of committed service availability targets

EG08 Agile responses to a changing business environment

- Level of board satisfaction with enterprise responsiveness to new requirements

- Number of critical products, and services supported by up-to-date business processes

- Average time to turn strategic enterprise objectives into an agreed and approved initiative

EG09 Information-based strategic decision-making

- Degree of board and executive management satisfaction with decision-making

- Number of incidents caused by incorrect business decisions based on inaccurate information

- Time to provide supporting information to enable effective business decisions


47

EG10 Optimization of service delivery costs

- Frequency of service delivery cost optimization assessments

- Trend of cost assessment vs. service level results

- Satisfaction levels of board and executive management with service delivery costs internal

Internal EG11 Optimization of business process functionality

- Frequency of business process capability maturity assessments


- Satisfaction levels of board, and executives with business process capabilities

EG12 Optimization of business process costs

- Frequency of business process cost optimization assessments

- Trend of cost assessment vs. service level results

- Satisfaction levels of board and executive management with business processing costs

EG13 Managed business change programs

- Number of programs on time and within budget

- Percent of stakeholders satisfied with program delivery

- Level of awareness of business change induced by IT-enabled business initiatives

EG14 Operational and staff productivity

- Number of programs/projects on time and within budget

- Cost and staffing levels compared to benchmarks

EG15 Compliance with internal policies



- Percent of policies supported by effective standards, and working practices

- Learning and growth

Learning and growth

EG16 Skilled and motivated people

- Level of stakeholder satisfaction with staff expertise and skills

- Percent of staff whose skills are insufficient for the competency required for their role

- Percent of satisfied staff


48

EG17 Product and business innovation culture

- Level of awareness and understanding of business innovation opportunities

- Stakeholder satisfaction with products, innovation expertise, and ideas

- Number of approved product and service initiatives resulting from innovative ideas


49

Appendix C: Full COBIT IT Goals and Sample Metrics

BSC

Dimensions


Financial ITG01 Alignment of IT and business strategy

- Percent of enterprise strategic goals, and requirements supported by IT strategic goals

- Stakeholder satisfaction with scope of the planned portfolio of programs, and services

- Percent of IT value drivers mapped to business value drivers

ITG02 IT compliance, and support for business compliance with external laws and regulations

- Cost of IT non-compliance, including settlements and fines

- Number of IT-related non-compliance issues reported to the board or causing public comment or embarrassment

- Number of non-compliance issues relating to contractual agreements with IT service providers

ITG03 Commitment of executive management for making IT-related decisions

- Percent of executive management roles with clearly defined accountabilities for IT decisions

- Number of times IT is on the board agenda in a proactive manner

- Frequency of IT strategy (executive) committee meetings

- Rate of execution of executive IT-related decisions

ITG04 Managed IT-related business risks

- Percent of critical business processes, IT services, and IT-enabled business programs covered by risk assessment

- Number of significant IT-related incidents that were not identified in risk assessment

- Percent of enterprise risk assessments including IT-related risks



50

BSC

Dimensions


ITG05 Realized benefits from IT-enabled investments, and services portfolio

- Percent of it-enabled investments where benefit realization monitored through full economic life cycle

- Percent of IT services where expected benefits realized

- Percent of IT-enabled investments where claimed benefits met or exceeded

Customer ITG06 Transparency of IT costs, benefits, and risk

- Percent of investment business cases with clearly defined, and approved expected IT-related costs, and benefits

- Percent of IT services with clearly defined, and approved operational costs, and expected benefits

- Satisfaction survey of key stakeholders regarding the transparency, understanding, and accuracy of IT financial information

ITG07 Delivery of IT services in line with business requirements

- Number of business disruptions due to it service incidents

- Percent of business stakeholders satisfied that it service delivery meets agreed-upon service levels

- Percent of users satisfied with quality of IT service delivery

- Adequate usage of applications, information, and technology solutions

ITG08 Percent of business process owners satisfied with supporting IT products, and services

- Level of business user understanding of how technology solutions support their processes

- Satisfaction level of business users with training, and user manuals


51

BSC

Dimensions


ITG09 IT agility - Level of satisfaction of business executives with IT’s responsiveness to new requirements

- Number of critical business processes supported by up-to-date infrastructure, and applications

- Average time to turn strategic IT objectives into an agreed and approved initiative

ITG10 Security of information, and processing infrastructure, and applications

- Number of security incidents causing business disruption or public embarrassment

- Number of IT services with outstanding security requirements

- Time to grant, change, and remove access privileges, compared to agreed service levels

- Frequency of security assessment against latest standards and guidelines

Internal ITG11 Optimization of IT assets, resources, and capabilities

- Frequency of capability maturity and cost optimization assessments


- Satisfaction levels of business and IT executives with IT-related costs and capabilities

ITG12 Enablement and support of business processes by integrating applications and technology into business processes

- Number of business processing incidents caused by technology integration errors

- Number of business process changes that need to be delayed or reworked because of technology integration issues

- Number of IT-enabled business programs delayed or incurring additional cost due to technology integration issues

- Number of applications or critical infrastructures operating in silos, and not integrated


52

BSC

Dimensions


ITG13 Delivery of programs on time, on budget, and meeting requirements, and quality standards

- Number of programs/projects on time, and within budget

- Percent of stakeholders satisfied with program/project quality

- Number of programs needing significant rework due to quality defects

- Cost of application maintenance vs. overall IT cost

ITG14 Availability of reliable and useful information

- Level of business user satisfaction with quality of management information

- Number of business process incidents caused by non-availability of information

- Ratio and extent of erroneous business decisions where erroneous or unavailable information was key factor

ITG15 IT compliance with internal policies



- Percent of policies supported by effective standards and working practices

- Frequency of policy review and update

Learning and growth

ITG16 Competent and motivated IT personnel

- Percent of staff whose IT-related skills are sufficient for the competency required for their role

- Percent of staff satisfied with their IT-related roles

- Number of learning/training hours per staff

ITG17 Knowledge, expertise, and initiatives for business innovation

- Level of business executive awareness, and understanding of IT innovation possibilities

- Stakeholder satisfaction with levels of IT innovation expertise, and ideas

- Number of approved initiatives resulting from innovative IT ideas

White Paper on Key Performance Indicators (KPI) for … · which initial Key Performance Indicators...

Documents

Transcript of White Paper on Key Performance Indicators (KPI) for … · which initial Key Performance Indicators...