NCHRP 3-94: Systems Operations and Management Guide Steve Lockwood, Phil Tarnoff, John Conrad and...
-
Upload
myles-gilbert -
Category
Documents
-
view
216 -
download
0
Transcript of NCHRP 3-94: Systems Operations and Management Guide Steve Lockwood, Phil Tarnoff, John Conrad and...
NCHRP 3-94: Systems Operations and Management Guide
Steve Lockwood, Phil Tarnoff, John Conrad and Rich Margiotta
Presentation to AASHTO SSOM
Manchester, NH
June 16, 2009
Agenda
Introduction and Approach
The Black Box – How it works
Example of Pilot Application
Web Product Concept
Sample Strategies for Review
Introduction
Objective: Mainstreaming SO&M as formal SDOT program
Guidance for SDOT managers
Develop existing material into an accessible and user-friendly product
Web-based approach
Benefits of Web-Based Approach
Avoids lengthy paper documents with big charts
Relationships among elements built in
Custom tailored to user
Users self-evaluate SDOT state of play
Increasing levels of detail displayed on automated basis
Hyperlinks to supporting documents
Approach
Covers interrelated elements critical to improving SO&M performance
Process and scope
Institutional elements (including organization)
Operations Capability Maturity Levels:Process Dimension Levels and Supporting Institutional Dimension Levels
Ad Hoc
Managed
Integrated
Level 1
Level 2
Level 3
Ad hoc operation. Relationships not coordinated
Processes fully documented & staff trained
Fully coordinated, performance-driven
TransitioningAgencies (most)
Goal for the future
A few Leaders
SupportArch.
SupportArch.
SupportArch.
The Process Dimension
1. Business processes
Scoping/Planning
Program development
Programming and Budgeting
Procurement
Operations
2. Systems and Technology
Regional architectures
Project systems engineering (design) process
Standards/interoperability
Testing and validation
Maintenance
3. Performance
Measures definition
Data acquisition management
Measures utilization
The Institutional Dimension
1. Mission Mission commitment Operations Culture SDOT authorities (laws) Continuous improvement
acceptance (culture)
2. Leadership In-reach Outreach (policy-makers,
stakeholders)
3. Organization Status/authority (equivalence) Unit relationships (consolidation) Authority/responsibility allocation
4. Staffing Core capacities definition/filled Technical capacities (staff
development) Career path (incentives), Recruitment and retention
5. Resources Funding sources Budgeting/resource allocation
6. Partnerships Interagency (cooperation, co-
training) Intergovernmental (cooperation,
coordination) Participation in MPO activities PPP (rationalized outsourcing)
Agenda
Introduction and Approach
The Black Box – How it works
Example of Pilot Application
Web Product Concept
Sample Strategies for Review
Framework: The Capability Maturity Model
Performance-critical elements identified (process, institutional)
Self-evaluation based by user type (administrator, project manager, program manager)
Presumes continuous improvement in critical elements
Strategies provided for levels of user to get to next level
(different strategies for different user types and their levels)
Each strategy has guidance including , examples, reference
Criteria for levels, strategies, guidance are in an invisible data base
Agenda
Introduction and Approach
The Black Box – How it works
Example of Pilot Application
Web Product Concept
Sample Strategies for Review
A recent "manual" application of the guide logic
State of Old Hampshire
Medium size, some SO&M programs
Some Policy-based strategy applications
Start-up Performance measurement context
Has an aggressive SO&M vision
Example below:
Program Assessment: Strengths and Weaknesses
Overall assessment in terms of maturity levels (for selected critical process elements)
Scope
Staff Management
Customer Service
Business Processes
Technology and Systems
Performance Measurement
Ranking and rationale
Action Plan -- starting with lowest level
Strategies to move up next level
Investment implications
LEVELS OF OPERATIONAL MATURITY
Level Name Characteristics
1 Incomplete Ad hoc processes
2 Performed Procedures defined and tracked
3 Managed Process is managed and measured
4 Established Continuous Analysis
CRITERIA FOR ASSESSING OPERATIONAL MATURITY
Basic Dimension
Level 1 – Incomplete
Level 2 – Performed Level 3 – Managed Level 4 - Established
Scope Narrow, opportunistic Needs-based, standardized
Addresses full range of highway M&O
Includes multi-modal & freight
Staff Management
On-the-job training Staff supervision, active training program, standardized procedures
Performance-based management program
Opportunities for advancement, risk taking for new procedures encouraged
Customer Service
Ad hoc information dissemination
Rudimentary dissemination of roadway condition information using website
Call in phone number, 511 and travel time on signs
Call center operation, interaction with media, frequent publication of performance
Business Processes
Informal, undocumented
Planned, mainstreamed Integrated, documented
Coordinated with external entities
Technology and Systems
Project oriented, no use of systems engineering
Generalized platforms based on systems engineering analysis
Extensive use of standards, provision for continuous validation
Coordinated with external systems and requirements
Performance Measurement
Outputs reported Outcomes reportedEmphasis on dissemination to the public
Outcomes used Performance accountability
SUMMARY OF THE EVALUATION OF DOT'S OPERATIONAL MATURITY
Dimension Rank Comments
Scope 2 Missing arterial flow monitoring and incident management
Staff Management
2 Absence of performance based management process
Customer Service
Low 2 Website data incomplete and/or inaccurate. Absence of well-defined call-in process
Business Processes
2 Expanded guidelines for VMS messages required. ConOps incomplete
Technology and Systems
4 Careful adherence to standards and systems engineering processes
Performance Measurement
Low 2 Absence of useful outcome measuresFailure to use results to influence the program
Overall Performance
Low 2 Based on the guideline that the agency’s maturity level is equal to the dimension at the lowest level
Agenda
Introduction and Approach
The Black Box – How it works
Example of Pilot Application
Web Product Concept
Sample Strategies for Review
Indicate User’s position (check one):
Top management (CO or District)
Program manager (CO or District)
Project manager (CO or district)
Step 1 – User Role Self-identification
Evaluation goes element-by-element
PROCESSDIMENSION
CRITERIA DEFINING LEVEL OF MATURITY
L-1 L-2 L-3 L-4
1. BUSINESS PROCESSES ELEMENT
Scoping/Planning
Program development
Programming and Budgeting
Procurement
Operations
2. SYSTEMS AND TECHNOLOGY ELEMENT
Regional architectures
Need for architectural standardization among state and regional systems are ignored
Regional architecture developed and subjected to periodic reviews
System implementations recognize need for future upgrades as systems installed by others offer increased functionality and expanded data interchange requirements
Regional architecture recognized as a dynamic tool requiring continuing review and updates as regional systems mature and new technologies and applications become available.
Systems engineering
Standards/interoperability
Testing and validation
3. PERFORMANCE ELEMENT
Measures definition
Data acquisition management
Measures utilization
Ex: Systems and Technology – Regional Architecture
Need for architectural standardization among state and regional systems are ignored
Regional architecture developed and subjected to periodic reviews
System implementations recognize need for future upgrades for increased functionality
Regional architecture recognized as a dynamic tool requiring continuing review and updates
Step 2: User self-evaluation of current level of capability in all key elements
Note: User will repeat step for other 7 elements later
Answer similar questions regarding:
1. PROCESS DIMENSION
1. Business processes (including scope)
2. Systems and Technology
3. Performance
2. INSTITUTIONAL DIMENSION
4. Mission
5. Leadership
6. Organization / Staffing
7. Resources
8. Partnerships
Step 3: Display of user’s level for each element at lowest level
REGIONAL ARCHITECTURES -- CRITERIA DETERMINE EXISTING AGENCY LEVEL
L-1 L-2 L-3 L-4
Need for architectural standardization among state and regional systems are ignored
Regional architecture developed and subjected to periodic reviews
System implementations recognize need for future upgrades as systems installed by others offer increased functionality and expanded data interchange requirements
Regional architecture recognized as a dynamic tool requiring continuing review and updates as regional systems mature and new technologies and applications become available.
Step 4 System Determines Appropriate Strategy
GENERAL STRATEGY TO ADVANCE LEVELS OF MATURITY – TOP MANAGMENT
L-1 L-2 L-2 L-3 L-3 L-4
Ensure that agency is an active participant in the development and maintenance of the regional architecture.
Monitor ongoing system developments as well as changing needs to ensure that the architecture is both followed and updated as needed
Coordinate architectural activities with performance measurement to assess architectural effectiveness as compared with vision developed for relevant concepts of operations.Identify areas needing improvement and develop recommended architectural enhancements.
Step 5 – Guidance for Appropriate level transition displayed
TECHNICAL PROCESSES -- TOP MANAGMENT (A) Systems Architecture
Strategy from Level 1 to Level 2 A. Identify an individual (designated here as the Architecture Lead) with a knowledge of appropriate traffic management and traveler information functions. Allocate a minimum of ½ time for the individual to lead the process. Provide individual with opportunity to attend training in the development of ITS architecture. Responsibility: Implemented by senior management. Responsible individual is at the project level of responsibility. Relationships: N/A References: N/A B. Architecture Lead is given responsibility for assembling and convening an architectural review committee of individuals representing stakeholder organizations related to the regional architecture. Stakeholders include organizations representing all of the surface transportation modes within the region. Architecture review committee is first acquainted with the principles of the regional architecture and then introduced to the draft architecture being considered for the region. Responsibility: The creation and operation of the architecture review committee is the responsibility of the Architecture Lead. C. Monitor system developments and other agency activities to ensure that architectural relevancy is maintained. This is performed through periodic stakeholder meetings, during which the architecture is reviewed in terms of the technological developments and needs of the participating agencies. . Responsibility: Periodic reviews are the responsibility of the architecture
(Note: Other layers are stored by level and position)
TECHNICAL PROCESSES -- TOP MANAGMENT(A) Systems Architecture
Strategy from Level 2 to Level 3
A. Upper management requires periodic reports (semi-annual is recommended) from the Architecture lead that define the manner in which the architecture is being used to influence the design and implementation of regional systems. The reports should also include a description of the manner in which the architecture is being modified to reflect changes in regional systems and requirements.
Responsibility: Upper management must be involved in the review and critique of the Architecture lead reports.
B. Agency commitment to the concepts of the regional architecture is demonstrated through mid-level management support of the efforts of the Architecture Lead. This requires periodic (at least every second meeting) attendance by the immediate supervisor(s) of the Architecture lead as well as the supervisor(s) of other agency personnel participating in the development of the regional architecture. In this way, the agency is assured that its long term requirements and operational philosophies are reflected in the ongoing evolution of the architecture.
Responsibility: Supervisory personnel affiliated with the staffing of the architectural review committee
(Note: Other layers are stored by level and position)
TECHNICAL PROCESSES -- TOP MANAGMENT(A) Systems Architecture
Strategy from Level 3 to Level 4
It is important to continually assess the degree to which the regional architecture meets the requirements of the region’s surface transportation agencies. This can be done through the use of performance measures that 1 – measure the architecture’s impact on the development of regional systems and 2 – measure the satisfaction of the stakeholders with its use. The first step in the performance measurement process is to identify appropriate measures. The measures should be selected by the architecture review committee. Examples include stakeholder satisfaction (a qualitative measure), the reliability of systems developed using the architecture, and the degree to which information identified by the architecture is shared among participating agencies.
Responsibility: The identification of performance measures is performed by the architecture review committee. The implementation of the performance measures is performed by the Architecture Lead working with the appropriate group within the agency responsible for performance measurement.
B. Level 4 performance is based on the continuous assessment of the effectiveness of agency policies. For this reason, the architecture itself must be continuously assessed to determine whether it is meeting the stakeholder needs, or whether modifications are required. This process is undertaken through the review of new stakeholder requirements, modifications or additions to stakeholder systems, and/or changes in technology that might have an impact on the architecture. At the periodic stakeholder meetings, time should be allocated to identify changes in any of these areas, and to discuss their potential impact on the architecture.
Responsibility: Responsibility of the architecture review committee as lead by the Architecture lead.
Step 6: Store and continue with other elements
System shows accounting matrix for components checked and next component
User checks “next”
System goes to next most critical element
Step 7 to N: system repeats steps 3-6
Step N + 1: completion high level summary display of change management strategies
“You were ranked as xxxxxxxxx, and we recommended the following”
Print-out functionality
Agenda
Introduction and Approach
The Black Box – How it works
Example of Pilot Application
Web Product Concept
Other Sample Strategies for Review
Performance Measurement : General Strategies
GENERAL STRATEGY TO ADVANCE LEVELS OF MATURITYPerformance Element: (A) Measures Definition
L-1 L-2 L-2 L-3 L-3 L-4
Develop a PM program based on output measures only
Develop a PM program based on output measures and a few outcome measures only
Develop a PM program based on full set of both output/outcome measures, linked to agency operations
Performance: Level 1 to Level 2
PERFORMANCE MEASUREMENT Measures Definition
Strategy from Level 1 to Level 2
A. The philosophy here is, “if you’re going to manage it, you’re going to measure it”. For Level 1 to Level 2, the limited number of operational activities that will be monitored must first be identified. For Level 2 to 3, all operations activities engaged in will have performance measures defined for them. Output measures for both Level transitions should be relatively simple and easy to collect. These are often referred to as “activity based” measures since they monitor the extent of activities undertaken, and their immediate consequences. Therefore, the measures selected will include at a minimum:incident durationincident and work zone characteristics (number, type, severity)operational activity (website hits, service patrol stops, messages)equipment locations and statusequipment downtimeResponsibility: Operations staffRelationships: N/AReferences: Guide to Benchmarking Operations Performance Measures, http://www.trb.org/TRBNet/ProjectDisplay.asp?ProjectID=1218NCHRP Web-Only Document 97, "Guide to Effective Freeway Performance Measurement: Final Report and Guidebook", http://www.trb.org/news/blurb_detail.asp?id=7477
Performance: Level 2 to Level 3
PERFORMANCE MEASUREMENT Measures Definition
Strategy from Level 2 to Level 3
GuidanceA. Development of congestion-based outcome performance measures should begin at Level 3 These
measures can be developed through available data and modeling methods, i.e., they can be “indirect measurements” of congestion. For example, a variety of methods are available for converting v/c ratios to delay-based measures.
The congestion measures should be based on travel time estimation. In addition vehicle-miles of travel should be monitored. To the extent that the data allow, these measures should be developed for different time periods (e.g., peak hour, peak period, offpeak).
Responsibility: Operations staff, but data may be borrowed from other units (e.g., traffic counts, capacity).
Relationships: N/AReferences: Guide to Benchmarking Operations Performance Measures, http://
www.trb.org/TRBNet/ProjectDisplay.asp?ProjectID=1218NCHRP Web-Only Document 97, "Guide to Effective Freeway Performance Measurement: Final
Report and Guidebook", http://www.trb.org/news/blurb_detail.asp?id=7477
Performance: Level 3 to Level 4
PERFORMANCE MEASUREMENT Measures Definition
Strategy from Level 3 to Level 4
A. Develop a formal Operations Strategic Plan. The Operations Strategic Plan should include vision, goals, and objectives for what operations are trying to achieve. Performance measures and targets should be crafted to monitor the progress toward meeting these. For example:
Responsibility: Operations staffRelationships: N/AReferences: see belowB. Additional performance measures should be added at Level 4. These include a variety of travel time-based congestion
measures, including reliability, as well as customer satisfaction. The complete “incident timeline” should be measured. Lane and shoulder blockage times should be tracked, so that lane- and shoulder-hours lost due to incidents and work zones can be developed.
Responsibility: Operations staffRelationships: N/AReferences:NCHRP Web-Only Document 97, "Guide to Effective Freeway Performance Measurement: Final Report and Guidebook",
http://www.trb.org/news/blurb_detail.asp?id=7477Guide to Benchmarking Operations Performance Measures, http://www.trb.org/TRBNet/ProjectDisplay.asp?ProjectID
=1218C. Coordinate performance measures used for operations with those used by safety and planning. In order to establish a
comprehensive and integrated performance measurement program, it is important that practitioners apply performance measures (metrics) uniformly across all types of applications. It is desirable to maintain performance measures that are used for specific applications, but a core set of measures should be used across all applications. This is particularly useful for congestion/mobility metrics, and it is not all that difficult to implement.
Responsibility: Operations staffRelationships: Safety and planning personnel nee to be involvedReferences:
Performance: Level 3 to Level 4 (continued)
PROGRAMMINGAND BUDGETING
USERS CURRENT LEVEL
Level 1 Level 2 Level 3 Level 4
Individual project capital costs. Outside central office formal budgeting process
Defined ”program” wish list of projects in some districts, integrated into district capital budgets
Statewide staged SO&M multi-year program of projects(Capital, operating and maintenance requirements)
SO&M integrated into standard formal capital budgeting process/cycles
Step 3: Display of user’s levels and indication of lowest level
PROGRAMMING AND BUDGETING
LEVEL TRANSITION STRATEGY
L 1 L l 2- L 2 L 3- L 3 L 4-
A.Secure agreement from top management on the need and utilization of an SO&M budgetB.Organize an interactive central office/district task force to develop program as basis for budgetC.Develop program of “next step” projects by priority and aggregate to total program estimate, statewide and by district – using such plan material as may be available
A.Organize an interactive central office/district task force to develop program budgetB.Utilize output from Statewide SO&M plan to develop staged SO&M program and related budge (capital, operating maintenance – including staffing, trainingC.Determine administrative or legal adjustments necessary in state budgeting for DOTs regarding eligible use of funds and appropriate budget categories for SO&M program components
A.Request/receive mandate from top management to integrate SO&M in the formal SDOT budgeting processB.Organize an agency-wide task force to determine a process for investment trade-offs among capacity vs. operations vs, maintenance investments – as part of regular budgeting cycleC.Integrate SO&M program into standard Formal SW programming and budgeting processD.Work with stakeholders to make necessary adjustments in state law an administrative procedures for use of state funds for SO&M
Step 4: Display of Strategies to get to next level for lowest level (“critical” component)
Step 5: Detailed Guidance for moving selected component to next level
PROGRAMMING/BUDGETING GUIDANCE FOR MOVING FROM LEVEL 2TO LEVEL 3
Function Develop sustainable budget for continuously improving SO&M applications
General Strategy Develop budget estimate for statewide program next steps
Guidance A. Organize an interactive central office/district task force to develop program budgetDevelop central office/district task force to develop budget –including SO&M and budgeting unit staffs
B. Develop Budgeting processReview standard budgeting process for other core programsEstablish appropriate analogue process steps
C. Utilize output from Statewide or district SO&M plan to develop staged SO&M program and related budge (capital, operating maintenance – including staffing, training
Analyze updated program into budget in terms of stagesEstablish cost components, including capital, operating and maintenance
C. Determine administrative or legal adjustments necessary in state budgeting for DOTs regarding eligible use of funds and appropriate budget categories for SO&M program components
Review proposed projects in terms of funding source eligibility restrictions – including capital, operating, staffing and maintenance componentsMeet with, develop support among, key legislative stakeholders as necessary -- depending on administrative discretionAllocate costs as appropriate
Responsibility: top management working through combined central office/district task force including budgeting staffRelationships: Mandate from top management to SO&M and planning staffReferences: Ex: NDSHA CHART, WSDOT
MDSHA CHART "07 Budget
Capital
Field and IM Equipment $3,855,000
Network Engineering $2,687,000
Leased Circuit Costs $1,000,000
CHART System and Network Connectivity $4,162,000
Plan., Develpmt, Engrg, & Coord. $3,000,000
CHART System Integration $3,100,000
Overhead $1,157,000
Total Capital $18,961,000
Staff and Operating Expenses
Operations salaries, overhead, overtime, and expenses (staff 64) $6,324,742
Systems Maintenance: emergency, preventive, and routine maintenance $1,200,000
Administrative: supplies and contractual salaries and other expenses $260,000
Miscellaneous Operating Expenses :travel expenses and meals $40,000
Total Operating $7,816,276
Grand Total $26,777,000