IT Infrastructure Partnership Program (ITP) 0 IT Infrastructure Partnership Program Fred Duball and...

1

IT Infrastructure Partnership Program (ITP)

IT Infrastructure Partnership Program

Fred Duball and Joe Fay

Program Directors

July 18, 2007ITIB IT Infrastructure Committee

2


Agenda

• Service Delivery

– Service Delivery Metrics

– Customer Satisfaction

– Request for Service

– P2P Procurement

• Transformation

• Partnership Financials

• Independent Program Maturity Assessment

3


Service Delivery

4


Jun

• Significant Incidents:

– Controller switch failure disrupted communication between mainframe and tape drives- mainframe services down all day 6/20, primary impact DMV, DSS, VEC + 12 other agencies – Interim fix 6/20 Final fix 6/24

– User executable caused the Unisys mainframe multiple brief outages on 6/22, 7/3 and 7/4. Primary Impact DSS. Final patch applied 7/15.

– DCR call center bandwidth not adequate for peak period VOIP Calls –customer calls garbled for 1.5 weeks. Resolved 6/8 with new DS3 circuit.

– VDOT Bristol Email and BlackBerry E-mail Failure – e-mail down 610 users for 5 days, BB e-mail down 16 users for 22 days. Resolved 6/12

– Symantec uninstall at Library of Virginia did not work– 40 PCs down 1 day, 7 PCs down 1.5 days. Resolved 5/15

– Combination of server and network configuration issues results in DFP slow network – response for primary applications slow for 3 months. Planned remediation complete 9/14

TargetGreen -

Meet or exceed target within 10%

Blue -Exceed target by > 10% or reach maximum of 100%

Yellow -Fail to reach target

within 10%

Red -Fail to reach target

by > 10%

Legend

Central Metrics

Service Delivery Dashboard

Field Metrics

7.1%

0%

64.3%

28.6%

Apr

0%

0%

57.1%

42.9%

May

0%

7.1%

42.8%

50.0%

Apr May Jun

R

GY

B

0.0%

0.0%

68.8%

31.25%

0.0%

6.25%

62.5%

31.25%

R

GY

B

3.9%

1.7%

88.2%

6.2%

5


Domain MeasureMOU-SLO

A S O N D J F M A M J

End User

Average Speed to Answer

<30 sec 33 22 14 22 18 26 28 47 47 28 18

Call Abandon Rate < 5% 6.1% 2.1% 0.8% 2.0% 1.5% 2.2% 2.1% 3.1% 3.3% 3.7% 2.5%

Email Response <60 mins

15 16 18 16 16 18 18 25 22 28 46

Voicemail Response <30 mins

15 16 16 16 16 18 18 25 22 28 28

First Call Resolution >70% 20% 21% 61% 67% 73% 74% 77% 79% 83% 86% 83%

VITA Messaging System Availability

>99.0% 99.9% 100% 99.9% 100% 99.9% 100% 99.7% 99.8% 100% 100% 99.9%

Shared Messaging System Availability

>99.0% 100% 99.9% 100% 100% 99.9% 100% 100% 99.4% 100% 100% 99.5%

Data Center

IBM Mainframe Availability

>99.9% 99.9% 100% 100% 100% 100% 100% 100% 100% 99.9% 99.9% 98.6%

Unisys Mainframe Availability

>99.9% 100% 99.9% 100% 100% 100% 100% 100% 100% 100% 100% 99.9%

UNIX Server Availability

>99% 99.8% 99.9% 99.8% 99.9% 99.9% 99.9% 99.8% 99.9% 99.9% 99.9% 99.9%

Windows Server Availability

>99% 99.9% 99.3% 100% 99.9% 99.4% 99.9% 99.9% 99.8% 99.9% 99.9% 99.9%

Network Circuits Availability 99.2% 99.5% 99.4% 99.8% 99.7% 99.7% 99.6% 99.7% 99.4 99.6% 99.6% N/A

Security

ACF2 Logon Requests

95% 100% 99% 100% 100% 100% 100% 100% 100% 100% 100%

Security Incident Reporting

95% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100%

FCR – Start Oct measure is (Resolved on First Call/Can be Resolved on First Call. Previous measure was (Resolved on First Call/Total Calls). May NG/VITA agreed to First Call Resolvable categories

TargetGreen -




within 10%


by > 10%

Legend

Central Operation Metrics

6


Speed to Answer

0

5

10

15

20

1/27

2/10

2/24

3/10

3/24

4/7

4/21

5/5

5/19

6/2

6/16

6/30

AgentsStaffing

Slots

Actions to address problem:

• Apr/May – incentives to agents to meet goals

• May/Jun –Added interim support through overtime and temp assignment of up to 3 field techs

• May/Jun/Jul – For longer term fix, increased agent staffing from 11 (Apr) to 21 (Jul)

• 11 Jun – Jan Weiner, Lafayette QA Manager, is Interim VCCC Manager to standardize processes/procedures and optimize Peregrine utilization.

0

200

400

600

800

1000

1200

1400

1600

1/27

2/10

2/24

3/10

3/24

4/7

4/21

5/5

5/19

6/2

6/16

6/30

Call VolumeCalls

0

10

20

30

40

50

60

70

80

1/27

2/10

2/24

3/10

3/24

4/7

4/21

5/5

5/19

6/2

6/16

6/30

Seconds

Week Average

Month AverageGoal

Actual

Plan

Speed to Answer

7


SLO#

AgenciesMOU S O N D J F M A M J

Average Speed to Answer 3 30 Sec 16 16 20 20 22 21 16 22 22

Call Abandonment Rate 3 5.0% 7.0% 8.4% 9.4% 5.3% 4.3% 6.4

First Call Resolution 12 70.0% 76.0% 79.3% 84.6% 84.7% 86.9% 87.7% 84.7% 88.0% 92.8% 91.9

End User

Average Time On-hold 2 90 Sec 5 4.5 5.5 5 3 5 6.5 3 3 3

Help Desk Password Resets

24 90.0% 87.6% 94.8% 93.9% 96.8% 92.8% 98.1% 94.6% 96.8% 97.5% 97.4

Service via Incident Ticket 22 78.0% 81.8% 82.7% 79.2% 82.6% 77.8% 79.1% 82.6% 85.9% 90.0% 85.4

Service via Service Request

25 80.0% 86.6% 86.9% 89.7% 91.4% 90.7% 92.5% 89.6% 94.2% 95.3% 96

Incident Repair 23 80.0% 84.3% 83.4% 82.6% 88.1% 86.4% 89.4% 88.1% 91.0% 92.0% 92.1

Messaging Service 27 99.0% 99.6% 100% 99.9% 100% 99.9% 100% 100% 99.9% 99.9% 99.9

Windows Mission Critical Servers

41 99.0% 99.1% 100% 100% 100% 100% 99.9% 100% 99.9% 99.8% 99.9

RISC/Unix Mission Critical Servers

14 99.0% 100% 99.8% 100% 99.9% 100% 100% 100% 99.9% 99.8% 100

Data Center

Windows Other Server 34 90.0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100

RISC/Unix Other Servers 11 90.0% 100% 100% 100% 100% 100% 99.9% 100% 100% 100% 100

QA/Test Systems and Servers

11 90.0% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100

Development Servers 17 90.0% 100% 99.9% 100% 100% 100% 100% 99.5% 100% 100% 100

Network Internet Access 16 99.8% 99.5% 100.0% 100% 100% 99.9% 100% 99.9% 99.9% 99.9% 99.9

Call Abandonment Rate recalculated in Apr to account for calls received only during Help Desk operating hours and not count abandons during first 30 seconds. Dec-Mar identifies previous calculation above, recalculated below.

11.2%

4.0% 6.6% 5.3%

12.7% 11.0% 12.7%

5.5%

TargetGreen -




within 10%


by > 10%

Legend

Field Metrics

8


Partnership continues to reinforce importance of customer satisfaction and customer service mind-set

• Service delivery - Measuring and improving– Reduced customer wait time - average speed-to-answer for VCCC at 18 seconds for June

– Reduced down time - Verizon mean-time-to-repair on circuit outages under four hour SLA/4th month running

– 93.2% on-time installation - processed 929 telecommunication service requests in May

• Request for Service (RFS) and Procure to Pay (P2P) improvements reduced process time

– Combined tasks in parallel

– Consolidating desktop orders through Product Support Center (PSC)

– Drill down reporting of tasks and steps

• Actively engaging customers

9


Question 2: Timeliness74.9%

16.0%

2.3%

2.8%

2.1%

1.9%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Question 4: Staff Knowledge76.6%

18.0%

2.4%

1.7%

0.7%

0.7%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Question 3: Staff Professionalism78.9%

17.8%

1.2%

0.9%

0.0%

1.2%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Question 5: Overall Satisfaction76.1%

17.1%

2.1%

2.1%

0.9%

1.6%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Question 1: Effectiveness76.0%

17.7%

1.9%

1.9%

0.9%

1.6%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Q1 Avg. = 5.61

Customer satisfaction is high for desktop support

Q2 Avg. = 5.53

Q5 Avg. = 5.60

Q3 Avg. = 5.71

Q4 Avg. = 5.66

• Period of Analysis: 04/07/07 – 06/29/07

• Number of Responses: 435

• Number of Resolved Tickets: 3807

• Response Rate: 11.4%

10


Question 4: Staff Knowledge60.1%

25.7%

7.9%

2.2%

1.7%

2.4%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Question 3: Staff Professionalism69.5%

24.1%

3.4%

1.0%

0.4%

1.6%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Question 5: Overall Satisfaction63.4%

23.1%

5.6%

2.0%

2.2%

3.7%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Question 2: Timeliness61.8%

24.9%

5.1%

2.0%

2.7%

3.4%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Question 1: Effectiveness62.4%

26.1%

4.8%

1.8%

2.3%

2.7%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Overall

Q1 Avg. = 5.36

Customer satisfaction is high for help desk services

Q2 Avg. = 5.31

Q5 Avg. = 5.32

Q3 Avg. = 5.57

Q4 Avg. = 5.33

• Period of Analysis: 02/09/07 – 06/29/07


• Number of Resolved Tickets: 27172


11


Customer satisfaction is high for messaging services

E-mail Easy to Use26.1%

58.7%

8.8%

4.4%

1.8%

0.3%

0% 10% 20% 30% 40% 50% 60%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Q5 Avg. = 5.02(341 Responses)

Availability of E-mail Support/Resources

12.0%

49.2%

21.3%

8.7%

7.5%

1.2%

0% 10% 20% 30% 40% 50% 60%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t


Overall Satisfaction with E-mail16.5%

57.6%

16.2%

5.0%

4.1%

0.6%

0% 10% 20% 30% 40% 50% 60%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t


• Survey Period: 04/30/07 – 05/14/07


• Number of Recipients: 3,567


Webmail Meets Needs21.6%

43.1%

20.3%

5.2%

8.5%

1.3%

0% 10% 20% 30% 40% 50% 60%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t

Security of E-mail

17.3%

44.9%

20.8%

7.6%

6.2%

3.2%

0% 10% 20% 30% 40% 50% 60%

Strongly Agree

Agree

Slightly Agree

Slightly Disagree

Disagree

Strongly DisagreeCu

sto

me

r A

gre

em

en

t


Q14A Avg. = 4.60(153 Responses)

12

IT Infrastructure Partnership Program (ITP)Request for Service (RFS) Process Monthly Totals

Request for Service Monthly Totals

45

6 9 14 917

21

3 25

20

0

10

20

30

40

50

60

70

Jul-Dec 2006 Jan-07 Feb-07 Mar-07 Apr-07 May-07

Time Period

Quantity

Status Totals Total Value

Total RFS Requests 67

Proposals in development 22

Proposals submitted to customer 45 $9.42M

Proposals under review by customer 18 $2.99M

Proposals authorized to proceed 27 $6.43M

RFS projects in progress 11 $1.68M

RFS projects completed 16 $4.75M

• RFS is new “project” work, incremental to the contract, such as:

– Providing infrastructure in a new Commonwealth facility or supporting an agency move

– Standing up infrastructure to support a new information system

13


Improved P2P (Procure to Pay) Process

• Near-term Improvements: Communicate standards to the agency community

Streamline approvals; parallel vs. sequential

Move Northrop Grumman Procurement workflow to work parallel with eVA workflow

Use Help Desk to initiate and verify installation – June 15

- Utilize Peregrine to track PRs/POs from submission to installation and generate automated reports

• Long-term Improvement Strategy:– Develop and deploy service catalog

– Consolidate contracts - 4Q07

*Assumption is that no approvals are escalated

14


P2P end-to-end processing time reduced

5.6

Note: The end-to-end processing time includes the time from submission of the eVA request until the proof of delivery is received from the vendor.

65.771.5

77.2

51.9

36.3

15.8

0

10

20

30

40

50

60

70

80

Average Number of Days

January February March April May June

Month

TOTAL NUMBER OF LINE ITEMS

Jan 531

Feb 967

Mar 623

Apr 332

May 603

June 1006

Line items = quantity of products (1 to n) on a single line of PO

15


End-to-end processing for standard products

Total number of items procured

Product May June

PCs 591 439

Monitors 430 206

Printers 84 82


31.9

13.0 14.816.7

0

5

10

15

20

25

30

35

Average Days

PCs &Monitors Std Printers

Product

May June

16


End-to-end processing for non-standard products

Total number of items procured

Product May JuneServers 61 42

Network 193 1,275

Software --- 3,837

Maintenance 902 506

Misc. 3,060 3,533

17.115.4

23.0

14.0 14.1

7.4

15.4

39.7

15.5

0

5

10

15

20

25

30

35

40

Average Days

Products

May June


17


Service Catalog• When

– Initial production phase start: November 15, 2007

• Benefit– Ability for end user to select “pre-packaged” bundles for quick order

– Allows standardization across the Commonwealth

– Easy approval process

– For common systems “extra” and “additional” features easily identified and selected

• Impact– End user training for a new order process

– Agency mapping for approval process

– Custom orders still may require current P2P process

– Possible changes to firewall to allow access to Web-based tool

18

IT Infrastructure Partnership Program (ITP)Service Delivery ImprovementsService Delivery Agency Focused

Improvements

‾ Library of Virginia now using Enterprise E-mail

‾ Migrated Tax i-File to new server hardware/software – hosted in RPB – can accommodate more concurrent users when filing taxes.

‾ Migrated VDOT to new e-mail software – more reliable and many new features.

Transformation Deployed to Operations – Desktops (5000+ deployed)

• Standard platform with anti virus and firewall security

• Reduces support calls required for older equipment

– Global Address List

• Single e-mail address list for the Commonwealth of VA reducing support calls

– Peregrine live at 7 sites (14 waiting agency signoff)

• Centralized ticket reporting to support teams

• Easier metrics consolidation and monitoring

– Server consolidation

• Standardizing hardware platforms and added redundancy to the architecture allows for fewer problem outages.

– Disk Consolidation – storage

• Removes older hardware from the architecture.

• Rapid deployment of new storage requirements

• Data replication to improve recovery timeframes

– Piloted ITIL

• Integrated change management to increase successful change and coordination for infrastructure teams

– Network MPLS Points of presence

• Provides redundancy for main core infrastructure.

• Less down time and allows for maintenance to be performed while keeping network available

19


Enterprise Exchange/Outlook EmailEnterprise Collaboration Tools

Active Directory, DNS$25M Investment

Messaging ServicesDesktop

Mass Desktop Refresh ProjectsNetwork Printer Consolidation and RefreshEnterprise Desktop Management Systems

$35M Investment

Enterprise Help Desk in Russell and ChesterfieldField Based Agents and Technicians for Level 3

Enterprise Help Desk System (Peregrine)$10M Investment

Help Desk

New IBM and Unisys Mainframes Consolidation and Refresh of Servers

Migration of servers to the Data Center$50M Investment

Mainframeand Servers

Enterprise Security Operations CenterComputer Security Incident Response Center

Secure Internet Gateway$10M Investment

Security

New Data Center/Office Building in ChesterfieldNew Disaster Recovery Center and Help Desk

in Russell County$60M Investment

Facilities Tier 3 and Tier 2

New Commonwealth-wide MPLS Core WANLAN upgrades to Local Switches/Routers as Needed

Network Re-addressing of IP, DHCP$60M Investment

Network

Voice over IP Network Optimized for Voice and Video Traffic

$20M Investment

Voice / Video

People – Process – ToolsReliable, High Performance,

Enterprise-Wide IT Infrastructure

$270 Million Investment

20


2006 2007 2008 2009+SAJ DNO FJ M SAJM DNOJA

Service Commencement Date 7/1/06General

Transformation Phase (36 Months to June 2009)Domains

ESOC Complete SCD+23 (6/1/08)

Security Interim SecurityIncident trackingand Mgmt SystemSCD+3 (10/1/06)

Enterprise Vulnerability Assessment Program Operational SCD+20 (3/1/08)

Enterprise Vulnerability Assessment Program Operational SCD+20 (3/1/08)

CSIRC CompleteSCD+20 (3/1/08)CSIRC CompleteSCD+20 (3/1/08)

MPLS Core Complete SCD+14 (9/1/07)

Complete Agency LAN migration (90%)SCD+30 (1/1/09)

Enterprise NOCSCD+16 (11/1/07)

DataNetwork

Temp. NOC(11/1/06)

VoiceVoice VoIP Architecture DesignAnd Recommendations

SCD+9 (4/1/07)

VoIP Architecture DesignAnd Recommendations

SCD+9 (4/1/07)

VoIP Completion (90%)SCD+63 10/01/11

VoIP Completion (90%)SCD+63 10/01/11

Procedures Manual (10/1/06)Procedures Manual (10/1/06)

ITIL Process Optimization Complete SCD+23 (6/1/08)ITIL Process Optimization

Complete SCD+23 (6/1/08)

DR Test at SWESC SCD+22 (5/1/08)DR Test at SWESC SCD+22 (5/1/08)

FJ M SAJM DNOJA FJ M A

Messaging

monthsmonths

+24+24 +30+30

DC

SD

CS

NV

SN

VS

SS

SS

RPB MigrationComplete

SCD+19 (2/1/08)

CESC Ready for Occupancy

SCD+12 (7/1/07)

SWESC ReadyFor Occupancy

SCD+16 (11/1/07)

FacilitiesFacilities

Server Consolidation90% Complete

SCD+35 (6/1/09)

Mainframe /Server

Mainframe / server workload migration from RPB to CESC

SCD+18 (1/1/08)

Help Desk Production Incident Mgmt System / SPOC Help Desk (SWESC)

SCD+24 (7/1/08)

Complete Desktop RefreshSCD+32 (3/1/09)

Desktop

Single Statewide Address List SCD+9 (P4/1/07-A3/22/07)

Enterprise messaging 90% complete

SCD+ 35 (6/1/09)

DNS / WINS InfrastructureSCD+13 (9/1/07)

EU

SE

US

Infrastructure Ops Center(Interim) 11/1/06)

Begin Desktop Refresh (P3/01/07-A3/15/07)

Incident Mgmt. Web Accessible (8/1/06)

Knowledge Mgt. System Operational P10/1/06-A4/01/07

Procedures Manual Plan (8/1/06)

Process Cutover (Internal Apps)

Desktop & asset mgmt system operational (P1/1/07-A1/29/07)

Arch Network Blueprint Addressing Plan (2/1/07)

+12+12 +18+18+6+6SCDSCD

= Delivered, awaiting final VITA acceptance

= Delivered = Delayed

21


Internal AppsGeneral Services Cross

Functional

SecuritySecurity Services

Voice

DataNetwork Services

Mainframe and Server

FacilitiesData Center Services

Messaging

Desktop

Help Desk

End User Services

Q4 07Q3 07Q2 07Technology Domain

Service Area

Qtr 2 Desktop Refresh (7/1)


RPB Server consolidation complete(5/15/07)

CSIRC System Detailed Design (5/15/07)

ESOC System detailed Design (6/1/07

CESC Ready for Occupancy (7/1/07)

MPLS Core Complete Begin Agency Migration (9/1/07)

Change Management “go live”

15% LAN MigrationSCD+15 (10/1/07)

Begin Messaging Refresh 10/1)

Standup Messaging Back End (8/30)

Connectivity to CESC (7/1/07)

VITA security Dashboard (09/10/07)

Installation of Telephony and Dedicated Incident Mgt System and Agent workstations installed (12/07)



Commonwealth GAL Update (4/1/07)

SWESC Ready for Occupancy (11/1/07)

ESOC Transitional Complete (11/1/07)

Enterprise NOC (11/1/07)

Connectivity to SWESC (11/1/07)

Move Mainframe/Server workload from RPB to CESC (1/1/08)

Commonwealth Server Consolidation 25% complete

(11/1/07)

RPB Server testing begins (6/06/07)

Phase 1 Change, Config and Release Mgt begins (4/01/07) Phase 2 Incident Mgt Begins

(8/20/07)

1st Mainframe Mock cutover test (10/1/07)

Service Catalog Production Demo (9/30/07)

Service Catalog Production Demo (11/15/07)

22

IT Infrastructure Partnership Program (ITP)Transformation Dashboard

EUS

Desktop

1) Moved from red to yellow May 21 – Will move from yellow to green with achievement of scheduled refresh rate

2) Asset management system delivered and accepted

3) Q1 desktop refresh completed. Target 3484+ systems

4) Q2 desktop refresh on target. Target additional 2500+ systems

Helpdesk

1) Knowledge management system in post live test. Target July 15th for testing complete

2) Agency migration schedule to Peregrine system has been defined and started

3) Project upgrade to green status based on achievement of scheduled rate, planned by August 1

Messaging

1) Change in storage solution to EMC will impact stand-up of backend infrastructure in CESC

2) AITR approval to proceed with pilot migration for Department for the Aging and Virginia Department of Health

3) Project will go from yellow to green when timeline from storage solution change is brought back onto baseline

DCS

Server

1) Phase 1 server consolidation complete at RPB

2) Phase 2 server consolidation begins for greater Richmond area data centers

3) Monitor server move into CESC. Will move from yellow to green after successful execution of first phase moves

Mainframe1) New mainframe deployment at CESC in progress

2) Monitor mainframe test at CESC. Will move from yellow to green after successful execution of first phase of testing

Facilities 1) CESC on schedule. SWESC on schedule

NWS

Data Network

1) Pilot deployments began – June 30

2) Modifying network rollout schedule for ABC and VEC requirements

3) Will move from yellow to green after successful execution of first agency to the MPLS infrastructure

Voice Network 1) Awaiting business decision on VoIP Engineering Change Proposal for CESC

Security 1) Two week delay on Internet security gateway beta agency testing

2) Beta testing for three agencies schedule in progress; will move from yellow to green after successful testing

GEN

Internal Apps 1) Delay in briefing VITA on the “managed service” chargeback offering warrants yellow status

Cross-Functional1) ITP change management training goals achieved2) VITA change management pilot in progress3) Phase 2 process design complete for incident, problem, capacity, availability processes

23


Desktop Refresh• When

– Q3 refresh targeting 4500+ additional systems

• Benefit– New PCs for workers

– Standardized environment

– Altiris system management tool deployed

– Standard antivirus software deployed

– Firewall for desktops deployed

• Impact– Potential of 2-4 hours of end user down time during PC upgrade

– Agency testing of application packages prior to deployment

– Site operations staff assistance during refresh activity

– Equipment storage and agency logistic support may be required

24


Desktop Quarterly Run Rate

Desktop Refresh Plan

0

10000

20000

30000

40000

50000

60000

Milestone Quarters

Cu

mu

lati

ve

To

tal

Un

its

Proposed Plan

Actual Run Rate

Proposed Plan 3484 5984 10484 18484 26884 35284 43284 51284

Actual Run Rate 3484 6056

Q107Q207 2500

Q307 4500

Q407 8000

Q108 8400

Q208 8400

Q308 8000

Q4+ Q1-09 8000

Q207 “Actual Run Rate” as of June 30, 2007

Quantity

25


Desktop Weekly Run Rate

Desktop Refresh Weekly Progress

0

500

1000

1500

2000

2500

3000

Week of

PC

Co

un

ts

Planned

Actual

26


Helpdesk Peregrine Tool Rollout• When

– Q3 rollout targeting 12,942+ additional supported users

– Q3 rollout targeting 25 additional agencies

• Benefit– Standardized environment for problem/change management

– Allows multiple support groups to cover multiple agency/systems from a single management tool

– Allows for centralized metrics reporting / trend analysis

– Centralized SLA management

• Impact– No impact to end users

– Agency participation for retained application support encouraged

– Operations staff availability for training

– Documentation of environment and critical level for systems to be defined

27


Peregrine Rollout(Incident Management Tool)

0

10000

20000

30000

40000

50000

60000

70000

En

d U

sers

Su

pp

ort

ed

0

5

10

15

20

25

Ag

en

cie

s i

n t

hat

Qu

art

er

Forecast Agencies 6 21 15 11 6 4

Actual Agencies 0 9

Actual Desktops 628 4138

Forecast Desktops 628 5143 10649 28711 32334 57736

Q1 CY07

Q2 CY07

Q3 CY07

Q4 CY07

Q1 CY08

Q2 CY08

Helpdesk – Peregrine Tool Coverage

Dept. of Mental Health (pilot site) VITA Dept. of Veterans Services Virginia Museum of Natural History Dept. of Minority Business

Enterprise Dept. of Historic Resources Virginia Museum of Fine Arts

Dept. of Mental Health Office of the Governor Dept. of Employment Dispute

Resolution Dept. of General Services Dept. of Business Assistance Virginia Department of Health

Agencies Complete

Larger Agencies in Progress

28


Messaging Client Rollout• When

– Q3-Q4 Pilot for 5,029 users at Virginia Department of Health and Virginia Department for the Aging

• Benefit– Standardized environment for e-mail systems

– Redundant architecture with complete failover capability

– Disaster recovery planned for SWESC to support e-mail

– Centralized e-mail environment for finding addresses, rooms, etc.

• Impact– End users will get a new mail client installed

– Agency participation for communicating mail policies

– Service delivery staff participation

– Agency documentation of application environment that uses e-mail systems

29


Messaging Quarterly Run RateExchange End User Migration Approach

0

10000

20000

30000

40000

50000

60000

70000

En

d U

sers

Mig

rate

d

Planned 5029 5029 5401 7562 23097 34855 47213 57213

Q3 CY07 Q4 CY07 Q1 CY08 Q2 CY08 Q3 CY08 Q4 CY08 Q1 CY09 Q2 CY09

Q3 Pilots

Q4 No ActivityMessaging follows

Network Refresh

Q108 Starts M

essaging Full R

ollout

30


Network Refresh Rollout• When

– Began Q3 with pilot at Department of Juvenile Justice (DJJ)

– Planned Virginia Department of Transportation and Alcohol Beverage Control headquarters

– Alcohol Beverage Control stores (230+) begin refresh process

• Benefit– Standardized equipment for reliability and management


– Connectivity to new network allows logical extension of agency connection to new data centers

– Higher security, monitoring, speed and allows rapid expansion

• Impact– End users get new IP addresses automatically

– Agency participation for ensuring application readiness


– Agency documentation of application environment that uses hard-coded IP addresses

31


Server Consolidation Rollout• When

– Began Q3 server consolidation site surveys

– Working with VDOT to assist in virtualizations

• Benefit

– Standardized equipment for reliability and management


– Expand the use of tape robotics to improve data access performance

– Upgrade to current disk technology will enable data replication of virtual tape and disk

• Impact

– Agency participation for ensuring application readiness


– Agency documentation of application environment that uses hard-coded IP addresses

32


Server Consolidation Metrics

– Scheduling the following RPB servers for virtualization• 37 Federally-funded servers• 34 Agency released servers• 12 VITA servers

– Working with VDOT to consolidate 70 servers – Planning to consolidate and move 10 Dept. of Environmental Quality servers to RPB – Developing weekly view for server consolidation

Server Consolidated

0

500

1000

1500

2000

2500

3000

Se

rve

r c

ou

nt

Planned Target 0 250 500 750 1000 1250 1500 1750 2000 2250 2500

Examined Forecast 0 250 600 750 1000 1250 1500 1750 2000 2250 2500

Servers Consolidated 0 49 400

Jan-07 Mar-07 Jun-07 Sep-07 Dec-07 Mar-08 Jun-08 Sep-08 Dec-08 Mar-09 Jun-09

33


Data Center Move

34

IT Infrastructure Partnership Program (ITP)Data Center Move

June June

7/1/07 8/1/07 9/1/07 10/1/07 11/1/07 12/1/07 1/1/08 2/1/08 3/1/08 4/1/08 5/1/08 6/1/08

7/15/07Mainframe

Install CESC

6/1/07

MPLS Cnctd CESC

Network/SecurityMainframe/ServerFacility/Move

7/13/07NG Move

into CESC

7/24/07VITA Move into CESC

11/11/07

Enterprise NOC/TSOC Oper CESC

3/11/08Agency Server

Consldtn Comp

11/12/07Mainframe/

Server Prod Cutover

CESC

9/1/07

MPLS Core Active CESC

& SWESC

10/30/07Server Install CESC

12/31/07Mainframes/Servers out

of RPB

1/29/08Servers Install

SWESC

1/4/08Mainframes

Install SWESC

4/17/08DR Test Comp

11/1/07

Help Desk Oper

Cutover11/16/07

ESOC Oper

SWESC

11/1/07SWESC Ready Occup

• Moving the data center from RPB to new, secure CESC facility is complex

– Significant coordination with agencies, minimizing business disruptions, technical complexity

• Robust, detailed data center move plan in place

– Move strategy, test objectives, risk mitigation, communication plan with agencies

35


Data Center Move Strategy• Mainframe Move Strategy

– Replicated mainframe and related hardware required for full application testing

– Coordinate all mainframe users (15 agencies)

– Provide agencies isolated external connectivity to CESC test environment

– Test environment available for 60-90 days

– Multiple mock cutover test prior to final go-live

• Server Move Strategy

– Virtualize/consolidate servers at RPB

– Consolidate multiple SAN/DISK storage systems at RPB

– Replicate RPB internal network (LAN) at CESC

– A two-phased cutover approach will be utilized

• Phase 1 is movement of servers onto an extended duplicate RPB network at CESC

• Phase 2 will require that servers be moved from duplicate RPB network to “New” network at CESC

– Servers will be moved in 7 logical groups, based primarily on agency usage

36


Data Center Move Test Objectives• Operations Team objectives and control

Test

– All systems will IPL/boot and communicate with peripherals

– Administrative functions (monitoring and management) operate as expected

– Data replication between CESC and RPB functions properly

– Internal CESC network (LAN) and firewalls function properly

– Print infrastructure functions properly

– Tape backup infrastructure functions properly

– Control-M infrastructure functions properly for support of batch operations

– Point-to-point connections function properly

• Retained Application support Team objectives

– Applications will initiate and connect with database(s)

– Applications will update data and print reports as expected

– Regression test of all applications components on the mainframe systems

37


Data Center Move Risk MitigationTop 5 risks identified during the server and mainframe move from RPB and CESC.

• Standup of an Isolated Test Environment

– Replicate mainframe hardware and software infrastructure

– Replicate servers running tier 2 applications that interface with mainframes

– Replicate DASD and tape storage infrastructure and data via high speed data links

• Extended Test Period

– Provide agencies with at least 60 days to complete application testing

– Mock move weekends scheduled to accommodate thorough integration testing of complex, interdependent applications

• Test Coordination / Command Center Support

– Provide rapid response team to quickly address problems that surface during testing

– Testing coordination teams will be assigned to each key mainframe using agency

– Weekly reporting of testing progress by agency and application to be shared with agency managers

• Stand-by Hardware

– Mission critical application hardware can be made available if hardware problems arise due to move

• Fallback Contingency

– RPB processing infrastructure will remain intact for at least 2-3 weeks following the move to provide fall-back capability

38


Mainframe and Server Communication Plan• Communications is key to the successes of the RPB Move to CESC.

• Detailed weekly planning meetings with all Service Delivery Teams – started May 28

• Overview and kick-off meeting with each affected agency – started June 1

– Review move plans and get feedback on proposed move dates and process

– Agencies need to assign POC for application testing and coordination

• Detailed planning meetings with agency application teams to develop test scenarios

• Detailed weekly status reviews with all agency/VITA test teams throughout test window

• Monthly status/update meetings with agency management directly affected by the RPB move

• Dedicated test coordinators from the Transformation Team, Current Ops Team, and the agency

• 24x7 command center setup before, during, and post move/cutover

– Multiple locations linked by phone and/or video conferencing (agency, RPB, CESC)

– Participation by agency application staff, Service Delivery, VITA, Transformation, and vendors (as needed)

– Representation by network, security, mainframe, server, applications, etc

39


ITP Financials

40


$0.0

$50.0

$100.0

$150.0

$200.0

$250.0

Funded Obligation $36.1 $51.0 $65.0 $81.6 $108.4 $122.4 $136.4 $152.7 $172.4 $196.0 $210.5 $230.0

ACTL+FCST $31.8 $48.6 $63.3 $80.6 $106.9 $122.3 $139.2 $156.8 $171.4 $191.1 $211.9 $230.0

Jul '06 Aug Sep Oct Nov Dec Jan '07 Feb Mar Apr May Jun

$M

Actuals

Forecast

Partnership Budget and Forecast – Year 1

$230.0M Forecast Total:

• $81.8M Current Ops

• $1.4M Facilities

• $57.5M Milestones

• $89.3M Managed Exp

Cumulative expenses through April continue to track closely to the budget; this is largely due to the delay of several milestones

$57.5M for milestones based on Northrop Grumman’s proposed schedule, which includes performance bond, relaxed refresh and service level agreement (help desk & messaging) adjustments

41


Independent Program Maturity Assessment

42


Independent Program Maturity Assessment

• To date CACI has completed 3 of 4 currently scheduled assessments

– July and November 2006 with net 192 positive findings and 115 minor negative findings – no major negative findings relative to program best practices

– May 2007 with net 70 consolidated minor negative findings and 45 minor positive findings – no major negative findings relative to program best practices

– Remaining assessment scheduled for November.

– Slower than expected rate by which the program governance and stakeholder management processes should have matured.

• ITP Program Governance Council next steps

– Focus on scope, timing and targets relative to best practices for program management maturation

– Balance demands of tactical program performance, management and best practices

– Determine long-term focus for program assessment

43


IT Infrastructure Partnership Program

Fred Duball and Joe Fay

Program Directors

July 18, 2007ITIB IT Infrastructure Committee

IT Infrastructure Partnership Program (ITP) 0 IT Infrastructure Partnership Program Fred Duball and...

Documents

Transcript of IT Infrastructure Partnership Program (ITP) 0 IT Infrastructure Partnership Program Fred Duball and...