Data CenterConsolidations

Webinar 09.16.11

Bethann Canada, VirginiaPeter Tamayo, WashingtonBaron Rodriguez, PTAC

Data Center Consolidation Considerations & Best Practices

Agenda

• Overview of data center consolidation considerations.

• Virginia’s data consolidation lessons learned – Bethann Canada

• Washington’s data center consolidation effort – Peter Tamayo

• Questions: Please send your questions to AEM Host through the chat box feature.

2

3

Political Reality

• Dwindling state resources

• Political drivers

• Savings?

• Increased security?

• Increased reliability?

• Centralized control of state assets

• Increased capacity needs as a result of SLDS initiatives

Issues to consider…

4

• Physical & Logical Access• Data & System Inventories• Consolidated Data Center Staffing• Business Continuity/Disaster

Recovery Plans

Security Policies

5

• Data Access & Use Policy – This policy specifies data access controls and defines appropriate data use. Such policies often break down specific roles and job responsibilities related to educational data.

• Non-Disclosure Policy – Educational agencies should require individuals who have access to personally identifiable education records to sign affidavits of non-disclosure.

• Data Retention/Destruction Policy – The data center provider generally is responsible for backup and archival of electronic systems. Therefore, a data retention and destruction policy should be crafted based on a careful review of local, state and federal policies.

• Data Breach Policy – This policy outlines procedures to follow in the event of a data breach. It should include references to any applicable state, local, and federal laws and provide appropriate escalation procedures to follow in the event of a data breach.

Other considerations…

• Access/Audit Log Review

• Legal Ramifications

6

“It’s like building your house with your own hands and then watching someone take it apart piece-by-piece with a crowbar.”

Bethann Canada on VDOE’s infrastructure consolidation

2003-20091. Personnel2. Assets3. Network4. Re-IP5. Messaging6. Desktop7. Server

Legislation requires Executive Branch agencies to turn over IT infrastructure to a central IT Agency.

Services are then outsourced to Northrup Grumman.

Governor directs agencies to “play nice”.

The Emperor is still naked…

Service Center Offerings 24x7x365 Security Desk All movement in/out through main lobby Automated access control Controlled visitor access Personnel badges, camera surveillance,

camera analytics Support from local fire and police Backup data center in Southwest VA

Security Offerings Firewalls Intrusion Detection/Prevention Web Proxy for Internet usage and Anti-Virus Email Proxy for Spam and Malware Security Information and Event

Management system Vulnerability and compliance scanning

Network Offerings Single and Dual-factor VPN Site-to-Site VPN Encryption

Router to router Remote Access Site-to-Site VPN E-mail Full Disk Web Proxy

Moving to a consolidated data center: Runbook TOC

Change Log Vendor and Agency Contacts (work, home, cell) Issues Register Resource Plan Application and Database Overview Application Dependencies Backup Overview Server Details Storage Details Network Details Server Runbook Hour-by-Hour Rollback

VDOE’s server move was outsourced to EMC.

Planning, documentation, and execution were extremely thorough and the move was carried out flawlessly.

Moving to a consolidated data center – Hour-by-Hour Plan

Task Posted start time

Estimated duration

Posted completion

time

Estimated Start Time Actual start Percent

completetime In process

Time Remaining

Estimate completion

Actual completion

Application Shutdown

10/16/09 18:00 0:15 10/16/2009

18:1510/16/2009

18:0010/16/09

18:00 100% 2:57 0:00complete 10/16/09 18:15

Database Shutdown

10/16/09 18:00 0:10 10/16/2009

18:1010/16/2009

18:1510/16/09

18:00 100% 2:57 0:00complete 10/16/09 18:15

Server Backup

10/16/09 18:00 10:00 10/17/2009

4:0010/16/2009

18:1510/16/09

18:30 100% 2:27 0:00complete 10/17/09 4:45

Server Shutdown

10/17/09 5:00 4:25 10/17/2009

9:2510/17/2009

4:4510/17/09

4:45 100% 16:12 0:00complete 10/17/09 5:18

Server derack

10/17/09 6:05 3:00 10/17/2009

9:0510/17/2009

5:1810/17/09

5:18 100% 15:39 0:00complete 10/17/09 6:45

load into truck

10/17/09 6:05 3:00 10/17/2009

9:0510/17/2009

3:4510/17/09

6:15 100% 14:42 0:00complete 10/17/09 7:30

Transport equipment to CESC

10/17/09 8:20 1:00 10/17/2009

9:2010/17/2009

7:3010/17/09

7:30 100% 13:27 0:00complete 10/17/09 8:00

Rack equipment

10/17/09 10:20 3:30 10/17/2009

13:5010/17/2009

9:0010/17/09

8:50 100% 12:07 0:00complete 10/17/09 12:02

Start Servers10/17/09 13:50 3:00 10/17/2009

16:5010/17/2009

12:0210/17/09

12:02 100% 8:55 0:00complete 10/17/09 19:24

Start Databases

10/17/09 16:40 0:30 10/17/2009

17:1010/17/2009

19:2410/17/09

14:45 100% 6:12 0:00complete 10/17/09 15:29

Start Applications

10/17/09 17:10 1:30 10/17/2009

18:4010/17/2009

15:2910/17/09

15:50 100% 5:07 0:00complete 10/17/09 18:54

Test Applications

10/17/09 18:40 6:00 10/18/2009

0:4010/17/2009

18:5410/17/09

17:33 100% 3:24 0:00complete 10/17/09 20:24

Moving to a consolidated data center: Hour-by-Hour Rollback Plan Failed lift and ship (truck crashes,

servers destroyed) Database failure Storage failure

3 days Larry Pathrow Request new Storage device 4 hoursLarry Pathrow Install and prepare new array . (initialize the replacement,

carve up the SAN Storage and reallocate the NAS Storage) 1 hour Larry Pathrow Verify/update the zoning on the switches if there are new

WWNs on the storage processors.     BackUp Task(s)   Karen Lusk Restore Data from Backup Tapes (4 TB data)     Application Task(s)

5 mins Application Owner Startup Application

15 mins

Application Owner Test Application

Fast-forward two years… No MOU or SLAs No evidence that backups are taking place 7-month turnaround on work requests for

new servers 2-month turnaround on work requests for

additional storage No knowledge of who has access to data Annual audit points DR cost-prohibitive

Costs 399 PC’s and Laptops $37,422 “help desk” $5,354 357 Mailboxes $5,244 30 Network Printers $1,569 26 Servers $17,776 Storage $28,340

Total monthly bill for June, 2011 $90,969

Benefits Relief – no servers in building

COOP advantage No midnight calls when the AC fails

Shift in focus to business needs Information Management Data Stewardship Accountability Systems Longitudinal Data System

PTAC Webinar, September 16, 2011

18

Fast Facts◦ Number of Schools / Districts / ESDs: 2,253 / 295 / 9◦ Number of K-12 students: 1,023,000◦ Number of K-12 school staff: 90,000◦ Number of SEA employees: 400 staff located in a building built

in 1895◦ State Funding in K-12: More than $6B annually◦ LEAs have Local control◦ Elected Superintendent◦ Dedicated K-20 network and State Government Network◦ Multiple billion dollar budget gaps over the past several years

19September 2011

Why? ◦ IT asset value in excess of $1B◦ State and local governments have to live within their means◦ No growth in near future◦ Opportunities for improving service to constituents◦ Need to be more “Green”

How?◦ Leverage economies of scale to drive down costs◦ Fully utilize assets◦ Standardize infrastructure◦ Consolidate data centers◦ Use the state’s buying power to purchase commoditized products◦ Cloud computing

20September 2011

New WA State Data Center (SDC) and Office Building◦ $255 million dollar construction cost◦ Consolidates at least 32 data centers◦ RFI in Summer 2010 for vendor information on a “turnkey”

approach for data center operations◦ New Data Center will have four data halls within the 50,000 sq.

ft. SDC◦ Office Building completed in June 2011◦ SDC is partially operational serving building staff◦ Plans to migrate agencies to the new data center have lost

momentum due to the reorganization of the state’s IT department

21September 2011

Engage with agency leadership, legal and financial staff and legislators early and often with the K-12 requirements and business cases

Discuss with WA LEAs and other SEAs on lessons learnedo NCES Forum, MIS

Participate with the other WA state agencies and the central IT department in the planning process

o Lift and Shift or “End of Life” replacement migrationo Server virtualization projecto Negotiate preliminary agency migration sequenceo Review and provide feedback to preliminary documents

Utilize sound and rigorous project management practices

22September 2011

23September 2011

Key items on the project plan◦ Review policy, security and legal requirements and perform gap

analysis WA State Policies and Standards: Personnel Security, Physical

Security, Data Security, Network Security, Access Security, Records Retention

◦ Stakeholder Management Coordinate with the K-12 application development staff Coordinate with OSPI program staff and user community Peer communication in a shared environment

◦ Patch hardware with the latest software updates◦ Clean up folders and files◦ Freeze environment

24September 2011

Key items on the project plan (continued)◦ Update IT Portfolio and other as-is and to-be documentation◦ Conduct proof-of-concept projects, pilots or other phased-in

approaches◦ Develop test plans◦ Develop cut-over and support plan with updated SLAs◦ Update and exercise Business Continuity Plans ◦ Implement a safety net

Full backups Develop roll-back plans Get another set of eyes on the plans and documentation

25September 2011

Peter Tamayo, Chief Information Officer◦peter.tamayo@k12.wa.us◦(360) 725-6134

26September 2011

Data Center Consolidations – Final Thoughts

• Efforts are partnerships!

• Work with your state attorney general to ensure compliance with federal, state, and local laws.

• Contact the Family Policy Compliance Office should you need assistance.

27

PTAC Help Desk & Website

28

Send PTAC your questions on privacy, confidentiality, and data security related to longitudinal data systems (LDSs)

Contact the Help Desko PrivacyTA@ed.gov

o Toll Free Phone: 855-249-3072

o Toll Free FAX: 855-249-3073

Get copies of PTAC resources, join our email list @ http://nces.ed.gov/programs/ptac

Data Center Consolidation

Webinar 09.16.11

Data Center Consolidation Considerations & Best Practices

Thank You for Participating