W-04: Experiences in Virtualization

Panelists: Randy Moore, ECY

1

Ecology Virtualization

IPMA Briefing 2012

Ecology Virtualization• Where we started…

– Agency IT Steering Committee mandate to eliminate servers in Program areas and centralize server administration function

• Our objectives– Reduce hardware, licensing, server administration and facilities costs– Free up Program IT resources to provide additional line of business or

application support• Obstacles

– Program resistance • Our servers, our funds, service concerns

– Staff resistance • We will lose control, what happens to our jobs

• So where’s the virtualization???

Ecology Virtualization • Where we really started…

– Central IT realization that we needed to apply the same principles to the agency’s centralized IT infrastructure

• Our objectives– Reduce hardware, licensing, server administration and facilities costs of

centralized IT infrastructure– Meet Governor’s GMAP goals for virtualization – As much as possible align with State IT transformation efforts

• Obstacles– Lack of trained IT staff with knowledge of virtualization– My application can’t or shouldn’t be virtualized or my vendor won’t support it if

it’s virtualized, Sorry my SW uses dongles for security– Fear of all eggs in one basket– It costs money – how are we going to pay for it?

• So where we are now….?

A Pictures Worth…

20.5%

79.5%

Phy sical 20.5%Virtual 79.5%

Total: 100.0%

Percent of HQ Servers Virtualized

By the Numbers…

• Ecology as of Today– 221 servers agency wide

• 185 at HQ in Thurston County, 79.5% virtualized• 7 more servers targeted for retirement

– 1 SAN enabled virtual farm with 3 VMWare clusters• Prod, Dev and Oracle

• Anticipated Windows server growth of 10% -20% per year -… just throw up another VM is the current mantra

ECY Virtual Platform Today

• VMware ESXi 4.1• Cisco UCS Blades with 1.5+ TB of RAM • Mix of NetApp NFS and HP FC Storage • 60TB RAW NFS RAID DP SAS and 60TB RAW of FC

RAID5 SATA• Mix of VMFS and NFS datastores• Windows licensed at the processor level• Certified FlexPod

Storage

• NFS based 10GB Ethernet storage (NetApp)• FC based 4GB FC storage (HP)• Thin provisioned, over provisioned• Fully virtualized (no constraints)• Legacy HP EVA 5000 and HP EVA 8000 still in

place but on track to be retired

ECY Virtual Platform

NFS and VMFS Datastores for VMs

How Did We Get Here?

• Multiple Phases – Learning phase

• Used equipment (added RAM)• Existing HP SANs for storage• Internal staff only • Gain knowledge and experience• Only a few dev/test servers virtualized

– Growth/Adoption phase• Purchase a couple new servers• More RAM • Existing HP SANs for storage• Internal staff only • Significant number of dev/test servers virtualized

How Did We Get Here? (cont.)– Adoption Phase

• Two Clusters (prod/dev) • Existing HP SANs for storage• Internal staff only • Most dev/test servers virtualized, couple of Prods• Where did all the storage go?

– Time to do it for real Phase• Brought in contractor - INX• Developed long range plan• Purchased new compute and storage platforms• SolarWinds tools for monitoring • Stood it up - worked out the kinks – yes there were kinks• Turn it back over to Ecology Staff to complete the P2Vs

What Did We Virtualize?• Everything we could (Prod and Dev)

– 16 MS SQL Servers (30+ instances )– SharePoint – Agency Oracle Platform– GIS including SDE servers– Remote Desktop Services– Domain (GCs) servers– SCCM & SolarWinds servers– Web/Application /COTs/Web Services servers– Print servers – A couple of file servers– Some Developer “desktops”– Almost everything

What Didn’t We Virtualize? (yet)

• 1 MS SQL Server • Our primary domain controller (DC)• Facility Servers (HVAC, Doors, etc)• Largest Windows File Servers• 10 - Modeling Servers owned by Programs• Desktop Imaging servers• Servers owned by other building tenants

Work in Progress

• Elimination of Window File Servers – Contain over 55 million documents– Consume over 27TB of storage– Decision made not to virtualize due to size and

storage impacts– Worked with INX to develop a Storage Strategy – Solution included purchase of F5 ARX 2500 and

additional 96TB of CIFs enabled NetApp storage. – Transparent migration of data to new storage and

elimination of file severs

Lessons Learned

• Communicate & educate even if you don’t think they need to know

• Be prepared to roll back (just in case) • Better to overbuild than be caught short of resources• You can never have too much RAM or too much Storage• Staff training and knowledge of the infrastructure is critical• Don’t put all your eggs in one people basket – cross train• Implement a disciplined change management approach• Don’t assume something works – validate/test connections,

failover, throughput, etc. • Monitor and take proactive action to prevent failures