Post on 28-Dec-2015
W-04: Experiences in Virtualization
Panelists: Randy Moore, ECY
1
Ecology Virtualization• Where we started…
– Agency IT Steering Committee mandate to eliminate servers in Program areas and centralize server administration function
• Our objectives– Reduce hardware, licensing, server administration and facilities costs– Free up Program IT resources to provide additional line of business or
application support• Obstacles
– Program resistance • Our servers, our funds, service concerns
– Staff resistance • We will lose control, what happens to our jobs
• So where’s the virtualization???
Ecology Virtualization • Where we really started…
– Central IT realization that we needed to apply the same principles to the agency’s centralized IT infrastructure
• Our objectives– Reduce hardware, licensing, server administration and facilities costs of
centralized IT infrastructure– Meet Governor’s GMAP goals for virtualization – As much as possible align with State IT transformation efforts
• Obstacles– Lack of trained IT staff with knowledge of virtualization– My application can’t or shouldn’t be virtualized or my vendor won’t support it if
it’s virtualized, Sorry my SW uses dongles for security– Fear of all eggs in one basket– It costs money – how are we going to pay for it?
• So where we are now….?
A Pictures Worth…
20.5%
79.5%
Phy sical 20.5%Virtual 79.5%
Total: 100.0%
Percent of HQ Servers Virtualized
By the Numbers…
• Ecology as of Today– 221 servers agency wide
• 185 at HQ in Thurston County, 79.5% virtualized• 7 more servers targeted for retirement
– 1 SAN enabled virtual farm with 3 VMWare clusters• Prod, Dev and Oracle
• Anticipated Windows server growth of 10% -20% per year -… just throw up another VM is the current mantra
ECY Virtual Platform Today
• VMware ESXi 4.1• Cisco UCS Blades with 1.5+ TB of RAM • Mix of NetApp NFS and HP FC Storage • 60TB RAW NFS RAID DP SAS and 60TB RAW of FC
RAID5 SATA• Mix of VMFS and NFS datastores• Windows licensed at the processor level• Certified FlexPod
Storage
• NFS based 10GB Ethernet storage (NetApp)• FC based 4GB FC storage (HP)• Thin provisioned, over provisioned• Fully virtualized (no constraints)• Legacy HP EVA 5000 and HP EVA 8000 still in
place but on track to be retired
ECY Virtual Platform
NFS and VMFS Datastores for VMs
How Did We Get Here?
• Multiple Phases – Learning phase
• Used equipment (added RAM)• Existing HP SANs for storage• Internal staff only • Gain knowledge and experience• Only a few dev/test servers virtualized
– Growth/Adoption phase• Purchase a couple new servers• More RAM • Existing HP SANs for storage• Internal staff only • Significant number of dev/test servers virtualized
How Did We Get Here? (cont.)– Adoption Phase
• Two Clusters (prod/dev) • Existing HP SANs for storage• Internal staff only • Most dev/test servers virtualized, couple of Prods• Where did all the storage go?
– Time to do it for real Phase• Brought in contractor - INX• Developed long range plan• Purchased new compute and storage platforms• SolarWinds tools for monitoring • Stood it up - worked out the kinks – yes there were kinks• Turn it back over to Ecology Staff to complete the P2Vs
What Did We Virtualize?• Everything we could (Prod and Dev)
– 16 MS SQL Servers (30+ instances )– SharePoint – Agency Oracle Platform– GIS including SDE servers– Remote Desktop Services– Domain (GCs) servers– SCCM & SolarWinds servers– Web/Application /COTs/Web Services servers– Print servers – A couple of file servers– Some Developer “desktops”– Almost everything
What Didn’t We Virtualize? (yet)
• 1 MS SQL Server • Our primary domain controller (DC)• Facility Servers (HVAC, Doors, etc)• Largest Windows File Servers• 10 - Modeling Servers owned by Programs• Desktop Imaging servers• Servers owned by other building tenants
Work in Progress
• Elimination of Window File Servers – Contain over 55 million documents– Consume over 27TB of storage– Decision made not to virtualize due to size and
storage impacts– Worked with INX to develop a Storage Strategy – Solution included purchase of F5 ARX 2500 and
additional 96TB of CIFs enabled NetApp storage. – Transparent migration of data to new storage and
elimination of file severs
Lessons Learned
• Communicate & educate even if you don’t think they need to know
• Be prepared to roll back (just in case) • Better to overbuild than be caught short of resources• You can never have too much RAM or too much Storage• Staff training and knowledge of the infrastructure is critical• Don’t put all your eggs in one people basket – cross train• Implement a disciplined change management approach• Don’t assume something works – validate/test connections,
failover, throughput, etc. • Monitor and take proactive action to prevent failures