IBM Inter connect2016 Konferansı - SUPERMICRO
-
Upload
sunucudeposu-supermicro-sistem-entegrasyonu -
Category
Business
-
view
102 -
download
0
Transcript of IBM Inter connect2016 Konferansı - SUPERMICRO
Shifting a Global Applications Services Business to Public CloudSpeaker: Paul Rafferty, Director of Infrastructure Architecture, Silverpop
Silverpop
• Cloud-based global marketing automation services business
• 2500 customers; 500+ employees; 2 managing storage
• Multitenant SaaS product -- 100s of customers per deployment
• Cutting edge big data analytics to analyze customer behavior
• Pre SoftLayer: 4 data centers; After SoftLayer: 10 data centers
1
Video: Silverpop Spectrum Accelerate Use Casehttps://youtu.be/y2uumq9IAZ0
Our cloud challenge
• Goal: Strategically shift from traditional data center to public cloud
storage to grow globally faster, be compliant, and save costs
• Challenges:
– Swiftly migrate mission critical customer data out of physical data centers
– Locate our services closer to our global customer base
– Meet new global regulations for privacy, data protection and transfer
– Meet our data protection and performance needs in a cloud model
• Solution:
– SoftLayer + IBM Spectrum Accelerate software defined storage (with
Supermicro as the underlying server hardware)2
Why we chose Spectrum Accelerate (1/4)
Data protection and performance
• We are paranoid about data protection – SaaS application hosting OUR
customers’ data – close to 1PB in total
• Typically maintain three online copies of all data on disparate arrays
• Primary production copy
• Secondary production copy
• Primary disaster recover copy
• All three must be able to support production workloads
• Very high IOP needs per system (25-75k IOPs)
• Very high throughput per system (1GB/s)
3
Why we chose IBM Spectrum Accelerate (2/4)
• over VMware Virtual SAN™ (VSAN)
– Enterprise management features
– Scale – need 100s of TBs per location
– Rich support matrix for diverse workloads (hosts,
clusters, files systems), not just VMware
• over EMC ScaleIO®
– Simpler, lower cost client model of iSCSI versus
driver: no need to buy FC HBAs
– XIV foundation is a more natural fit for
transitioning from on-premises arrays
– No replication and no data migration tools
– We do run about 30TB of ScaleIO in our on-
premises data centers
4
• over Open Source
– Maturity, scalability, proven reliability
– Quick to launch, low barrier to entry for SAN
admin
– For scaling, ease of use, predictable
performance, and features
• over SoftLayer storage
– Management of 1000s of LUNs is difficult
– Aggregate performance of an XIV array
versus 1000s of individual SoftLayer devices
– lower cost!
– LUN growth and migration
– Performance monitoring
Lots of options out there…
Why we chose Spectrum Accelerate (3/4)
It fits our workloads...
• We are very write heavy
– Workloads with over 20k+ random 8kb overwrites per second per database server
– XIV (and enterprise arrays) are very good at write workloads compared to mid-tier
solutions
– Data consistency is very important – XIV lineage has years of dealing with the
vagaries of SATA drives
• We have large databases – support and manage 100s of LUNs.
– Smallest Oracle databases are 20TB with an average over 40TB
– Smallest DB2 databases are 2TB with an average over 8TB
5
Why we chose Spectrum Accelerate (4/4)
It fits our workloads…
• Have very bursty and concentrated workloads
– Multitenant product – anyone at anytime can be “busy”
– Large read caches are very important – a full Spectrum Accelerate cluster can have
close to 2TB of memory
• We are very sensitive to read latency
– SSD cache is very important to keeping read response times under 2ms
– Typical query may require 10,000 or more 8Kb reads
– A full cluster has access to 12TB of SSD read cache – we know our hot working sets
will be on fast SSD while the cooler ‘historical’ data will remain on the less expensive
SATA drives
6
Get ready, get set, deploy!
• April: Prepared SoftLayer infrastructure
• July: 125 TB migrated to Spectrum Accelerate
• August: 270 TB “fully loaded”
• October: 500 TB in US, Canada, Europe
• October: 400 TB for another business unit
• November: Adding new cluster every 3 weeks
• 2016: Deploy 20 clusters in 7-8 more countries
Cookie cutter pattern! Deploy fast! Leverage SoftLayer IaaS!
April 2015
2016
Our Spectrum Accelerate solution in SoftLayer (1/3)
Hardware – standard SoftLayer 12 drive dual socket systems (Supermicro) with:
• 128GB RAM
• 2 x E5-26xx CPU (typically 2650s)
• 2x10GBps private network connections
• 1 800GB SSD – OS
• 1 800GB SSD Cache
• 10 x 4TB SATA drives
• ESX 5.5u2 – Leverage VMware Service Provider Program pricing from
SoftLayer for licensing
• Prefer LSI RAID controllers in JBOD but have had no issues with Adaptec
8
Our Spectrum Accelerate solution (2/3)
Size of clusters
– We like to build either 7 node or full 15 node clusters
– In SoftLayer we like to pre-provision a spare server
– We almost always provision all 12 drives per server
Why?
– Our smallest locations typically need about 100TB of space. Our SoftLayer
bare metal hardware choice provides us with about 120TB for a 7 node
cluster or 255TB for a full 15 node cluster
– Hardware will fail – we like to have a node ready
9
Our Spectrum Accelerate solution (3/3)
Features in use
• Obvious, but unique in a cloud environment – full host group & LUN masking
functionality.
• We have 1000s of LUNs in SoftLayer – XIV unified interface makes this
intuitive to storage and system administrators
• Replication – forming the basis for our geographical disaster recovery
leveraging the SoftLayer internal backbone (no cost for transit)
• SSD and Memory – we are seeing very good cache hit rates
• Performance tools – XIVTop versus limited or no visibility to cloud storage
options
10
Our Spectrum Accelerate solution: Workloads
• Oracle
– Primary databases and Data Guard copies
– Very high IOPs – on order of 70-100k IOPs per DB server
– 100+ LUNs per Oracle server in some cases
• VMware
– Providing primary storage for all VMDKs in our SoftLayer based private clouds
– We also do iSCSI LUNs mapped directly to Linux and Windows guest VMs
• DB2 on Linux
– ETL and DSS workloads – high read throughput
• Windows
– Clustered SQL server in SoftLayer
– Clustered Application Servers in SoftLayer
11
Best Practices: What we have learned about… (1/12)
• Deploying Tier 1 Oracle and VMware on Spectrum Accelerate
in public cloud
• Setting up SoftLayer
• Using VMware ESXi as the hypervisor
• Using Supermicro hardware
13
Best Practices: Client Hosts in SoftLayer (2/12)
One of our larger single Linux Oracle hosts in SoftLayer
• 130 assigned LUNs
• Over 250TB of assigned Spectrum Accelerate disk
• Using udev – managed by DBAs (well trained )
• Using HAK
• Sustained 1GB/s workloads
• 50-75k IOPs
• 40/60 Read to Write (~600MB/s writes)
• 4-6ms overall latency
• 70% IOPs @ 8kb
• multipathd: “ls /dev/sd* | wc –l = 801”
14
Best Practices: Client Hosts in SoftLayer (3/12)
• Leverage the XIV host attachment kit – it is excellent
– Team went from running a pure on-premises fiber channel storage
infrastructure to an iSCSI-based overnight
• Basic networking
– We use LACP on the host side
– Use iPerf to test, verify, and baseline your hosts and network configurations
– Tune the standard TCP tunables for Linux – net.core.*
• (See the old but still useful http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf)
– We have no issues in SoftLayer maxing out our 10Gbps links to Spectrum
Accelerate with the Supermicro AOC-STG ixgbe based NICs out of the box
15
Best Practices: Client Hosts in SoftLayer (4/12)
16
Typical day for one of our database servers in SoftLayer… 1GB/s
Best Practices: SoftLayer (5/12)
Networking
• Get your networking right – high volumes of traffic!
• Use large private portable networks for your iSCSI traffic
– We request and use /24 portable and primary subnets and keep all traffic
within a flat layer 3 segment: Avoid transiting network gateways!
– Add additional iSCSI subnets if needed
• Dedicated subnets for management and the interconnect IPs
– We use dedicated portable subnets for each
– No outside traffic allowed into the interconnect network
– Can apply network ACLs to our management network
17
Best Practices: SoftLayer + ESX – SA Hypervisor (6/12)
• Pay attention to interconnect and client traffic ports
– We are using an off-the-shelf SoftLayer configuration with two 10Gbps
private NICs per server
– We “break” bonding at the switch and at the ESX level
– Dedicate one interface to interconnect and one to client/management traffic
and set these to fail over for each other for the rare cases an access switch
or NIC fails
– Use jumbo frames – MTU of 9000 on client, ESX vSwitch, and Spectrum
nodes
• Leverage VLANs and Tagging
– Clients live on a default “app” VLAN and iSCSI VLAN is trunked in
18
Best Practices: SoftLayer + ESX – SA Hypervisor (7/12)
Hardware burn in…
• Allocate some burn in time for the SATA drives
• A full cluster will have 180 SATA drives of “various provenance”
• We find we will lose 2-3 drives in the first 1-2 weeks of a built out
• Rarely do we encounter a bad SSD, but we do check the smart data
• After 4 weeks our clusters achieve very good stability
– Bad disks have failed and any server issues (e.g. pink screens) have been
identified and rectified/replaced
– Run workloads against new cluster with your favorite disk IO tool
19
Best Practices: SoftLayer + ESX – Client Hypervisors (8/12)
We use Spectrum Accelerate to provide storage for our VMware ESX
clusters hosted on SoftLayer bare metal servers
• Create a single distributed virtual switch
• Trunk in our iSCSI VLAN to ESX server
• Create two iSCSI port groups
– A and B Side – each assigned to a single unique physical NIC
– Single VMkernel port on each
– Set single active uplink and disabled standby uplink on each
– Create software iSCSI adapter with the two iSCSI port groups
20
Best Practices: SoftLayer + ESX – Client Hypervisors (9/12)
21
Best Practices: SoftLayer + ESX – Client Hypervisors
(10/12)
We also have some “big” VMs
that actually connect directly to
our Spectrum Accelerate
clusters
• We cheat…
– Create a “storage port group”
– Assign single NIC to VM
– Expose 4 “target” paths to
VM’s iqn
– Let VMware load balance for
us
22
Best Practices: Using Supermicro hardware (11/12)
• Standardize on a firmware and driver for LSI
– Versions:
– Don’t trust SoftLayer to always deliver the same firmware – even within the
same order!
– Obviously watch the HCL for Spectrum with each version
However we have found it to be lenient on being slightly out of spec
• RAID Card
– OS disk is a proper “virtual disk”
– Cache SSD and SATA disks are in pass through/JBOD
– Typically have to “re-do” storage configuration on freshly ordered systems –
SoftLayer likes to deliver with all drives in a VD even if you didn’t ask for it
23
Best Practices: Using Supermicro hardware (12/12)
We really like to use the 12 drive 2RU systems
• Why?
– Our experience in SoftLayer has shown an order of magnitude fewer soft
and hard disk errors under load with the Supermicro BPN-SAS3-826EL1
back plane versus the larger BPN-SAS3-846EL1 back plan in the 24+ drive
systems
• What do we compromise on then?
– We can’t build maximum capacity clusters (1 OS + 1 SSD + 10x4TB)
– We use a single SSD for the ESX OS install and VMDK space for the
Spectrum Accelerate guest
24
Benefits in using Spectrum Accelerate
• Speed to deploy
– Cannot stress this one enough: leverage the IaaS of SoftLayer and the
power of Software Defined Storage
– 24-48 hours to spin up a 250 TB cluster – includes SoftLayer tasks
• Performance
– All performance benefits of XIV in our cloud environments
• Overall: Reliability, consistent performance, ease of use
– Cloud is naturally un-reliable – Spectrum Accelerate addresses that
25
Benefits in using Spectrum Accelerate
• TCO
– Can better match capacity to budget – no one shot at “capex”
– Deploy small cluster and add disks as needed – match expense to actual
load – less idle capacity
– Improve our Storage Admin efficiency – manage 2-3x the space per SAN
administrator
Important as cloud deployments tend to “balloon” footprints
– Have retired two physical data centers and one more is in our 2016 roadmap
– As faster servers, disks, and features come to SoftLayer and Spectrum
Accelerate we can immediately upgrade – no write-off/depreciation
26
What’s next
• Deploying our application in additional countries as SoftLayer opens
new data centers
• Continue to meet our growth – over 10TB per month of primary data
• Migrate all on premise legacy tape backups to disk to disk based
Spectrum Accelerate clusters in public cloud
• Evaluating moving to XIV in our current physical data centers
– Speed migration of products into public cloud
– Leverage replication for disaster recovery
– Leverage replication for dev/test at scale in public cloud
27
Summary
28
“Spectrum Accelerate is a Tier 1 array that we can
use on the cloud, spin up fast, and we know will be
reliable.
It gives us the right blend of speed to provision,
performance, operational simplicity and integration
with, again, just the whole ecosystem we are running
in SoftLayer.”
Paul Rafferty
Director, Infrastructure Architecture, Silverpop
Supermicro® Hyperconverged Appliance with IBM® Spectrum Accelerate™Speaker: Marc The’berge, Director, Rack Solutions, Supermicro
Email: [email protected]; Twitter: @tbird960
The hardware below
• Silverpop use case involves:
– Spectrum Accelerate on SoftLayer® with underlying Supermicro hardware
– Future: XIV® integrated systems for fast data migration to SoftLayer
• Supermicro and IBM now offer a hyperconverged appliance with IBM
Spectrum Accelerate for your data center
– All the benefits of SDS for commodity servers and the ease and confidence of
a pre-tested, pre-integrated, validated appliance
– Proven enterprise-class storage capabilities for your data center and sites
30
Supermicro & IBM: Perfect synergy for a hyperconverged building block
31
• A leader in x86 infrastructure
• $2B/year publicly traded US
company
• WW sales and 24x7
service/support Manufacturing in
US, Europe and Asia
• First with dense servers and
storage
• Flexible and agile solutions
integration
• #1 in Software Defined Storage
platform by IDC
• $1B investment in SDS & Storage
• IBM Spectrum Accelerate is
based on proven XIV enterprise
storage
• #1 cloud computing provider in
the US by IDC
• #1 in US Patents for 22 straight
years
32
• Fast, low risk deployment
• Hardware, firmware, VMware® ESXi™ and IBM Spectrum
Accelerate stack configured and tested
• Complete appliance stress tested
• All hardware and cables labelled
• 6, 9 and 15 node standard configurations
• Compute, storage and networking cabled in the rack
• Customized configurations available
• With 3-15 nodes per instance
• More/different processors
• More RAM
• 800GB SSD; 1, 2, 3, 4 or 6TB NL SAS drives
• Customer choice rack, “rack ready,” or built into available
racks
Supermicro Hyperconverged Appliance powered by
IBM Spectrum Accelerate
33
Supermicro Hyperconverged Appliance powered by
IBM Spectrum Accelerate
Appliance SKU SRS-42USSG-SPAC-01 SRS-42USSG-SPAC-02 SRS-42USSG-SPAC-03
Node 6x 2U 12x 3.5" SSG 9x 2U 12x 3.5" SSG 15x 2U 12x 3.5" SSG
Processor/Node Intel® Xeon® E5-2620v3 Intel Xeon E5-2620v3 Intel Xeon E5-2620v3
Memory/Node 4x 16GB DDR4 ECC Reg 4x 16GB DDR4 ECC Reg 4x 16GB DDR4 ECC Reg
SSD/Node 1x 600GB Intel DC S3500 1x 600GB Intel DC S3500 1x 600GB Intel DC S3500
Total Data Drive 72 SAS3 108 SAS3 180 SAS3
Total Effective
Capacity114TB (4TB NL SAS) 186TB (4TB NL SAS) 330TB (4TB NL SAS)
Disk Controller LSI® 3008 SAS3 LSI 3008 SAS3 LSI 3008 SAS3
10G Interface 4x 10G SFP+ 4x 10G SFP+ 4x 10G SFP+
GbE Switch 1x SSE-G2252P 1x SSE-G2252P 1x SSE-G2252P
10GbE Switch 2x SSE-X3348SR 2x SSE-X3348SR 2x SSE-X3348SR
Notes:• Can be customized with 2nd processor, more powerful processor, more RAM, 1, 2 3, 4 or 6TB NL SAS
drives, 800GB SSD and 3-15 nodes per module• 3 nodes for test/development environments only, 4-15 nodes for production workloads
Who needs it and why: Use cases
34
Use case Who Why
Easy to manage
hyperconverged infrastructure
for VMware workloads
Small to large
businesses
• Powerful VMware integration lets VMware admins control
infrastructure, simplify deployment & management, avoiding
need for storage professionals
Development and test
environment/ ad-hoc projects
Medium to large
companies
• Fast deploy at lower risk as is tested and delivered ready to
deploy
• Need to save costs but want reliability and compute
• Can leverage in-house skills to build what else is needed
Powerful building block for
hyperconverged solutions
Medium to large
companies
• Fast deploy and at lower risk if using an appliance tested and
delivered ready to deploy
• Need to save costs but want reliability and compute
• Can leverage in-house skills to build what else is needed
Lower capacity enterprise-class
storage capabilities
Large companies
with many branch
offices
• Easy to centrally manage, replicate data between branch
office and headquarters
• Excellent price/performance and flexibility for backup and
remote sites
Hybrid cloud solutions Any • Increase use of public cloud in single managed on/off
premises environment
What we offer
• Two options:
– Rack option: Cabled and racked at the Supermicro factory
– Rack-ready option: Loaded with Spectrum Accelerate,
cabled and tested, with nodes, switches and cables labelled
and shipped individually. For customers who self-rack their
hardware
• Spectrum Accelerate enterprise-class capabilities:
– Scale-out, No tuning and no hotspots, management scaling
and mobility, integration with VMware, OpenStack®,
Microsoft® and IBM
• Support:
– 24x7, 4-hour response time from Supermicro hardware
support
– Software support from IBM and VMware
35
10GbE
Hypervisor
Spectrum
Accelerate
Spectrum
Accelerate
Spectrum
Accelerate
iSCSI
iSCSI
iSCSI
Hypervisor
Hypervisor
Supermicro Hyperconverged Appliance summary
• Perfect hardware/software synergy
• Appliance available with 3 standard configurations and custom
configurations
• Fast deploy for range of real needs
• 24/7 support
• Get one today!
36
Thank YouYour Feedback is Important!
Access the InterConnect 2016 Conference Attendee
Portal to complete your session surveys from your
smartphone,
laptop or conference kiosk.