Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines...
-
Upload
hoangtuong -
Category
Documents
-
view
221 -
download
4
Transcript of Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines...
![Page 1: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/1.jpg)
Anatomy of an IT Outage(and ways to prevent them)
IT on Wall Street SeminarSep 22, 2016
Doron Pinhas, CTO
![Page 2: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/2.jpg)
Or it can look like this:
“System outage grounds
2300 Delta flights”
![Page 3: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/3.jpg)
» Always unexpected
» Usually takes too long to resume service
» Way too often…• Root cause remains unknown
• Same problems reoccur
Anatomy of an IT outage
3
![Page 4: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/4.jpg)
» Technology evolution and environment complexity don’t make it any easier
*Based on Continuity Software annual surveys
Are we getting better?
When was your last downtime event?
4
![Page 5: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/5.jpg)
Can this be the real reason?
Can this be the real reason?
Can this be the real reason?
A closer look into recent events
5
Delta airlines Reported damages reach $150M (est.)
Power switch failed
Where Impact Reason given
Worldpay A million Etsy transactions affected (est.)
Server update
Southwest Airlines
2300 flights canceled Network router failed
Deutsche Bank Unable to report swap data for five days
???
![Page 6: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/6.jpg)
6
“The issue arose from a hardware failure in the main database of the equities’ trading system”
![Page 7: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/7.jpg)
Why does it keep happening?
Hint: Not [necessarily] for lack of trying…
7
![Page 8: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/8.jpg)
Possible reasons
8
Design issues
Implementation issues
Testing issues
Measurement(of Quality and Risk)
![Page 9: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/9.jpg)
• Setting up a resilient infrastructure is costly and complex– Cloud orchestration, HA (virtual HA/FT, clustering, App LB), redundant /
active-active storage, multi-pathing, teaming, Network LB, replication, Geo-HA (SRM, Stretch/Metro compute & storage), …
• The result: multiple technologies, vendors and teams
The resilient datacenter – blueprint
Site 1 Site N
Active-active / Active-passive
9
![Page 10: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/10.jpg)
Site 1 Site N
The challenge? Simple math…
OS Configuration
ApplicationConfiguration
HAConfiguration
SANConfiguration
LB sessionconfiguration
VMware HA
VMware SRM
• Some changes slip through the cracks…• IT stability & quality cannot be fully tested following every change
SnapshotConfiguration
Mirror / Replica
Configuration
PuppetManifest
Active-active / Active-passive
10
![Page 11: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/11.jpg)
Ris
k
Time
New build / Test / Audit
Every day that goes by…
11
![Page 12: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/12.jpg)
If disaster strikes tomorrow…
12
How confident are you that your IT will recover smoothly?
12
![Page 13: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/13.jpg)
How to prevent your next outage?
13
![Page 14: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/14.jpg)
Lessons learned from the best run shops
14
1 Design right• Manage knowledge• Rely on community (vendors, other
users)
2 Implement right • Test quality immediately
3 Make sure it stays that way (see next slides!)
Put quality control at the centerTransforms IT operations from reactive to proactive
![Page 15: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/15.jpg)
Making sure your environment stays ready
15
Transform IT operations from this:
… to that
Must be automated!
Ris
k
Time
New build / Test / Audit
Daily validation
![Page 16: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/16.jpg)
About quality & risk automation
» Stamina & focus more important than pace• Each small addition goes a long way
• Start with your:– Existing manual checklists
– Most recurring issues
– Any newly discovered (& significant) risk
» Ways to automate & “shortcuts”• Use vendor scripts & built-in
validation tools
• Create your own scripts
• Use automation / configuration management tools
16
Fast ROI – quickly frees time to fuel your journey
Limitation: cross-domain issues will not be caught!
![Page 17: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/17.jpg)
Tracking & enumeration
» Record results over time• Create score-cards for servers / objects
• Will allow comparing status before and after a change, and examine trends
» Mining the data enables numerous benefits• Understand what works and what does not
• Benchmark your vendors
• Make it part or your decision making process
• …
17
![Page 18: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/18.jpg)
Case study – large bank
» Automating checks in design labs, pre-prod and prod
» Store results in a repository correlating:• What has changed
• Score cards
• Open issues & trending
• Business dependencies
» Customized feeds for:• IT teams
• Business owners
18
![Page 19: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/19.jpg)
Case study – results
» 90% reduction in downtime
» 70% reduction in firefighting costs
» Dramatic improvement in predictability & confidence
• Tests and actual workload shifts work
• When gaps do exist, it’s easy to understand: what to do
» Better collaboration
• From finger-pointing to constant improvement
» Significant return on investment
• New use-cases uncovered regularly
19
![Page 20: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/20.jpg)
The right program – key to constant improvement
» Change control process:• Better decision making through quality & risk measurement
• Shorter cycles
» Stay ahead of trouble• Automate
• Handle violations immediately
» Cross-team collaboration & visibility• Single-pane of glass for IT configuration quality, health & risk
20
![Page 21: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/21.jpg)
About us
21
![Page 22: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/22.jpg)
About us
Helping many of the world’s largest
enterprises prevent outages and data
loss in their critical IT infrastructure.
22
![Page 23: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/23.jpg)
Our technology & services
23
Early detection of availability risks and single-points-of-failure
Actionable alerts to relevant teams
Automated, cross-layer configuration validation
Easy measurement and visualization of resiliency metrics
AvailabilityGuard Services
Resilience health checks, best-practice validation
Managed Service Availability Assurance
![Page 24: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/24.jpg)
To learn more
» Come talk to us....
» One-time health check
24
![Page 25: Doron Pinhas, CTO - wsta.org · Doron Pinhas, CTO. Or it can look like this: ... Delta airlines Reported damages reach $150M (est.) Power switch failed Where Impact Reason given Worldpay](https://reader031.fdocuments.us/reader031/viewer/2022021820/5ad98d9f7f8b9a53618b69cf/html5/thumbnails/25.jpg)
Thank you!(Questions?)
app.continuitysoftware.com
25