Keeping it cool in a crisis (Relate Live Sydney)
Transcript of Keeping it cool in a crisis (Relate Live Sydney)
#RelateLive
#RelateLive
Keeping it Cool in a Crisis
Dave DysonZendesk Senior Customer Service Evangelist
@dave_dyson
Crisis CommunicationWhat we’ll cover
• Definitions • Goals • Ingredients for Success • Our Red Alert Process • Metrics
Definitions
What is this “Red Alert” that you speak of?Categories of Bad
• Service Disruption • Security Incident • Legal Entanglement • Public Relations Nightmare • Physical Emergency
Goals
The “Prime Directive”Your Mission
• Repair Trust with Customers • Restore Service ASAP • Consistent Response • Efficiency
WARMTH
CO
MPE
TEN
CE
WARMTH
CO
MPE
TEN
CE
Ingredients for Success
Planning AheadA Haiku
A Red Alert is not the time to figure out how to handle one.
KEEPCALM
AND
HAVE A PLANDOCUMENT IT
TRAINTAKE OWNERSHIP
AND
COMMUNICATE
Stay CalmEmotions are Contagious
As a leader, your emotions carry extra weight.
If you’re nervous, your team will be too.
EmpathizeShow You Care
Active ListeningHelps Defuse Anxiety
• Leave your ego behind • Frustration is normal • Let them speak • Verbal “nods” • Reflect what they say • Validate their emotions
“If you fail to plan, you are planning to fail” - Benjamin Franklin
“vaj nab luj SoH, luj nab SoH!” - Benjamin Franklin (original Klingon)
Red Alert PlansWhat to Include
• Process • Roles & Duties • Staffing • Tools • Communications • Special Cases • Metrics
StakeholdersThis isn’t just about you.
• Support Team • Engineering • Operations • Security • Marketing / PR • Sales • Account Management • Customer Success • Executives
Your Plan is Not Set in StoneHone & Refine
DocumentationIf it’s not written, it won’t be remembered
• Complete • Process & People • Clear • Accessible • Up to Date
TrainingPractice Makes Perfect
• Onboarding • Shadowing • Video • Drills
Take Ownership
• Empathize • Apologize • Don’t Shift Blame
Communication“All decks, this is the bridge…”
• Internal • External • Timely • Accurate • Compassionate • Honest • Transparent • One Voice
Red Alerts at ZendeskThe Plan in Action
Red Alert Process DocumentationKeep it secure, keep it safe
• Process Documentation • Checklist • Twitter & Flowdock • How to get Notified • Training Materials • On-Call Schedules
Zendesk Red Alert Process Overview
1. Assess the Situation 2. Alert the Incident Team 3. Communicate to Stakeholders 4. Public Acknowledgement 5. Status Updates 6. Resolve the Issue 7. Wrap Up
Assessment - CriteriaIs This a Red Alert?
Security Event A current, ongoing threat to the security of customer or Zendesk data
OR
Service Event Loss of critical Zendesk functionality
- AND -
The event impacts multiple customers
Impact:
Scope:
AssessmentSpecific Criteria
Critical Functionality: • Account Access • Performance Degradation • Channels • Core Partner Services • Business Rules / Routing • Agent Collision • Reporting • Apps Framework
Security Threat: • Compromised data • Spoofing • Stolen passwords • DDoS Attack
Internal AlertsTools and recipients
PagerDuty: Incident Response Team
(Support, Operations)
Email: Internal Stakeholders
(Engineering, Operations, Marketing)
Flowdock:Wider Team Visibility (Support, Operations)
Company-Wide Visibility
The Incident Response Team
Customers
Customer Advocates
Incident Lead
Support Duty
ManagerOperations
Manager
Engineering & Operations
Staff
Incident Response Team
Incident Response Team - StaffingOn-Call Duty Rotation
Weekly on-call shifts for each role • 8 hours x 7 days – based in AMER, APAC, EMEA
Avoid consecutive duty shifts • Standard: one shift every 6 weeks
Support Duty Manager has on-call Backup as well • Allows for escalation when primary duty manager is unavailable
PagerDuty used for scheduling & alerts • Dashboard shows who is on call & upcoming schedule • Individuals can customizable alerts for incidents and upcoming shifts
The Zencident Room
Red Alert - Problem & Incident Ticket
Problem Ticket • Created internally, owned by Incident Lead • Uses Service Disruption ticket form • Technical updates & discussions recorded here • No risk of inadvertent customer communication • Solves attached incidents automatically
Incident Tickets • Customer reports - attached to the Problem ticket • Proactive tickets attached as well
Service Disruption Ticket FormAdditional Fields
Added to the Problem Ticket: • Impacted Pods • Alert Duration • Number of Incidents • Link to public Help Center article • Link to internal Operations incident record • Checkbox: Post-mortem published
Communicating to CustomersKey Points
• Acknowledge Quickly • Take Ownership • Provide Scope and Impact • Regular Progress Updates • Be Transparent
Communication CadenceWhat We Share, and When
Within 15 Minutes: • @ZendeskOps: Acknowledgement • Create internal Problem ticket • Respond to customers & attach to Problem ticket
Within 30 Minutes: • @ZendeskOps: Scope & Impact • Publish Help Center article • Proactive Notifications to top customers
Every 30 Minutes: • @ZendeskOps: Status update • Update Help Center article • Update Top MRR Customers • (Reset cadence if necessary)
“Tell them what you would want to know”
- Susan Griffin-Black, EO Products
Resolution & Wrap-UpOnce service is deemed restored by the incident team:
1. Ticket Resolution • Solving the Problem ticket solves all attached Incidents • Include link to Help Center article for post-mortem
2. Send “All Clear” Tweet • Include link to Help Center article for post-mortem
3. Post-Mortem (within 3 business days) • Operations team writes internal version • Support Operations Incident Manager edits for public
consumption, and publishes to Help Center article
Service Disruption Post-MortemWhat should it include?
• Scope and Customer Impact • Incident Duration • Communication Timeline • Process / Training Gaps • Recommendations for Improvement
Special SituationsExceptions to the Rule
Security Incidents Involve the Security Team
Zopim Outage Communicate via Zopim Twitter & Facebook
Partner Outage (Twilio, GoodData)
Report incident to Partner Refer to Partner’s system status page
Zendesk Support Instance is Down
Use backup voice service (IfByPhone) Outgoing communication via Tweets only
Shift Handoff Live handoffs only - no email
Overlapping Incidents Separate Red Alerts & Problem Tickets
The Red Alert AppCustom App in our Zendesk Instance:
1. Displays links to active Red Alerts 2. Turn a ticket into a Red Alert ticket
• PagerDuty, Ticket Form & fields 3. Send Social Media Communications
• Twitter, Facebook • Composition, templates, character counts • Salutation and closing • “All Clear” option adds link to Help Center article • Creates or updates Help Center article
4. Proactive Ticket Communications • Lists top customers, with Pod & feature info • Composition, templates, all-clear option • Creates or updates Incident tickets • Updates Yammer
5. All actions add Internal Notes to Problem ticket
MetricsRed Alert Impact Report (Insights)
• Number of Red Alerts • Number of attached Incidents • Total Support Handle Time • Estimated Support Cost ($/ticket) • Number Proactive Tickets • Top Red Alerts (by # of incidents) • Red Alerts by “About” field • Customer Satisfaction over time
• Process Scorecard
Customer SatisfactionRed Alert Tickets
CSAT (Last 7 weeks):
100%
The Future!Where No One Has Gone Before
System Status Page (released) • More detail, account-specific Improved Red Alert App • More capabilities • Possible public version Wider Proactive Communication • More channels • More customers Incident Management as a Job • Not just a role
Conclusion
WARMTH
CO
MPE
TEN
CE
KEEPCALM
AND
PLAN AHEADDOCUMENT IT
TRAINTAKE OWNERSHIP
AND
COMMUNICATE
Smooth Sailing“Second star to the right, and straight on ‘till morning…”
#RelateLive
Q & A