Mean Time to Sleep: Quantifying the On-Call Experience

@lozzd • @ryan_frantz

Mean Time to SleepQuantifying the on-call experience

Laurie Denness@lozzd

Ryan Frantz@ryan_frantz

Who is in an on-call rotation?

Who is on call right now?

Who feels like on-call sucks?

Welcome. How is on call?

Let’s help our people sleep

Make on-call more bearable

Incremental Changes

Email toAcknowledge

Email to Acknowledge• Replying “ack” with some context makes it appear in

IRC too

Email Only Alerts• Do you care if RAID becomes degraded in the middle of

the night?

• Do you care if one of your web/hadoop/X boxes dies in the middle of the night?

the night?

• Do you care if one of your web/hadoop/X boxes dies in the middle of the night?

• Can it wait until the morning?

Added Context• Previous service state

• Duration in that state

• Previous service state

• Alert recipients

• Notes

• Link to Runbook

• Notes

• Link to Runbook

• Notes

• Link to runbook

Alert Storms• Reduce noise when 200 things go wrong by aggregating

• Trigger alert percentage of pool over threshold

Low friction downtime• IRC commands to downtime hosts/sets of hosts

Downtime Reminders• Help prevent false notifications

Event Handlers• Teach Nagios to augment the team

• Restarting services (nscd)

• Re-running jobs (transient errors)

• Duplicate crons (Chef)

Incremental Improvements?• Maybe

• More ideas; hoped they’d stick

• We didn’t know because we didn’t measure

Measure Everything• “You can’t manage what you can’t measure.”

- Deming (not really)

Measure Everything• “You can’t manage what you can’t measure.”

- Deming (not really)

• But, we weren’t measuring anything

What should we measure?

What should we measure?• Volume of alerts (total, by severity)

• Alert categorization (actionable vs not)

• Alert times: Off-hours?

• Noisy hosts/services

Opsweekly

@lozzd • @ryan_frantz We have data.

Aggregate alerts1. Look at reports

2. Wow, look at all those alerts for the same thing

3. Aggregate alerts

4.Profit

Parent relationships• Prevent alerts due to upstream issues (downed switch)

• Standard Nagios feature

• Computers can do this for us!

Parent relationships• signalvnoise.com

• LLDP on host shows switch info

• Put switch info into Chef using ohai

• Create Nagios host configs based on data

Service Dependencies• Hundreds of Graphite-sourced checks

• Created new template that sets a servicegroup that depends on the Graphite service.

Keep on analyzing• It’s okay to just identify and delete alerts that don’t

mean anything!

Keep on analyzing• It’s okay to just identify and delete alerts that don’t

mean anything!

• Or move them to email only

More Quantification!

Reviewing the Year• Use reports

• Use search

• Identify noisiest alerts

Reviewing the YearYEARLY REPORT SCREENSHOTS

• Great time to look at this data and make improvements

Nagios Hack Day/Week

• Great time to look at this data and make improvements

• If Disk Space is the worst. Can we rethink that?

Nagios Hack Day/Week

Outsource Your Alerts• Etsy’s Search Team has on-call rotation

• A whole subset of alerts that don’t go to Ops

• More teams starting this but Search Team is at 100%

Sleep Tracking

“Track your life!” - @ph

Did it work?

Did it work?• Yes.

• Signal to noise ratio is much better

• Okay, so it’s a little more complicated than that

• Adding alerts all the time means new “annoying” things

• Keep monitoring

What’s next?

• We focus on people’s sleep

The Effect of Sleep

• But not the effect on the person when they come to work the next day

The Effect of Sleep

• How do we measure the impact of sleep loss/deprivation?

The Effect of Sleep

• How do we measure the impact of sleep loss/deprivation?

The Effect of Sleep

• Subjective: Pittsburgh Sleepiness Scale

• Objective: Psychomotor vigilance task (PVT) to measure alertness

Beyond Opsweekly• Employee wellness program

• Security have started using past sleep data to check for weird logins to systems

More context: nagios-herald

More reports• We have a bunch of data, we can build better reports,

drill down to analyze alerting trends

More reports• We have a bunch of data, we can build better reports,

drill down to analyze alerting trends

• Can we attribute particular actions to reduced noise volume?

• Aggregate alerts

• Non-downtimed alerts

Thanks

Etsy Ops Team

SewMona

Open Source/Links• http://ryanfrantz.com/mtts

• https://github.com/etsy/opsweekly

• https://github.com/etsy/nagios-herald

• https://github.com/jonlives/jawboneup_to_graphite

• http://codeascraft.com

Questions?

Mean Time to Sleep: Quantifying the On-Call Experience

Engineering

Transcript of Mean Time to Sleep: Quantifying the On-Call Experience

Quantifying Mangrove

Quantifying mean inner potential of ZnO nanowires by off-axis ...

Ontario Public Service Guidance Document for Quantifying ......10. appendix d - survey sampling and statistical analysis 106 10.1population mean 106 10.2population total 108 10.3population

Mean deformation metrics for quantifying 3D cell matrix ... · PDF fileMean deformation metrics for quantifying 3D cell–matrix interactions without requiring information about matrix

Quantifying Innovation

BEHAVIORAL HEALTH ECHO MODULE 4: SLEEP DISORDERSApr 11, 2019 · , or a multiple sleep latency test showing a mean sleep latency less than or equal to 8 minutes and two or more sleep-onset

Sleep, Sleep, Sleep

Quantifying and Measuring Morphological Complexityclml.uchicago.edu/~max/wccfl-slides.pdfMotivation Quantifying Complexity Measuring Complexity: Morphology Measurements Summary Quantifying

Quantifying Reconnection*

SLEEP-RELATED LARYNGOSPASM - SciELO · SLEEP-RELATED LARYNGOSPASM FLAVIO S. ALOE*, MICHAEL J. THORPY** SUMMARY - Seven patients (mean age 46.6; range 33-58; 6M.1F) presented with

Lynne M. Bird A Randomized, Double ... - Il Sorriso Angelman · Sleep Onset Latency, min n LS mean change from baseline (SE) Total Sleep Time at Night, min n LS mean change from baseline

Irish Music & Dance Association Eanáir...in their sleep of sleep," you mean that they are very fast asleep indeed. Circle January 30 on your calendar for Gaeltacht Minnesota's annual

Quantifying mybikerides

CHRISTMAS CAROLS FROM POLAND Polish trad., transl. Douglas … · Sleep, sleep, sleep, sleep. Qui - et ass and Let theBabe sleepwhile Ma - ry rocks Him, Sleep, sleep, sleep, sleep.

Quantifying Extinctions

The association of sleep-disordered breathing with …...groups of sleep-disordered breathing (P = .40) or in the mean apnea/hypopnea index across groups of persons with none-to-mild

Quantifying fitness

СПИ, ІСУСЕ, СПИ SLEEP, JESUS, SLEEP...Спи серденько, спи. SLEEP, JESUS, SLEEP Sleep, Jesus, sleep Go to sleep I will rock you with songs to sleep Sleep, Jesus,

Sympathetic skin response in obstructive sleep apnea syndrome · OSAS group were compared with those from7 non- apneic heavy snorers whose mean age was 48.0 + 8.6 years, mean BMI-

New Relic Infrastructure Intro: Increasing Mean Time Between Loss of Sleep [FutureStack16]