SlideShare ist ein Scribd-Unternehmen logo
1 von 56
Downloaden Sie, um offline zu lesen
Relaxing picture of Yoga
PagerDuty Alert
Ignite (10m)   how to not burn out your monitoring team
hunt through logs for 2
hours
Ignite (10m)   how to not burn out your monitoring team
Ignite (10m)   how to not burn out your monitoring team
Ignite (10m)   how to not burn out your monitoring team
How to not burn out your
production team
Gil Zellner
(CloudifyDev at Gigaspaces)
Twitter: @Heathenaspargus
Who am I?
Now:
Past:
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
cost of hiring new employee is 1.5-3x their monthly
salary
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
Next day
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
frustration - I am unable to complete my task
@Heathenaspargus
Ignite (10m)   how to not burn out your monitoring team
Time spent inefficiently
@Heathenaspargus
Repetitive tasks
@Heathenaspargus
Working Alone
@Heathenaspargus
Yak
Shaving
@Heathenaspargus
https://www.ergoflex.co.uk/blog/category/sleep-research/sleeponomics-could-sleep-deprivation-be-the-real-reason-politicians-make-bad-decisions
@Heathenaspargus
Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus
Mandatory Half day-off after night production issue
@Heathenaspargus
Allocate weekly time to resolve or automate issues that kept us up at night
@Heathenaspargus
Wider rotation (more people do on-call)
@Heathenaspargus
Knowledge Matrix
Deploy System Mobile Link Backend
Gil V V
Karen V V
Ari V V
@Heathenaspargus
@Heathenaspargus
Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus
Ignite (10m)   how to not burn out your monitoring team
Ignite (10m)   how to not burn out your monitoring team
@Heathenaspargus
solution: alert only things that meet the following
criteria:
1) Alert on symptoms, not suspected "causes"
2) Actionable
3) Business breaking
@Heathenaspargus
Alerte générale!
@Heathenaspargus
Solution: direct alerts to relevant parties
@Heathenaspargus
@Heathenaspargus
Ignite (10m)   how to not burn out your monitoring team
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
What are your KPIs ?
@Heathenaspargus
@Heathenaspargus
Netflix stream starts per second
@Heathenaspargus
Picking how to measure things
Ignite (10m)   how to not burn out your monitoring team
Diagnosis
@Heathenaspargus
Make heal script
@Heathenaspargus
@Heathenaspargus
@Heathenaspargus
Facebook Auto Remediation
https://www.facebook.com/notes/facebook-engineering/making-facebook-self-healing/10150275248698920
@Heathenaspargus
Easy (days) Intermediate
(months)
Hard (years)
- no changes to
infrastructure
- just policy
- Small changes
to apps
- logging
- light
automation
- Design for
better
operability
- long term
@Heathenaspargus
Ignite (10m)   how to not burn out your monitoring team
@Heathenaspargus
Bad artists copy, great artists steal
email:
Gil.Zellner@gmail.com
Twitter:
@Heathenaspargus

Weitere ähnliche Inhalte

Andere mochten auch

Become a Learning Masterchef
Become a Learning MasterchefBecome a Learning Masterchef
Become a Learning MasterchefKineoPacific
 
Social Media Report - Big 5 Canadian Banks September - October 2016
Social Media Report - Big 5 Canadian Banks September - October 2016Social Media Report - Big 5 Canadian Banks September - October 2016
Social Media Report - Big 5 Canadian Banks September - October 2016Unmetric
 
Getting Started - Console Program and Problem Solving
Getting Started - Console Program and Problem SolvingGetting Started - Console Program and Problem Solving
Getting Started - Console Program and Problem SolvingHock Leng PUAH
 
How to Kick Ass on Google When You're All Out of Bubblegum
How to Kick Ass on Google When You're All Out of BubblegumHow to Kick Ass on Google When You're All Out of Bubblegum
How to Kick Ass on Google When You're All Out of BubblegumGreg Gifford
 
Борьба с биологическими инвазиями.
Борьба с биологическими инвазиями.Борьба с биологическими инвазиями.
Борьба с биологическими инвазиями.kulibin
 
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...Marcel Lebrun
 
Common Mistakes Travelers Make
Common Mistakes Travelers MakeCommon Mistakes Travelers Make
Common Mistakes Travelers MakeVisitorsCoverage
 
Stepping into Internet-Virtual Worlds and Future of Value Creation
Stepping into Internet-Virtual Worlds and Future of Value CreationStepping into Internet-Virtual Worlds and Future of Value Creation
Stepping into Internet-Virtual Worlds and Future of Value CreationRobin Teigland
 
How to deal with bitter criticism
How to deal with bitter criticismHow to deal with bitter criticism
How to deal with bitter criticismXenia Y
 
SLA School Librarian of the Year 2012
SLA School Librarian of the Year 2012SLA School Librarian of the Year 2012
SLA School Librarian of the Year 2012Candy Gourlay
 
Présentation Xebicon15 - Akamis - Marketing & IT
Présentation Xebicon15 -  Akamis - Marketing & ITPrésentation Xebicon15 -  Akamis - Marketing & IT
Présentation Xebicon15 - Akamis - Marketing & ITAKAMIS
 
Accelerated Startup - #Idea-to-IPO
Accelerated Startup - #Idea-to-IPOAccelerated Startup - #Idea-to-IPO
Accelerated Startup - #Idea-to-IPOVitaly Golomb
 
Desempeño del ingeniero industrial en la empresa centrales1
Desempeño del ingeniero industrial en la empresa centrales1Desempeño del ingeniero industrial en la empresa centrales1
Desempeño del ingeniero industrial en la empresa centrales1Annabel Lee
 
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...Kapost
 
Time is Life. Life is Time
Time is Life. Life is TimeTime is Life. Life is Time
Time is Life. Life is TimeShirley Williams
 
Centros De Responsabilidad Intra (ExposicióN)
Centros De Responsabilidad Intra (ExposicióN)Centros De Responsabilidad Intra (ExposicióN)
Centros De Responsabilidad Intra (ExposicióN)Julio Nuñez
 
The Science of Webinars
The Science of WebinarsThe Science of Webinars
The Science of WebinarsHubSpot
 
23 frases de liderazgo para inspirarte
23 frases de liderazgo para inspirarte23 frases de liderazgo para inspirarte
23 frases de liderazgo para inspirarteAna Sek
 

Andere mochten auch (20)

Become a Learning Masterchef
Become a Learning MasterchefBecome a Learning Masterchef
Become a Learning Masterchef
 
Social Media Report - Big 5 Canadian Banks September - October 2016
Social Media Report - Big 5 Canadian Banks September - October 2016Social Media Report - Big 5 Canadian Banks September - October 2016
Social Media Report - Big 5 Canadian Banks September - October 2016
 
Getting Started - Console Program and Problem Solving
Getting Started - Console Program and Problem SolvingGetting Started - Console Program and Problem Solving
Getting Started - Console Program and Problem Solving
 
How to Kick Ass on Google When You're All Out of Bubblegum
How to Kick Ass on Google When You're All Out of BubblegumHow to Kick Ass on Google When You're All Out of Bubblegum
How to Kick Ass on Google When You're All Out of Bubblegum
 
Борьба с биологическими инвазиями.
Борьба с биологическими инвазиями.Борьба с биологическими инвазиями.
Борьба с биологическими инвазиями.
 
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
Classes inversées, classes "à l'envers" ? Et si c’était simplement Enseigner ...
 
Common Mistakes Travelers Make
Common Mistakes Travelers MakeCommon Mistakes Travelers Make
Common Mistakes Travelers Make
 
Stepping into Internet-Virtual Worlds and Future of Value Creation
Stepping into Internet-Virtual Worlds and Future of Value CreationStepping into Internet-Virtual Worlds and Future of Value Creation
Stepping into Internet-Virtual Worlds and Future of Value Creation
 
How to deal with bitter criticism
How to deal with bitter criticismHow to deal with bitter criticism
How to deal with bitter criticism
 
SLA School Librarian of the Year 2012
SLA School Librarian of the Year 2012SLA School Librarian of the Year 2012
SLA School Librarian of the Year 2012
 
Présentation Xebicon15 - Akamis - Marketing & IT
Présentation Xebicon15 -  Akamis - Marketing & ITPrésentation Xebicon15 -  Akamis - Marketing & IT
Présentation Xebicon15 - Akamis - Marketing & IT
 
Accelerated Startup - #Idea-to-IPO
Accelerated Startup - #Idea-to-IPOAccelerated Startup - #Idea-to-IPO
Accelerated Startup - #Idea-to-IPO
 
Desempeño del ingeniero industrial en la empresa centrales1
Desempeño del ingeniero industrial en la empresa centrales1Desempeño del ingeniero industrial en la empresa centrales1
Desempeño del ingeniero industrial en la empresa centrales1
 
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
The Content Marketing Platform: Why Kapost is Critical to Content Marketing S...
 
Time is Life. Life is Time
Time is Life. Life is TimeTime is Life. Life is Time
Time is Life. Life is Time
 
Centros De Responsabilidad Intra (ExposicióN)
Centros De Responsabilidad Intra (ExposicióN)Centros De Responsabilidad Intra (ExposicióN)
Centros De Responsabilidad Intra (ExposicióN)
 
The Science of Webinars
The Science of WebinarsThe Science of Webinars
The Science of Webinars
 
KZero Radar chart Q1 2013
KZero Radar chart Q1 2013KZero Radar chart Q1 2013
KZero Radar chart Q1 2013
 
Disc
DiscDisc
Disc
 
23 frases de liderazgo para inspirarte
23 frases de liderazgo para inspirarte23 frases de liderazgo para inspirarte
23 frases de liderazgo para inspirarte
 

Mehr von Gil Zellner

Lessons in video games that have prepared me for a life in engineering
Lessons in video games that have prepared me for a life in engineeringLessons in video games that have prepared me for a life in engineering
Lessons in video games that have prepared me for a life in engineeringGil Zellner
 
5m the unsung glory of internal tools (1)
5m   the unsung glory of internal tools (1)5m   the unsung glory of internal tools (1)
5m the unsung glory of internal tools (1)Gil Zellner
 
Full talk from ops to dev and back again
Full talk   from ops to dev and back againFull talk   from ops to dev and back again
Full talk from ops to dev and back againGil Zellner
 
How to get your engineers to care about the AWS Bill
How to get your engineers to care about the AWS BillHow to get your engineers to care about the AWS Bill
How to get your engineers to care about the AWS BillGil Zellner
 
Rinse and repeat
Rinse and repeatRinse and repeat
Rinse and repeatGil Zellner
 
A newborn as a production system
A newborn as a production systemA newborn as a production system
A newborn as a production systemGil Zellner
 
How to do monitoring that won't make your engineers quit
How to do monitoring that won't make your engineers quitHow to do monitoring that won't make your engineers quit
How to do monitoring that won't make your engineers quitGil Zellner
 

Mehr von Gil Zellner (8)

HA coffee
HA coffeeHA coffee
HA coffee
 
Lessons in video games that have prepared me for a life in engineering
Lessons in video games that have prepared me for a life in engineeringLessons in video games that have prepared me for a life in engineering
Lessons in video games that have prepared me for a life in engineering
 
5m the unsung glory of internal tools (1)
5m   the unsung glory of internal tools (1)5m   the unsung glory of internal tools (1)
5m the unsung glory of internal tools (1)
 
Full talk from ops to dev and back again
Full talk   from ops to dev and back againFull talk   from ops to dev and back again
Full talk from ops to dev and back again
 
How to get your engineers to care about the AWS Bill
How to get your engineers to care about the AWS BillHow to get your engineers to care about the AWS Bill
How to get your engineers to care about the AWS Bill
 
Rinse and repeat
Rinse and repeatRinse and repeat
Rinse and repeat
 
A newborn as a production system
A newborn as a production systemA newborn as a production system
A newborn as a production system
 
How to do monitoring that won't make your engineers quit
How to do monitoring that won't make your engineers quitHow to do monitoring that won't make your engineers quit
How to do monitoring that won't make your engineers quit
 

Kürzlich hochgeladen

A Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationA Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationMohsinKhanA
 
Landsman converter for power factor improvement
Landsman converter for power factor improvementLandsman converter for power factor improvement
Landsman converter for power factor improvementVijayMuni2
 
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
Engineering Mechanics  Chapter 5  Equilibrium of a Rigid BodyEngineering Mechanics  Chapter 5  Equilibrium of a Rigid Body
Engineering Mechanics Chapter 5 Equilibrium of a Rigid BodyAhmadHajasad2
 
Guardians and Glitches: Navigating the Duality of Gen AI in AppSec
Guardians and Glitches: Navigating the Duality of Gen AI in AppSecGuardians and Glitches: Navigating the Duality of Gen AI in AppSec
Guardians and Glitches: Navigating the Duality of Gen AI in AppSecTrupti Shiralkar, CISSP
 
Lecture 1: Basics of trigonometry (surveying)
Lecture 1: Basics of trigonometry (surveying)Lecture 1: Basics of trigonometry (surveying)
Lecture 1: Basics of trigonometry (surveying)Bahzad5
 
Popular-NO1 Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialis...
Popular-NO1 Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialis...Popular-NO1 Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialis...
Popular-NO1 Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialis...Amil baba
 
Basic Principle of Electrochemical Sensor
Basic Principle of  Electrochemical SensorBasic Principle of  Electrochemical Sensor
Basic Principle of Electrochemical SensorTanvir Moin
 
How to Write a Good Scientific Paper.pdf
How to Write a Good Scientific Paper.pdfHow to Write a Good Scientific Paper.pdf
How to Write a Good Scientific Paper.pdfRedhwan Qasem Shaddad
 
Power System electrical and electronics .pptx
Power System electrical and electronics .pptxPower System electrical and electronics .pptx
Power System electrical and electronics .pptxMUKULKUMAR210
 
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...amrabdallah9
 
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdfSummer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdfNaveenVerma126
 
Multicomponent Spiral Wound Membrane Separation Model.pdf
Multicomponent Spiral Wound Membrane Separation Model.pdfMulticomponent Spiral Wound Membrane Separation Model.pdf
Multicomponent Spiral Wound Membrane Separation Model.pdfGiovanaGhasary1
 
me3493 manufacturing technology unit 1 Part A
me3493 manufacturing technology unit 1 Part Ame3493 manufacturing technology unit 1 Part A
me3493 manufacturing technology unit 1 Part Akarthi keyan
 
cloud computing notes for anna university syllabus
cloud computing notes for anna university syllabuscloud computing notes for anna university syllabus
cloud computing notes for anna university syllabusViolet Violet
 
SATELITE COMMUNICATION UNIT 1 CEC352 REGULATION 2021 PPT BASICS OF SATELITE ....
SATELITE COMMUNICATION UNIT 1 CEC352 REGULATION 2021 PPT BASICS OF SATELITE ....SATELITE COMMUNICATION UNIT 1 CEC352 REGULATION 2021 PPT BASICS OF SATELITE ....
SATELITE COMMUNICATION UNIT 1 CEC352 REGULATION 2021 PPT BASICS OF SATELITE ....santhyamuthu1
 
Modelling Guide for Timber Structures - FPInnovations
Modelling Guide for Timber Structures - FPInnovationsModelling Guide for Timber Structures - FPInnovations
Modelling Guide for Timber Structures - FPInnovationsYusuf Yıldız
 
Dev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingDev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingMarian Marinov
 

Kürzlich hochgeladen (20)

Litature Review: Research Paper work for Engineering
Litature Review: Research Paper work for EngineeringLitature Review: Research Paper work for Engineering
Litature Review: Research Paper work for Engineering
 
A Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software SimulationA Seminar on Electric Vehicle Software Simulation
A Seminar on Electric Vehicle Software Simulation
 
Landsman converter for power factor improvement
Landsman converter for power factor improvementLandsman converter for power factor improvement
Landsman converter for power factor improvement
 
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
Engineering Mechanics  Chapter 5  Equilibrium of a Rigid BodyEngineering Mechanics  Chapter 5  Equilibrium of a Rigid Body
Engineering Mechanics Chapter 5 Equilibrium of a Rigid Body
 
Guardians and Glitches: Navigating the Duality of Gen AI in AppSec
Guardians and Glitches: Navigating the Duality of Gen AI in AppSecGuardians and Glitches: Navigating the Duality of Gen AI in AppSec
Guardians and Glitches: Navigating the Duality of Gen AI in AppSec
 
Lecture 1: Basics of trigonometry (surveying)
Lecture 1: Basics of trigonometry (surveying)Lecture 1: Basics of trigonometry (surveying)
Lecture 1: Basics of trigonometry (surveying)
 
Popular-NO1 Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialis...
Popular-NO1 Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialis...Popular-NO1 Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialis...
Popular-NO1 Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialis...
 
Basic Principle of Electrochemical Sensor
Basic Principle of  Electrochemical SensorBasic Principle of  Electrochemical Sensor
Basic Principle of Electrochemical Sensor
 
How to Write a Good Scientific Paper.pdf
How to Write a Good Scientific Paper.pdfHow to Write a Good Scientific Paper.pdf
How to Write a Good Scientific Paper.pdf
 
Power System electrical and electronics .pptx
Power System electrical and electronics .pptxPower System electrical and electronics .pptx
Power System electrical and electronics .pptx
 
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
Strategies of Urban Morphologyfor Improving Outdoor Thermal Comfort and Susta...
 
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdfSummer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
Summer training report on BUILDING CONSTRUCTION for DIPLOMA Students.pdf
 
Multicomponent Spiral Wound Membrane Separation Model.pdf
Multicomponent Spiral Wound Membrane Separation Model.pdfMulticomponent Spiral Wound Membrane Separation Model.pdf
Multicomponent Spiral Wound Membrane Separation Model.pdf
 
me3493 manufacturing technology unit 1 Part A
me3493 manufacturing technology unit 1 Part Ame3493 manufacturing technology unit 1 Part A
me3493 manufacturing technology unit 1 Part A
 
Présentation IIRB 2024 Marine Cordonnier.pdf
Présentation IIRB 2024 Marine Cordonnier.pdfPrésentation IIRB 2024 Marine Cordonnier.pdf
Présentation IIRB 2024 Marine Cordonnier.pdf
 
cloud computing notes for anna university syllabus
cloud computing notes for anna university syllabuscloud computing notes for anna university syllabus
cloud computing notes for anna university syllabus
 
Présentation IIRB 2024 Chloe Dufrane.pdf
Présentation IIRB 2024 Chloe Dufrane.pdfPrésentation IIRB 2024 Chloe Dufrane.pdf
Présentation IIRB 2024 Chloe Dufrane.pdf
 
SATELITE COMMUNICATION UNIT 1 CEC352 REGULATION 2021 PPT BASICS OF SATELITE ....
SATELITE COMMUNICATION UNIT 1 CEC352 REGULATION 2021 PPT BASICS OF SATELITE ....SATELITE COMMUNICATION UNIT 1 CEC352 REGULATION 2021 PPT BASICS OF SATELITE ....
SATELITE COMMUNICATION UNIT 1 CEC352 REGULATION 2021 PPT BASICS OF SATELITE ....
 
Modelling Guide for Timber Structures - FPInnovations
Modelling Guide for Timber Structures - FPInnovationsModelling Guide for Timber Structures - FPInnovations
Modelling Guide for Timber Structures - FPInnovations
 
Dev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingDev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & Logging
 

Ignite (10m) how to not burn out your monitoring team

Hinweis der Redaktion

  1. before we begin, lets all relax a bit, dim the lights
  2. Pagerduty Alert - add alert
  3. What is this nonsense ?
  4. Find something not worth waking me up over.
  5. Find some app sending nonsense errors
  6. Why: lower capacity when starting equipment etc. External cost to other engineers
  7. 1st on-call
  8. 10th on call if nobody is dying leave me alone
  9. Why mandatory: people don’t allocate rest enough for themselves. A person would simply accumulate more vacation days.
  10. Dedicated time in the week to prevent things from waking us up at night. Simple fixes, even things like a jenkins job to do reboots of components. At similarWeb, production engineers spend their on-call week only working on ease-of-operation related issue.
  11. Alert all the things! Why this is bad: boy who cried wolf syndrome, alert fatigue
  12. Attention is a depletable resource
  13. Alert everyone! why this is bad tragedy of the commons: everyone ends up turning off their phones thinking “someone else will take of this”
  14. only people who can fix the problem get alerted, others get emails system needs to be smart enough to make the choice, and fixed when it makes a mistake in waking up the wrong person
  15. Log nothing Log to /dev/null Log to ephemeral storage
  16. measuring the wrong things. hosts are irrelevant, so is cpu and memory.
  17. mention multi AZ
  18. Tweet at me! Tell me your monitoring horror stories!