Incidents

What is an incident?

Definitions vary, but it’s important to agree on one. Examples include:

“Any unplanned disruption or degradation of service that is actively affecting customers ability to use [us].”

“An incident is anything that takes you away from planned work with a degree of urgency.”

I definitely don’t like definitions that are too narrow, e.g. “an outage of our service” or similar. There are lots of situations where you might want coordinated response and communication and post-hoc analysis and writeup.

Process

My two go-to references for good incident response process are:

• The PagerDuty Incident Response Documentation

• The incident.io guide to incident management

Other process guides:

• Incident Response at Heroku

• Atlassian Incident Management Handbook

• How To Establish a High Severity Incident Management Program

Post-Incident

• It’s not a Post Mortem if nobody died

• Incident Write-ups They Want to Read

  • “Focus on narrative, not metadata”

  • “Support your readers”

  • “Be visual”

  • “Don’t be afraid of analysis”

• Some Observations On the Messy Realities of Incident Reviews – Adaptive Capacity Labs

• Incident Review and Postmortem Best Practices - by Gergely Orosz - The Pragmatic Engineer

• Howie: The Post-Incident Guide

Assorted Notes

• Against Incident Severities and in Favor of Incident Types | Honeycomb

• Assembly time is where you have the most control of an incident

• 3 mistakes I’ve made at the beginning of an incident (and how not to make them) | FireHydrant

• Running More Low-Severity Incidents Is Improving Our Culture – The New Stack

• There Is No Shame in Customer-Reported Incidents - The New Stack

• Keep Calm and Respond: A Beginner’s Heuristic to Incident Response - DZone DevOps

• Incident benchmark report | FireHydrant

• Incident travel time - by Robert Ross - The Thought Drop

• How We Manage Incident Response at Honeycomb - The New Stack

• Align platform and product engineering teams over incidents | FireHydrant

• A guide to Incident Command | by Jonathan Word | Medium

• Your guide to better incident status pages | FireHydrant

Incident Writeups

Some of the below are here because the incident itself was interesting, some because the writeup is particularly insightful.

• High Scalability - High Scalability - Troubles with Sharding - What can we learn from the Foursquare Incident?

• Twilio incident and Redis

• Incident report on memory leak caused by Cloudflare parser bug

Misc

• Wolf Incident Postmortem - LessWrong - I don’t like how “post mortem” gets commonly used for non-fatal incident reviews/reports in the tech industry, but it makes sense here(!)

• Ransomware incidents now make up majority of British government’s crisis management COBRA meetings - The Record by Recorded Future