Production Incident
Overview
This playbook covers the response to a live production incident affecting users or business operations.
Why It Matters
The response quality matters as much as the fix. Good incident handling reduces impact and future confusion.
Core Concepts
- Stabilize first.
- Assign ownership.
- Verify the mitigation.
Mental Models
During an incident, the first job is to stop the bleeding, not to solve every long-term problem.
Best Practices
- Announce the incident.
- Track actions in one place.
- Roll back or mitigate if possible.
Common Mistakes
- Investigating without stabilizing.
- Losing track of attempted fixes.
- Failing to communicate status.
Trade-offs
Fast mitigation may not be the final root-cause fix, but it protects users while the deeper work continues.
Decision Framework
flowchart TD
A[Incident] --> B[Stabilize]
B --> C[Communicate]
C --> D[Diagnose]
D --> E[Mitigate]
E --> F[Verify]
F --> G[Follow up]Examples
- Roll back a bad deploy, then investigate why it passed checks.
Checklists
- Has user impact been assessed?
- Is the team aligned on the mitigation?
- Have we verified recovery?
Senior Engineer Notes
Senior engineers stay calm, organized, and evidence-driven. The incident response becomes part of the team’s reputation.