Incident Management Process Design

Incident Severity Classification

Define severity levels before incidents occur so the response is proportional to the impact. A common framework uses four levels: SEV1 (critical -- service is down for all users), SEV2 (major -- significant feature is unavailable or degraded), SEV3 (minor -- limited impact on a subset of users), and SEV4 (low -- cosmetic issue or minor inconvenience). Each severity level should have defined response time expectations, escalation procedures, and communication requirements.

The classification should be based on user impact, not technical cause. A database failover that completes automatically in 30 seconds might technically be a complex event but is a non-incident from the user perspective. A CSS change that breaks the checkout button is technically trivial but is a SEV1 because it blocks revenue. Classify based on what users experience, not on what engineers find technically interesting or complex.

PagerDuty's incident response framework recommends that only SEV1 and SEV2 incidents trigger the full incident response process. SEV3 and SEV4 issues should be handled through normal engineering workflows -- bug tickets, sprint planning, and regular release cycles. Applying the full incident process to minor issues creates alert fatigue and devalues the severity system. Reserve the formal process for incidents that genuinely require coordinated, urgent response.

The Incident Response Workflow

The incident response workflow has five phases: detection, triage, mitigation, resolution, and review. Detection can be automated (monitoring alerts) or manual (customer reports, team member observation). The faster detection happens, the shorter the total incident duration. Invest in monitoring that catches issues before customers report them -- a 2024 Datadog study found that organizations with automated incident detection resolved issues 40% faster than those relying on customer reports.

Triage assigns an incident commander, determines severity, and establishes a communication channel. The incident commander coordinates the response, keeps stakeholders informed, and makes decisions about mitigation strategy. This role should not be the person debugging the technical issue -- the incident commander manages the process while responders focus on the problem. Separating these roles prevents the person deepest in the technical details from losing track of communication and coordination.

Mitigation takes priority over root cause analysis during an active incident. The goal is to restore service as quickly as possible, even if the fix is temporary. Rolling back a deployment, restarting a service, failing over to a secondary system, or enabling a circuit breaker are mitigation actions that restore service without understanding the root cause. Root cause analysis happens in the postmortem after the incident is resolved and the team is no longer under time pressure.

Communication During Incidents

Internal communication should flow through a dedicated incident channel -- a Slack channel or a similar space created for each SEV1/SEV2 incident. All investigation, decisions, and status updates happen in this channel, creating a real-time record that is invaluable for the postmortem. Keep the channel focused: general discussion happens elsewhere, and the incident channel is reserved for information directly relevant to resolution.

External communication -- status page updates, customer notifications, support team briefings -- should happen within 15 minutes of incident declaration for SEV1 and within 30 minutes for SEV2. Customers tolerate outages better when they are informed promptly and honestly. A status page that says 'We are investigating increased error rates and will update within 30 minutes' is vastly better than silence that forces customers to speculate and flood the support team with tickets.

Avoid blame in incident communications. Saying 'A code change caused the outage' is factual and useful. Saying 'An engineer pushed bad code' is blaming and counterproductive. Blameless communication during the incident sets the tone for the blameless postmortem that follows. Etsy's blameless postmortem culture, documented by John Allspaw, demonstrates that psychological safety during incident response improves both the quality of the response and the quality of the subsequent analysis.

Blameless Postmortems That Drive Improvement

A postmortem should answer four questions: What happened? Why did it happen? How was it detected and resolved? What will we do to prevent recurrence? The answers must be specific. 'Improve monitoring' is not an action item. 'Add an alert on payment processing error rate exceeding 1% with a 5-minute evaluation window, assigned to the payments team, due by Marzo 15' is an action item. Vague action items are never completed; specific ones are tracked and verified.

Use the timeline as the postmortem's backbone. Document every significant event with timestamps: when the incident started, when it was detected, when the incident commander was engaged, what mitigation actions were taken and when, when service was restored. The timeline reveals gaps -- a 45-minute delay between detection and response, a 2-hour debugging session that could have been shortened with better tooling -- that become improvement opportunities.

Conduct postmortems within 48 hours of incident resolution while details are fresh. Invite everyone involved in the response plus stakeholders who were affected. The postmortem meeting should review the written document rather than create it from scratch -- the document author (typically the incident commander) prepares the draft and the meeting focuses on filling gaps, correcting inaccuracies, and prioritizing action items. Google's SRE book provides a detailed postmortem template that has been widely adopted across the industry.

Building Incident Readiness

Game days -- simulated incidents run during business hours -- build incident response muscle memory. Inject failures into the staging or production environment and run the full incident response process. Netflix's Chaos Monkey, which randomly terminates production instances, is the most famous example, but simpler approaches work for smaller teams: disable a database replica, introduce artificial latency on an API endpoint, or revoke a service's credentials. The goal is to practice the response process under controlled conditions.

Runbooks reduce diagnosis time by documenting common incident scenarios and their resolution steps. A runbook for 'database connection pool exhausted' should include: how to verify the diagnosis, how to identify the root cause (connection leak, traffic spike, slow queries), short-term mitigation steps (restart the application, increase pool size), and long-term fixes (connection timeout configuration, query optimization). Maintain runbooks in a searchable location accessible to on-call engineers at 3 AM.

Review incident metrics monthly: mean time to detection (MTTD), mean time to mitigation (MTTM), mean time to resolution (MTTR), and incident frequency by severity. Track these metrics over rolling quarters to identify trends. A decreasing MTTD indicates improving monitoring. A stable MTTR with increasing incident frequency indicates that the team is getting faster at response but not preventing recurrence. Use these metrics to allocate improvement effort where it will have the most impact on reliability.