KAHIBARO
Discord Login Register ☰
☰

28.9. Incident Response

Table of Contents

Why Incident Response Matters

Incidents are those moments when your backend system behaves in a way that hurts users or the business. Examples:

Incident response is the structured way you detect, handle, communicate, and learn from these events.

Good incident response does three things:

  1. Minimize impact on users and the business.
  2. Restore service quickly and safely.
  3. Produce learning so the same problem is less likely next time.

Important rule
Do not treat incidents as random bad luck. Treat them as signals about weaknesses in your system and process. Every significant incident must produce some form of improvement.

This chapter focuses on how to handle incidents in production, not on how to fully prevent them. Prevention and resilience are covered by other chapters like Monitoring Production Systems, Fault Tolerance, and High Availability.

What Is an Incident?

Incident vs Bug vs Outage

Example: A function calculates a discount incorrectly, but customers do not notice yet.

Example: Checkout requests begin failing with 500 errors and orders cannot be completed.

Example: Your API gateway is down and all requests fail.

You can think of it like this:

$$\text{Incident} = \text{Problem} + \text{User / Business Impact}$$

If there is no impact yet, you are dealing with a risk or a latent bug, not an incident.

Incident Severity

Not all incidents are equal. You need a severity scale to decide how urgently to respond and who to involve.

A common 4-level scale:

LevelNameTypical ImpactExample
SEV1CriticalMajor business function down, many users affected, urgent response neededCheckout API is down, no orders can be placed
SEV2HighImportant functionality broken, significant set of users affectedPayments work but are extremely slow or flaky
SEV3MediumDegraded experience, workarounds exist, limited user group affectedAdmin dashboard returns 500s for some queries
SEV4LowMinor issues, no immediate impact, cosmetic or background processes degradedDaily analytics job failed, no user-facing impact today

Severity rule
Always assign a severity at the beginning of an incident and upgrade it if you are unsure. It is safer to treat an issue as more serious and downgrade later.

For a beginner-friendly rule of thumb:

The Incident Response Lifecycle

Most organizations use a similar lifecycle:

  1. Detection
    You notice something is wrong. Often via alerts or monitoring.
  2. Triage
    You assess severity and decide whether this is an incident, and which type.
  3. Containment
    You stop things from getting worse. Sometimes you disable features or roll back.
  4. Mitigation / Remediation
    You fix or work around the root cause to restore service.
  5. Recovery
    You return systems to the normal, stable state and confirm they are healthy.
  6. Post‑incident review
    You analyze what happened and create follow‑up actions.

You can visualize it as a simple state machine:

$$
\text{Healthy} \rightarrow \text{Incident Detected} \rightarrow \text{Investigating} \rightarrow \text{Mitigated} \rightarrow \text{Learning}
$$

At beginner level, the key idea is: you follow a structured process every time, not random improvisation.

Detection and Alerting

You cannot respond to incidents you do not notice. Detection depends on monitoring and alerts.

Sources of Detection

Typical signals:

If you have good monitoring, most incidents should be detected by alerts before users report them.

Good Alert Design

You want alerts that are useful and actionable, not noisy. Examples:

Bad AlertGood Alert
"CPU > 80% for 5 minutes""Checkout API p95 latency > 2s for 5 minutes for more than 10% of traffic"
"Database connections > 80% used""Database 'orders' connection pool exhausted, 5% of requests failing"
"Error rate increased""500 error rate on POST /orders > 2% for 10 minutes"

Actionable alert rule
Every alert should have a clear owner and lead to a specific action. If no one knows what to do when an alert fires, you should not have that alert.

Triage and Classification

Once something suspicious is detected you perform triage:

  1. Is this a real incident or just noise?
    • A single 500 might not be an incident.
    • 10% of requests failing for 5 minutes likely is.
  2. What is the severity?
    • Use your severity table and decide SEV1 to SEV4.
  3. What is the scope?
    • Which services, endpoints, or regions are affected?
    • Is this affecting internal users, external users, or both?
  4. Who needs to be involved quickly?
    • On-call backend engineer.
    • Database specialist.
    • Security or networking person, if needed.

Example Triage Scenario

You see an alert:

"500 error rate for POST /orders is 12% for last 10 minutes."

You check logs and confirm:

Triage outcome:

Roles During an Incident

Even in small teams, it helps to separate roles so people are not doing everything at once.

Typical roles:

RoleResponsibility
Incident Commander (IC)Owns the response process, makes decisions, keeps track of progress, manages communication
Subject Matter ExpertsInvestigate technical causes, try fixes, gather data
Scribe / Note takerRecords timeline, actions taken, and important observations
Communications ownerUpdates stakeholders (support, product, management, maybe public status page)

In a very small team, one person might play several roles. But you should still think in roles:

Role rule
During a serious incident, avoid "everyone debugs everything". Assign an Incident Commander and let them coordinate so work is not duplicated and communication is clear.

First Steps: Stabilize and Contain

When you notice an incident, your first goal is not to find the deep root cause. Your first goal is to stop the bleeding.

Containment Strategies

Examples of containment actions:

Example

After a new version of the Orders API is deployed:

The safest containment step is:

You can then debug the new validation logic offline, in a staging environment.

Gathering Information Safely

After containment or in parallel, you need data to understand the incident. Key sources:

You should:

  1. Confirm when the problem started.
  2. Check what changed around that time.
  3. Identify which components are affected.

Do no harm rule
During an incident, be careful with experiments. Never run dangerous scripts or schema changes directly in production as a "test". Always ask, "Could this make things worse?"

Fixing the Incident

After containment, you move to mitigation and remediation.

Example: Temporarily disable a heavy report feature to reduce database load.

Example: Fixing a missing database index that caused timeouts.

Example: Database Connection Exhaustion

Symptoms:

Mitigation:

  1. Increase connection pool size slightly, if safe.
  2. Scale out application servers carefully if they are under capacity.
  3. Apply rate limiting on the heaviest endpoints.

Remediation (after the fire is under control):

  1. Identify endpoints that open too many connections or leak connections.
  2. Add or fix connection pooling in the ORM or driver.
  3. Review and reduce long-running queries.
  4. Add or improve database indexes.

Communication During Incidents

Technical work is only half of incident response. Communication is the other half.

Internal Communication

You need to keep:

Good internal updates have:

Avoid:

External Communication

For user-facing incidents, you might:

Even a beginner team can write short, clear messages:

We are currently experiencing elevated error rates for order creation affecting some users in the EU region. Our team is investigating and working on a fix. We will provide an update in 30 minutes.

Later, after mitigation:

We have rolled back a recent change and error rates have returned to normal. We are monitoring the system closely and will publish a detailed summary after our investigation.

Post‑Incident Review (Postmortem)

The most valuable part of incident response is what you do afterward.

A post‑incident review (often called a postmortem) is a structured document and meeting where you:

  1. Reconstruct the timeline
    • When did it start?
    • When was it detected?
    • What actions were taken and when?
  2. Explain technical cause
    • What precisely failed?
    • Why did this cause the specific symptoms?
  3. Understand impact
    • Which users or systems were affected?
    • How many requests failed?
    • How long did it last?
  4. Analyze the response
    • What went well?
    • What delayed detection or response?
    • Were alerts and runbooks helpful?
  5. Decide improvements
    • Code or configuration changes.
    • Monitoring and alerting improvements.
    • Process changes or training.
    • Documentation or runbook updates.

Blameless rule
Post‑incident reviews must be blameless. Focus on systems, processes, and incentives, not on blaming individuals. This is essential for honest learning.

Example Post‑Incident Questions

Runbooks and Playbooks

A runbook is a step‑by‑step guide for handling a known incident type.

For example, a runbook for "Database connection exhaustion" might include:

  1. Check database connections metric.
  2. Check application instance count and traffic pattern.
  3. Temporarily reduce heavy batch jobs.
  4. If needed, increase pool size by a small, predefined amount.
  5. Notify database owner if pool has been increased.
  6. After the incident, schedule index/optimization review.

Runbooks help:

You can have runbooks for:

For each runbook, include:

FieldDescription
NameShort name of the incident type
SymptomsWhat you see in alerts, dashboards, logs
Quick checksFast checks to confirm or rule out this problem
Safe actionsLow‑risk steps for immediate mitigation
EscalationWho to contact if safe actions do not work
Follow‑upItems to check after the incident ends

Coordinating with Other Production Practices

Incident response interacts with several other production topics from this course:

Incident response does not replace these topics. It is what you do when all your protective mechanisms are not enough.

Building an Incident Response Culture

Even small backend teams can establish a minimal incident response culture:

  1. Define what "incident" means
    For example, any issue that impacts users for more than 5 minutes or blocks purchases is an incident.
  2. Define severity levels
    Create a simple SEV1 to SEV4 table that fits your product.
  3. Define ownership
    Decide who is on‑call or who is responsible for responding to which systems.
  4. Improve after every incident
    Always produce at least one improvement, such as:
    • Better logs.
    • A new or improved alert.
    • A small code change.
    • A new runbook step.
  5. Practice
    Occasionally simulate incidents in a test environment, for example:
    • "The database is down, what do we do?"
    • "The payments provider is timing out, how do we handle this?"

Over time, your incident response will become faster, more confident, and more predictable, which is exactly what a production backend needs.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3512 / 24h
Impressum / Imprint
Datenschutz / Data Policy