20.11. Alerting

Why Alerting Matters

Monitoring tells you what is happening. Alerting tells someone that action is needed.

Alerting connects your system metrics, logs, and checks to humans or automated responders. Without alerting, you only find incidents when a user complains or when you look at dashboards manually.

Good alerting helps you:

Detect problems quickly.
Notify the right people.
Respond before users are heavily impacted.
Avoid burnout from constant noisy alerts.

Important: A monitoring system without alerting is almost useless in production. A bad alerting system can be worse than none, because it causes alert fatigue and real incidents get ignored.

Types of Alerts

You can think of alerts in several categories. This helps you design them more clearly.

By Source

Source	Examples	Tools (typical)
Metrics	CPU, memory, request rate, error rate	Prometheus + Alertmanager
Logs	Specific error patterns, spikes in exceptions	Loki, Elasticsearch, Datadog Logs
Traces	Slow spans, failing services	Jaeger, Tempo, X-Ray, etc.
Synthetic	Health check endpoints from outside	UptimeRobot, Pingdom, custom

Metrics-based alerts are usually the core of backend alerting. Log- and trace-based alerts are often used for specific conditions or debugging recurring failures.

By Purpose

Purpose	Description	Example
Availability alerts	Service is down or unusable	Health check failing for 5 minutes
Performance alerts	Service is slow, but still works	p95 latency exceeds 500 ms for 10 minutes
Error-rate alerts	Too many errors compared to normal	5xx rate > 2% of all responses for 10 minutes
Capacity / saturation	Running out of resources	DB connections > 90% of pool size for 15 minutes
Business / product alerts	Business KPI problems, fraud detection	Checkout success rate drops below 80%
Security alerts	Suspicious or malicious behavior	Many failed logins from same IP

By Urgency

You cannot treat all alerts the same. If you do, your on-call engineers will burn out.

Typical severities:

Level	Meaning	Example
Critical	Immediate user impact, wake someone up	API completely down, payments failing globally
High	Serious, fix during working hours	Error rate increased, but system still mostly usable
Medium	Needs investigation soon	Background worker queue growing unusually fast
Low	Observation or suggestion	Disk usage > 70%, plan to increase volume size

Rule: Only Critical alerts should wake people at night. Everything else should be routed as lower-priority notifications.

Basic Alerting Concepts

To work with alerting tools, you should understand some common concepts.

Alert Rules

An alert rule is a condition plus some metadata about what to do.

Example in human language:

Trigger "High error rate" alert when 5xx responses are more than 5% of total responses for 10 minutes.

Common fields in an alert rule:

Field	Description
Name	Human-readable name of the alert
Expression	The condition, often based on metrics
Threshold	The value that indicates a problem
Duration	How long the condition must be true
Severity	How urgent it is
Labels/tags	Info for routing (service, team, environment)
Description	What the alert means and how to handle it

In Prometheus, for example, the condition is a PromQL expression like:

text

rate(http_requests_total{status=~"5.."}[5m]) 
/
rate(http_requests_total[5m]) 
> 0.05

You usually also specify a for: 10m duration so it only fires if the condition is true for 10 minutes.

Thresholds and SLOs

A threshold is the value where you decide something is too bad to tolerate.

Instead of picking random thresholds, production teams often define SLOs (Service Level Objectives), for example:

99.9% of requests should succeed.
p95 latency for GET /api/orders should be under 400 ms.

Then they build alerts around violations of these SLOs, or around being close to violating them.

Example pattern:

Pattern: Alert on symptoms, not just causes.
Instead of "CPU > 90%", prefer "Error rate > 2%" or "Latency > SLO".

High CPU alone might not hurt users. High error rate definitely does.

Alert States: Firing, Pending, Resolved

Most alerting systems have states like:

Inactive: Condition is false.
Pending: Condition is true, but not long enough yet.
Firing: Condition has been true longer than the configured for duration.
Resolved: Condition was firing, now condition is false again.

You typically only notify humans when the alert is firing. Some systems also send a "resolved" notification when things recover.

What to Alert On

For backend systems, some alert types are especially common.

Uptime and Health Checks

Use health endpoints that your service exposes, for example /healthz or /live.

Typical alerts:

HTTP 5xx from health endpoint for X minutes.
Health endpoint not reachable from public internet.

Example service-level alert:

"If health check fails from 3 different locations for 5 minutes, send Critical alert."

Error Rates

Instead of only counting errors, compare them to all requests.

Let:

$E(t)$ be the number of error responses in time window $t$.
$R(t)$ be the total number of responses in time window $t$.

Then the error rate is:

$$
\text{error\_rate}(t) = \frac{E(t)}{R(t)}
$$

Rule: Alert on ratio, not just raw counts.
5 errors per second could be trivial if you handle 10,000 requests per second, but huge if you only handle 20.

Example alert:

If error_rate(5m) > 0.05 (5%) for 10 minutes, severity = Critical.

Latency (Response Time)

Latency describes how long requests take. Instead of an average, you want percentiles like p95 or p99, because outliers hurt users even if the average looks okay.

Example:

"p95 latency for POST /api/orders > 800 ms for 15 minutes."

Resource Saturation

Monitor when you are close to limits:

CPU usage as a percentage.
Memory usage, or high GC activity.
Disk space usage.
Number of open file descriptors.
Database connections in use.
Queue length for background jobs.

You usually want capacity alerts to be warning-level, giving you time to act before a hard failure.

Examples:

Disk usage > 85% for 30 minutes: High severity.
DB connection usage > 90% for 10 minutes: High severity.

Business and Security Signals

Over time, you add alerts closer to business impact, for example:

Drop in successful logins.
Sudden spike in failed payments.
Many failed login attempts from same IP or user account.

These are just metrics like any other, but they require thinking about business behavior.

Alert Destinations and Routing

An alert that no one sees is useless. You must send alerts to places where someone will react.

Common Destinations

Destination	Use case
Email	Non-urgent notifications, summaries
Chat (Slack etc)	Team visibility, most alerts
Pager apps	Critical alerts, wake on-call engineer
Ticketing (Jira)	Track follow-up work on recurring problems
Webhooks	Trigger automation, runbooks, or other tools

For Critical alerts in production, teams usually integrate alerting with:

An on-call rotation system (PagerDuty, Opsgenie, etc).
Phone calls, push notifications, or SMS.

Routing by Labels or Tags

Alerts often carry labels such as:

service = "payments-api"
team = "payments"
environment = "production"
severity = "critical"

Then your alert manager uses routing rules:

All alerts with environment = production and severity = critical go to on-call.
Alerts with team = payments go to a specific Slack channel.
Alerts from environment = staging only send to low-priority channel.

This way, the right team sees the right alerts.

Avoiding Alert Fatigue

Alert fatigue happens when engineers receive so many alerts that they stop paying attention. This is very common in badly configured systems.

Common Causes of Bad Alerts

Alerting directly on every single exception event.
Using thresholds that are too low, causing alerts for normal behavior.
No for duration, so any short spike creates alerts.
Duplicate alerts from multiple rules for the same problem.
Alerts firing for non-production environments with same urgency as production.

Good Practices

Use these guidelines to keep alerts useful:

Start with a small, critical set.
Focus on availability, error rates, and very bad performance first.
Use a "for" period.
Require the condition to be true for several minutes before firing.
Group related alerts.
Many alerting systems can group alerts by service or cluster, sending a single message about multiple instances failing.
Tune thresholds over time.
Look at real metric histories before picking values. Adjust if you see frequent false positives.
Silence during maintenance.
When deploying or doing maintenance, silence relevant alerts, or mark deployments to avoid false alerts.
Review incidents.
After a real incident or a noisy alert, update the alert rules. Bad alerts should be fixed or removed.

Rule: Every firing alert should be:

Actionable: You can do something specific.
Owned: A team is clearly responsible.
Relevant: It indicates real or likely user impact.
If an alert fails any of these, change or delete it.

Writing Useful Alert Messages

The content of an alert is as important as the condition. At 3 AM, someone half-asleep must understand it quickly.

A good alert should answer:

What is broken?
Which service, which environment, which region.
How bad is it?
Severity, error rate, affected users if known.
What should I check first?
Links to dashboards, logs, and maybe a runbook.
What changed recently?
Often the cause is a recent deployment or config change.

Example: Bad vs Good Alert

Bad message:

"Error alert: high error count"

Good message:

`[CRITICAL][payments-api][production] 5xx error rate > 10% for 15m
Affected: /api/payments/*
Check: https://grafana.example.com/d/abcd/payments-api
Runbook: https://wiki.example.com/payments-api-runbook#high-error-rate`

The good example includes context, severity, and pointers to next steps.

Basic Alerting with Prometheus and Alertmanager

In many modern backends with Prometheus and Grafana, alerting works like this:

Prometheus collects metrics from your services.
You define alert rules in Prometheus.
When a rule fires, Prometheus sends alert data to Alertmanager.
Alertmanager groups, deduplicates, and routes alerts to email, Slack, paging tools, etc.

An abbreviated example of an alert rule (in YAML) could look like:

yaml

groups:
  - name: backend.rules
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m])
              /
              rate(http_requests_total[5m]) > 0.05
        for: 10m
        labels:
          severity: critical
          service: payments-api
          environment: production
        annotations:
          summary: "High 5xx error rate on payments-api"
          description: "Error rate > 5% for more than 10m in production."

Alertmanager configuration then decides:

Where to send alerts with severity = critical.
How to group alerts.
Who is on-call for service = payments-api.

You do not need to master these tools yet, but understanding this flow helps you design your backend metrics and health checks so they can feed into alerting later.

Alerting and On-Call

In many teams, real-time alerting is connected to an on-call rotation. That is a schedule where one engineer is responsible for handling Critical alerts.

Key ideas:

Only truly urgent alerts should page on-call.
On-call engineers should have access to:

Dashboards.
Logs.
Runbooks (step-by-step guidance).

Incidents that wake people up should lead to:

A review of the alert.
A fix to avoid similar alerts if they were not helpful.

As a backend developer, you contribute by:

Exposing good metrics and health checks.
Suggesting useful alert rules.
Writing simple runbooks for recurring failure modes in your service.

Summary

Alerting is the bridge between your backend systems and human or automated responses. It builds on monitoring and observability, uses thresholds on metrics and logs, and routes information to the right people and tools.

Keep in mind:

Start with a small, critical set of alerts about availability, error rates, and latency.
Use severity levels, "for" durations, and routing rules to prevent noise.
Ensure every alert is actionable, owned, and relevant.
Improve your alerts whenever they misfire during real incidents.

With well-designed alerting, your backend can fail gracefully, and your team can respond quickly without being overwhelmed.