KAHIBARO
Discord Login Register ☰
☰

28.8. Monitoring Production Systems

Table of Contents

Why Monitoring Matters in Production

When your backend goes to production, it stops being “just code” and becomes a service that real people depend on. Monitoring is how you know:

Without monitoring, you only learn about problems when users complain, data is lost, or money is lost. With monitoring, you can:

Think of monitoring as your “nervous system” in production. Logging, metrics and traces are different senses that together tell you how healthy your application is.

Important: A production backend without proper monitoring is not truly “in production,” no matter how well the code is written.

Key Monitoring Concepts

Before looking at tools, it helps to know the common concepts used in monitoring. These will appear in many tools and dashboards.

Black-box vs White-box Monitoring

Examples:

Examples:

In practice, you want both. Black-box tells you “something is wrong for the user.” White-box helps you find out “what exactly is wrong inside.”

Logs, Metrics, Traces

Most monitoring setups are built on three pillars:

PillarWhat it isExample questions it answers
LogsText records of eventsWhat error happened for this specific request?
MetricsNumeric time series valuesIs error rate increasing? Is CPU usage too high?
TracesEnd-to-end request paths across servicesWhich service is slow? Where is time spent in this request?

You do not need to implement everything at once. A typical path:

  1. Start with structured logging and basic health checks.
  2. Add metrics and simple dashboards.
  3. Add tracing when you have multiple services or complex flows.

SLIs, SLOs, and Error Budgets

Large production systems often use three related concepts:

Examples:

Example with availability:

Rule: Define a few clear SLOs for your main APIs, and use them to guide alerts and release decisions.

Health Checks

A health check is a simple endpoint or command that returns the health status of your service. It is used by:

Types of Health Checks

You will commonly see at least two levels.

Liveness Checks

Liveness checks answer: “Is the process alive at all?” If this check fails, the system may restart the container or process.

Examples:

A simple liveness endpoint in FastAPI:

python 
from fastapi import FastAPI
app = FastAPI()
@app.get("/healthz")
def liveness():
    return {"status": "ok"}

This only proves that the app is running and can respond. It does not tell you that the database works or that dependencies are healthy.

Readiness Checks

Readiness checks answer: “Can this instance handle real traffic correctly?” A service might be alive but not ready, for example if:

Example FastAPI readiness endpoint:

python 
from fastapi import FastAPI
import asyncpg
app = FastAPI()
@app.get("/readyz")
async def readiness():
    try:
        conn = await asyncpg.connect("postgres://user:pass@db/app")
        await conn.execute("SELECT 1")
        await conn.close()
        return {"status": "ready"}
    except Exception:
        # Failure status and code for readiness
        return {"status": "not_ready"}

A load balancer can be configured to:

Designing Health Checks

Good health checks are:

Choosing What to Monitor

Monitoring everything is impossible and noisy. You must choose what matters most. For backends, there are common categories.

Infrastructure Metrics

These are about the servers or containers:

MetricWhy it matters
CPU usageHigh CPU can cause slowness, thread pool exhaustion.
Memory usageMemory leaks, out-of-memory kills, increased latency.
Disk usageFull disks stop logs, databases and queues.
Disk I/OSlow disks hurt database and file operations.
Network I/OCan reveal DDoS attacks or misconfigured clients.

Example: You might alert if CPU is above 85% for 5 minutes, or disk usage is over 90%.

Application Metrics

These are tied to your API and business logic:

MetricExample
Request rate (RPS)120 requests per second to /api/orders
Latency (p50, p95, p99)p95 latency = 400 ms for /api/login
Error rate3% of requests return 5xx in the last 5 minutes
Queue length200 pending jobs in the email queue
Job processing timeAverage job takes 2 seconds

Percentiles:

Rule: Always monitor at least request rate, error rate, and latency for your main APIs.

Business Metrics

Business metrics show if the system is delivering value:

These metrics are often critical, sometimes more than pure technical metrics. An app may be technically “healthy” but broken if, for example, payments always fail due to a new bug.

The “Four Golden Signals”

A common rule from SRE practice is to always monitor:

  1. Latency
    How long each request takes.
  2. Traffic
    How many requests you get per time unit.
  3. Errors
    How many requests fail.
  4. Saturation
    How “full” your system is, for example CPU, memory, or queue length.

If you have these four, you can usually detect and debug most production issues.

Dashboards

Dashboards are visual collections of your important metrics. A good dashboard lets you answer questions quickly without digging into raw logs.

Designing Useful Dashboards

Do not create dashboards with 50 tiny graphs. Focus on clarity.

Useful patterns:

  1. Top-level service dashboard
    • Overall request rate.
    • Error rate.
    • p50, p95 latency.
    • CPU and memory usage.
    • Database connection count.
  2. Per-endpoint dashboard
    For critical endpoints like /login, /checkout:
    • Requests per second.
    • Error rate.
    • Latency percentiles.
    • Rate of specific errors, such as validation errors vs 500s.
  3. Backend worker dashboard
    • Queue length for each queue.
    • Job processing rate.
    • Job failure rate.
    • Job runtime distributions.

Example table of typical graphs:

Dashboard sectionGraph example
TrafficTotal requests per second
ErrorsHTTP 5xx per minute
LatencyOverall p95 latency for all endpoints
DBQueries per second, slow queries count
QueuesJobs queued vs processed per minute

Making Dashboards Actionable

A dashboard is useful only if it helps you make decisions such as:

To be actionable:

Alerts and On-call

Monitoring is not only about graphs. You also need alerts, which notify you when something is wrong.

What Makes a Good Alert

Good alerts:

Bad alerts:

Example of a good alert:

“Checkout API 5xx error rate > 5% for 5 minutes. Possible user impact. Check database connectivity and recent deployments.”

Example of a bad alert:

“CPU > 70% for 1 minute.”
(Might be normal under load and self-resolving.)

Basic Alert Examples

Some starter alarms for an API backend:

ConditionWhy it matters
5xx error rate > 2% for 10 minutesUsers are hitting server errors
p95 latency > 1 second for 10 minutesSystem is slow for many users
Health check failing on any instanceAn instance is unhealthy
DB connection failures > X per minuteDatabase issues or misconfiguration
Queue length > N for more than 15 minutesBackground jobs falling behind

Start with a small set of alerts that directly relate to user impact. Expand slowly, and review alerts after incidents.

On-call Basics

In teams, someone is usually “on-call”, meaning they receive alerts outside normal working hours. Even in small projects, you might:

Guidelines:

Monitoring Logs in Production

Logs are a major part of monitoring in production. They help you see detailed events and debug specific failures.

Structured Logging

In production, plain text logs are hard to search. It is better to use structured logging, usually JSON, where each field is a key-value pair.

Example of a structured log entry:

json 
{
  "timestamp": "2026-08-28T15:23:10Z",
  "level": "INFO",
  "message": "Created new order",
  "order_id": 12345,
  "user_id": 678,
  "endpoint": "/api/orders",
  "duration_ms": 85,
  "status_code": 201
}

This type of log can be easily filtered, for example:

Centralized Logging

In production you usually have:

Reading logs directly from each machine is not practical. You need centralized logging, where:

  1. Each instance writes logs to stdout or a local file.
  2. A logging agent or service collects logs.
  3. Logs are sent to a central system like Elasticsearch, Logstash, Kibana (ELK), or Loki, or a hosted logging platform.

Centralized logging lets you:

Correlation IDs

For debugging user requests, it helps to tag all logs from a single request with a correlation id or request id.

Flow:

  1. A request arrives with no id.
  2. The gateway or application generates a random id, for example req-abc123.
  3. This id:
    • Is added to the response header.
    • Is included in every log line created while handling that request.
    • May be passed to downstream services.

This way, when a user reports a problem with a specific request, you can search logs for that id and see the whole story.

Tracing and Distributed Tracing

As systems grow into microservices and multiple backends, a single request might:

If the request is slow, which part is responsible? Logs are helpful but can be complex. Tracing records a flow of sub-operations for each request.

Basic Concepts

Each span has:

Distributed Tracing

In distributed tracing, spans from different services are linked using:

Example:

In dashboards, you see a timeline that shows where time is spent. If the payment service is slow, you see a long bar for that span.

OpenTelemetry and Tools

Modern systems often use OpenTelemetry to:

Step by step:

  1. Add OpenTelemetry SDK to your application.
  2. Instrument your HTTP server and client, database calls, and background workers.
  3. Export traces to a backend that can visualize them.

This is especially valuable once you have more than one service involved in a request.

Monitoring Databases and Queues

Many production incidents start with the database or background processing. You should monitor these separately.

Database Monitoring

Key metrics:

MetricWhy it matters
Connections in useHitting max connections causes failures
Slow queries countCan reveal missing indexes or bad queries
Query throughputCapacity and load on the database
Replication lag (if replicas)Stale reads or risk of inconsistency
Cache hit rate (if DB caching)Efficiency of caching strategies

PostgreSQL, for example, can expose:

Monitoring these metrics can help you detect:

Queue and Worker Monitoring

If you use background workers such as Celery with Redis:

Key metrics:

MetricWhy it matters
Queue lengthToo long means jobs are delayed
Job throughputJobs processed per minute
Job failuresHigh failure rate means bugs or external issues
Job runtimeSlow jobs may cause backlogs

You can set alerts when, for example:

Continuous Improvement of Monitoring

Monitoring is not something you finish once. It evolves as your application, architecture and user traffic change.

Learn from Incidents

After each incident, ask:

Then:

Monitor Your Monitoring

Monitoring systems themselves can fail or become overloaded. Consider:

Start Simple, Grow Gradually

For a small backend, a realistic starting point:

  1. Health checks for liveness and readiness.
  2. Structured logging and centralized log storage.
  3. Basic metrics:
    • Request rate.
    • Error rate.
    • p95 latency.
    • CPU and memory.
  4. A few essential alerts tied to user impact.

As you grow:

Key Principle: Monitoring should help you sleep better, not create constant noise. Start with the most important signals and refine them based on real incidents.

With solid monitoring in place, your production backend becomes observable. You can see what is happening, understand why it is happening, and respond quickly when things go wrong.

Views: 6

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3056 / 24h
Impressum / Imprint
Datenschutz / Data Policy