KAHIBARO
Discord Login Register ☰
☰

20.8. Monitoring

Table of Contents

Why Monitoring Matters

Monitoring is how you see what your backend is doing in real time. Logs tell you what happened. Monitoring tells you how things are going right now, and whether you need to act.

Some problems you can only catch by monitoring:

Without monitoring you often only learn about problems when users complain.

Important: Monitoring is about continuous measurement of system health and performance, not about debugging single requests. Use logs for details, use monitoring for trends and alerts.

In this chapter we focus on the concepts of monitoring. Specific tools like Prometheus and Grafana are covered in their own chapters.

Key Concepts in Monitoring

Metrics vs Logs vs Traces

You already saw logging earlier. Monitoring mainly works with metrics and sometimes traces.

Monitoring focuses mostly on metrics.

Types of Metrics

You will see the same few metric types again and again.

1. Counters

A counter only goes up, never down, except when the process restarts.

Examples:

Example in pseudo code (Python style):

python 
http_requests_total += 1
if response.status_code >= 500:
    http_errors_total += 1

You do not reset counters manually. Monitoring systems calculate rates from them, like requests per second.

2. Gauges

A gauge can go up and down.

Examples:

Example:

python 
db_connections_open = get_current_connection_count()

3. Histograms / Summaries

These measure distributions, such as:

They let you answer:

Example labels:

text 
http_request_duration_seconds_bucket{
  path="/api/orders",
  method="GET",
  le="0.1"
}  1234

This says: 1234 GET /api/orders requests finished in ≤ 0.1 seconds.

Rule of thumb:
Use counters for counts, gauges for "current values", and histograms for latencies and sizes. Do not store every value as a log if you only need aggregated behavior, use metrics.

What You Should Monitor

You cannot monitor everything. Start with the most important signals.

The “Four Golden Signals”

A very common model is the four golden signals of monitoring:

SignalQuestion it answersExamples
LatencyHow fast are requests?Request duration, DB query time
TrafficHow much work is the system doing?Requests per second, messages per second
ErrorsHow often do things fail?Error rate, failed jobs, 5xx responses
SaturationHow “full” is the system?CPU, memory, queue length, DB connections

These give a high level picture of system health.

Latency

You care about:

Example:

This tells you some users have a bad experience, even if the average looks fine.

Traffic

Examples:

Use traffic metrics to:

Errors

Important metrics:

Often you define an error budget, for example:

Example error budget:
Up to 0.1% of all requests may fail in a month. If you exceed that, you must spend time on reliability before adding new features.

Saturation

Saturation shows how close you are to the limits:

If saturation is always high, you may need more instances or optimization.

Application-Level Monitoring

System metrics like CPU and RAM are useful. But you also need application metrics that reflect your business and logic.

Technical Application Metrics

Common examples:

MetricDescription
http_requests_totalTotal HTTP requests
http_requests_in_flightRequests currently being processed
http_request_duration_secondsRequest duration histogram
db_queries_totalTotal DB queries
db_query_duration_secondsDB query time histogram
jobs_queued_totalJobs pushed to background queue
jobs_failed_totalFailed jobs

Add labels to metrics, for example:

Example in pseudo code:

python 
increment("http_requests_total", labels={
    "method": request.method,
    "endpoint": "/api/orders",
    "status_code": response.status_code,
})

Do not create labels with unbounded values such as user IDs, email addresses, or random strings. That will explode your metric storage.

Important rule:
Metric label values should come from small, fixed sets like HTTP methods or a limited set of endpoints.
Never put user IDs, request IDs, or arbitrary strings in labels.

Business Metrics

Business metrics describe what the application achieves, not just how it runs.

Examples:

These help answer:

Business metrics are often more important than purely technical ones, because they show real user impact.

Infrastructure Monitoring

Monitoring should not stop at your application. You also monitor the environment it runs in.

Host and Container Metrics

Examples:

MetricDescription
cpu_usage_percentCPU usage per host or container
memory_used_bytesRAM usage
disk_used_percentDisk usage
network_rx_bytes_totalReceived bytes over network
network_tx_bytes_totalSent bytes over network

With containers and orchestration (for example Docker, Kubernetes), you also monitor:

Dependency and External Service Monitoring

Dependencies often fail:

For each important dependency, measure:

Example:

text 
external_api_requests_total{service="payments", status="success"} 1532
external_api_requests_total{service="payments", status="error"}    27

When external services become slow, your own system may become slow too. Monitoring helps you detect this quickly.

Health Checks and Probes

Monitoring tools and load balancers often call health endpoints to check if your application is alive and ready.

Liveness vs Readiness

Two useful concepts:

Typical FastAPI style example:

python 
@app.get("/health/live")
def liveness():
    # If code runs, return OK
    return {"status": "ok"}
@app.get("/health/ready")
def readiness():
    if not db_is_connected():
        raise HTTPException(status_code=503, detail="DB not ready")
    return {"status": "ready"}

Monitoring systems can watch these endpoints and alert if they fail.

Rule:
Expose simple, fast health endpoints.
Liveness should only check if the process is alive.
Readiness should check if dependencies are usable.

Dashboards

Dashboards present metrics in a way humans can understand quickly.

What to Put on a Dashboard

You usually create multiple dashboards:

  1. Overview dashboard
    • Requests per second
    • Error rate
    • p95 latency
    • CPU, memory, database connections
  2. API performance dashboard
    • Latency per endpoint
    • Error rate per endpoint
    • Successful vs failed calls to external services
  3. Background jobs dashboard
    • Jobs queued vs processed
    • Failed jobs per worker
    • Job processing latency
  4. Business dashboard
    • Orders per minute
    • Payment success rate
    • Signups per hour

The main rule: keep them simple enough to understand in a stressful situation.

Example Dashboard Layout

RowGraphs
1Requests per second, Error rate
2p95 latency overall, p95 latency by main endpoints
3CPU and memory, DB connections and query duration
4Queue length, failed jobs per minute

Alerting

Monitoring is useless if nobody reacts. Alerting notifies you when metrics cross dangerous thresholds.

Alert Rules

Typical alert rules:

Example rule in plain language:

If http_5xx_error_rate is above 5% for 10 minutes, send a critical alert to the on-call channel.

Alert Channels

Common channels:

You want:

Alert Priorities

Usually you define levels:

LevelExampleAction
InfoNew version deployedNo immediate action
WarningLatency slightly higher than normalInvestigate during working hours
CriticalHigh error rate affecting many usersImmediate action, wake up on-call engineer

Monitoring in Different Environments

You usually have at least:

Monitoring is most important in production, but other environments are useful too.

Production

Staging / Testing

Example:

Local Development

You normally do not need full monitoring locally, but you might:

Common Monitoring Pitfalls

Too Many Metrics

It is easy to instrument everything. That can create problems:

Guidelines:

Missing Context

If you only see "error rate high", that is not enough.

Combine metrics with:

Example: after a deployment at 12:05, latency jumps at 12:07. This gives you a strong hint that the deployment caused the issue.

No Clear Ownership

If everyone owns monitoring, sometimes nobody does. Decide:

How Monitoring Fits With Logging and Observability

Monitoring is one part of a bigger picture often called observability.

A simple mental model:

LayerPurpose
LoggingUnderstand specific events and errors
MonitoringWatch system health and performance over time
TracingFollow a single request through the system

You will often:

  1. Receive an alert from monitoring.
  2. Look at dashboards to confirm there is a problem.
  3. Check logs or traces to find the exact cause.

Monitoring connects high level symptoms (error rate up, latency up) to detailed investigation (logs, traces).

Summary

Next chapters will show concrete tools like Prometheus and Grafana, and how to implement monitoring for your backend.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3051 / 24h
Impressum / Imprint
Datenschutz / Data Policy