KAHIBARO
Discord Login Register ☰
☰

20.9. Prometheus

Table of Contents

Why Prometheus Matters for Backend Developers

Prometheus is one of the most popular tools for collecting and querying metrics from backend systems. It is used to answer questions like:

Prometheus is focused on metrics, not logs or traces. Metrics are numeric measurements over time, such as $requests\_total$ or $cpu\_usage\_percent$.

Key idea: Prometheus is a pull-based, time-series metrics database. It regularly scrapes (pulls) metrics from your services, stores them, and lets you query them with a special language called PromQL.

Compared to traditional monitoring tools, Prometheus is:

In this chapter, you will learn what Prometheus is, how it works, which metrics you should export from backends, and how to query them.

Core Concepts of Prometheus

Time Series and Metrics

Prometheus stores time series: a sequence of values for the same metric over time.

Each time series is identified by:

  1. Metric name, for example:
    • http_requests_total
    • process_cpu_seconds_total
    • api_response_time_seconds
  2. A set of labels, which are key-value pairs that describe dimensions of the metric, such as:
    • method="GET"
    • status="200"
    • endpoint="/users"
    • instance="api-1"

Together, these define one time series, for example:

Every time Prometheus scrapes your service, it gets the current value for each time series and attaches a timestamp.

Metric Types

Prometheus has 4 main metric types:

TypeDescriptionTypical usage example
CounterValue that only increasesTotal HTTP requests, total errors
GaugeValue that can go up and downMemory usage, current active users
HistogramBuckets that count observations by sizeRequest duration, response size distribution
SummarySimilar to histogram, client-side quantilesRequest duration with percentiles

Rule:

  • Use counters for things that are counted over time and only increase, like *_total.
  • Use gauges for values that can go up and down, like memory or queue length.
  • Use histograms or summaries for durations and sizes if you care about distribution (p50, p90, p99).

Example metric names:

Pull-based Scraping

Prometheus does not wait for applications to push metrics. Instead, it pulls metrics from them.

Each instrumented service exposes an HTTP endpoint, usually /metrics. Prometheus periodically sends an HTTP GET request to that endpoint and parses the metrics.

Example request from Prometheus to your service:

Your service responds with text like:

text 
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",status="200"} 1523
http_requests_total{method="GET",status="404"} 12
http_requests_total{method="POST",status="201"} 231

Prometheus:

  1. Reads this text.
  2. Updates the internal time series.
  3. Stores them in its time series database.

This pull model has benefits:

If you cannot expose a /metrics endpoint, you can use the Pushgateway, but that is a special case and less common for backend web services.

Targets and Jobs

Prometheus scrapes targets. A target is a single endpoint that exposes metrics, such as:

Targets are grouped into jobs. A job is a group of related targets that logically belong together.

Example jobs:

Prometheus automatically adds some labels:

Example time series:

up is a built-in gauge that is 1 when the target is reachable and 0 when it is not.

Prometheus Data Model and Labels

Labels and High Cardinality

Labels are powerful because they let you slice and filter metrics. For example:

You can use labels to answer questions like:

However, you must be careful about high cardinality.

Cardinality is the number of unique label combinations.

Rule: Avoid labels that can take many different values, such as:

  • User IDs
  • Request IDs
  • Full URLs with path parameters
  • Random tokens or hashes
    High cardinality explodes the number of time series and can make Prometheus slow or even unusable.

Good labels:

Bad labels:

Instead of labeling by user_id, you would use aggregate metrics like:

Examples of Metric Naming and Labeling

Metric names should be:

Good nameMeaning
http_requests_totalTotal HTTP requests
http_request_duration_secondsDuration of HTTP requests in seconds
queue_sizeCurrent size of some queue
db_queries_totalNumber of database queries
db_query_duration_secondsDuration of database queries in seconds

Example for an API:

text 
http_requests_total{method="GET",status="200",endpoint="/users"} 1523
http_requests_total{method="GET",status="404",endpoint="/users"} 12
http_requests_total{method="POST",status="201",endpoint="/users"} 231

This allows you to ask:

Prometheus in Practice for Backend Services

Typical Setup Overview

A common basic monitoring setup for a backend application looks like this:

High level flow:

  1. Your app exports metrics at /metrics.
  2. Prometheus scrapes /metrics every few seconds.
  3. Prometheus stores the data.
  4. Grafana visualizes metrics and dashboards.
  5. Alertmanager (an extra component) sends alerts based on rules.

Exporting Metrics from a Web Backend

Your job as a backend developer is usually to:

  1. Instrument your code with metrics.
  2. Expose a /metrics endpoint.
  3. Make sure Prometheus can reach that endpoint.

In Python, a typical approach is:

  1. Use the prometheus_client library to define metrics:
python 
   from prometheus_client import Counter, Histogram
   REQUESTS_TOTAL = Counter(
       "http_requests_total",
       "Total HTTP requests",
       ["method", "endpoint", "status"],
   )
   REQUEST_DURATION_SECONDS = Histogram(
       "http_request_duration_seconds",
       "HTTP request duration in seconds",
       ["method", "endpoint", "status"],
   )
  1. Update them in your request handling code (for example in FastAPI middleware):
python 
   import time
   from fastapi import FastAPI, Request
   from prometheus_client import generate_latest, CONTENT_TYPE_LATEST
   from starlette.responses import Response
   app = FastAPI()
   @app.middleware("http")
   async def metrics_middleware(request: Request, call_next):
       method = request.method
       endpoint = request.url.path
       start = time.perf_counter()
       try:
           response = await call_next(request)
           status = str(response.status_code)
       except Exception:
           status = "500"
           raise
       finally:
           duration = time.perf_counter() - start
           REQUESTS_TOTAL.labels(method=method, endpoint=endpoint, status=status).inc()
           REQUEST_DURATION_SECONDS.labels(
               method=method, endpoint=endpoint, status=status
           ).observe(duration)
       return response
   @app.get("/metrics")
   def metrics():
       data = generate_latest()
       return Response(data, media_type=CONTENT_TYPE_LATEST)
  1. Prometheus scrapes http://your-api:8000/metrics.

Example content of /metrics in text form:

text 
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",endpoint="/users",status="200"} 1523
http_requests_total{method="GET",endpoint="/users",status="404"} 12
http_requests_total{method="POST",endpoint="/users",status="201"} 231
# HELP http_request_duration_seconds HTTP request duration in seconds
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.1",method="GET",endpoint="/users",status="200"} 1300
http_request_duration_seconds_bucket{le="0.5",method="GET",endpoint="/users",status="200"} 1490
http_request_duration_seconds_bucket{le="1",method="GET",endpoint="/users",status="200"} 1510
http_request_duration_seconds_bucket{le="+Inf",method="GET",endpoint="/users",status="200"} 1523
http_request_duration_seconds_sum{method="GET",endpoint="/users",status="200"} 48.2
http_request_duration_seconds_count{method="GET",endpoint="/users",status="200"} 1523

You do not have to write these buckets yourself. The client library generates them.

Prometheus Configuration for Your Service

Prometheus uses a YAML configuration file, typically prometheus.yml. A very simple configuration that scrapes one backend app might look like this:

yaml 
global:
  scrape_interval: 15s  # how often to scrape by default
scrape_configs:
  - job_name: "api"
    scrape_interval: 5s  # override for this job
    static_configs:
      - targets:
          - "api-1:8000"
          - "api-2:8000"

This directs Prometheus to scrape the /metrics endpoint from both instances every 5 seconds.

If you use Docker Compose, this might look like:

yaml 
services:
  api:
    image: my-api-image
    ports:
      - "8000:8000"
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

Prometheus will then be accessible at http://localhost:9090.

PromQL: Querying Metrics

Prometheus comes with its own query language called PromQL (Prometheus Query Language). You use it to:

Basic PromQL Examples

Selecting a Metric

Return all time series for http_requests_total:

promql 
http_requests_total

Filter by label, for example only GET requests:

promql 
http_requests_total{method="GET"}

Filter out 2xx responses:

promql 
http_requests_total{status!~"2.."}

Rates and Counters

Counters only increase, so you are usually interested in the rate of increase.

PromQL has functions like rate and irate.

Example: requests per second over the last 5 minutes:

promql 
rate(http_requests_total[5m])

Sum across all instances:

promql 
sum(rate(http_requests_total[5m]))

Requests per second per status code:

promql 
sum by (status) (rate(http_requests_total[5m]))
promql 
sum(rate(http_requests_total{status!~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Formula:
To compute error rate, define:
$$
\text{error\_rate} = \frac{\text{error\_requests\_per\_second}}{\text{total\_requests\_per\_second}}
$$
In PromQL:

Gauges and Averages

For gauges, you use functions like avg, max, min.

Example: average memory usage across instances:

promql 
avg(process_resident_memory_bytes)

Maximum CPU usage per instance:

promql 
max by (instance) (rate(process_cpu_seconds_total[5m]))

Histograms and Latency

Histograms store values in buckets. For request duration, you can estimate percentiles using histogram_quantile.

Assume you have:

To compute p90 latency for GET requests to /users:

promql 
histogram_quantile(
  0.90,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{method="GET", endpoint="/users"}[5m])
  )
)

This returns the 90th percentile in seconds.

Similarly, p99 for all requests:

promql 
histogram_quantile(
  0.99,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

These queries are commonly used in Grafana dashboards.

Examples of Useful Backend Queries

Here are some queries that are typical for monitoring a REST API.

Total Requests per Second

promql 
sum(rate(http_requests_total[5m]))

Requests per second by method:

promql 
sum by (method) (rate(http_requests_total[5m]))

Error Rate

5xx error requests per second:

promql 
sum(rate(http_requests_total{status=~"5.."}[5m]))

Percentage of non-2xx requests:

promql 
sum(rate(http_requests_total{status!~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))
* 100

Latency

p95 latency for all requests:

promql 
histogram_quantile(
  0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

p95 latency for /orders endpoint:

promql 
histogram_quantile(
  0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{endpoint="/orders"}[5m]))
)

Instance Health

Check if all instances are up:

promql 
up

Only API instances:

promql 
up{job="api"}

Instances that are down:

promql 
up == 0

Example: Full Flow from Code to Graph

To connect the whole picture, consider this small scenario.

1. Instrumentation in the API

You add metrics in your API using a library, for example in Python:

python 
from prometheus_client import Counter
ERRORS_TOTAL = Counter(
    "http_errors_total",
    "Total HTTP errors",
    ["method", "endpoint", "status"],
)
# inside your error handler or middleware:
ERRORS_TOTAL.labels(method=method, endpoint=endpoint, status=status).inc()

2. Metrics Endpoint

You expose a /metrics endpoint that returns all metrics (including your custom ones).

3. Prometheus Scrape Configuration

In prometheus.yml:

yaml 
scrape_configs:
  - job_name: "api"
    scrape_interval: 5s
    static_configs:
      - targets:
          - "api-1:8000"
          - "api-2:8000"

4. Prometheus Stores Data

Every 5 seconds, Prometheus pulls metrics and keeps appending the newest values to the time series for http_errors_total.

5. Grafana Dashboard

In Grafana, you create a panel with the query:

promql 
sum(rate(http_errors_total[5m]))

This shows you how many errors per second happen, averaged over the last 5 minutes.

You can also break it down by endpoint:

promql 
sum by (endpoint) (rate(http_errors_total[5m]))

Now you can see which endpoint is most problematic.

Integration with Other Components

Prometheus rarely runs alone in a production backend. It is usually part of a monitoring stack:

ComponentPurpose
PrometheusScrapes metrics, stores time series
AlertmanagerSends alerts based on Prometheus rules
GrafanaVisualizes metrics and creates dashboards
ExportersExpose metrics from systems like databases, Redis, OS

Examples of exporters you might use:

Your backend services just become additional targets in this ecosystem.

Practical Tips and Common Pitfalls

Tips for Designing Metrics

Avoiding Common Problems

A rough rule: if you have:

Then total time series ≈ $N \times M \times K$.

Keep this number reasonable or Prometheus may struggle.

Rule of thumb: Try to keep time series count manageable. Many production setups stay under a few million time series. As a beginner, if you are in the tens of thousands, you are already fine and should avoid exploding it with high-cardinality labels.

Summary

In this chapter you learned:

For a backend developer, Prometheus is a key tool to understand how your application behaves in production, detect problems early, and verify that changes did not break performance or reliability.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3506 / 24h
Impressum / Imprint
Datenschutz / Data Policy