KAHIBARO
Discord Login Register ☰
☰

23.8. Production Logging

Table of Contents

Why Production Logging Is Different

In development you log mainly for yourself. In production you log for future you, your teammates, and often for 3 a.m. you that has to fix a broken system fast.

Production logging must:

Think of production logs as your system’s black box recorder.

In production, unstructured print() logs, local-only log files, and logs containing secrets or personal data are unacceptable.

We will focus on patterns that apply to any backend stack, with concrete examples in Python-like pseudocode where helpful.

Logging Goals in Production

Observability and Debugging

In production you often cannot reproduce issues locally. Logs must:

Example of helpful context:

json 
{
  "level": "ERROR",
  "timestamp": "2026-08-28T09:15:22.123Z",
  "service": "orders-api",
  "env": "prod",
  "request_id": "a0a1f774-6dae-4d88-9b1b-0b0f86377c2b",
  "user_id": 42,
  "path": "/api/orders",
  "method": "POST",
  "message": "Failed to create order",
  "error": "psycopg2.IntegrityError: duplicate key value violates unique constraint"
}

Compare that to:

text 
Something went wrong

Only the first one is useful in production.

Auditing and Compliance

For some systems you must be able to answer questions like:

That means keeping audit logs for sensitive operations, for example:

json 
{
  "level": "INFO",
  "type": "audit",
  "timestamp": "2026-08-28T10:02:11.011Z",
  "service": "admin-api",
  "action": "user_role_changed",
  "performed_by": 1,
  "target_user": 42,
  "old_role": "user",
  "new_role": "admin",
  "ip": "203.0.113.7"
}

Audit logs:

Performance and Capacity Planning

Logs can help you see:

For example, logging slow requests:

json 
{
  "level": "WARNING",
  "event": "slow_request",
  "path": "/api/orders",
  "method": "GET",
  "duration_ms": 1200,
  "user_id": 42
}

You can then build dashboards on top of this information.

What to Log in Production

Request and Response Information

At a minimum, log for each incoming request:

Example structure:

json 
{
  "timestamp": "2026-08-28T09:30:01.500Z",
  "level": "INFO",
  "event": "request_completed",
  "service": "shop-api",
  "env": "prod",
  "request_id": "3a4e...",
  "method": "GET",
  "path": "/api/products",
  "status_code": 200,
  "duration_ms": 45,
  "client_ip": "198.51.100.23",
  "user_id": 123
}

Do not log full request or response bodies by default in production, especially not:

Business Events

Log important business events such as:

Example:

json 
{
  "level": "INFO",
  "event": "order_created",
  "order_id": 987,
  "user_id": 123,
  "total": 149.99,
  "currency": "USD",
  "timestamp": "2026-08-28T11:00:21.044Z"
}

These logs help with:

Errors and Exceptions

When an error happens:

Example (Python-style):

python 
logger.exception(
    "Failed to process payment",
    extra={
        "event": "payment_error",
        "user_id": user_id,
        "order_id": order_id,
        "payment_provider": "stripe",
    }
)

This should produce a log with:

Never log raw secrets such as:

  • Passwords
  • API keys
  • Access tokens
  • Private keys
  • Full credit card numbers
    Mask or omit them before logging.

Log Levels and How to Use Them in Production

Most logging systems have at least these levels:

LevelPurpose in productionExample
DEBUGDetailed internal information for troubleshootingVariable values, low-level function calls
INFONormal operation eventsStartup complete, request completed, user logged in
WARNINGSomething unexpected, but system still worksSlow request, near limit, retry succeeded
ERRORA request or action failedUnhandled exception in endpoint, DB error, third-party fail
CRITICAL / FATALApplication or subsystem is unusableService cannot start, configuration missing

Production defaults:

Example usage in Python style:

python 
logger.debug("Cache miss", extra={"key": cache_key})
logger.info("User logged in", extra={"user_id": user.id})
logger.warning("Slow query", extra={"duration_ms": 900, "query": "SELECT ..."})
logger.error("Failed to send email", exc_info=True)
logger.critical("Database unavailable, shutting down")

Define clear rules for when to use each level and stick to them. Mixed or random use of log levels makes your logs noisy and almost useless.

Structured Logging in Production

Why Structured Logging

Plain text logs like:

text 
[INFO] 2026-08-28 10:01:03 User 42 logged in from 203.0.113.7

are hard for machines to parse reliably.

Structured logs such as JSON:

json 
{
  "timestamp": "2026-08-28T10:01:03.501Z",
  "level": "INFO",
  "event": "user_login",
  "user_id": 42,
  "ip": "203.0.113.7",
  "service": "auth-api",
  "env": "prod"
}

are much easier to:

Structured logging is the standard for modern production systems.

Common Fields to Include

You can define a common schema for all logs:

FieldDescription
timestampISO 8601 UTC time
levelDEBUG / INFO / WARNING / ERROR / CRITICAL
serviceService or application name
envEnvironment, for example prod, staging
hostHostname / container id
request_idCorrelation id for the request
user_idAuthenticated user, if any
eventShort event name
messageHuman readable description

Then each log can add its own custom fields.

Adding Structure in Code

Python-like example with a JSON logger:

python 
import logging
import json
import sys
class JsonFormatter(logging.Formatter):
    def format(self, record):
        log = {
            "timestamp": self.formatTime(record, datefmt="%Y-%m-%dT%H:%M:%S.%fZ"),
            "level": record.levelname,
            "message": record.getMessage(),
            "service": "orders-api",
            "env": "prod",
        }
        if hasattr(record, "request_id"):
            log["request_id"] = record.request_id
        if hasattr(record, "user_id"):
            log["user_id"] = record.user_id
        return json.dumps(log)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(JsonFormatter())
logger = logging.getLogger("orders-api")
logger.setLevel(logging.INFO)
logger.addHandler(handler)

Then:

python 
logger.info("Order created", extra={"user_id": 123, "request_id": "abc"})

Outputs JSON with these fields.

Correlation IDs and Traceability

When a single user action touches multiple services or components, you want to connect all related logs. This is where correlation IDs (or trace IDs) help.

How It Works

  1. When a request enters your system, you:
    • Look for an existing correlation ID header, for example X-Request-ID.
    • If none, generate a new unique ID.
  2. You attach this ID to:
    • The request context
    • All logs produced while handling this request
    • Any outgoing requests to other services (as a header).
  3. Downstream services also log this ID.

Now you can search the log system by that ID and follow the whole flow.

Example log fields:

json 
{
  "timestamp": "2026-08-28T10:20:00.111Z",
  "level": "INFO",
  "service": "auth-api",
  "event": "request_completed",
  "request_id": "a3f4...",
  "path": "/login",
  "status_code": 200
}
json 
{
  "timestamp": "2026-08-28T10:20:00.210Z",
  "level": "INFO",
  "service": "orders-api",
  "event": "request_completed",
  "request_id": "a3f4...",
  "path": "/orders",
  "status_code": 201
}

Search for request_id:a3f4... and you see both logs.

Implementing in a Web Backend

In many frameworks, you do this with middleware:

  1. Generate or read a request_id on each request.
  2. Store it in a request-scoped context.
  3. Configure your logger to read from that context on each log call.

Even simple frameworks allow you to pass it manually in extra fields.

Log Storage and Centralization

In production you often have:

You cannot SSH into each instance and read /var/log/app.log.

Centralized Logging Pattern

Typical pattern:

  1. Applications print logs to stdout/stderr as structured lines, often JSON.
  2. The container runtime or host collects logs.
  3. A log agent (Fluent Bit, Filebeat, Vector, etc.) ships logs to a central system.
  4. A log backend stores and indexes logs, for example:
    • Elasticsearch / OpenSearch
    • Loki
    • Cloud logging (AWS CloudWatch, GCP Logging, etc.)
    • Hosted log platforms

Then you use a UI or API to:

In containerized production systems, log to stdout/stderr, not to random local files. Let the platform handle collection and rotation.

Log Rotation and Retention

Logs grow forever if you let them. In production you must define:

Choices often depend on:

Example policy:

Log Volume, Sampling, and Cost

In production, logs:

You should avoid both extremes:

Reducing Log Noise

Some strategies:

Example bad pattern:

python 
logger.info("User details: %s", user.to_dict())

On a busy site this can generate millions of logs per hour.

Better:

python 
logger.debug("Loaded user", extra={"user_id": user.id})

Then DEBUG can be disabled in production or enabled only for specific components.

Sampling

For extremely high volume events, you can sample:

Example: Log at most 1 out of 100 successful requests, but log all errors.

Pseudocode:

python 
import random
if random.random() < 0.01:  # 1% sampling
    logger.info("Request completed", extra=fields)

Errors and warnings should usually not be sampled, or sampled very lightly.

Security and Privacy in Production Logs

Logs can easily become a security and privacy risk.

Do Not Log Sensitive Data

Sensitive examples:

Typical mistakes:

Masking example:

python 
safe_body = {**body}
if "password" in safe_body:
    safe_body["password"] = "***redacted***"
logger.warning("Invalid registration data", extra={"body": safe_body})

Everything you log might be seen by many people or systems. Never treat logs as private or safe for secrets.

Access Control and Encryption

Data Retention and Anonymization

If your system handles personal data, you may need:

For example, log user_id instead of email, and keep the mapping in your main database, which is already protected and has its own retention rules.

Designing a Production Logging Strategy

When preparing for production deployment, define and implement a clear logging strategy.

Define What You Log

For each service, write down:

  1. Minimal request log fields.
  2. Important business events to log.
  3. Which errors to log and at which level.
  4. What must never be logged.

Example checklist for an API service:

Standardize Log Format

Across services, agree on:

This makes cross-service debugging easier.

Integrate with Monitoring and Alerting

Use logs for:

This usually involves:

Example: Minimal Production Logging Setup for a Simple API

Imagine a FastAPI or similar web service running in Docker.

  1. Application logging:
    • Use the standard logging library.
    • Configure a JSON formatter.
    • Include service, env, and request_id fields in all logs.
    • Log to stdout only.
  2. Middleware:
    • Generate or read X-Request-ID.
    • Log request completion with path, method, status code, duration, user_id.
    • Attach request_id to a per-request context used by the logger.
  3. Log levels:
    • Set global level to INFO.
    • Use WARNING for slow requests or retries.
    • Use ERROR and CRITICAL for real failures.
  4. Centralization:
    • Use Docker logging driver or a small agent to send container logs to a central log system (for example, CloudWatch, Loki, or Elastic).
    • Set a retention policy (for example, 30 days for general logs, 90 days for errors).
  5. Security:
    • Filter out or mask sensitive fields from request/response logs.
    • Restrict access to the logging system to the operations and backend team.
  6. Documentation:
    • Write a short document that explains:
      • Which events are logged.
      • Where to find logs for each environment.
      • How to search by request_id or user_id.
      • What each log level means in your system.

Summary

Production logging is not just “print some messages.” It is:

If you design production logging carefully before deployment, you will save hours or days every time something goes wrong in production.

Views: 6

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3000 / 24h
Impressum / Imprint
Datenschutz / Data Policy