KAHIBARO
Discord Login Register ☰
☰

20.3. Structured Logging

Table of Contents

Why Structured Logging Matters

Traditional logs are often just plain text lines. They look readable to humans, but they are hard for tools to search, filter, and analyze in a reliable way.

Structured logging means that each log entry has a clear machine readable structure, usually as key value pairs, most commonly JSON. This makes your logs much more useful in real applications.

Some examples of where structure helps:

Key idea: Structured logging turns every log entry into data, not just text.

In backend systems that run across multiple instances and services, structured logging is not optional. It is the only realistic way to make sense of large volumes of logs.

Unstructured vs Structured Logs

Let us compare a simple example.

Unstructured log

text 
[2026-08-01 10:15:23] ERROR: Failed to create order for user 42: stock not available

To a human this is clear. To a machine, it is just one long string. If you want to find all logs for user 42, you must do a text search for user 42 and hope that the format never changes.

Structured log

The same event as JSON:

json 
{
  "timestamp": "2026-08-01T10:15:23.456Z",
  "level": "ERROR",
  "message": "Failed to create order: stock not available",
  "user_id": 42,
  "order_id": null,
  "product_id": "P-123",
  "stock_available": 0,
  "service": "orders",
  "endpoint": "POST /orders"
}

Now a log system can:

Side by side comparison

AspectUnstructured textStructured log (JSON)
Human readabilityGoodGood (if formatted)
Machine parsingHard, fragileEasy, reliable
Field evolutionBreaks parsersAdd new keys without breaking old ones
SearchingFull text search onlyFilter by fields, combine with conditions
AnalyticsLimited, manualRich, can aggregate by fields

Rule: Prefer structured logs for any backend that might need centralized logging, search, or metrics.

Log Fields and Context

The power of structured logs comes from consistent fields. You still have a human readable message, but you add context as separate keys.

Common base fields

Most backends benefit from including these fields in every log entry:

FieldDescriptionExample
timestampWhen the event happened2026-08-01T10:15:23.456Z
levelSeverity (DEBUG, INFO, WARN, ERROR)INFO
messageHuman readable descriptionUser logged in
serviceName of the serviceauth-service
envEnvironmentprod, staging, dev
hostnameMachine or container nameapi-1, pod-xyz
loggerLogger name / moduleauth.handlers

Request and user context

For web backends, add context for each incoming request and user:

FieldDescriptionExample
request_idUnique id per requestreq-8f2c5a...
trace_idCorrelation id across servicestrace-12ab...
methodHTTP methodPOST
pathRequest path/orders
routeLogical route namePOST /orders
status_codeResponse status201
duration_msTime taken to handle the request123.4
user_idAuthenticated user id42
ipClient IP address203.0.113.10
user_agentClient user agentMozilla/5.0 ...

Example with request context:

json 
{
  "timestamp": "2026-08-01T10:16:01.111Z",
  "level": "INFO",
  "message": "Request completed",
  "service": "orders",
  "env": "prod",
  "request_id": "req-01H7R9WFZ4W6C7HAXKD4",
  "trace_id": "trace-7b8c123",
  "method": "POST",
  "path": "/orders",
  "status_code": 201,
  "duration_ms": 87.32,
  "user_id": 42
}

Rule: Always add correlation fields like request_id or trace_id so you can follow a single request across many log lines and services.

Domain specific context

You should also add fields that make sense for your domain, for example:

Example for a failed payment:

json 
{
  "timestamp": "2026-08-01T10:20:00.000Z",
  "level": "ERROR",
  "message": "Payment declined",
  "service": "payments",
  "env": "prod",
  "user_id": 42,
  "order_id": "ORD-1001",
  "payment_id": "PAY-987",
  "amount": 49.99,
  "currency": "USD",
  "gateway": "stripe",
  "reason": "insufficient_funds",
  "request_id": "req-01H7R9..."
}

JSON Logging in Practice

Most modern backends log in JSON. The exact API depends on the language, but the pattern is always similar.

You will see two main shapes:

  1. Plain JSON per line
  2. JSON inside a wrapper format from the logging library

Example: simple Python JSON logs

A minimal pattern, not using any special libraries:

python 
import json
import sys
from datetime import datetime, timezone
def log(level, message, **fields):
    entry = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "level": level,
        "message": message,
        **fields,
    }
    sys.stdout.write(json.dumps(entry) + "\n")
    sys.stdout.flush()
log("INFO", "User logged in", user_id=42, service="auth", env="dev")

This produces:

json 
{"timestamp": "2026-08-01T10:30:00.123456+00:00", "level": "INFO", "message": "User logged in", "user_id": 42, "service": "auth", "env": "dev"}

Each line is a complete JSON object. Log collectors like Fluent Bit or Filebeat can easily parse it.

Logging frameworks with JSON formatters

In real applications you use a logging framework, but you configure it to output JSON.

High level steps are similar for any language:

  1. Configure a formatter that outputs JSON
  2. Attach that formatter to handlers (console, file, etc.)
  3. Log using regular methods, but pass extra fields as structured data

Pseudo code to show the concept:

python 
logger.info(
    "Created order",
    extra={
        "order_id": "ORD-1001",
        "user_id": 42,
        "amount": 49.99,
    },
)

Your logging configuration then embeds those extra fields directly in the JSON.

One log event per line

A very important detail: keep one JSON object per line in your log output. This is often called JSON Lines or NDJSON.

Correct:

text 
{"level": "INFO", "message": "a"} 
{"level": "INFO", "message": "b"}
{"level": "ERROR", "message": "c"}

Incorrect:

text 
[
  {"level": "INFO", "message": "a"},
  {"level": "INFO", "message": "b"}
]

Log systems almost always expect each line to be a full event.

Rule: Write exactly one JSON object per line. Do not pretty print JSON in logs in production.

Designing a Log Schema

A log schema is a convention for what fields appear in your logs and how they are named.

Why a schema matters

Without a schema, each team or developer invents their own field names:

This makes search and analysis painful.

With a schema:

Basic schema example

You do not need something huge. Start simple. For a small backend you might define a schema like:

CategoryFields
Coretimestamp, level, message
Environment / hostservice, env, hostname, version
HTTP requestrequest_id, method, path, status_code, duration_ms, ip, user_agent
User / authuser_id, session_id, auth_method
Error / exceptionerror_type, error_message, stack_trace

You can then document this schema for your team.

Example log that fits the schema:

json 
{
  "timestamp": "2026-08-01T11:00:00.000Z",
  "level": "ERROR",
  "message": "Unhandled exception while processing request",
  "service": "orders",
  "env": "prod",
  "hostname": "api-3",
  "version": "1.4.0",
  "request_id": "req-123",
  "method": "POST",
  "path": "/orders",
  "status_code": 500,
  "duration_ms": 102.7,
  "ip": "203.0.113.5",
  "user_agent": "Mozilla/5.0 ...",
  "user_id": 42,
  "error_type": "DatabaseError",
  "error_message": "connection timeout",
  "stack_trace": "Traceback (most recent call last): ..."
}

Naming conventions

Simple rules keep things consistent:

Rule: Design your log fields like you design a database schema, with consistent names and clear types.

Adding Context Without Repeating Yourself

It is common to have values that should appear in every log within a request, such as request_id, user_id, service, and env.

You do not want to manually add them to every log call:

python 
logger.info("Something", extra={"request_id": request_id, "user_id": user_id, ...})

Instead, use per request context.

Concept: logging context

The idea is:

In different languages this has different names:

Conceptually it behaves like:

python 
with log_context(request_id="req-123", user_id=42):
    logger.info("Started processing")
    logger.info("Doing step 1")
    logger.error("Something failed")

All three lines will receive request_id and user_id automatically in the JSON.

Example outputs for all three log lines:

json 
{"level": "INFO", "message": "Started processing", "request_id": "req-123", "user_id": 42}
{"level": "INFO", "message": "Doing step 1", "request_id": "req-123", "user_id": 42}
{"level": "ERROR", "message": "Something failed", "request_id": "req-123", "user_id": 42}

This pattern is essential in web backends, where a single request can trigger many log entries.

Logging Levels and Structured Data

Structured logging does not change the semantics of log levels, but it makes them easier to analyze.

Typical levels:

With structured logs, you can:

Example, two error logs with different types:

json 
{
  "timestamp": "2026-08-01T11:10:00.000Z",
  "level": "ERROR",
  "message": "Database connection failed",
  "service": "orders",
  "error_type": "DatabaseError",
  "retryable": true
}
json 
{
  "timestamp": "2026-08-01T11:10:01.000Z",
  "level": "ERROR",
  "message": "User permission denied",
  "service": "orders",
  "error_type": "AuthorizationError",
  "retryable": false,
  "user_id": 42
}

You can build alerts like:

This becomes trivial with structured data.

Privacy and Security in Structured Logs

Structured logs are easy to search and export, which is very powerful but also risky.

You must be careful not to log sensitive information.

Examples of data you should avoid logging:

Logs that break this rule are dangerous because:

Redacting fields

A common pattern is to define sensitive fields and redact them automatically.

Example of redaction:

Input data:

json 
{
  "email": "user@example.com",
  "password": "super-secret",
  "card_number": "4111111111111111"
}

Redacted in logs:

json 
{
  "email": "user@example.com",
  "password": "***REDACTED***",
  "card_number": "***REDACTED***"
}

You can apply redaction when:

Rule: Never log secrets, passwords, or full payment data. If in doubt, do not log the value, or redact it.

Using Structured Logs in Aggregation Systems

In many production systems, your application logs to standard output. A separate agent collects these logs and sends them to a central place.

Examples of central systems:

Structured logging makes life easier:

Example aggregation query

If your logs look like this:

json 
{"timestamp": "...", "level": "ERROR", "service": "orders", "error_type": "DatabaseError"}
{"timestamp": "...", "level": "ERROR", "service": "orders", "error_type": "ValidationError"}
{"timestamp": "...", "level": "INFO", "service": "auth"}

You can query:

You did not need special parsing logic. You just use the structured fields.

Practical Examples of Structured Log Entries

Here are several complete example entries to show realistic patterns.

Successful request log

json 
{
  "timestamp": "2026-08-01T11:30:00.000Z",
  "level": "INFO",
  "message": "Request completed",
  "service": "orders",
  "env": "prod",
  "request_id": "req-abc123",
  "trace_id": "trace-xyz789",
  "method": "POST",
  "path": "/orders",
  "route": "POST /orders",
  "status_code": 201,
  "duration_ms": 95.3,
  "user_id": 42,
  "ip": "203.0.113.7"
}

Business event log

json 
{
  "timestamp": "2026-08-01T11:31:00.000Z",
  "level": "INFO",
  "message": "Order created",
  "service": "orders",
  "env": "prod",
  "request_id": "req-abc123",
  "user_id": 42,
  "order_id": "ORD-1005",
  "amount": 89.00,
  "currency": "USD",
  "items_count": 3,
  "payment_method": "credit_card"
}

Background job log

json 
{
  "timestamp": "2026-08-01T11:32:00.000Z",
  "level": "INFO",
  "message": "Job executed",
  "service": "worker",
  "env": "prod",
  "job_type": "send_email",
  "job_id": "job-7788",
  "attempt": 1,
  "duration_ms": 210.5,
  "status": "success",
  "email_type": "order_confirmation",
  "user_id": 42,
  "order_id": "ORD-1005"
}

Error with stack trace

json 
{
  "timestamp": "2026-08-01T11:33:00.000Z",
  "level": "ERROR",
  "message": "Failed to reserve stock",
  "service": "inventory",
  "env": "prod",
  "request_id": "req-xyz555",
  "order_id": "ORD-1006",
  "product_id": "P-999",
  "quantity": 2,
  "error_type": "StockReservationError",
  "error_message": "Not enough stock available",
  "stack_trace": "Traceback (most recent call last): ...",
  "retryable": false
}

In all these examples, you can see that:

Summary

Structured logging treats logs as data, not just text. By logging in a consistent structured format, usually JSON, with predictable fields, you can:

To use structured logging effectively in backends:

These practices make logging a powerful tool for understanding, debugging, and operating backend systems.

Views: 15

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3040 / 24h
Impressum / Imprint
Datenschutz / Data Policy