KAHIBARO
Discord Login Register ☰
☰

20.13. Observability Basics

Table of Contents

Why Observability Matters

When your backend runs on your laptop, you can watch the console, restart quickly, and debug by trial and error. In production, your app runs on remote servers, often in multiple containers, behind load balancers, with many concurrent users.

You cannot just “ssh in and print stuff” whenever something goes wrong. You need a way to understand what is happening inside your system from the outside.

That is what observability is about: designing systems so that you can ask questions about their behavior and get useful answers, even for problems you did not predict in advance.

Key idea:
Observability is the ability to understand the internal state of a system by looking at its outputs, without changing the running system.

Good observability lets you:

You will use three main types of data, often called the three pillars of observability: logs, metrics, and traces.

The Three Pillars of Observability

Logs

Logs are timestamped text records that describe events that happened in your application or infrastructure.

Examples:

Typical log fields:

FieldExample
Timestamp2026-08-28T09:32:15.123Z
LevelINFO, WARNING, ERROR
Serviceorders-service
MessageOrder created
Context{"order_id": 456, "user_id": 123}

Logs are great for:

However, raw logs can be large and hard to search. This is why we also use metrics.

Metrics

Metrics are numeric measurements over time.

Examples:

Metrics are usually stored as time series:

TimeValue (Requests/sec)
10:00:00120
10:00:10135
10:00:20160

Metrics are great for:

Metrics are compact and easy to visualize, but they do not show detailed “per request” information. That is where traces help.

Traces

Traces track the path of a single request as it flows through your system.

Consider a user calling /api/create-order. Internally this may call:

  1. Orders service
  2. User service
  3. Inventory service
  4. Payments service
  5. Database

A trace links all these operations together. Each step in the trace is a span with information like:

A simple trace structure:

SpanParentDuration (ms)
HTTP request(root)300
Orders serviceHTTP request200
DB queryOrders service50

Traces are great for:

Together, logs, metrics, and traces give you different views of the same reality.

Important:

  • Logs answer: “What happened?”
  • Metrics answer: “How often and how much?”
  • Traces answer: “Where in the flow did it happen?”

Observability vs Monitoring

The terms monitoring and observability are related but not the same.

You can think of it like this:

ConceptFocus
MonitoringKnown problems, predefined checks
ObservabilityUnknown problems, open‑ended questions

In practice:

You will often see teams say “we are improving observability” when they:

Core Concepts: Signals, Context, and Correlation

To make observability useful, three ideas are important: signals, context, and correlation.

Signals

A signal is any piece of data that describes your system’s behavior. The main signals are:

Some people also include:

As a beginner backend developer you should focus on the three main signals first.

Context

Context is extra information that makes a signal meaningful.

A log message "Payment failed" is not very helpful alone. With context it becomes usable:

json 
{
  "timestamp": "2026-08-28T10:00:00Z",
  "level": "ERROR",
  "message": "Payment failed",
  "user_id": 123,
  "order_id": 456,
  "payment_provider": "stripe",
  "error_code": "card_declined",
  "trace_id": "abc-123"
}

You will often add context like:

Context is what turns a pile of data into something you can reason about.

Correlation

Correlation means linking different signals that refer to the same thing.

Example questions:

To do this you usually:

This is one of the simplest but most powerful practices for observability.

Rule:
Always include a request / trace ID in logs and propagate it across services.
This makes it possible to follow a single request across the whole system.

Basic Observability Setup for a Backend

You do not need a complex stack on day one. Start small and grow over time.

A typical beginner‑friendly setup looks like this:

LayerExample ToolPurpose
LogsApplication logs to stdoutSee what the app is doing
Log collectionDocker logging / simple agentSend logs to a file or service
MetricsPrometheus + client libraryCollect app and system metrics
DashboardsGrafanaVisualize metrics over time
TracesOpenTelemetry + JaegerTrace requests across services

You might not use all of these at first, but you can still follow the ideas:

Designing for Observability

Good observability is not something you “bolt on at the end.” It is part of your design.

Here are simple practices you can apply as a beginner.

Log Important Events, Not Everything

Do not log every tiny detail such as every loop iteration. Focus on:

Example for an order creation flow:

  1. Receive request to create order.
  2. Validate input.
  3. Check inventory.
  4. Charge payment.
  5. Save order.
  6. Return success.

You might log:

This gives you a clear story when something goes wrong.

Use Log Levels Wisely

Common levels:

LevelWhen to use
DEBUGDetailed information for local debugging
INFONormal events in the app’s lifecycle
WARNINGSomething unusual, but the app can still continue
ERRORAn operation failed, but the app is still running
CRITICALThe application or a major part is unusable

Example:

You will often configure production to ignore DEBUG logs, but still store INFO and above.

Add Metrics to Critical Paths

Identify your most important features and add metrics, for example:

These metrics help answer questions like:

Trace Requests Across Services

If your application is small and monolithic, you might delay full tracing. As soon as you have:

it becomes worth introducing distributed tracing.

The basic idea:

Even simple traces can tell you things like, “The slowdown came from the payments service, not the database.”

From Symptoms to Root Cause

Observability is most useful when something breaks. Here is how the three pillars help you go from symptom to root cause.

Imagine you receive an alert: “Error rate > 5 percent on /api/orders for 10 minutes”.

  1. Start with metrics
    • Check the error rate and latency metrics for /api/orders.
    • Look for patterns:
      • Does it affect all regions or only one?
      • Did traffic suddenly increase?
      • Did it start after a deployment?
  2. Drill down into traces
    • Open traces for failed /api/orders requests.
    • Compare traces that succeeded vs failed.
    • Look for:
      • Spans that have errors.
      • Spans that take much longer than usual.
    • You might see that:
      • The payments-service span fails with HTTP 502.
      • The db.query span suddenly takes 2 seconds.
  3. Inspect logs for context
    • Filter logs by trace ID from a failing trace.
    • Read what the services logged during that request.
    • Look for stack traces, timeouts, or configuration issues.
    • Example findings:
      • Logs show that a new API key is invalid.
      • Logs show timeouts only in one region.

This flow is typical:

Common investigation flow:
Metric alert → Traces for failing requests → Logs for a specific trace → Root cause

As a backend developer, your job is to make sure the data you emit makes this flow possible.

Measuring and Improving Observability

You cannot directly measure “observability,” but you can watch outcomes related to it.

Common indicators:

When observability improves:

For your own practice on projects:

How This Connects to the Rest of the Course

In this course you will see observability ideas show up in many places:

For now, remember the fundamental mindset:

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3505 / 24h
Impressum / Imprint
Datenschutz / Data Policy