KAHIBARO Bookshelves
Discord Login Register

12.2. Metrics and alerts

Types of Metrics in OpenShift

In OpenShift, most “monitoring” data you work with falls into a small set of metric types. Understanding them helps you create meaningful alerts instead of noisy ones.

Core metric types

  • Counter
    • Monotonically increasing value (only goes up, or resets).
    • Examples:
      • container_cpu_usage_seconds_total
      • http_requests_total
    • Typical use: rates per second or per minute using PromQL rate() or irate().
  • Gauge
    • Value that can go up and down.
    • Examples:
      • node_memory_MemAvailable_bytes
      • kube_pod_container_status_restarts_total (often behaves like a counter but is modeled as a gauge)
      • cluster:usage:cpu_cores:sum
    • Typical use: snapshots, thresholds (e.g., alert if > X).
  • Histogram
    • Measures distributions of observations (size, latency, duration).
    • Implemented as multiple series with buckets, e.g. _bucket, _sum, *_count.
    • Examples:
      • apiserver_request_duration_seconds_bucket
    • Typical use: SLIs like response time percentiles (p90, p95, p99).
  • Summary
    • Similar to histogram but calculates quantiles on the client side.
    • Less commonly used for cluster-level alerting in OpenShift; histograms are preferred.

High-level categories of metrics

  • Platform / cluster metrics
    • Collected by the cluster monitoring stack, focused on:
      • Nodes (CPU, memory, disk, network)
      • Kubernetes components (API server, controller manager, etcd)
      • OpenShift components (SDN, OAuth server, Router, etc.)
    • Primary for platform SRE / cluster admins.
  • Application metrics
    • Emitted by workloads you deploy in namespaces.
    • Typical patterns:
      • Business metrics (orders processed, jobs completed).
      • Technical metrics (request latency, error rate, queue depth).
    • Scraped by user workload monitoring (if configured) or external Prometheus.

Metrics Collection and Exposure

Prometheus as the core metrics engine

OpenShift’s metrics and alerting are centered on Prometheus:

  • Cluster Monitoring stack (managed by the Cluster Monitoring Operator):
    • Operates in openshift-monitoring.
    • Contains:
      • Prometheus instances (platform metrics)
      • Alertmanager
      • Thanos components (in some configurations)
      • Grafana (depending on version and configuration)
  • User Workload Monitoring:
    • Optional.
    • For scraping and alerting on application metrics in user namespaces, managed under openshift-user-workload-monitoring when enabled.

How metrics get into Prometheus

Prometheus relies on a pull model (scraping):

  • Targets expose metrics on an HTTP endpoint, typically /metrics, in Prometheus text format.
  • Prometheus scrapes these endpoints on a fixed interval (e.g., every 30s).

In OpenShift, you generally don’t hand-edit prometheus.yml. Instead, you use:

  • ServiceMonitor
    • Custom resource (CRD) that describes how to scrape metrics from a Service.
    • Example fields:
      • selector (which Services to match)
      • endpoints (path, port, scheme, interval)
    • Belongs to user workload monitoring; cluster admins manage the cluster-level equivalents.
  • PodMonitor
    • Similar to ServiceMonitor but targets Pods directly.
    • Useful when you don’t expose a Service or want pod-level selection.
  • Scrape configurations for platform components
    • Managed automatically by the Cluster Monitoring Operator for core OpenShift components (you do not modify these directly).

Example: ServiceMonitor (conceptual example)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-monitor
  namespace: my-namespace
spec:
  selector:
    matchLabels:
      app: my-app
  namespaceSelector:
    matchNames:
      - my-namespace
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Working with Metrics via the OpenShift Console

The OpenShift web console provides multiple entry points for metrics:

  • Observe → Metrics:
    • PromQL query interface.
    • You can:
      • Run queries against the cluster Prometheus.
      • View graphs over time.
      • Inspect series labels.
  • Workloads and Nodes detail pages (dashboards):
    • Surface key metrics (CPU, memory, network, restarts) for:
      • Pods
      • Deployments
      • Nodes
    • These views are built on top of Prometheus queries.

Metrics visible here are primarily platform-level metrics; application metrics appear when user workload monitoring and the relevant ServiceMonitor/PodMonitor are configured.

Alerting Concepts in OpenShift

Alerts in OpenShift are defined in Prometheus alerting rules and handled by Alertmanager. The Cluster Monitoring Operator manages both for the platform stack.

Key components

  • Alerting rules
    • Expressions written in PromQL that evaluate metrics and produce alert states.
    • Each rule defines:
      • A name
      • A PromQL expression
      • A for duration (optional)
      • Labels (severity, scope, etc.)
      • Annotations (human-readable message, runbook link)
  • Alertmanager
    • Receives alerts from Prometheus.
    • Responsible for:
      • Grouping alerts
      • Silencing (temporarily disabling notifications)
      • Routing to receivers (email, webhooks, etc.)
      • Deduplication and inhibition
  • Alerting integration in the console
    • Observe → Alerts shows:
      • Active, pending, and silenced alerts
      • Alert details, labels, history
    • You can:
      • Filter by severity, namespace, or component
      • Create silences for specific alert label sets

PromQL for Alerting

PromQL (Prometheus Query Language) is used both for dashboards and alerts. For alerts, you usually:

  1. Transform raw metrics into a useful signal, often using:
    • rate() / irate() for Counters
    • sum, avg, max, min, count for aggregation
    • histogram_quantile() for latency metrics from histograms
  2. Apply a condition:
    • Compare against thresholds:
      • $metric > threshold
    • Check ratios:
      • $error_rate / $total_requests > 0.05
  3. Use label filters to scope what you alert on:
    • metric{namespace="my-namespace", app="my-app"}

Common helper examples:

  • Rate of HTTP requests over 5 minutes:
    $$ \text{rate}(http\_requests\_total[5m]) $$
  • 5xx error ratio:
    $$
    \frac{
    \text{rate}(http\_requests\_total\{code=~"5.."}[5m])
    }{
    \text{rate}(http\_requests\_total[5m])
    }
    $$
  • p95 latency from a histogram over 5 minutes:
    $$
    \text{histogram\_quantile}\left(
    0.95,
    \sum \text{ by }(le) (
    \text{rate}(request\_duration\_seconds\_bucket[5m])
    )
    )
    $$

Defining Alerting Rules

Cluster-level alerting rules for OpenShift components are managed by the platform. For extending alerting (especially for applications), OpenShift uses Kubernetes custom resources:

  • PrometheusRule CRD:
    • Declares groups of alerting and recording rules.
    • Managed by the Cluster Monitoring Operator (for platform) and, in some configurations, by user workload monitoring (for apps).

Structure of an alert rule

An alert rule contains:

  • alert – alert name (identifier)
  • expr – PromQL expression
  • for – required duration the condition must hold before firing
  • labels – metadata for routing and classification (e.g. severity, namespace)
  • annotations – human-friendly information (summary, description, runbook)

Conceptual example (application-oriented):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-app-rules
  namespace: my-namespace
spec:
  groups:
    - name: my-app-availability
      rules:
        - alert: MyAppHighErrorRate
          expr: |
            (
              sum(rate(http_requests_total{app="my-app", status=~"5.."}[5m]))
            /
              sum(rate(http_requests_total{app="my-app"}[5m]))
            ) > 0.05
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "High error rate for my-app"
            description: "More than 5% of my-app requests have been failing for 10 minutes."

This example illustrates typical alert design practices:

  • Uses rate() for counters over a 5-minute window.
  • Compares a ratio, not raw counts (helps on low traffic).
  • Requires the condition to hold for: 10m to reduce flapping.
  • Labels include severity for downstream routing.

Alert Lifecycle and States

Prometheus internally tracks alert states before passing them to Alertmanager:

  • Inactive
    • Condition is not met (expr evaluates to false or empty).
  • Pending
    • Condition is true, but the for duration has not yet elapsed.
  • Firing
    • Condition has been true for at least the for duration.

Alertmanager processes firing alerts:

  • Groups them (e.g. by namespace and alertname).
  • Applies routing rules and templates.
  • Sends notifications to receivers.
  • Allows silencing and inhibition rules (e.g. suppressing pod-level alerts when a node-level alert indicates a higher-level issue).

In the console under Observe → Alerts, you can see:

  • Which alerts are firing or pending.
  • Their labels and annotations.
  • History of state changes.

Metrics, SLIs, SLOs, and Alerting

Metrics form the basis of Service Level Indicators (SLIs), which you use to implement Service Level Objectives (SLOs). Alerts should typically be SLO-driven, not just raw metric thresholds.

Typical SLIs in OpenShift environments:

  • Availability / error rate
    • Example SLI: ratio of successful to total requests.
    • Metrics: http_requests_total, status code labels.
  • Latency
    • Example SLI: 95th percentile request duration.
    • Metrics: histograms like *_duration_seconds_bucket.
  • Resource saturation
    • CPU, memory, disk I/O, and queue depths.
    • Metrics: node and pod resource metrics, work queue metrics.

SLO-aligned alerting patterns:

  • Fast-burn alerts
    • Trigger if you will violate your SLO quickly (e.g., very high error rate).
  • Slow-burn alerts
    • Trigger if you are slowly consuming your error budget over a longer time.

These patterns are implemented using different PromQL windows and thresholds, all backed by the same base metrics.

Best Practices for Metrics and Alerts in OpenShift

Metric design

  • Expose low-cardinality labels
    • Avoid labels with unbounded values (e.g., user IDs, request IDs).
    • High cardinality can overload Prometheus and increase storage cost.
  • Use consistent naming and units
    • Suffix metrics with units: _seconds, _bytes, _total.
    • Make it obvious whether it’s a counter or gauge.
  • Instrument business-relevant metrics
    • Don’t limit yourself to CPU/memory; track rates that matter to your users (jobs, transactions, messages processed).

Alert design

  • Prefer symptom-based alerts over cause-based
    • Example:
      • Alert on “application error rate high” rather than “pod restarted once.”
    • Use platform alerts for underlying health (nodes, etcd, API server) and application alerts for user-visible symptoms.
  • Avoid noisy alerts
    • Always use appropriate windows and for durations.
    • Consider traffic volume: percentages or ratios instead of absolute counts.
  • Use severities consistently
    • E.g.:
      • critical for user-visible outages or data loss.
      • warning for degraded service or approaching capacity.
      • info for non-urgent signals.
  • Include clear runbooks or hints
    • Use annotations.description to:
      • Explain likely impact.
      • Suggest first troubleshooting steps.
      • Link to internal docs if available.

Operational considerations

  • Be aware of the scope:
    • Platform alerts (managed by OpenShift) vs application alerts (your responsibility).
    • Avoid duplicating platform-level alerts in your own rules.
  • Observe resource usage of metrics and alerts:
    • Too many extremely detailed metrics or high-scrape-frequency targets can strain the Prometheus instances.
    • Tune:
      • Scrape intervals
      • Retention periods
      • Label cardinality
  • Validate alerts before relying on them:
    • Use the console Metrics explorer to:
      • Run the alert expression as a query.
      • Check that it behaves as expected during normal and failure scenarios.
    • Dry-run new alerts by setting them to lower severity or only logging initially.

Putting It Together in an OpenShift Environment

In practice, using metrics and alerts on OpenShift typically looks like:

  1. Instrument your application with metrics (e.g. Prometheus client libraries).
  2. Expose /metrics and create a Service pointing to that port.
  3. Create a ServiceMonitor so user workload monitoring scrapes your app.
  4. Explore metrics using the console’s Metrics tab, refine PromQL queries.
  5. Define PrometheusRule objects with well-designed alerting rules.
  6. Use the Alerts view to:
    • Observe alert states.
    • Tune thresholds and durations.
    • Create silences during maintenance.

This workflow builds on the OpenShift monitoring stack that is already collecting cluster metrics and shipping default alerts, allowing you to extend it with application-specific metrics and alerting tailored to your workloads.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Course Overview
    • ☰ 1.1. What is OpenShift
      • ☰ 1.2. Why OpenShift is used in modern IT and cloud environments
        • ☰ 1.3. OpenShift vs Kubernetes
          • ☰ 1.4. Typical OpenShift use cases
            • ☰ 1.5. Structure and goals of the course
            • ☰ 2. Containers and Cloud-Native Fundamentals
              • ☰ 2.1. What is a container
                • ☰ 2.2. Containers vs virtual machines
                  • ☰ 2.3. Introduction to Docker and container images
                    • ☰ 2.4. Container registries
                      • ☰ 2.5. Basics of cloud-native applications
                      • ☰ 3. Kubernetes Fundamentals
                        • ☰ 3.1. What is Kubernetes
                          • ☰ 3.2. Kubernetes architecture
                            • ☰ 3.3. Control plane components
                              • ☰ 3.4. Worker node components
                                • ☰ 3.5. Kubernetes objects and resources
                                  • ☰ 3.6. Declarative configuration model
                                  • ☰ 4. OpenShift Architecture
                                    • ☰ 4.1. Overview of OpenShift architecture
                                      • ☰ 4.2. Control plane in OpenShift
                                        • ☰ 4.3. Nodes and machine roles
                                          • ☰ 4.4. OpenShift Operators
                                            • ☰ 4.5. OpenShift networking model
                                              • ☰ 4.6. OpenShift storage model
                                              • ☰ 5. OpenShift Installation and Deployment Models
                                                • ☰ 5.1. OpenShift deployment options
                                                  • ☰ 5.2. Installer-Provisioned Infrastructure
                                                    • ☰ 5.3. User-Provisioned Infrastructure
                                                      • ☰ 5.4. Managed OpenShift services
                                                        • ☰ 5.5. Single-node OpenShift
                                                          • ☰ 5.6. Cluster lifecycle management
                                                          • ☰ 6. Accessing and Using OpenShift
                                                            • ☰ 6.1. OpenShift web console
                                                              • ☰ 6.2. OpenShift CLI (oc)
                                                                • ☰ 6.3. Authentication and authorization
                                                                  • ☰ 6.4. Projects and namespaces
                                                                    • ☰ 6.5. Resource quotas and limits
                                                                    • ☰ 7. Building and Deploying Applications
                                                                      • ☰ 7.1. Application deployment concepts
                                                                        • ☰ 7.2. Source-to-Image (S2I)
                                                                          • ☰ 7.3. Container-based deployments
                                                                            • ☰ 7.4. DeploymentConfigs and Deployments
                                                                              • ☰ 7.5. Rolling updates and rollbacks
                                                                              • ☰ 8. Networking in OpenShift
                                                                                • ☰ 8.1. Services and service discovery
                                                                                  • ☰ 8.2. Routes and ingress
                                                                                    • ☰ 8.3. Network policies
                                                                                      • ☰ 8.4. Load balancing
                                                                                        • ☰ 8.5. External access to applications
                                                                                        • ☰ 9. Storage in OpenShift
                                                                                          • ☰ 9.1. Persistent storage concepts
                                                                                            • ☰ 9.2. Persistent Volumes
                                                                                              • ☰ 9.3. Persistent Volume Claims
                                                                                                • ☰ 9.4. StorageClasses
                                                                                                  • ☰ 9.5. Dynamic provisioning
                                                                                                    • ☰ 9.6. Stateful applications
                                                                                                    • ☰ 10. Configuration and Secrets Management
                                                                                                      • ☰ 10.1. ConfigMaps
                                                                                                        • ☰ 10.2. Secrets
                                                                                                          • ☰ 10.3. Environment variables
                                                                                                            • ☰ 10.4. Managing sensitive data
                                                                                                              • ☰ 10.5. Best practices for configuration
                                                                                                              • ☰ 11. Scaling and High Availability
                                                                                                                • ☰ 11.1. Horizontal pod autoscaling
                                                                                                                  • ☰ 11.2. Vertical scaling
                                                                                                                    • ☰ 11.3. Self-healing and pod restarts
                                                                                                                      • ☰ 11.4. High availability concepts
                                                                                                                        • ☰ 11.5. Multi-zone and multi-region clusters
                                                                                                                        • ☰ 12. Monitoring, Logging, and Observability
                                                                                                                          • ☰ 12.1. Built-in monitoring stack
                                                                                                                            • ☰ 12.2. Metrics and alerts
                                                                                                                              • ☰ 12.3. Logging architecture
                                                                                                                                • ☰ 12.4. Distributed tracing
                                                                                                                                  • ☰ 12.5. Troubleshooting applications
                                                                                                                                  • ☰ 13. Security in OpenShift
                                                                                                                                    • ☰ 13.1. Security model overview
                                                                                                                                      • ☰ 13.2. Role-Based Access Control
                                                                                                                                        • ☰ 13.3. Security Context Constraints
                                                                                                                                          • ☰ 13.4. Image security and scanning
                                                                                                                                            • ☰ 13.5. Network security
                                                                                                                                              • ☰ 13.6. Compliance and auditing
                                                                                                                                              • ☰ 14. CI/CD and DevOps with OpenShift
                                                                                                                                                • ☰ 14.1. DevOps concepts
                                                                                                                                                  • ☰ 14.2. CI/CD pipelines overview
                                                                                                                                                    • ☰ 14.3. OpenShift Pipelines (Tekton)
                                                                                                                                                      • ☰ 14.4. GitOps with Argo CD
                                                                                                                                                        • ☰ 14.5. Integrating OpenShift with external CI tools
                                                                                                                                                        • ☰ 15. Operators and Platform Services
                                                                                                                                                          • ☰ 15.1. What are Operators
                                                                                                                                                            • ☰ 15.2. Operator Lifecycle Manager
                                                                                                                                                              • ☰ 15.3. Installing and managing Operators
                                                                                                                                                                • ☰ 15.4. Common platform Operators
                                                                                                                                                                  • ☰ 15.5. Custom Operators
                                                                                                                                                                  • ☰ 16. Upgrades, Maintenance, and Operations
                                                                                                                                                                    • ☰ 16.1. OpenShift upgrade process
                                                                                                                                                                      • ☰ 16.2. Backup and restore
                                                                                                                                                                        • ☰ 16.3. Node maintenance
                                                                                                                                                                          • ☰ 16.4. Cluster troubleshooting
                                                                                                                                                                            • ☰ 16.5. Capacity planning
                                                                                                                                                                            • ☰ 17. OpenShift on HPC and Specialized Workloads
                                                                                                                                                                              • ☰ 17.1. Running batch workloads on OpenShift
                                                                                                                                                                                • ☰ 17.2. OpenShift and GPUs
                                                                                                                                                                                  • ☰ 17.3. MPI and parallel workloads in containers
                                                                                                                                                                                    • ☰ 17.4. Hybrid HPC and cloud workflows
                                                                                                                                                                                    • ☰ 18. OpenShift in Practice
                                                                                                                                                                                      • ☰ 18.1. Typical OpenShift workflows
                                                                                                                                                                                        • ☰ 18.2. Development to production lifecycle
                                                                                                                                                                                          • ☰ 18.3. Case studies from industry
                                                                                                                                                                                            • ☰ 18.4. Common pitfalls and best practices
                                                                                                                                                                                            • ☰ 19. Future Directions and Ecosystem
                                                                                                                                                                                              • ☰ 19.1. Kubernetes ecosystem overview
                                                                                                                                                                                                • ☰ 19.2. OpenShift roadmap
                                                                                                                                                                                                  • ☰ 19.3. Serverless and event-driven workloads
                                                                                                                                                                                                    • ☰ 19.4. AI and data platforms on OpenShift
                                                                                                                                                                                                    • ☰ 20. Final Project and Hands-On Exercises
                                                                                                                                                                                                      • ☰ 20.1. Deploying a containerized application
                                                                                                                                                                                                        • ☰ 20.2. Implementing CI/CD with OpenShift
                                                                                                                                                                                                          • ☰ 20.3. Securing and scaling an application
                                                                                                                                                                                                            • ☰ 20.4. Monitoring and troubleshooting report