KAHIBARO Bookshelves
Discord Login Register

9.6. Stateful applications

Understanding Stateful Applications in OpenShift

Stateful applications are those where the data and identity of the application instances matter over time. In OpenShift, this means paying special attention to how pods are named, how storage is attached, and how data survives failures, reschedules, and upgrades.

This chapter focuses on how OpenShift supports stateful workloads on top of the storage primitives already introduced elsewhere (PersistentVolumes, PersistentVolumeClaims, StorageClasses, and dynamic provisioning).

What Makes an Application Stateful?

A workload is typically considered stateful if one or more of these apply:

  • It stores data that must persist beyond the life of a pod (databases, message queues, file stores).
  • Individual instances (replicas) are not interchangeable; each has an identity (e.g., db-0, db-1, db-2) and may own specific data.
  • The application maintains in‑memory or on‑disk state that is important for the correctness of the system (leader elections, consensus logs, checkpoints).
  • It needs stable network identities or well-known endpoints for peer-to-peer communication.

In contrast to stateless apps, you cannot safely scale or restart stateful apps without considering the impact on data consistency and availability.

Challenges of Running Stateful Workloads on OpenShift

Containers and Kubernetes/OpenShift were originally focused on stateless microservices. Stateful applications introduce additional challenges:

  • Pod lifecycle vs. data lifecycle: Pods are ephemeral; data must not be.
  • Scheduling constraints: Data may live on specific nodes or in specific zones; pods might need to follow that data.
  • Ordering and identity: Some distributed systems require that certain replicas start first, or that nodes keep a stable name over time.
  • Consistency and durability: Storage performance characteristics (latency, IOPS, durability guarantees) directly affect application behavior.
  • Backups and restore: You must treat persistent volumes as part of your backup strategy, not just application images.

OpenShift provides patterns and APIs to address these challenges while keeping deployment as declarative as possible.

StatefulSets: Core Pattern for Stateful Workloads

For stateful workloads, the primary OpenShift/Kubernetes workload resource is the StatefulSet. While Deployments/DeploymentConfigs are suitable for stateless applications, StatefulSets add features specifically for stateful scenarios.

Key properties of a StatefulSet:

  • Stable, ordinal pod identity
    Pods are named predictably, e.g., my-db-0, my-db-1, my-db-2. This ordinal index is preserved across restarts and reschedules.
  • Persistent volume association per replica
    Each replica can have its own persistent volume claim, typically using volumeClaimTemplates. The my-db-0 pod always mounts the same PVC, even if it is rescheduled to another node.
  • Ordered operations
    By default, StatefulSets support ordered pod creation, updates, and termination. For example:
    • Create my-db-0 first, then my-db-1, then my-db-2.
    • Terminate my-db-2 first, then my-db-1, then my-db-0.
      Some stateful systems rely on this for safe cluster bootstrapping and rolling updates.
  • Stable network identity
    Combined with a Headless Service (Service without a cluster IP), each pod gets a stable DNS name such as my-db-0.my-db-headless.my-namespace.svc.

In OpenShift, you create StatefulSets using the same YAML model as Kubernetes, and they integrate with the storage model (PVs, PVCs, StorageClasses) you already learned.

Typical StatefulSet Layout

A minimal pattern for a stateful application looks like this (conceptually):

  • A StatefulSet with:
    • A spec.serviceName referencing a headless service for stable DNS.
    • A spec.volumeClaimTemplates section so that each replica gets its own PVC.
    • A spec.replicas defining the number of instances.
  • A Service with clusterIP: None (headless) to provide per-pod DNS.

OpenShift then:

  1. Creates and attaches PVCs from the template, usually using dynamic provisioning.
  2. Ensures that pod-0 always uses pvc-0, pod-1 uses pvc-1, and so forth.
  3. Preserves these relationships across pod restarts and node changes.

Storage Design for Stateful Applications

Because stateful apps rely heavily on storage, your choice of storage backing and configuration is critical.

Access Modes and Their Impact

Different applications have different expectations about how their volumes can be used:

  • RWO (ReadWriteOnce): Only one node can mount the volume for read-write at a time.
    • Common for single-node databases like postgres, mysql.
    • Works well when each replica has its own volume.
  • RWX (ReadWriteMany): Multiple nodes can mount the volume read-write simultaneously.
    • Useful for shared file stores, content repositories, or apps requiring shared storage.
    • Often backed by NFS, CephFS, or similar.
  • ROX (ReadOnlyMany): Multiple nodes can mount read-only.
    • Less common for typical application data, more for shared reference data.

When defining volumeClaimTemplates in StatefulSets, ensure the requested accessModes match the application design and the capabilities of the underlying StorageClass.

Performance and Consistency Considerations

For stateful workloads, you should match the storage to the workload characteristics:

  • Databases: Usually need low-latency, high-IOPS, durable storage. Use storage classes designed for database workloads.
  • Message queues: Benefit from fast writes; evaluate throughput and durability trade-offs.
  • Analytics and batch: Often use large volumes; throughput may matter more than latency.

On OpenShift, this often means:

  • Choosing specialized StorageClasses (e.g., “fast-ssd”, “gp2/gp3”, “replicated”).
  • Tuning resources.requests and resources.limits on pods so the storage backend is not overloaded.
  • Understanding the replication and failure behavior of the underlying storage (e.g., Ceph, cloud block storage).

Pod Placement and Data Locality

For some stateful systems, where data is stored locally on nodes or where network topology matters, pod scheduling is important.

Common techniques:

  • Node affinity / anti-affinity
    Ensure certain replicas run on different nodes or zones for availability, or close to other services for performance.
  • Pod anti-affinity
    Spread replicas across nodes to avoid a single node failure impacting all instances.
  • Topology-aware provisioning
    Use storage classes that are aware of zones/regions, so volumes and pods are co-located to avoid cross-zone latency penalties.

In OpenShift, these are configured via the standard affinity and topologySpreadConstraints fields on StatefulSet pods, combined with storage classes that support topology.

Patterns for Common Stateful Applications

Relational Databases (MySQL/PostgreSQL)

Common pattern:

  • StatefulSet with 1 or a small number of replicas (master/primary and read replicas).
  • One PVC per replica backed by RWO block storage.
  • Application-specific replication (e.g., streaming replication) configured via environment variables and init scripts.
  • A Service pointing to the primary instance, possibly supplemented by Services for read replicas.

NoSQL Stores (Cassandra, MongoDB, etc.)

Typical needs:

  • Multiple replicas for data distribution.
  • Each replica with its own PVC and stable identity (e.g. cassandra-0, cassandra-1).
  • Peer discovery via headless services and the stable DNS names that StatefulSets provide.
  • Explicit handling of scaling up/down to avoid data loss or cluster imbalance.

File Services and Shared Storage

For workloads requiring shared file access:

  • Use RWX volumes (via appropriate StorageClasses).
  • Sometimes Deployment or DeploymentConfig is sufficient if the app does not need per-instance identity.
  • For applications combining shared and per-instance data, mix RWX and RWO volumes as needed.

Backup, Restore, and Data Protection

Stateful workloads require a strategy beyond just backing up container images and manifests.

Key aspects:

  • Volume snapshots
    Some storage backends support PVC snapshots via VolumeSnapshot resources. Use them to capture point-in-time copies of data volumes.
  • Application-consistent backups
    For databases and similar systems, coordinate snapshots with the application:
    • Flush or pause writes before snapshot.
    • Use native backup tools (e.g., pg_dump, mysqldump, logical backups) when consistency guarantees are required.
  • Disaster recovery strategies
    Common patterns on OpenShift:
    • Periodic export of application data to object storage (e.g., S3, internal object store).
    • Replication between clusters or regions, if the storage solution supports it.
  • Testing restores
    For stateful apps, regularly test restore procedures in a separate namespace or test cluster to ensure that both data and configuration restore correctly.

Scaling and Upgrades for Stateful Workloads

Scaling stateful apps is more complex than stateless ones.

Horizontal Scaling

When increasing replicas:

  • New pods will get new PVCs (for StatefulSets with volumeClaimTemplates).
  • The application may need explicit rebalancing (e.g., data redistribution in Cassandra, Kafka).
  • Some systems require configuration changes or cluster membership updates.

When decreasing replicas:

  • Removing replicas may leave behind PVCs, which is by design for data safety.
  • You must decide whether and when to delete these PVCs; automatic deletion can cause irreversible data loss.

Rolling Updates

For stateful workloads:

  • Ensure the update strategy is compatible with the application’s high-availability model.
  • Many StatefulSets use RollingUpdate strategy with ordered updates. This can:
    • Update pod-0 first, verify health, then pod-1, etc.
    • Minimize the risk to the entire cluster.
  • Respect application-specific steps:
    • Some databases require failover or leader changes before upgrading.
    • Quorum-based systems must maintain a majority during updates.

Operating Stateful Apps on OpenShift

Running stateful applications successfully is as much about operational practices as it is about YAML definitions.

Recommended practices:

  • Isolate stateful workloads
    Use dedicated namespaces and potentially dedicated nodes or node pools, especially for I/O-intensive workloads.
  • Monitor storage-specific metrics
    Pay attention to:
    • Volume latency and throughput.
    • Node disk pressure and filesystem health.
    • Storage backend capacity and performance.
  • Define clear ownership
    Clarify who manages:
    • Application configuration and replication.
    • Storage class selection and capacity planning.
    • Backup and restore procedures.
  • Use Operators where available
    For complex stateful systems (databases, message queues, data platforms), Operators can:
    • Automate cluster creation, scaling, and upgrades.
    • Encode safe operational procedures.
    • Integrate backup/restore workflows with OpenShift primitives.

When Not to Use StatefulSets

Not all applications that touch storage are truly stateful in the cluster sense. You might not need a StatefulSet if:

  • All replicas are identical and interchangeable, and shared storage (e.g., RWX volume) is sufficient.
  • The application’s state is entirely external (e.g., uses a managed database service outside the cluster).
  • You only need ephemeral storage for caching or temporary data; emptyDir or ephemeral volumes are enough.

In these cases, a Deployment or DeploymentConfig plus appropriate volumes may be simpler and more appropriate.

Summary

Stateful applications on OpenShift rely on:

  • StatefulSets to provide stable identities and ordered management.
  • PersistentVolumes and PVCs to persist data across pod and node lifecycles.
  • Appropriate StorageClasses and access modes to meet performance and semantics needs.
  • Operational practices—backup, restore, scaling, and upgrades—tailored to the specifics of each stateful system.

Understanding these patterns allows you to confidently design and run databases, queues, and other data-intensive workloads on OpenShift in a reliable and maintainable way.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Course Overview
    • ☰ 1.1. What is OpenShift
      • ☰ 1.2. Why OpenShift is used in modern IT and cloud environments
        • ☰ 1.3. OpenShift vs Kubernetes
          • ☰ 1.4. Typical OpenShift use cases
            • ☰ 1.5. Structure and goals of the course
            • ☰ 2. Containers and Cloud-Native Fundamentals
              • ☰ 2.1. What is a container
                • ☰ 2.2. Containers vs virtual machines
                  • ☰ 2.3. Introduction to Docker and container images
                    • ☰ 2.4. Container registries
                      • ☰ 2.5. Basics of cloud-native applications
                      • ☰ 3. Kubernetes Fundamentals
                        • ☰ 3.1. What is Kubernetes
                          • ☰ 3.2. Kubernetes architecture
                            • ☰ 3.3. Control plane components
                              • ☰ 3.4. Worker node components
                                • ☰ 3.5. Kubernetes objects and resources
                                  • ☰ 3.6. Declarative configuration model
                                  • ☰ 4. OpenShift Architecture
                                    • ☰ 4.1. Overview of OpenShift architecture
                                      • ☰ 4.2. Control plane in OpenShift
                                        • ☰ 4.3. Nodes and machine roles
                                          • ☰ 4.4. OpenShift Operators
                                            • ☰ 4.5. OpenShift networking model
                                              • ☰ 4.6. OpenShift storage model
                                              • ☰ 5. OpenShift Installation and Deployment Models
                                                • ☰ 5.1. OpenShift deployment options
                                                  • ☰ 5.2. Installer-Provisioned Infrastructure
                                                    • ☰ 5.3. User-Provisioned Infrastructure
                                                      • ☰ 5.4. Managed OpenShift services
                                                        • ☰ 5.5. Single-node OpenShift
                                                          • ☰ 5.6. Cluster lifecycle management
                                                          • ☰ 6. Accessing and Using OpenShift
                                                            • ☰ 6.1. OpenShift web console
                                                              • ☰ 6.2. OpenShift CLI (oc)
                                                                • ☰ 6.3. Authentication and authorization
                                                                  • ☰ 6.4. Projects and namespaces
                                                                    • ☰ 6.5. Resource quotas and limits
                                                                    • ☰ 7. Building and Deploying Applications
                                                                      • ☰ 7.1. Application deployment concepts
                                                                        • ☰ 7.2. Source-to-Image (S2I)
                                                                          • ☰ 7.3. Container-based deployments
                                                                            • ☰ 7.4. DeploymentConfigs and Deployments
                                                                              • ☰ 7.5. Rolling updates and rollbacks
                                                                              • ☰ 8. Networking in OpenShift
                                                                                • ☰ 8.1. Services and service discovery
                                                                                  • ☰ 8.2. Routes and ingress
                                                                                    • ☰ 8.3. Network policies
                                                                                      • ☰ 8.4. Load balancing
                                                                                        • ☰ 8.5. External access to applications
                                                                                        • ☰ 9. Storage in OpenShift
                                                                                          • ☰ 9.1. Persistent storage concepts
                                                                                            • ☰ 9.2. Persistent Volumes
                                                                                              • ☰ 9.3. Persistent Volume Claims
                                                                                                • ☰ 9.4. StorageClasses
                                                                                                  • ☰ 9.5. Dynamic provisioning
                                                                                                    • ☰ 9.6. Stateful applications
                                                                                                    • ☰ 10. Configuration and Secrets Management
                                                                                                      • ☰ 10.1. ConfigMaps
                                                                                                        • ☰ 10.2. Secrets
                                                                                                          • ☰ 10.3. Environment variables
                                                                                                            • ☰ 10.4. Managing sensitive data
                                                                                                              • ☰ 10.5. Best practices for configuration
                                                                                                              • ☰ 11. Scaling and High Availability
                                                                                                                • ☰ 11.1. Horizontal pod autoscaling
                                                                                                                  • ☰ 11.2. Vertical scaling
                                                                                                                    • ☰ 11.3. Self-healing and pod restarts
                                                                                                                      • ☰ 11.4. High availability concepts
                                                                                                                        • ☰ 11.5. Multi-zone and multi-region clusters
                                                                                                                        • ☰ 12. Monitoring, Logging, and Observability
                                                                                                                          • ☰ 12.1. Built-in monitoring stack
                                                                                                                            • ☰ 12.2. Metrics and alerts
                                                                                                                              • ☰ 12.3. Logging architecture
                                                                                                                                • ☰ 12.4. Distributed tracing
                                                                                                                                  • ☰ 12.5. Troubleshooting applications
                                                                                                                                  • ☰ 13. Security in OpenShift
                                                                                                                                    • ☰ 13.1. Security model overview
                                                                                                                                      • ☰ 13.2. Role-Based Access Control
                                                                                                                                        • ☰ 13.3. Security Context Constraints
                                                                                                                                          • ☰ 13.4. Image security and scanning
                                                                                                                                            • ☰ 13.5. Network security
                                                                                                                                              • ☰ 13.6. Compliance and auditing
                                                                                                                                              • ☰ 14. CI/CD and DevOps with OpenShift
                                                                                                                                                • ☰ 14.1. DevOps concepts
                                                                                                                                                  • ☰ 14.2. CI/CD pipelines overview
                                                                                                                                                    • ☰ 14.3. OpenShift Pipelines (Tekton)
                                                                                                                                                      • ☰ 14.4. GitOps with Argo CD
                                                                                                                                                        • ☰ 14.5. Integrating OpenShift with external CI tools
                                                                                                                                                        • ☰ 15. Operators and Platform Services
                                                                                                                                                          • ☰ 15.1. What are Operators
                                                                                                                                                            • ☰ 15.2. Operator Lifecycle Manager
                                                                                                                                                              • ☰ 15.3. Installing and managing Operators
                                                                                                                                                                • ☰ 15.4. Common platform Operators
                                                                                                                                                                  • ☰ 15.5. Custom Operators
                                                                                                                                                                  • ☰ 16. Upgrades, Maintenance, and Operations
                                                                                                                                                                    • ☰ 16.1. OpenShift upgrade process
                                                                                                                                                                      • ☰ 16.2. Backup and restore
                                                                                                                                                                        • ☰ 16.3. Node maintenance
                                                                                                                                                                          • ☰ 16.4. Cluster troubleshooting
                                                                                                                                                                            • ☰ 16.5. Capacity planning
                                                                                                                                                                            • ☰ 17. OpenShift on HPC and Specialized Workloads
                                                                                                                                                                              • ☰ 17.1. Running batch workloads on OpenShift
                                                                                                                                                                                • ☰ 17.2. OpenShift and GPUs
                                                                                                                                                                                  • ☰ 17.3. MPI and parallel workloads in containers
                                                                                                                                                                                    • ☰ 17.4. Hybrid HPC and cloud workflows
                                                                                                                                                                                    • ☰ 18. OpenShift in Practice
                                                                                                                                                                                      • ☰ 18.1. Typical OpenShift workflows
                                                                                                                                                                                        • ☰ 18.2. Development to production lifecycle
                                                                                                                                                                                          • ☰ 18.3. Case studies from industry
                                                                                                                                                                                            • ☰ 18.4. Common pitfalls and best practices
                                                                                                                                                                                            • ☰ 19. Future Directions and Ecosystem
                                                                                                                                                                                              • ☰ 19.1. Kubernetes ecosystem overview
                                                                                                                                                                                                • ☰ 19.2. OpenShift roadmap
                                                                                                                                                                                                  • ☰ 19.3. Serverless and event-driven workloads
                                                                                                                                                                                                    • ☰ 19.4. AI and data platforms on OpenShift
                                                                                                                                                                                                    • ☰ 20. Final Project and Hands-On Exercises
                                                                                                                                                                                                      • ☰ 20.1. Deploying a containerized application
                                                                                                                                                                                                        • ☰ 20.2. Implementing CI/CD with OpenShift
                                                                                                                                                                                                          • ☰ 20.3. Securing and scaling an application
                                                                                                                                                                                                            • ☰ 20.4. Monitoring and troubleshooting report