Configuration and Secrets Management

Why Configuration and Secrets Management Matters on OpenShift

Configuration and secret handling are central to running applications securely and reliably on OpenShift. Instead of baking configuration into images or hardcoding sensitive values in source code, OpenShift provides dedicated resources and patterns to:

• Separate configuration from code and images
• Adapt the same image to multiple environments (dev, test, prod)
• Rotate credentials and keys without rebuilding images
• Enforce security and compliance around sensitive data

This chapter gives the conceptual foundation and cross‑cutting patterns. The specific resource types (ConfigMap, Secret, etc.) are discussed in their own chapters.

Key goals of configuration and secrets management in OpenShift:

• Decouple: Images stay generic; configuration is provided at deploy time.
• Standardize: Use Kubernetes/OpenShift-native mechanisms so tools and Operators can manage them.
• Secure: Protect sensitive values and control who can access or modify them.
• Automate: Make configuration reproducible and manageable through Git, CI/CD, and Operators.

Types of Configuration and Where They Fit

OpenShift applications typically rely on several layers of configuration:

• Build-time configuration
  Applied during image build (e.g., in Dockerfile or S2I). Fixed once the image is built.
• Deploy-time configuration
  Bound when creating or updating workloads (Deployments, DeploymentConfigs, Jobs, etc.), usually via environment variables, volumes, or annotations.
• Runtime/operational configuration
  Tunables used by the platform or Operators (e.g., resource limits, probes, autoscaling, network policies). These often live alongside application configuration but serve a different purpose.

The chapter’s focus is on application-facing deploy-time configuration and secret management, not on cluster‑level tuning.

Common categories:

• Non-sensitive configuration
• App feature flags
• Service URLs
• Non-secret user-facing text (banners, messages)
• Timeouts, limits, tuning parameters
• Sensitive configuration
• Database passwords
• API tokens and access keys
• TLS private keys and certificates
• SSH keys, encryption keys
• License files with confidential data

A core design choice is deciding what belongs in non-secret configuration versus secrets, and enforcing that separation.

Design Principles: Separating Configuration from Code

When designing applications for OpenShift:

1. Make applications configurable from the environment
   Applications should consume configuration via environment variables, files, CLI flags, or a combination thereof. This makes them easy to wire to OpenShift resources.
2. Avoid image rebuilds for config changes
   If changing a feature flag or endpoint requires rebuilding the image, you lose agility and complicate operations. Prefer externalized configuration so a Deployment update is enough.
3. Use convention over ad-hoc patterns
   Stick to patterns that Operators, templates, and Helm/GitOps tools understand:
• Use named configuration resources (ConfigMap, Secret)
• Mount them via volumes or environment variables
• Use labels/annotations consistently
4. Plan for multiple environments
   Design configuration so you can:
• Share a base configuration across environments
• Override environment-specific values (URLs, credentials, limits)
  without changing code.
5. Version and review configuration
   Treat configuration (especially non-secret) as code:
• Store in Git (infrastructure-as-code, GitOps)
• Apply via pipelines
• Code review for configuration changes

Patterns for Providing Configuration to Applications

On OpenShift, the common ways to inject configuration into Pods are:

• Environment variables
• Simple key/value
• Works well for scalar values and feature flags
• Easily read by most languages and frameworks
• Can be filled from configuration resources or literal values
• Configuration files (mounted via volumes)
• Better for structured configuration formats: YAML, JSON, INI, properties
• Avoids long/complex values in environment variables
• Allows apps that expect config files to run unchanged
• Command-line arguments
• Often templated based on environment variables or config files
• Less flexible for large configurations, but good for critical flags

Typical patterns:

• Twelve-Factor style: almost all configuration via environment variables.
• Configuration file pattern: mount a file and point the application to it via a single environment variable or command-line flag.

When designing an application for OpenShift, it’s common to combine both: environment variables for small configuration values and a mounted config file for more complex or structured settings.

Configuration Lifecycles and Reloading

A crucial aspect is how configuration changes are applied:

• Immutable pod pattern
• Configuration changes trigger a new rollout (new Pods with updated configuration).
• Common for traditional Deployments and DeploymentConfigs.
• Easy to audit and reason about: configuration is tied to a Pod version.
• Hot-reload pattern
• Application watches mounted configuration and reloads without a restart (e.g., on file change).
• Reduces downtime but adds complexity.
• Needs careful testing to avoid inconsistent state across replicas.

Considerations when choosing:

• Risk profile: Critical services may prefer explicit rollouts and staged deployments.
• Complexity: Hot reload often requires extra application logic or libraries.
• Statefulness: For stateful applications, reloading config may require careful coordination.

On OpenShift, rollouts are typically managed via Deployments/DeploymentConfigs; how they react to config changes is important when designing configuration update processes.

Secrets vs Ordinary Configuration: Conceptual Differences

In OpenShift, secrets are not just “configuration with base64”; they are different in intent and handling:

• Access control: Secrets usually have more restrictive RBAC than non-sensitive configuration.
• Auditing and compliance: Access to secrets may be monitored more strictly.
• Rotation: Secrets often require regular rotation; non-sensitive configuration may not.
• Storage and backup considerations: Secrets may need encrypted storage, special backup handling, or key management integration.

Design questions you should answer for each application:

• What counts as a secret for this app?
• Who needs read/write access to it?
• How will credentials be rotated?
• How will secrets be injected into Pods (env vs files)?
• What is the blast radius if a particular secret is leaked?

Access Control, RBAC, and Team Boundaries

Configuration and secrets are not just technical artifacts; they encode organizational boundaries:

• Application-level config
  Usually managed by application teams; may live within application namespaces.
• Shared services config
  For things like databases, message brokers, or external SaaS endpoints. Owned by platform or shared-service teams.
• Cluster-level credentials and keys
  Typically restricted to cluster administrators or platform SREs.

On OpenShift, projects/namespaces, RoleBindings, and service accounts combine to define:

• Who can create/update/delete configuration or secret resources
• Which Pods (via service accounts) can mount or read them
• Which automation (Operators, pipelines) can manage them

Configuration management must align with your organizational model:

• Per‑team namespaces with their own configuration
• Shared namespaces and config objects for common infrastructure
• Central security/pki/infra teams governing critical secrets

Configuration and Secrets in CI/CD and GitOps

Configuration management interacts closely with delivery workflows:

• Config as code
• Non-sensitive configuration is typically stored in Git.
• CI/CD jobs or GitOps tools apply changes to the cluster.
• Enables pull‑request reviews for configuration updates.
• Secrets handling in pipelines
• Pipelines need access to certain secrets (container registry tokens, deployment keys, etc.).
• Avoid storing raw secrets in Git; use:
• Secret management tools
• Encrypted files in Git (e.g., sops) with decryption at deploy time
• External secret stores integrated with OpenShift
• Control which service accounts and namespaces the pipeline can access.
• Environment promotions
• Promote the same image through dev → test → prod while:
• Keeping configuration in separate repositories or branches
• Or using overlays (e.g., Kustomize) to adjust config per environment
• Secrets are often unique per environment (different credentials, keys).

Within OpenShift, Operators and GitOps tools leverage the native resource model, so making configuration and secrets first‑class Kubernetes/OpenShift resources is key for automation.

Configuration Validation and Safety

Misconfiguration is a frequent source of outages. To reduce risk:

• Validation at deploy time
• Use resource schemas and validation hooks where available.
• Use CI jobs to run application-level config validation (linting, dry-runs).
• Runtime safeguards
• Health checks (liveness/readiness probes) surface misconfigurations quickly.
• Monitoring/alerts around startup failures, configuration-related logs.
• Guardrails via policy
• Admission control and policies can:
• Block Pods from referencing prohibited secrets
• Enforce labels/annotations or naming conventions for configuration objects
• Restrict which namespaces can access certain configuration types
• Safe rollout strategies
• Canary or blue‑green deployments for risky configuration changes.
• Gradual rollout to a subset of replicas or environments before full production rollout.

Managing Configuration at Scale

As the number of applications and teams grows, ad‑hoc practices don’t scale. Consider:

• Naming and labeling conventions
• Consistent names for config and secrets (e.g., {app}-config, {app}-db-credentials).
• Labels/annotations indicating owner, purpose, environment.
• Reuse vs fragmentation
• Decide when configuration should be shared across applications (e.g., common API endpoints) versus dedicated per app.
• Avoid coupling unrelated apps via shared configuration that must change together.
• Template and Operator usage
• Templates, Helm charts, or Operators encapsulate repeated configuration patterns.
• Make opinionated defaults and structures (e.g., “this Operator always creates and manages its own secrets and config objects”).
• Documentation
• Document required configuration keys, defaults, and semantics.
• Provide sample configuration resources for new teams.

Security and Compliance Considerations for Secrets

Even though details of security models and tools are covered elsewhere, secrets management intersects with broader security and compliance:

• Least privilege
• A Pod’s service account should only access the secrets it needs.
• Avoid “one giant secret” pattern that aggregates unrelated data.
• Segregation of duties
• Different roles for creating/updating secrets vs deploying applications that use them.
• Approvals required for sensitive secret changes.
• Auditing
• Track who modified or accessed secret resources (via API server audit logs, external SIEM).
• Integrate secret changes into change management processes where needed.
• Rotation and expiry
• Plan how often secrets must be rotated (e.g., database passwords, tokens).
• Ensure rotation mechanisms don’t require deploy-time manual work every time.
• Coordinate rotation with dependent services to avoid outages.
• Integration with external secret stores
• Many organizations centralize secrets (HSMs, cloud KMS, Vault, etc.).
• OpenShift workloads may consume secrets that originate from these systems via dedicated controllers or Operators.

Operational Runbooks and Incident Response

Effective configuration and secret management includes operational practices:

• Runbooks for common tasks
• Changing a configuration value and safely rolling it out.
• Rotating a credential or key with minimal downtime.
• Revoking compromised credentials and propagating changes.
• Incident response
• Playbooks for suspected secret leaks:
• Revoke or rotate affected credentials.
• Re-deploy workloads with new secrets.
• Inspect and limit access to affected namespaces or service accounts.
• Procedures for rollback when a configuration change breaks production.
• Backup and restore
• Ensure that configuration and secrets are included in backups, but:
• Protect secret backups with encryption and strict access controls.
• Validate that restores correctly reestablish configuration and secret state without conflicts.

Summary

Configuration and secrets management in OpenShift is about more than just creating specific objects; it’s about:

• Designing applications to consume external configuration cleanly.
• Distinguishing between non-sensitive configuration and secrets.
• Applying appropriate access control, rotation, and audit practices.
• Integrating configuration and secrets into CI/CD and GitOps flows.
• Scaling patterns across teams and applications with conventions and automation.

Subsequent chapters will detail the concrete mechanisms OpenShift provides—ConfigMap, Secret, environment variables, and best practices for managing sensitive data—and how to apply these design principles using real resources and examples.