KAHIBARO Bookshelves
Discord Login Register

10.5. Best practices for configuration

Principles of Good Configuration Design

Configuration in OpenShift should be:

  • Declarative – stored as YAML manifests in Git, not edited by hand in the cluster.
  • Environment-specific, but app-consistent – same app, same config shape across environments; only values differ.
  • Immutable at runtime (as much as possible) – changes are made via new versions, not by manual edits in a running pod.
  • Separation of concerns – application code, configuration data, and secrets are strictly separated.

Applied to OpenShift, this means using ConfigMap, Secret, Deployment, BuildConfig, etc. in a consistent way and avoiding “magic” configuration inside images.

Key goals:

  • Make configuration auditable (via Git and cluster resources).
  • Make configuration reproducible (same manifests deployed repeatedly give the same result).
  • Make configuration safe (no secrets in the wrong places; safe defaults).

What Belongs Where: ConfigMaps, Secrets, and Images

A core best practice is deciding where each piece of configuration lives.

Split configuration types clearly

  • Use ConfigMaps for:
    • Non-sensitive configuration (feature flags, timeouts, logging levels).
    • Templates, config files (e.g. nginx.conf, application YAML/INI).
    • Command-line arguments or environment variables that aren’t secret.
  • Use Secrets for:
    • Credentials (DB passwords, API keys, tokens).
    • TLS certificates and keys.
    • SSH keys, private keys, encryption keys.
  • Use images for:
    • Application code.
    • Static assets that don’t change per-environment.
    • “Base defaults” but not environment-specific values.

Anti-patterns to avoid

  • Secrets baked into images
    Never embed credentials in container images, even if the registry is private.
  • Environment-specific images
    Do not build separate images for dev/stage/prod just to change config.
  • Shared credentials across environments
    Each environment must have its own secrets; never reuse prod credentials elsewhere.

Structuring Configuration in OpenShift

Prefer environment variables for simple values

For simple key-value pairs:

  • Use env or envFrom with ConfigMap / Secret.
  • Keep env names consistent across environments (e.g. always DB_HOST, DB_USER).

Advantages:

  • Easy to inspect via oc describe pod.
  • Clear mapping between app code and configuration.

Examples:

env:
  - name: APP_LOG_LEVEL
    valueFrom:
      configMapKeyRef:
        name: myapp-config
        key: log.level
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: myapp-db-credentials
        key: password

Use mounted files for structured config

When configuration is more complex:

  • Use ConfigMaps or Secrets as mounted files:
    • Application config files (application.yaml, settings.json).
    • CA bundles, keystores, certificates.
  • Mount them read-only and at a known path.

Example pattern:

volumes:
  - name: app-config
    configMap:
      name: myapp-config
containers:
  - name: myapp
    volumeMounts:
      - name: app-config
        mountPath: /etc/myapp
        readOnly: true

Prefer this when:

  • The config has many keys or nested structures.
  • Multiple containers share the same config file.

Avoid configuration sprawl

Configuration should be organized, not scattered:

  • Define one primary ConfigMap per application component (e.g. myapp-config), with optional specialized ones (e.g. myapp-feature-flags).
  • Keep keys namespaced by logical area:
    • db.host, db.port, db.poolSize
    • logging.level, logging.format
  • Avoid one huge “global” ConfigMap used by many unrelated apps.

Managing Configuration Across Environments

Keep configuration shape identical

Across environments (dev, test, prod):

  • Keep the same keys and overall structure.
  • Only values differ.

This enables:

  • Reusing the same manifests with different overlays.
  • Easier promotion pipelines (same config contract, different values).

Bad:

  • Dev uses DB_HOST, prod uses DATABASE_URL.

Good:

  • All environments use DB_HOST, DB_PORT, DB_NAME, etc., with different values.

Use clear naming conventions

Common patterns:

  • Namespaced by app and environment:
    • myapp-config-dev, myapp-config-prod
    • myapp-db-credentials-dev, myapp-db-credentials-prod
  • Or same resource names per environment but separated by namespace:
    • myapp-config in myapp-dev
    • myapp-config in myapp-prod

Pick one pattern and apply it consistently.

Use Git and overlays (GitOps-friendly)

Store all configuration manifests in Git:

  • Base manifests defining common resources and structure.
  • Environment overlays or kustomizations for specific values:
    • Different ConfigMap/Secret data.
    • Different replica counts, resource limits.

Benefits:

  • Versioning and history of configuration changes.
  • Review via pull requests.
  • Easy to roll back to a known good state.

Even without a full GitOps setup, treat configuration as code.

Security-Focused Configuration Practices

Never store secrets in plain text sources

Avoid:

  • Secrets in Git (even private repos).
  • Secrets in CI logs, console output, or shared chat.
  • Secrets in OpenShift resource YAML files stored in SCM.

Preferred approaches:

  • Use separate secret management for source (e.g. sealed secrets, external secret stores).
  • Keep the actual Secret resources generated from secure storage, not hand-written.

Use least privilege and separation

  • Different credentials per:
    • Environment (dev, test, prod).
    • Application or team (avoid app sharing root DB passwords).
  • Limit secret access via:
    • Namespaces and RBAC.
    • ServiceAccounts bound to particular workloads.

Application pods should only be able to read the Secrets they actually need.

Validate that configuration doesn’t leak secrets

Practices:

  • Do not surface secrets in:
    • Metrics.
    • Logs.
    • Debug endpoints.
  • When troubleshooting:
    • Avoid oc describe secret (shows data as base64) unless truly needed.
    • Prefer listing references to secrets, not the content.

Use separate ConfigMaps and Secrets instead of mixing sensitive and non-sensitive values in one resource. This reduces the risk of exposing secrets by accident.

Operational Best Practices

Make configuration changes controlled and trackable

  • Always change configuration via:
    • Git commits and CI/CD pipelines, or
    • Well-documented oc commands/scripts.
  • Avoid manual oc edit in production namespaces.

Use:

  • Labels and annotations to document:
    • Owner and purpose.
    • Ticket or change request IDs.
    • Version or timestamp.

Example:

metadata:
  name: myapp-config
  labels:
    app: myapp
    component: backend
  annotations:
    owner: "team-payments"
    change-ticket: "INC-12345"

Minimize required restarts for config changes

Some configuration requires pod restarts, some not:

  • If config values are changed often:
    • Consider implementing dynamic reload in the app (e.g. file watchers, SIGHUP).
    • Use mounted config files and reload logic.
  • If dynamic reload is not supported:
    • Document that configuration changes cause a rollout.
    • Use controlled rolling updates with readiness checks.

Balance between:

  • Operational simplicity.
  • Need for fast configuration changes.

Validate configuration before applying

Avoid pushing broken configs into production:

  • Validate configuration in lower environments first.
  • Add automated checks in pipeline:
    • Lint YAML files.
    • Run unit tests using sample configuration.
    • Run smoke tests post-deployment.

If configuration is used by multiple tenants or teams, treat changes as backward-incompatible unless proven safe.

Use defaults and fallbacks carefully

In the application:

  • Provide sensible defaults for non-critical options.
  • Explicitly require critical values (e.g., DB credentials) to be present; fail fast if missing.

In OpenShift manifests:

  • Keep default values in ConfigMap when:
    • They are safe and non-secret.
  • Avoid “hidden” defaults in multiple places; centralize into:
    • A single ConfigMap.
    • Or code-level defaults with environment overrides.

Designing for Multi-Tenancy and Teams

Namespace-level isolation

Use namespaces to:

  • Separate configuration for different:
    • Applications.
    • Teams.
    • Environments or tenants.

Each namespace:

  • Has its own ConfigMaps and Secrets.
  • Can have its own resource quotas and policies.

This prevents accidental cross-usage of configuration between unrelated workloads.

Align configuration ownership and responsibilities

Define:

  • Who owns which ConfigMaps/Secrets (teams, roles).
  • Who may change production configuration.
  • Approval workflows for changes.

Common models:

  • Application team owns application ConfigMaps.
  • Platform team owns cluster-wide config (logging, ingress, monitoring).
  • Security team controls policies for secrets.

Patterns for Complex Applications

Shared vs. per-service configuration

For microservices:

  • Put service-specific config in that service’s own ConfigMap/Secrets.
  • Use shared ConfigMaps for:
    • Common endpoints (logging, tracing, service discovery).
    • Cross-cutting feature flags.

But:

  • Avoid huge global ConfigMaps with many unrelated entries.
  • Use separate shared configs per domain, e.g.:
    • observability-config
    • auth-config
    • platform-endpoints

Configuration for jobs and batch workloads

Batch jobs often:

  • Use different parameters per run (dates, input sources).

Best practices:

  • Use parameters as environment variables derived from:
    • ConfigMaps for defaults.
    • Job-specific overrides.
  • Avoid copying entire ConfigMaps per job; override only what changes.

Example pattern:

  • Base ConfigMap: reporting-job-config (common values).
  • Each Job manifest:
    • Reuses this ConfigMap.
    • Adds run-specific env variables or CLI args.

Practical Tips and Anti-Patterns

Practical tips

  • Always label ConfigMaps and Secrets with:
    • app, component, environment.
  • Clean up unused configuration:
    • Delete orphan ConfigMaps/Secrets after application retirement.
  • Keep config small and focused:
    • Avoid very large blobs; consider splitting into logical pieces.

Common anti-patterns

  • Configuration via image rebuilds
    Every config change requiring a new image build slows feedback and mixes concerns.
  • Huge “global” config per cluster
    Makes it hard to know which apps depend on which values.
  • Mounting the same Secret/ConfigMap in every pod “just in case”
    Violates least privilege, increases blast radius.
  • Environment-specific logic in app code (e.g. if ENV == "prod" then…)
    Prefer environment-specific values via configuration instead.

Summary Checklist

When designing configuration for an OpenShift workload, verify:

  • Separation:
    • Non-sensitive config in ConfigMaps.
    • Sensitive data in Secrets.
    • No secrets in images or Git.
  • Structure:
    • Consistent naming and keys across environments.
    • Clear separation of per-service and shared config.
  • Security:
    • Least-privilege access to Secrets.
    • No accidental exposure via logs or UIs.
  • Operations:
    • Configuration managed via Git or repeatable commands.
    • Changes validated in lower environments.
    • Clear owner and labeling of resources.

Following these practices produces configurations that are safer, easier to reason about, and friendlier to automation and GitOps workflows in OpenShift.

Views: 0

Comments

Please login to add a comment.

Don't have an account? Register now!

Where to Move

Move chapter:

  • ☰ 1. Course Overview
    • ☰ 1.1. What is OpenShift
      • ☰ 1.2. Why OpenShift is used in modern IT and cloud environments
        • ☰ 1.3. OpenShift vs Kubernetes
          • ☰ 1.4. Typical OpenShift use cases
            • ☰ 1.5. Structure and goals of the course
            • ☰ 2. Containers and Cloud-Native Fundamentals
              • ☰ 2.1. What is a container
                • ☰ 2.2. Containers vs virtual machines
                  • ☰ 2.3. Introduction to Docker and container images
                    • ☰ 2.4. Container registries
                      • ☰ 2.5. Basics of cloud-native applications
                      • ☰ 3. Kubernetes Fundamentals
                        • ☰ 3.1. What is Kubernetes
                          • ☰ 3.2. Kubernetes architecture
                            • ☰ 3.3. Control plane components
                              • ☰ 3.4. Worker node components
                                • ☰ 3.5. Kubernetes objects and resources
                                  • ☰ 3.6. Declarative configuration model
                                  • ☰ 4. OpenShift Architecture
                                    • ☰ 4.1. Overview of OpenShift architecture
                                      • ☰ 4.2. Control plane in OpenShift
                                        • ☰ 4.3. Nodes and machine roles
                                          • ☰ 4.4. OpenShift Operators
                                            • ☰ 4.5. OpenShift networking model
                                              • ☰ 4.6. OpenShift storage model
                                              • ☰ 5. OpenShift Installation and Deployment Models
                                                • ☰ 5.1. OpenShift deployment options
                                                  • ☰ 5.2. Installer-Provisioned Infrastructure
                                                    • ☰ 5.3. User-Provisioned Infrastructure
                                                      • ☰ 5.4. Managed OpenShift services
                                                        • ☰ 5.5. Single-node OpenShift
                                                          • ☰ 5.6. Cluster lifecycle management
                                                          • ☰ 6. Accessing and Using OpenShift
                                                            • ☰ 6.1. OpenShift web console
                                                              • ☰ 6.2. OpenShift CLI (oc)
                                                                • ☰ 6.3. Authentication and authorization
                                                                  • ☰ 6.4. Projects and namespaces
                                                                    • ☰ 6.5. Resource quotas and limits
                                                                    • ☰ 7. Building and Deploying Applications
                                                                      • ☰ 7.1. Application deployment concepts
                                                                        • ☰ 7.2. Source-to-Image (S2I)
                                                                          • ☰ 7.3. Container-based deployments
                                                                            • ☰ 7.4. DeploymentConfigs and Deployments
                                                                              • ☰ 7.5. Rolling updates and rollbacks
                                                                              • ☰ 8. Networking in OpenShift
                                                                                • ☰ 8.1. Services and service discovery
                                                                                  • ☰ 8.2. Routes and ingress
                                                                                    • ☰ 8.3. Network policies
                                                                                      • ☰ 8.4. Load balancing
                                                                                        • ☰ 8.5. External access to applications
                                                                                        • ☰ 9. Storage in OpenShift
                                                                                          • ☰ 9.1. Persistent storage concepts
                                                                                            • ☰ 9.2. Persistent Volumes
                                                                                              • ☰ 9.3. Persistent Volume Claims
                                                                                                • ☰ 9.4. StorageClasses
                                                                                                  • ☰ 9.5. Dynamic provisioning
                                                                                                    • ☰ 9.6. Stateful applications
                                                                                                    • ☰ 10. Configuration and Secrets Management
                                                                                                      • ☰ 10.1. ConfigMaps
                                                                                                        • ☰ 10.2. Secrets
                                                                                                          • ☰ 10.3. Environment variables
                                                                                                            • ☰ 10.4. Managing sensitive data
                                                                                                              • ☰ 10.5. Best practices for configuration
                                                                                                              • ☰ 11. Scaling and High Availability
                                                                                                                • ☰ 11.1. Horizontal pod autoscaling
                                                                                                                  • ☰ 11.2. Vertical scaling
                                                                                                                    • ☰ 11.3. Self-healing and pod restarts
                                                                                                                      • ☰ 11.4. High availability concepts
                                                                                                                        • ☰ 11.5. Multi-zone and multi-region clusters
                                                                                                                        • ☰ 12. Monitoring, Logging, and Observability
                                                                                                                          • ☰ 12.1. Built-in monitoring stack
                                                                                                                            • ☰ 12.2. Metrics and alerts
                                                                                                                              • ☰ 12.3. Logging architecture
                                                                                                                                • ☰ 12.4. Distributed tracing
                                                                                                                                  • ☰ 12.5. Troubleshooting applications
                                                                                                                                  • ☰ 13. Security in OpenShift
                                                                                                                                    • ☰ 13.1. Security model overview
                                                                                                                                      • ☰ 13.2. Role-Based Access Control
                                                                                                                                        • ☰ 13.3. Security Context Constraints
                                                                                                                                          • ☰ 13.4. Image security and scanning
                                                                                                                                            • ☰ 13.5. Network security
                                                                                                                                              • ☰ 13.6. Compliance and auditing
                                                                                                                                              • ☰ 14. CI/CD and DevOps with OpenShift
                                                                                                                                                • ☰ 14.1. DevOps concepts
                                                                                                                                                  • ☰ 14.2. CI/CD pipelines overview
                                                                                                                                                    • ☰ 14.3. OpenShift Pipelines (Tekton)
                                                                                                                                                      • ☰ 14.4. GitOps with Argo CD
                                                                                                                                                        • ☰ 14.5. Integrating OpenShift with external CI tools
                                                                                                                                                        • ☰ 15. Operators and Platform Services
                                                                                                                                                          • ☰ 15.1. What are Operators
                                                                                                                                                            • ☰ 15.2. Operator Lifecycle Manager
                                                                                                                                                              • ☰ 15.3. Installing and managing Operators
                                                                                                                                                                • ☰ 15.4. Common platform Operators
                                                                                                                                                                  • ☰ 15.5. Custom Operators
                                                                                                                                                                  • ☰ 16. Upgrades, Maintenance, and Operations
                                                                                                                                                                    • ☰ 16.1. OpenShift upgrade process
                                                                                                                                                                      • ☰ 16.2. Backup and restore
                                                                                                                                                                        • ☰ 16.3. Node maintenance
                                                                                                                                                                          • ☰ 16.4. Cluster troubleshooting
                                                                                                                                                                            • ☰ 16.5. Capacity planning
                                                                                                                                                                            • ☰ 17. OpenShift on HPC and Specialized Workloads
                                                                                                                                                                              • ☰ 17.1. Running batch workloads on OpenShift
                                                                                                                                                                                • ☰ 17.2. OpenShift and GPUs
                                                                                                                                                                                  • ☰ 17.3. MPI and parallel workloads in containers
                                                                                                                                                                                    • ☰ 17.4. Hybrid HPC and cloud workflows
                                                                                                                                                                                    • ☰ 18. OpenShift in Practice
                                                                                                                                                                                      • ☰ 18.1. Typical OpenShift workflows
                                                                                                                                                                                        • ☰ 18.2. Development to production lifecycle
                                                                                                                                                                                          • ☰ 18.3. Case studies from industry
                                                                                                                                                                                            • ☰ 18.4. Common pitfalls and best practices
                                                                                                                                                                                            • ☰ 19. Future Directions and Ecosystem
                                                                                                                                                                                              • ☰ 19.1. Kubernetes ecosystem overview
                                                                                                                                                                                                • ☰ 19.2. OpenShift roadmap
                                                                                                                                                                                                  • ☰ 19.3. Serverless and event-driven workloads
                                                                                                                                                                                                    • ☰ 19.4. AI and data platforms on OpenShift
                                                                                                                                                                                                    • ☰ 20. Final Project and Hands-On Exercises
                                                                                                                                                                                                      • ☰ 20.1. Deploying a containerized application
                                                                                                                                                                                                        • ☰ 20.2. Implementing CI/CD with OpenShift
                                                                                                                                                                                                          • ☰ 20.3. Securing and scaling an application
                                                                                                                                                                                                            • ☰ 20.4. Monitoring and troubleshooting report