Managing sensitive data

Why Managing Sensitive Data Matters

OpenShift makes it easy to deploy and scale applications, but that also makes it easy to accidentally leak credentials or other sensitive information. Managing sensitive data is about:

• Ensuring secrets are stored and transported securely
• Limiting who and what can access them
• Reducing the risk and impact of leaks
• Making rotation and revocation practical

This chapter focuses on how to handle sensitive data correctly in OpenShift, building on the mechanics of Secrets, ConfigMaps, and environment variables from previous sections.

Typical sensitive data includes:

• Application credentials: database passwords, API keys, OAuth tokens
• Infrastructure credentials: cloud provider keys, SSH keys
• Certificates and keys: TLS certificates, private keys, CA bundles
• Personal or regulated data: some configuration values might be derived from GDPR/PII context (e.g., encryption keys for protected data)

Principles for Handling Sensitive Data

Least privilege

• Only give each pod, service account, and user the minimum secrets they actually need.
• Use separate secrets per application or per microservice rather than one large, shared secret.
• Use separate service accounts for different workloads and bind secrets only where required.

Avoid hard‑coding secrets

Avoid:

• Baking secrets into container images
• Committing secrets to Git
• Putting secrets in plain-text configuration files
• Passing secrets via command-line arguments (often visible in process listings and logs)

Instead:

• Store sensitive data in OpenShift Secrets or in an external secret management system, and reference them from your workloads.
• Use placeholders (e.g. environment variables) in code or configuration that are resolved from secrets at runtime.

Encryption in transit and at rest

• Ensure TLS is used when applications communicate using secrets (e.g., DB connections using passwords).
• Use OpenShift features that provide encryption at rest for secrets in etcd (usually enabled by cluster operators) and for storage backing Persistent Volumes.

Secret Usage Patterns

Secrets as environment variables vs volumes

You can consume secrets:

• As environment variables:
• Convenient for simple key/value credentials (e.g., DB_USER, DB_PASSWORD).
• Risk: values may be visible in /proc, debug logs, crash dumps, or app error pages.
• As mounted volumes:
• Better for multi-line, structured data: TLS keys, JSON blobs, kubeconfigs.
• Supports file permissions and separation from process environment.

Guidelines:

• Prefer volumes for highly sensitive or complex data (certs, keys).
• Use env vars for less-sensitive, short-lived secrets and where the application is environment-driven.
• Avoid exposing the same secret both as env vars and volume in the same pod unless needed.

Minimizing exposure in logs and diagnostics

• Ensure your application does not log sensitive fields.
• When using oc describe and oc get:
• Be aware that some values may appear in plain text in the output.
• Use role-based access control (RBAC) so only trusted users can run commands that reveal secret metadata or contents.

Lifecycle of Secrets

Managing sensitive data is not a one-time step; it’s a lifecycle.

Creation

• Create secrets separately from application manifests when possible.
• Use oc with file-based or stdin inputs instead of inline literals to reduce shell history leakage.

Example:

oc create secret generic db-credentials \
  --from-literal=username=dbuser \
  --from-literal=password="$(pass show prod/db-password)"

or:

oc create secret generic tls-cert \
  --from-file=tls.crt=server.crt \
  --from-file=tls.key=server.key

Avoid:

• Entering secret values directly in command history without precautions.
• Committing the YAML with base64-encoded secret values into public or shared repositories.

Rotation

Secret rotation is essential when:

• Credentials are compromised or suspected to be.
• Team members leave, or access policies change.
• You follow a scheduled rotation policy (e.g., every 90 days).

Approach:

1. Create a new secret with updated values (e.g., db-credentials-v2).
2. Update deployments to use the new secret:
• Change secretName in pod templates or env references.
3. Roll out changes (oc rollout restart or by updating the deployment).
4. Verify all pods are using the new secret.
5. Remove or disable old credentials in the backend system (DB, external API).
6. Delete the old secret from OpenShift if no longer needed.

For short-lived secrets (tokens, signed URLs), consider:

• Using secrets that are auto-generated or managed by Operators.
• Reusing mechanisms like OAuth tokens, ServiceAccount tokens, or external identity providers that handle rotation.

Revocation and incident response

When a secret is leaked or suspected:

1. Revoke or change the credential in the external system (DB, cloud, API).
2. Create new secrets with new values.
3. Update workloads to the new secrets and restart pods.
4. Audit:
• Check who accessed the secret (via audit logs if available).
• Identify where the secret might be present (logs, backups, scripts).
5. Clean up:
• Remove old secrets.
• Sanitize logs or archives if possible.
• Update policies to prevent recurrence.

Storing Secrets Securely

Git and configuration management

Storing OpenShift manifests in Git is standard, but raw secrets should not be committed in plain text.

Options:

• References-only in Git:
• Store only the names of secrets in Git; create the secret out-of-band (manually, script, or CI job).
• Encrypted secrets in Git:
• Use tools such as sealed-secrets, sops, or similar solutions to encrypt secret data while keeping manifests version-controlled.
• Decryption happens in the cluster or controlled CI environment, not on developer machines by default.

General guidelines:

• Never rely on base64 encoding in standard Kubernetes/OpenShift secret manifests as a security measure; it is only encoding, not encryption.
• Restrict who can read the Git repo and who can decrypt if using encrypted secrets.

Environment access and shell history

• Avoid typing secrets directly in interactive shell commands:
• Use --from-file or stdin.
• Pipe from external secret stores or password managers.
• Clear or disable Bash history when working with secrets, or use:
• HISTCONTROL=ignorespace and prefix sensitive commands with a space (where supported).

Access Control and Policy

Controlling who can read or modify secrets is central to managing sensitive data.

Limiting user access to secrets

Use RBAC to:

• Allow only specific roles or groups to:
• Create, update, or delete secrets.
• Read secret data (get on secrets).
• Provide read access on a per-project basis or narrower where possible.

Examples of role patterns:

• Developers:
• Can reference certain secrets in their deployments.
• May not have permission to read the contents of all secrets.
• Platform admins:
• Can manage cluster-wide or infrastructure-related secrets.

Be cautious with “view all” permissions:

• view and edit roles may allow reading secrets depending on cluster policy; check and tighten if needed.

Limiting in-cluster access

Pods themselves should not be able to read arbitrary secrets:

• Avoid mounting the default service account token with broad permissions.
• Use dedicated service accounts with minimal permissions.
• Ensure that applications do not expose secrets via debug endpoints, health checks, or logs.

Integrating with External Secret Stores

In many environments, OpenShift is not the only place secrets are managed.

Common external systems:

• HashiCorp Vault
• Cloud provider secret managers (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager)
• HSMs or on-premise secret stores

Integration patterns:

• Sync controllers/Operators:
• An Operator periodically reads from an external store and writes corresponding OpenShift Secrets.
• Applications consume regular OpenShift secrets, unaware of the external source.
• Sidecar containers:
• A sidecar fetches secrets at runtime and writes them to a shared volume.
• Apps read from the volume, not from a Kubernetes secret.
• Direct in-app access:
• Application code contacts the external secret store using its own credentials.
• Reduces reliance on Kubernetes secrets but increases app complexity.

Security considerations:

• Ensure the bootstrap credentials (what the pod uses to talk to the external store) are well-controlled.
• Use short-lived tokens and identity-based access (e.g., workload identity, service principal) where possible.

Application-Level Practices

Avoiding overexposure inside the pod

Inside the container:

• Use fine-grained configuration:
• Only pass the secrets that this specific process needs.
• Separate configuration from secrets:
• Non-sensitive config should be in ConfigMaps.
• Sensitive values go into Secrets, even if they conceptually relate to configuration.

Reducing accidental disclosure

• Do not dump full environment variables or config structures in logs.
• Limit information shown in error messages that end up in client responses.
• Be cautious when enabling debug modes in production; they may reveal more than intended.

Testing with non-production data

• Use dummy or low-privilege credentials in development and testing environments.
• Make it obvious when you are using production-grade secrets (naming, documentation, tagging).

Auditing and Compliance Considerations

For environments with compliance requirements:

• Enable and retain audit logs capturing:
• Secret creation, update, deletion.
• Access to secrets (especially get and list operations).
• Tag and document:
• Which secrets are related to regulated data (PII, PHI, payment).
• Rotation intervals and responsible owners.

Processes:

• Regularly review:
• Which projects and service accounts have access to which secrets.
• Whether any dormant or unused secrets can be retired.
• Integrate secret checks into:
• CI (e.g., scanning for secrets accidentally committed to Git).
• Periodic security assessments.

Practical Guidelines and Checklists

Do’s

• Use OpenShift Secrets (or external stores) for all sensitive values.
• Apply least privilege: per-application secrets, minimal roles.
• Prefer volume mounts for certificates and private keys.
• Automate secret rotation where possible.
• Keep secrets out of source control, or encrypt them when they must be versioned.
• Use RBAC and auditing to track and control secret access.

Don’ts

• Do not commit raw secrets (even base64) to public or broadly shared repositories.
• Do not hard-code secrets in images, Helm charts, or manifests.
• Do not echo secrets in logs, error messages, or monitoring dashboards.
• Do not reuse the same credential across many unrelated services or environments.
• Do not grant broad access to secrets via generic “admin” or “view everything” roles unless strictly necessary.

By following these practices, you can use OpenShift’s built-in mechanisms and surrounding ecosystem to handle sensitive data in a way that is secure, auditable, and maintainable over the lifetime of your applications.