KAHIBARO
Discord Login Register ☰
☰

24.12. Production Environments

Table of Contents

Understanding Production Environments

A production environment is where real users interact with your application and real data is stored. Everything you do in CI/CD ultimately targets this environment, so you must treat it very differently from development and testing.

This chapter focuses on what is unique about production environments in a CI/CD context: how they are structured, protected, and updated safely.

What Makes Production Different

In many teams you will see at least three main environments:

EnvironmentTypical UsersPurpose
LocalIndividual developersDaily coding, experiments
StagingQA, developers, productFinal testing before release
ProductionReal customersLive system that must stay reliable

Production has special constraints:

In production, stability and safety are more important than speed of change.
Never treat production as a place to “try things and see what happens”.

Typical Production Environment Setup

A basic production environment often includes:

For small systems, some of these can be combined on a single virtual machine, but the logical separation still exists. In CI/CD, you treat them as separate components, each with their own configuration and deployment flows.

Separation of Concerns in Production

In production you separate concerns more strictly than in development:

CI/CD pipelines must respect these differences. For example, rolling out a new application image is cheap, but applying a database migration is not.

Protecting Production in CI/CD

You rarely want every push to automatically update production. Instead, you add safeguards.

Common protection mechanisms:

  1. Protected branches
    • Only certain branches, for example main or release/*, can trigger production deployments.
    • Only authorized users can merge to these branches.
  2. Manual approvals
    • Pipelines for production include a manual “approval” step.
    • Ops or senior engineers confirm that the release is ready.
  3. Restricted runners or agents
    • Only specific CI runners, with limited credentials, are allowed to deploy to production.
    • Even if someone misconfigures a job, it cannot push to production without the right runner.
  4. Least privilege credentials
    • CI uses a special service account with exactly the permissions it needs.
    • For example, it can deploy containers and run migrations, but not drop databases.

Production deployment jobs should never use your personal admin credentials.
Always use scoped, revocable service accounts with minimum required permissions.

Release Strategies for Production

CI/CD to production is not just “build and deploy”. The way you release changes affects user experience and risk.

1. Direct deployment

The simplest form:

  1. Build image.
  2. Stop old application.
  3. Start new application.

This is acceptable for small internal systems, but it causes downtime.

2. Blue / Green deployments

You maintain two environments:

Process:

  1. Deploy new version to Green.
  2. Run tests and smoke checks on Green.
  3. Switch traffic from Blue to Green, for example by changing load balancer config.
  4. Keep Blue around for quick rollback, then remove it when confident.

Advantages:

3. Rolling deployments

You update a subset of instances at a time:

  1. Have $N$ application instances behind a load balancer.
  2. Take 1 instance out of rotation.
  3. Update that instance to the new version.
  4. Put it back in rotation.
  5. Repeat until all $N$ are updated.

Rolling deployments minimize downtime and can be partly automated by your orchestrator (for example Kubernetes).

4. Canary releases

You send a small percentage of traffic to the new version first:

  1. Deploy the new version alongside the old.
  2. Route, for example, 5% of traffic to the new version.
  3. Monitor metrics and errors.
  4. If everything is fine, gradually increase to 100%.
  5. If not, route back to the old version.

Canary releases are very useful when releasing risky changes in production.

Environment Configuration in Production

Configuration management in production has stricter rules than in development.

Typical pattern:

  1. CI builds a Docker image from a specific Git commit.
  2. CI pushes this image to a registry with a version tag, for example my-api:1.4.0.
  3. CI deploys the image to staging.
  4. After tests and approvals, CI deploys the same image to production, but with different environment variables.

Never embed production secrets in:

  • Source code
  • Docker images
  • Public CI logs
    Use environment variables, secret managers, or encrypted configuration files managed outside the code repository.

Handling Database Changes in Production

Databases in production are sensitive. Schema changes must be coordinated with application releases.

Typical approach in CI/CD:

  1. Migration step in the pipeline
    • Before or during deployment, a job runs migration scripts.
    • For example, using Alembic for SQLAlchemy.
  2. Backward‑compatible migrations
    • First migration: add new columns or tables.
    • Second migration: update application to use new columns.
    • Later migration: remove old columns when no longer used.
  3. Safe failure behavior
    • If migration fails, the deployment should stop.
    • Application should not start with a partially updated schema.

Example sequence:

You design releases like this:

  1. Release A (migration only)
    • Add first_name and last_name columns, keep name.
    • Optionally backfill from name.
  2. Release B (application update)
    • Application writes to and reads from first_name and last_name.
    • Still writes name for safety.
  3. Release C (cleanup)
    • Remove name column, after verifying nothing uses it.

This multi‑step plan avoids production outages caused by incompatible schema changes.

Versioning in Production

Every production deployment should be traceable:

json 
{
  "version": "1.3.2",
  "git_commit": "abc1234",
  "build_time": "2026-08-27T10:15:00Z"
}

This helps you answer:

Observability Requirements in Production

Production environments must be observable. CI/CD deployments should be coupled with checks of:

In practice:

  1. After deployment, CI triggers basic smoke tests.
  2. Team members check dashboards for spikes in:
    • HTTP 5xx errors.
    • Response times.
    • Database query time.
  3. If metrics degrade beyond a threshold, you consider rollback.

You can even automate this:

Rollback Strategies

No matter how careful you are, some releases will fail in production. Plan rollbacks ahead of time.

Common rollback options:

  1. Roll back to previous container image
    • Keep at least one previous release available.
    • CI/CD can redeploy the earlier version quickly.
  2. Blue / Green rollback
    • If you use Blue / Green, you just switch traffic back to the previous environment.
  3. Feature flag kill switch
    • If a particular feature causes problems, you disable it via configuration without a full rollback.

Key guidelines:

In production incidents, time matters.
Have a predefined rollback plan instead of inventing it during an outage.

Access Control and Auditing

In production, you control who can:

Core practices:

This is important for both security and debugging.

Example: Simple Production Pipeline Flow

A minimal, safe flow for production might look like this:

  1. Developer merges feature branch to main.
  2. CI builds Docker image my-api:1.0.0 and runs tests.
  3. CI deploys my-api:1.0.0 to staging automatically.
  4. Staging tests pass, QA verifies features manually.
  5. A release manager clicks “Promote to production” in CI.
  6. CI:
    • Applies database migrations in production.
    • Performs a rolling deployment of my-api:1.0.0 to application servers.
    • Runs smoke tests against production.
  7. Monitoring shows stable metrics for 30 minutes.
  8. Release is marked successful in the release log.

If something goes wrong at step 6 or 7:

Summary

Production environments are special because:

When you design CI/CD for production, always optimize for predictability, safety, and traceability, even if that seems slower. In real systems, controlled change is far more valuable than fast, chaotic change.

Views: 8

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3517 / 24h
Impressum / Imprint
Datenschutz / Data Policy