24.10. Deployment Pipelines

Understanding Deployment Pipelines

A deployment pipeline is the automated path that your code follows from a Git commit to running in a live environment. It connects building, testing, packaging, and deploying your backend into a repeatable process.

You can think of it as a factory line: code goes in at one end, a deployed application comes out at the other, with automatic checks at every station.

This chapter focuses on the structure and logic of deployment pipelines, not on specific CI/CD tools. Those are covered elsewhere in this course.

A deployment pipeline is a series of automated stages that transform and verify your code changes until they are safely deployed to an environment.

Typical stages include: build, test, package, deploy, and verify.

Core Ideas of a Deployment Pipeline

A deployment pipeline has a few key goals:

Repeatable: The same change, same inputs, same environment config, always gives the same result.
Automated: No manual steps where humans might forget or misclick.
Fast feedback: You know quickly if a change is broken.
Safe deployments: Reduce risk of breaking production.

A pipeline usually runs on every push, merge, or tag in your Git repository. The behavior might differ per branch:

Branch / Tag	Typical Behavior
`feature/*`	Build + unit tests only
`develop`	Build + unit + integration tests + deploy to dev
`staging`	Full pipeline + deploy to staging
`main` / `master`	Full pipeline + deploy to production (maybe gated)
`v1.2.3` tag	Production release for version 1.2.3

Pipeline Stages from Code to Production

Below is a common high level flow. Each step can be mapped to a CI/CD job or stage.

Code checkout
Build
Static checks
Tests
Package / artifact build
Deploy to non-production environments
Automated checks in environment
Production deployment
Post-deploy verification

Not every project needs every step, but most real systems include at least a subset.

1. Code Checkout

The pipeline always starts with the source code.

Typical tasks:

Fetch the repository at a specific commit.
Optionally fetch submodules.
Download any private dependencies (from internal registries).

Important points:

The pipeline should use exactly the same commit that was reviewed and merged.
You normally do not run git pull on servers manually. The CI/CD system does it for you and ships built artifacts.

2. Build Stage

The build stage converts source code into something that can run or be deployed.

Examples for backend projects:

Python:

Install dependencies with pip install -r requirements.txt.
Optionally build a wheel package.
Build a Docker image that contains your code and dependencies.

Node.js:

npm install and npm run build if you have compiled assets.

Go, Java, etc.:

Compile source code into binaries or JARs.

For Docker-based deployments, the build stage often:

Logs into a container registry (if private).
Builds a Docker image with a tag, for example:

my-api:commit-sha
my-api:v1.2.3

Pushes the image to the registry.

Always tie a build artifact (like a Docker image) to a specific commit hash so you can reproduce and roll back deployments.

3. Static Checks

Static checks run without executing the application. They catch problems early.

Common static checks:

Code format: black, isort, prettier.
Linting: flake8, pylint, eslint.
Type checks: mypy, pyright, tsc.
Security scanning: bandit, dependency vulnerability scanners.

In a deployment pipeline:

Fail the pipeline if any static check fails.
Keep this stage relatively fast to give quick feedback.

Example grouping of static check jobs:

Job name	Purpose
`lint`	Style and basic errors
`type-check`	Type-related issues
`security-scan`	Known vulnerabilities in deps

4. Test Stages

Tests are usually split into stages according to speed and scope:

Unit tests: Fast, isolated tests of functions and classes.
Integration tests: Interact with real or test databases, queues, etc.
End-to-end (E2E) tests: Run the whole system against a test environment.

A common pattern:

Run unit tests after build and static checks.
Only if those pass, build deployment artifacts (Docker image).
After deployment to a test environment, run integration or E2E tests.

You might also:

Collect test reports and coverage reports as pipeline artifacts.
Fail the pipeline if coverage falls below a threshold.

5. Packaging and Artifacts

A deployment pipeline often produces artifacts that are then reused in later stages.

For example:

Compiled code.
Python wheels or .whl files.
Docker images.
Configuration templates.
Database migration bundles.

Key rule:

Build an artifact once and promote it through environments.
Do not rebuild from source separately for staging and production.

This ensures that staging and production are running exactly the same artifact.

6. Deploying to Environments

The same artifact usually moves through multiple environments:

Typical sequence:

Development environment
Staging or QA environment
Production environment

Each deployment step should be:

Automated by the pipeline.
Configured through environment-specific configuration, not separate code.

Configuration per Environment

You should not change the code when you deploy to production. Instead, you swap configuration values, usually through environment variables.

Examples:

Setting	Dev value	Staging value	Prod value
`DATABASE_URL`	local PostgreSQL	staging PostgreSQL cluster	production PostgreSQL cluster
`REDIS_URL`	local Redis	staging Redis	production Redis
`DEBUG`	`true`	`false`	`false`
`ALLOWED_ORIGINS`	`http://localhost:3000`	`https://staging.example.com`	`https://app.example.com`

The pipeline passes these values through Docker, Kubernetes manifests, or systemd service files, depending on your platform.

7. Automated Checks in Each Environment

After deployment to a non-production environment, you usually want automatic validation.

Common checks:

Health checks:

Hit /health or /live endpoint.
Confirm the service returns 200 OK.

Smoke tests:

Simple, high-level checks that major features work:

Can you create a test user?
Can you log in?
Can you call a key API endpoint?

These checks can be part of the pipeline and must pass before you:

Promote the build to the next environment.
Or allow manual QA to start.

Pipelines for Different Environments

You usually define pipelines or pipeline branches that map to environments in a consistent way.

Dev / Feature Branch Pipelines

For feature branches, your goal is fast feedback, not full deployment.

Common behavior:

Trigger on push to feature/*.
Run:

Build
Static checks
Unit tests

Possibly build a Docker image and push to a non-production registry.
No deployment to shared environments, or deploy to ephemeral per-branch environments.

Ephemeral environments are temporary environments created for a branch, then destroyed when the branch is merged.

Staging Pipelines

Staging pipelines are closer to production:

Trigger on merge into staging or a specific release branch.
Steps usually include:

Build (if not already built on earlier stages).
Run full test suite.
Build or reuse Docker image, push to registry.
Deploy to staging environment.
Run smoke tests or integration tests against staging.

Some teams also run manual QA or user acceptance testing on staging.

Production Pipelines

Production pipelines are the most cautious:

Possible triggers:

Merge into main / master.
A Git tag like v1.2.3.
Manual approval after staging passes.

Common production deployment strategies (concept only, implementation is tool specific):

Strategy	Description
Direct (all at once)	Update all instances at the same time
Rolling	Update instances one by one or in batches
Blue-green	Deploy to a new set of servers, switch traffic over
Canary	Deploy to a small subset first, then expand if healthy

After production deploy, the pipeline often:

Checks health endpoints.
Optionally runs a small set of smoke tests.
Notifies the team via Slack, email, or other channels.

Manual Approvals and Gates

For critical environments like production, many teams use manual approval steps.

A gate might:

Require a human to click "approve" in the CI/CD system.
Require passing test status and quality checks first.
Optionally require a change request or ticket id.

Typical flow:

Code merges into main.
Pipeline builds and tests.
Successfully built artifact is deployed to staging.
Tests pass in staging.
Pipeline pauses at a manual approval step: "Deploy to production?"
Engineer reviews changes and production status.
Engineer approves, pipeline continues to production deploy.

Rollbacks and Versioning in Pipelines

A good deployment pipeline not only deploys forward but also helps you roll back when something goes wrong.

Common rollback approaches:

Redeploy the last known good artifact (for example Docker image with a previous tag).
Switch traffic back in blue-green deployments.

You need a clear versioning scheme to do this reliably.

Examples:

Docker tags:

my-api:commit-sha
my-api:1.2.3

Git tags:

v1.2.3

Always make it easy and fast to deploy a previous known good version from your pipeline, without manual rebuilding.

This often means:

Keeping build artifacts in a registry or artifact store.
Having a pipeline job that can deploy a specific version by tag or commit.

Pipeline Configuration as Code

Deployment pipelines are usually described in files inside your repository, for example:

.github/workflows/*.yml for GitHub Actions.
.gitlab-ci.yml for GitLab CI.
azure-pipelines.yml for Azure DevOps.
Jenkinsfile for Jenkins.

Benefits of storing pipeline configuration in version control:

You can review changes with code review.
You can roll back pipeline changes.
Pipeline changes are tied to application changes, which improves traceability.

Core ideas you often express in these files:

Stages: build, test, deploy.
Jobs inside stages.
Dependencies between jobs.
Triggers on branches, tags, or pull requests.
Secrets references for environment configuration.

Secrets in Deployment Pipelines

Deployment usually needs secrets, for example:

Database passwords.
API keys.
Cloud credentials.
JWT signing keys.

You must:

Never commit secrets in the repository.
Store them in the CI/CD system’s secret store, or in a dedicated secret management system.
Inject them at runtime into jobs as environment variables.

The pipeline should:

Access secrets only in jobs that need them, for example deployment jobs.
Avoid printing secrets in logs.

Example Logical Pipeline Flow

Here is a simplified, tool-agnostic flow for a backend with Docker deployment:

Trigger

On push to any branch.

Build & Unit Test Stage

Checkout code at commit X.
Install dependencies.
Run static checks (lint, type).
Run unit tests.
If any step fails, stop here.

Build Artifact Stage

Build Docker image my-api:X.
Push image to registry.
Store commit hash and image tag mapping.

Staging Deployment Stage

Only for staging and main branches.
Deploy my-api:X to staging.
Run smoke tests against staging.
If tests fail, mark pipeline as failed.

Production Deployment Stage

Only for main branch, maybe after manual approval.
Deploy my-api:X to production using rolling updates.
Run basic health checks.
Notify team of deployment status.

Rollback Option

A separate job or pipeline can redeploy my-api:Y where Y is the last known good tag.

Metrics and Feedback from Pipelines

A deployment pipeline is also a source of process metrics:

Lead time for changes: Time from commit to production.
Deployment frequency: How often you deploy.
Change failure rate: How often a deployment causes incidents.
Mean time to restore: How fast you can recover from a bad deployment.

Even as a beginner, you can start simple:

Track how long your pipeline takes to run.
Track how often builds or deployments fail.
Gradually improve the slowest stages.

Starting Simple with Deployment Pipelines

You do not have to build a complex pipeline on day one. A minimal useful pipeline might be:

On push:

Run lint and unit tests.

On push to main:

Build Docker image and push to registry.
Deploy container to a test server via script.

Later:

Add staging environment.
Add manual approval for production.
Add smoke tests and rollback strategies.

Focus on these principles:

Automate repetitive steps.
Keep the pipeline reliable and transparent.
Treat pipeline configuration as code.
Make rollback possible and quick.

These ideas will apply regardless of the specific CI/CD tool or hosting platform you use.