24.3. Continuous Delivery
Table of Contents
Understanding Continuous Delivery
Continuous Delivery, usually shortened to CD, is a software engineering practice where your code is always in a deployable state and can be released to users at any time with a small, repeatable, low‑risk process.
You will often see CI/CD used together. Continuous Integration (CI) focuses on merging and testing code frequently. Continuous Delivery (CD) focuses on making that tested code ready for production release at all times.
Core idea of Continuous Delivery:
Every change that passes automated tests is automatically prepared for release to production, and releasing it is a business decision, not a technical challenge.
In backend development, this is especially important because servers, databases, and APIs must be updated safely without downtime or data loss.
Continuous Delivery vs Continuous Deployment
These two terms are close but not the same.
| Practice | What gets automated | Who decides to release? |
|---|---|---|
| Continuous Integration | Build and automated tests on each change | Developer merges when tests pass |
| Continuous Delivery | Build, test, package, and prepare for release | Human (team, product owner, SRE) |
| Continuous Deployment | Build, test, package, and auto release to prod | System releases automatically |
With Continuous Delivery:
- After code is merged to the main branch, the pipeline:
- runs tests,
- builds containers or artifacts,
- runs extra checks,
- deploys to staging,
- waits for a manual approval to deploy to production.
With Continuous Deployment:
- All the above happens and then the last step, deployment to production, is also automatic if everything passes.
Important distinction:
Continuous Delivery automates up to production and requires manual approval.
Continuous Deployment automates through production with no manual gate.
In this course, you will mostly focus on Continuous Delivery, because it is a safer and more common initial step for backend teams.
Why Continuous Delivery Matters for Backend Developers
For backend services, manual and infrequent releases often cause:
- Large batches of changes that are hard to debug.
- Long release windows and nighttime deployments.
- Rollbacks that are painful or impossible.
- Fear of releasing, which slows down development.
Continuous Delivery aims to solve this by:
- Releasing small changes frequently.
- Having repeatable, automated pipelines.
- Making releases boring and predictable.
Some concrete benefits:
- Faster feedback
You can deploy new backend endpoints or changes to staging quickly and test them with real clients or QA. - Reduced risk
Each deployment contains fewer changes. If something breaks, it is easier to find and fix. - Easier rollbacks
If each release is versioned and automated, rolling back often becomes a single command or button. - Better collaboration
Backend, frontend, QA, and product can all work against the same, frequently updated environments.
What Does “Always Deployable” Mean?
In Continuous Delivery, the main branch of your repository should always be:
- Green (all tests passing),
- Buildable (Docker images, binaries, or packages can be created),
- Deployable (can be rolled out without manual fixes).
This does not mean the application is perfect. It means that at any point, if the business asks for a release, you can:
- Take the current main commit.
- Trigger the release pipeline.
- Deploy to production safely.
Example in a Backend Context
Imagine a FastAPI application with a PostgreSQL database.
In a Continuous Delivery setup:
- Every merge to
main: - runs unit tests and API tests with pytest,
- builds a Docker image,
- runs database migrations against a test database,
- deploys the backend to a staging environment,
- runs smoke tests against staging (simple checks that the API is up and basic endpoints work).
If all steps pass, the pipeline marks this version as ready for production.
You might then:
- Review metrics or do manual exploratory testing on staging.
- Press an “Approve” button in your CI/CD platform to deploy the same version to production.
The Continuous Delivery Pipeline
A CD pipeline is a sequence of automated steps that move code from “just built” to “ready for production” and often beyond into “actually deployed”.
A typical backend Continuous Delivery pipeline might have these stages:
- Build
- Install dependencies.
- Run static checks (like linters or formatters).
- Build a Docker image of your backend.
- Test
- Run unit tests.
- Run integration tests with a real database (like PostgreSQL in Docker).
- Run API tests against the running backend.
- Package and Publish
- Tag the build (for example,
v1.4.3). - Push the Docker image to a container registry.
- Save artifacts like migration scripts or documentation.
- Deploy to Staging
- Deploy the new image to a staging environment.
- Apply database migrations on staging.
- Run smoke tests on staging (basic endpoints, health checks).
- Manual Approval
- Developer, QA, or product reviews staging.
- Someone clicks “approve” for production deployment.
- Deploy to Production
- Deploy the same image and configuration used in staging to production.
- Apply migrations (carefully, often with special strategies).
- Run health checks.
- Post‑Deployment Checks
- Monitor logs and metrics.
- If something goes wrong, roll back.
Simple Pipeline Flow
You can think of a Continuous Delivery pipeline as a function:
$$
\text{Pipeline}(commit) \rightarrow \text{deployable version}
$$
Where a deployable version includes:
- A built artifact (for example, Docker image),
- A version tag,
- All checks passing,
- A record of how to deploy it to environments.
Key Practices that Enable Continuous Delivery
You cannot just “turn on” Continuous Delivery. You need some practices in place.
Automated Testing
Without automated tests, you cannot safely rely on a pipeline to tell you if code is good enough.
In backend development you typically need:
- Unit tests for small pieces of logic.
- Integration tests that hit your database, caches, and message queues.
- API tests that call endpoints and verify responses.
- Smoke tests that check basic API functionality after deployment.
Rule:
If a bug is found after release, add or improve an automated test that would have caught it. Over time, your test suite becomes a safety net for Continuous Delivery.
Infrastructure as Code
Your environments should be created and updated via code:
- Docker Compose files,
- Kubernetes manifests,
- Terraform or similar tools,
- Ansible or shell scripts.
This makes environments:
- Repeatable (staging is similar to production),
- Versioned (you can see changes),
- Automated (no “secret manual steps”).
Configuration Management
Backend services often have environment-specific settings:
- Database URLs,
- API keys,
- Redis hosts,
- Feature flags.
In Continuous Delivery, configuration must be:
- Separated from code, for example via environment variables.
- Automated, so the pipeline can set these values for each environment.
You will cover this in more depth in the CI/CD and deployment chapters, but for CD it is essential.
Versioning and Artifacts
Every successful pipeline run should produce:
- An identifiable version, for example:
- Git commit hash, or
- Semantic version like
1.3.0, or - Combination like
1.3.0+build.42. - An artifact:
- Docker image pushed to a registry,
- Or a binary, or a package.
This lets you:
- Deploy exactly what passed tests.
- Roll back to a known good version.
Continuous Delivery Across Environments
You will often have multiple environments:
| Environment | Purpose | Who uses it |
|---|---|---|
| Dev | Developer playground, often local | Individual developers |
| Test | Run CI/CD tests, integration, performance | CI pipeline, QA |
| Staging | Production-like environment, final verification | QA, product, sometimes clients |
| Prod | Real users and real data | Customers, systems |
In Continuous Delivery, the goal is:
- Use the same build artifact (for example, the same Docker image) across test, staging, and production.
- Change only configuration and environment resources, not the code artifact.
Example: FastAPI Service Through Environments
- Code merged to
main. - CI builds
my-api:1.0.5Docker image, pushes it to a registry. - CI deploys
my-api:1.0.5to test and runs automated tests. - If successful, CI deploys
my-api:1.0.5to staging. - After manual approval, CI deploys the same image
my-api:1.0.5to production.
This is safer than rebuilding for every environment, because you avoid “it worked on staging but the production build is different”.
Release Strategies in Continuous Delivery
How you actually switch traffic to new backend versions is also part of Continuous Delivery.
Common strategies:
1. Rolling Updates
- Replace old instances of your backend with new ones gradually.
- Example in Kubernetes:
- Have 4 pods of version 1.
- Gradually start pods of version 2 and stop pods of version 1.
- If issues appear, stop and roll back.
2. Blue‑Green Deployments
- Maintain two identical production environments:
- Blue: currently serving traffic.
- Green: new version, not yet receiving traffic.
- Steps:
- Deploy new version to green.
- Run tests against green.
- Switch the load balancer to point from blue to green.
- If something goes wrong, switch back to blue.
3. Canary Releases
- Release the new version to a small percentage of traffic.
- Watch metrics and errors.
- If all is good, gradually increase the percentage.
These strategies reduce risk and integrate well with Continuous Delivery pipelines.
Database Changes in Continuous Delivery
Backend deployments often include database schema changes. Handling these safely is critical.
Some common practices:
- Database Migrations as Code
- Use tools like Alembic (for SQLAlchemy) to define migrations in version control.
- Migrations run automatically in the pipeline or as part of deployment.
- Backward Compatible Changes
- Try to make schema changes that work with both old and new code for some time.
- For example:
- Add a new column but do not remove the old one immediately.
- Deploy code that writes to both columns.
- Later deploy code that reads from the new column.
- Finally, remove the old column.
- Safe Rollbacks
- Consider what happens to data if you roll back the application but the database schema is newer.
- Sometimes you need rollback migrations or clear steps to fix data.
Rule for database changes:
Assume application and database may be on slightly different versions during deployment. Migrations must not break older or newer versions unexpectedly.
Example: A Simple Continuous Delivery Workflow
Here is a simplified Continuous Delivery pipeline for a FastAPI + PostgreSQL project using Docker and GitHub Actions. This is not full YAML, but it gives a sense of the flow.
- On merge to main:
1. Checkout code
2. Set up Python
3. Install dependencies
4. Run linters and formatters
5. Run unit tests
6. Build Docker image: my-api:${GIT_SHA}
7. Push Docker image to registry
8. Run integration tests using that image in Docker Compose (FastAPI + Postgres)
9. If all pass, deploy image to staging (for example via Docker Compose or Kubernetes)
10. Run smoke tests against staging (health endpoint, one main endpoint, etc.)
11. Mark build as "ready for production"- On manual approval for production:
1. Take same Docker image tag (my-api:${GIT_SHA})
2. Update production environment to use this image
3. Run database migrations
4. Run health checks
5. Notify team of deploymentNo code changes happen between staging and production. Only configuration (like database URLs and secrets) differ.
Team Practices Around Continuous Delivery
Technology is only part of Continuous Delivery. Team habits matter a lot.
Helpful practices:
- Trunk‑based development
- Developers merge small changes to
mainfrequently. - Feature branches are short‑lived.
- Avoid long‑running branches that diverge.
- Small, incremental changes
- Avoid huge PRs that are hard to review and test.
- Break features into small backend changes that can be deployed safely.
- Feature flags
- Use configuration switches to turn features on or off.
- You can deploy code for a feature that is disabled in production, then enable it later.
- Strict pipeline rules
- If the pipeline is red (failing), fix it before adding new changes.
- Do not bypass tests or manual approvals except in emergencies.
- Monitoring and observability
- After each deployment, you should have:
- Logs to inspect errors,
- Metrics (like response times, error rate),
- Health checks for services.
Challenges and How to Handle Them
Continuous Delivery is powerful but not trivial.
Typical challenges:
| Challenge | Mitigation idea |
|---|---|
| Flaky tests | Stabilize tests, isolate side effects, use fixed test data |
| Long pipelines | Parallelize steps, optimize Docker builds and test suites |
| Complex manual steps | Script and automate them, use Infrastructure as Code |
| Risky database changes | Use incremental migrations, test thoroughly in staging |
| Resistance to frequent deployments | Start with non‑critical services, show reliability improvements |
It is normal for a team to adopt Continuous Delivery gradually. You do not need perfection to start gaining value. Even automating build, tests, packaging, and staging deployment is already a big win.
How This Fits the Rest of the Course
In other parts of this course, you will:
- Use GitHub Actions or GitLab CI to build CI/CD pipelines.
- Dockerize your FastAPI backend and PostgreSQL database.
- Configure staging and production deployments.
- Add tests, logging, monitoring, and rollback strategies.
Continuous Delivery is the conceptual goal that explains why we set up pipelines in a particular way. It is about ensuring that your backend can be released safely and quickly whenever needed.
Views: 8
KAHIBARO