Table of Contents
Integrating Docker into CI/CD Pipelines
Continuous Integration and Continuous Delivery or Deployment, often shortened to CI/CD, benefits a lot from Docker, but it also introduces its own patterns and pitfalls. In this chapter you will focus on how Docker fits into automated build, test, and deployment pipelines, without trying to cover all of CI/CD as a topic on its own.
Why Docker Fits Naturally in CI/CD
CI/CD systems run your code on remote machines that can change frequently. These machines need a predictable environment, repeatable builds, and a way to run tests that does not depend on what is installed on the CI worker.
Docker images give you a standardized build and runtime environment. If your CI pipeline uses a Docker image as its build environment, that same image can be used locally by developers. This reduces the “it works on my machine” problem and makes failures more reproducible.
In many setups the pipeline itself is described as a sequence of steps, and each step can run in a specific Docker image. This lets you choose a lightweight image specialized for building your application, another for security scanning, and another for deployment.
Key idea: Use Docker images to make the CI environment identical or very close to the local development and production environments.
Common Patterns for Docker in CI Stages
In a CI/CD pipeline Docker usually shows up in three main stages. The first is the build stage, where a Docker image for your application is created. The second is the test stage, where this image is used to run automated tests. The third is the deploy stage, where the same image is pushed to a registry and later pulled by the production environment.
In a typical build stage the pipeline checks out your source code, then runs a docker build command to produce an image, usually tagged with both a commit reference and a human friendly version. In many systems you will then push this image to a registry so later stages, or other environments, can fetch it by tag or digest.
The test stage often runs your tests inside containers built from this new image. Sometimes the pipeline starts multiple containers, for example your app container and a database container, using Docker commands or an orchestration tool. The tests run inside this controlled environment, which makes them independent of the CI runner’s operating system.
Finally, the deploy stage uses that same image tag to update a staging or production environment. The pipeline usually does not rebuild the image at deployment time. Instead, it pulls the previously built image from the registry and starts containers from it. This guarantees that what you tested is exactly what you deploy.
Important rule: Build an image once in CI, test that exact image, and deploy the same image without rebuilding it.
Using Docker as the CI Build Environment
Many CI services let you define which Docker image each job should use. Instead of installing tools on the CI runner on each run, you bake them into a custom Docker image. The job then starts from that image so your toolchain is already present.
For example, if you build a Node application, you might use an official node image as your base, add your build tools, and store this as a custom builder image. Your CI job then declares this image and runs npm install or other commands inside it. Every pipeline run gets a consistent version of Node, npm, and build tools, with no need for manual setup.
This pattern also lets you keep the CI environment under version control. When you update dependencies in the builder image, you tag and store it in a registry. You can then choose when to switch your pipelines to use the new image by updating the image tag in your CI configuration.
Best practice: Encapsulate your CI toolchain inside versioned Docker images so builds are reproducible and easy to roll back.
Building and Tagging Images in CI
Inside a CI job that handles the build stage, the process usually follows a few consistent steps. The job first authenticates with your container registry so it has permission to push images. It then runs docker build to build the image from your Dockerfile, using context from the checked out repository.
Tagging is especially important in CI/CD. A typical strategy is to tag images using both a branch or version name and a commit identifier. For example, you might tag an image as myapp:1.4.0 and also as myapp:1.4.0-<short-commit-hash>. For main or master branches, some teams also push a myapp:latest tag.
Once the docker build completes successfully, the pipeline pushes these tags to the registry with docker push. Future stages should reference the same tags and should not rebuild the image. Some systems also record the image digest, a content based identifier, and store it as part of the build metadata.
Important rule: Use clear, consistent tags, such as version plus commit, so you can always trace a deployed container back to the exact source code revision.
Running Tests Inside Containers
After building the image, most pipelines run automated tests that use it. One approach is to run tests inside the same image you built, by using docker run with a suitable command. Another approach creates a test specific image that extends the main image with tools like test runners, but still reuses the same base.
If your application depends on services such as databases or message queues, the pipeline can start those as separate containers. For example, a job might start a database container, wait until it is ready, run your app container linked to that database, and then run integration tests. When the job completes, the CI system tears down the containers.
When tests run inside containers, developers can repeat failing tests locally by running the same docker run commands or using the same configuration files. This is one of the major benefits of using Docker throughout the CI and development workflows.
Pushing to Registries During the Pipeline
Once tests have passed, the pipeline usually pushes the image to a registry if it has not already done so. Sometimes the push happens right after a successful build, other times it is delayed until all tests succeed. If you push after tests, the image is guaranteed to represent a passing build, which is helpful for production usage.
The CI job must authenticate with the registry using credentials such as tokens or user and password pairs. These credentials should be stored securely in the CI system, not directly in the repository. The job uses these secrets to run docker login, then pushes the tags.
Many teams push images to different repositories or use different tags for staging and production. For example, a build from the main branch might be tagged as myapp:staging and deployed to a staging environment, while a tagged release might produce myapp:1.4.0 and be deployed to production.
Deploying Containers from CI
The final step in a Docker enabled CI/CD process is deployment. The core idea is to pull the already built image from the registry to the target environment and start containers from it. Exactly how this happens depends on your deployment platform, but the CI job usually orchestrates the steps.
In a simple setup, the CI pipeline connects to a remote server and runs commands that stop existing containers, pull the new image, and start new containers. In more advanced systems, the pipeline might interact with an orchestrator, but at the core it still passes the image tag and asks for an update.
Because the same image that passed tests is now being deployed, reproduction of issues becomes simpler. If you detect a bug, you can roll back by deploying an earlier image tag. This reinforces the importance of consistent tagging and avoiding rebuilds in deploy steps.
Managing CI Performance with Docker Caching
Building images in CI can be slow if you do not use caching wisely. Many CI systems support Docker layer caching, which lets subsequent builds reuse previously built layers if nothing changed in those steps. This can speed up docker build significantly.
To benefit from caching, you should keep frequently changing files, such as your application code, in later Dockerfile instructions, and static parts like base OS packages and tooling in earlier instructions. In CI this means that if you only change application code, the base layers can still be reused.
Some CI services provide shared cache volumes or caching of the Docker build directory. Others allow you to export and import build caches between jobs. When configuring your pipeline, it is worth enabling these features because they can reduce build times by a large margin.
Handling Secrets in Docker Based Pipelines
Secrets are common in CI/CD pipelines. You need credentials for registries, access tokens for deployment targets, and possibly keys for third party services. When Docker is involved you must ensure these values do not leak into images or logs.
A general approach is to let the CI platform manage secrets and inject them into jobs as environment variables or temporary files. The docker build itself should not bake secrets directly into the image. Instead, you can pass build time secrets through special mechanisms or only use secrets at runtime during docker run, not at image creation.
During deployment steps, the CI job may use secrets to authenticate to servers or orchestration systems, but these values should never appear in Dockerfiles or be written to files that become part of the image. Keeping secrets out of images ensures that even if an image is shared widely, it does not contain sensitive data.
Critical rule: Never bake long lived secrets into Docker images. Use your CI system’s secret management and inject them only at build or deploy time in a safe way.
Typical Pitfalls When Using Docker in CI/CD
There are several common mistakes seen when teams start using Docker in their pipelines. One mistake is rebuilding images in multiple stages, for example once for tests and again for deployment. This leads to differences between the tested and deployed containers. Another is relying on the latest tag without recording an immutable tag or digest, which makes it hard to know exactly what is running.
Resource contention is another issue. Many containers running in parallel on the same CI worker can cause memory or CPU exhaustion, which leads to flaky builds. Configuring sensible parallelism and cleanup of stopped containers and unused images is important for stable pipelines.
Finally, teams sometimes forget to clean up old tags in registries. CI builds that create many unique image tags can quickly fill a registry with unneeded images. Regular pruning, retention policies, and clear tag strategies help keep registries manageable over time.