KAHIBARO
Discord Login Register ☰
☰

24.5. CI/CD Pipelines

Table of Contents

Understanding CI/CD Pipelines

A CI/CD pipeline is the automated path that your code follows from a Git commit to running in a real environment. Instead of doing steps manually, you describe them once, and a pipeline runs them in the same way every time.

This chapter focuses on what a CI/CD pipeline is, what typical stages look like, and how it fits into backend development. The next chapters will show concrete examples with GitHub Actions and GitLab CI/CD.

What Is a CI/CD Pipeline?

A CI/CD pipeline is a series of automated steps that run whenever code changes. These steps usually include:

You define these steps in a configuration file, usually in YAML, stored in the same repository as your code.

Key idea: A CI/CD pipeline is code that automates how you build, test, and deploy your application. If you cannot run it with one command or one button, it is not a proper pipeline.

Basic Structure of a Pipeline

Most pipeline systems share the same core ideas:

A typical minimal pipeline has at least these stages:

  1. Checkout / setup
  2. Test
  3. Build
  4. Deploy

The exact names differ between tools, but the idea is the same.

Example: Conceptual Pipeline Flow

StageExample jobPurpose
setupCheckout repo, install PythonPrepare environment
testRun unit and API testsProve code still works
buildBuild Docker imageCreate deployable package
deployDeploy to staging or prodRelease new version of application

For a backend project, your pipeline will usually at least run Python tests and build Docker images.

Triggers and Branch-Based Pipelines

You rarely want to run the same actions for every branch. For example, you might:

Typical triggers include:

In your config file you usually describe something like:

This is how pipelines enforce safe workflows.

Common Pipeline Stages for Backend Projects

1. Checkout and Setup Stage

This stage prepares everything other stages need.

Typical steps:

Example (conceptual, tool-agnostic):

yaml 
- stage: setup
  jobs:
    - name: setup-python
      steps:
        - checkout-code
        - install-python: "3.11"
        - pip-install: "requirements.txt"

The exact syntax differs between tools, but the meaning is the same: get code, install Python, install packages.

2. Test Stage

This is the core of Continuous Integration. The goal is to reject broken changes automatically.

Typical jobs:

Many teams separate "tests" into multiple jobs for speed and clarity.

Example layout:

JobCommand examplePurpose
unit-testspytest tests/unitFast feedback
integrationpytest tests/integrationDB, Redis, external systems
style-checkflake8 appCode style
type-checkmypy appType safety

A pipeline can run these jobs in parallel within the same stage. The stage is successful only if all jobs succeed.

Rule: A pipeline must fail fast when any test fails. Do not allow broken tests to "pass" or be ignored. If tests are flaky, fix them, do not just rerun manually.

3. Build Stage

Once your code passes tests, you create something that can be deployed. For backend applications this is often:

In Docker-based backend projects, a common pattern is:

  1. Build a Docker image using your Dockerfile.
  2. Tag it with the commit SHA and/or a version.
  3. Push it to a container registry, for example GitHub Container Registry or Docker Hub.

Conceptual example:

yaml 
- stage: build
  needs: [test]  # only run if tests passed
  jobs:
    - name: build-and-push-image
      steps:
        - docker-build: "my-api:${COMMIT_SHA}"
        - docker-push: "registry.example.com/my-api:${COMMIT_SHA}"

The important part is the dependency: the build stage must not run if tests fail.

4. Deploy Stage

The deploy stage uses the artifact from the build stage to update a real environment.

Common patterns:

For Docker-based deployments, a deploy job might:

Pipeline systems usually allow:

Rule: Never deploy directly from a developer machine to production. Always deploy from a CI/CD pipeline using a versioned artifact that has passed tests.

Parallelism and Dependencies

Pipelines are often represented as graphs. Jobs can run:

For example:

  1. lint and unit-tests run in parallel.
  2. integration-tests depend on unit-tests.
  3. build depends on all tests.
  4. deploy depends on build.

Visual idea:

This structure gives fast feedback, since simple checks finish quickly and can fail the pipeline before slower jobs even start.

Artifacts and Caching

Pipelines often run on clean, temporary machines. They start from zero each time. Two concepts help avoid repeated work:

Artifacts

Artifacts are files produced by one job and passed to others. Examples:

For example, an integration-tests job might upload a test report as an artifact, then a later "collect-reports" job could combine them or publish them as a web page.

Caches

Caches speed up repeated work, such as:

You usually configure the cache key using the requirements.txt checksum and possibly the Python version. If the dependencies file has not changed, the pipeline can reuse cached dependencies.

Rule: Use artifacts for results that must be exact and tracked, like build outputs, and caches only for things that can be safely regenerated, like dependencies.

Environments and Promotion

A key practice in CI/CD is promoting the same artifact through multiple environments.

Common environments:

Instead of rebuilding for each environment, you:

  1. Build one Docker image in the pipeline.
  2. Test it.
  3. Deploy the same image to staging.
  4. If staging is good, deploy exactly the same image to production.

This avoids "works in staging but not in prod" due to different builds or dependency versions.

Typical pipeline design:

Manual vs Automatic Steps

Not every pipeline step must be fully automatic.

Common patterns:

Most CI/CD systems let you mark a job as "manual" or "requires approval" and still keep it part of the pipeline graph.

Handling Secrets in Pipelines

Pipelines usually need secrets:

You must never put them in your repository.

Common approaches:

Example behavior:

Rule: Secrets must be stored securely by the CI/CD system and never committed to Git or printed in pipeline logs.

Designing a Simple Backend Pipeline

To tie everything together, here is a conceptual pipeline for a FastAPI + PostgreSQL project using Docker. The syntax is generic, but the flow can be implemented in GitHub Actions or GitLab CI/CD.

Stages:

  1. test
  2. build
  3. deploy_staging
  4. deploy_production

Jobs:

yaml 
stages:
  - test
  - build
  - deploy_staging
  - deploy_production
jobs:
  test:
    stage: test
    runs-on: linux
    steps:
      - checkout-code
      - install-python: "3.11"
      - pip-install: "requirements.txt"
      - run: "pytest --maxfail=1 --disable-warnings -q"
  build:
    stage: build
    needs: [test]
    rules:
      - if: branch in ["develop", "main"]
    steps:
      - checkout-code
      - docker-login: "registry.example.com"
      - docker-build: "registry.example.com/my-api:${COMMIT_SHA}"
      - docker-push: "registry.example.com/my-api:${COMMIT_SHA}"
  deploy-staging:
    stage: deploy_staging
    needs: [build]
    rules:
      - if: branch == "develop"
    environment: "staging"
    steps:
      - ssh: "staging-server.example.com"
      - run: |
          docker pull registry.example.com/my-api:${COMMIT_SHA}
          docker-compose up -d
  deploy-production:
    stage: deploy_production
    needs: [build]
    rules:
      - if: tag matches "v*"
    environment: "production"
    when: manual
    steps:
      - ssh: "prod-server.example.com"
      - run: |
          docker pull registry.example.com/my-api:${COMMIT_SHA}
          docker-compose up -d

This example is not tied to a specific CI/CD tool, but it shows how you can:

How CI/CD Pipelines Fit Into Your Backend Workflow

In a typical backend workflow:

  1. You create a branch, for example feature/add-tasks-endpoint.
  2. You commit and push changes.
  3. The CI pipeline runs tests and checks.
  4. You open a pull request.
  5. The pipeline runs again on the pull request.
  6. If everything is green, you merge to develop or main.
  7. On merge, the CD part of the pipeline builds and possibly deploys.
  8. If a production deploy breaks something, you roll back using the pipeline, for example by redeploying a previous image.

The pipeline becomes a central part of your development process. Over time, you will add:

Future chapters will show how to express these ideas in real CI/CD tools, specifically GitHub Actions and GitLab CI/CD.

Views: 7

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3042 / 24h
Impressum / Imprint
Datenschutz / Data Policy