KAHIBARO
Discord Login Register ☰
☰

32.11. CI/CD Pipeline

Table of Contents

Why You Need a CI/CD Pipeline for the Final Project

In your final project you are building a production backend. A CI/CD pipeline is how you move from “it runs on my laptop” to “it is reliably deployed to real users without breaking things every week”.

CI/CD stands for:

For the final project we will focus on a simple but realistic setup:

  1. Every push or pull request triggers:
    • Code checks and tests.
    • Docker image build.
    • Image push to a container registry.
  2. On a special branch or tag, the pipeline also:
    • Deploys to a staging or production environment.

Core idea: Every change goes through the same automated steps: build → test → package → deploy. You should never deploy code that has skipped the pipeline.

You will not learn CI/CD from scratch here, that is covered in the CI/CD module. In this chapter you apply those ideas to this specific project.

Defining Pipeline Goals for the Final Project

Before writing any configuration, define what your pipeline must guarantee.

Minimal goals for this project

For a small production backend, a practical CI/CD pipeline should:

  1. Check code quality
    • Run formatting checks (for example black --check, isort --check).
    • Run linting and static checks (for example ruff, flake8, mypy).
  2. Run automated tests
    • Run pytest against your test suite.
    • Fail fast if any test fails.
  3. Build a production Docker image
    • Build from your project Dockerfile.
    • Tag the image with something meaningful, for example commit SHA or version.
    • Push it to a container registry (GitHub Container Registry, Docker Hub, or GitLab Registry).
  4. Deploy consistently
    • Use the same image in staging and production.
    • Use defined environment variables and secrets from the CI/CD platform, not hard-coded values.

Example stages

A simple pipeline can have these stages:

StagePurposeWhen it runs
lintFormat and style checksEvery push and pull request
testRun test suiteEvery push and pull request
buildBuild and push Docker imageMain branch and release tags
deployDeploy or trigger deploy script on the serverOnly on main branch or release tag

Rule: The same artifact that passed tests should be the one you deploy. Do not rebuild “by hand” on the server with different code.

Choosing a CI/CD Platform

For your final project you can pick any common CI/CD provider. The configuration syntax changes, but the structure stays the same.

Typical choices:

PlatformWhere config livesCommon use case
GitHub Actions.github/workflows/*.ymlProjects hosted on GitHub
GitLab CI/CD.gitlab-ci.ymlProjects hosted on GitLab
Other (CircleCI…)Provider-specific config filesOptional, not required for this course

In examples below we will use GitHub Actions, because GitHub is common for personal and portfolio projects. You can translate the idea to another provider later.

Structuring Your CI/CD Configuration

Think of your CI/CD config as “code that runs in response to Git events”. You want it as clear and modular as your application code.

A typical GitHub Actions workflow file for this project might be:

yaml 
name: ci-cd
on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]
jobs:
  lint:
    # ...
  test:
    # ...
  build:
    needs: [lint, test]
    # ...
  deploy:
    needs: [build]
    # ...

Key ideas:

You should keep the workflow file:

Continuous Integration for the Final Project

CI is about checking that your changes are safe to merge. For the final project, your CI should at least run:

  1. Dependency installation
  2. Code quality tools
  3. Test suite

Example CI job for linting

Using GitHub Actions and Python:

yaml 
jobs:
  lint:
    name: Lint and format
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
          # or requirements-dev.txt if you split dev deps
      - name: Run format checks
        run: |
          black --check .
          isort --check-only .
      - name: Run lint
        run: |
          ruff check .
          # or flake8, pylint, etc.

If any of these commands returns a non-zero exit code, the job fails and the pipeline stops moving forward.

Example CI job for tests

yaml 
  test:
    name: Run tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: app
          POSTGRES_PASSWORD: app
          POSTGRES_DB: app_test
        ports:
          - 5432:5432
        options: >-
          --health-cmd="pg_isready -U app"
          --health-interval=10s
          --health-timeout=5s
          --health-retries=5
      redis:
        image: redis:7
        ports:
          - 6379:6379
    env:
      DATABASE_URL: postgresql://app:app@localhost:5432/app_test
      REDIS_URL: redis://localhost:6379/0
      ENVIRONMENT: test
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Run tests
        run: |
          pytest -q

This job starts temporary PostgreSQL and Redis containers, configures environment variables, then runs your tests against them.

Rule: The CI environment should be as close as possible to your real environment: same Python version, same database engine, same Redis, same environment variables.

Building and Publishing Docker Images in CI

Your final project backend will run inside Docker in production, so your CI must:

  1. Build the Docker image from your Dockerfile.
  2. Tag it with a useful tag.
  3. Push it to a container registry.
  4. Use that tag for deployment.

Choosing a container registry

Typical options:

RegistryURL format
GitHub Container Registryghcr.io/USERNAME/IMAGE:TAG
Docker HubUSERNAME/IMAGE:TAG
GitLab Container Registryregistry.gitlab.com/…/IMAGE:TAG

In examples we will use GitHub Container Registry (ghcr.io).

Required secrets

In your GitHub repository settings, you will need:

These are stored securely and used in the pipeline.

CI job to build and push the image

yaml 
  build:
    name: Build and push Docker image
    runs-on: ubuntu-latest
    needs: [lint, test]
    env:
      REGISTRY: ghcr.io
      IMAGE_NAME: ${{ github.repository }}  # e.g. username/project
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ secrets.GHCR_USERNAME }}
          password: ${{ secrets.GHCR_TOKEN }}
      - name: Extract metadata (tags, labels)
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha
            type=raw,value=latest,enable={{is_default_branch}}
      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

This job:

You can inspect images afterward with:

bash 
docker pull ghcr.io/USERNAME/PROJECT:latest

Integrating Deployment with CI/CD

Once the image is pushed, you want your deployment environment to use that image. There are two common patterns:

  1. CI deploys directly to the server
    • CI connects via SSH and runs deployment scripts.
  2. Server pulls on demand
    • CI updates metadata (for example push a tag, update a manifest).
    • A tool on the server watches for changes and updates containers.

For a simple final project, SSH-based deployment from CI is often the easiest to understand.

Basic deployment flow

  1. CI builds and pushes image ghcr.io/USERNAME/PROJECT:sha-abcdef.
  2. CI connects to the server via SSH.
  3. CI runs a script on the server that:
    • Pulls the new image.
    • Restarts the Docker Compose stack (or container) using that image.
    • Performs database migrations if needed.

Example deployment script on the server

Put this on the server as deploy.sh:

bash 
#!/usr/bin/env bash
set -euo pipefail
APP_IMAGE="$1"  # e.g. ghcr.io/USERNAME/PROJECT:sha-abcdef
echo "Pulling image: $APP_IMAGE"
docker pull "$APP_IMAGE"
export APP_IMAGE
echo "Running database migrations"
docker compose run --rm backend alembic upgrade head
echo "Restarting stack"
docker compose up -d

Your docker-compose.yml should use the APP_IMAGE:

yaml 
services:
  backend:
    image: "${APP_IMAGE}"
    env_file:
      - .env
    depends_on:
      - db
      - redis
  db:
    image: postgres:16
    # ...
  redis:
    image: redis:7
    # ...

CI deployment job using SSH

yaml 
  deploy:
    name: Deploy to production
    runs-on: ubuntu-latest
    needs: [build]
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Get image tag
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ghcr.io/${{ github.repository }}
          tags: type=sha
      - name: Deploy via SSH
        uses: appleboy/ssh-action@v1.1.0
        with:
          host: ${{ secrets.PROD_HOST }}
          username: ${{ secrets.PROD_USER }}
          key: ${{ secrets.PROD_SSH_KEY }}
          script: |
            cd /opt/your-app
            ./deploy.sh "${{ steps.meta.outputs.tags }}"

Required secrets:

Rule: Never hard-code server IPs, passwords, or private keys in your repository. Always use encrypted CI/CD secrets.

Handling Environments: Staging vs Production

For a real project you usually have at least:

You can support both by:

Example: different branches deploy to different environments

yaml 
on:
  push:
    branches: [ develop, main ]
jobs:
  # lint, test, build as before ...
  deploy-staging:
    runs-on: ubuntu-latest
    needs: [build]
    if: github.ref == 'refs/heads/develop'
    # similar SSH deployment but to STAGING_* secrets
  deploy-production:
    runs-on: ubuntu-latest
    needs: [build]
    if: github.ref == 'refs/heads/main'
    # SSH deployment with PROD_* secrets

You might use:

The same Docker image can run in both, only configuration differs.

Managing Secrets and Configuration in CI/CD

Your backend needs secrets, for example:

These must never be committed to the repository.

Where to store secrets

In GitHub Actions:

In your jobs you can pass secrets to the environment:

yaml 
env:
  DATABASE_URL: ${{ secrets.DATABASE_URL }}
  JWT_SECRET: ${{ secrets.JWT_SECRET }}

On the server you can also have a .env file that is not committed, used by Docker Compose:

env 
DATABASE_URL=postgresql://user:pass@db:5432/app
JWT_SECRET=super-secret
REDIS_URL=redis://redis:6379/0

Rule: Configuration that changes per environment (staging vs production) should be in environment variables, not in your code.

Making the Pipeline Fast and Reliable

Even a small pipeline can become painful if it is slow or flaky. A few tips tailored for this project:

Cache dependencies

For Python projects you can cache your virtual environment or pip cache to avoid re-downloading packages each run.

Example (GitHub Actions):

yaml 
      - name: Cache pip
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-

Limit what triggers full pipelines

You can:

Keep tests focused

If tests are slow:

Make failure obvious

Configure your Git platform so that:

Putting It All Together: Example CI/CD Workflow File

Here is a compact example that you can adapt for the final project. It focuses on core steps and omits optional optimizations:

yaml 
name: final-project-ci-cd
on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]
env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}
jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Format and lint
        run: |
          black --check .
          isort --check-only .
          ruff check .
  test:
    runs-on: ubuntu-latest
    needs: [lint]
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: app
          POSTGRES_PASSWORD: app
          POSTGRES_DB: app_test
        ports:
          - 5432:5432
      redis:
        image: redis:7
        ports:
          - 6379:6379
    env:
      DATABASE_URL: postgresql://app:app@localhost:5432/app_test
      REDIS_URL: redis://localhost:6379/0
      ENVIRONMENT: test
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Run tests
        run: pytest -q
  build:
    runs-on: ubuntu-latest
    needs: [test]
    if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/develop'
    steps:
      - uses: actions/checkout@v4
      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ secrets.GHCR_USERNAME }}
          password: ${{ secrets.GHCR_TOKEN }}
      - name: Extract metadata (tags)
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha
            type=raw,value=latest,enable={{is_default_branch}}
      - name: Build and push
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
  deploy-staging:
    runs-on: ubuntu-latest
    needs: [build]
    if: github.ref == 'refs/heads/develop'
    steps:
      - name: Get image tag
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: type=sha
      - name: Deploy to staging
        uses: appleboy/ssh-action@v1.1.0
        with:
          host: ${{ secrets.STAGING_HOST }}
          username: ${{ secrets.STAGING_USER }}
          key: ${{ secrets.STAGING_SSH_KEY }}
          script: |
            cd /opt/final-project-staging
            ./deploy.sh "${{ steps.meta.outputs.tags }}"
  deploy-production:
    runs-on: ubuntu-latest
    needs: [build]
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Get image tag
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: type=sha
      - name: Deploy to production
        uses: appleboy/ssh-action@v1.1.0
        with:
          host: ${{ secrets.PROD_HOST }}
          username: ${{ secrets.PROD_USER }}
          key: ${{ secrets.PROD_SSH_KEY }}
          script: |
            cd /opt/final-project
            ./deploy.sh "${{ steps.meta.outputs.tags }}"

You do not need to copy this file exactly. Instead, adapt it to your repository:

If you keep the core ideas in place, you will have a robust, automated path from code commit to running application, which is exactly what you want for a real production backend.

Views: 8

Comments

Please login to add a comment.

Don't have an account? Register now!

© 2025 Kahibaro
Visitors: 3037 / 24h
Impressum / Imprint
Datenschutz / Data Policy