24.7. GitLab CI/CD

Table of Contents

GitLab CI/CD

GitLab CI/CD is GitLab's built-in system for running tests, building artifacts, and deploying your backend projects automatically whenever something changes in your repository.

In this chapter you will learn how GitLab CI/CD works, how to configure pipelines with .gitlab-ci.yml, and how to use it specifically for backend projects, especially those that use Docker.

GitLab CI/CD Concepts

GitLab CI/CD has a few core concepts you must understand before you write any configuration.

Key building blocks

Concept	What it is	Simple example
Pipeline	A full run of CI/CD that contains multiple stages and jobs	"test, then build, then deploy"
Stage	A group of jobs that run in parallel, stages run in order	`test` stage then `build` stage
Job	A single task that runs in a runner	"run unit tests", "build Docker image"
Runner	The machine or container that executes jobs	GitLab shared runner, or your own VM
Artifact	Files created by jobs and stored by GitLab for later stages or download	Test reports, coverage, built packages
Cache	Temporary storage to speed up jobs, not a permanent artifact	Caching `venv`, `node_modules`, or `.m2`
Environment	A named deployment target	`staging`, `production`

A pipeline is created when a relevant event happens, for example:

A push to a branch
A merge request is opened or updated
A tag is pushed
A scheduled pipeline runs

The behavior of the pipeline is defined by a single file in the root of your repository: .gitlab-ci.yml.

The .gitlab-ci.yml File

The .gitlab-ci.yml file is the heart of GitLab CI/CD. It is where you define stages, jobs, images, variables, and rules.

Basic structure

A minimal structure usually has:

yaml

stages:
  - test
  - build
  - deploy
unit_tests:
  stage: test
  script:
    - echo "Running tests"
build_image:
  stage: build
  script:
    - echo "Building..."
deploy_prod:
  stage: deploy
  script:
    - echo "Deploying..."

Indentation and syntax

YAML uses spaces, not tabs.
Nested elements must be indented consistently, usually 2 spaces.
Job names must be unique in the file.

Important rule
If .gitlab-ci.yml is invalid YAML, no pipelines will run at all. Always validate the file before pushing, for example with an online YAML validator or GitLab's built-in CI Lint.

Defining stages

You define the order of stages in a top-level stages list:

yaml

stages:
  - lint
  - test
  - build
  - deploy

All jobs in the same stage run in parallel, as long as there are available runners. Stages run sequentially: all lint jobs must finish before any test job starts.

If a job fails in a stage, by default the pipeline stops and later stages are skipped.

Jobs, Scripts, and Images

Each job runs in its own isolated environment, usually a container created from a Docker image.

Using Docker images

You can define a default image for all jobs:

yaml

image: python:3.11-slim
stages:
  - test
tests:
  stage: test
  script:
    - python --version

Or you can set the image per job:

yaml

stages:
  - test
  - build
python_tests:
  stage: test
  image: python:3.10
  script:
    - pip install -r requirements.txt
    - pytest
build_frontend:
  stage: build
  image: node:20
  script:
    - npm ci
    - npm run build

The script section

The script key is a list of shell commands executed in the job. Example for a Python backend project:

yaml

tests:
  stage: test
  script:
    - pip install -r requirements.txt
    - pytest -q

GitLab automatically runs these commands using the default shell (for example /bin/sh or bash) inside the job's container or runner environment.

Before script and after script

You can define commands that run before or after every job, or for a specific job.

Global:

yaml

before_script:
  - python --version
  - pip install -U pip
after_script:
  - echo "Job finished at $(date)"

Per job:

yaml

tests:
  stage: test
  before_script:
    - pip install -r requirements.txt
  script:
    - pytest

The global before_script can be overridden or extended in the job.

Using Variables

Variables let you avoid repeating hard-coded values and allow secure handling of secrets.

Types of variables

YAML-defined variables in .gitlab-ci.yml:

yaml

   variables:
     PYTHON_VERSION: "3.11"
     PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
   tests:
     image: "python:${PYTHON_VERSION}"
     script:
       - echo $PIP_CACHE_DIR

Project-level or group-level variables configured in GitLab UI under
Settings > CI/CD > Variables.
Predefined variables that GitLab provides automatically, such as:

| Variable | Meaning |
|----------------------|------------------------------------------|
| $CI_COMMIT_BRANCH | Name of the branch for the current job |
| $CI_COMMIT_TAG | Tag name if pipeline is for a tag |
| $CI_PROJECT_PATH | namespace/project |
| $CI_JOB_STAGE | Current job's stage |
| $CI_PIPELINE_SOURCE| Type of pipeline trigger (push, web, etc.) |

Using variables in scripts

Variables are environment variables in the job:

yaml

tests:
  stage: test
  variables:
    ENV: "test"
  script:
    - echo "Environment is $ENV"
    - echo "Branch is $CI_COMMIT_BRANCH"

Protecting secrets

Access secrets through GitLab's CI/CD variables rather than hardcoding them.

For example, you can set POSTGRES_PASSWORD in GitLab UI and use:

yaml

test_db:
  stage: test
  script:
    - echo "$POSTGRES_PASSWORD" | wc -c  # length, do not print secret

Important rule
Never commit secrets into your repository. Always use GitLab CI/CD variables or an external secret manager.

Runners and Execution Environments

A runner is a machine that picks up jobs and executes them. GitLab supports:

Shared runners, provided by GitLab (available on GitLab.com)
Specific runners that you install and register for your projects

Types of runners

Common executor types:

Executor type	How it runs jobs	Typical use case
Docker	Starts a new container per job	Most common for backend projects
Shell	Runs commands directly on the runner machine	Internal networks, custom tools
Kubernetes	Runs jobs as pods in a Kubernetes cluster	Larger organizations, cloud native setups

Choosing an image with Docker executor

When using Docker executor, image is important because it defines the environment. For a Python FastAPI backend:

yaml

image: python:3.11-slim
tests:
  stage: test
  script:
    - pip install -r requirements.txt
    - pytest

For building Docker images inside CI you typically use an image with Docker CLI or use the GitLab Docker-in-Docker (DinD) service, which we will see later.

Conditional Jobs: only, except, rules

You usually do not want to run every job on every branch. GitLab offers only, except, and rules to control when jobs run.

only and except (older style)

Example:

yaml

deploy_prod:
  stage: deploy
  script:
    - ./deploy.sh
  only:
    - main       # only run on main branch
  except:
    - tags       # do not run for tags

You can also use special keywords:

yaml

test_merge_requests:
  stage: test
  script:
    - pytest
  only:
    - merge_requests

rules (newer, more flexible)

rules gives fine-grained control:

yaml

deploy_staging:
  stage: deploy
  script:
    - ./deploy_staging.sh
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
deploy_prod:
  stage: deploy
  script:
    - ./deploy_prod.sh
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual   # require a button click

rules can check various variables:

yaml

test_mr:
  stage: test
  script:
    - pytest
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

Comparison operators in rules use standard equality or inequality. For example:

$CI_COMMIT_BRANCH == "main"
$CI_COMMIT_BRANCH != "main"

Artifacts and Cache

Artifacts and cache both store files from jobs, but they are used in different ways.

Artifacts

Artifacts are meant to be passed to later stages or downloaded from the pipeline UI.

Example: Archive test reports or built packages.

yaml

tests:
  stage: test
  script:
    - pytest --junitxml=report.xml
  artifacts:
    paths:
      - report.xml
    expire_in: 1 week

Example: Use build artifacts in a deploy job.

yaml

build_package:
  stage: build
  script:
    - python -m build
  artifacts:
    paths:
      - dist/
deploy:
  stage: deploy
  needs:
    - build_package
  script:
    - ls dist/
    - ./deploy.sh dist/

Cache

Cache is intended to speed up jobs by reusing files like dependencies.

Example: Cache Python dependencies:

yaml

cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths:
    - .venv/
    - .cache/pip/
tests:
  stage: test
  script:
    - python -m venv .venv
    - source .venv/bin/activate
    - pip install -r requirements.txt
    - pytest

Note:

cache is best-effort; it can be pruned or invalidated.
Artifacts are more reliable for passing build outputs between stages.

A common pattern for dependencies is to use a broader key so it can be reused across branches:

yaml

cache:
  key: "deps"
  paths:
    - .cache/pip/

Example: Simple Python Backend Pipeline

Here is a basic CI setup for a Python backend project with tests and linting.

yaml

image: python:3.11-slim
stages:
  - lint
  - test
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
  key: "pip-cache"
  paths:
    - .cache/pip/
before_script:
  - python --version
  - pip install -U pip
lint:
  stage: lint
  script:
    - pip install flake8
    - flake8 src tests
tests:
  stage: test
  script:
    - pip install -r requirements.txt
    - pytest --maxfail=1 --disable-warnings -q

How this works:

lint and tests use the same Python image.
lint runs first, then tests because of stage order.
The pip cache is shared so the second job installs faster.

Building Docker Images in GitLab CI

Many backend applications are deployed as Docker containers. GitLab CI can build and push images automatically.

Using Docker-in-Docker (DinD)

A common approach is to use Docker-in-Docker with a docker image and a docker:dind service.

yaml

image: docker:25
services:
  - docker:25-dind
variables:
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_DRIVER: overlay2
stages:
  - build
build_image:
  stage: build
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"

Here:

$CI_REGISTRY, $CI_REGISTRY_IMAGE, $CI_REGISTRY_USER, and $CI_REGISTRY_PASSWORD are predefined variables in GitLab when using its built-in container registry.
The image is tagged with the commit short SHA, which is unique per commit.

You could also tag by branch or tag name:

yaml

script:
  - export TAG="${CI_COMMIT_TAG:-$CI_COMMIT_BRANCH}"
  - docker build -t "$CI_REGISTRY_IMAGE:$TAG" .
  - docker push "$CI_REGISTRY_IMAGE:$TAG"

Example: Build and test FastAPI app

yaml

stages:
  - test
  - build
variables:
  DOCKER_TLS_CERTDIR: "/certs"
test_app:
  stage: test
  image: python:3.11-slim
  script:
    - pip install -r requirements.txt
    - pytest
build_image:
  stage: build
  image: docker:25
  services:
    - docker:25-dind
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - docker build -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
  needs:
    - test_app

The needs keyword ensures build_image waits for tests and can start as soon as they pass.

Deployments and Environments

GitLab CI/CD can track deployments to environments such as staging and production.

Defining environments

yaml

deploy_staging:
  stage: deploy
  script:
    - ./deploy_to_staging.sh
  environment:
    name: staging
    url: https://staging.example.com
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
deploy_production:
  stage: deploy
  script:
    - ./deploy_to_production.sh
  environment:
    name: production
    url: https://api.example.com
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual

Notes:

The environment name groups deployments in GitLab's Environments page.
The URL gives you a link from the pipeline to the environment.
Marking production deploys as manual helps avoid accidental deployments.

Example: Docker-based deployment script

Your deploy_to_production.sh might do something like:

bash

#!/bin/bash
set -e
TAG="$CI_COMMIT_SHORT_SHA"
docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
docker pull "$CI_REGISTRY_IMAGE:$TAG"
docker stop myapp || true
docker rm myapp || true
docker run -d --name myapp \
  -p 80:8000 \
  -e ENV=production \
  "$CI_REGISTRY_IMAGE:$TAG"

You would commit this script and call it from the GitLab CI job.

Merge Request Pipelines and Code Quality

CI is especially useful for merge requests, where you want to run tests and checks before merging.

Running pipelines only for merge requests

yaml

test_mr:
  stage: test
  image: python:3.11
  script:
    - pip install -r requirements.txt
    - pytest
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

Enforcing passing pipelines

In GitLab project settings you can require merge requests to have green pipelines before merging, which is a common practice for backend teams.

Example: Complete Backend CI/CD Flow

Here is a more complete example for a FastAPI backend with:

Lint and tests on every branch
Docker image build on main and develop
Automatic deploy to staging from develop
Manual deploy to production from main

yaml

image: python:3.11-slim
stages:
  - lint
  - test
  - build
  - deploy
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  DOCKER_TLS_CERTDIR: "/certs"
cache:
  key: "pip-cache"
  paths:
    - .cache/pip/
before_script:
  - python --version
  - pip install -U pip
lint:
  stage: lint
  script:
    - pip install flake8
    - flake8 app tests
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH'
tests:
  stage: test
  script:
    - pip install -r requirements.txt
    - pytest
  artifacts:
    reports:
      junit: report.xml
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH'
build_image:
  stage: build
  image: docker:25
  services:
    - docker:25-dind
  script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
    - export TAG="${CI_COMMIT_SHORT_SHA}"
    - docker build -t "$CI_REGISTRY_IMAGE:$TAG" .
    - docker push "$CI_REGISTRY_IMAGE:$TAG"
  needs:
    - tests
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
    - if: '$CI_COMMIT_BRANCH == "main"'
deploy_staging:
  stage: deploy
  image: alpine:3
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - apk add --no-cache openssh
    - ./deploy_staging.sh
  rules:
    - if: '$CI_COMMIT_BRANCH == "develop"'
deploy_production:
  stage: deploy
  image: alpine:3
  environment:
    name: production
    url: https://api.example.com
  script:
    - apk add --no-cache openssh
    - ./deploy_production.sh
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual
      allow_failure: false

You would implement deploy_staging.sh and deploy_production.sh to perform SSH, run Docker commands on servers, or trigger any other deployment mechanism.

Practical Tips and Common Pitfalls

Validate .gitlab-ci.yml using the CI Lint tool in GitLab (CI/CD > Pipelines > CI Lint).
Use small, focused jobs instead of a single huge job, this gives better feedback and caching.
Use rules for fine control instead of mixing only and except.
Do not print secrets in job logs.
Keep Docker images small, for example use python:3.11-slim instead of full images.
Cache dependencies but do not cache build outputs that must be reliable, use artifacts for those.

With this knowledge you can set up GitLab CI/CD for your backend projects, automate testing and builds, and connect it to your deployment process.

24.7. GitLab CI/CD

GitLab CI/CD

GitLab CI/CD Concepts

Key building blocks

The .gitlab-ci.yml File

Basic structure

Indentation and syntax

Defining stages

Jobs, Scripts, and Images

Using Docker images

The script section

Before script and after script

Using Variables

Types of variables

Using variables in scripts

Protecting secrets

Runners and Execution Environments

Types of runners

Choosing an image with Docker executor

Conditional Jobs: only, except, rules

only and except (older style)

rules (newer, more flexible)

Artifacts and Cache

Artifacts

Cache

Example: Simple Python Backend Pipeline

Building Docker Images in GitLab CI

Using Docker-in-Docker (DinD)

Example: Build and test FastAPI app

Deployments and Environments

Defining environments

Example: Docker-based deployment script

Merge Request Pipelines and Code Quality

Running pipelines only for merge requests

Enforcing passing pipelines

Example: Complete Backend CI/CD Flow

Practical Tips and Common Pitfalls

Comments

Where to Move