Automated testing

Why Automated Testing Matters in DevOps

Automated testing is essential in DevOps because:

Code is deployed frequently (often multiple times per day).
Manual testing cannot keep up with rapid changes.
Reliable feedback must be fast, repeatable, and consistent.
Failures must be detected before code reaches production.

In a DevOps workflow, automated tests are typically integrated into:

Local development workflows (pre-commit hooks, local test runs).
CI pipelines (GitHub Actions, GitLab CI, etc.).
CD pipelines (smoke tests, health checks after deployment).

The key idea: treat tests as code, keep them in version control, run them automatically on every relevant change.

Types of Automated Tests in a DevOps Context

You will encounter many test types; not all are used in every project. Common categories:

Unit tests

Test individual functions/classes in isolation.
Run very fast.
Usually language/framework specific (e.g., pytest, unittest, go test, npm test).
Main goal: catch logic errors close to the code.

In CI, unit tests are usually the first and most frequent tests to run.

Integration tests

Test how components work together (e.g., your app + database).
Often require external services or containers.
Slower than unit tests, but closer to real usage.

Typical DevOps pattern: spin up dependencies with Docker Compose or containers in CI, then run integration tests against them.

End-to-end (E2E) tests

Test full user flows through the system (e.g., browser clicking through a web app).
Often use tools like Selenium, Cypress, Playwright.
Slowest and most fragile; run less frequently (e.g., nightly, pre-release).

API tests

Specialized integration/E2E tests for HTTP APIs.
Verify status codes, response bodies, headers, auth flows.
Tools: curl, httpie, Postman collections, newman, language test frameworks.

Performance and load tests

Measure response times, throughput, resource usage.
Tools: ab, wrk, JMeter, k6.
Often run on dedicated environments (staging or perf environment).

Security and quality checks

Not tests in the “unit test” sense, but still automated checks that run in pipelines:

Static Application Security Testing (SAST): scan code for vulnerabilities.
Dependency scanning: check libraries for known CVEs.
Linters and formatters: flake8, eslint, gofmt, etc.
Container image scanning: Trivy, Clair, etc.

What “Automated” Means on Linux

On Linux, automating tests revolves around:

Command-line tools and scripts.
Return/exit codes ($0$ = success, non-zero = failure).
Standard input/output streams.

Any test command, no matter how sophisticated, boils down to:

You run a command: run-tests.sh.
The command prints information to stdout/stderr.
The command exits with an exit code:

$0$ → all tests passed.
Non-zero → at least one test failed.

CI/CD systems use this exit code to decide whether a job or pipeline passes or fails.

Common Testing Tools on Linux

You will use different test tools depending on language and stack. A non-exhaustive overview:

General Linux tools

These are not “test frameworks”, but often used in automated tests:

bash / sh scripts for simple checks.
curl / wget for HTTP/API checks.
diff, cmp for output comparison.
grep, awk, jq to validate outputs.
timeout to fail tests that hang.

Using these together lets you build lightweight tests even when no full framework exists.

Language-specific test frameworks

Examples you’ll frequently see on Linux servers:

Python: pytest, unittest.
JavaScript/TypeScript: jest, mocha, vitest, cypress (for E2E).
Java: JUnit, TestNG.
Go: built-in go test.
Ruby: rspec, minitest.
PHP: phpunit, pest.

DevOps tasks often involve running these via CI configs, not writing them in detail.

Infrastructure and configuration tests

In DevOps, you also test infrastructure and configuration:

Ansible: ansible-lint, molecule.
Terraform: terraform validate, tflint, terratest.
Docker: docker build --pull --no-cache, image tests with docker run + checks.

These tests help ensure that infrastructure-as-code and config changes are safe to apply.

Structuring Tests in a Repository

To integrate with CI and team workflows, you typically:

Keep tests in the same repository as the code.
Use consistent directory names, for example:

tests/ at the project root.
Language defaults, like src/ + tests/, or pkg/ + internal/ + cmd/ (for Go).

Use naming conventions the framework understands:

Python: test_*.py.
Go: *_test.go.
Many frameworks: files ending with .spec. or .test..

A common layout for a web service project:

my-service/
  src/
  tests/
    unit/
    integration/
  docker/
  .github/workflows/

This separation lets you run “fast tests” and “slow tests” differently in CI.

Running Tests Locally on Linux

To keep feedback fast, developers run tests before pushing. Common patterns:

Use simple commands:

pytest tests/unit
npm test
go test ./...

Provide helper scripts:

./scripts/test-unit.sh
./scripts/test-all.sh

Use make targets:

  make test        # run all tests
  make test-unit   # run unit tests
  make lint        # linting and formatting checks

Pre-commit hooks

Pre-commit hooks run locally before each commit. On Linux:

Install pre-commit or write your own .git/hooks/pre-commit.
Run lightweight checks:

Linters.
Formatters.
A small subset of tests.

Automated pre-commit checks improve quality before code even reaches CI.

Integrating Tests in CI Pipelines (Linux Focus)

In a DevOps environment using Linux build agents/runners, integration typically follows this pattern:

Checkout code.
Set up environment:

Install dependencies.
Pull Docker images if needed.

Run tests using CLI tools.
Collect and publish results:

JUnit XML, coverage reports, artifacts.

Fail the job if any test command returns non-zero.

Basic example in a shell-based CI job

You often see scripts like:

#!/usr/bin/env bash
set -euo pipefail
echo "Running unit tests..."
pytest tests/unit
echo "Running integration tests..."
docker compose up -d
pytest tests/integration
docker compose down

set -e ensures the script stops at the first failing command, making CI fail accordingly.

Parallelizing tests

Linux-based CI runners can run multiple test jobs in parallel:

Separate jobs by type:

lint, unit, integration, e2e.

Or by test shards:

TEST_PART=1, TEST_PART=2, etc.

Parallelization keeps pipelines fast even as test suites grow.

Testing Services with Docker in CI

Containers are common in DevOps, and automated tests often use them:

Build your service’s image.
Spin it up along with dependencies (DB, cache).
Run tests against the running stack.

A simple pattern:

docker compose up -d
Wait for services to be ready (health checks, curl loop).
Run tests from a separate container or the CI runner.
docker compose down after tests.

On Linux hosts this integrates naturally with system tools like systemd (for Docker daemon) and the filesystem (for mounting volumes, logs).

Test Artifacts and Reports

CI environments on Linux can store and analyze test outputs:

JUnit XML reports: many frameworks can output these for CI to parse.
Coverage reports: coverage.py, nyc, go test -cover, etc.
Logs and screenshots (especially for E2E and UI tests).

Why this matters:

Historical view of test failures.
Ability to see which tests are flaky.
Easy navigation from CI UI to failing tests.

From a Linux perspective, these are usually just files under ./reports or ./artifacts directories that CI uploads.

Testing Infrastructure and Deployments

Automated testing in DevOps is not limited to application code.

Smoke tests after deployment

After a deployment to a Linux server or Kubernetes cluster:

Run simple scripts that check:

Main HTTP endpoint returns $2xx$.
Health check endpoints are OK.
Basic workflow (e.g., login) works.

These are often lightweight shell scripts using curl and exit codes.

Example smoke test:

#!/usr/bin/env bash
set -euo pipefail
URL="https://my-service.example.com/health"
for i in {1..10}; do
  if curl -fsS "$URL" >/dev/null; then
    echo "Health check passed"
    exit 0
  fi
  echo "Waiting for service..."
  sleep 5
done
echo "Service did not become healthy"
exit 1

Configuration and policy tests

You might automatically test:

That required Linux services are enabled/active (e.g., using systemctl is-active).
That firewall rules match expectations.
That certain ports are not open.

These can be encoded as automated checks in scripts or with specialized tools (like InSpec or Goss).

Best Practices for Automated Testing in DevOps on Linux

Fail fast, run fast tests early:

Run unit tests and linters first to quickly reject bad changes.

Keep tests reliable:

Avoid brittle timeouts and environment assumptions.
Use containers or standardized environments to reduce “works on my machine” issues.

Make tests easy to run locally:

Same commands locally and in CI.
Document make test or ./scripts/test.sh.

Separate tests by speed and purpose:

Fast vs slow; app vs infra; smoke vs exhaustive.

Use exit codes consistently:

Ensure test scripts return non-zero when anything fails.

Log and observe:

Save logs from failing tests.
Add enough output to debug without re-running repeatedly.

Example: A Simple End-to-End Flow

Putting it together, a typical DevOps automated testing flow on Linux might look like:

Developer edits code on a Linux workstation.
Runs make test locally:

Linters.
Unit tests.

Pushes changes.
CI pipeline on Linux runner:

Job lint → runs static analysis.
Job unit → runs unit tests.
Job integration → spins up Docker services, runs integration tests.

On merge to main branch:

Additional e2e and performance checks may run.

On deployment:

Smoke tests run against the deployed Linux servers or containers.
If smoke tests fail, deployment is rolled back automatically.

This continuous, automated testing loop is what enables safe, frequent changes in a DevOps environment.

Comments

Please login to add a comment.

Don't have an account? Register now!