Table of Contents
Understanding Why Debugging Shell Scripts Is Different
Debugging shell scripts is trickier than debugging many other languages because:
- The shell silently ignores many errors unless you ask it not to.
- Many operations are external commands, not built‑ins.
- Exit codes and pipelines have special rules.
- Subshells and redirections can hide what’s really happening.
The goal of debugging is to see what the script is actually doing (commands, values, and control flow) and stop or highlight mistakes early, instead of chasing mysterious side effects.
This chapter focuses on techniques and tools for debugging, not on writing general scripts or control structures (covered elsewhere).
Enabling Shell Debug Options
The main built‑in debugging aids in bash/sh are controlled by set options and - flags when invoking the shell.
Core Options: `-e`, `-u`, `-x`, `-o pipefail`
You can turn these on in three main ways:
- In the shebang:
#!/usr/bin/env bash
set -euo pipefail- At the command line:
bash -euxo pipefail your_script.sh- Temporarily in an interactive shell:
set -x # later: set +xKey options:
`-e` / `set -e` — Exit on Error
- Script exits if a simple command returns a non‑zero status.
- Helps you catch failures early, instead of continuing with bad state.
But there are exceptions where -e is ignored (by POSIX definition):
- In
if,while,untiltests:
if some_command; then # 'some_command' failure does not exit the script
echo "ok"
fi- In
||and&&lists:
cmd1 || echo "cmd1 failed" # 'cmd1' failure will not exit with -e- In some subshell contexts.
Therefore, -e is helpful but not sufficient as a complete error‑handling strategy.
`-u` / `set -u` — Treat Unset Variables as Errors
- Expanding an unset variable causes the script to exit (with
-eor non‑zero $?), unless${var-...}or${var:-...}is used.
Example:
#!/usr/bin/env bash
set -u
echo "User is: $USER"
echo "Foo: $FOO" # exits: unbound variable FOOThis helps catch typos and forgotten variable initializations.
`-x` / `set -x` — Trace Commands
- Prints each command to stderr just before it is executed, with variables expanded.
- Very useful to see execution flow and argument values.
Example:
#!/usr/bin/env bash
set -x
name="Alice"
echo "Hello, $name"Output looks like:
+ name=Alice
+ echo 'Hello, Alice'
Hello, Alice
You almost never want set -x in production scripts by default; enable it selectively (see below).
`-o pipefail` — Catch Failures in Pipelines
By default, in bash, the exit status of a pipeline is the status of the last command only. With pipefail:
- The pipeline’s exit code is the first non‑zero exit code in the pipeline.
- This makes failing commands in the middle of pipelines visible.
Example without pipefail:
false | true
echo $? # prints 0 (success), error hidden
With pipefail:
set -o pipefail
false | true
echo $? # prints 1 (failure)Running Scripts in Debug Mode
Rather than editing the script to add debugging options, you can run it in a debug shell:
bash -x your_script.sh # trace commands
bash -xe your_script.sh # trace commands + exit on error
bash -xv your_script.sh # also show input lines as readMeaning of flags:
-x— print commands and arguments as they are executed.-v— print each line of the script as it is read (before expansion).- Combine them to compare source vs executed commands.
This is useful when:
- You don’t want to modify the original script.
- You suspect issues before variable expansion (
-v).
Selective Debugging with `set -x` / `set +x`
Tracing the entire script can be noisy. Instead, enable it around the suspicious parts:
#!/usr/bin/env bash
echo "Starting backup"
# Enable tracing only for this block
set -x
tar czf backup.tar.gz /important/data
scp backup.tar.gz backup@server:/backups
set +x
echo "Backup complete"This gives you visibility into just the backup portion.
You can also wrap a function:
debug_on() { set -x; }
debug_off() { set +x; }
some_func() {
debug_on
# function body...
debug_off
}Inspecting Exit Codes and Errors
Exit codes are central to debugging shell scripts.
Checking `$?`
After running a command, $? holds its exit status:
cmd
status=$?
echo "cmd exited with $status"Use this when:
- You temporarily turn off
-eto inspect a failure. - You need to distinguish between specific error codes.
Example:
#!/usr/bin/env bash
set -e
if ! some_command; then
echo "some_command failed with $?"
fiUsing `trap` for Error Diagnosis
trap lets you run code on signals or special pseudo‑signals like ERR and EXIT.
Trap on `ERR`
With bash, you can print debugging info whenever a command fails:
#!/usr/bin/env bash
set -Eeu -o pipefail # -E makes ERR propagate in functions/subshells
trap 'echo "Error on line $LINENO: command exited with status $?"' ERR
dangerous_command$LINENO— current line number in the script.$?— exit status of the failing command.
This gives you line‑level error messages without manually checking every command.
Trap on `EXIT`
Run cleanup or a summary when the script ends:
trap 'echo "Script finished with exit code $?"; cleanup_temp_files' EXIT
This runs whether the script exits normally or due to set -e or exit.
Adding Debug Logging
Printing internal state at key points is often the most effective debugging strategy.
Use a Dedicated Debug Function
Instead of scattering echo everywhere, define a debug helper:
#!/usr/bin/env bash
DEBUG=${DEBUG:-0} # set to 1 to enable
debug() {
if [[ "$DEBUG" -eq 1 ]]; then
# >&2 sends to stderr so logs don’t mix with normal output
printf '[DEBUG] %s\n' "$*" >&2
fi
}
debug "Script started"
name="Alice"
debug "Name is: $name"Then run:
DEBUG=1 ./your_script.shThis lets you:
- Toggle debug logs without editing the script.
- Send debug logs to stderr, preserving stdout for real output.
Including Context: Time, Script Name, Line
Improve debug to include more context:
debug() {
[[ "${DEBUG:-0}" -eq 1 ]] || return 0
local ts
ts=$(date '+%Y-%m-%d %H:%M:%S')
printf '%s [%s:%d] %s\n' "$ts" "${BASH_SOURCE[1]}" "${BASH_LINENO[0]}" "$*" >&2
}Here:
${BASH_SOURCE[1]}— file that calleddebug.${BASH_LINENO[0]}— line number in the caller.
This greatly helps when debugging larger scripts.
Debugging Functions, Subshells, and Pipelines
Advanced scripts often use functions, subshells, and pipelines in ways that make debugging non‑obvious.
Functions and Error Propagation
With set -e, behavior in functions can surprise you. Example:
set -e
foo() {
echo "Doing foo"
false # this will cause script exit if not in conditional or || / &&
echo "This will not run"
}
foo
echo "unreachable"While debugging, you may temporarily wrap the failing part in a conditional to inspect status:
foo() {
echo "Doing foo"
if ! false; then
echo "false failed with $?"
fi
echo "This will run"
}
Or temporarily disable -e around suspicious areas:
set +e
some_command
status=$?
set -e
echo "some_command exited with $status"Subshells: `(...)` vs `{ ...; }`
Subshell ( ... ):
- Runs in a separate process.
- Variables modified inside do not affect the parent shell.
- Exit code of the subshell is the exit code of the last command inside.
This can hide changes you expect to persist. Example:
count=0
(
count=5
)
echo "$count" # still 0When debugging unexpected variable values, search for:
- Subshells:
(...),$(...), pipelines. whileloops usingreadwith a pipe; those run in a subshell in many shells.
Prefer redirection to avoid subshell loops when you need state:
# Bad for state: read loop in a pipe (subshell)
count=0
grep pattern file | while read -r line; do
((count++))
done
echo "$count" # often 0 (subshell)
# Better: redirection, same shell
count=0
while read -r line; do
((count++))
done < <(grep pattern file)
echo "$count" # correctWhen debugging, mentally track which shell instance variables live in.
Using `bash -n` for Syntax Checking
To quickly check for syntax errors without executing:
bash -n your_script.sh-n— “noexec”: read and parse, but do not run.
This catches issues such as:
- Missing
fi,done,}. - Unmatched quotes.
- Some syntax typos.
It does not detect logic errors or run‑time failures.
Integrate it into your workflow:
- Run
bash -n script.shafter editing complex conditionals or loops. - Add it to a pre‑commit hook in version control.
Using `shellcheck` for Static Analysis
shellcheck is a widely used static analysis tool for shell scripts.
Running ShellCheck
Install it from your distribution or from https://www.shellcheck.net.
Then run:
shellcheck your_script.shIt reports:
- Common pitfalls (e.g.,
for f in $(ls)). - Quoting issues.
- Undefined variables.
- Bad glob patterns.
- Non‑portable constructs.
Example output (simplified):
In script.sh line 5:
for f in $(ls *.txt); do
^-- SC2045: Iterating over ls output is fragile. Use globs.
Each warning has an ID (SC2045) you can look up with detailed explanations and fixes.
Suppressing or Annotating Warnings
Sometimes you know a warning is acceptable (e.g., intentional portability break). You can disable a rule locally:
# shellcheck disable=SC2045
for f in $(ls *.txt); do
...
doneOr disable for a block:
# shellcheck disable=SC2044
find . -name '*.txt' -print0 | while IFS= read -r -d '' file; do
...
done
# shellcheck enable=SC2044Use this sparingly; usually, it’s better to fix the underlying issue.
Debugging Input/Output and Quoting Issues
Many shell bugs are due to quoting or word splitting.
See Exact Arguments with `printf '%q'`
When you’re not sure what a variable expands to:
printf 'ARG: "%s"\n' "$var"To see shell‑escaped form (how the shell would write it):
printf 'DEBUG var=%q\n' "$var"This reveals spaces, newlines, and special characters.
Temporarily Replace Destructive Commands
To debug scripts that modify or delete files, temporarily replace dangerous commands:
# Replace rm with echo
rm() {
command printf 'rm would delete: %q\n' "$@"
}Or change calls in the suspect block:
# For debugging only
# rm "$file"
echo "Would remove: $file"This lets you inspect what would happen without risk.
Debugging with `PS4` and Enhanced Tracing
The set -x tracing format can be customized using PS4.
Custom `PS4` Prompt
By default, PS4 is + . You can include timestamp, line number, and function name:
#!/usr/bin/env bash
PS4='+ ${BASH_SOURCE}:${LINENO}:${FUNCNAME[0]}: '
set -x
some_function() {
echo "Inside function"
}
some_functionExample trace line:
+ script.sh:5:some_function: echo 'Inside function'
This makes set -x output far more useful, especially in larger scripts.
Debugging with External Tools
While most debugging is done with the shell itself, a few simple external tools can help.
`strace` for System Call Tracing
strace traces system calls made by your script and its child processes.
Useful to debug:
- Permission issues.
- “File not found” despite being sure it exists.
- Network calls failing.
Example:
strace -f -o trace.log ./your_script.sh-f— follow child processes.-o trace.log— write output totrace.logfor later inspection.
Then search for ENOENT, EACCES, or failed open() calls in the log.
`set -o xtrace` with `tee`
Sometimes you want to save the set -x output:
bash -x your_script.sh 2>trace.logOr view it live and save:
bash -x your_script.sh 2> >(tee trace.log >&2)This can be handy during longer or intermittent runs.
Systematic Debugging Strategy for Shell Scripts
A practical approach when a script misbehaves:
- Verify syntax:
bash -n script.sh- Run ShellCheck and fix obvious issues:
shellcheck script.sh- Narrow down the problem area:
- Insert
debuglogs at high‑level steps. - Use
set -xonly around suspect blocks. - Enable strict mode (if not already):
set -Eeuo pipefail
trap 'echo "Error at ${BASH_SOURCE[0]}:${LINENO}: exit $?"' ERR- Inspect inputs and critical variables:
- Print arguments and environment variables near problem code.
- Use
printf '%q\n'when dealing with paths and user input. - Watch for subshells and pipelines:
- Check whether loops or assignments happen in a subshell.
- Check external commands:
- Examine their exit codes.
- Run them manually with the same arguments.
- Use
straceor logging for tricky I/O or permission issues.
By combining these techniques, you can reliably identify where and why shell scripts fail, even as they become more complex.