6.1.3 Debugging tools (gdb, strace)

Why Debugging Tools Matter on Linux

Debugging tools let you see what your program is really doing instead of guessing. For C/C++ (and many compiled languages on Linux), the two most common low‑level tools are:

gdb: an interactive debugger that can pause, inspect, and control your program.
strace: a tracer for system calls and signals (what your program asks the kernel to do).

They solve different problems:

Use gdb when you need to inspect variables, step through code, and understand logic bugs or crashes.
Use strace when you need to see interactions with the OS: files opened, permissions errors, network calls, hangs on I/O, etc.

This chapter shows how to apply both tools in simple, practical ways as a developer on Linux.

Preparing Programs for Debugging

Most compiled languages can add debug information to generated binaries. For C/C++ using gcc:

Compile with debug symbols:
gcc -g -O0 main.c -o main

-g includes debug symbols; -O0 disables optimizations that can make debugging confusing. For production, you typically build with optimizations and optionally -g (debug builds vs release builds).

Many distributions also ship -dbg / -dbgsym packages with debug symbols for system libraries; these help you debug deeper into library code.

gdb: The GNU Debugger

gdb lets you:

Run and pause programs under debugger control.
Set breakpoints (places to stop).
Step line by line.
Inspect variables, memory, and the call stack.
Attach to a running process.
Debug post‑mortem using core dumps.

You’ll most often use gdb for C/C++, but it’s also used indirectly for other languages or mixed-language projects.

Starting gdb

Debugging an executable directly

gdb ./myprogram
(gdb) run arg1 arg2

You can also specify arguments in one go:

gdb --args ./myprogram arg1 arg2
(gdb) run

Attaching to a running process

First find the PID (e.g., with ps, pgrep, or top), then:

gdb -p <PID>

Examples:

gdb -p 12345

This is useful if your program is hung or misbehaving and you want to inspect it without restarting.

On some systems, attaching to other users’ processes is restricted by ptrace settings. You may need sudo or to adjust /proc/sys/kernel/yama/ptrace_scope (system admin concern).

Basic gdb Workflow

Inside gdb, you type commands at the (gdb) prompt. Key commands:

run (or r): start the program.
continue (or c): resume execution after a breakpoint.
break (or b): set a breakpoint.
next (or n): step to the next line (skip over function calls).
step (or s): step into function calls.
finish: run until the current function returns.
print (or p): show value of an expression/variable.
backtrace (or bt): show the call stack.
quit (or q): exit gdb.

Example: Debugging a Simple Crash

Suppose you have:

#include <stdio.h>
int main() {
    int *p = NULL;
    *p = 42;  // crash
    printf("Value: %d\n", *p);
    return 0;
}

Compile with debug info:

gcc -g -O0 crash.c -o crash

Run in gdb:

gdb ./crash
(gdb) run

The program will crash (SIGSEGV). Then:

bt — show where it crashed and the call stack.
list — show the source around the current line.
p p — inspect the pointer value.

Example session:

(gdb) run
Starting program: /path/crash
Program received signal SIGSEGV, Segmentation fault.
0x0000555555555156 in main () at crash.c:5
5       *p = 42;  // crash
(gdb) print p
$1 = (int *) 0x0
(gdb) bt
#0  0x0000555555555156 in main () at crash.c:5

From this, you clearly see you dereferenced a null pointer.

Breakpoints and Stepping

Breakpoints let you stop execution at interesting points.

Setting breakpoints

Common ways:

By function name:

  (gdb) break main
  (gdb) break my_function

By file and line:

  (gdb) break myfile.c:42

By current line (when stopped):

  (gdb) break

Managing breakpoints:

info break — list breakpoints.
delete 1 — remove breakpoint #1.
disable 1 / enable 1 — temporarily turn a breakpoint off/on.

Stepping through code

When you hit a breakpoint:

next — execute the current line; step over function calls.
step — step into function calls.
finish — run until the current function returns.
continue — run freely until the next breakpoint or program exit.

Inspecting Variables and the Stack

When stopped at a breakpoint (or crash):

print var — print a variable.
print *ptr — dereference a pointer.
print var[3] — access array element.
print mystruct.field — structure field.
set var x = 10 — change variable value.

You can use C expressions:

(gdb) print a + b * 2

For the call stack:

backtrace (or bt) — show the stack frames.
frame 0 / frame 1 — select a specific frame.
info locals — list local variables in the current frame.
info args — show function arguments.

This is extremely helpful for understanding how you reached a certain state.

Debugging Core Dumps (Post‑Mortem)

If your program crashes, the kernel can produce a core dump file, which contains the memory image at crash time.

Enable core dumps (per shell):

   ulimit -c unlimited

Run your program until it crashes. A core (or core.<pid>) file will appear according to kernel.core_pattern.
Open the core file with gdb:

   gdb ./myprogram core

Then use bt, frame, info locals, etc. to inspect the state at crash time.

This is often used on production or test systems where you can’t reproduce the problem interactively.

Using gdb with Other Build Systems

With make, you’ll often have make debug targets that compile with -g -O0.
With CMake, -DCMAKE_BUILD_TYPE=Debug typically adds -g.
Many IDEs (e.g., VS Code, CLion, Eclipse) wrap gdb and give you a graphical front-end; understanding the raw gdb commands helps troubleshoot when the IDE hides details.

strace: Tracing System Calls

strace shows every system call a process makes and the signals it receives. While gdb shows what your code is doing, strace shows what the process is asking the kernel to do.

Common uses:

Find why a program fails to open a file (ENOENT, EACCES, etc.).
See what configuration files a program is reading.
Diagnose hangs (stuck on I/O, waiting on a lock, DNS, etc.).
Understand process startup behavior.
Trace child processes.

Running a Program Under strace

Basic usage:

strace ./myprogram arg1 arg2

You’ll see a stream like:

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
mmap(NULL, 12345, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f...
close(3)                                = 0
write(1, "Hello\n", 6)                  = 6
exit_group(0)                           = 0

Key things:

First part is the syscall, e.g. openat, mmap, write.
Then arguments.
Then = <return value>.

Error codes are shown using errno names:

openat(AT_FDCWD, "/does/not/exist", O_RDONLY) = -1 ENOENT (No such file or directory)

Attaching to a Running Process

Just like gdb, you can attach by PID:

strace -p <PID>

Example:

strace -p 12345

To see system calls from child processes too:

strace -f -p 12345

Press Ctrl+C in the strace terminal to detach/stop tracing.

Making strace Output Manageable

strace can be very verbose. Useful options:

-o file: write output to a file instead of the terminal.

  strace -o trace.log ./myprogram

-f: follow forks and threads (child processes/threads).

  strace -f ./server

-e trace=: filter which syscalls to display:

File I/O:

    strace -e trace=file ./myprogram

Network:

    strace -e trace=network ./client

Specific calls:

    strace -e trace=open,close,read,write ./myprogram

-e trace=!foo,bar: show everything except named calls.
-T: show time spent in each syscall.
-tt: show timestamps with microseconds.
-s N: increase the maximum string length printed (default is small):

  strace -s 200 -e trace=read,write ./myprogram

You can combine options, e.g.:

strace -f -tt -o trace.log ./myprogram

Common strace Debugging Scenarios

“File not found” or “Permission denied”

If your program prints a vague error like “could not load configuration”:

strace -e trace=file ./myprogram

Look for lines like:

openat(AT_FDCWD, "/home/user/.myapprc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/myapp.conf", O_RDONLY) = -1 EACCES (Permission denied)

You now know which path is failing and why (ENOENT, EACCES, etc.).

Program hangs

If a program is “frozen”:

Get its PID (e.g., pidof myprogram).
Attach:

   strace -p <PID>

See which syscall it’s blocked on.

For example:

read(3,  <unfinished ...>

connect(3, {sa_family=AF_INET, sin_port=htons(80), ...}, 16 <unfinished ...>

This tells you it’s waiting on reading input or attempting a network connection.

strace vs gdb: When to Use Which

Use gdb when:

You need to inspect variable values, control flow, and logic.
You have crashes (segfaults, etc.) and want stack traces.
You want to step through specific lines and functions.

Use strace when:

You suspect file, network, or permission issues.
The program hangs and you want to know what syscall it’s stuck on.
You’re debugging startup behavior, configuration discovery, or external dependencies.
You don’t have debug symbols or source code (e.g., third‑party binaries).

They also complement each other: you might use strace to locate the broad area of trouble (e.g., failing file open), then gdb to inspect why the code is trying that path or using bad arguments.

Practical Tips and Patterns

Always build debug versions of your code (-g, no heavy optimization) when actively developing or investigating difficult bugs.
Keep a gdb cheat sheet nearby; the basic commands (run, break, next, step, print, bt) cover most day‑to‑day needs.
Use strace -f -o trace.log around problematic runs; you can later search trace.log for ENOENT, EACCES, etc.
For intermittent bugs in longer‑running services, consider:

Enabling core dumps and analyzing them with gdb.
Attaching gdb or strace to a live process during or just before the problem.

With these two tools in your daily workflow, you can move beyond “add more print statements” and systematically understand what your programs are doing on Linux.

Comments

Please login to add a comment.

Don't have an account? Register now!