4.3.1 What is the kernel?

Big-picture view: where the kernel fits

The kernel is the core program of an operating system that runs with the highest privilege and mediates all interaction between software and hardware.

User programs (your shells, browsers, databases, containers, etc.) never touch hardware directly. Instead, they:

1. Run in user space (restricted mode).
2. Ask the kernel (in kernel space) to do privileged work through system calls.
3. The kernel:
• Schedules CPU time,
• Manages memory,
• Talks to devices,
• Enforces permissions and isolation.

Visually:

• Hardware (CPU, RAM, disks, NICs, etc.)
• Kernel (Linux)
• System libraries (glibc, musl, etc.)
• User processes (shell, apps, servers, containers)

The Linux kernel is a single, continuously running program that starts very early in the boot process and only stops when the system shuts down or reboots.

Core responsibilities of the Linux kernel

1. Process and thread management

The kernel decides who runs when on the CPU.

Key ideas you should connect with the kernel:

• Processes and threads
  Processes have their own address spaces; threads share a process’s memory. The kernel:
• Creates and destroys them (fork, execve, exit),
• Tracks their states (running, sleeping, zombie, etc.),
• Handles thread creation (e.g. via clone).
• Scheduling
  The scheduler chooses which runnable task gets CPU time, based on policies (e.g. CFS – Completely Fair Scheduler for normal tasks, real-time schedulers for RT tasks). From user land, you notice this via:
• nice and renice,
• chrt for real‑time policies,
• how responsive the system feels under load.
• Context switches
  The kernel saves and restores registers, program counters, and other CPU state when switching between tasks. Context switching is necessary but has overhead, which becomes relevant in performance tuning.

2. Memory management

The kernel manages all physical RAM and presents each process with the illusion of its own large, contiguous memory space.

Main roles:

• Virtual memory
  Every process has a virtual address space, which the kernel maps to:
• physical RAM,
• or swap space on disk,
• or memory‑mapped files.
• Paging
  Memory is divided into fixed-size units called pages (commonly 4 KiB). The kernel:
• Maps/unmaps pages,
• Loads pages from disk on demand (page faults),
• Evicts less-used pages when memory is tight.
• Access control and protection
  The kernel sets per-page permissions (read, write, execute) so:
• A process cannot read/write memory belonging to another process,
• Executable code and writable data can be separated (important for security).
• Kernel memory vs user memory
  Kernel memory is not directly visible to user processes. Interaction happens through system calls and controlled interfaces (e.g. /proc, /sys), not via arbitrary pointers.

3. Device and driver management

The kernel is the layer that knows how to talk to hardware.

• Device drivers
  Each hardware device (disk controllers, network cards, GPUs, USB devices, etc.) needs specific logic. That logic lives as:
• Built-in code inside the kernel image, or
• Separate kernel modules that can be loaded/unloaded (covered in a later chapter).
• Unified device model
  The kernel exposes devices via:
• /dev nodes (e.g. /dev/sda, /dev/ttyUSB0, /dev/null),
• Class and bus abstractions inside the kernel (PCI, USB, etc.),
• Sysfs (/sys) for structured device information and control.
• Character vs block vs network devices
  Kernel drivers implement specific interfaces for:
• Character devices (byte-stream oriented),
• Block devices (random-access blocks, like disks),
• Network interfaces (packets).

From user space, you typically do not talk to drivers directly; you use system calls (read, write, ioctl, send, recv, etc.) and libraries, and the kernel routes those operations to the right driver.

4. Filesystems and storage

The kernel provides the abstraction of files and directories on top of raw storage devices.

• VFS (Virtual Filesystem Switch)
  The kernel includes a generic filesystem layer (VFS) that provides a uniform interface (open, read, write, stat, etc.) irrespective of the underlying filesystem type.
• Multiple filesystem types
  The kernel knows how to handle many filesystem implementations, such as:
• ext4, XFS, Btrfs, F2FS, etc.
• Network filesystems (NFS, CIFS/SMB clients),
• Pseudo-filesystems: procfs (/proc), sysfs (/sys), tmpfs, etc.
• Mounting
  The kernel is responsible for:
• Attaching filesystems to the single global directory tree (mounting),
• Handling unmounting and ensuring data consistency,
• Enforcing permissions for mount operations.

The entire concept of “everything is a file” is realized and enforced by the kernel’s VFS and filesystem drivers.

5. Networking

Linux networking is implemented almost entirely inside the kernel.

• Network stack
  The kernel implements:
• Link layer interaction with NIC drivers,
• IP layer (IPv4, IPv6),
• Transport protocols (TCP, UDP, etc.),
• Routing, NAT, firewall hooks, and packet filtering.
• Sockets
  User programs use the socket API (via system calls) to:
• Create socket descriptors,
• Bind to ports,
• Connect, send, and receive data.
  The kernel then handles all packet transmission, retransmission, congestion control, and delivery.
• Routing and firewalling
  Tools like ip, iptables/nftables, and advanced features (bridging, VLANs, tunneling, namespaces, etc.) all configure kernel networking behavior; they don’t implement networking themselves.

6. Security, isolation, and resource control

The kernel is the final authority on what any program is allowed to do.

• Permissions and ownership
  The kernel enforces file permissions (owner, group, mode bits), capabilities, and system call access. It checks:
• User IDs and group IDs,
• Permission bits and ACLs,
• Whether an operation is permitted for that user.
• Authentication hooks
  While authentication logic is mainly in user space (e.g. PAM), the kernel:
• Maintains user ID and group information at runtime,
• Enforces credential-based access on each system call.
• Security frameworks
  Mechanisms like:
• Capabilities (fine-grained splitting of root’s powers),
• LSM (Linux Security Modules) hooks for systems like SELinux and AppArmor,
• Seccomp (restricting which system calls a process may use),
  are implemented and enforced in the kernel.
• Containers and resource control
  Advanced isolation and resource control features:
• namespaces,
• cgroups,
  are kernel features that underpin containers and resource limits (they’re covered in detail elsewhere, but conceptually they are about separating and limiting what processes can see and use).

The Linux kernel’s design style

Monolithic kernel with modularity

Linux is a monolithic kernel:

• Most core services (scheduling, memory, filesystems, networking, drivers) run inside one large kernel binary in a single address space (kernel space).
• This is different from microkernels where many services run as separate user-space servers.

However, Linux is also modular:

• Many subsystems and drivers are built as loadable kernel modules that can be inserted or removed at runtime.
• This allows:
• Hardware support to be added without rebooting,
• Smaller base kernel images,
• Distribution-specific tailoring.

From a conceptual point of view: it’s monolithic by architecture, modular by implementation.

Kernel vs user space ABI

The kernel provides a relatively stable ABI (Application Binary Interface) to user space:

• System calls (like open, read, write, clone, ioctl, etc.) form the main public interface.
• This interface changes very slowly to avoid breaking existing binaries.
• Internal kernel APIs between kernel subsystems and drivers are not stable and can change between releases—driver code must be kept in sync with kernel versions.

As an admin or developer, you mostly care about:

• User-space ABI stability: programs compiled for a given architecture and ABI generally continue to run across kernel updates.
• Kernel–module API instability: out-of-tree modules (like proprietary drivers) must provide builds specific to kernel versions.

How users and applications interact with the kernel

System calls

System calls (syscalls) are the boundary between user programs and the kernel:

• User code requests the kernel to perform privileged operations by invoking syscalls.
• On x86_64, this is often via the syscall CPU instruction; on other architectures, similar mechanisms exist.
• Examples include:
• Process control: fork, execve, wait4, exit
• File I/O: openat, read, write, close
• Memory: mmap, brk, munmap
• Networking: socket, bind, connect, sendto, recvfrom
• Time: clock_gettime, nanosleep

Most higher-level functions you use in C or other languages (like fopen, printf, etc.) are provided by libraries (e.g. glibc) and internally make system calls when needed.

Pseudo-filesystems: `/proc` and `/sys`

The kernel exposes information and controls to user space via special virtual filesystems:

• /proc
  Shows process and system information as files:
• /proc/<pid>/ directories for each process,
• /proc/meminfo, /proc/cpuinfo, /proc/uptime, etc.
• /sys (sysfs)
  Represents devices, drivers, and kernel attributes hierarchically:
• /sys/class/, /sys/block/, /sys/devices/, etc.
• Many tunables can be adjusted by writing to files in /sys.

When you read or write these files, you are indirectly asking the kernel to provide data or change settings.

ioctls and netlink

Beyond regular read/write:

• ioctl (input/output control)
  A generic system call for device-specific configuration. For example:
• Changing terminal modes,
• Adjusting advanced network interface parameters.
• Netlink sockets
  Used by user-space tools (like ip) to communicate structured messages to networking code in the kernel for routing, firewall, and link management.

Linux kernel as a project and artifact

Versioning and releases

The Linux kernel is developed as an open-source project, mainly at kernel.org.

Version numbers generally follow:

$$
\text{major.minor.patch}
$$

For example:

• 6.8.1 → major 6, minor 8, patch 1.

Distributions often:

• Take an upstream kernel version,
• Apply their own patches and configuration options,
• Ship it as a packaged kernel (e.g. 5.15.0-100-generic in Ubuntu).

Understanding that you are usually running a distribution kernel, not a raw upstream kernel, is important when tracking features and bugs.

Configuration and build-time choices

Before compilation, the kernel is configured:

• Which drivers are built-in vs as modules vs disabled,
• Which subsystems are enabled (filesystems, networking features, namespaces, security frameworks, etc.),
• Which debug options are turned on.

This configuration profoundly shapes what your kernel can do and how it behaves. Compiling and customizing a kernel is handled in a separate chapter, but conceptually:

• The same Linux kernel source tree can produce very different kernels:
• Tiny kernels for embedded devices,
• Feature-rich, general-purpose server kernels,
• Hardened kernels for security-sensitive environments.

Why the kernel matters for advanced work

Even if you never touch kernel code, understanding what the kernel is helps you:

• Interpret system behavior:
• Why a process can be "running" but not using CPU (blocked in the kernel),
• Why I/O waits or memory pressure slow things down.
• Use tools effectively:
• strace, perf, top, ps, vmstat, iostat, etc., all reveal kernel-level states.
• Make good configuration decisions:
• Scheduler tuning, memory overcommit settings, swap strategies,
• Filesystem choices and mount options.
• Understand the limits of user-space:
• Certain behaviors can only be changed via kernel configuration, sysctls, or different kernel builds.

In later chapters on kernel modules, parameters, and custom kernels, you’ll work more directly with specific mechanisms. Here, the key takeaway is:

• The Linux kernel is the privileged, always-running core of the system that implements processes, memory, devices, filesystems, networking, and security.
• Everything you do on a Linux system passes through the kernel at some point.