23.5 HPC Networking (InfiniBand, RDMA)

Introduction

High Performance Computing, often shortened to HPC, focuses on solving very large or very complex problems by using many computers that work together. Typical examples are weather prediction, scientific simulations, machine learning training at large scale, and financial risk modeling. In these environments, the network is not just a way to reach the internet. It is a critical part of the overall performance and must move data between machines extremely fast with very little waiting time.

HPC networking is therefore quite different from the normal enterprise networks discussed in other chapters. It uses special hardware, special protocols, and special techniques that prioritize low latency, high bandwidth, and very efficient communication between thousands of servers. Two of the most important ideas in this world are InfiniBand and RDMA. They are used in many supercomputers and large clusters, and also appear inside modern data centers that run large scale AI workloads.

This chapter focuses on what is specific about HPC networking and how InfiniBand and RDMA enable very fast communication, without repeating general routing or switching concepts that belong to other parts of the course.

Goals of HPC Networking

In a normal office network, if a file transfer takes a few seconds longer, no one really notices. In an HPC cluster, small delays can multiply across thousands of nodes and millions of messages. The main goals of HPC networking are:

Very low latency. The time it takes for a small message to travel from one node to another must be extremely small, often measured in microseconds.

Very high throughput. The network must move huge amounts of data, often many gigabytes per second per node, and aggregate bandwidth across the cluster can reach terabits per second.

Scalability. The same technology should work when connecting tens of nodes or tens of thousands of nodes. It must handle large topologies while trying to preserve low latency.

Efficient CPU usage. The network should move data without forcing the main CPUs to spend much time on copying or processing packets. This is essential for performance and power efficiency.

Predictable performance. For scientific and engineering workloads, it is important that communication patterns and timings are predictable, so simulations behave consistently.

InfiniBand and RDMA-oriented Ethernet technologies are built specifically to meet these goals.

Overview of InfiniBand

InfiniBand is a high speed, low latency interconnect technology designed originally for servers and storage systems. Instead of using normal Ethernet frames and IP routing, InfiniBand defines its own physical links, frames, and fabric behaviors that are all optimized for HPC workloads.

From a user perspective, an InfiniBand network looks like a fabric that connects many compute nodes. Applications that use special communication libraries, such as MPI (Message Passing Interface), can send messages directly over this fabric with very low overhead. While IP can be layered on top of InfiniBand if needed, the main power of InfiniBand comes from its native model of communication between hosts.

InfiniBand networks use specialized adapters, called Host Channel Adapters or HCAs, in the servers. They connect to InfiniBand switches that form the fabric. This is conceptually similar to Ethernet NICs and switches, but InfiniBand introduces its own addressing, flow control, and reliability mechanisms that are tailored for HPC.

InfiniBand Fabric Characteristics

An InfiniBand fabric is the entire set of InfiniBand devices and links that can communicate with each other. Within this fabric, nodes exchange messages in a way that is quite different from normal IP routing.

InfiniBand uses the idea of channels and queue pairs instead of simple sockets. Applications post send and receive operations to queues that are managed mostly by the hardware. The HCA on each node then moves data across the fabric by interacting directly with memory regions that the application has prepared.

Fabric design in InfiniBand aims to reduce the number of hops between nodes and to keep paths symmetric and predictable. Common topologies include fat-tree networks, which try to provide the same bandwidth at all levels, and more recent layouts such as dragonfly that try to reduce latency even for large clusters.

Congestion control and flow control are handled within the fabric so that packet loss is minimized. Reliable delivery is often managed by the hardware, so the CPUs on the nodes do not need to constantly resend or reorder messages.

InfiniBand Performance Aspects

InfiniBand links are defined in terms of speeds and widths. The speed per lane has increased over time, and multiple lanes can be combined to form a wider link. For example, an InfiniBand link might use 4 lanes, and each lane might support a certain gigabits per second rate, so the total bandwidth is the sum of all lanes in one direction.

Although exact numbers change with each generation, the general pattern is that InfiniBand aims to stay ahead of or at least competitive with the latest Ethernet speeds, while also providing much lower latency and lower CPU overhead.

In HPC environments, latency is often just as important as raw bandwidth. InfiniBand networks can reach latencies in the microsecond range for small messages. This is achieved by offloading most of the protocol work into the HCA hardware and by using very efficient switches that are optimized for cut-through forwarding and simple, fast decision logic.

Power efficiency is another key aspect. Because much of the communication work, including reliability and flow control, is handled in hardware, CPUs can spend a larger fraction of time on actual computation, not on network processing.

What is RDMA?

Remote Direct Memory Access, or RDMA, is a method that allows one computer to read from or write to the memory of another computer directly, with very little involvement from the CPUs on either side. This is fundamentally different from normal networking, where data is copied many times between application buffers and kernel buffers, and is processed by the operating system stack at each step.

With RDMA, an application registers certain regions of memory to be accessible for RDMA operations. Once this is done, the network adapter can move data directly between these memory regions on different nodes. The CPU sets up the transfer but does not need to copy the data byte by byte.

RDMA allows networks to achieve both low latency and low CPU overhead. It is used heavily in InfiniBand, but RDMA ideas have also been brought into Ethernet environments.

In HPC contexts, RDMA is often used by communication libraries such as MPI, and also by storage technologies that provide very fast access to remote disks or memory.

RDMA Operations and Queue Pairs

RDMA communication is based on the concept of queue pairs. A queue pair is made of a send queue and a receive queue. Applications post work requests to these queues. The RDMA hardware then processes these requests and updates completion queues when operations are finished.

There are several basic types of RDMA operations. RDMA write lets a node write data directly into a memory region on a remote node. RDMA read lets a node read data from remote memory. There are also send and receive operations where the remote side must have posted a receive buffer. Each of these operations uses memory keys or tokens that the remote side shares during connection setup, which control which memory regions are accessible.

Because applications can post many operations at once, and the hardware can process them in the background, RDMA can keep the network busy while CPUs continue to compute. This is called overlapping communication with computation and is very important for efficient HPC applications.

InfiniBand and RDMA Together

InfiniBand was designed from the start with RDMA in mind. The InfiniBand HCA is essentially a specialized RDMA engine that is well integrated with the InfiniBand fabric.

In an InfiniBand environment, RDMA operations are carried across the fabric using InfiniBand verbs, which are the low-level operations exposed by the InfiniBand programming interface. The verbs API lets applications register memory, create queue pairs, and post RDMA work requests. While most users interact with higher-level libraries such as MPI or SHMEM, underneath these libraries rely on InfiniBand verbs to perform efficient data transfers.

Because InfiniBand carries RDMA operations natively, it avoids many of the inefficiencies that appear when layering RDMA over a protocol stack that was not originally designed for it. This is one reason why InfiniBand remains a top choice in supercomputers that have extremely strict performance requirements.

RDMA over Converged Ethernet (RoCE)

Although InfiniBand is very powerful, many data centers and enterprise environments prefer Ethernet for compatibility reasons. To bring RDMA benefits into Ethernet networks, RDMA over Converged Ethernet, known as RoCE, was introduced.

RoCE allows RDMA operations to run over Ethernet links rather than pure InfiniBand links. There are different versions of RoCE, but the general idea is that the Ethernet fabric is configured to behave in a way that supports lossless or nearly lossless transport, because RDMA hardware expects very low packet loss.

To achieve this, RoCE environments often use techniques like priority based flow control. Some classes of traffic are given special treatment by switches to avoid drops. As a result, the Ethernet fabric becomes more suitable for RDMA and can provide performance that approaches InfiniBand for certain workloads.

RoCE is popular in environments where operators want to use standard Ethernet switches but still want RDMA acceleration for storage or for distributed workloads like large machine learning training runs.

RDMA and CPU Offload

One of the main advantages of RDMA is the reduction of CPU overhead. To understand this, consider a traditional data transfer using standard sockets. Data travels through several layers of the operating system stack, which involves multiple copies and context switches. The CPU must process headers, manage buffers, and handle acknowledgments.

With RDMA, once the connection is established and memory is registered, the CPU gives a carefully defined description of what should be sent or received to the RDMA hardware. The RDMA engine on the adapter then moves the data between the local and remote memory regions. CPU intervention is limited to posting work requests and processing completion notifications.

This means that even when the network is very busy, CPU usage can stay relatively low. More CPU cycles are free to run the actual scientific or analytic computation, which is exactly the point in HPC.

In practical terms, this allows a cluster to handle more communication without scaling CPU resources just for the network stack.

HPC Topologies for InfiniBand Fabrics

InfiniBand fabrics in HPC clusters use specific topologies that try to balance cost, scalability, and performance. Two of the most common are fat-tree and dragonfly.

A fat-tree is built from multiple layers of InfiniBand switches arranged so that bandwidth increases closer to the root, similar to a tree whose trunk is thicker than its branches. The intention is to provide enough bandwidth in the core so that there are no bottlenecks when many nodes communicate at the same time. This design can provide full bisection bandwidth, which means that any half of the nodes can communicate with the other half at full link rate.

Dragonfly and related designs arrange nodes into groups and connect these groups with high bandwidth links between group routers. The goal is to minimize the number of hops for distant communication and to reduce the total amount of wiring required. These topologies are often used in very large supercomputers where cable management and cost become significant.

Regardless of the exact topology, HPC network designers must consider oversubscription ratios, path diversity, and the ability to handle common communication patterns used by parallel applications. InfiniBand switches include features that support routing across these topologies efficiently.

Collective Communication and HPC Workloads

HPC applications differ from typical client server applications in their communication patterns. Many HPC programs use collective communication, where all processes or a large group of processes participate in operations such as broadcast, reduce, or gather. For example, a broadcast sends data from one node to all nodes, while a reduce operation combines data from all nodes into one result.

InfiniBand and RDMA based networks provide support that makes these operations efficient. Some switches and adapters support offload of certain collective operations, which means that sums or logical reductions can be partially computed inside the network rather than only on the CPUs.

This concept, often called in-network computing, is becoming increasingly important in both traditional HPC and modern AI training clusters. It reduces the time needed for synchronization steps that would otherwise slow down the overall computation, because network delays accumulate when many nodes must wait for each other.

RDMA in Storage and File Systems

RDMA is not only used for pure compute tasks. It is very important in HPC storage as well. Many parallel file systems and distributed storage solutions rely on RDMA to move data between compute nodes and storage servers quickly.

By using RDMA for storage access, the system can move file data and metadata with very low latency and high throughput. This is important because many scientific workloads need to read or write huge amounts of data. Without fast storage networking, even the most powerful compute nodes would spend a lot of time waiting.

Some block storage protocols and file system protocols have RDMA aware versions. They use RDMA write and read operations to move data, while higher level semantics such as file locks and permissions are handled by control protocols. This combination provides the scalability needed for large HPC clusters.

Comparing InfiniBand and High Speed Ethernet for HPC

Both InfiniBand and high speed Ethernet are used in modern compute clusters. The choice between them depends on performance goals, cost, vendor ecosystem, and operational familiarity.

InfiniBand typically offers lower latency, very mature RDMA support, and strong integration with HPC software stacks. It is widely used in top supercomputers. However, it requires specialized switches and adapters and operates as a separate fabric from regular Ethernet unless gateways are used.

High speed Ethernet, especially when combined with RoCE, provides a path to RDMA benefits while using more general purpose hardware. Many cloud providers and large enterprises prefer Ethernet because it fits with their existing tools and processes. However, to achieve RDMA performance comparable to InfiniBand, Ethernet fabrics must be carefully engineered with features like lossless queues and suitable buffer configurations.

In practice, many HPC and AI clusters use InfiniBand for their main compute fabric and Ethernet for management and external connectivity. Others choose a fully Ethernet based architecture with RoCE for simplicity. The core requirement in both cases is that the network supports RDMA or similar offload features to reach the desired level of performance.

Practical Considerations for Beginners

For someone new to networking, HPC networking might initially seem intimidating because it introduces unfamiliar terms like InfiniBand, HCA, verbs, and RDMA. It is helpful to think of HPC networking as a specialized environment where the same fundamental ideas still apply, such as links, switches, addressing, and routing, but they are optimized and offloaded to hardware more aggressively.

At a basic level, you can remember the following simple picture. Traditional networks move data in many small steps between application and network hardware and back again. HPC networking tries to remove as many of those steps as possible. InfiniBand provides a fabric that is designed from the ground up for this, and RDMA is the technique that allows direct memory to memory transfers across that fabric with minimal CPU work.

As you explore more advanced topics, you will encounter programming interfaces such as MPI or InfiniBand verbs, and tools to monitor InfiniBand fabrics and RDMA performance. However, the foundation remains the same: reduce latency, increase bandwidth, save CPU cycles, and scale to very large numbers of nodes.

Summary

HPC networking focuses on enabling very fast communication among many compute nodes in a cluster. InfiniBand provides a specialized fabric that supports high bandwidth and very low latency, and it integrates RDMA directly into its design. RDMA allows data to move directly between memory on different hosts with minimal CPU involvement, which is crucial in large parallel applications.

RoCE brings RDMA concepts into Ethernet based data centers, while keeping many of the same performance advantages when the network is designed correctly. HPC topologies such as fat-tree and dragonfly help preserve performance as the number of nodes grows.

Although HPC networking may seem different from everyday enterprise networking, it is built on the same basic ideas and simply pushes them to the limit in order to support the most demanding compute and storage workloads.