Communicators

Overview

In MPI, a communicator is the fundamental object that defines who can talk to whom. Every point to point or collective operation takes a communicator argument. This tells MPI which group of processes participate in the operation and provides a local numbering of those processes. Understanding communicators is essential for building modular, robust, and scalable distributed memory applications.

This chapter focuses on what communicators are, how they relate to process groups and ranks, why you might want more than one communicator, and the basic MPI routines used to create and manage them. Details of point to point and collective calls themselves are covered in their own chapters.

The Role of Communicators

Each MPI process belongs to one or more communicators. A communicator contains two key pieces of information, a set of processes called a group, and a context that isolates communication within that group. When you call an MPI routine such as MPI_Send, you always pass a communicator. MPI interprets the destination rank as local to the communicator, not as a global process identifier.

When your program starts, MPI automatically provides at least one communicator, MPI_COMM_WORLD. This communicator contains all the processes started for your MPI job, and it is usually where beginners start. However, most nontrivial applications need additional communicators to reflect logical structure, to separate algorithmic components, or to enable safe use of libraries.

The presence of a communicator argument in essentially every MPI communication routine is a constant reminder that communication always happens in some communication context, not globally.

Groups, Ranks, and Communicators

A communicator is built on top of a group. A group is an ordered list of processes. Each member of the group has an integer rank in the range $0$ to $N-1$, where $N$ is the number of processes in the group. The communicator then associates this group with a communication context so that operations are restricted to this group.

You can think of the relationship as:

$$\text{Communicator} = \text{Group} + \text{Context}.$$

The group determines set membership and rank mapping. The context ensures that messages sent using one communicator cannot accidentally match messages expected in another communicator, even if buffer tags and ranks appear identical.

A single process can have rank 0 in one communicator and rank 5 in another communicator. It is therefore meaningless to talk about the rank of a process without naming the communicator.

In code, you usually obtain your rank and the size of the communicator with

MPI_Comm_rank(comm, &rank);
MPI_Comm_size(comm, &size);

where comm is, for example, MPI_COMM_WORLD or some user created communicator.

Why Use Multiple Communicators

Although MPI_COMM_WORLD is convenient, using a single global communicator for everything quickly becomes limiting. There are several important reasons to introduce additional communicators.

One reason is modular design. Suppose you call a third party library that internally uses MPI. If both your code and the library do all communication in MPI_COMM_WORLD, there is a risk of message interference if tags or assumptions about message ordering collide. By giving the library its own communicator that contains the same set or a subset of processes, you isolate the library communication context from your own.

Another reason is algorithmic structure. Many algorithms decompose processes into subgroups such as rows and columns of a virtual process grid, or layers in a multilevel method. Each subgroup can have its own communicator, for example a row communicator and a column communicator in a 2D grid. Collective operations within each subgroup then become natural and efficient.

A third reason is dynamic reconfiguration. Some parts of the computation might engage only a subset of processes. Rather than repeatedly checking conditions inside every collective call, you can create a communicator that contains exactly the processes that participate in a given phase, and restrict communication to that communicator.

Finally, communicators are useful for fault tolerance strategies and for managing multiple independent simulations inside a single MPI job. Each simulation can have its own communicator, even if all processes run in the same job allocation.

Predefined Communicators

MPI defines a few standard communicators that are always available after MPI_Init. The most important are MPI_COMM_WORLD and MPI_COMM_SELF.

MPI_COMM_WORLD contains all the processes in your MPI job. It is the default communicator for most simple programs. When you request the size and rank from this communicator, you obtain the global job size and the rank of each process in this global context.

MPI_COMM_SELF is a communicator that contains only the calling process. Every process has its own instance of MPI_COMM_SELF, with size 1 and rank 0. This communicator is useful when you want to perform collective operations locally on a single process, for example if a routine has a collective interface but you want to call it in a single process context, or when you develop algorithms that treat each process as an independent unit.

There are other predefined communicators for specific purposes, such as intercommunicators created by certain advanced MPI functions, but as a beginner you will mostly work with MPI_COMM_WORLD and custom intracommunicators derived from it.

Duplicating Communicators

A common first step beyond MPI_COMM_WORLD is to create a duplicate communicator. A duplicate communicator has the same group of processes and the same rank mapping as the original, but it has a separate communication context. This separates messages conceptually, so that communication in the duplicate cannot accidentally match messages in the original communicator.

You create a duplicate communicator with MPI_Comm_dup:

MPI_Comm new_comm;
MPI_Comm_dup(MPI_COMM_WORLD, &new_comm);

After this call, every process has a valid new_comm communicator. Calls to MPI_Comm_rank and MPI_Comm_size on new_comm will return the same values as on MPI_COMM_WORLD. However, sending a message with a given tag and destination using MPI_COMM_WORLD will never be received by a matching receive that waits on new_comm, and vice versa.

Duplicating communicators is particularly useful when you want to pass a communicator into a library or component while preserving the isolation of your own communication. The library can safely use the duplicated communicator internally.

Splitting Communicators into Subgroups

To create subgroups of processes, MPI provides MPI_Comm_split. This routine partitions an existing communicator into one or more disjoint communicators based on user supplied keys. All processes in the original communicator must call MPI_Comm_split, but they can specify different colors and keys.

The basic usage is:

MPI_Comm subcomm;
MPI_Comm_split(parent_comm, color, key, &subcomm);

The color argument is an integer that determines group membership. All processes in parent_comm that pass the same color will end up in the same new communicator. Processes that pass MPI_UNDEFINED as color do not participate in any new communicator and receive MPI_COMM_NULL in subcomm.

Within each color group, the key argument determines the ordering of processes in the new communicator. The ranks in the new communicator are assigned by sorting processes by key values in increasing order. This allows you to preserve or change rank ordering as needed.

A common pattern is to split MPI_COMM_WORLD into communicators that correspond to different roles. For example, you might use the first half of processes for one task and the second half for another:

int world_rank, world_size;
MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
MPI_Comm_size(MPI_COMM_WORLD, &world_size);
int color = (world_rank < world_size / 2) ? 0 : 1;
MPI_Comm subcomm;
MPI_Comm_split(MPI_COMM_WORLD, color, world_rank, &subcomm);

After this call, processes with color == 0 share one communicator, and processes with color == 1 share another. Within each subcommunicator, ranks start from 0 again and are assigned according to the key ordering, here simply the original world rank.

Creating Communicators from Explicit Groups

In some cases you want finer control than MPI_Comm_split provides. MPI allows you to form groups explicitly, then create communicators from those groups. This is done in two stages: first manipulate groups with routines like MPI_Group_incl or MPI_Group_excl, then create a communicator with MPI_Comm_create or related functions.

You start from the group of an existing communicator, for example:

MPI_Group world_group;
MPI_Comm_group(MPI_COMM_WORLD, &world_group);

Then you derive a new group. For instance, to include a specific set of ranks:

int ranks[3] = {0, 2, 4};
MPI_Group new_group;
MPI_Group_incl(world_group, 3, ranks, &new_group);

Now you create the communicator from this group:

MPI_Comm new_comm;
MPI_Comm_create(MPI_COMM_WORLD, new_group, &new_comm);

Every process in MPI_COMM_WORLD must call MPI_Comm_create with the same parent communicator and group arguments. Processes that are not in new_group will receive MPI_COMM_NULL in new_comm. Processes in new_group will obtain a valid communicator that contains exactly the selected ranks.

Explicit group based construction is useful when the membership of a communicator is irregular or when it depends on data computed at runtime. It gives you direct control over which processes participate.

Communicators and Communicators Topologies

Communicators can be associated with process topologies, which are logical structures that describe neighbor relationships, such as Cartesian grids or general graphs. The details of topologies are handled in a separate chapter, but it is important to understand the connection at a high level.

When you create a Cartesian communicator, for example with MPI_Cart_create, MPI takes an existing communicator and the desired grid dimensions, and returns a new communicator that embeds grid coordinates into ranks. From that point on, you can ask for your coordinates and neighbors using this special communicator.

The key idea is that topological information is bound to communicators. Operations that rely on topology, like neighborhood collectives, work with the communicator that encodes that structure. Your code can switch between different communicators to move between different logical views of the process set, such as a global view and a grid based view.

Intercommunicators and Intracommunicators

So far the discussion has focused on intracommunicators. An intracommunicator contains a single group of processes and is used for communication within that group. This is the type of communicator you use most of the time.

MPI also defines intercommunicators. An intercommunicator connects two disjoint groups of processes and is used for communication between the groups. Within an intercommunicator, each process has a local group and a remote group. Intercommunicators are useful for advanced patterns such as client server models or when one group of processes spawns another at runtime.

The exact mechanisms for creating and using intercommunicators are more advanced and typically appear later in an MPI course. For now it is sufficient to know that MPI has this concept, and that functions often specify whether they expect an intra or an intercommunicator.

Lifetime and Cleanup of Communicators

Communicators, like many MPI objects, occupy internal resources. Although MPI usually reclaims all resources when MPI_Finalize is called, good practice is to free communicators that you no longer need. This helps avoid resource exhaustion in long running or complex applications.

You free a communicator with MPI_Comm_free:

MPI_Comm_free(&comm);

After this call, the handle comm becomes invalid and can no longer be used. All processes in the communicator must participate in MPI_Comm_free. MPI ensures that cleanup is safe and coordinated.

You should not free predefined communicators such as MPI_COMM_WORLD or MPI_COMM_SELF. MPI reserves these for the duration of the program. Only communicators that you have explicitly created or duplicated should be freed.

Design Considerations and Best Practices

Good communicator design supports clarity, modularity, and performance. A few guidelines are particularly helpful in practice.

First, avoid overusing MPI_COMM_WORLD. It is fine to begin with it, but as soon as you introduce modules, libraries, or multiple algorithmic phases, consider creating dedicated communicators. This prevents accidental interference and makes it easier to reason about which processes are active at any point.

Second, plan the structure of your communicators to match the structure of your algorithm. For example, if your method uses a 2D process grid, create row and column communicators. Use row communicators for row wise collectives and column communicators for column wise collectives. Similarly, if you run several independent simulations, give each simulation its own communicator so that collectives do not cross simulation boundaries.

Third, maintain clear documentation about which communicator a given routine expects. Passing the wrong communicator can lead to subtle deadlocks or incorrect behavior, especially in collective operations. It is often useful to define function interfaces that explicitly accept a communicator parameter and to avoid implicitly relying on MPI_COMM_WORLD.

Finally, be consistent about communicator cleanup. Free communicators when the associated algorithmic phase or object lifetime ends. This practice not only saves resources but also helps you detect logic errors since using a freed communicator is invalid and likely to be caught by debugging builds or tools.

By treating communicators as first class design elements rather than incidental details, you gain powerful control over structure and correctness in distributed memory parallel programs.