Point-to-point communication

Core ideas of point-to-point communication

In distributed-memory parallel programming, and MPI in particular, point-to-point communication means data is transferred explicitly between exactly two processes: one sender and one receiver. This is in contrast to collective operations, where many processes participate in one call.

Point-to-point communication is used for:

• Exchanging boundary data between neighboring subdomains in domain decompositions
• Sending work units from a master process to workers (and getting results back)
• Building custom communication patterns that are not covered by collectives

At a high level, each message has:

• A source process (who sends)
• A destination process (who receives)
• A buffer (the actual data)
• A datatype
• A message tag (an integer used to distinguish different kinds of messages)
• A communicator (the group of processes and their context)

MPI provides matching pairs of calls – send on one process, receive on another – that together complete a point-to-point operation.

Basic MPI point-to-point operations

Blocking send and receive

The simplest operations are blocking:

• MPI_Send: send data, return when the user buffer can be safely reused
• MPI_Recv: receive data, return when the buffer has been filled

Typical usage pattern between two processes:

• Process 0 calls MPI_Send(...) to process 1
• Process 1 calls MPI_Recv(...) from process 0

Example skeleton in C-like pseudocode:

int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0) {
    double x = 3.14;
    MPI_Send(&x, 1, MPI_DOUBLE, 1, 0, MPI_COMM_WORLD);
} else if (rank == 1) {
    double y;
    MPI_Recv(&y, 1, MPI_DOUBLE, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}

Key points about blocking operations:

• The call does not return until the library guarantees the user’s buffer is safe:
• For MPI_Send: the data has either been delivered or copied into an internal buffer
• For MPI_Recv: the data has arrived and been copied into the receive buffer
• They are easy to reason about but can lead to synchronization and waiting.

Matching rules

For a message transfer to succeed correctly, the following must match between the send and receive:

• Same communicator
• Same datatype and count (same number and type of elements)
• The receiving side:
• source matches the sender’s rank (or uses wildcards, see below)
• tag matches the tag used by the sender (or uses wildcards)

If any of these are inconsistent, you may get incorrect results, deadlocks, or runtime errors.

Wildcards: `MPI_ANY_SOURCE` and `MPI_ANY_TAG`

Point-to-point communication allows flexible reception using wildcards:

• MPI_ANY_SOURCE for the source rank
• MPI_ANY_TAG for the tag

These can be specified in MPI_Recv when you don’t know exactly which process or tag will send the next message.

Example:

MPI_Status status;
MPI_Recv(buf, n, MPI_DOUBLE, MPI_ANY_SOURCE, MPI_ANY_TAG,
         MPI_COMM_WORLD, &status);
/* After the receive, you can inspect who sent the message and with which tag */
int src  = status.MPI_SOURCE;
int tag  = status.MPI_TAG;
int count;
MPI_Get_count(&status, MPI_DOUBLE, &count);

Wildcards are powerful for dynamic or irregular communication patterns, but:

• They make reasoning about correctness harder
• They can interact badly with buffering and message ordering if not used carefully

Common communication patterns

Point-to-point operations are used to implement a variety of patterns:

• Neighbor exchange (halo / ghost cell exchange)
  Each process exchanges data only with a few known neighbors (e.g., left/right in 1D, 4 or 8 neighbors in 2D, 6 in 3D).
• Master–worker (task farm)
  One or a few processes send tasks to many workers, then receive results back.
• Pipeline
  Processes are arranged in a line; data flows through them as in an assembly line.
• Ring communication
  Each process sends to (rank + 1) mod P and receives from (rank - 1 + P) mod P.

These patterns can be built from simple sequences of sends and receives. Higher-level MPI routines (collectives and neighborhood collectives) often internally rely on point-to-point operations.

Blocking vs non-blocking communication

Point-to-point APIs are often provided in both blocking and non-blocking variants. Knowing when to use which is central to performance and correctness in distributed-memory programs.

Blocking operations: simplicity first

Blocking operations (MPI_Send, MPI_Recv) are conceptually straightforward:

• The program order in the code largely reflects the communication order
• Variables passed as buffers are safe to read/write again once the call returns
• Useful prototyping tools: good for getting the first correct version

But they can easily cause:

• Idle time: processes wait in a blocking call while others are still computing
• Unintended synchronization: one slow process makes many others wait
• Deadlocks if operations are not ordered carefully (covered in detail elsewhere)

Non-blocking operations: overlapping communication and computation

Non-blocking operations, such as MPI_Isend and MPI_Irecv, allow you to initiate a communication and then continue doing useful work while the transfer proceeds in the background. Completion is checked later using MPI_Wait or MPI_Test.

Basic workflow:

1. Post receives with MPI_Irecv
2. Initiate sends with MPI_Isend
3. Perform independent computation
4. Complete communication with MPI_Wait / MPI_Waitall (or MPI_Test variants)

Example skeleton:

MPI_Request reqs[2];
/* Post non-blocking receive */
MPI_Irecv(recvbuf, n, MPI_DOUBLE, src, tag, MPI_COMM_WORLD, &reqs[0]);
/* Post non-blocking send */
MPI_Isend(sendbuf, n, MPI_DOUBLE, dst, tag, MPI_COMM_WORLD, &reqs[1]);
/* Do some useful computation here that does not touch sendbuf/recvbuf */
/* Ensure communication is completed before using the data */
MPI_Waitall(2, reqs, MPI_STATUSES_IGNORE);

Important rules for non-blocking operations:

• Buffers must not be reused or modified until the non-blocking operation has completed (i.e., after MPI_Wait* returns successfully).
• Non-blocking operations are still logically blocking at the moment you wait; overlap arises only if:
• There is work to do between start and wait, and
• The implementation and hardware enable progress in the communication.

Message tags and logical channels

Message tags are integer labels attached to each point-to-point message. They act like separate "channels" on the same source/destination pair.

Applications commonly use them for:

• Distinguishing different types of messages (e.g., control vs data)
• Distinguishing different timesteps or iterations
• Separating messages for different algorithms or modules

Design considerations:

• Keep tag usage simple and documented; e.g.:
• Tag 0: configuration
• Tag 1: boundary exchange
• Tag 2: convergence check
• Avoid relying on subtle tag patterns that are hard to maintain.
• Respect the valid tag range (MPI_TAG_UB gives the maximum allowed value).

Ordering guarantees and message matching

MPI guarantees certain ordering properties for point-to-point messages:

• For a fixed pair of processes, and a fixed communicator and tag:
• If the sender posts two sends in order, S1 then S2,
• And the receiver posts matching receives that could receive from either,
  then S1 will be received before S2.

More precisely:

• Messages sent from process A to process B with the same tag and communicator are received in the order they were sent, provided the receives are non-overtaking (i.e. able to match them).
• If you use different tags, the library is free to deliver them in any order unless the receives explicitly constrain the matching.

Practically:

• Do not rely on ordering across different tags.
• Prefer simple, well-structured message patterns (e.g., exchange by stage, or use tags consistently per stage).

Common pitfalls in point-to-point communication

Deadlocks from mismatched ordering

Deadlocks can easily arise when processes wait on each other in a cycle. For example, two processes both call blocking sends to each other before posting receives.

Simple ways to avoid this in point-to-point patterns:

• Use a consistent ordering: some processes send first, others receive first.
• Or use non-blocking operations for at least one side of the communication.
• For neighbor exchanges, a common trick is to alternate the direction per rank (even ranks send then receive, odd ranks receive then send).

Deadlocks and detailed patterns are covered more deeply elsewhere, but they frequently originate in incorrect point-to-point logic.

Buffer misuse with non-blocking operations

Misusing send or receive buffers with non-blocking calls is a frequent error:

• Writing to a send buffer before the corresponding MPI_Isend is completed
• Reading from a receive buffer before the corresponding MPI_Irecv is completed
• Reusing request handles incorrectly

Defensive habits:

• Group communication-related variables and requests together.
• Immediately follow up communications with appropriate MPI_Wait* calls before leaving a scope where buffers could be reused or deallocated.
• Use debugging tools and runtime checks when the MPI library supports them.

Incorrect message sizes or datatypes

Point-to-point operations require agreement on the data layout:

• Sender and receiver must agree on:
• Count (number of elements)
• Datatype (e.g., MPI_INT, MPI_DOUBLE, or a matching custom type)
• If they don’t, you may see:
• Truncated messages
• Uninitialized memory being read
• Subtle data corruption

When message size may vary:

• The sender can send the size first (e.g., an integer count), then the receiver allocates and issues a second receive for the actual data.
• Alternatively, the receiver can post a large-enough receive and pass a status object to MPI_Recv, then use MPI_Get_count to obtain the actual number of elements received.

Designing point-to-point communication in applications

Point-to-point communication design has a big impact on scalability and maintainability. When planning communication in a distributed-memory application:

• Identify communication partners
• Nearest neighbors? Master–worker? Arbitrary graph of dependencies?
• Define a clear protocol
• Which process sends what to whom, and when?
• What tags represent what kinds of messages?
• Minimize the number and volume of messages
• Fewer, larger messages are often more efficient than many small ones.
• Consider overlap and asynchrony
• Can you use non-blocking calls to hide latency behind computation?
• Think about robustness
• Avoid fragile reliance on MPI_ANY_SOURCE unless necessary.
• Make order and matching as explicit and deterministic as possible.

Higher-level communication mechanisms (neighbor collectives, graph communicators, domain-specific libraries) are often built atop these same point-to-point concepts. A solid understanding of point-to-point behavior is therefore essential to use these tools effectively and to debug issues when they arise.