Performance considerations

Overview

Performance in shared memory parallel programming is not only about adding more threads. It is about how threads interact with each other and with the memory system of a node. In this chapter, the focus is on how to think about performance when you use OpenMP or similar thread based models on modern multicore CPUs. Concepts like race conditions and synchronization have their own chapters. Here the interest lies in how they influence speed, scalability, and hardware utilization.

Throughput, latency, and speedup

When you add threads to a shared memory program, you usually expect some speedup. A simple measure is the ratio of serial runtime to parallel runtime. If $T_1$ is the time with one thread and $T_p$ is the time with $p$ threads, then the speedup is

$$S_p = \frac{T_1}{T_p}.$$

Parallel efficiency compares the achieved speedup to ideal linear scaling:

$$E_p = \frac{S_p}{p} = \frac{T_1}{p \, T_p}.$$

In a shared memory context, $T_p$ is influenced strongly by memory behavior and coordination among threads. High efficiency means that most thread time is spent doing useful work instead of waiting on memory, locks, or other threads. A key goal in this chapter is to understand how your choices in OpenMP constructs, data structures, and scheduling affect $T_p$.

Costs of creating and managing threads

In shared memory parallel programming, threads are relatively lightweight, but they are not free. Each #pragma omp parallel region has a creation and teardown cost. If you enter and leave parallel regions very frequently, this overhead can dominate total runtime, especially for small workloads per region.

A common performance pattern is to create a parallel region once, then keep it active while executing multiple parallel sections or work sharing constructs such as for or sections. For example, repeated parallel loops are often written as a single parallel that encloses several for constructs, instead of starting a new parallel region for each loop.

Nested parallel regions add further overhead. They may be useful in some hybrid or very specific situations, but they typically reduce performance on standard multicore nodes, because thread pools are split into smaller groups and management overhead increases. In many cases, it is beneficial to keep nesting disabled until you have a clear performance reason to enable it.

In summary, thread management should be amortized over a substantial amount of work. Many small, short lived parallel regions tend to give poor speedup.

Load balance and scheduling choices

Even if thread management overhead is small, performance can be lost when the work is not evenly distributed across threads. Shared memory models like OpenMP offer different loop scheduling options, each with performance trade offs.

Static scheduling assigns loop iterations to threads in a fixed pattern before execution starts. This has minimal scheduling overhead and usually predictable memory access patterns, which favor cache performance. It works well when each iteration takes roughly the same time. If some iterations are much more expensive than others, static scheduling can lead to idle threads toward the end of the loop, and the overall runtime is then determined by the thread that received the heaviest portion of the work.

Dynamic scheduling assigns chunks of iterations to threads as they request them during execution. This can balance irregular workloads, but has extra overhead due to dynamic assignment and can disturb locality if iterations that touch related data are processed by different threads in an unpredictable order.

Guided scheduling is a compromise, where large chunks are assigned at first, then the chunk size gradually decreases. This can reduce overhead compared to fully dynamic scheduling, while still responding to differences in iteration cost.

Chunk size is another performance lever. A chunk that is too small increases scheduling overhead and may break up useful memory locality. A chunk that is too large can reintroduce load imbalance. There is no universal best choice. For regular loops, default static scheduling is often good, while dynamic or guided schedules with modest chunk sizes tend to work better for loops where execution time per iteration is highly variable.

False sharing and cache effects

Because threads in a shared memory system access a common address space, they also share parts of the memory hierarchy. Caches are usually organized into lines, for example 64 bytes per line, and coherence protocols keep these lines consistent across cores. False sharing occurs when different threads write to different variables that happen to reside in the same cache line. Even though there is no conceptual sharing of the logical variables, the hardware must treat the cache line as shared and repeatedly invalidate and transfer it between cores. This can generate a large amount of coherence traffic and degrade performance severely.

A classic example is an array where each thread updates its own element, but the elements are tightly packed and several of them fall within one cache line. Incrementing those elements from different threads can lead to continuous ping pong of the line between cores.

To avoid false sharing, you can use padding so that frequently updated per thread data structures are separated by at least one cache line, or align arrays so that different threads work on regions that do not share cache lines. OpenMP’s private and reduction constructs can also help structure per thread data so that it is not unintentionally shared at the cache line level.

Cache locality is another essential performance aspect. If each thread repeatedly touches data that fits in its local caches, performance is high. If threads walk through memory with long strides or irregular patterns, cache misses increase, and performance becomes limited by memory bandwidth and latency instead of compute capability. Loop interchange, tiling, and reordering of data can improve locality but the exact techniques belong to other chapters. In shared memory, you must also consider that several cores share parts of the memory hierarchy, so poor locality in one thread can hurt others, because all compete for the same shared caches and memory bandwidth.

Memory bandwidth and scalability limits on a node

Multicore CPUs typically have many cores that share some level of the memory subsystem. For many data intensive workloads, the total rate at which data can be fetched from memory is bounded by the node’s memory bandwidth. When only one or two threads run, the bandwidth does not yet saturate and adding more threads increases throughput. Beyond a certain number of threads, the aggregate memory traffic reaches the physical limit of the memory system. Additional threads continue to compete for the same bandwidth, so performance stops scaling and may even degrade due to contention.

This effect is especially visible in memory bound kernels, such as simple vector operations or stencil codes with limited computation per byte of data loaded. On a single node, it is common to see strong scaling up to a modest core count, followed by a plateau where extra threads produce little gain.

The practical consequence is that you cannot judge scalability only by counting cores. You must understand whether your code is compute bound or memory bound. If it is memory bound, optimizations that increase arithmetic intensity, improve cache reuse, or reduce unnecessary memory traffic can extend the region where additional threads improve performance.

Sometimes, using all logical cores, including hardware threads created by simultaneous multithreading, can hurt performance for bandwidth bound workloads. Two hardware threads on the same core share execution and cache resources. If they both execute bandwidth hungry code, they may only add contention. In such scenarios, binding one software thread per physical core can be faster than filling all hardware threads.

NUMA effects and thread placement

Modern multicore systems often use NUMA architectures. The total memory is physically divided among several memory controllers, each attached to a group of cores. Accessing memory that is local to a core’s NUMA domain is faster than accessing memory in another domain. The operating system and hardware present a unified view of memory, but the non uniform access costs still affect performance.

This has two implications for shared memory programming. First, if threads frequently access data in a remote NUMA domain, the latency and bandwidth penalties can reduce performance. Second, if memory allocation and thread placement are not coordinated, large arrays may end up placed in one domain while threads that use them run on other domains.

To mitigate this, you can use first touch allocation patterns. When memory is first written, it is typically placed in the NUMA domain of the core that performs the write. Initializing large arrays in parallel, using the same loop partitioning that will be used in the main computation, can help ensure that data is distributed across NUMA nodes in a way that matches the thread mapping. Thread affinity mechanisms, including OpenMP’s proc_bind clause or environment variables provided by vendors and runtimes, let you pin threads to specific cores so that their execution is stable and predictable.

The interaction between NUMA and thread placement is a performance consideration specific to shared memory systems. Unlike distributed memory systems, where data is explicitly partitioned, NUMA penalties arise implicitly if data and threads end up on different domains.

Synchronization overhead and contention

Synchronization primitives such as barriers, locks, and atomic operations are necessary for correctness when threads cooperate on shared data. However, each such primitive introduces a performance cost. A full barrier forces all threads to wait until the slowest thread arrives, which combines synchronization overhead with any existing load imbalance. Placing barriers inside tight loops can dramatically slow down execution.

Locks and mutual exclusion protect critical sections, but they also serialize access to data. If many threads compete for the same lock, they spend considerable time waiting instead of computing. This can make an apparently parallel region behave almost like serial code. In some cases, it is possible to redesign algorithms so that shared state is reduced or accessed less frequently. Techniques such as using thread local buffers and combining results later, or employing reductions instead of explicit locks, can significantly reduce contention.

Atomic operations provide finer grained synchronization than locks, but they still involve hardware level synchronization and may serialize updates to a single memory location. If a single shared counter is incremented by thousands of threads, the atomic increments can become a bottleneck. Distributing counters per thread and aggregating them at the end often scales better.

From a performance perspective, the general guideline is to minimize the amount of time threads spend in synchronized regions and to avoid central shared structures that all threads touch frequently.

Tasking overhead and granularity

Shared memory models include tasking features that allow you to express irregular or recursive parallelism. Creating and scheduling tasks has overhead similar in spirit to thread creation overhead. If tasks are too fine grained, that is, if each task performs very little work, the cost of managing them can exceed the useful work they perform.

A key concept is task granularity. Coarse grained tasks, where each task performs a relatively large amount of computation, amortize the creation and scheduling costs better. However, if tasks are too large and their workloads vary, you may reintroduce load imbalance. The challenge is to choose a granularity that balances these effects.

The tasking subsystem can be very effective for irregular problems, but it must be used with an awareness of management overhead. In some cases, simple parallel loops or work sharing constructs are faster than task based parallelism, because they have lower runtime overhead.

Choosing the number of threads

On a shared memory system, you can control the number of threads that your program uses. While using the maximum number of cores on a node seems appealing, it is not always optimal. For compute bound workloads that fit well in caches and have good vectorization, using one thread per core will often provide close to linear speedup, at least within a socket. For memory bound workloads, there is often a saturation point where adding more threads does not reduce runtime further.

You can explore this by measuring performance for different thread counts. Plotting runtime or throughput versus the number of threads often reveals a curve that rises quickly and then levels off. The practical choice for production runs may be near the knee of that curve, where additional threads no longer pay off.

It is also important to consider that on shared clusters, using more cores per job than necessary may reduce fairness and system throughput. Allocating a moderate number of threads per process and using multiple processes on a node, in combination with distributed memory techniques, can sometimes yield better node utilization. That topic connects to hybrid programming and is addressed elsewhere. Within this chapter, the main point is that in pure shared memory codes, the optimal thread count is a performance parameter that you should treat as tunable, not fixed.

Measuring and reasoning about performance

To make informed decisions about performance considerations, you need measurements. Timers around parallel regions reveal how runtime changes as you adjust thread counts, scheduling policies, and data layouts. Simple microbenchmarks can help you characterize which parts of the code are compute bound and which are memory bound.

A useful mental model is to view total runtime as a combination of useful work and overhead:

$$T_p \approx T_{\text{compute}} + T_{\text{memory}} + T_{\text{sync}} + T_{\text{overhead}},$$

where $T_{\text{compute}}$ is the time spent in arithmetic operations, $T_{\text{memory}}$ is time stalled on memory, $T_{\text{sync}}$ is time in synchronization, and $T_{\text{overhead}}$ covers thread management and scheduler overheads. While you cannot always measure each term directly, profiling tools and targeted experiments let you infer which component dominates. Optimizations in shared memory codes usually target one or more of these components. For example, changing loop scheduling affects $T_{\text{sync}}$ and load balance, data layout affects $T_{\text{memory}}$, and restructuring critical sections affects $T_{\text{sync}}$ and contention.

Putting it all together

Performance considerations for shared memory programming arise from the interaction between your parallel code, the thread runtime, and the underlying multicore hardware. Effective use of OpenMP or similar models requires more than correct syntax. You must structure parallel regions to minimize overhead, choose scheduling that balances work without destroying locality, avoid false sharing and excessive contention, respect NUMA boundaries, and select thread counts that match the memory and compute characteristics of your workload.

These considerations are local to a single node but they form the foundation for efficient hybrid and large scale parallel programs. A well tuned shared memory component is a prerequisite for getting the most out of modern HPC clusters.