Exascale computing

What “Exascale” Means

Exascale computing refers to systems capable of sustaining on the order of $10^{18}$ floating‑point operations per second (1 exaFLOP). In practice:

• Peak performance: around 1–10 exaFLOP/s (or more) in 64‑bit precision.
• Real application performance: much lower than peak, but still orders of magnitude beyond previous petascale systems.
• Scale: millions of cores and/or tens of thousands of GPUs or other accelerators.

Key ideas:

• Exascale is not just “faster hardware” – it changes how we must program, manage data, and think about reliability.
• Power, parallelism, and resilience become dominant constraints.

Key Challenges at Exascale

Power and Energy Constraints

At exascale, energy is often the primary design limit rather than raw performance:

• A naive design could require >100 MW; practical systems must target ~20–30 MW or less.
• Energy per operation and per data movement must be minimized:
• Compute is relatively cheap in energy.
• Moving data (especially across nodes) is expensive.
• Algorithms and codes need to be energy‑aware:
• Fewer memory accesses and less communication.
• Fewer, smarter I/O operations.
• Avoiding unnecessary recomputation or data movement.

You will increasingly see metrics like “FLOPs per watt” and “simulated time per joule” alongside “time to solution”.

Extreme Parallelism and Concurrency

Exascale systems may have:

• O($10^6$–$10^7$) hardware cores.
• O($10^7$–$10^9$) hardware threads.
• Massive numbers of GPU streaming multiprocessors and vector units.

Implications:

• Parallelism exists at many levels simultaneously:
• Across nodes (MPI, distributed memory).
• Within nodes (threads, tasks, shared memory).
• Within accelerators (warps/wavefronts, SIMT).
• Within cores (SIMD/vector units).
• Simple “flat MPI” or “one big OpenMP region” designs can hit scaling limits.
• Load balancing and scheduling become much harder:
• More heterogeneous workloads.
• More dynamic behavior (AMR, irregular graphs, etc.).

Programming models must expose and manage this deep hierarchy of parallelism.

Memory, Data Movement, and Bandwidth

Memory is no longer a single, homogeneous pool:

• Multi‑level memory hierarchies:
• High‑bandwidth memory (HBM) near accelerators.
• DDR or other “capacity” memory.
• Possibly non‑volatile memory tiers.
• Growing gap between compute capability and memory bandwidth.

Consequences:

• Data layout and locality can dominate performance.
• Algorithms are redesigned to maximize arithmetic intensity (FLOPs per byte moved).
• “Communication‑avoiding” and “memory‑efficient” variants of classical algorithms become essential.

You often trade extra compute for less data movement, because compute is cheap, bandwidth is not.

Resilience and Fault Tolerance

As system size grows:

• Mean time between failures (MTBF) for the whole system can drop to hours or even minutes.
• Components fail continuously: nodes, links, memory, storage.

Traditional resilience strategy:

• Periodic global checkpoints to a parallel filesystem.
• Restart from last checkpoint when a failure occurs.

At exascale:

• Full checkpoints can be too slow and too large to write frequently.
• Naive checkpoint/restart can consume a large fraction of runtime and I/O bandwidth.

This drives research and adoption of:

• Algorithm‑based fault tolerance (ABFT).
• Local/partial checkpointing (e.g., node‑local storage, asynchronous snapshots).
• Redundant computations or data encoding.
• Resilient programming models that can handle process/node loss without aborting the whole job.

Heterogeneity and Complexity

Exascale systems are typically heterogeneous:

• Mix of CPUs, GPUs, and sometimes other accelerators.
• Multiple memory technologies and interconnects.
• Vendor‑specific features and proprietary software stacks.

Challenges:

• Portability: a code optimized for one exascale system may not run efficiently (or at all) on another without adaptation.
• Performance portability: using the same source code to achieve good performance across different architectures.
• Software complexity: multiple layers (MPI + OpenMP/OpenACC/CUDA + math libraries + I/O libraries, etc.).

Abstractions and portable programming models are needed to shield applications from low‑level hardware differences.

Architectural Trends in Exascale Systems

Node‑Level Design

Common characteristics of exascale nodes:

• Manycore CPUs: dozens to hundreds of cores per socket.
• Multiple GPUs or accelerators per node:
• High bandwidth, high compute density.
• Tight CPU–GPU integration (e.g., shared address space, fast links).
• Complex NUMA layouts and memory tiers.

For HPC applications, this typically means:

• Hybrid programming:
• MPI across nodes.
• Threads or GPU kernels within nodes.
• Careful mapping of data and work to specific devices and memory types.

System‑Level Architecture

At system scale:

• Hierarchical interconnects:
• Node‑internal (PCIe, NVLink, similar high‑speed links).
• Rack‑level networks.
• Global high‑performance networks (fat‑tree, dragonfly, etc.).
• Network features:
• High bisection bandwidth.
• Low latencies.
• Offload engines for collective operations, reductions, and sometimes atomic operations or storage interactions.

Topology awareness becomes more important:

• Mapping MPI ranks or tasks to network topology can significantly affect performance.
• Some job schedulers and runtimes expose topology info for applications to exploit.

Energy‑Efficient Hardware Features

To stay within power budgets, hardware introduces:

• Dynamic voltage and frequency scaling (DVFS).
• Power capping interfaces and APIs.
• Deep sleep states and clock gating.

Applications and runtimes that adapt to these features can:

• Reduce energy consumption at small performance cost.
• Sometimes even improve performance by avoiding thermal throttling or contention.

Programming Models and Software for Exascale

Next‑Generation Programming Models

Existing models (MPI, OpenMP, CUDA, etc.) remain central but are evolving:

• MPI:
• Support for larger scales (more ranks, bigger collectives).
• Fault tolerance extensions (ULFM and others).
• Asynchronous progress and improved one‑sided communication.
• Shared‑memory/tasking models:
• Task‑based parallelism (OpenMP tasks, TBB, Kokkos tasks, HPX, etc.).
• Better support for irregular and dynamic workloads.
• Accelerator‑focused models:
• CUDA, HIP, SYCL, OpenACC, OpenMP target offload.
• Emphasis on standard, vendor‑neutral approaches (e.g., SYCL, OpenMP offload) for performance portability.

Trend: combining message passing, node‑local tasks, and accelerator offload into flexible, composable programming models.

Performance Portability Frameworks

To deal with heterogeneity and frequent hardware changes, exascale software effort focuses on:

• Abstraction libraries:
• Kokkos, RAJA, SYCL‑based frameworks, etc.
• Domain‑specific libraries:
• Exascale‑ready linear algebra, FFT, and PDE solvers that target multiple backends.
• Unified memory and data management:
• Abstractions to move data between CPU and GPU memories efficiently and transparently when possible.

For learners, this means:

• Knowing how to express parallelism is more important than targeting a specific vendor API.
• Understanding the underlying patterns (data parallelism, task parallelism, communication patterns) allows you to use these frameworks effectively.

Software Stack Complexity

The exascale software stack includes:

• Low‑level runtime layers (MPI, communication libraries, GPU drivers).
• Math and solver libraries tuned for exascale architectures.
• I/O, checkpointing, and workflow tools that scale to exascale data.
• Performance analysis and tuning tools that can handle millions of threads.

Practical implications:

• More reliance on system‑provided modules and containers.
• Greater importance of build systems (CMake, Spack, etc.) to manage complex dependencies on large systems.
• Need for automated testing and continuous integration that can operate at scale (e.g., testing for race conditions and scalability issues).

Algorithmic and Numerical Considerations

Communication‑Avoiding and Hierarchical Algorithms

At exascale, communication costs (both latency and bandwidth) can be dominant. Algorithms are redesigned to:

• Minimize global synchronization (e.g., fewer global reductions).
• Reduce communication volume, even if more floating‑point work is done.
• Exploit hierarchical machine structure:
• Intra‑node vs inter‑node communication.
• Local vs global operations.

Examples of trends (without going into full detail):

• Communication‑avoiding Krylov solvers and factorizations.
• Hierarchical collectives (tree‑based, topology‑aware reductions).
• Multi‑level domain decompositions matching hardware hierarchy.

Mixed Precision and Novel Arithmetic

To gain more performance and reduce energy:

• Mixed‑precision algorithms:
• Most operations in lower precision (e.g., FP16, BF16).
• Corrective steps in higher precision (e.g., FP32 or FP64).
• Use of hardware features like tensor cores for dense linear algebra and some PDE and ML‑inspired kernels.

For scientific computing, this requires:

• Careful analysis of numerical stability and accuracy.
• Validation that mixed‑precision results meet domain‑specific error tolerances.

In Situ and In Transit Data Processing

At exascale, writing all raw data to disk becomes impractical:

• I/O bandwidth and storage capacity cannot keep up with simulation rates.
• Storing full resolution outputs for every time step is often impossible.

Instead:

• In situ analysis: compute diagnostics, statistics, derived fields, or visualizations while the simulation runs, before data is discarded.
• In transit analysis: data is streamed to dedicated analysis resources, reducing storage pressure on the main system.

This changes typical workflows:

• You design what you want to analyze and visualize before running.
• You trade some compute resources to significantly reduce I/O and storage costs.

Exascale Applications and Use Cases

Traditional Domains at New Scales

Many classical HPC fields are being pushed to new limits:

• Climate and weather:
• Higher resolution, better physics, more ensemble members.
• Long‑term climate projections with finer regional detail.
• Materials and chemistry:
• More accurate quantum simulations.
• Larger systems and longer timescales.
• Astrophysics and cosmology:
• Galaxy formation and large‑scale structure simulations with unprecedented detail.
• Engineering and energy:
• Full‑system simulations (e.g., entire aircraft or reactors) with high fidelity.

The move to exascale often involves:

• Rewriting or refactoring legacy codes to exploit new architectures.
• Adopting new algorithms that scale to millions of cores.

Coupled Multiphysics and Multiscale Simulations

Exascale performance enables:

• Coupling multiple physical models:
• For example, fluid dynamics + chemistry + radiation + structures.
• Bridging multiple scales:
• From molecular to macroscopic, or from microseconds to years.

These simulations can require:

• Sophisticated coupling frameworks.
• Flexible time‑stepping and load balancing.
• Robust I/O and workflow orchestration to manage complex runs.

Integration with AI and Data‑Driven Methods

Modern exascale applications increasingly combine:

• Simulation:
• To generate high‑fidelity data and explore physical regimes.
• Data analytics and machine learning:
• To build surrogate models, accelerate solvers, or steer simulations adaptively.

Examples of emerging patterns:

• Neural networks used as fast surrogates for expensive physical components.
• ML‑guided mesh refinement, parameter exploration, and uncertainty quantification.
• Large‑scale training of scientific ML models on exascale resources.

AI workloads themselves can be exascale‑class, using massive clusters of accelerators for training large models.

What Exascale Means for You as a Beginner

Skills That Age Well in the Exascale Era

Even if you never directly run on a top exascale system, the same ideas appear in smaller clusters and cloud environments. Useful long‑term skills include:

• Thinking in parallel:
• Identifying concurrency in your problem.
• Decomposing work and data across many processing elements.
• Being aware of data movement:
• Minimizing communication and unnecessary memory traffic.
• Designing data structures with locality in mind.
• Writing performance‑portable code:
• Using abstractions and libraries rather than hard‑coding for a single architecture.
• Practicing good software engineering:
• Testing, version control, modular design, performance regression tests.

How to Prepare Practically

As you continue through this course and beyond:

• Pay attention to:
• Parallel programming models (MPI, OpenMP, GPU programming).
• Concepts like scalability, load balancing, and memory hierarchy.
• Experiment with:
• Small‑scale versions of exascale ideas: hybrid parallelism, task‑based parallelism, and simple mixed‑precision code.
• Follow developments in:
• Performance portability frameworks.
• Exascale software projects and open benchmarks.

The same principles that enable exascale computing will help you write efficient, robust HPC applications at any scale.