Scientific software frameworks

What are scientific software frameworks?

Scientific software frameworks are large, reusable software systems that provide:

• Core numerical capabilities (often built on BLAS/LAPACK/FFT libraries)
• Data structures for common scientific objects (vectors, matrices, meshes, grids, particles)
• Abstractions for parallelism and hardware (MPI, OpenMP, GPUs, etc.)
• Tools for building full applications (I/O, configuration, logging, validation)

They sit “above” basic numerical libraries: instead of just giving you a matrix-matrix multiply, they help structure an entire scientific code or simulation while hiding much of the low-level HPC complexity.

Typical use cases:

• PDE solvers and multi-physics simulations
• Large-scale optimization and inverse problems
• Molecular dynamics and materials science
• Climate, fluid dynamics, astrophysics, and more

You rarely write these frameworks yourself; you use existing ones and extend them with your own physics, models, or algorithms.

Why use frameworks instead of “just” libraries?

Using only basic libraries (BLAS, LAPACK, FFT, etc.) gives fine-grained control but quickly becomes unmanageable for complex applications. Frameworks help by:

• Raising the level of abstraction
• Work with “field on a mesh” or “distributed vector” instead of raw arrays and MPI calls.
• Managing parallelism and hardware
• Hide MPI communicators, rank bookkeeping, domain decomposition, and sometimes GPU offload.
• Enforcing structure
• Encourage modularity: separate physics, discretization, solvers, and I/O.
• Improving portability
• Run on laptops, clusters, or GPUs with minimal code changes.
• Leveraging community effort
• Benefit from years of optimization and testing by domain experts.

Frameworks are not just performance tools; they are also about software engineering and maintainability for large scientific codes.

Categories of scientific software frameworks in HPC

PDE and multi-physics simulation frameworks

These focus on solving partial differential equations on meshes or grids, often supporting multiple physics coupled together.

Typical features:

• Mesh and grid management (structured/unstructured, adaptive mesh refinement)
• Discretizations (finite element, finite volume, finite difference)
• Time-stepping schemes
• Parallel data distribution and load balancing
• Coupling of multiple physics components (e.g., fluid + structure + heat)

Representative examples (conceptually, not full tutorials):

• deal.II – finite element framework with strong MPI+threads support, interfaces to many solvers.
• FEniCS – high-level PDE framework using variational formulations; generates low-level code.
• Firedrake – similar philosophy to FEniCS with a focus on composability and advanced code generation.
• MFEM – lightweight finite element library supporting GPU backends and advanced discretizations.
• OpenFOAM – C++ framework for CFD, providing solvers and utilities for fluid and related problems.
• MOOSE – multi-physics framework for coupled PDEs, widely used in nuclear, geoscience, etc.

Using such frameworks, you typically:

1. Describe your PDEs and boundary conditions.
2. Choose discretizations and solvers (often from integrated libraries).
3. Configure problem parameters (in input files or scripts).
4. Run in parallel using the framework’s parallelization infrastructure.

Linear algebra and solvers as frameworks

Some “libraries” are actually full frameworks for scalable linear algebra, preconditioning, and nonlinear solvers. They:

• Provide distributed vectors/matrices abstracting away MPI details.
• Wrap underlying BLAS/LAPACK/ScaLAPACK/FFT libraries and more.
• Implement scalable Krylov solvers, multigrid, nonlinear solvers, and time integrators.
• Integrate with PDE and multi-physics frameworks.

Key examples:

• PETSc – a very widely used framework for linear, nonlinear, and time-dependent problems. It offers:
• Distributed data structures (Vec, Mat)
• Krylov solvers (KSP), nonlinear solvers (SNES), time-stepping (TS)
• Interfaces to many external packages (hypre, MUMPS, SuperLU, etc.)
• Trilinos – a large collection of interoperable packages for:
• Distributed linear algebra (Epetra, Tpetra)
• Solvers and preconditioners (Belos, Ifpack, ML, MueLu)
• Nonlinear and transient solvers
• Embedded UQ, optimization, and more

These are “frameworks” in the sense that:

• You embed them into your application.
• They manage much of the parallel communication and performance tuning for solvers.
• They often define the architecture of your code (e.g., how you store and assemble systems).

Domain-specific simulation frameworks

Some frameworks target particular scientific domains, providing much more than generic PDE or linear algebra tools:

• Predefined models (e.g., standard force fields in molecular dynamics)
• Specialized algorithms
• Ready-to-use workflows and analysis tools

Examples (again conceptually):

• LAMMPS, GROMACS, NAMD – molecular dynamics frameworks
• WRF, ICON, CESM – weather and climate models
• FLASH, Enzo, RAMSES – astrophysical simulation codes
• CP2K, Quantum ESPRESSO, VASP – electronic structure and quantum chemistry packages

These are often used as end-user applications, but architecturally they are large frameworks:

• You configure simulations via input files or scripts.
• You can often extend them with custom potentials, models, or analysis modules.
• They are tightly coupled to HPC environments and rely on underlying numerical libraries.

How frameworks build on numerical libraries and software stacks

In a typical HPC stack:

• Low level: BLAS, LAPACK, ScaLAPACK, FFT libraries, vendor math libraries (MKL, AOCL, etc.)
• Middle level: PETSc, Trilinos, hypre, domain-decomposition and multigrid libraries
• High level (this chapter): PDE frameworks, multi-physics frameworks, domain-specific codes

Scientific software frameworks:

• Link against optimized math libraries provided by the HPC system.
• Use environment modules and software stacks so that:
• You load the right compiler/MPI/numerical libraries before building or running.
• You can select CPU-only vs. GPU-enabled builds.
• Often offer multiple backends:
• Different sparse matrix formats
• Different solvers / preconditioners
• Different parallel paradigms (MPI-only, MPI+threads, MPI+GPU, etc.)

Understanding that layering is crucial: if performance is poor, you must ensure:

• The framework is using the optimized libraries on the target machine.
• The build configuration matches the hardware (e.g., GPU support enabled).

Abstractions and data structures in frameworks

Frameworks typically provide specialized abstractions that differ from “raw arrays”:

• Distributed vectors and matrices
• Global indexes vs. local ownership
• Ghost/halo regions for stencil operations
• Mesh / grid / topology objects
• Elements, faces, edges, nodes
• Connectivity and neighborhood information
• Fields and function spaces
• Scalar/vector/tensor fields associated with mesh entities
• Function spaces for finite element or spectral methods
• Operator interfaces
• Abstract matrices or operators: user provides “apply” routines instead of explicit matrices.
• Facilitates matrix-free methods and GPU acceleration.

These abstractions:

• Encapsulate the parallel layout.
• Allow the same user code to run on different architectures.
• Support advanced algorithms (e.g., multigrid, AMR) without exposing every low-level detail.

Parallelism and accelerators in frameworks

Modern frameworks are designed with parallelism as a first-class concern:

• MPI-based distributed memory
• Domain decomposition is often automatic or minimally configured.
• Communication patterns (halo exchanges, global reductions) are pre-implemented.
• Shared-memory parallelism
• Internal use of OpenMP, threads, or task-based runtimes.
• Thread-safety and data partitioning are handled in the framework.
• GPU and accelerator support
• Backends using CUDA, HIP, SYCL, OpenACC, or Kokkos/Raja-like abstraction layers.
• Data structures that can live on GPU memory with automatic transfers when needed.
• Kernel abstractions so you don’t write low-level CUDA for common operations.

From the user’s perspective, you may:

• Enable GPU support at configure/build time.
• Choose GPU-aware data structures or solvers in input files.
• Possibly modify small portions of your code to respect framework guidelines for GPU correctness.

Workflow: building applications using frameworks

Using a scientific software framework usually follows a pattern:

1. Install or load the framework stack
• Use your cluster’s module system:
• Load compiler, MPI, and math libraries.
• Load the framework module (e.g., module load petsc or module load fenics).
2. Configure your project
• Use CMake, Make, or Python packaging to find the framework and link against it.
• Ensure your build uses the same compilers/MPI as the framework.
3. Develop your application components
• Implement:
• Problem definition (geometry, mesh)
• Physics/model equations
• Boundary and initial conditions
• Data assimilation or optimization logic (if needed)
• Use framework-provided APIs and data structures.
4. Select numerical methods and solvers
• Choose discretizations, solvers, and preconditioners via:
• Input files
• Command-line options
• Configuration objects
5. Run on the HPC system
• Use job scripts and the scheduler to request resources.
• The framework handles most of the parallel low-level details.
6. Post-process and analyze
• Use built-in I/O formats and tools (e.g., VTK, HDF5 output).
• Possibly integrate with Python, ParaView, VisIt, or domain-specific tools.

A key advantage is flexibility: many design choices become runtime options rather than fixed in code, enabling rapid experimentation.

Choosing an appropriate framework

When selecting a scientific software framework for an HPC project, consider:

• Scientific scope
• Does it support your type of equations (elliptic, parabolic, hyperbolic)?
• Does it have models for your domain (e.g., CFD, structural mechanics, electromagnetics)?
• Programming language
• C, C++, Fortran, Python interfaces, or mixed?
• Does it fit your team’s skills?
• Parallel and hardware support
• Scales to the core counts and node counts you expect?
• Supports GPUs or other accelerators on your target system?
• Integration with existing tools
• Works with the solvers and libraries you already use?
• Supports your I/O formats and analysis workflows?
• Community and sustainability
• Active development and user community?
• Documentation, tutorials, examples?
• Long-term maintenance and support?

On large HPC systems, it is often practical to:

• Start with frameworks that are already installed and supported by the center.
• Use provided examples from the framework as a template for your own codes.

Trade-offs and limitations

While powerful, scientific software frameworks come with trade-offs:

• Learning curve
• Large APIs and complex abstractions take time to understand.
• Flexibility vs. constraints
• You must work within the framework’s data structures and workflows.
• Exotic algorithms or data layouts may be hard to implement.
• Performance transparency
• Much performance-critical behavior is hidden.
• Tuning may require understanding both the framework and underlying libraries.
• Version and dependency complexity
• Large frameworks depend on many third-party packages.
• Different systems or projects may require different versions.

Understanding these trade-offs helps you decide when to adopt a framework vs. building lighter-weight, custom solutions on top of basic libraries.

Getting started with scientific software frameworks

For absolute beginners approaching HPC:

1. Start with high-level interfaces where possible
• Python-based or DSL-based frameworks can reduce boilerplate.
2. Study official examples
• Frameworks usually provide minimal examples: “hello world” PDE problem, simple solver setups.
3. Use the HPC center’s documentation
• Many centers provide step-by-step guides to run specific frameworks on their clusters.
4. Incrementally extend examples
• Change boundary conditions, materials, parameters.
• Add a new equation or coupling.
5. Profile and scale slowly
• Begin on few cores/nodes; once correct, test larger scales.
• Use the framework’s logging and diagnostic tools to understand performance.

Over time, you will learn both the framework concepts and how they relate to the broader numerical libraries and software stacks that underpin HPC applications.