Efficient resource usage

Why Efficient Resource Usage Matters

In HPC, “efficient resource usage” is about getting the maximum useful science or engineering done per unit of:

• CPU/GPU time
• Memory and storage
• Energy
• Human time (queue waiting, debugging, etc.)

Inefficient use of resources wastes money and energy, increases queue times for everyone, and can reduce overall scientific output. This chapter focuses on concrete practices users can adopt to use HPC resources more effectively and responsibly.

Matching Resources to Your Workload

Right-sizing jobs

Request only what you actually need, but enough that your job runs effectively.

Key dimensions:

• Number of CPU cores:
• Avoid running single-threaded codes on many cores unless using job arrays or multiple tasks.
• For multi-threaded or MPI programs, measure how performance scales and choose a point with good efficiency, not just the maximum cores allowed.
• Memory per node / per task:
• Estimate memory needs from smaller test runs.
• Add a modest safety margin (e.g., 20–30%), not a factor of 10 “just in case.”
• Over-requesting memory can leave CPUs idle because the scheduler must reserve memory that you don’t use.
• GPU resources:
• Request a GPU only if your code uses it.
• Ensure your job is actually running computations on the GPU (profilers and logs help).
• Wall-clock time:
• Use short test jobs to estimate runtime.
• Request realistic time + some buffer (e.g., 20–30%).
• Overestimating by a large factor keeps your job in longer queues and can reduce backfilling opportunities for the scheduler.

Job arrays vs. big monolithic jobs

Many workflows involve many independent runs (parameter sweeps, Monte Carlo, etc.).

• Use job arrays to:
• Map one parameter set to one array task.
• Allow the scheduler to place tasks flexibly across idle resources.
• Avoid:
• Packing many unrelated tasks into a single job with manual scripting unless the system specifically encourages that pattern.

Improving Utilization Within a Job

Avoid idle hardware

Once you’ve been granted nodes, you pay (in allocation and energy) for the entire reservation, whether you use the resources or not.

• Multi-threaded programs:
• Set OMP_NUM_THREADS (or equivalent) to match the cores you requested.
• Avoid oversubscription (more threads than cores), which can hurt performance and waste energy.
• MPI programs:
• Choose --ntasks, --nodes, --ntasks-per-node, etc. so that cores per process match your code’s design.
• Hybrid MPI+OpenMP:
• Ensure MPI ranks × threads per rank ≈ cores per node requested.
• Avoid leaving many cores unused on each node.

Balancing performance and efficiency

Maximum speed is not always maximum efficiency.

• Diminishing returns in scaling:
• Speedup may flatten as you add more cores.
• Extra cores might yield very small time savings but large extra resource and energy cost.
• Prefer high parallel efficiency:
• Define efficiency $E = \frac{\text{speedup}}{\text{number of cores}}$.
• Choose a configuration where $E$ is still reasonably high, instead of blindly using the maximum core count allowed.

Queue-aware and Cluster-aware Behavior

Choosing the right partition/queue

Clusters often have multiple partitions (short, long, GPU, bigmem, etc.).

• Use short queues for test runs and quick jobs; this improves turnaround for everyone and for you.
• Use big-memory or GPU partitions only when your job genuinely needs them.
• If possible, schedule long, non-urgent jobs in low-priority / scavenger queues that use idle capacity.

Backfilling-friendly jobs

Schedulers can “backfill” small or short jobs into gaps.

• Submit some small, short jobs when appropriate; they fit into gaps and keep the cluster busier without delaying large jobs.
• Avoid requesting “just under” the maximum wall time for routine runs; that blocks backfilling.

Checkpointing and Failure-Aware Usage

Why checkpointing helps resource efficiency

Jobs sometimes fail: bugs, node failures, time limits, power events. If your program has no checkpointing:

• All compute time up to the failure is effectively wasted.
• Restarting from scratch amplifies energy and resource waste.

Implement or use:

• Periodic checkpoints:
• Save state at safe intervals matched to your runtime and failure likelihood.
• Use formats that are efficient for both writing and reading.
• Restart capability:
• Ensure your code can resume from a checkpoint reliably.

Choosing checkpoint frequency

Too frequent:

• Wastes time and I/O bandwidth.
• Produces excessive files.

Too infrequent:

• Loses too much work on failure.

Aim for a compromise, often guided by:

• Typical runtime
• Checkpoint cost (time and data size)
• System reliability and time limits

Efficient Use of Storage and I/O

Using the right filesystem

Clusters may have:

• Fast parallel filesystems (for active jobs)
• Home directories (for code, configs)
• Scratch or temporary storage (large working data)
• Archival systems (backups, long-term storage)

Efficient usage patterns:

• Put large, high-I/O data on scratch or parallel filesystems, not in home.
• Move old, rarely used data to archive or delete it.
• Clean up scratch areas after a project to avoid consuming quota and backups.

Reducing I/O overhead

I/O can dominate runtime and energy.

• Aggregate small writes:
• Buffer output and write fewer, larger chunks.
• Avoid writing logs every iteration unless necessary.
• Avoid unnecessary checkpoints:
• Store only what is needed to restart or analyze results.
• Use compressed or binary formats when appropriate:
• Less data moved = less energy and time.

Code and Workflow Practices that Save Resources

Start small, then scale

Before large production runs:

• Debug on:
• Small input sizes.
• Fewer cores/nodes.
• Validate correctness and stability:
• Avoid wasting massive allocations on trivial bugs.
• Once stable and validated, increase problem size and resources.

Profiling and basic optimization

Even modest performance tuning can significantly cut resource usage:

• Use profiling tools to:
• Identify hot spots and bottlenecks.
• Avoid pathologically slow algorithms or unnecessary work.
• Simple optimizations:
• Avoid redundant calculations inside loops.
• Use appropriate numerical libraries (BLAS, FFT, etc.).
• Ensure vectorization where possible.

This reduces runtime per job, and by extension, total resource and energy consumption over multiple runs.

Avoiding unnecessary runs

• Plan parameter studies:
• Use coarse sweeps first.
• Narrow down to promising regions before large-scale fine sweeps.
• Reuse intermediate results:
• Where possible, design workflows that avoid recomputing expensive steps.

Energy-Aware Usage Patterns

Choosing when and how to run

Some systems or centers:

• Have energy-aware schedulers or power caps.
• Provide lower-priority queues that run opportunistically when renewable energy is abundant.

When options exist:

• Select energy-aware or “green” queues for non-urgent workloads.
• Consider slightly lower clock speeds or fewer cores if they give better energy-per-result, not just fastest time.

Monitoring energy usage (when available)

If tools or job summaries expose energy metrics (e.g., Joules, average power):

• Compare configurations:
• Same job with different core counts or node types.
• Look at energy-per-simulation or energy-per-time-step.
• Prefer configurations that balance:
• Acceptable runtime.
• Lower total energy consumption.

Cooperative Behavior in Shared Environments

Respecting fair-share and allocations

Allocations and fair-share policies try to distribute capacity equitably.

As a user:

• Avoid monopolizing:
• Don’t submit thousands of large jobs at once without coordination.
• Coordinate with your group to avoid bursts that overwhelm queues.
• Throttle non-urgent jobs:
• Limit the number of concurrent jobs when the system is busy.

Cleaning up and documenting

• Delete:
• Temporary files.
• Redundant data and checkpoints after they’re no longer needed.
• Document:
• Your job scripts and workflows.
• Parameters used for production runs.

This reduces storage pressure and makes it easier to reuse or reproduce work without re-running unnecessary computations.

Practical Checklist for Efficient Use

Before submitting a job:

• Have I:
• Tested on a small problem?
• Measured rough runtime and memory use?
• Requested only the needed CPUs, GPUs, memory, and time with a reasonable buffer?
• Chosen the correct partition/queue?

During development:

• Am I:
• Using job arrays for many independent tasks?
• Implementing or using checkpoint/restart?
• Minimizing unnecessary I/O and logging?

For production runs:

• Have I:
• Checked scaling to choose an efficient core count?
• Ensured full use of allocated resources (no idle cores/GPUs)?
• Planned data management (what to keep, move, or delete)?

Using this mindset consistently converts raw HPC power into scientifically useful work with less waste, lower energy consumption, and better access for everyone sharing the system.