Energy consumption of HPC systems

Table of Contents

Understanding Energy Use in HPC

High‑performance computing systems consume significant electrical power. This chapter focuses on how and where that energy is used, how it is measured, and what concepts you need to understand before thinking about optimization or “green” strategies.

Where Energy Is Consumed in an HPC System

An HPC installation (often called a data center or machine room) has multiple layers of energy consumption:

1. Compute Hardware

These are the components you usually think of as “the cluster”:

CPUs and GPUs

Typically the largest single contributors to IT power.
Power draw can vary widely depending on utilization, frequency, and voltage.
Modern CPUs and GPUs have power caps and dynamic frequency scaling.

Memory (DRAM)

Consumes static power just being powered on, plus extra during heavy access.
In large-memory nodes, DRAM can be a substantial fraction of node power.

Storage (local SSDs/HDDs)

SSDs usually consume less power than spinning disks, especially at idle.
High‑performance distributed filesystems add many disks and controllers, which collectively draw considerable power.

Network/Interconnect

Network interface cards (NICs), InfiniBand adapters, switches, and routers.
High‑bandwidth, low‑latency networks can be significant energy consumers at the cluster scale.

2. Supporting Infrastructure (Non‑IT Load)

The “overhead” required to keep the IT hardware running:

Cooling systems

Computer Room Air Conditioners (CRAC) or air handlers.
Chillers, cooling towers, pumps, and fans.
In some modern systems, direct liquid cooling or rear‑door heat exchangers.

Power delivery

Uninterruptible Power Supplies (UPS).
Power distribution units (PDUs), transformers, cabling.
These add conversion and distribution losses.

Facility overhead

Lighting, building management systems, monitoring equipment, etc.

3. System‑Level vs Node‑Level Perspective

When talking about energy consumption, it’s useful to distinguish:

Node‑level

Power drawn by a single compute node (or GPU node) in watts.
Used to understand per‑job or per‑application behavior.

System‑level

Total power for all nodes plus networking, storage, and cooling.
Relevant for data center planning, electricity contracts, and sustainability metrics.

Power vs Energy: Basic Quantities

Two key physical concepts appear repeatedly in HPC energy discussions:

Power ($P$): rate of energy use, measured in watts (W).

1 W = 1 joule per second.
“This node draws 400 W under full load.”

Energy ($E$): total amount of work done or heat produced, measured in joules (J) or kilowatt‑hours (kWh).

1 kWh = 3.6 million J.
“This simulation consumed 50 kWh.”

Time relates the two:
$$
E = P \times t
$$
where $E$ is energy, $P$ is power, $t$ is time.

In HPC, you will see:

Instantaneous power: what the machine is drawing right now.
Average power over a period: e.g., average power during a job.
Total energy for a job: power integrated over runtime.

Typical Power Scales in HPC

Orders of magnitude (approximate) for context:

Single server CPU at load: 100–300 W.
One high‑end GPU: 250–700 W.
Fully loaded dual‑socket CPU node: 300–800 W.
GPU‑accelerated node (4–8 GPUs): 1–5 kW.
Small cluster (a few racks): tens of kW.
Large academic supercomputer: hundreds of kW to a few MW.
Top‑tier leadership‑class supercomputer: 10–30+ MW.

Electricity cost, emissions, and cooling requirements scale with this power draw and the number of hours the system runs.

Metrics Used to Describe Energy Efficiency

Several metrics are widely used to quantify and compare energy use.

1. PUE (Power Usage Effectiveness)

A facility‑level metric:
$$
\text{PUE} = \frac{\text{Total Facility Power}}{\text{IT Equipment Power}}
$$

Total Facility Power: everything the building uses (IT + cooling + power losses + lights, etc.).
IT Equipment Power: power used by servers, storage, and networks alone.

Interpretation:

PUE = 1.0 would mean all power goes to IT equipment, with zero overhead (practically impossible).
Typical traditional data centers: PUE ≈ 1.5–2.0.
Well‑optimized HPC facilities: PUE ≈ 1.1–1.3 or better.

Lower PUE ⇒ more of the electrical power is used for computation rather than overhead.

2. Energy to Solution

From an application / job point of view, the main quantity is:

Energy to solution: total energy consumed to complete a given scientific or engineering task.

If an application runs for time $t$ with average power $\bar{P}$, then:
$$
E_{\text{solution}} = \bar{P} \times t
$$

Two systems might show:

System A: short runtime but very high power.
System B: longer runtime but lower power.

The more sustainable system for that task is the one with lower $E_{\text{solution}}$, not necessarily the one with shorter runtime.

3. FLOPS per Watt

HPC commonly uses performance per watt as a hardware and system metric:

FLOPS/W: floating‑point operations per second per watt.
Used in benchmarks such as the Green500 list.

Higher FLOPS/W means higher computational throughput for the same power.

At the code level, you may also see:

Energy efficiency expressed as FLOPS per joule or operations per joule.

4. Utilization and “Energy Productivity”

For a cluster or machine room:

Utilization: fraction of time and capacity that is doing useful work.

A mostly idle system with high fixed background power is energy inefficient.

“Energy productivity” can be thought of as:

Useful work per unit of energy, over long periods.

High utilization and appropriate job sizing help prevent energy waste.

How Energy Is Measured in HPC Environments

You will encounter several types of measurements and tools:

1. Hardware-Level Sensors

Many components expose internal power sensors, such as:

CPU package power (e.g., via Intel RAPL interfaces).
GPU power (via vendor APIs: NVIDIA nvidia-smi, AMD tools).
Node‑level power measured by baseboard management controllers (BMC / IPMI).

These allow:

Per‑job energy estimation.
Fine‑grained profiling during application runs.

2. Rack and Facility-Level Measurement

Intelligent PDUs provide per‑outlet (per‑node) or per‑rack power readings.
Facility meters monitor:

Total building power.
Cooling system power.
Dedicated feeds to specific clusters.

These are necessary to compute PUE and to understand how close the facility operates to power and cooling limits.

3. Integration with Schedulers

Schedulers (like SLURM and others) can integrate with power measurement:

Attaching energy use information to jobs.
Enforcing node power caps or total cluster power budgets.
Enabling energy‑aware job scheduling (e.g., spreading or concentrating jobs to optimize cooling and power distribution).

Sources of Inefficiency in Energy Use

Energy waste in HPC systems comes from various technical and operational choices:

1. Idle and Low-Utilization Power

Hardware consumes non‑trivial power even when idle.
A lightly used system has:

High baseline (idle) power.
Little useful work done per kWh.

Long periods of low utilization increase energy per useful output.

2. Overprovisioning and Overclocking

Hardware run at maximum frequency/voltage might:

Increase power superlinearly with frequency.
Deliver modest performance gains but much higher energy use.

Overprovisioned capacity (more nodes than are commonly needed) increases idle overhead.

3. Poor Application Efficiency

Inefficient codes can waste energy by:

Doing unnecessary computation.
Waiting on I/O or communication (“stalled” cores still draw power).
Thrashing caches and memory, leading to more energy‑expensive memory accesses.

Better algorithms and better implementations can reduce energy to solution significantly.

4. Imbalanced System Design

Disproportionately powerful CPUs or GPUs paired with too little memory bandwidth or too slow of an interconnect.
Applications spend more time stalled, not fully using the expensive, power‑hungry components.

5. Inefficient Cooling and Power Distribution

Older cooling systems with low coefficient of performance.
Multiple stages of power conversion with high losses.
Poor airflow management leading to hot spots and overcooling.

Even if the IT equipment is efficient, inefficient facility infrastructure increases total energy consumption.

Energy Consumption Across the System Lifecycle

Energy considerations are not limited to runtime.

1. Manufacturing and Embodied Energy (High-Level View)

Producing chips, servers, and infrastructure consumes energy and resources.
From a lifecycle perspective, replacing hardware too frequently can be wasteful even if newer hardware is more energy‑efficient during operation.

2. Operational Phase

For large systems, most environmental impact typically comes from years of continuous operation.
Electricity source matters:

Carbon intensity can vary by country, region, and time of day.
Same kWh can correspond to different greenhouse gas emissions depending on the grid mix.

3. End-of-Life

Decommissioning, recycling, and reuse.
Extending useful life through repurposing for less demanding workloads can improve overall energy and resource utilization.

Trade-offs: Performance, Energy, and Cost

In HPC, there is usually a three‑way trade‑off:

Performance (time to solution): shorter runtimes are often scientifically or commercially valuable.
Energy (energy to solution): lower energy consumption is better for sustainability.
Cost (electricity, cooling, infrastructure): operators pay for power and the systems to deliver and remove it.

Examples of trade‑offs:

Running at a slightly lower CPU or GPU frequency may:

Reduce power noticeably.
Increase runtime only a little.
Decrease total energy to solution and cost.

Using more nodes may:

Decrease time to solution.
Increase or decrease energy to solution depending on scaling efficiency.

Real systems often choose an operating point that balances these aspects based on:

Scientific priorities.
Budget constraints.
Sustainability goals.

Why Energy Consumption Matters in HPC

Energy use is central to the future of high‑performance computing because:

Physical limits

Power and cooling capacity limit how large and fast systems can grow.
Exascale systems already push toward tens of megawatts.

Economic limits

Electricity is a major fraction of total cost of ownership (TCO).
Energy‑inefficient systems may be unaffordable to operate at full capacity.

Environmental and ethical considerations

Large HPC centers can have significant carbon footprints.
Responsible use of public or institutional resources requires attention to energy.

Understanding where energy is consumed and how it is quantified is the foundation for later discussions on: