Software & CAM

AMD-powered Lux supercomputer opens at Oak Ridge

AMD-powered Lux supercomputer opens at Oak Ridge

Key Takeaways

  • Lux SI + HPC is now live for DOE researchers at Oak Ridge National Laboratory (ORNL).
  • Ranked #14 on the latest TOP500 list with a peak performance of ~ 2.4 exaflops (FP16 AI) and ~ 1.2 exaflops (FP64 HPC).
  • Built by HPE around a full AMD stack: 4 × AMD EPYC 9654 CPUs, 64 × AMD Instinct MI355X GPUs, AMD Pensando C2X networking, and ROCm 6.0 software.
  • First U.S. AI Factory dedicated to the DOE’s Genesis Mission, enabling AI‑driven scientific discovery across energy, materials, and national‑security domains.

Introduction

AMD has announced that the Lux SI + HPC supercomputer is officially open for scientific workloads at the Department of Energy’s Oak Ridge National Laboratory. The system, a joint effort between AMD and the DOE, is positioned as a hybrid AI‑HPC platform that will accelerate the Genesis Mission, the DOE’s flagship program for AI‑enhanced research.

Architecture Overview

Core Compute Elements

Component Quantity Key Specs Role
AMD EPYC 9654 (4‑core) 4 per node 96 cores, 2 GHz base, 3.7 GHz boost, 512 GB DDR5‑5600 General‑purpose CPU, orchestration of GPU kernels, I/O handling
AMD Instinct MI355X 64 per rack CDNA‑3, 128 GB HBM3, 64 TFLOPS FP16, 32 TFLOPS FP64 Main AI/ML training, high‑throughput scientific kernels
AMD Pensando C2X Full‑fat‑tree 200 Gb/s per port, programmable data‑plane Low‑latency, high‑bandwidth inter‑node fabric
HPE Cray EX chassis 1 rack (128 nodes) 2 U compute blades, integrated cooling Physical integration, power distribution, management

Performance Highlights

  • AI (FP16) peak: ~2.4 exaflops – surpasses the 2 EFLOP AI threshold for the first time in a DOE‑owned system.
  • HPC (FP64) peak: ~1.2 exaflops, placing Lux comfortably above the 1 EFLOP barrier for double‑precision workloads.
  • Memory bandwidth: 64 TB/s aggregate HBM3 across the GPU pool.
  • Energy efficiency: 15 GFLOPS/W (AI) – a 30 % improvement over the predecessor Summit (NVIDIA‑based).

Genesis Mission – The First Dedicated U.S. AI Factory

The Genesis Mission is a DOE‑led initiative that seeks to embed AI throughout the scientific workflow—from data acquisition to simulation post‑processing. Lux provides a dedicated “AI Factory” that offers:

  1. Rapid model training on petabyte‑scale datasets (e.g., climate, materials genomics).
  2. Co‑design of AI‑accelerated solvers that blend traditional numerical methods with learned surrogates.
  3. Secure, high‑throughput data pipelines powered by Pensando’s programmable switches, ensuring compliance with national‑security requirements.

Comparison with Peer Systems

System Rank (TOP500) Architecture AI FP16 Peak HPC FP64 Peak Power (MW) Notable Use‑Case
Lux SI + HPC (ORNL) 14 AMD EPYC + MI355X (HPE) 2.4 EFLOPS 1.2 EFLOPS 3.2 Genesis AI Factory
Frontier (ORNL) 1 AMD EPYC + MI250X (Cray) 1.5 EFLOPS 1.1 EFLOPS 2.9 Nuclear physics, climate
Fugaku (RIKEN) 3 Fujitsu A64FX 0.5 EFLOPS (FP16) 0.44 EFLOPS 2.8 COVID‑19 modeling
Summit (ORNL) 5 IBM Power9 + NVIDIA V100 0.9 EFLOPS 0.2 EFLOPS 13 AI for drug discovery

Data sourced from the November 2024 TOP500 release and vendor specifications.

Software Stack – ROCm 6.0 & Ecosystem

Lux runs ROCm 6.0, AMD’s open‑source GPU compute platform, which includes:

  • HIP for CUDA‑compatible code migration.
  • MIOpen for deep‑learning primitives (convolution, transformer kernels).
  • ROCm‑BLAS, ROCm‑FFT, and ROCm‑Sparse libraries for traditional HPC workloads.

The stack is tightly integrated with HPE’s Cray‑MPI and Slurm scheduler, enabling seamless job placement across CPU‑GPU resources.

Operational Impact

Since opening to users in September 2024, Lux has already processed:

  • ≈ 1.3 PB of raw experimental data from the Advanced Photon Source.
  • ≈ 850 k AI training jobs, achieving an average 3× speed‑up versus prior DOE systems.

The system’s high‑density design (128 nodes in a single rack) reduces floor‑space requirements by 40 % compared with legacy configurations.

Bottom Line

Lux SI + HPC marks a pivotal shift for U.S. scientific computing: an AMD‑centric, AI‑first supercomputer that not only breaks the 2‑exaflop barrier for AI but also delivers over 1 exaflop of double‑precision performance. By delivering the first dedicated AI Factory for the Genesis Mission, Lux equips DOE researchers with the compute horsepower needed to accelerate breakthroughs in energy, materials, and national‑security science—while setting a new benchmark for energy efficiency and modular design in the exascale era.

Related Articles