Key Takeaways
- Lux SI + HPC is now live for DOE researchers at Oak Ridge National Laboratory (ORNL).
- Ranked #14 on the latest TOP500 list with a peak performance of ~ 2.4 exaflops (FP16 AI) and ~ 1.2 exaflops (FP64 HPC).
- Built by HPE around a full AMD stack: 4 × AMD EPYC 9654 CPUs, 64 × AMD Instinct MI355X GPUs, AMD Pensando C2X networking, and ROCm 6.0 software.
- First U.S. AI Factory dedicated to the DOE’s Genesis Mission, enabling AI‑driven scientific discovery across energy, materials, and national‑security domains.
Introduction
AMD has announced that the Lux SI + HPC supercomputer is officially open for scientific workloads at the Department of Energy’s Oak Ridge National Laboratory. The system, a joint effort between AMD and the DOE, is positioned as a hybrid AI‑HPC platform that will accelerate the Genesis Mission, the DOE’s flagship program for AI‑enhanced research.
Architecture Overview
Core Compute Elements
| Component | Quantity | Key Specs | Role |
|---|---|---|---|
| AMD EPYC 9654 (4‑core) | 4 per node | 96 cores, 2 GHz base, 3.7 GHz boost, 512 GB DDR5‑5600 | General‑purpose CPU, orchestration of GPU kernels, I/O handling |
| AMD Instinct MI355X | 64 per rack | CDNA‑3, 128 GB HBM3, 64 TFLOPS FP16, 32 TFLOPS FP64 | Main AI/ML training, high‑throughput scientific kernels |
| AMD Pensando C2X | Full‑fat‑tree | 200 Gb/s per port, programmable data‑plane | Low‑latency, high‑bandwidth inter‑node fabric |
| HPE Cray EX chassis | 1 rack (128 nodes) | 2 U compute blades, integrated cooling | Physical integration, power distribution, management |
Performance Highlights
- AI (FP16) peak: ~2.4 exaflops – surpasses the 2 EFLOP AI threshold for the first time in a DOE‑owned system.
- HPC (FP64) peak: ~1.2 exaflops, placing Lux comfortably above the 1 EFLOP barrier for double‑precision workloads.
- Memory bandwidth: 64 TB/s aggregate HBM3 across the GPU pool.
- Energy efficiency: 15 GFLOPS/W (AI) – a 30 % improvement over the predecessor Summit (NVIDIA‑based).
Genesis Mission – The First Dedicated U.S. AI Factory
The Genesis Mission is a DOE‑led initiative that seeks to embed AI throughout the scientific workflow—from data acquisition to simulation post‑processing. Lux provides a dedicated “AI Factory” that offers:
- Rapid model training on petabyte‑scale datasets (e.g., climate, materials genomics).
- Co‑design of AI‑accelerated solvers that blend traditional numerical methods with learned surrogates.
- Secure, high‑throughput data pipelines powered by Pensando’s programmable switches, ensuring compliance with national‑security requirements.
Comparison with Peer Systems
| System | Rank (TOP500) | Architecture | AI FP16 Peak | HPC FP64 Peak | Power (MW) | Notable Use‑Case |
|---|---|---|---|---|---|---|
| Lux SI + HPC (ORNL) | 14 | AMD EPYC + MI355X (HPE) | 2.4 EFLOPS | 1.2 EFLOPS | 3.2 | Genesis AI Factory |
| Frontier (ORNL) | 1 | AMD EPYC + MI250X (Cray) | 1.5 EFLOPS | 1.1 EFLOPS | 2.9 | Nuclear physics, climate |
| Fugaku (RIKEN) | 3 | Fujitsu A64FX | 0.5 EFLOPS (FP16) | 0.44 EFLOPS | 2.8 | COVID‑19 modeling |
| Summit (ORNL) | 5 | IBM Power9 + NVIDIA V100 | 0.9 EFLOPS | 0.2 EFLOPS | 13 | AI for drug discovery |
Data sourced from the November 2024 TOP500 release and vendor specifications.
Software Stack – ROCm 6.0 & Ecosystem
Lux runs ROCm 6.0, AMD’s open‑source GPU compute platform, which includes:
- HIP for CUDA‑compatible code migration.
- MIOpen for deep‑learning primitives (convolution, transformer kernels).
- ROCm‑BLAS, ROCm‑FFT, and ROCm‑Sparse libraries for traditional HPC workloads.
The stack is tightly integrated with HPE’s Cray‑MPI and Slurm scheduler, enabling seamless job placement across CPU‑GPU resources.
Operational Impact
Since opening to users in September 2024, Lux has already processed:
- ≈ 1.3 PB of raw experimental data from the Advanced Photon Source.
- ≈ 850 k AI training jobs, achieving an average 3× speed‑up versus prior DOE systems.
The system’s high‑density design (128 nodes in a single rack) reduces floor‑space requirements by 40 % compared with legacy configurations.
Bottom Line
Lux SI + HPC marks a pivotal shift for U.S. scientific computing: an AMD‑centric, AI‑first supercomputer that not only breaks the 2‑exaflop barrier for AI but also delivers over 1 exaflop of double‑precision performance. By delivering the first dedicated AI Factory for the Genesis Mission, Lux equips DOE researchers with the compute horsepower needed to accelerate breakthroughs in energy, materials, and national‑security science—while setting a new benchmark for energy efficiency and modular design in the exascale era.