The EUROPA consortium, formed by Domyn and Fraunhofer IAIS and Fraunhofer IIS, trained a 951.9-billion-parameter Mixture-of-Experts model on JUPITER, the EuroHPC exascale supercomputer hosted at Germany's Forschungszentrum Jülich, boosting throughput from 90 to approximately 1,300 tokens per second per GPU across 384 NVIDIA GH200 GPUs per data-parallel replica1.
The work was carried out under the EuroHPC Frontier AI Grand Challenge, which required the consortium to build an open European MoE model of at least 400 billion parameters, support all 24 official EU languages, and comply with the EU AI Act.
Architecture and hardware
The model uses 74 layers, a hidden size of 6,144, 384 routed experts with top-8 routing, expert hidden size of 1,792, a vocabulary of 256,000 tokens, and a pretrain sequence length of 65,536 tokens. Of the 951.9 billion total parameters, 32.8 billion are active per token. Each JUPITER Booster node contains four NVIDIA GH200 Grace Hopper Superchips, each pairing an NVIDIA Hopper GPU with 96 GB HBM3 and a 72-core Grace CPU with 120 GB LPDDR5X. Intra-node GPU communication runs over NVLink 4 at 150 GB/s per direction, while each node has four InfiniBand NDR200 adapters providing roughly 25 GB/s per GPU for inter-node traffic.
The training setup uses 384-way FSDP with four-way expert parallelism inside each node and no pipeline parallelism. The MoE token all-to-all is kept on NVLink because it is difficult to overlap. Total training scale spans 1,536 GPUs across four data-parallel replicas, with a batch size of 384 sequences of 65,536 tokens, approximately 25 million tokens per step.
Optimization path from 90 to 1,300 tokens/s
The throughput gains came in stages. Full activation checkpointing lifted throughput from 90 to 370 tokens per second per GPU at a 32,768-token sequence length. Offloading boundary activations to the Grace CPU after the forward pass and copying them back to the Hopper GPU one layer before backward recomputation raised throughput from 370 to 508 tokens per second per GPU, while extending the sequence length first to 49,152 and then to 65,536 tokens with Muon. At 65,536 tokens, boundary activations alone consume approximately 55 GiB per GPU.
Memory pressure was a separate bottleneck. After a software stack upgrade, the CUDA caching allocator began retrying allocations on every rank at roughly 70 percent memory utilization. Side streams held 27.3 GiB of HBM at peak, of which only 6.4 GiB was in use. Using unshard_async_op moved the all-gather setup onto the compute stream and let NCCL run on its internal stream. Peak reserved memory fell from 88.1 to 74.6 GiB, throughput rose to 1,210 tokens per second per GPU, and allocator retries dropped from 384 ranks to zero.
The optimizer itself was a constraint: with FSDP's CPU offload, Newton-Schulz iterations on 72 Arm cores took approximately 35 seconds per step. FlexShard let the team control which rank owns which parameters, sharding experts by whole expert so each rank owns complete expert matrices and runs Newton-Schulz iterations with no communication. The optimizer step fell from 41.7 seconds to 2.2 seconds, and throughput rose from 441 to 597 tokens per second per GPU. Muon is used for every weight matrix except the router and embeddings, covering approximately 99.5 percent of the parameters, with AdamW handling the rest.
Upstream contributions
The consortium upstreamed all fixes to PyTorch and TorchTitan, including a shared RoPE cache that skips the embedding re-gather in the backward pass, configurable seed-offset mesh axes, and a fix for CPU initialization resetting the RNG state. The offloading strategy uses TorchTitan's activation-checkpointing wrapper and hooks contributed by the authors.
ANALYSIS The throughput gains were achieved on publicly shared infrastructure rather than a hyperscaler's private cluster, testing the premise that sovereign compute can sustain frontier-scale training.