Meta has open-sourced an MXFP8 extension of FlashAttention-4 that delivers end-to-end block-scaled attention on Nvidia Corp. Blackwell GPUs, reaching 2.85 PF/s on forward passes and 2 PF/s on backward passes for LLM shapes1. On Meta's internal shapes, FA4 MX8 achieves 2.54 PF/s forward and 1.58 PF/s backward, yielding up to 1.6× and 1.52× gains over BF16. Meta states the module is already used internally for GEM training, describing it as one of the first MXFP8 FA4 implementations deployed in production training workloads. The code is available in Meta's Ads Model Kernel Library on GitHub.
Meta Open-Sources MXFP8 FlashAttention-4 for Blackwell, Hits 2.85 PF/s
Meta released an open-source MXFP8 extension of FlashAttention-4 for Nvidia Blackwell GPUs, reaching 2.85 PF/s forward and up to 1.6× gains over BF16.