AWS Shares MoE Fused Kernel Optimization Practices
Understand the story
The AWS team implemented kernel engineering optimizations for Fused MoE models on Amazon SageMaker AI. One technique, called Swap-AB, flips the operand order of matrix multiplications before entering PTX assembly to fit NVIDIA Hopper architecture WGMMA instructions' M=64 dimension requirement. According to the report, the optimization delivers a 2–4% end-to-end throughput improvement on H200. AWS has upstreamed this optimization into vLLM 0.24.0 and later.
AI overview of reporting · Updated 5 days ago
Reporting timeline
Explore different aspects of the story through its reporting.
- AWS Machine Learning BlogMixture-of-Experts kernel engineering on Amazon SageMaker AI
The AWS team implemented kernel optimizations for Fused MoE models on Amazon SageMaker AI; the Swap-AB trick flips matrix multiplication before entering PTX assembly to fit the M=64 dimension requirement of NVIDIA Hopper's WGMMA instructions, delivering a 2–4% end-to-end throughput gain on H200, and has been upstreamed into vLLM 0.24.0 and later.
Story heat trend
Current heat3·Comparable peak10（10/14 00:00）·Comparable change over 24 hours -50%
The trend compares the same continuously observed sources, which may cover fewer sources than the current heat score. Hover or select the chart to inspect each hour; use the left and right arrow keys to navigate.