vLLM Introduces Push-Mode NIXL KV Transfer to Optimize TTFT

The AWS Machine Learning Blog published a post introducing a new push-mode NIXL KV connector for vLLM's disaggregated inference. The design moves registration and transfer preparation on the decode side off the critical path to reduce the time-to-first-token latency between prefill and decode. In a 1P1D (one prefill, one decode) deployment on Amazon EC2 P5en instances with EFA, official test data shows that using the Qwen3-32B model at 8 QPS with 16k input length, TTFT dropped from about 1744 ms to about 540 ms, roughly a 3x improvement, and TPOT also improved by up to 30%. The feature was contributed to vLLM by the AWS team and is aimed at users adopting disaggregated inference (prefill/disagg separation); it can be enabled via the push mode of the NIXL KV connector. The optimization is now available in vLLM, and all publicly reported validation data comes from the above AWS test environment; no comparison data from other independent environments has been reported.

AI overview of reporting · Updated 48 min前

Not enough continuous observations to show a trend yet.