An AWS engineering post describes a push-based connector for the NVIDIA Inference Xfer Library (NIXL) designed to improve disaggregated inference performance in vLLM. In this architecture, prefill and decode run on separate engine instances, requiring the key-value (KV) cache generated during prefill to be transferred before token generation begins.

The default NIXL workflow is pull-based: after prefill, the router sends metadata to the decode instance, which allocates destination blocks, prepares the transfer, and retrieves the KV data. AWS says these steps can extend time to first token (TTFT). The push-based design registers destination blocks on the decode side while prefill is still running. Once both registration and prefill are complete, a background thread on the prefill instance prepares and submits an NIXL write directly to the registered memory.

According to AWS, this approach moves transfer preparation away from the decode engine’s latency-sensitive execution loop. In a benchmark using the Qwen3-32B model on a 1P1D deployment with Amazon EC2 P5en instances and AWS Elastic Fabric Adapter, push mode delivered up to three times better TTFT and up to 30% better time per output token than pull mode. The test disabled prefix caching and varied input lengths from 1,000 to 16,000 tokens and request rates from one to eight queries per second. AWS notes that results depend on workload and configuration.

The post also identifies a tradeoff: pull mode allows the router to choose a decode instance after prefill, while push mode selects that instance earlier to enable overlapping setup.