llm-d released version 1.0 on 20 October, giving operators a stable configuration API for distributed model serving. Earlier versions already developed its routing and caching mechanisms; this release concentrates on operating a shared service.[1]
The flow-control documentation describes holding requests at the gateway until backend capacity is available. This leaves the scheduler able to choose their destination later, instead of trapping them in an overloaded server’s local queue.[2]
Selection has three stages: choose the highest traffic priority, choose a customer or workload within that band, then choose a request within its queue. Fairness therefore operates within a priority class; it does not give every request equal precedence.[2]
A service could put live chat above an overnight batch, then share turns among customers within the chat tier. That is a choice about service commitments as well as computing.[2]
The queues live in memory. A graceful shutdown returns a retryable HTTP 503 response; an abrupt crash can drop connections. The caller still needs to handle interruption.[2]
For upgrades, the release’s DisaggregatedSet coordinates the servers that process prompts and generate answers. Revision-aware routing waits for compatible pairs to be ready. Operators must opt into multi-tenant policy; the defaults preserve first-come-first-served behaviour.[1]
Sources & further reading
- llm-d v1.0: State-of-the-art inference, ready for Day 2 llm-d · 20 Oct 2026 · Accessed 21 Oct 2026
- Flow Control llm-d · Accessed 21 Oct 2026
Report an error or suggest a correction