llm-d 1.0 adds traffic priorities and upgrade controls for AI serving

Two model requests may contain the same long instructions and documents. If the second reaches a server that retains the processed prefix in its key-value cache, it can reuse some work. If it reaches another server, that computation may be repeated. Distributed inference software must balance that opportunity against a different problem: the busiest server may already have a queue.[1]

llm-d, which coordinates inference engines such as vLLM and SGLang, released version 1.0 on 20 October. The project is now a Cloud Native Computing Foundation sandbox project, with more than 500 contributors from over 25 organisations. The release adds operating controls around techniques such as cache-aware routing and separate prefill and decode workers.[1][2]

Sharing capacity and replacing workers

Priority-aware flow control lets operators distinguish interactive traffic from background work and allocate shared capacity across tenants. It is an opt-in capability: the default remains first-come, first-served. A deployment does not acquire a suitable priority policy merely by installing the release.[1]

Session-aware routing seeks to keep a conversation near useful cached state until that worker is saturated. High-availability routing adds a hot standby, while coordinated rolling upgrades manage separate prefill workers, which process input, and decode workers, which generate output. A stable versioned API also gives operators a clearer integration target.[1]

These controls address a continuing service rather than a single fast request. Cache locality can reduce recomputation, but routing everything to one warm worker can increase waiting time. The useful outcome is a policy that balances those costs for the service’s actual traffic.[1]

Putting the throughput example in context

IBM’s 1.0 announcement cites an earlier GLM-5.2 demonstration. The underlying 8 September report used 544 H100 GPUs and reached about 6.6 million output tokens per minute at a peak of 3,000 concurrent coding agents. That is evidence from a specified deployment, not a measurement of the effect of the new 1.0 features.[2][3]

IBM estimated per-token costs five to ten times lower than commercial API pricing for its selected input-heavy, cache-friendly workload and assumed cloud rental rates. That comparison depends on traffic, hardware utilisation, token mix and API prices; it does not establish equivalent answer quality or every operator’s full service cost.[3]

The release also names production uses: DigitalOcean’s serverless inference and managed agents, and JetBrains’ Junie Lite coding service. These put the project in a practical operating context. Version 1.0’s contribution is the machinery around those workloads—how requests share capacity and how a service keeps running while its components change.[2]

Report an error or suggest a correction