A chatbot request arrives with a problem that is invisible to the server scheduling it. The system knows the prompt, the available processors and perhaps the customer's latency target. It does not know whether the answer will take ten tokens or ten thousand. That uncertainty is costly because a short response and a long one compete for memory and computational time in different ways. A new paper proposes an AI scheduler called URSA that plans around a distribution of possible output lengths instead of pretending it already knows the final number.[1]
The Scientific Reports study evaluates URSA in trace-driven simulations. Its primary BurstGPT test, modelling a 32-GPU heterogeneous cluster, achieved 90.4 percent service-level-objective attainment with URSA-CQL, compared with 71.3 percent for first-come-first-served and 84.1 percent for a learning-to-rank scheduler. These figures are from that test, not an average across all the datasets used in the paper.[1]
Why output length matters twice
BurstGPT simulation: service and tail latency
| Scheduler | Requests meeting SLO | P99 first-token latency |
|---|---|---|
| First-come-first-served | 71.3% | 7.9s |
| Learning-to-rank | 84.1% | 3.1s |
| URSA-CQL | 90.4% | 2.4s |
Inference involves a prefill phase, in which the system processes the incoming prompt, and a decode phase, in which it generates output token by token. These phases use resources differently. During generation, a model typically stores a key–value cache containing attention state for the active sequence. Longer outputs consume more of that memory. If a scheduler underestimates demand and packs too many sessions onto a GPU, it may have to evict data, recompute work or delay requests at exactly the moment users notice.[1]
A simple first-in-first-out queue ignores these differences. More elaborate heuristics may classify jobs or predict an average length, but an average hides the possibility of unexpectedly large responses. URSA predicts a calibrated distribution of output length and the resulting cache footprint. It can reserve memory against a high quantile selected for the service class rather than a single best guess. This matters when some customers have strict time-to-first-token limits, while others prioritise steady time per generated token.[1]
The proposed policy also considers heterogeneous GPU instances: different accelerators can have different compute capability, memory availability and current occupancy. Rather than treating the queue as a flat list, the design represents relationships among waiting requests, processors and cached state as a graph processed by a neural network. The resulting model helps decide which request should be dispatched to which instance, given current load and the risk of a bad length prediction.[1]
Training a scheduler without disrupting customers
URSA learns from simulated transitions generated by replaying historical traffic under existing heuristic policies. Its conservative offline reinforcement-learning objective balances requests meeting their service targets, useful completed work and memory risk. The method therefore avoids learning through experiments on live customers, although its training states and scheduling actions are generated by the simulator.[1]
The paper uses the datasets for different jobs. BurstGPT provides the primary length-uncertainty and burstiness workload; Azure traces inform experiments separating prompt processing from generation. Alibaba cluster traces supply arrival patterns and hardware heterogeneity, with token-length profiles attached from the language-model traces. They are not themselves logs of equivalent LLM requests.[1]
In the primary comparison, 99th-percentile time to the first token fell from 7.9 to 2.4 seconds relative to first-come-first-served dispatch. That is about a 3.3-fold speed-up at the tail; requests requiring eviction or recomputation fell to 0.9 percent. The simulator does not measure energy, and those results do not establish savings in a deployed service.[1]
The useful idea is to reserve resources for a range of plausible answers, then schedule against the user’s deadline. A short expected response is not automatically urgent, and a long one is not automatically dispensable. URSA’s simulations show why treating length and urgency separately can matter when the cluster fills up.[1]
Sources & further reading
- Uncertainty-aware offline reinforcement learning for LLM inference serving with heterogeneous SLOs Scientific Reports · 26 Oct 2026 · Accessed 26 Oct 2026