AMD’s September 2026 UltraQuant evaluation reported about 28% more output tokens per second at its highest tested load: 32 simultaneous requests on eight MI355X GPUs. The change was a smaller numerical format for cached context. The model weights stayed fixed, making this a test of serving efficiency rather than a larger or newly trained model.[1]
An AI assistant produces its answer a piece at a time. A key-value cache retains intermediate attention information from earlier text, allowing the model to reuse work. That cache can also support successive turns in a conversation. Compressing it changes how the information is represented; it does not require writing a shorter prose summary of the conversation.[3]
Preserve the relationships the model needs
The TurboQuant research offers one explanation of how this can work. Its April 2025 paper describes rotating numerical vectors before assigning their coordinates to a small set of values. Rotation makes the distribution easier to represent economically. A second variant adds a one-bit correction for the error in inner-product estimates: the comparisons that help an attention mechanism weigh information.[2]
The representation is lossy. The useful question is whether the discarded numerical precision changes performance on the task. In the paper’s Llama-3.1-8B-Instruct experiment, the average LongBench score matched the full-cache baseline at 3.5 effective bits per channel. Individual task scores differed. A matching average therefore describes that evaluation, rather than identical answers or universal preservation of quality.[2]
TurboQuant’s reported LongBench results
| Cache representation | Bits per channel | Average score |
|---|---|---|
| Full cache | 16 | 50.06Source 2 |
| TurboQuant | 3.5 | 50.06Source 2 |
| TurboQuant | 2.5 | 49.44Source 2 |
Make the hardware benefit
UltraQuant uses a rotation and a hardware-compatible four-bit format. AMD’s MI355X can consume these values directly during matrix operations, avoiding a software lookup-and-reconstruction stage. Each group of 32 cached values occupies 17 bytes, including its scale, against 32 in the eight-bit format. Model weights remained unchanged between cache configurations; only 23 of Qwen3.8’s 92 layers hold this cache.[1]
AMD’s Qwen3.8-MXFP4 replay
| Measure | 8-bit ROCm-FA | UltraQuant |
|---|---|---|
| Total decode throughput | 338 tokens/second | 434 tokens/secondSource 1 |
| Time between output tokens | 77 milliseconds | 58 millisecondsSource 1 |
Light-load results were similar. On GPQA-Diamond, UltraQuant answered 184 of 198 questions correctly against 187 for eight-bit caching; AMD interpreted the difference as sampling variation. Publication-time plans included bringing the measured serving path into the main vLLM project.[1]
Where the gain becomes useful
Hugging Face’s implementation guidance supplies an important practical boundary: cache quantisation can increase latency when context is short and sufficient GPU memory is available. Its different cache classes trade memory consumption against generation speed. Smaller storage is one part of the decision; the time spent producing a useful answer remains the outcome to measure.[3]
For someone evaluating an assistant, a sensible trial would keep the model and task fixed, then compare complete conversations at the expected workload. Include tasks that revisit an early detail. Record whether the final work is correct, as well as how quickly it arrives. That proposed evaluation connects the engineering gain to the reason for keeping a long context at all.
Sources & further reading
- UltraQuant: Qwen3.8 serving evaluation AMD ROCm Blog · 25 Sep 2026 · Accessed 12 Oct 2026
- TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate Zandieh and colleagues / arXiv, version 1 · 28 Apr 2025 · Accessed 12 Oct 2026
- Cache strategies Hugging Face Transformers documentation · Accessed 12 Oct 2026