Why fast storage networks can still leave AI GPUs waiting

The network had room to spare. The GPUs still had to wait. In a May 2026 technical report, Lablup analysed a production cluster containing 504 Nvidia B200 GPUs, with 480 used for the training job. Across 20 measured starts, loading checkpoints and datasets took a median 31 minutes. Storage links rated at 200 gigabits per second per node averaged roughly one-tenth of their theoretical capacity during loading.[1]

The authors traced a bottleneck to a limit of 128 outstanding storage requests per connection. This was a case study of one installation, and it did not measure overall training efficiency. Its wider lesson is nevertheless valuable: buying more bandwidth will not necessarily get data to a processor sooner.[1]

The queue before the cable

A network file system lets a computer read and write files held on another machine. Each request must be admitted, processed and completed. The number allowed to remain outstanding at once is its concurrency. Microsoft’s NFS guidance explains that these limits can apply at the connection or session level, depending on the protocol. Both the client and the server matter.[2]

Imagine a hypothetical store that admits only ten collection orders at a time. If each occupies its place for one second, it can complete about ten orders per second at steady state. Widening the road outside does nothing while that admission limit remains the constraint. Faster service or more usable places would change the result. The same queueing relationship links outstanding storage requests, response time and throughput.[2]

That is why a low bandwidth reading can be misleading. It may mean there is little work to do. It may also mean work is waiting somewhere the bandwidth meter cannot see. For someone assessing an AI facility, the consequential question is where the application’s time goes.

Saving has two clocks

Checkpointing preserves the state needed to resume training. It also illustrates another reason the speed printed on a storage specification is an incomplete guide. In asynchronous saving, the training process first secures a stable copy of its state; storage work can then continue while the GPUs return to computation.[3]

A 2024 PyTorch account of IBM Research’s work reported that a seven-billion-parameter model’s average checkpoint-related training pause fell from 148.8 seconds to 6.3 seconds. The reported improvement concerned time taken away from training. The bytes still had to be serialised and written: asynchronous processing moved that work off the immediate training path.[3]

Current PyTorch documentation distinguishes completion of staging—the protected copy—from completion of the upload. Its fully asynchronous example waits for staging before the optimiser changes the model’s parameters, then separately manages the remaining save. The copy needs CPU memory; additional simultaneous saves can add memory pressure. The tutorial generally recommends one pending checkpoint request at a time.[4]

A useful operational view therefore follows three moments: when training can continue, when the saved state is available from persistent storage, and when a restart has actually resumed useful computation. Improving one does not establish that the others improved by the same amount.

Measure the whole route

The Lablup report’s storage diagnosis points towards request handling and response time. It does not demonstrate a tested universal tuning fix.[1]

Nor does raising every concurrency limit guarantee more speed. Google Cloud’s guidance for its NetApp service warns that excessive outstanding requests can overwhelm the server’s ability to process a workload and cause it to restrict the connection. Settings have to fit the service and the number of clients sharing it.[5]

For a data-centre customer, a useful acceptance test would include the actual training framework loading a representative checkpoint across the intended number of machines. Ask for elapsed restart time alongside bandwidth, request latency and GPU activity. Repeat with the cache conditions the contract is supposed to cover. These are proposed purchasing questions, not tests performed for this article.

A room full of accelerators is only productive when work reaches them. Sometimes the next increment of capacity is waiting to be released by understanding a queue.

Report an error or suggest a correction