How OpenAI cut Habitat’s median global read latency from 120 ms to 10 ms

OpenAI says it reduced Habitat’s median global read latency from roughly 120 milliseconds to roughly 10 milliseconds by bringing frequently used data closer to requests and reducing service overhead. The engineering account explains an often invisible part of AI performance: before a model answers, the product may need settings, permissions and conversation state from storage.[1][2]

Habitat supports products used by more than a billion people weekly and serves over 500 petabytes of data across almost 40 regions, according to OpenAI. Its reported 70 million requests per second are storage-platform operations, not language-model prompts. That distinction matters when interpreting the scale of the system.[1]

Habitat began as a Python client library connected to database systems such as Azure Cosmos DB. As more products came to depend on it, the library evolved into a central service that handles routing, authorisation, caching and other data operations. Centralisation reduced the need for individual product teams to understand every underlying database detail. Yet every new source of traffic introduced another opportunity for a local spike to affect unrelated services.[1]

Why sharing a database became dangerous

The team's October account describes a circular architectural journey. Shared database containers initially simplified resource management but allowed so-called noisy neighbours: one busy workload could exhaust capacity shared with another. The company responded by creating dedicated containers for important services, reducing the radius of an outage. That offered better isolation while forcing teams to estimate and provision capacity before launches. A product moving rapidly from internal testing to large-scale use could encounter an avoidable bottleneck if nobody had enlarged its container in time.[1]

The eventual solution was not simply to choose shared or dedicated databases forever. OpenAI worked with Azure on more careful controls for shared workloads: throttling unusually hot keys, prioritising interactive requests over background jobs, and improving serverless scaling behaviour. According to the company, those changes allowed it to return to shared tenancy for many uses without the same provisioning problems. The published narrative is a case study in engineering trade-offs, not a universal prescription for every application's database.[1][2]

The performance problem had a second dimension: distance. Some data lived far from the region serving the user's request. Replicating everything everywhere would create huge costs and collide with limited regional storage capacity. Instead, the team prioritised replicating the smallest, most latency-sensitive metadata and caching frequently accessed information near the serving computers. The system also uses change feeds to invalidate cached entries shortly after the underlying data changes, reducing the tension between fast reads and stale answers.[1]

Selective replication, integrated caching and the Rust rewrite together produced the reported change from about 120 ms to 10 ms at the median, or p50. Dividing the former by the latter gives a 12-fold ratio. The slowest uncached reads can still cross distant regions, so the figure describes the middle request rather than a guarantee for every read.[1]

Optimising the common read without hiding the slow one

OpenAI reports more than 20 million queries per second on its largest Azure Cosmos DB account, alongside millions of captured changes per second feeding caches and other downstream systems. These figures explain why the company describes a close engineering partnership with Azure rather than a simple hand-off to a managed database.[1]

OpenAI also outlined an in-development exabyte-scale storage system for bulky data, using components associated with RocksDB and replicated state management. That future design should not be confused with the current operational system or treated as a delivered capacity milestone. At this scale, planned architectures are bets against future constraints, and the test is whether they continue working under load, failures and geographic requirements.[1]

The lesson is visible to users even when the machinery is not. Fast model inference has limited value if loading a conversation or checking access becomes the slow step. Habitat’s redesign tackles that surrounding work, while its remaining cross-region reads show why improving the median does not finish the latency problem.[1]

Report an error or suggest a correction