The economics of an AI token, from GPU time to customer price

A token price tells a customer what model input and output will cost under a tariff. It tells much less about the economics of producing them. Together AI illustrates the distinction: its serverless service uses a shared fleet and charges for tokens, while dedicated model inference runs on GPUs allocated to one customer. These are two ways to buy access to computing.[1]

The underlying business problem is turning available machine time into work people will pay for. A server can be ready throughout an hour and receive almost no requests. Another hour can keep the same allocation busy. Dividing each hour’s capacity cost by the work delivered produces very different unit costs.

Being ready has a price even during quiet periods

Together bills dedicated model inference by the minute for replicas that are ready to serve—a replica being one running instance of the model. Each ready replica is charged independently, whether requests arrive or not. Provisioning and cold-start time before a replica is ready are excluded. Stopping the deployment ends that ready-capacity bill.[3]

The contract changes who carries the risk of empty capacity. A customer buying tokens mainly sees a variable model-access bill. A customer keeping a dedicated allocation ready must either find enough useful work for it or accept paying for availability. Availability can have value even when it produces no tokens: a team may need a system waiting for an unpredictable request.

A worked example makes the difference visible

Together’s published rates, checked on 15 October 2026, list managed H100 inference at US$5.49 per GPU-hour. Its separate H100 cluster rental is $3.99 per GPU-hour, for a different service. Standard serverless Llama 3.3 70B costs $1.04 per million input tokens and the same for output tokens.[2]

For an illustrative comparison, assume two managed H100s remain ready for one hour. Every request contains 1,000 input tokens and produces 200 output tokens. The traffic volumes below are hypothetical completed work; the hardware allocation is an assumption, not a benchmarked or certified deployment. Caching discounts, batch discounts, retries, tools, taxes and other application costs are excluded.

The arithmetic crosses at $10.98 ÷ $1.04 = approximately 10.56 million combined tokens in the hour, or about 8,800 requests at 1,200 tokens each. Below that completed volume, the token tariff gives the smaller bill; above it, the assumed dedicated allocation does. This is a bill-equivalence point, conditional on the same model quality and the allocation actually completing the work within the hour.[2]

Whether that allocation can serve the work promptly is a separate engineering question. Together’s configuration documentation binds a deployment to particular model weights, numerical precision, serving software, GPU types and GPU counts. A configuration that needs more hardware changes the calculation. So does a bursty workload requiring extra capacity to prevent queues from becoming too long.[4]

Serving software changes how much the hour can deliver

The research system Orca, presented at OSDI in 2022, explains one useful mechanism. In a fixed batch of requests, a short response can finish while a long response continues occupying the batch. Orca schedules work between individual model iterations, allowing finished requests to leave and new ones to enter without waiting for the longest request to end.[5]

Orca illustrates a serving technique whose economic effect comes from fitting more useful work into available computing time. Particular services use their own engines and configurations. The wider connection is that scheduling decisions influence how much paid work a given allocation can complete.[5]

Capacity can also change with demand. Together’s autoscaling documentation describes replica limits and the delay between deciding to add a replica and having it ready. Its optional idle shutdown stops a quiet deployment and releases the hardware, but the stopped deployment does not restart on its own. Lower idle cost therefore comes with an operational choice about how service resumes.[6]

A provider’s total cost includes more than the running bill

CoreWeave’s second-quarter 2026 filing provides a useful accounting example. It places data-centre rent, electricity and certain facility costs in cost of revenue. Depreciation of servers and networking equipment appears in technology and infrastructure expenses, alongside related engineering costs. Interest is reported separately again. Looking at one expense line would leave out substantial parts of operating and financing the computing business.[7]

Those are CoreWeave’s categories, not a disclosure of Together’s margins. Our worked example calculates what a customer would pay for an assumed allocation at public rates. It cannot reveal the provider’s equipment purchase price, financing, actual fleet occupancy or negotiated contracts.

This is why a cheaper token and a cheaper computing hour are different developments. The link between them runs through the amount and timing of demand, the software serving it, the resources required and the service promised to the customer. Understanding that link makes a token price useful evidence about a product without mistaking it for the supplier’s production cost.

Report an error or suggest a correction