Providers sell the same models at roughly half their synchronous price to anyone willing to wait — this page is what a queue must guarantee before that discount is worth taking, and where it does not apply at all.
Most of what a production system sends to a model is not urgent. Nightly enrichment, backfills, evaluation sweeps, summarising a corpus, re-scoring records that changed yesterday — the output is read by a person the next morning, or by another program on a schedule. All of it is billed at the price of work somebody is actively waiting for.
Providers offer an alternative: submit the same requests as a file, accept a turnaround measured in hours instead of seconds, and pay roughly half. The price and the window are the provider's, not ours. What stops most teams taking the trade is not the economics; it is the second integration, the second place credentials live, and a job runner you now have to operate and monitor.
| Synchronous | Batch | |
|---|---|---|
| Price | The provider's standard rate | Roughly half — the provider's batch rate |
| Turnaround | Seconds | Up to 24 hours; the provider's window, not a target |
| Integration | An OpenAI-compatible client | The same client, a different call |
| Credentials | Yours, held once | Yours, the same ones |
| The job runner | Not needed | Ours — submission, polling, retries, result storage |
The batch surface is the OpenAI-compatible one you already know: upload a file of requests, create a batch, read its status, download the output. Point the official client at the gateway and those objects behave as the SDK documents them.
# Same client, same key — one call changes
from openai import OpenAI
client = OpenAI(base_url="https://api.ai-ctrl.net/v1", api_key="$AI_CTRL_API_KEY")
f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
b = client.batches.create(input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h")
Batch pricing applies where the provider publishes it, on the models it publishes it for. Coverage widens as providers are added. more providers soon
A batch is the largest single spend an inference API can commit in one call, and the easiest to duplicate. The component that submits it runs as more than one copy, and every copy sees the same queued work. Send the same file twice and the provider does exactly what it was asked: runs it twice, charges twice, returns two identical result sets. Nothing errors. You find out on the invoice.
So taking a job and marking it taken are one indivisible step, not two. One copy wins the claim and holds it for the length of the submission; any other copy reaching for that job finds it held and moves on. The claim is settled before the money is spent, never after — a marker written once the request is in flight says nothing about the request that left a moment earlier.
It is a held claim rather than a plain flag because a sender can die mid-submission, and both naive versions fail in opposite directions. A flag never released strands the batch forever — unsent, unbilled, undone. A flag released the moment a sender looks unhealthy sends the batch again while the first request is still in flight, which is exactly the charge you were avoiding.
Once submitted, the batch belongs to the provider. Our side does one thing: ask whether it has finished, at a rate matched to how likely the answer is to have changed. The first check comes half a minute after submission, then a minute, five, fifteen, and from there once an hour until it ends.
Small batches often come back inside the first few checks, which is why those are close together. A large one will not, and asking every thirty seconds for a day spends request quota to learn nothing. Neither number says when your work finishes. They say when we look.
Three terminal outcomes, and the difference between them is operational rather than cosmetic. Completed — results fetched, stored, yours to download. Failed — the work did not come back, and the provider's own reason travels with the outcome instead of being flattened into a generic message. That includes the case where the provider's window elapsed with the work unfinished: it surfaces as a failure carrying the expiry reason, so the cause is still legible even though the status is shared. Cancelled — you asked us to stop; the cancellation is picked up on the next check and passed upstream, so a batch you no longer want stops costing you as soon as the provider allows, rather than running to completion because nothing was listening.
An elapsed window and a rejected request share a status but not a remedy. One says the work did not fit the time available — split the batch, or accept that this workload wants the synchronous path. The other says the request itself was wrong. The reason field is what tells them apart, which is why it is carried through rather than summarised: retrying on the status alone is how a team re-submits something that will run out of time again for exactly the same reason.
The saving is real and so is the constraint, and the constraint is the part most descriptions leave out. Run a workload through these four questions in order. The first one that stops you, stops you.
If your traffic already goes through an OpenAI-compatible client, the deferrable half of it is one call away from a different price. We are onboarding early users now.