Skip to content
Docs menu

Batches, files and service tiers

Run offline work with OpenAI's Batch API, turn single requests into batch requests at batch prices, upload files, and ask for flex or priority tiers.

View .md

Work that doesn't need an answer now can run cheaper and out of the way of interactive traffic. The orchestrator offers three ways: OpenAI's Batch API over uploaded files, single requests marked as batch requests, and provider service tiers. Files and batches belong to the client key that made them; another key gets 404.

Files

curl -s http://127.0.0.1:8787/v1/files -H "Authorization: Bearer $VENOM_API_KEY" \
  -F purpose=batch -F file=@requests.jsonl
Endpoint What it does
POST /v1/files Upload (multipart/form-data: file, purpose one of batch, user_data, assistants, vision, evals, fine-tune), at most 200 MB, stored under $VENOM_DATA_DIR/files/. Returns the file object (id: "file-…", bytes, filename, purpose, status: "processed").
GET /v1/files?purpose=&limit=&after= List, newest first.
GET /v1/files/{id} One file object.
GET /v1/files/{id}/content The bytes. Generated images are served here too, through a signed link that works without a key for an hour.
DELETE /v1/files/{id} Delete; 409 file_in_use while an unfinished batch reads it.

Batch results are files of purpose batch_output.

Batches

Each line of the input file is {"custom_id", "method": "POST", "url": <endpoint>, "body"}:

{"custom_id": "doc-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "venom-fast", "messages": [{"role": "user", "content": "Summarise: ..."}]}}
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="my-local-key")
f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(input_file_id=f.id, endpoint="/v1/chat/completions", completion_window="24h")
print(client.batches.retrieve(batch.id).status)
Endpoint What it does
POST /v1/batches {input_file_id, endpoint, completion_window: "24h", metadata?}. endpoint is /v1/chat/completions, /v1/completions, /v1/embeddings or /v1/responses. Returns OpenAI's batch object (id: "batch_…", status, request_counts, output_file_id, error_file_id).
GET /v1/batches/{id}, GET /v1/batches?limit=&after= Read and list.
POST /v1/batches/{id}/cancel cancelling, then cancelled; 409 batch_not_cancellable once it ended.

Validation (unique custom_id, the batch's endpoint, a model, no stream, at most 50,000 lines) fails the batch with line-numbered errors. Output lines are {id, custom_id, response: {status_code, request_id, body}, error}: 2xx in the output file, other statuses in the error file, a line that never got an answer with response: null and an error. A batch past its 24-hour window ends expired with what finished.

How it runs: one batch at a time, oldest first, each line served in-process as the batch's key through the normal routing, budgets and request log, on the background lane (flex on OpenAI), with the batch id as its session so the Sessions view sums its cost. Up to 4 lines run at once while no interactive request is in flight, 1 otherwise; while idle, a line for an alias whose first target is in the cloud goes to its first local target. A 429 or 5xx is retried up to 5 times. State is in SQLite, so a restart resumes a batch where it stopped.

On an upstream with native_batch: true (kinds openai, dashscope, anthropic), a batch whose every line resolves there is handed to the provider's own Batch API instead (polled every 30 s, cancel forwarded, results copied back), and each line is logged at half the list price. Anthropic's Message Batches take chat completions only.

Batch requests

Add x-venom-urgency: batch (or x-venom-service-tier: batch) to a single /v1/chat/completions, /v1/completions, /v1/embeddings or /v1/responses request and it becomes a batch request. When the model's first target has native_batch, it is collected with the key's other requests for that model (for batch_routing.window, 30 s by default, or until max_lines, 1000) into one provider batch at the batch price. Anything else (and any request under a route profile, or from a key with a hard spend limit) runs at once on the background lane.

curl -s -i http://127.0.0.1:8787/v1/chat/completions -H "Authorization: Bearer $VENOM_API_KEY" \
  -H "x-venom-urgency: batch" -H "x-venom-batch-wait: 120" -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-5.4-mini","messages":[{"role":"user","content":"Classify: ..."}]}'

The answer is 202 with a batch_request object (id: "breq_…", status queued, in_progress, completed, failed, cancelled or expired, mode provider_batch or background, and response once done), Location: /v1/batch_requests/{id} and x-venom-batch-request.

Header or endpoint What it does
x-venom-batch-wait: <seconds> Wait up to that long (at most batch_routing.max_wait, 10 min) and answer like the synchronous call, or 202 if it isn't back.
stream: true (chat completions) 200 text/event-stream at once, SSE comments every 15 s, then the result as chunks and [DONE].
x-venom-callback-url POST the finished object to that URL; its host must be in batch_routing.callback_hosts (else 400 callback_host_not_allowed).
x-venom-urgency: background Serve now on the background lane; now changes nothing; any other value is 400.
GET /v1/batch_requests?limit=&after= List, newest first, without responses.
GET /v1/batch_requests/{id}?wait=<seconds> Read one; long-poll up to max_wait.
POST /v1/batch_requests/{id}/cancel Cancel while queued; 409 batch_request_not_cancellable after.

Batch requests are kept 7 days after they finish.

Service tiers

x-venom-service-tier: auto | default | flex | priority asks for a provider tier when the body has no service_tier (it wins over a route profile's default). OpenAI gets it as is (flex only on models that offer it; a flex 429 is resent at once on the default tier). Anthropic: priority and auto become auto, default becomes standard_only. Other kinds ignore it. Any other value is 400 invalid_service_tier. The answer carries x-venom-service-tier with the tier sent. Flex and priority requests are priced at the table's flex and priority rates, and GET /admin/v1/spend reports per tier what was spent and saved against the standard price. Gears can set a tier per role (Profiles, side requests and gears).