The batch API for AI agent pipelines
Most of an agent's model calls have no user waiting on them. Run them as batch jobs: OpenAI-compatible, one job per step across every work item, billed at 50% below realtime.
- Fan-out agent steps
- Multi-step loops
- Judge passes built in
- 50% off realtime
Your code plans. The batch executes.
An agent normally calls the model once per step, item by item. Batch changes only that part. Your code still holds the state, runs the tools, and decides what happens next. It just collects the next model call for every item and submits them all as one job.
When the job completes, your code applies the results, runs whatever tool calls the models asked for, and submits the next round. Every request carries a custom_id with its item and step, so each result lands back on the right item automatically.
Nothing here is proprietary. It is the standard OpenAI-compatible Batch Inference API driven by an agent planner instead of a static file.
{"custom_id":"lead-0041:research","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gpt-oss-120b","messages":[{"role":"system","content":"You are the research step of a sales agent. Given a company profile, produce {\"summary\", \"signals\": [], \"disqualifiers\": []} as JSON."},{"role":"user","content":"Company: Meridian Freight, 240 employees, WMS migration announced in June..."}]}}
{"custom_id":"lead-0042:research","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gpt-oss-120b","messages":[{"role":"system","content":"You are the research step of a sales agent. Given a company profile, produce {\"summary\", \"signals\": [], \"disqualifiers\": []} as JSON."},{"role":"user","content":"Company: Halcyon Labs, 45 employees, hiring two ML platform engineers..."}]}}# round N: run one agent step across every work item, as one job
plan = [agent.next_request(item) for item in items if not item.done]
write_jsonl("round.jsonl", plan) # custom_id = item:step
f = client.files.create(file=open("round.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
# when the job completes, fold results back into agent state
for row in read_jsonl(client.files.content(batch.output_file_id)):
items[row["custom_id"]].apply(row["response"])
# tool calls requested by the model run in your orchestrator,
# and their results go into round N+1's JSONLSix agent workloads that batch.
Wide fan-out, nobody watching a spinner: half price and no rate-limit juggling.
Fan-out steps
Research every lead, read every file, triage every ticket. One agent design, thousands of independent runs. This is most of your token spend, and none of it needs a streaming connection.
Multi-step loops
Submit a round of model calls, run the requested tool calls in your orchestrator, submit the next round. Per-step latency stops mattering when the pipeline runs overnight anyway.
Judge and gate passes
Score every agent trajectory with an LLM-as-a-judge batch before results reach production. The judge pass costs a rounding error next to the run it audits.
Memory compaction
Summarize histories, distill scratchpads, refresh entity memory: one nightly job across every session instead of inline work during conversations.
Backfills and replays
New agent version, old inputs. Replay last quarter's tickets through the new pipeline and diff the outcomes before cutover. At batch prices, full-history replays are routine.
Eval sweeps
Grid-search prompts, models, and tool configs by expanding the matrix into JSONL. A 10x20 sweep over 500 tasks is one file, not a week of rate-limited loops.
Batch the pipeline, stream the conversation.
Split by call, not by agent. Most products have both kinds.
- Pipelines where each item is independent (leads, documents, tickets, repos)
- Agent steps that tolerate hours of latency: enrichment, triage, research, review
- Anything you would run nightly, weekly, or per-release
- A user watching the agent work: keep that on realtime chat completions
- Deep sequential chains over a single item with no fan-out to amortize
The steps your agents are already doing.
Every workload page ships a runnable JSONL example and model picks with batch pricing.
Agentic batch, answered.
What is an agentic batch API?
A batch API used as the execution layer for AI agent pipelines. Instead of calling a realtime endpoint once per agent step, your orchestrator collects the current step for every work item into a JSONL file, submits one job, and applies the results. Independent items run as one job; sequential steps run round by round. The economics are the point: agent pipelines multiply request volume, and batch cuts the per-token price in half while removing rate-limit orchestration.
How do tool calls work in a batch job?
The model side of a tool loop batches; the tools run where they always ran, in your orchestrator. Each round's responses include the tool calls the agents requested; your code executes them and writes the results into the next round's messages. Batch executes model requests, so anything the model needs mid-request has to be in the request.
When should an agent use batch instead of realtime?
Split by who is waiting. A user watching the agent needs realtime streaming. A pipeline processing a queue (enrich these 10,000 leads, triage this backlog, review every PR from last sprint) is batch-shaped: independent items, no latency requirement, and enough volume for the 50% discount to be a line item.
Does this work with my agent framework?
If the framework separates planning from execution, yes: anything that can emit its next model request as JSON instead of firing it can batch. Teams typically add a batch executor alongside the realtime one and route by workload. The surface is the standard OpenAI batches API, so the SDK plumbing already exists.
How do I evaluate agents at scale?
Serialize each trajectory (messages, tool calls, outcomes) into one grading record and run an LLM-as-a-judge batch over all of them. The same pattern gates deploys: replay a fixed task set through the new agent version, judge both runs, and diff the scores. See the LLM-as-a-judge workload page for prompts and costs.
Point your orchestrator at a batch endpoint.
Same OpenAI SDK, same JSONL, half the price. Odd pipeline shape? Tell us what it does and we will help you batch it.