← Back to Blog

Why You Cannot Replay an Entire LLM Batch Job

Learn why replaying entire finished LLM batches causes duplicate costs and data corruption, and how item-level reconciliation prevents systematic failures.

The dual-file fields below refer to OpenAI native Batch. Claude uses its own results-stream protocol. HTTP 5xx alone does not prove non-execution.

The Illusion of the Completed Batch Status

When orchestrating high-volume asynchronous inference with frontier models such as GPT, Claude, or Gemini, receiving a completed status event on a batch job is not equivalent to business-level acceptance. Upstream provider platforms mark a batch as completed when the batch processing lifecycle reaches that state. Inspect individual results separately; Claude uses processing_status ended rather than the same status names.

Furthermore, batch processing architectures process records out of sequence. In native batch pipelines like the OpenAI Batch API, the order of returned records in the output file is not guaranteed to match the submission line order. Merging responses back into your internal domain model requires joining specifically on the custom identifier field submitted with each line. Treating a batch job as a single atomic unit is an anti-pattern that leads to redundant spend, corrupted application state, and unhandled silent drops.

The Risks of Wholesale Batch Replays

When a batch finishes with partial errors, engineers are often tempted to resubmit the original batch payload in its entirety. Executing a blanket replay introduces three immediate failure modes:

  1. Duplicate execution and billing: Successfully processed records are charged a second time, and downstream databases risk inserting duplicate records or triggering race conditions.
  2. Permanent error amplification: Requests that failed due to invalid prompt structures, unauthorized model parameters, or context-length violations will deterministically fail again upon replay, incurring processing overhead without changing the outcome.
  3. State collisions: Partially processed or cancelled batches often contain successfully fulfilled prompts. If a cancelled job produced usable outputs, those records must be parsed and stored rather than discarded and re-executed.

To safeguard pipeline integrity, you must isolate stable business identifiers from transient execution attempt identifiers. If your ingestion system identifies duplicated custom identifiers or schema conflicts, those lines must be quarantined immediately rather than propagated into re-execution queues.

Item-Level Reconciliation and Four-State Classification

Production-ready batch systems operate on row-level reconciliation rather than file-level restarts. Once a batch enters a terminal state, for OpenAI the client downloads artifacts referenced by non-null output_file_id and error_file_id. Claude instead exposes a result stream with per-item result.type. Any item submitted in the manifest that appears in neither file must be flagged as unknown rather than assumed to be dead.

To illustrate this triage mechanism, consider four synthetic requests evaluated during a batch post-mortem:

  • Item A: Accepted. The model returned a valid completion, payload validation passed, and the business data was committed downstream.
  • Item B: Quality Failed. The model generated a completion, but the output failed strict business schema parsing or hallucinated critical constraints. A blind replay does not guarantee better quality; investigate acceptance rules and prompts under a separate budget.
  • Item C: Expired Confirmed Non-Executed Eligible. The batch engine terminated or timed out before processing this specific item, and the specific result confirms it was not executed and is eligible for replay.
  • Item D: Absent Unknown. The identifier appeared in the input file but was missing from both output and error manifests upon completion. Its execution state cannot be verified from the file download alone.

In this synthetic triage, only Item C is an immediate retry candidate. Item A must be preserved, Item B must be routed to human review or quarantine, and Item D must be queried against execution logs or quarantined before any replay decision is made.

Operational Workflows and Client-Side Reconciliation

Row-level reconciliation is strictly an application-layer procedure. While developers frequently aggregate upstream access across models such as gpt-6.2-sol, claude-sonnet-5, or gemini-3.8-flash, third-party routing infrastructure does not replace your local ledger. For instance, developers testing unified gateway interfaces can create an account at APIBox Registration and connect to APIBox v1 Base. However, querying an external catalog endpoint or reviewing model lists via pricing verifies model availability only; it does not indicate managed asynchronous batch capabilities or native /batches routing support.

When managing long-running agent workflows and resilient background tasks, combine client-side idempotency keys with strict retry policies. Review our architectural analysis on Agent Tool Timeout and Idempotency for strategies on separating request identity from execution attempts.

External Sources and Official Documentation

Runnable offline check

Synthetic data only; no model calls or production retries.

# Offline synthetic normalized states, not a provider parser.
def classify(expected, rows):
    if len(expected) != len(set(expected)):
        raise ValueError("duplicate input ID")
    seen = {}
    allowed = {"accepted", "quality_failed", "confirmed_retryable"}
    for key, state in rows:
        if key not in expected or key in seen or state not in allowed:
            raise ValueError("unknown ID, duplicate result, or invalid state")
        seen[key] = state
    return {key: seen.get(key, "unknown") for key in expected}

result = classify(["a", "b", "c", "d"], [
    ("c", "confirmed_retryable"), ("a", "accepted"),
    ("b", "quality_failed")])
assert result["d"] == "unknown"
assert [k for k, v in result.items() if v == "confirmed_retryable"] == ["c"]
try:
    classify(["a"], [("a", "accepted"), ("a", "accepted")])
except ValueError:
    pass
else:
    raise AssertionError("duplicates must be quarantined")
print(result)

Sources and live catalog

FAQ

Why does output line ordering differ from input line ordering in batch files?

Batch processing systems distribute individual requests across parallel worker nodes to maximize infrastructure throughput. Because different prompts vary in length and generation time, requests complete out of order, requiring clients to join responses via custom_id.

How should missing records absent from both output and error files be handled?

Missing records must be classified as unknown rather than automatically queued for replay. Re-running missing requests without confirming execution status risks duplicate inference costs and potential write conflicts in downstream stores.

Can validation or authentication failures in a batch be retried automatically?

No. Errors caused by invalid request schemas, unsupported parameters, or authentication issues are deterministic. Automatic retries will fail continuously and consume pipeline quota without changing the underlying error condition.

Try it now, sign up and start using 30+ models with one API key

Sign up free →