Coordinated Omission in LLM Load Testing: Closed vs. Open Arrival Queue Delays
Analyze coordinated omission in LLM load testing. Compare closed-loop and open-loop arrival rates via discrete simulation, interpret queue delay, and cap budgets.
In LLM performance benchmarking, when an initial request encounters a 300 ms stall, the second request physically begins execution at the exact same moment (t = 300 ms) in both architectures. However, measured metrics diverge completely: a closed-loop single client logs a queue wait of 0 ms with a finish time of 350 ms; an open-loop model with a fixed 100 ms arrival interval logs a queue wait of 200 ms with a finish time of 350 ms. Both report an end-to-end p95 of 300 ms, but only the open loop generates the independent arrival at 100 ms that has to wait.
This article focuses on queuing theory and arrival rate mechanisms; it does not measure Time to First Token (TTFT) or benchmark live endpoints.
Discrete Event Simulation: Closed vs. Open Loops
Consider a single serial FIFO service desk operating over a [0, 1000) ms horizon. Both models share identical service time rules: the first request takes 300 ms, and all subsequent requests take 50 ms. Percentiles use Nearest Rank, using the one-based rank ceil(0.95 * n).
- Closed-Loop Model: Dispatches the next request immediately upon completing the prior one. Request 0 begins at t = 0 ms and completes at t = 300 ms; request 1 starts at t = 300 ms and finishes at t = 350 ms; continuing iteratively through request 14, which finishes at t = 1000 ms. All 15 requests record a queue wait of 0 ms.
- Open-Loop Model: This example assumes an unbounded queue with no drops. Requests arrive according to a predetermined schedule at t = 0, 100, …, 900 ms (10 requests total). Execution timing follows
start = max(scheduled, free)andfree = start + service. Request 1 arrives at t = 100 ms but must wait until t = 300 ms for the server to free up, experiencing 200 ms of queue wait. Request 4 waits 50 ms, while requests 5 through 9 experience 0 ms wait. The final request completes at t = 950 ms.
| Metric | Closed-Loop Client | Open-Loop Fixed Arrival (Unbounded Queue Assumption) | Comparative Dynamic |
|---|---|---|---|
| Completed Requests | 15 requests | 10 requests | Closed loops dispatch faster as service clears; open loops follow fixed clock |
| Final Finish Time | 1000 ms | 950 ms | Both finish within the 1000 ms window |
| Queue Wait p95 | 0 ms | 200 ms | Only the open loop schedules work while the server is busy |
| Service Time p95 | 300 ms | 300 ms | Identical service time execution rules |
| End-to-End Latency p95 | 300 ms | 300 ms | Peak isolated slow request elevates tail percentile in both samples |
import math
def calculate_p95(values):
if not values:
return 0
sorted_v = sorted(values)
return sorted_v[math.ceil(0.95 * len(sorted_v)) - 1]
def simulate_serial_queue():
get_svc = lambda idx: 300 if idx == 0 else 50
# 1. Closed-loop
closed_reqs, clock, req_id = [], 0, 0
while clock < 1000:
svc = get_svc(req_id)
closed_reqs.append({"id": req_id, "wait": 0, "svc": svc, "e2e": svc, "finish": clock + svc})
clock += svc
req_id += 1
# 2. Open-loop (Unbounded queue assumption)
open_reqs, free_at = [], 0
for req_id, sched in enumerate(range(0, 1000, 100)):
svc = get_svc(req_id)
start = max(sched, free_at)
finish = start + svc
open_reqs.append({"id": req_id, "wait": start - sched, "svc": svc, "e2e": finish - sched, "finish": finish})
free_at = finish
return closed_reqs, open_reqs
if __name__ == "__main__":
c_reqs, o_reqs = simulate_serial_queue()
print(f"Closed: n={len(c_reqs)}, wait_p95={calculate_p95([r['wait'] for r in c_reqs])}ms, e2e_p95={calculate_p95([r['e2e'] for r in c_reqs])}ms")
print(f"Open: n={len(o_reqs)}, wait_p95={calculate_p95([r['wait'] for r in o_reqs])}ms, e2e_p95={calculate_p95([r['e2e'] for r in o_reqs])}ms")Discrete sample calculations illustrate queuing mechanics qualitatively and must not be used as production Service Level Objectives (SLOs). If you encounter connection resets or truncated streaming responses during load tests, see the troubleshooting guide for LLM streaming HTTP 200 incomplete responses.
k6 Mechanics and Metric Dimensions
In Grafana k6, arrival-rate scenarios diverge from idealized mathematical queues:
- According to official documentation on k6 Open vs Closed Scenarios and k6 Dropped Iterations, arrival rates are sustained by Virtual Users (VUs). When an iteration is scheduled but all
maxVUsare occupied, k6 recordsdropped_iterations. - The
dropped_iterationsmetric denotes unstarted iterations caused by client worker saturation; it is not an HTTP 5xx error, nor is it equivalent to a failed HTTP request. - Having
dropped_iterations > 0indicates the test did not sustain the target arrival rate, but it does not invalidate executed samples. Conversely,dropped_iterations == 0alone does not prove compliance with online SLOs. - Iterations match HTTP requests only when a scenario is configured to execute strictly one request per iteration. Benchmark reports must independently log scheduled iterations, started iterations,
dropped_iterations, alongside completed business successes, timeouts, HTTP failures, and connection aborts.
Client-Side Budget Guards and APIBox Gateway Configuration
When evaluating external model gateways, callers should enforce strict client-side guards: set hard caps on total attempts (including retries) and constrain input/output token limits per request. Once the planned budget ceiling is reached, the client should halt request dispatch immediately.
The APIBox base URL is https://api.apibox.cc/v1. The live pricing catalog lists these model and group multipliers (commercial billing ratios only; not SLAs or availability guarantees):
- gpt-6-astra (group
gpt-vip-1): multiplier 0.1 - claude-sonnet-5 (group
claude-vip-2): multiplier 0.3 - gemini-3.8-flash (group
gemini-vip): multiplier 0.2
Billing deposits support Alipay and WeChat Pay, as documented on the APIBox pricing overview page. After completing offline queuing simulations and establishing client-side budget caps, developers can create evaluation credentials via the APIBox platform registration page.
Frequently Asked Questions
What is coordinated omission in LLM load testing?
Coordinated omission occurs when a closed-loop client delays issuing subsequent requests because a prior call is stalled, misaligning emission timing from independent arrival traffic and omitting queue wait times from metrics.
Are k6 dropped_iterations equivalent to HTTP errors?
No. The dropped_iterations metric indicates that active VU capacity was exhausted to start a new iteration on schedule; it is neither an HTTP error nor directly equivalent to a failed HTTP call.
Why can’t small-sample discrete event simulation results be used as online SLO targets?
Small-sample discrete simulations follow fixed mathematical rules to demonstrate queuing mechanics qualitatively; they cannot account for live thousand-user concurrency, network jitter, or tail latencies in production.
Try it now, sign up and start using 30+ models with one API key
Sign up free →