Using Promptfoo to Prevent Ticket Routing Regressions
Test ticket routing with Promptfoo: compare two prompts, catch billing misclassification and case mismatches, and inspect a small real API evaluation.
We use synthetic ticket negative controls to catch routing errors prior to rollout.
The candidate is deliberately broken as a negative control, not a customer incident.
The Configuration
We use promptfoo to compare our baseline prompt against a candidate version. We target the Gemini 3.8 Flash model via the APIBox gateway. Ensure your APIBOX_API_KEY is exported in your environment variables rather than hardcoded. Always include your .env and results.json files in .gitignore to protect sensitive data.
description: Ticket routing regression example
prompts:
- id: baseline
raw: 'Classify the ticket as billing or technical. Payment, invoices and refunds are billing. Crashes and connectivity are technical. Return only the label. Ticket: {{ticket}}'
- id: candidate
raw: 'Classify the ticket as billing or technical. If it mentions API, choose technical even for an invoice dispute; otherwise payment and refunds are billing. Return only the label. Ticket: {{ticket}}'
providers:
- id: openai:chat:gemini-3.8-flash
config:
apiBaseUrl: https://api.apibox.cc/v1
apiKeyEnvar: APIBOX_API_KEY
temperature: 0
max_tokens: 256
timeout: 30000
maxRetries: 0
tests:
- vars:
ticket: 'I was charged twice for my API subscription. Please refund the duplicate payment.'
assert:
- type: equals
value: billing
- vars:
ticket: 'The API endpoint returns connection refused.'
assert:
- type: equals
value: technical
- vars:
ticket: 'Please send me an invoice for last month.'
assert:
- type: equals
value: billingRunning the Test
Execute the test suite using the CLI. We use --no-cache to ensure fresh inference and --max-concurrency 1 for stable, sequential evaluation:
npx --yes promptfoo@0.125.1 eval --no-cache --max-concurrency 1 -o results.jsonAnalysis of Results
6 inference evaluations using Promptfoo 0.125.1 via APIBox gemini-3.8-flash show: Baseline returned ‘Billing’ for the duplicate-charge ticket, failing on strict equality with ‘billing’. Candidate returned ‘technical’ for the same ticket, failing due to semantic misrouting. The other 4 evaluations passed; errors were 0. The process exited with code 100 due to assertion failures, not provider failure.
Small sample sizes do not prove production accuracy or deterministic model outputs, even at temperature 0 with no cache. —no-cache only ensures fresh requests; normalization should be evaluated separately only if supported by downstream business contracts.
Add anonymized boundary tickets, inspect failures for each label, and repeat evaluation before rollout.
Get a key from the APIBox console; check pricing and your actual bill.
References: OpenAI-compatible provider · Assertions · APIBox model directory
Try it now, sign up and start using 30+ models with one API key
Sign up free →