Flagship case study · AI operations
AI Support Operations Copilot
A human-in-the-loop support triage system designed around a stricter question than whether an LLM can classify a ticket: what evidence would justify trusting the workflow, and what should happen when that evidence fails?
- Prototype complete
- Not deployed
- Tested with synthetic fixtures only
- Human review required
One request, step by step
From a support request to a reviewable draft.
Recorded example from the Python rule-based baseline, using synthetic test data.
-
A request arrives
Urgent: the synthetic demo service has been unavailable since ten minutes ago.
-
The system prepares triage
- Category
- Service disruption
- Priority
- High
- Security warnings
- None flagged in this example
-
A person reviews it
The output requires human review. The operator checks the classification and draft before deciding what to do.
No reply is sent and no service action is taken by the copilot.
Read the suggested reply
Draft for human review: We received your service disruption request. A support operator will review the details before any action is taken.
This is the baseline's fixed reply template.
- offline automated tests
- 94
- LLM calls / unvalidated outputs passed
- 308 / 0
- baseline → best measured snapshot, category / 33
- 14 → 33
- contract + human-review invariants on produced outputs
- 100%
Problem
The operational problem
Support requests are repetitive, but safe handling still depends on context, policy awareness and human judgment. A useful system must improve interpretation without quietly inventing facts, weakening deterministic safeguards or turning a draft into an autonomous customer-facing decision.
Can a semantic layer improve recall while the operational contract remains deterministic, reviewable and fail-closed?
Operating model
Safety is structure, not model behavior
Contract before model
Input validation runs before tokens are spent. Every result must satisfy a strict versioned output contract, including human_review_required: true.
Model adds, never subtracts
Deterministic security flags form a floor. Model-reported flags are merged as a union, so the semantic layer cannot remove a rule-based warning.
Degrade explicitly
The LLM attempt is bounded before fallback begins. The timed-out worker is not cancelled and may finish later; baseline execution and validation happen afterward, so this is not an end-to-end caller deadline.
Human review stays mandatory
Replies are suggestions only. Generated drafts are off by default and pass through a deterministic guard before they can enter the contract.
System flow
One contract across two classification engines
"human_review_required": true
- 01 Synthetic fixture untrusted input
- 02 Validate + scan deterministic
- 03 Classify baseline or optional LLM
- 04 Reply guard optional, off by default
- 05 Contract guard reject nonconforming output
- 06 Human review always required
Measured results
The semantic layer improved recall. The guard kept authority.
33 out-of-distribution paraphrases · snapshot from 2026-08-04
| Engine | Category | Security cases | Guard rejects | Cost |
|---|---|---|---|---|
| Keyword baseline | 14 / 33 | 0 / 5 | 0 | $0 |
| claude-haiku-4.5 | 33 / 33 | 4 / 5 | 0 | $0.033 |
| gpt-4o-mini | 31 / 33 | 5 / 5 | 0 | $0.003 |
| gemini-2.5-flash | 23 / 33 | 4 / 5 | 8 | $0.006 |
| deepseek-chat | 32 / 33 | 5 / 5 | 0 | $0.004 |
- The 33-case paraphrase set was intentionally out of distribution for the keyword rules; it is synthetic evidence, not a production accuracy estimate.
- Gemini produced schema-nonconforming output on eight paraphrase cases and two aligned cases. Every rejected call failed closed; no unvalidated output reached a caller.
- Missing-information judgments diverged sharply between models and labels. The case study reports per-field results instead of hiding that disagreement inside a single accuracy number.
Engineering-honesty arc
The findings that made the project stronger were not flattering
-
01
A review exposed a defect class
Substring matching made “not urgent” urgent and found “date” inside “update.” The fix applied word boundaries and negation handling across both Python and generated workflow JavaScript, then re-verified exact parity.
-
02
The labels lowered the score
A frozen-rubric audit corrected nine fields in eight paraphrase cases. The baseline's full-assertion score fell from 5/33 to 4/33; the lower number was published rather than patched away.
-
03
A documented fallback did not exist
Review found that the docs claimed a deterministic fallback that production code did not implement. The claim was corrected visibly, then a time-bounded wrapper was built and tested.
-
04
The reply guard measured false positives
Across 28 adversarial drafts, neither model produced an unsafe reply. The guard's unsafe-draft catch rate therefore remains unmeasured; all three rejections were safe refusals.
Build scope
What was implemented and verified
- Core
- Dependency-free Python classifier and validation
- Workflow
- Inactive, manual-only n8n artifact generated from reviewable sources
- Evaluation
- 44 aligned + 33 paraphrased synthetic cases
- LLM layer
- Anthropic SDK or OpenRouter behind the same contract
- Verification
- Python 3.10 / 3.13 CI, artifact drift and parity checks
Evidence trail
Inspect the records behind the claims
Results, corrections and limitations are committed beside the implementation rather than summarized from memory.