Project / 04
AI Customer Support Agent
Problem / Why I built it
Support needs evidence and boundaries
A useful support assistant must do more than generate fluent text. It needs to identify the request, ground its guidance in relevant history, and escalate when evidence or confidence is insufficient.
- Role
- Author
- Dates
- 2026
- Stack
- Python · Naive Bayes · BM25 · evaluation
- Status
- Complete
- Verified
- 95.95% escalation recall on 150 held-out cases
- Repository
- View repository ↗
A completed offline support agent built from real AmazonHelp conversations. It predicts customer intent, retrieves relevant historical cases, drafts grounded guidance, and applies deterministic guardrails before deciding whether to escalate.
Architecture / Decision path
From customer message to a reviewable decision.
- 01Customer message
- 02Intent classifier
- 03BM25 retrieval
- 04Reply guidance
- 05Route guardrails
02 / Dataset
Built around real conversations
The pipeline audits 2.8 million Customer Support on Twitter records and isolates 154,976 direct customer-to-AmazonHelp pairs. The repository includes a redacted 5,600-case subset for local retrieval and training.
03 / Implementation
What is working now
- 01
Eight data-derived intents covering delivery, returns, orders, payments, accounts, digital services, product issues, and unclear requests
- 02
Naive Bayes intent classification and a local Okapi BM25 retrieval index
- 03
Historical-response-conditioned reply guidance with an AUTO_HANDLE or ESCALATE decision and reason
- 04
Deterministic confidence, evidence, and safety guardrails that do not depend on an LLM judge
- 05
CLI, local demo UI, automated tests, reproducible pipelines, and B0/B1/B2 evaluation tooling
04 / Decisions
Engineering choices
Run offline first
The core prototype needs no API key or package installation, making its baseline behavior reproducible and easy to inspect.
Protect the test boundary
Conversation roots are split chronologically, duplicates are removed across partitions, and retrieval uses training data only.
Keep humans in the loop
The project refuses to present machine suggestions as human ground truth. Uncertain or risky requests are routed for review.
Measured results / B2
- 01
- 53.33% intent accuracy
- 02
- 95.95% escalation recall
- 03
- 4.0% auto coverage
Final verdict
- All 200 labels passed validation and the 150-case held-out evaluation completed.
- All six B2 automatic decisions were unsafe under the route labels, so automatic handling remains disabled.
- Human reviewers passed 20/20 B2 replies; the judge passed 3/20, exposing meaningful evaluator disagreement.
Future iteration
Improve retrieval and routing on the observed failures, add current policy evidence, and run a larger shadow evaluation before enabling automatic handling.
What I learned
Measure the route before trusting the reply.
- All 200 labels passed validation and the 150-case held-out evaluation completed.
- All six B2 automatic decisions were unsafe under the route labels, so automatic handling remains disabled.
- Human reviewers passed 20/20 B2 replies; the judge passed 3/20, exposing meaningful evaluator disagreement.