Support Email Automation with Human Escalation
Automate 500 support emails a day currently handled by ten agents - classification, routing, drafted replies, calibrated confidence, and an escalation policy that keeps humans where they matter.
Last updated:
Ten agents handle 500 customer-support emails a day and the client wants that automated with a human-in-the-loop escalation path. The model is the easy part - at this volume any competent LLM drafts a decent reply. The design is the escalation policy: what sends without a human, what an agent approves, and what never leaves the queue. Get that wrong in the permissive direction and you multiply complaints; get it wrong in the conservative direction and you have built an expensive autocomplete.
“The design is the escalation policy, not the model. Decide what auto-sends, what a human approves, and what never leaves the queue - then measure deflection against reopen rate and CSAT together, because deflection alone can be raised by sending confident nonsense.”
Clarifying Questions (Ask These First ~5 min)
| Question | Why it matters |
|---|---|
| What is the intent mix, and how concentrated is it? | If five intents are sixty percent of volume, release one is those five and nothing else |
| May a reply ever be sent without a human seeing it? | The single biggest scope question in the room |
| Current handle time and first-contact resolution? | Without a baseline there is no way to prove value later |
| Do agents use canned responses today? | Existing macros are labelled training data, already curated |
| Which systems must be read to answer - orders, billing, CRM? | Most answers need account context, not just the text of the email |
| What is the SLA, and who is accountable when the reply is wrong? | Sets confidence thresholds and the audit trail |
| How many languages? | Changes model choice and the reviewer staffing plan |
Architecture (Draw This)
Say this as you draw it
Key Components (30-second pitch each)
- 1Classification first, generation second - Routing is where reliability lives - a wrong route cannot be rescued by a good draft. A small fine-tuned classifier is cheaper, faster and more measurable than an LLM call per email, and it gives you a confidence number that means something.
- 2Three lanes, with sensitive intents hard-wired - Auto-send, agent-approved and human-only. Complaints, cancellations, legal and regulatory topics go to the human lane regardless of model confidence, because that routing is a business rule and not a model decision.
- 3Grounded replies, never invented policy - Retrieval over the help centre and past resolved tickets, with the reply constrained to what the sources support. The failure that destroys trust is a confident, well-written statement of a refund policy the company does not have.
- 4Confidence that is actually calibrated - Raw model probabilities are not calibrated. Fit calibration on a held-out set so that a 0.9 means roughly nine in ten correct, otherwise every threshold you set is arbitrary and the auto-send lane is a coin toss with extra steps.
- 5Explicit escalation triggers - Negative sentiment, a second contact on the same thread, an explicit request for a human, legal or media keywords, and account value above a threshold. These fire independently of the classifier and override it.
- 6The feedback loop is the product - Agent edit distance is the cheapest quality signal you will ever get, and it arrives on every assisted reply for free. Combined with reopen rate and CSAT it tells you which intents are ready for the auto-send lane.
What Separates a Strong Answer
- 1Calibrate classifier confidence - A strong candidate fits calibration on a held-out set so that confidence scores are meaningful, preventing arbitrary thresholds and unreliable auto-sends.
- 2Implement explicit escalation triggers - They define clear rules for when an email should be escalated to a human, independent of model confidence, ensuring sensitive cases are handled appropriately.
- 3Ground replies in real sources - By using retrieval over the help center and past tickets, a strong design prevents the generation of misleading or incorrect policy statements.
- 4Leverage feedback loop effectively - A strong candidate uses agent edit distance, reopen rates, and CSAT as continuous feedback to improve system accuracy and identify intents ready for auto-send.
The Three Lanes
| Lane | Criteria | Realistic share at maturity |
|---|---|---|
| Auto-send | Top intents, calibrated high confidence, no commitment made (order status, reset, opening hours) | 20-35% |
| Agent-approved | Known intent, context resolved, but the reply commits the business to something | 40-50% |
| Human only | Complaint, legal, cancellation, unknown intent, or a repeat contact | Remainder |
Escalation Triggers
| Trigger | Why it overrides the classifier |
|---|---|
| Negative sentiment or profanity | An automated reply to an angry customer multiplies the complaint |
| Second contact on the same thread | The first automated answer already failed |
| Explicit request for a human | Never override this - it destroys trust and shows up directly in CSAT |
| Legal, regulator, media or chargeback keywords | One wrong answer here costs more than every deflection saved that month |
| Account value above threshold | A business rule about who gets a person, not a model decision |
3 Biggest Risks
- 1Confident replies on money topics - Refunds, delivery dates and entitlements are where a wrong sentence costs real money. Mitigated by keeping every commitment-bearing intent in the approved lane, plus a forbidden-phrase check that blocks promises the sources do not support.
- 2Measuring deflection in isolation - Automation rate rises happily while reopen rate and CSAT quietly fall. Mitigated by reporting deflection, reopen rate, CSAT and first-response time as one set - a lift in the first that moves the others the wrong way is not a win.
- 3The staffing question nobody asks - Ten agents and a system that drafts most replies changes what those ten people do. The honest answer is redeployment into the human lane and quality review rather than headcount reduction; saying that plainly is better than letting the client discover it in month three.
Google Stack: Gmail or Zendesk webhook → Pub/Sub → Cloud Run classifier (fine-tuned small model) → Vertex AI Search over the help centre and resolved tickets → Gemini drafting → Firestore for thread state → Looker for the feedback loop
Azure Stack: Exchange Online + Logic Apps → Azure AI Language custom text classification → Azure AI Search over the help centre → Azure OpenAI drafting → Dynamics 365 Customer Service as the agent surface → Application Insights for edit distance and lane metrics
AWS Stack: SES or WorkMail → SQS → Comprehend or a SageMaker classifier → Bedrock Knowledge Bases over the help centre → Bedrock Claude drafting → Amazon Connect Cases → QuickSight dashboards
Other Options: Zendesk AI, Intercom Fin or Front AI if buying beats building - at 500 emails a day that is a genuine conversation and the interviewer will respect it being raised | Self-hosted: a small classifier plus Llama and Qdrant behind the existing helpdesk API.
Frequently asked questions
What accuracy do you need before enabling auto-send?
State it as business cost rather than F1. If a wrong auto-reply costs a reopen plus a CSAT hit, and the intent is a known share of volume, the threshold is computable. In practice, start with zero auto-send, run fully assisted for a few weeks, and let the measured per-intent accept rate choose the first lane to open.
Where does training data come from on day one?
The last year of resolved tickets is already a labelled corpus - the agent's chosen macro or disposition is the intent label and the sent reply is the target. Clean it first: closed-without-reply, spam and duplicate threads poison both the classifier and the retrieval corpus.
Fine-tune, or prompt with retrieval?
Prompt with retrieval for generation, and fine-tune the classifier. Revisit generation fine-tuning only once edit distance shows the model is consistently wrong in the same direction, since a tone problem is usually a template problem rather than a weights problem.
500 a day is small. Why all this machinery?
Cost is trivial at this volume and you should say so. The machinery exists for governance - routing, thresholds, audit and the feedback loop - not for scale. If the client only wanted drafts in an inbox, that is a much smaller project and worth naming as an option.
How do you handle attachments?
Classify the attachment type first, extract with OCR where it matters (receipts, screenshots of errors), and treat anything unparsed as a reason to route to a human rather than to reply around it. Attachments are also the most common injection vector in inbound mail.
What does success look like at six months?
Not a percentage automated. A fall in median first-response time, stable or rising CSAT, flat reopen rate, and agent time visibly shifted into the human lane. Report those four together, because any one of them alone can be gamed.