The failure mode of chatbot customer service isn’t that the AI can’t answer. It’s that the AI answers when it shouldn’t have.
In February 2024, the British Columbia Civil Resolution Tribunal ruled against Air Canada after its website chatbot told a customer he could apply for bereavement fares retroactively, advice contradicted by the airline’s own policy pages. Air Canada argued the chatbot was effectively a separate entity responsible for its own statements. The tribunal rejected that, holding the airline responsible for all information on its website and finding it had not taken reasonable care over accuracy (Moffatt v. Air Canada, 2024 BCCRT 149).
In April 2025, the pattern repeated in a different industry. Cursor’s front-line AI support bot informed users that the product restricted logins to a single device. No such policy existed; co-founder Michael Truell confirmed publicly that users were “of course free to use Cursor on multiple machines.” The actual trigger, by Truell’s account, was a session security change that likely caused a race condition logging users out across devices; the bot filled the explanatory gap with an invented policy. Customers canceled subscriptions over a rule that had never been written.
Neither incident was a model problem. Both were escalation and governance problems; an AI operating in territory it should never have entered, with no threshold to stop it and no review to catch it. Getting live chat automation and AI helpdesk ticketing system rules right is mostly a matter of deciding, in advance and in writing, where the boundary sits. This is how you design it properly.
KEY TAKEAWAYS
- AI should fully handle low-risk, documented issues, assist with judgment-based tickets, and immediately escalate sensitive, legal, emotional, or high-risk situations.
- Proper escalation needs confidence thresholds, restricted topics, customer-requested handoff, sentiment signals, and loop detection working independently.
- A quality handoff gives the agent full context, including the transcript, summary, customer history, escalation reason, confidence score, and previously taken actions.
- AI support accuracy depends on clear, up-to-date, well-organized documentation that covers common questions, exceptions, and known issues.
- Treat AI conversations like agent conversations by sampling, scoring, tracking paired metrics, disclosing AI use, and reviewing both missed and unnecessary escalations.
- Start with ticket analysis and a narrow pilot, clean the relevant knowledge base, test in suggest-only mode, and expand only when quality holds.
Where AI should handle, and where it must escalate
Start by sorting your contact drivers into three tiers. Do this from your own ticket data, not from a template. Export your last 90 days, group by reason, and rank by volume.
Tier 1: automate fully
Characteristics: high volume, single correct answer, documented, low emotional stakes, reversible.
- Order status and delivery timelines
- Returns and refund policy questions (the policy, not the exception)
- Password resets, login troubleshooting, MFA setup
- Business hours, pricing, plan comparisons, feature availability
- Invoice retrieval, receipt resending, updating payment details
- How-to questions with an existing knowledge base article
For e-commerce, this tier is commonly 60–75% of volume. For B2B SaaS it’s more often 30–45%.
Tier 2: AI-assisted, human-approved
Characteristics: the AI can draft or diagnose, but a person should own the outcome.
- Technical troubleshooting beyond documented steps
- Billing disputes and proration questions
- Partial refunds and goodwill gestures
- Bug reports needing reproduction
- Anything requiring a judgment call about an exception to policy
Here the AI summarizes the thread, surfaces relevant history, drafts a reply, and suggests next actions. The agent edits and sends. Measured properly, this is often where most of the economic value sits; it compresses handle time across the entire remaining queue rather than removing a slice of it.
Tier 3: human only, escalate immediately
Characteristics: irreversible, legally consequential, emotionally charged, or reputationally exposed.
- Cancellations, downgrades, and retention conversations
- Complaints, threats of legal action, regulator mentions
- Account security incidents and suspected fraud
- Bereavement, medical, accessibility, and hardship situations
- Anything touching a vulnerable customer
- Data subject requests: deletion, access, portability
- Media inquiries
Write this list down as an explicit restriction list in your platform configuration. The Air Canada case sat squarely in Tier 3: a bereavement fare question is exactly the category where a confidently wrong AI answer becomes both a customer harm and a legal liability.
Designing escalation rules that actually fire
Three triggers, all of which should exist independently.
1. Confidence threshold. The AI escalates when its own certainty falls below a set level. Start conservative, escalate readily, and loosen only where you have evidence. The instinct to tune for a higher containment rate is exactly what produces the Cursor failure: a bot that would rather assert something than admit it doesn’t know.
2. Topic and action restrictions. Hard rules that override confidence entirely. Regardless of how certain the model is, it does not discuss legal matters, does not process refunds above a threshold, does not delete accounts, does not speculate about outages. Restriction lists are more reliable than confidence scores because they don’t degrade as your content changes.
3. Explicit customer request. “Talk to a human” must work on the first attempt, from any point in the conversation, without an interrogation first. This is increasingly a regulatory expectation, not just a courtesy. Gartner projects that regulatory changes will increase assisted-service volume by 30% by 2028, partly because rules are trending toward mandating easy access to a person.
Add two more signals if your platform supports them:
Sentiment escalation: frustration, repetition, or profanity detected in the customer’s messages. A customer rephrasing the same question three times is a customer who needs a person.
Loop detection: if the AI has given substantially the same answer twice, it isn’t going to work the third time. Escalate.
What a good handoff carries
An escalation that loses context is barely better than no escalation. When the transfer fires, the agent should receive, already attached:
- The full transcript, not a summary alone
- A generated summary of what the customer wants and what’s been tried
- The customer record: plan, order history, previous tickets, lifetime value
- The escalation reason and the AI’s confidence score
- Any actions the AI already took
The reason this matters is measurable. Zendesk’s 2026 CX Trends research, over 11,000 respondents across 22 countries, found 74% of consumers are frustrated when they have to repeat information, and 81% want conversations to continue without backtracking. A handoff that forces the customer to start over converts a rescued interaction back into a bad one.
Two rules for the transfer itself: tell the customer plainly that they’re being moved to a person and roughly how long it will take, and never make them re-enter details they already gave. If nobody is available, say so and set an expectation rather than leaving them watching a typing indicator.
Knowledge base quality: the ceiling on everything
Your AI cannot be more accurate than your documentation. Every hour spent on knowledge base quality returns more than an hour spent tuning prompts.
Audit for contradictions first. If two pages disagree about your returns window, the AI will confidently pick one, and it may not be the current one. Search for your top 20 policy terms and reconcile every result.
Write for retrieval, not for reading. One topic per article. A specific title that matches how customers phrase the question, not internal terminology. The direct answer in the first paragraph, before the context and caveats. Include the phrasings customers actually use; pull them from ticket subject lines.
Document the exceptions. Most escalations are exception cases. “Returns accepted within 30 days” is easy; “returns on sale items, gift purchases, and international orders” is where the AI fails. If the exception isn’t written down, it cannot be handled.
Close the loop from failures. Every escalation caused by missing knowledge is a content ticket. Review escalation reasons weekly, and route the gaps to whoever owns documentation. This single feedback loop does more for resolution rate over a quarter than any configuration change.
Date and own every article. A stale article is worse than a missing one, because the AI will use it. Assign an owner and a review date to each.
QA and monitoring: auditing what the AI says
Treat AI conversations exactly as you’d treat a new agent’s: sampled, scored, and coached.
Sample deliberately. A random 2–5% of AI-resolved conversations weekly, plus 100% of a targeted set: every reopened AI resolution, every conversation with CSAT of 1 or 2, every escalation from a restricted topic, and every conversation touching money.
Score on five dimensions.
- Accuracy: Was the information correct against current policy?
- Completeness: Did it fully answer, or partially?
- Tone: Did it match your brand voice, and read appropriately for the situation?
- Escalation judgment: Should this have been handed off, and was it?
- Compliance: Was AI involvement disclosed, was PII handled correctly, were required disclaimers present?
Watch the metric pairs, never single numbers. Resolution rate alongside CSAT. Containment alongside abandonment. Automation percentage alongside reopen rate. Any of these rising while its partner falls means the AI is closing conversations rather than solving problems. Zendesk found only 21% of low-maturity organizations track automation success rates at all, against 66% of high-maturity ones. The measurement gap is the maturity gap.
Disclose AI involvement. Since 2 August 2026, Article 50 of the EU AI Act has required that people be informed they’re interacting with an AI system, at the latest at the first interaction, in a clear and distinguishable manner that meets accessibility requirements. Faint labels and briefly flashing notices do not satisfy it. Customer expectation runs the same direction: 95% of consumers in Zendesk’s research expect an explanation of AI-driven decisions, and 79% want it in plain language. Notably, Cursor’s own response to its incident was to commit to labeling AI-assisted support replies. The fix for a trust failure was transparency.
Build the escalation-quality review. Once a month, review a sample of escalations for the opposite error: conversations the AI handed off that it could have handled. Over-escalation is a real cost, just a less visible one than over-confidence.
Operational playbook: day 1 to week 4
Day 1–3 – baseline and scope. Export 90 days of tickets and rank contact drivers by volume. Assign each to Tier 1, 2, or 3. Record current FRT, AHT, CSAT, first-contact resolution, and reopen rate. Pick two or three Tier 1 drivers for the pilot. Write the Tier 3 restriction list.
Day 4–10 – knowledge base sprint. Audit and fix articles covering the pilot drivers only. Reconcile contradictions, document exceptions, rewrite titles to match customer phrasing. Resist the urge to fix everything.
Week 2 – suggest-only mode. The AI drafts; agents approve or reject before sending. Log every rejection with a reason. This gives you a real accuracy read at zero customer risk, and the rejection log is your knowledge base backlog. Target: agents accepting 70%+ of drafts unedited before going live.
Week 3 – go live, narrow. Enable autonomous resolution on the pilot drivers only, with a conservative confidence threshold, the Tier 3 restriction list active, one-click human handoff visible throughout, and clear AI disclosure. Start the QA sampling cadence immediately.
Week 4 – first review and expand. Check resolution rate by driver, escalation reasons, reopen rate, and CSAT split. Fix the knowledge gaps the escalation log surfaced. Add one new contact driver, one, not five. Then repeat the week-4 review every two weeks, expanding scope only when the current scope holds both resolution rate and CSAT.
The teams that get this right treat the AI as a team member in onboarding: narrow responsibilities, close supervision, clear rules about when to ask for help, and expanded scope earned through demonstrated performance.
FAQs
When should an AI chatbot escalate to a human?
On any of five signals: model confidence below your threshold, a restricted topic or action, an explicit customer request for a person, detected frustration or repeated rephrasing, or the same answer already given twice without resolution. Restricted topics should override confidence entirely; cancellations, complaints, legal and regulatory mentions, security incidents, bereavement and hardship, and data subject requests belong with a human regardless of how certain the model is.
What is a good escalation rate for AI customer support?
There isn’t a universal target, because it depends entirely on ticket mix. Judge it against CSAT and reopen rate instead: an escalation rate that looks high but produces strong CSAT and few reopens is working correctly, while a low escalation rate paired with rising reopens and falling CSAT means the AI is over-reaching.
How do you stop an AI support bot from making things up?
Ground it in a curated knowledge base rather than open generation, set a conservative confidence threshold, maintain hard topic and action restrictions, and require citations to source articles so agents can verify. Then audit sample conversations weekly and score for accuracy against current policy. The Cursor incident happened because a bot filled a knowledge gap with an invention instead of escalating; the structural fix is making “I don’t know, let me get someone” a first-class outcome.
Who is liable if a chatbot gives wrong information?
The business. Moffatt v. Air Canada established that a company is responsible for information its chatbot provides on its own website; the tribunal explicitly rejected the argument that the chatbot was a separate entity, and found the company had failed to take reasonable care over accuracy. Treat AI output as your published statements, because legally that is close to how it is treated.
Should customers be told they’re talking to AI?
Yes, and in the EU it’s mandatory. Article 50 of the EU AI Act has applied since 2 August 2026 and requires clear, distinguishable disclosure at or before the first interaction. Beyond compliance, disclosure improves outcomes. Customers calibrate their expectations and escalate sooner when the AI can’t help, instead of feeling misled.
Sources
- Moffatt v. Air Canada, 2024 BCCRT 149 (British Columbia Civil Resolution Tribunal, 14 February 2024)
- The Register, “Cursor AI’s own support bot hallucinated its usage policy,” 18 April 2025
- Zendesk CX Trends Report 2026 (11,000+ respondents across 22 countries, June 2025)
- European Commission, “Commission starts enforcing AI Act rules and new transparency requirements on 2 August”; EU AI Act Article 50 and the Commission’s Article 50 transparency guidelines, adopted 20 July 2026
- Gartner, “Gartner Predicts GenAI Cost Per Resolution for Customer Service Will Exceed Offshore Human Agent Costs by 2030,” 26 January 2026
