The Problem — The Bot Was Live, but Everything Got Worse
This client was a mid-sized e-commerce business receiving an average of 3,000 customer questions per day, covering everything from shipping status and product pricing to return policies and promotions.
Six months before we got involved, the client’s team had already deployed their first AI chatbot, hoping it would reduce the Customer Service team’s workload by around 50%.
The outcome was the opposite:
- Accuracy was only 60% — out of every 10 questions, 4 were answered incorrectly
- NPS dropped by 15 points — customers were frustrated by irrelevant responses and repetitive loops
- 40% of conversations had to be escalated to a human agent
- Agents were spending more time than before because they had to correct the bot’s wrong answers before they could actually help
- Cost per interaction increased by 23%, from ฿39 to ฿48 — the exact opposite of what had been promised
Total damage: ฿180,000/month spent on chatbot platform fees plus agent overtime required to clean up the resulting issues.
Here was the situation when we came in:
| Problem |
Number |
Impact |
| Low accuracy |
60% |
Customers received wrong answers 4 out of 10 times |
| High escalation rate |
40% |
Agents were just as overloaded as before |
| NPS decline |
-15 points |
Customers began leaving negative reviews |
| Increased cost |
+23% (฿48/interaction) |
Completely contrary to the business case |
| Wasted spend |
฿180,000/month |
Bot + unnecessary agent OT |
The client was close to removing the bot entirely by the time we started the conversation.
The Objective — Set Clear Targets Before Taking Action
We aligned on KPIs with the client before the project began. Every metric had to be measurable in real terms:
| Metric |
Before |
Target |
Why this number |
| Accuracy |
60% |
≥ 90% |
Below this, customers won’t trust the bot |
| Containment Rate |
60% |
≥ 80% |
The bot must resolve issues without human handoff |
| Cost per Interaction |
฿48 |
≤ ฿15 |
A 3x reduction from the current level |
| CSAT Score |
3.1/5 |
≥ 4.2/5 |
It needs to outperform the industry average |
| First Response Time |
45 seconds |
≤ 5 seconds |
E-commerce customers expect instant replies |
We told the client directly: even if we achieved only 3 out of these 5 targets, it would still be worthwhile. The goal was to make the bot genuinely useful, not just something nice to showcase.
The Approach — 3 Principles Before Touching a Single Line of Code
Before optimizing anything, we established 3 core principles:
1. Understand Before Answering
The bot had to truly understand customer intent before responding — not just detect keywords and fire back an answer.
2. Fail Fast to Human
If confidence is low, escalate immediately. A wrong answer causes more damage than no answer.
3. Learn from Every Conversation
Every conversation had to become training data so the system could improve week by week.
What we see again and again: many companies dump their entire FAQ into a chatbot and expect it to “understand.” Garbage in, garbage out. If the knowledge base is poorly structured, the bot will respond poorly too.
We designed the system in layers:
| Layer |
Role |
Why it matters |
| Intent Classification |
Identify what the customer is asking about |
Reduce misclassification, the primary cause of wrong answers |
| Knowledge Base (RAG) |
Retrieve relevant information to support responses |
Ensure fact-based answers instead of unreliable generation |
| Conversation Management |
Maintain context throughout the conversation |
Let customers ask follow-up questions without starting over |
| Human Handoff Intelligence |
Decide when to transfer to a human |
Escalating early at low confidence is better than answering incorrectly |
| Continuous Learning Pipeline |
Learn from every conversation |
Improve the system weekly without waiting for major updates |
The Process — 4 Phases in 12 Weeks
Phase 1: Conversation Audit (Weeks 1–3)
We pulled 15,000 real conversations from the previous 6 months for analysis. We didn’t guess, and we didn’t rely on stakeholder assumptions — we looked at the actual data.
Root cause analysis of incorrect answers:
| Cause |
Share |
Example |
| Intent Misclassification |
47% |
The customer asked about returns, but the bot interpreted it as a shipping inquiry |
| Knowledge Gap |
31% |
The bot had no information about a new promotion and responded with data that was 3 months old |
| Context Loss |
22% |
The customer asked a follow-up question, but the bot lost the context from the previous message |
The key finding: the biggest issue wasn’t that the model was unintelligent — it was that the training data did not reflect real customer questions. There was a massive gap between the company’s self-written FAQ and how customers actually asked for help.
Phase 2: Knowledge Restructuring (Weeks 4–6)
We completely rebuilt the knowledge base:
- Extracted 287 intent patterns from real conversations, compared to only 45 topics in the original FAQ
- Built a dynamic knowledge base synced with the client’s back-end systems — product pricing, stock status, and promotions updated in real time
- Added contextual variations for every intent, so phrases like “return item,” “request refund,” “send it back,” and “I don’t want it anymore” all mapped to the same intent
An example of a complex intent: a customer types, “I ordered yesterday and still haven’t received tracking.” This is not just a shipping status question — it also carries implied urgency. The bot had to respond with the status and, if the order had not yet shipped, escalate to the fulfillment team for verification instead of simply replying, “Please wait 3–5 business days.”
We organized the intents into 12 main groups and 287 sub-intents, based on real-world scenarios rather than what the marketing team imagined customers might ask.
Phase 3: Handoff Intelligence (Weeks 7–9)
We designed a confidence scoring system:
- Confidence ≥ 85% → The bot answers directly
- Confidence 60–84% → The bot answers, then asks the customer, “Did this answer your question?” If not → escalate to a human
- Confidence < 60% → Immediate handoff to a human, with a summarized context so the agent doesn’t have to ask everything again
The key detail: when escalating, the bot sends the agent a conversation summary + predicted intent. This meant agents did not have to start from zero, reducing handling time by 35%
Phase 4: Feedback Loop & Continuous Training (Weeks 10–12)
We built a pipeline where:
- Every escalated conversation is automatically reviewed
- Agents can flag incorrect bot answers with a single click
- The system retrains weekly on new data, without waiting for the dev team to perform manual updates
- A real-time dashboard shows accuracy trends to both the CS team and the dev team
One notable number: in the first week after launching this pipeline, agents flagged 127 incorrect answers. By week 4, that dropped to just 23 — the system learned very quickly when given high-quality feedback.
What made this pipeline work was simple: we made flagging as easy as possible. Agents didn’t need to write explanations — just click once and choose the error type. If they had needed to file long reports, nobody would have flagged anything.
The Results — But They Were Not Perfect from the Start
Critical Failure: Hallucinated Prices
In the first production iteration, we hit a serious problem: the RAG-based bot displayed incorrect product prices 8% of the time.
For general content, 8% inaccuracy might be tolerable. But in e-commerce, wrong pricing is a disaster. Customers saw a price from the bot and placed an order, only to find the actual price was different. Complaints started immediately. In some cases, customers demanded to buy at the price the bot had quoted.
We had to pause the system for 5 days to fix it.
This is something we want to say plainly: we made a mistake. We should have caught this before deployment. During testing, we used sample data where prices didn’t change often. But in production, promotions changed daily, and the system was surfacing stale cached pricing.
The fix: we added a verification layer for numeric data — price, inventory, shipping fees. All of it had to be queried directly from the database instead of being generated from context. Internally, we called this a “Structured Data Fallback”: if the question involves numbers, skip the generative path entirely and fetch the data directly.
After the fix: Price Accuracy reached 99.97% (only 3 errors in 10,000 cases, and every error was caused by database sync delay, not the bot)
Final Results After 12 Weeks
| Metric |
Before |
After |
Change |
| Accuracy |
60% |
94% |
+34 points |
| Containment Rate |
60% |
85% |
+25 points |
| Cost per Interaction |
฿48 |
฿15.40 |
-68% |
| CSAT Score |
3.1/5 |
4.3/5 |
+1.2 points |
| First Response Time |
45 seconds |
3 seconds |
-93% |
| NPS |
Down 15 points |
Recovered to +8 points vs. pre-bot baseline |
Full recovery |
Every target we set — all 5 were achieved
Cost Analysis — The Real Numbers That Proved ROI
| Item |
Before (Agent Only) |
After (AI + Agent) |
Difference |
| Agent cost (salary + OT) |
฿720,000 |
฿290,000 |
-฿430,000 |
| Chatbot platform cost |
฿0 |
฿55,000 |
+฿55,000 |
| Infrastructure & maintenance |
฿130,000 |
฿35,000 |
-฿95,000 |
| Total per month |
฿850,000 |
฿380,000 |
-฿470,000 |
Savings: ฿470,000/month = ฿5,640,000/year
The full development and optimization cost for the project was approximately ฿1,800,000, which meant payback in just over 4 months. After that, everything became net savings.
Just as importantly, the remaining agents were not reduced — they were upskilled to handle more complex cases that genuinely required human judgment, improving service quality for difficult cases as well.
Another notable metric: before optimization, each agent handled an average of 85 cases/day, most of them repetitive, low-complexity questions. After optimization, agents handled 32 cases/day, but they were cases that truly required judgment. As a result, the resolution rate for complex cases increased from 71% to 89% because agents had more time to focus on each one.
5 Lessons from This Project
1. Audit First, Optimize Later
“Auditing conversations before tuning the model is like a doctor diagnosing before prescribing medication.” If we hadn’t analyzed 15,000 conversations, we would never have known that 47% of the problem came from intent misclassification rather than model intelligence.
2. Teach the Bot How to Give Up
“If your chatbot gives wrong answers and keeps insisting, customers will be more frustrated than if there were no bot at all — teach it to say, ‘Let me connect you with an agent.’” A 100% containment rate is not the real goal. Customer satisfaction is.
3. Numbers and Money Must Never Hallucinate
In e-commerce, incorrect pricing creates real damage. A bot might get away with imperfect content responses, but price, stock, and shipping fees must always be queried directly from the source of truth. Never let a generative model guess.
4. The Feedback Loop Matters More Than the Starting Model
The first model you deploy doesn’t need to be perfect, but it must have a mechanism to improve every week. The system we delivered in week 12 was visibly smarter than it was in week 8 because the feedback loop was working continuously.
5. Agents Were Not Replaced — They Were Upgraded
The remaining CS team handled harder, higher-impact cases. No one lost their job, but everyone shifted from answering repetitive questions to solving real problems, which also improved team job satisfaction.
Timeline Summary
| Week |
Phase |
What happened |
| 1–3 |
Conversation Audit |
Analyzed 15,000 conversations to identify root causes |
| 4–6 |
Knowledge Restructuring |
Built 287 intent patterns + dynamic KB |
| 7–9 |
Handoff Intelligence |
Confidence scoring + agent context passing |
| 8 |
Critical Failure |
Price hallucination — system paused for 5 days |
| 10–12 |
Feedback Loop |
Continuous training pipeline + dashboard |
| 12+ |
Ongoing |
The system keeps learning weekly on its own |
What Still Needs Improvement
We are not claiming this system is perfect. There are still several areas that need further work:
- Voice Channel Support — currently the system supports only text-based chat. It is not yet integrated with the call center, which still receives 800–1,000 calls per day
- Proactive Outreach — the bot is still reactive, meaning it waits for customers to ask first. It cannot yet proactively notify customers when issues occur, such as out-of-stock products or shipping delays
- Multi-language Support — the current system supports Thai only. Customers who type in English or other languages still have to be escalated to a human, representing around 12% of all conversations
- Real-time Sentiment Analysis — there is still no system for detecting customer emotion during a conversation. If a customer is becoming frustrated, the bot should escalate faster than it does now
There is also the issue of deeper analytics. Right now, we can track whether the bot answered correctly or incorrectly, but we still cannot measure how much revenue each conversation generates. For example, if a customer asks the bot about a product, do they go on to make a purchase? To answer that, the system needs deeper integration with the e-commerce platform.
We have already aligned on a roadmap with the client for all 5 of these areas, and we expect them to be rolled out gradually over the next 6 months.
If You’re Facing a Similar Problem
Many organizations we speak with tell a similar story — they launched a chatbot, but the results fell short of expectations. Often, the problem is not the technology itself, but the thinking behind it and the data used to support it.
If you want to discuss AI strategy for customer service, or would like us to help audit your existing chatbot system, feel free to reach out. There is no charge for an initial consultation.
Contact the Enersys team
Sources
