E-commerce / Conversational Commerce
How a Conversational Commerce Business Turned Its AI Chat Assistant Into an Agent That Closes Sales
A Messenger-first commerce business now has an AI agent that closes sales on its own and handles 82% of conversations without a person, with sales holding level against an agents-only setup. A controlled test also replaced a gut-feel worry with data the team can trust.
- Client
- A Philippine e-commerce business that sells through Facebook and Messenger. Most of its buyers start by clicking a paid ad and finish by chatting with the brand.
Results
Measured: has a documented baselineObserved: a described change in how the work runs
- MeasuredAutonomous orders
After: ~8–9% of chats
Before: 0% (not designed to close)
- MeasuredChats with no human
After: 82% handled by AI agents end-to-end
Before: Every conversation gets handed over to human
- ObservedAI Agent visibility
After: Every decision traced
Before: Black box, no tracing
- MeasuredChats that blocked page
After: 23.6% (AI agent)
Before: 28.1% (sales team only)
On this page · 7 sections
The constraint
Then the goal changed. The business wanted the assistant to sell, not just qualify, so human agents would only handle conversations that needed a person. Under the original design, every sale still needed an agent. Agents spent their day answering the same questions about stock, sizes, legitimacy and delivery before they could take an order.
There was also no way to see inside the assistant. It had no tracing or observability. When a conversation went wrong, nobody could tell what the AI had understood, which tool it had called or why it had handed off. Problems came to light through unhappy customers, not alerts.
On top of that, a worry was spreading through the team: if the AI doesn't convert most chats, Meta will start sending the page worse leads. It came from an AI chatbot's answer during a busy product launch and was repeated on gut feel. With no data to check it against, it could have stopped the rollout.
The context
The business sells through Facebook and Messenger. Almost every buyer starts by clicking a paid ad and finishes by chatting with the brand. Orders are cash on delivery, so customers decide at the door whether an order becomes revenue.
Buyers write in Taglish, often send a photo instead of a product code, and expect a reply within seconds. If the reply is slow, they go back to the feed. As ad spend grew, chat volume grew with it.
An AI assistant was already running on the business's pages. It was built and implemented to the original design: greet customers, answer product questions, collect the first order details, then pass the conversation to a person who closed the sale. It did that job well.
How we approached it
We started by tracing the existing assistant end to end against the new goal. It was working as designed. It handed off once the first order details came in, and checkout conversations went to a part of the assistant built to answer questions, not to place orders. The design fit the old goal but couldn't reach the new one. So we treated this as a product decision, not a prompt tweak, and redesigned it as an AI agent with a new job: take a willing buyer all the way to a confirmed order, and hand off only when a person is actually needed.
We had to rebuild that picture from code, because the system recorded nothing about its own behavior. So before we changed what the AI agent did, we made sure we could see what it did.
We chose not to make the AI agent do everything. Photos it can't match confidently, voice notes and stickers still go to a person. And we didn't let it treat interest as intent. When the pilot showed it pushing checkout too early, we made it wait for a clear purchase signal, and accepted some short-term conversion loss to cut complaints and blocks.
We piloted on a single page with no human review, read real conversations every day and adjusted the AI agent based on what we saw.
On the lead-quality worry, we didn't argue either way. We broke it into hypotheses we could check:
- Does the AI agent lower the purchase rate?
- Does Meta deliver less while the AI agent replies?
- Do more people block the page?
- Does Meta rate the ad lower?
- Does the AI agent answer every chat Meta sends?
Data from the product launch showed no support for the worry. But the AI agent had been on the whole time, so it wasn't a clean comparison. So we designed a controlled test. One ad set ran with the AI agent on, then switched to the human sales team only, with the same ad and creative. The account's other pages served as a control. We checked Meta's numbers against the AI agent's own logs, hour by hour.
What we changed
The AI agent now closes sales. It collects the address, payment method and confirmation, then submits the order without a person stepping in. Returning buyers aren't asked again for details they've already given. Customers can send a photo, and the AI agent matches it to a real product after checking its confidence.
The AI agent works alongside the sales team. It steps back when a salesperson takes over, and it follows up on conversations that went quiet after a handoff so those leads don't drop.
The black box is gone. Every message, routing decision, tool call, extracted fact and handoff is traced and linked to its conversation, so any conversation can be replayed step by step. Error tracking and structured logs flag problems as they happen. Each stage from first message to checkout to order is measured, so drop-offs show up as numbers, not anecdotes.
The controlled test settled the worry. Purchase rate, cost per purchase and ad quality rankings were level with the AI agent on and off. Blocks went the other way from what the team feared: fewer customers blocked the page while the AI agent was replying (23.6% of chats) than with the sales team only (28.1%). The same work corrected a misleading dashboard: it had counted any reply from the sales team as a handoff. Measured properly, the AI agent hands off fewer than 1 in 10 conversations. The team now decides how far to take the AI agent from one source of truth.
Technology
The agent uses specialized sub-agents with tool calling for stock lookups, product matching, address parsing and order submission. It connects to Messenger, Meta Ads and the business's point-of-sale system, with tracing and monitoring built in.
What we'd tell you again
When the goal changes, change the design. The original assistant did the job it was built for. Zero autonomous orders was the expected result of that design, not a failure. Reaching a new goal meant redesigning the agent, not polishing its prompts.
You can't improve an agent you can't see. Without tracing, every fix to an AI agent is a guess. Observability came first, because it made each later change something we could check.
Turn gut feel into a test. A plausible worry repeated often enough starts to feel like a fact. Breaking it into hypotheses and testing them against a control replaced opinion with evidence.
The result
- Autonomous ordersMeasured
~8–9% of chats
- Chats with no humanMeasured
82% handled by AI agents end-to-end
- AI Agent visibilityObserved
Every decision traced
- Chats that blocked pageMeasured
23.6% (AI agent)
Have a similar constraint?
Tell us what's becoming harder as you grow. We'll help you figure out if it's worth solving.
Talk to ScaleXY