Resources · 3 min read
How to test a chatbot before launch: the scenario bank
A demo that went well isn't evidence. Evidence is a bank of hard conversations the agent passes before every change — one that grows with every real mistake.
The demo isn't evidence
The demo goes well because the demo was chosen: six reasonable questions, a fresh catalog, nobody insisting. Production is a customer asking at ten at night about a discontinued product, three times, with different spelling. The demo shows what the agent does well; testing means going after what it does badly — before a customer finds it.
What a scenario bank is
A list of hard conversations with an expected outcome: product out of stock, a size that doesn't exist, a price that changed yesterday, a customer asking for a discount, a forbidden topic. Each scenario defines what must happen — answer with the data, ask to narrow down, or escalate to a person. The criterion isn't “it answered nicely”: it's pass or fail, no adjectives.
When it runs: before every change
A change to the instructions, the catalog or the model: the whole bank runs again before the change touches production. The agent that passed yesterday can fail today without anyone touching “anything” — the model updated underneath, the catalog moved. Running the bank turns that surprise into a test result: it fails on your screen, not in a customer's WhatsApp.
A good bank is harvested, not written
The first one is written by hand: twenty or thirty scenarios drawn from how your customers actually buy. The good one is harvested later — every real conversation that went wrong becomes a new scenario, and that mistake can't come back without setting off an alarm. Three months in, the bank is the memory of everything the agent is no longer allowed to forget.
The business defines the expected outcome
What should happen when someone asks for a discount isn't decided by whoever programmed the agent: it's decided by whoever owns the money. Writing the scenarios forces that conversation — and the document it produces is worth as much as the system. And if you're evaluating vendors, ask for one thing: the scenario bank and its latest run. If they don't exist, you're buying a demo.
Rather go through it on your own case?
A 15-minute audit of your real operation. You leave with an honest read on what to automate first, and what to leave alone.
Book a 15-min audit