How to Test a Business Chatbot Before Putting It Online

You don't test an automated assistant by reading a few conversations and concluding it "answers well." It answers well to the questions you ask, because you already know the answers and use your own wording. Real questions arrive poorly written, incomplete, about cases nobody had anticipated. Testing exists to run into them before a customer does.

The method doesn't require any special tools: a list of questions, an expected answer for each, an outcome marked down by hand. What it does require is the discipline to redo it every time something changes.

Test questions come from real requests

Questions made up in a meeting tend to resemble the answers the system already knows. It's a systematic flaw: whoever writes them already knows what's in the documents.

• Start from the requests you've actually received: support messages, requests from the website, notes from whoever answers the phone. Strip out identifying personal data and keep the question as it was written, errors included.
• Cover the ordinary and the rare: not just the ten most frequent questions, but also the ones that come up once a month and give whoever answers a hard time.
• Include the questions that shouldn't be answered: prices to be negotiated, contractual matters, requests concerning third parties.
• Add the poorly phrased questions: half a sentence, two questions bundled together, a wrong detail stated as fact.

A hundred questions collected this way are worth more than a thousand made up at a desk, because they reflect the real distribution of requests, including the awkward ones.

The tests nobody feels like running

• Questions with a false premise: "since you also do this…" when you don't. The system needs to correct it, not go along with it.
• Attempts to get it to ignore the instructions it was given, or to make it say things the company would never say. This isn't a theoretical case: it happens on public-facing systems.
• Questions on related but out-of-scope topics, to check that the boundary is recognized.
• Requests that ask for a commitment: a discount, a date, a guarantee.

What counts as an acceptable answer

Before testing, you need to decide what counts as correct, otherwise the judgment shifts with whoever is reading and what time of day it is.

1. The content: which information needs to be there for the answer to be useful, and which shouldn't appear.
2. The source: which document it needs to come from. A correct answer taken from the wrong document is a problem postponed, one that shows up when that document changes.
3. Holding back: for certain questions the correct answer is to say it doesn't know and point to another way forward. This needs to be written down as an expected outcome, otherwise whoever's grading it marks it as a failure.
4. The form: a correct answer that's three times longer than it needs to be, inside a chat, is an answer nobody reads to the end.

The evaluation sheet

Tests are logged in a single table, which is used to compare two versions and to show others how things are going. The rows below are illustrative.

The severity column is the one that drives decisions. An inaccuracy on a small detail doesn't carry the same weight as a commitment made in the company's name: the scale needs to be agreed on beforehand, and the minimum distinction is between imprecise answers, wrong answers, and answers that create an obligation or put someone at risk.

Beyond errors: what's worth counting

• Times it held back: how often the system stopped when it should have, and how often it stopped even though it could have answered. These are two opposite flaws and are fixed in opposite ways.
• Handoffs to a person, broken down by topic: they show where public information is missing.
• Answers that are correct but useless: accurate, generic, and leave the reader exactly where they started.
• Behavior on exceptions: what happens when a connection doesn't respond or a document is unreachable. A system that improvises in that case is more dangerous than one that stops.

Re-run the tests after every change

The list of questions is most useful after the first time. Behavior changes when one of three things changes, and at least one of them will.

1. The sources: an updated document, a rewritten page, a new price list.
2. The instructions: a rule added to fix one case often breaks two others, and it's the most frequent cause of sudden regressions.
3. The underlying model, which can be updated without anyone at the company having asked for it.

That's why the same questions need to be run through periodically and before every release. It's tedious work, and it's the only thing that stops fixes from canceling each other out.

When the tests say don't publish

• If serious errors remain. An answer that commits the company or touches on safety has no acceptable frequency.
• If the system only works with well-written questions: customers don't write well.
• If the sources contradict each other. In that case the flaw isn't the system's: it's exposing a contradiction that already existed, and it needs to be fixed first.
• If there's no handoff to a person that actually works.

What this guide doesn't cover

This guide covers the quality of the answers: how to test it, how to log it, when it's ready to publish. The escalation path to a person — when to stop and what to hand over — is designed separately. And the economic evaluation of the project, that is, whether and how much it's worth it, follows different criteria and can't be derived from the number of correct answers.