Save a library of prompts you care about (opening hours, refund policy, pricing edge cases, tricky escalation triggers) and re-run them against the bot in one click. Each run gets a Pass, Fail, or Pending review verdict so you can catch regressions before your customers do.
How to get there: Go to Setup → Chatbot in the top menu → click your chatbot → Advanced → Test Automation in the sidebar.
A test case is a saved prompt plus an expected response. When you run it, the bot answers as it would for a real visitor, and Boei compares the answer to what you expected.
Each case tracks:
You can run a single case with the Test button on its row, or run all active cases with Run selected. Each case streams through the same pipeline your real widget uses, so tool calls, lead flows, and knowledge-base lookups all execute for real.
If a case triggers the Contact Form lead flow, you'll see a green "Contact Form Lead Flow opened" indicator next to the response. That confirms the bot correctly handed off to the form.
By default Boei uses a small AI model to compare the actual answer to the expected description and assign a verdict of Pass or Fail with a one-line reason. This is fast and covers 90% of real cases without eyeballing every response.
Toggle it off with the switch in the toolbar if you'd rather review every run yourself. When the judge is off:
The toggle applies to both UI runs and CLI runs (php artisan chatbot:run-test-cases), so a shared setting keeps human and machine paths in sync.
| Verdict | Meaning |
|---|---|
| Passed | Judge (or you) marked the answer correct |
| Failed | Judge (or you) marked the answer wrong |
| Pending review | Judge is off, waiting for you to Pass or Fail |
| Error | The bot couldn't answer (rate limit, low credits, upstream failure) |
| Pending | Case hasn't been run yet |
The verdict banner sits at the top of each case card in the case's status color, so you can scan a long list for red rows fast.
If the bot called any tools during the run (search knowledge base, open lead form, custom webhooks, etc.), a small chevron on the metadata footer expands the full tools list. Hidden by default to keep the card scannable.