Sample report
A shortened evaluation report for an AI support chatbot before a contract renewal.
This report comes from a real evaluation for a small company. We changed the details and numbers to keep the company anonymous.
Summary
- System
- Vendor AI chatbot answering customer support tickets
- Decision
- Renew the contract in six weeks
- Sample
- 400 conversations from the last 90 days
- Graded by
- Two senior support staff, checked against help center articles
The vendor reports 78% of conversations resolved. On our sample, 61% of customers had their problem solved, and most failures came from one outdated pricing page.
Results
| Measure | Vendor | Our test |
|---|---|---|
| Marked resolved | 78% | 78% |
| Problem solved | - | 61% |
| Answer cited an outdated article | - | 9% |
| Ticket reopened within 7 days | - | 14% |
Error causes
| Cause | Share of failures |
|---|---|
| Answers from a pricing page that changed in June | 56% |
| Marked resolved when the customer stopped replying | 28% |
| Account-access questions that needed a person | 16% |
Recommendation
- Renew for one year on the vendor's cheaper plan.
- Make the renewal conditional on retiring the outdated pricing content.
- Count a conversation as resolved only when the customer confirms it.
- Route account-access questions to a person.
- Rerun the test set after the vendor's next release.
You keep the 400 graded cases, the scoring method and the scripts to rerun it.