Sample report
A shortened evaluation report for an AI support chatbot before a contract renewal.
This report comes from a real evaluation for one of our clients, a small company. We changed the details and numbers to keep the client anonymous.
Summary
- System
- A vendor's AI chatbot that answers customer support requests
- Decision
- Whether to renew the contract, due in six weeks
- Sample
- 400 conversations from the last 90 days
- Graded by
- Two senior support staff members, using the company's help center articles
The vendor counted 78% of conversations as resolved. In our sample, 61% of customers had their problem solved, and most failures came from one outdated pricing page.
Results
One outdated pricing page caused more than half of the 156 failures
- 244 problems solved
- 87 quoted prices from a pricing page that changed in June (56% of failures)
- 44 customer stopped replying without an answer, and the vendor counted it as resolved (28% of failures)
- 25 login and account questions that needed a staff member (16% of failures)
| Share of the 400 conversations | Reported by the vendor | Measured by us |
|---|---|---|
| Counted as resolved by the vendor's rule | 78% | 78% |
| Customer's problem solved | Not reported | 61% |
| Counted as resolved, but the problem was not solved | Not reported | 17% |
| Answer quoted the outdated pricing page | Not reported | 22% |
| Customer came back within 7 days | Not reported | 14% |
Recommendation
- Renew for one year on the vendor's cheaper plan.
- Make the renewal conditional on the vendor removing the outdated pricing page from the chatbot's sources.
- Count a conversation as resolved only when the customer confirms it.
- Send login and account questions to a staff member.
- Rerun the evaluation after the vendor's next update.
You receive the 400 graded conversations, the scoring rules and the code to rerun the evaluation.