SeedSweep

Sample report

A shortened evaluation report for an AI support chatbot before a contract renewal.

This report comes from a real evaluation for one of our clients, a small company. We changed the details and numbers to keep the client anonymous.

Summary

System
A vendor's AI chatbot that answers customer support requests
Decision
Whether to renew the contract, due in six weeks
Sample
400 conversations from the last 90 days
Graded by
Two senior support staff members, using the company's help center articles

The vendor counted 78% of conversations as resolved. In our sample, 61% of customers had their problem solved, and most failures came from one outdated pricing page.

Results

One outdated pricing page caused more than half of the 156 failures

  • 244 problems solved
  • 87 quoted prices from a pricing page that changed in June (56% of failures)
  • 44 customer stopped replying without an answer, and the vendor counted it as resolved (28% of failures)
  • 25 login and account questions that needed a staff member (16% of failures)
Each square is one of the 400 conversations.
Share of the 400 conversations Reported by the vendor Measured by us
Counted as resolved by the vendor's rule 78% 78%
Customer's problem solved Not reported 61%
Counted as resolved, but the problem was not solved Not reported 17%
Answer quoted the outdated pricing page Not reported 22%
Customer came back within 7 days Not reported 14%

Recommendation

  • Renew for one year on the vendor's cheaper plan.
  • Make the renewal conditional on the vendor removing the outdated pricing page from the chatbot's sources.
  • Count a conversation as resolved only when the customer confirms it.
  • Send login and account questions to a staff member.
  • Rerun the evaluation after the vendor's next update.

You receive the 400 graded conversations, the scoring rules and the code to rerun the evaluation.