SeedSweep

Questions

The work

What does SeedSweep do?

We evaluate AI systems and data, whether you bought them or built them, and show you how often they are right, where and why they go wrong and what to change. Where the evidence supports it, we then help you improve them and find opportunities to increase your revenue.

Why not let the vendor or our own team test it?

Vendors measure their own systems on tests they choose, and internal teams are often grading work they built or bought. We have no stake in the result, and we agree on the grading rules with you before we start.

What do we get at the end?

You receive the share of your real questions and tasks the system gets right, the causes behind the errors, a recommendation for your decision and the graded examples, scoring rules and code to rerun the evaluation after every update. We can also set up a dashboard or connect the evaluation to your systems so you can monitor the results.

Do you also build systems?

Yes, after the evaluation and only where the evidence supports it. When an acquirer, investor or auditor hires us to evaluate another company's AI, our work ends after the evaluation. When we compare vendors for you, we never include our own products.

Can you clean our data or build a knowledge base?

Yes, when an evaluation shows that wrong answers come from outdated, duplicated or missing documents or records. We first show which documents cause which errors. Then we clean them up or write down what is missing with your team and rerun the evaluation.

Can you work with our diligence or audit firm?

Yes. Your diligence or audit firm can hire us directly, or we can work alongside them.

What is an AI evaluation?

An AI evaluation runs a system on a fixed set of real questions or tasks and grades every answer against rules agreed in advance. Because the examples and the grading stay fixed, you can rerun the same evaluation after every change and compare the results.

What does an AI audit from SeedSweep cover?

We test what your system produces on your own questions and tasks and trace each error to its cause. We do not run governance audits, which check your policies and records against standards and laws such as ISO 42001 (an international standard for managing AI), the EU AI Act and the NIST AI Risk Management Framework, and we do not certify against them.

How do you measure AI accuracy?

We grade a sample of your real questions or tasks against correct answers we agree on with you, then report the share the system gets right and the causes behind the rest. The sample report shows the result on 400 support conversations.

How do you test an AI chatbot's accuracy?

We grade a sample of its real conversations against your help center articles. We count how many customers had their problem solved, which can differ from how many conversations the vendor counted as resolved.

What is the difference between resolution rate and the share of problems solved?

A resolution rate is the share of conversations a vendor counts as resolved under its own rule, and some rules count a conversation as resolved when the customer stops replying. The share of problems solved counts the customers who got what they needed. In our sample report, which comes from an evaluation for one of our clients, the vendor counted 78% of conversations as resolved and 61% of customers had their problem solved. The guide to containment rate vs resolution rate explains how to check the number yourself.

Why does our AI assistant give wrong answers?

Outdated or conflicting documents are a common cause. In our sample report, 56% of the failed conversations came from answers that quoted a pricing page that changed in June. An evaluation finds the document behind each wrong answer.

How do you test a RAG system or a knowledge base?

A RAG system is an assistant that looks up your documents before it answers. We grade its answers to real questions against the current version of each document and list the documents behind the wrong answers so you can correct or replace them. Then we rerun the same evaluation.

How do you measure AI hallucinations?

A hallucination is an answer that your documents or records do not support. We measure the share of answers to your own questions that have no such support. Published hallucination rates come from other tasks and other data, so they do not give you this number.

How do you test an AI agent before it goes live?

An AI agent is an AI system that uses your other software to complete a task. We test it on your real tasks, including ones where one of those programs fails or returns wrong information, and check each step and the final result.

How do you test audio and video systems?

We compare what they produce, such as transcripts, notes or answers, with the original recordings. When a system runs live, we also measure how quickly it responds.

How do you evaluate an AI vendor?

We test each vendor's system on the same sample of your real questions or tasks with the same grading. The result shows how often each system is right and what each correct answer costs.

Can you help us choose an AI model?

Yes. We compare open models, which you can run on your own servers, with closed models that you use through a provider, on your own data. Public benchmarks measure other tasks, so we report accuracy and the cost of each correct answer on yours.

How do you check a company's AI automation claims during due diligence?

We take a sample of the company's customer conversations or tasks and record whether the AI finished each one alone, finished it with help from a person or left it to a person. Then we recalculate the automation rate under the seller's definition and under a strict one that counts only work finished with no human help.

Can you test an AI system before an audit or a regulator's review?

Yes. We check the patient notes it wrote, the insurance claims it reviewed or the loan decisions it supported against the recordings and records behind them.

Data and security

Does our data leave our systems?

It doesn't have to. If you need it, we can run every analysis on your own servers or in your cloud account. We follow GDPR and your rules for data retention and deletion. See security and data for the details.

How do you check data quality for AI?

We compare a sample of the data with its source and check which errors change what the AI model or assistant does. We report those errors first.

How do you check the quality of labels a vendor delivered?

We label a sample ourselves without seeing the vendor's labels, measure how often the two agree and compare the batch with the quality terms in your contract before you pay or renew.

Should we clean our data before using AI?

Yes, starting with the errors that lead to wrong answers. An evaluation on real questions shows which documents and records cause wrong answers, and we work with you to correct them.

Timing and cost

How long does it take?

It depends on the decision and on how quickly we can get access to the data. In the discovery call we tell you whether a useful evaluation can finish before your deadline.

What does it cost?

We quote after the discovery call, once we know the decision, the systems involved and the data.

How do you measure whether AI is worth what we pay?

We start from the share of real tasks it gets right and the cost of each correct answer, then compare that with what the same work cost before.

About SeedSweep

Why the name?

In machine learning, a seed sweep reruns an experiment with different random starting points to check whether a result is replicable or happened by chance. We apply the same idea to AI systems and check whether their results are replicable on your own data.