Knowledge Assistants
AI Knowledge Agent for Google Workspace
Answers staff questions from your documents with citations, connected natively to Google Workspace so it reads and cites Drive files, shared drives…
Knowledge Assistants
AI Chatbot Accuracy Audit
A clear, evidence-backed picture of where your AI chatbot answers well, where it goes wrong, and what to fix first.
An AI chatbot that sounds confident is not always right. Wrong or invented answers can mislead customers, and you may not notice until someone complains. AI Chatbot Accuracy Audit tests your bot with real questions and adversarial prompts. Every answer is scored by hand. You get a report with transcript evidence, accuracy and handoff rates, and 15 ranked fixes.
- 150 real questions and 30 adversarial prompts
- Every answer scored by hand, with transcripts
- 15 ranked fixes and a reusable test set
$2,925
7–17 working days
What it is
AI Chatbot Accuracy Audit is a one-time test of your live or staging AI chatbot. We ask it 150 real questions built from your FAQs, tickets and documents, plus 30 adversarial prompts. Every answer is scored by hand, and you get a report with transcript evidence and 15 ranked fixes. Making the fixes is not included.
Why businesses buy it
Most chatbots are tested with a handful of questions before launch, then trusted with thousands of customers. Problems such as invented answers, missed handoffs or unsafe replies stay hidden until they cause harm. This audit gives you evidence instead of a feeling. We test the bot with questions drawn from your own material, plus prompts designed to push it off course. Every score links to a transcript. The ranked fixes show where to start, and the test set lets you check again after any change.
Problems this solves
You do not know how often the chatbot gets answers wrong
We test the bot with 150 domain questions built from your FAQs, tickets and documents. Every answer is scored by hand, and the report gives accuracy and handoff rates.
The bot may invent answers or say something unsafe
Scoring flags invented and unsafe answers separately from simple mistakes. Each finding comes with transcript evidence, so your team can see exactly what the bot said.
Nobody has checked how it handles manipulation or personal data
We run 30 adversarial prompts covering off-topic requests, prompt injection, policy edge cases and personal-data requests. The report shows where the bot held firm and where it slipped.
You know something is wrong but not what to fix first
The report ranks 15 fixes by impact, so the most serious problems come first. The reusable test set lets you re-run the same checks after changes.
A good fit if
You run an AI chatbot or knowledge assistant and worry it gives wrong or invented answers.
You want independent evidence before expanding the bot to more customers or channels.
You need to know how it handles off-topic requests, prompt injection and requests for personal data.
You can give us test access and allow around 500 test messages without hitting rate limits.
Not the right fit if
You want the problems fixed as well as found, on a helpdesk AI agent. See Helpdesk AI Agent Setup.
You want ongoing monthly reviews and fixes, not a one-time test. See AI Agent Tuning Retainer.
You need a security penetration test or a compliance certification. This audit is neither.
Your chatbot answers poorly because the content is thin. See Help Center Content Build.
How it works
- You give us test access to the bot, your source FAQs and documents, and a sample of tickets.
- We build a test set of 150 domain questions and 30 adversarial prompts from your real material.
- We run every question and prompt against the bot and save the full transcripts.
- Each answer is scored by hand as correct, partial, wrong, invented, unsafe or a missed handoff.
- We write a report with accuracy and handoff rates, transcript evidence and 15 fixes ranked by impact.
- A recorded 45-minute readout walks your team through the findings and the reusable test set.
What you provide within 72 hours
Your portal lists these the moment you order. Delivery starts the working day after everything is in and your access is checked.
- Test access to the bot, live or staging, by invite or a key you control.
- Your source FAQs and documents, so we know what the right answers are.
- A sample of real customer tickets or questions.
- Rate limits that allow around 500 test messages.
- One person to join the readout.
The 72-hour rule. Send everything on your checklist, including access, within 72 hours of ordering so your build starts on time.
Timeline
Start
Order placed. Test access, source documents and tickets are due within 72 hours.
Week 1
Test set of 150 questions and 30 adversarial prompts built from your material.
Week 1
All tests run against the bot and full transcripts saved.
Week 1
Every answer scored by hand and the 15 fixes ranked.
Go-live
Report and test set delivered, and the recorded readout held.
After delivery
14 days of follow-up questions about the findings.
| Option | Delivery time | Price |
|---|---|---|
| Standard | 7–17 working days | $2,925 |
What you receive at handover
- A reusable test set of 150 domain questions and 30 adversarial prompts (CSV)
- Scored results for every answer: correct, partial, wrong, invented, unsafe or missed handoff
- A report with transcript evidence and accuracy and handoff rates
- 15 ranked fixes (recommendations; fixes are not included)
- A recorded 45-minute readout
Expected benefits and how to measure them
- Helps you see how accurate your chatbot really is
- Makes it easier to decide which fixes to make first
- Gives you a test set to check the bot again after changes
- Accuracy rate
- Take the share of correct answers from the audit report, then re-run the test set after fixes and compare.
- Invented or unsafe answers
- Count answers scored invented or unsafe in the audit, and again in each later test run.
- Missed handoffs
- Compare the number of missed handoffs in the audit with the number after handoff rules are changed.
Before you start: The audit itself is the baseline. Also note any recent customer complaints about wrong bot answers, so you can compare later.
These are things to track, not promised results. Outcomes depend on your data, your team and your market.
Running costs
There are no running costs. The audit is a one-time piece of work, and you do not need to buy any tools for it. The test messages run on your bot, so check whether your platform charges per conversation before we start.
Ask about running costsCustom requirements
Add a note at checkout if you need something specific. If it fits within the Scope Card it is simply included. If it goes beyond it, you get a written Change Quote within two working days, and that work starts only once you accept it, with its time added to your delivery date.
Request a custom versionKeep it running
This is a one-time audit, so there is nothing to keep running. The report and test set are yours, and you can re-run the test set after any change. If you want the fixes made, Helpdesk AI Agent Setup covers helpdesk AI agents. AI Agent Tuning Retainer reviews and fixes answers every month.
How this compares
| Do it yourself | Hire a traditional agency | Upmark Labs | |
|---|---|---|---|
| Testing | A few questions typed in by the team | Often part of a larger build project | 150 domain questions and 30 adversarial prompts |
| Scoring | A general feeling of whether it seems right | Varies | Every answer scored by hand against your source material |
| Evidence | Screenshots, if anyone saves them | Depends on the provider | A report with transcript evidence for each finding |
| Next steps | You work out what to fix | The fix list may assume they do the work | 15 ranked fixes your team or any provider can act on |
Questions about this product
How do you score whether a chatbot answer is right?
By hand, against your own source material. Each answer is marked correct, partial, wrong, invented, unsafe or a missed handoff. An invented answer states something your documents do not support. A missed handoff is when the bot should have passed the customer to a person. Every score links to the transcript, so you can check it.
What are the 30 adversarial prompts in the chatbot audit?
Messages designed to push the bot off course. They include off-topic requests and prompt injection attempts that try to override its instructions. Others probe policy edge cases or ask for personal data. They show how the bot behaves when customers, or bad actors, do not ask nicely. They are not a full security penetration test.
Does the chatbot audit include fixing the problems?
No. The audit finds and ranks the problems, and making the fixes is not included. The 15 ranked fixes are written so your team or any provider can act on them. If you run a helpdesk AI agent, Helpdesk AI Agent Setup can make the fixes. For a custom bot, ask for a Change Quote.
Can you audit any AI chatbot, or only certain platforms?
Most text chatbots and knowledge assistants, including Intercom, Zendesk, Gorgias, Freshdesk and Voiceflow bots, and custom web chat. We need test access, live or staging, and rate limits that allow around 500 test messages. Voice agents are not covered, and the audit covers one bot in one language.
Will the chatbot audit tell us if our bot is compliant?
No. The audit is not a compliance certification or a legal review. It shows how the bot answers, including how it handles personal-data requests and unsafe prompts. That evidence helps you prepare for your own compliance review, but it is not legal advice. Check specific obligations with a qualified adviser.
Can we re-use the chatbot test set after the audit?
Yes. The test set is delivered as a CSV of all 150 questions and 30 adversarial prompts, with the expected answers. You can re-run it after each change to see whether the bot improved or slipped. AI Agent Tuning Retainer can also re-run it for you every month as part of its reviews.
What happens after the chatbot audit readout?
You keep the report, the test set and the readout recording, and you can send follow-up questions for 14 days. After that, you decide what to fix and who fixes it. Many teams fix the top-ranked items first, then re-run the test set to check the effect.
Find out what your chatbot really says
Order AI Chatbot Accuracy Audit for 180 hand-scored tests, transcript evidence and 15 ranked fixes your team can act on.
Works well with
Knowledge Assistants
AI Knowledge Agent for Microsoft 365
Answers staff questions from your documents with citations, connected natively to Microsoft 365 so it reads and cites SharePoint, OneDrive and Teams…
Knowledge Assistants
AI Knowledge Agent for Confluence
Answers staff questions from your documents with citations, connected natively to Confluence so it reads and cites spaces and pages your users…