Extensions
Evaluation Test Suite
Before any change goes live, your agent is checked against a fixed set of questions with known good answers.
Changing a prompt or a document can quietly break an answer that used to be right, and nobody notices until a customer does. This add-on gives the agent you already run a fixed set of test questions with agreed answers. Every change runs the suite, so you see what got better or worse before it goes live.
- Checks every change against questions with known good answers
- Shows what improved and what broke before release
- Starts from a recorded baseline of your current agent
$2,500
10–20 working days
What it is
Evaluation Test Suite is an add-on that gives an existing Upmark Labs agent an automated set of test questions with agreed expected answers. Each time a prompt, document or setting changes, the suite runs and shows what got better or worse. A baseline run records where it stands today.
Why businesses buy it
AI agents change often: new documents, updated prices, adjusted prompts. Without a test set, each change is a guess, and mistakes reach customers first. An evaluation suite turns that guess into evidence, so your team can change things with confidence and roll back quickly when something slips. The written scope covers drafting the questions with you, agreeing the expected answers, a baseline run and documentation for running it, so the suite is useful from the first change.
Problems this solves
A small prompt change quietly breaks answers that used to work
Evaluation Test Suite runs a set of test questions with agreed expected answers each time a prompt, document or setting changes. It shows what got better or worse.
Nobody knows how good the agent is today in measurable terms
A baseline run records where the agent stands before any further changes. Later runs compare against it, so progress is clear rather than a feeling.
Checking the agent by hand after each update takes too long
The test questions run automatically, so nobody has to type them in after each change. Your team reviews the results instead of repeating the checks.
A good fit if
You plan to keep changing the agent's prompts, knowledge or settings after launch.
A past change quietly broke an answer that used to be right, and nobody noticed for days.
You can write down the questions that matter most and what a good answer looks like.
Your agent is an Upmark Labs build, or you are ordering one together with this.
Not the right fit if
You want test sets and automated evaluations across several AI features. AI Evaluation System is built for that.
You want two prompt versions tried side by side on real traffic. Prompt A/B Testing does that.
You want live conversations scored for quality after the fact. Conversation QA Scoring adds that.
You want a separate copy of the product to try changes in. Staging Environment Setup gives you one.
How it works
- The test set is written. We turn your priority questions and sample conversations into test cases, each with the answer or behavior you expect.
- Scoring is agreed. Some cases need an exact fact, others need the right tone or a refusal, and each gets a pass rule you approve.
- The suite is wired to the agent. It can be run on demand before any change, against the parent product's current setup.
- A baseline is recorded. The first full run shows where the agent stands today, so later runs can be compared to it.
- You review the cases and results with one revision round, then get a written walkthrough of how to run the suite.
What you provide within 72 hours
Your portal lists these the moment you order. Delivery starts the working day after everything is in and your access is checked.
- The order reference of the agent being tested
- The questions your agent must always get right, with what a good answer looks like
- A handful of real conversations, good and bad, to base test cases on
- Answers the agent must refuse or hand to a person
- One person who decides whether an answer counts as a pass
The 72-hour rule. Send everything on your checklist, including access, within 72 hours of ordering so your build starts on time.
Timeline
Start
Priority questions and sample conversations received
Weeks 1–2
Test cases and pass rules drafted
Week 2
Suite connected to the agent
Week 2
Baseline run, results report sent
Week 2
Your review and one revision round
Go-live
Suite handed over with a written walkthrough
| Option | Delivery time | Price |
|---|---|---|
| Standard | 10–20 working days | $2,500 |
What you receive at handover
- An automated test set of questions and expected answers for your agent
- A baseline run showing where the agent stands today
- Updated documentation on running the suite and reading results
- A test report, a handoff note and a short written walkthrough
Expected benefits and how to measure them
- Helps catch regressions before customers see them
- Makes it easier to judge whether a change helped
- Can save time on manual checks after updates
- Test pass rate
- Compare each run's pass count with the baseline run in the suite's results.
- Regressions caught before release
- Log each failure the suite found before a change went live.
- Manual checking time
- Ask the admin to note time spent checking after each change, before and after the suite.
Before you start: The baseline run is your starting point. Also note how your team checks the agent after changes today.
These are things to track, not promised results. Outcomes depend on your data, your team and your market.
Running costs
Each run of the suite sends its test questions through the agent, so it uses AI model calls in your own account. How much depends on how many cases you keep and how often you run them; the parent product's running costs are otherwise unchanged.
Ask about running costsCustom requirements
Running the suite automatically on a schedule against several agents, or scoring with a human panel, is beyond this add-on. Add a checkout note for a written Change Quote. Larger or bespoke projects go through the Custom AI request form instead.
Request a custom versionKeep it running
Adding cases as your agent grows is normal upkeep, covered by the change hours of the parent product's maintenance, which grows in line with the add-on's price.
Adds $125/mo to maintenance
Covered by the main product's maintenance, which grows by this amount.
How maintenance worksHow this compares
| Do it yourself with a SaaS tool | Hire a custom agency | Upmark Labs | |
|---|---|---|---|
| Price | An evaluation platform bills monthly, often per run | An agency quotes testing as project time | Ours is one fixed price, shown before you buy |
| Test cases | You write them yourself | Depends on the brief | Ours are drafted from your questions and approved by you |
| Starting point | An empty dashboard | Set per project | Ours includes a baseline run of your current agent |
Questions about this product
How many test cases are in the Evaluation Test Suite?
As many as your priority questions and sample conversations support, agreed at intake rather than fixed in advance. The aim is coverage of what matters: the answers that must be right, the refusals that must happen, and the handoffs that must trigger. Your approver signs off the final list before the baseline run.
Who decides whether an answer passes?
You do. We propose a pass rule for each case, such as a required fact, a required refusal or a required handoff, and your approver confirms or changes it. Where a case needs judgment, the rule describes what a good answer must contain rather than one exact wording.
When should I run the suite?
Before any change goes live. That includes editing the prompt, adding documents, changing models or adjusting settings. Comparing the new run with the baseline shows which answers changed. The written handover report shows your team how to start a run and read the results.
Can I add questions after handover?
Yes. New cases can be added as your agent takes on new topics. Small additions fit inside the monthly change hours of the parent product's maintenance. A large expansion, such as a whole new knowledge area, can be quoted separately through a Change Quote.
Can the Evaluation Test Suite be added to an agent already in use?
Yes. The suite can be ordered with a new Upmark Labs agent or added to one already running. For a live agent, the baseline run captures how it answers today, which is useful in itself. You choose the delivery speed at checkout, and each speed has a fixed price and date shown before you pay.
Does this replace checking real conversations?
No. The suite tests known questions before a change, while real conversations show what customers actually ask. The two work well together. If you also want live conversations scored automatically, Conversation QA Scoring covers that side and complements the suite.
Know what a change does before customers do.
Add the Evaluation Test Suite and every prompt or knowledge change is checked against your agreed questions, starting from a recorded baseline.
Works well with
Front Line
AI Customer Support Agent
Resolves customer questions across chat and email using your knowledge base and hands off complex cases.
AI Features & Bots
AI Website Chatbot
Adds a branded AI assistant to your website trained on your content.