Extensions

Prompt A/B Testing

Try two versions of your agent's prompt on real use and keep the one that performs better, with evidence.

Teams often argue about which prompt wording works better, and the loudest opinion usually wins. This add-on lets the product you already run try two prompt versions side by side on real use. Traffic is split between version A and version B, outcomes are recorded on a measure you agree, and you choose the winner from a clear summary.

Tests two prompt versions on real use, not guesses
Measures each version on an outcome you agree
Ends with a side-by-side summary to pick the winner

$1,500 10–20 working days

Choose delivery and add to cart Ask an expert

Two cups holding paper scrolls beside a tipped brass balance, two prompt versions weighed against each other.

What it is

Prompt A/B Testing is an add-on that lets an existing Upmark Labs product run two prompt versions side by side and compare the results. Traffic or requests are split between version A and version B, outcomes are recorded, and you choose the winner. A summary shows which version performed better.

Why businesses buy it

Small changes in wording can change how many customers book, buy or get a correct answer. Without a fair test, you either never change the prompt or change it on instinct and hope. A/B testing gives you evidence from real conversations, so improvements stick and bad ideas are dropped quietly. The written scope covers both versions, the traffic split, the measure, recording outcomes and a results summary, built into your existing product rather than bolted on.

Problems this solves

Teams argue over which prompt wording works better

Prompt A/B Testing runs two prompt versions side by side. Traffic or requests are split between version A and version B, and outcomes are recorded for each.

Prompt changes are judged on a hunch rather than results

Outcomes are recorded for both versions, and a summary shows which one performed better. You then choose the winner from the evidence, not from opinion or habit.

Testing a new prompt risks every user getting a worse version

Only part of the traffic or requests sees the new version while the test runs. If it performs worse, the original version keeps serving everyone else.

Scope card SKU UPM-A0033

Prompt A/B Testing

Everything this build includes, written down before you pay. Choose how you want it, own the code or license it yearly, then the delivery speed and any add-ons.

$1,500 one-time build price

What is included

Channel: As per the parent product
Integrations: As described in the add-on
Knowledge: As per the parent product
Workflows: Adds side-by-side testing of prompt versions
Languages: As per the parent product
Testing: Tested together with the parent product
Revisions: 1 round
Handoff: Written handover report
Support: 14 days after launch

A good fit if

Your team disagrees about how the agent or generator should be instructed, and opinion is not settling it.

You can name what better means for you, such as fewer handoffs, shorter replies or more approved drafts.

Your product is a content generator, a knowledge chatbot or an action agent.

You run an Upmark Labs product already, or are adding this to the order for one.

Not the right fit if

You want every change checked against known questions before release. Evaluation Test Suite does that.

You manage many prompts across several apps. AI Prompt Management gives you a central place for them.

You only want the tone and persona rewritten. Custom Persona and Brand Voice Tuning covers it.

Your product is a voice agent, workflow automation or another type outside the three this add-on fits.

How it works

Two versions are set. Your current prompt becomes version A, and version B is written with you around the change you want to test.
The measure is agreed. We pick one or two outcomes the parent product can already record, so the comparison is fair.
Requests are split. Each conversation or generation is assigned to A or B, and the assignment is logged with the outcome.
Results are compared. You get a side-by-side summary of how each version did on the agreed measures.
You pick the winner, which becomes the live prompt, after one revision round and a written walkthrough.

What you provide within 72 hours

Your portal lists these the moment you order. Delivery starts the working day after everything is in and your access is checked.

The order reference of the product being tested
What you want version B to change, in plain words
The outcome that decides the winner, such as fewer handoffs or more approved drafts
One person who approves both prompt versions and calls the result

The 72-hour rule. Send everything on your checklist, including access, within 72 hours of ordering so your build starts on time.

Timeline

Start Change to test and success measure agreed

Week 1 Version B written and approved

Weeks 1–2 Request split and outcome logging built

Week 2 Both versions tested with the parent product

Week 2 Results summary, your review and one revision

Go-live Testing handed over, walkthrough recorded

Delivery options, in working days (Monday to Friday) from completed intake. They can extend because of national or public holidays, technical issues outside our control (such as platform outages, third-party changes, account approvals or access delays), or natural disasters and other events beyond our control. If a delay happens, we tell you in writing with a new date.
Option	Delivery time	Price
Standard	10–20 working days	$1,500

What you receive at handover

Side-by-side prompt testing added to your existing product
A results summary showing which version performed better
Updated documentation on setting up and ending a test
A test report, a handoff note and a short written walkthrough

Expected benefits and how to measure them

Helps settle prompt debates with evidence
Can lower the risk of rolling out a weaker prompt
Makes ongoing improvement easier to manage

Outcome per version: Compare the tracked outcome for version A and version B from the results summary.
Tests completed: Count prompt tests run and concluded each quarter.
Outcome after rollout: Compare the outcome for the month before and after the winning version went live.

Before you start: Record the current prompt and the outcome you care about most, measured over the last month.

These are things to track, not promised results. Outcomes depend on your data, your team and your market.

Running costs

Splitting requests does not add many model calls, because each request still runs once, on either A or B. Usage stays in your own AI account at roughly the parent product's normal level.

Ask about running costs

Custom requirements

Testing more than two versions at once, or measuring outcomes the parent product does not record, needs extra work. Add it to the checkout note for a written Change Quote. Larger or bespoke projects go through the Custom AI request form instead.

Request a custom version

Keep it running

Running a fresh A/B round later, with a new version B, fits within the change hours of the parent product's maintenance, which grows in line with the add-on's price.

Adds $75/mo to maintenance

Covered by the main product's maintenance, which grows by this amount.

How maintenance works

How this compares

	Do it yourself with a SaaS tool	Hire a custom agency	Upmark Labs
Price	An experimentation tool bills monthly	An agency charges testing as consulting time	Ours is one fixed price, shown before you buy
Set-up	You wire it into the product yourself	Depends on the brief	Ours is built into your existing Upmark Labs product
Decision	Raw numbers to interpret	Set per project	Ours ends with a side-by-side summary on your agreed measure

Questions about this product

Which products can use Prompt A/B Testing?

Content and creative generators, knowledge chatbots and action agents. Those are the product types where the prompt shapes what the customer or your team sees, so comparing versions is meaningful. Voice agents, workflow automations and other types are not covered by this add-on.

How long should an A/B test run before I decide?

Long enough to collect a fair number of conversations or drafts on each version. That depends on your traffic, so we agree a sensible stopping point at intake. A quiet product needs longer. The results summary shows how many requests each version handled, so you can judge the evidence yourself.

Who writes version B?

We do, with you. You describe the change in plain words, such as shorter replies or asking for an order number earlier. We write version B, you approve it, and only then does the split start. Version A stays exactly as it runs today.

Can customers tell they are part of a test?

Not from the prompt itself. Each customer sees one version for their conversation, and both versions follow the same rules and knowledge as the parent product. Whether you tell customers you run experiments is your choice, and we can add a line to your privacy wording if you want.

Can Prompt A/B Testing run on a product that is already live?

Yes. That is where it is most useful, because it needs real traffic to compare. It can also be ordered with a new product and switched on once enough use builds up. You choose the delivery speed at checkout, and each speed has a fixed price and date shown before you pay.

What happens to the losing version?

It is switched off and kept in the documentation. The winning version becomes the live prompt for all traffic. If you later want to test another idea, the losing version stays on record so nobody repeats the same experiment by accident.