Extensions

Conversation QA Scoring

Every conversation your agent has is scored for quality, so weak answers surface before customers complain.

You cannot read every conversation your agent has, so weak answers often surface only when a customer complains. This add-on scores each conversation against criteria you agree, such as accuracy, tone and rule-following. Low scores are flagged with a reason, so your team reviews the conversations that need attention. The scoring guide is written down and checked against human judgment.

Scores every conversation, not a small sample
Flags low scores with a clear reason for review
Follows a written scoring guide checked against human judgment

$1,500 10–20 working days

Choose delivery and add to cart Ask an expert

Row of cards marked at different heights with one flagged, each conversation scored and weak ones raised.

What it is

Conversation QA Scoring adds automated quality scoring to the conversations of an agent you already run, or are buying now. Each conversation is scored against criteria you agree, such as accuracy, tone and whether it followed the rules, and low scores are flagged for review. Scores are checked against human judgment first.

Why businesses buy it

Manual review usually covers a handful of conversations a week, which means most mistakes go unseen. Customers who get a poor answer rarely report it. They just stop coming back. Automated scoring checks every conversation and points your team at the ones worth reading, so problems are fixed before they spread. The written scope covers the criteria, the scoring guide, the flags and a calibration check against human judgment, so you can trust the scores from the first week.

Problems this solves

Nobody has time to read every conversation the agent handles

Conversation QA Scoring scores each conversation against criteria you agree, such as accuracy, tone and rule-following. Your team can focus on the ones that need attention.

Poor answers are found only after a customer complains

Conversations with low scores are flagged for review automatically. Your team can spot and fix recurring problems sooner, rather than waiting for a complaint to arrive.

Automated scores feel untrustworthy if nobody tested them

Scores are checked against human judgment before launch. The test report shows where the scoring agreed with your reviewers, so you know how far to rely on it.

Scope card SKU UPM-A0023

Conversation QA Scoring

Everything this build includes, written down before you pay. Choose how you want it, own the code or license it yearly, then the delivery speed and any add-ons.

$1,500 one-time build price

What is included

Channel: As per the parent product
Integrations: As described in the add-on
Knowledge: As per the parent product
Workflows: Adds automated quality scoring of an agent's conversations
Languages: As per the parent product
Testing: Tested together with the parent product
Revisions: 1 round
Handoff: Written handover report
Support: 14 days after launch

A good fit if

Your agent handles more conversations than anyone can read.

You only find bad answers when a customer complains.

You can describe what a good conversation looks like in a few clear criteria.

Someone on your team will review flagged conversations and act on them.

Not the right fit if

You want your human support team's conversations scored for coaching. AI Support QA scores against your rubric with coaching points.

You want a fixed test set run before each change to the agent. Evaluation Test Suite checks answers before changes go live.

You mainly want to know how customers feel. Sentiment Analysis Layer detects sentiment and sends alerts.

Your product is not conversational. Scoring applies only to voice, messaging, chat and action agents.

How it works

Criteria are agreed. Usually accuracy against your knowledge, tone, rule-following and whether the customer's need was met.
A scoring guide is written so each criterion is judged the same way every time.
Conversations are scored automatically as they finish, or in batches on a schedule you choose.
Low scores are flagged with the reason, so a reviewer knows what to look at.
Scores are checked against human judgment on a sample, then reviewed by you with one revision round.

What you provide within 72 hours

Your portal lists these the moment you order. Delivery starts the working day after everything is in and your access is checked.

The order reference of the agent whose conversations are scored
What a good conversation looks like for your business
Examples of good and poor conversations, if you have them
Where flagged conversations should go for review
One person who reviews flags and owns the criteria

The 72-hour rule. Send everything on your checklist, including access, within 72 hours of ordering so your build starts on time.

Timeline

Start Criteria and example conversations gathered

Week 1 Scoring guide written and agreed

Week 2 Automated scoring built

Week 2 Scores compared with human judgment on a sample

Week 2 Your review and one revision round

Go-live Scoring live, guide documented

Delivery options, in working days (Monday to Friday) from completed intake. They can extend because of national or public holidays, technical issues outside our control (such as platform outages, third-party changes, account approvals or access delays), or natural disasters and other events beyond our control. If a delay happens, we tell you in writing with a new date.
Option	Delivery time	Price
Standard	10–20 working days	$1,500

What you receive at handover

Automated quality scoring on your existing agent's conversations
Agreed scoring criteria and flagging rules for low scores
A test report comparing automated scores with human judgment
Updated documentation, a handoff note and a short written walkthrough

Expected benefits and how to measure them

Helps your team find weak conversations without reading them all
Can catch recurring mistakes earlier
Gives a steady view of quality over time

Average quality score: Track the average score each month from the scoring records and watch the trend.
Flagged conversations: Count flagged conversations weekly and note how many led to a fix.
Quality complaints: Compare complaints about wrong or poor answers for the month before launch and the months after.

Before you start: Note how conversations are reviewed today and how many a person checks each week. Keep a list of recent known problem conversations.

These are things to track, not promised results. Outcomes depend on your data, your team and your market.

Running costs

Scoring uses AI model usage for each conversation scored, billed to your own model accounts. Scoring in batches rather than one by one can keep that lower. We charge once for the add-on.

Ask about running costs

Custom requirements

Scoring human agents as well as the AI, coaching reports, or many different scorecards by team are beyond this add-on. Note them at checkout and you receive a written Change Quote. Larger or bespoke projects go through the Custom AI request form instead.

Request a custom version

Keep it running

Scoring is covered by the parent product's maintenance, which grows in line with the add-on's price. Adjusting a criterion or the flag threshold uses its monthly change hours.

Adds $75/mo to maintenance

Covered by the main product's maintenance, which grows by this amount.

How maintenance works

How this compares

	Do it yourself with a SaaS tool	Hire a custom agency	Upmark Labs
Price	A QA platform bills monthly per seat	An agency quotes manual review hours	Ours is one fixed price, shown before you buy
Coverage	Often a sample of conversations	A sample, reviewed by hand	Ours can score every conversation
Consistency	Depends on reviewer settings	Depends on the reviewer	Ours follows a written scoring guide
Calibration	Left to you	Set per project	Ours is checked against human judgment before launch

Questions about this product

What criteria are conversations scored on?

The ones you agree at intake. Common choices are accuracy against your approved knowledge, tone, whether the agent followed its rules, and whether the customer's need was met. Each criterion is defined in a written scoring guide so scores mean the same thing every time.

How do we know the automated scores are fair?

Before launch, we score a sample of conversations automatically and compare the results with a person's judgment on the same sample. Where they disagree, the scoring guide is adjusted. The comparison is included in the test report, and your reviewer signs it off.

Can it score my human support team's conversations too?

No. This add-on scores your AI agent's conversations only. Scoring human agents for coaching is what AI Support QA does. If you want both on one scorecard, add a note at checkout and we send a written Change Quote with the price and timing.

When does Conversation QA Scoring start scoring?

Once it goes live with its parent product and the calibration check is done. It follows the delivery speed chosen for that agent, and faster speeds are available, each with a fixed price and date shown before you pay. From launch, new conversations are scored as they finish, and low scores are flagged for review.

What happens to a conversation with a low score?

It is flagged with the criterion it failed and a short reason, then sent to the place you chose for review, such as a shared inbox or a chat channel. Your reviewer decides whether to fix the knowledge, the rules or the persona.

Will scoring every conversation be expensive to run?

Each scored conversation uses some AI model usage in your own account. For high volumes, scoring in scheduled batches, or scoring a share of conversations, keeps that lower. We agree the approach at intake based on your volumes. We charge nothing per conversation scored.