How to Measure AI Customer Service Quality
Vendors report AI customer service quality as resolution rates and satisfaction scores. Those numbers are easy to inflate and hard to act on. The measure that changed how we built the system in our case study was simpler: of the replies the AI drafts, what does the team actually keep, and what do they change?
Count three groups of sent replies
Every reply that goes out with an AI draft falls into one of three groups. In the last 30 days:
- Sent by the AI on its own: 2,756.
- Approved by an agent without a single edit: 3,463.
- Edited by an agent before sending: 19,357.
Together that is 25,576 replies sent using an AI draft. The edited group is large, and that is useful rather than embarrassing: it is the richest source of information about what the AI gets wrong.
Store the draft and the final text side by side
None of this works unless the system keeps the AI's original draft and the text actually sent as two separate fields. We learned this the hard way: one field was overwritten with the final text at send time, so for a while every comparison showed the AI "matching" the agent perfectly. Keep both from day one.
Read the edits, then classify them
We classified 400 edited drafts by what the agent changed:
- 53 percent removed promises the agent did not want to make.
- 22 percent added stock status the draft had left out.
- 10 percent replaced "we have no information" with an answer the knowledge base already contained.
Each group pointed to a fix in what the AI receives, not to a different model. Over-promising pointed to rules that pushed the model to offer solutions without checking what had been agreed. Missing stock status pointed to a lookup the AI did not have. "No information" pointed to retrieval.
Separate wording edits from substance edits
Agents also change greetings, sign-offs and word choice. Those edits are real but cheap. An edit that changes the solution, the amount, the facts or a promise is expensive, because the draft would have misled the customer or cost money. Weight the two differently, and spend improvement time on substance.
Check the records, not only the replies
A reply can be right while the record behind it is wrong. Comparing AI drafts with final replies over 14 days, 38 of 276 compensation records did not match what was actually sent, because the ledger had taken amounts from the AI's proposal. 21 of those final replies gave no money at all. Measure whether ledgers, task cards and execution steps follow the version a person approved.
Use edit rates to decide what may send itself
Automatic sending should widen one category at a time, and only where agents have stopped changing drafts in that category. Before opening a new category, count the drafts already waiting in it: they were written under the old rules and should not go out automatically. One automatic send of an older draft taught us that.
Translate volume into hours, honestly
To size the work, we count 5 minutes for each email an agent would otherwise write and 3 minutes for each chat message. For the last 30 days that equals 3,900 hours of output. It is an output figure, not a saving claim: the same team absorbed the volume and moved to reviewing and deciding.
A measurement checklist
- Keep the AI draft and the sent text in separate fields.
- Report sent-by-AI, sent-unchanged and edited every week.
- Classify a sample of edits by what changed and fix the input behind the largest group.
- Audit that records follow the human-approved version.
- Open automatic sending only where edits have dropped.
For the hallucinations those edits uncovered, see AI customer service hallucination examples.