CSAT for AI Customer Service: What Low Scores Meant
When AI starts sending replies, the first question from management is whether customers like it less. The usual answer compares one average score before and after. That hides what is going on. We read every rating that came in over two days, from midday on 17 November 2026, for an e-bike brand's email support: 217 ratings, 52 of them low, meaning two points or fewer out of five. Then we grouped each rating by the message the customer was actually rating.
The four groups
- Replies the AI sent on its own: 64 ratings, 7 low. That is 11 percent.
- Tickets an agent closed, surveyed at closing: 107 ratings, 21 low. That is 20 percent.
- Tickets where an agent replied, the customer was silent for seven days, and the survey went out then: 31 ratings, 19 low. That is 61 percent.
- Notices the company sent first, before the customer had written: 15 ratings, 5 low. That is 33 percent.
What the groups say
The AI's own replies had the lowest share of low scores. This is not because the AI writes better than people. It is because of what it is allowed to send: simple answers with no pending action. The customer asked, got the answer, and was done. Agents handle the hard cases, where a refund or a part is involved, and those rate lower whoever writes the reply.
The third group is the finding. Six in ten ratings were low when the survey arrived after seven days of silence. Reading them one by one, the customers were not rating the reply. They were still waiting for a part, a refund or a follow-up, and the survey reached them in the middle of that wait. The score measured an unfinished job.
Read this way, most of the 52 low ratings were about two things: a survey sent before the matter was finished, and real delays. The wording of AI replies was a small part.
Rule 1: survey when the matter is finished
Our survey has two triggers: when a ticket closes, and when a ticket is still open and the customer has not replied for seven days. One exception was added after this review. When a reply promises that a colleague will do something, such as sending an invoice separately, issuing a voucher or shipping a part, the survey waits the full seven days even if the ticket is closed, so that the promised thing can arrive first.
Two further changes follow from the numbers: do not survey a customer about a notice they never asked for, and do not survey after seven days if we still owe the customer something.
Rule 2: group by the message that was rated
A rating belongs to a specific reply. Record which reply the survey followed, who sent it, AI or agent, and how long after the reply the survey went out. Without those three fields, the average mixes easy questions with hard ones and blames or credits the wrong sender.
Rule 3: read the low ones
A low score is a customer telling you something specific. Reading these 52 found one defect no dashboard showed: a reply had gone out in Vietnamese to a customer who wrote in Italian. The message was short and in capital letters, and the language detector misread it. We compared detection on 2,704 recent emails before and after the fix: 20 changed, all of them corrections.
Other low ratings pointed at replies that said a colleague would follow up "separately", where the follow-up was slow. The reply was accurate. The wait behind it was the problem.
Rule 4: do not judge AI by complaints alone
Agents and managers remember the bad AI reply someone forwarded. Count instead. In our checks in November 2026 the low share for AI-sent replies was 10 to 13 percent, and for agent replies 17 to 20 percent. A fair comparison also looks at what each side handles, because the AI gets the simple cases by design.
Rule 5: watch what the customer does next
Most customers never answer a survey. Every customer who is unhappy with a reply does something: writes again, repeats the question, or opens a chat. For replies the AI sends on its own, the system looks at the customer's next message and judges whether the reply settled the matter. To work, that check has to see the AI's reply and the customer's messages before and after it. A monitor that returns the same verdict for every case is not seeing what it is supposed to judge.
How to run this review yourself
- Take every rating from a few days. Two days gave us 217.
- For each, find the reply it followed, the sender, and the gap between reply and survey.
- Group, count the low share in each group, then read every low one.
- Sort what you read into: wording, wrong fact, unfinished action, delay, wrong language, survey timing.
- Fix the largest group first, and count again a week later.
Who this fits
Any team that sends a satisfaction survey and is about to let AI send replies, or already has. The review needs no new tooling beyond the three fields in Rule 2.
Related: how to measure AI customer service quality and after-hours customer service with AI. The free plan includes how we would measure quality on your tickets, within two working days.