How an AI Support Agent Learns From Agent Edits
Every time a support agent edits an AI draft before sending it, they show the system how the company actually handles that case. In the 30 days to 19 November 2026, agents in the deployment from our case study edited 19,339 AI drafts. That is the most valuable training material the system has, and it is easy to lose. This is the loop we built to use it, and what it found.
Keep both versions
The loop needs two texts for every reply: what the AI drafted and what was sent. Many systems store one reply field, and the agent's edit overwrites the draft. From that day nobody can tell what the AI wrote. The AI's draft is stored in its own field and never overwritten.
Before trusting any comparison, test it on replies the AI sent with no person involved. Draft and sent text must be identical there. If the comparison reports differences on those, the method is broken.
Five steps
- Mine the differences. For each edited reply, extract what the agent changed in substance: a different solution, an added fact, a removed promise. Group the changes by theme.
- Propose. A theme that passes a threshold of repeated, independent cases is compared with the existing rules and classified: already covered, needs a revision, new, in conflict with a rule, or skip.
- Verify. A candidate rule is tested by replaying real past cases with and without it.
- Measure the gap. Where the practice is already in the rules, rerun the current system on the same cases and count how often it does what the agents did.
- Diagnose. For the cases where it does not, find why.
What the first run found
The first run covered 300 complex after-sales replies sent by agents over seven days. Of the eight practices that passed the threshold, six were already covered by existing rules. The agents were not doing something the system had never been told. The system had been told, and did not do it consistently.
That changes the work. The answer was not to add experience. It was to find out why existing rules were not followed, and the usual cause was that the same matter was described in several places with small differences.
An example: one rule in five places
When a customer has a fault repaired at a local shop, the company reimburses up to a limit. Agents kept adding the limit to drafts that had left it out. The rule existed in five places, among them the reply principles, a constraint entry, the solution descriptions and a procedure document, and they did not say the same thing. We made all five say one thing: state the limit in the same email. Replayed on the original tickets, the drafts then carried the limit, and ordinary tickets were unchanged.
Agent practice wins
When the knowledge base, a rule or a procedure document conflicts with what agents consistently do, the agents are right and the document is changed. "Consistently" is the condition: the practice shows up across several agents and many replies. One agent's one-off choice, or a reply that was simply wrong, does not count.
Store rules, never raw text
What goes into the experience library is a distilled rule that makes sense without the original ticket: in which situation, do what, with which exceptions. No order numbers, no names, no addresses, no sentences copied from a customer or an agent. Similar cases merge into one entry.
We set this rule after an audit. Of 83 entries in the library, 34 had been saved as raw review comments, carrying order numbers, addresses, names, and in some cases a whole reply in English. Every new draft was searching through them. One group of entries said the opposite of the current policy. Every route that writes to the library now passes through a distilling step first.
Verify by replay, on frozen facts
A candidate change is tested on real past tickets. Two details decide whether the test means anything:
- Freeze what changes over time. If the system can search internal team messages, a replay today finds the correction someone posted after the original draft. The old version then looks as good as the new one. Tools like that are frozen to the moment of the original draft.
- Run each case twice. Model output varies from run to run. A change counts when it is clearly better across both runs of the same batch, compared side by side with the alternatives.
A change that does not show a clear improvement is not deployed. Several of ours were dropped at this step.
Fix by subtraction
When the loop finds a gap, the first move is to delete: repeated explanations, rules that contradict each other, steps that add nothing. A new rule is added only if it is a general principle. A rule written for one ticket makes the next ticket worse. Constraints that record a past incident are kept, merged with similar ones, and their tests still have to pass.
What agents change most
In a review of 400 edited drafts: 53 percent of the edits removed a promise the agent did not want to make, 22 percent added stock status the draft had left out, and 10 percent replaced "we have no information" with an answer the knowledge base already held. Three numbers, three specific pieces of work.
Who this fits
Teams where people review AI output every day. The loop needs volume: thousands of edited replies a month give it enough repeated cases to separate a practice from an accident.
Related: how to measure AI customer service quality and why AI agents fail in production. The free plan describes how this loop would run on your tickets, within two working days.