Dingbo AI
Dingbo AI · Field notes

How an AI Support Agent Learns From Agent Edits

Updated 2026-11-19 · Shenzhen Dingbo Intelligent Technology Co., Ltd.

Dingbo AI builds and runs AI agents for customer service. You pay per completed task.

Want AI to take over a repetitive job? Get a free assessment

Email only. We reply within 24 hours, free of charge.

Every time a support agent edits an AI draft before sending it, they show the system how the company actually handles that case. In the 30 days to 19 November 2026, agents in the deployment from our case study edited 19,339 AI drafts. That is the most valuable training material the system has, and it is easy to lose. This is the loop we built to use it, and what it found.

Keep both versions

The loop needs two texts for every reply: what the AI drafted and what was sent. Many systems store one reply field, and the agent's edit overwrites the draft. From that day nobody can tell what the AI wrote. The AI's draft is stored in its own field and never overwritten.

Before trusting any comparison, test it on replies the AI sent with no person involved. Draft and sent text must be identical there. If the comparison reports differences on those, the method is broken.

Five steps

What the first run found

The first run covered 300 complex after-sales replies sent by agents over seven days. Of the eight practices that passed the threshold, six were already covered by existing rules. The agents were not doing something the system had never been told. The system had been told, and did not do it consistently.

That changes the work. The answer was not to add experience. It was to find out why existing rules were not followed, and the usual cause was that the same matter was described in several places with small differences.

An example: one rule in five places

When a customer has a fault repaired at a local shop, the company reimburses up to a limit. Agents kept adding the limit to drafts that had left it out. The rule existed in five places, among them the reply principles, a constraint entry, the solution descriptions and a procedure document, and they did not say the same thing. We made all five say one thing: state the limit in the same email. Replayed on the original tickets, the drafts then carried the limit, and ordinary tickets were unchanged.

Agent practice wins

When the knowledge base, a rule or a procedure document conflicts with what agents consistently do, the agents are right and the document is changed. "Consistently" is the condition: the practice shows up across several agents and many replies. One agent's one-off choice, or a reply that was simply wrong, does not count.

Store rules, never raw text

What goes into the experience library is a distilled rule that makes sense without the original ticket: in which situation, do what, with which exceptions. No order numbers, no names, no addresses, no sentences copied from a customer or an agent. Similar cases merge into one entry.

We set this rule after an audit. Of 83 entries in the library, 34 had been saved as raw review comments, carrying order numbers, addresses, names, and in some cases a whole reply in English. Every new draft was searching through them. One group of entries said the opposite of the current policy. Every route that writes to the library now passes through a distilling step first.

Verify by replay, on frozen facts

A candidate change is tested on real past tickets. Two details decide whether the test means anything:

A change that does not show a clear improvement is not deployed. Several of ours were dropped at this step.

Fix by subtraction

When the loop finds a gap, the first move is to delete: repeated explanations, rules that contradict each other, steps that add nothing. A new rule is added only if it is a general principle. A rule written for one ticket makes the next ticket worse. Constraints that record a past incident are kept, merged with similar ones, and their tests still have to pass.

What agents change most

In a review of 400 edited drafts: 53 percent of the edits removed a promise the agent did not want to make, 22 percent added stock status the draft had left out, and 10 percent replaced "we have no information" with an answer the knowledge base already held. Three numbers, three specific pieces of work.

Who this fits

Teams where people review AI output every day. The loop needs volume: thousands of edited replies a month give it enough repeated cases to separate a practice from an accident.

Related: how to measure AI customer service quality and why AI agents fail in production. The free plan describes how this loop would run on your tickets, within two working days.

Which job eats the most hours on your team? Tell us in one line and we will sketch a plan for free

We connect to the systems and documents you already use, the AI works through your process, and your people review and approve. You see real numbers from a pilot before you pay for anything, then you pay for completed work.

Or email us directly: yzh317179958@163.com

Free assessmentReply within 24h