AI Agent Pilot: Running a Proof of Concept on Real Work
An AI agent demo proves that the agent can answer questions someone chose for it. A pilot has to prove something harder: that the agent can do one of your jobs, on your data, at a quality your team accepts. Every client of ours starts with a 2 to 4 week pilot on real work before paying anything. This is how we run it.
Pick one job, not a platform
A pilot that tries to cover "customer service" or "operations" ends with a slide deck. Pick one job with a clear start and finish: replying to order status emails, answering pre-sales chat, generating certificates from a spreadsheet. The job should happen every day, follow rules that can be written down, and leave a record in a system. A list of jobs that meet these conditions is in AI agent use cases running in production.
Record the baseline first
Before the agent touches anything, write down four numbers for the job: cases handled per day, average handling time per case, the size of the backlog, and time to first reply. Without them, the pilot ends in opinions. With them, the same four numbers measured the same way after the pilot give the answer.
Connect read-only
In a pilot the agent reads your systems and writes nothing into them. It looks up orders, shipments, tickets and documents. It does not edit an order, issue a refund or send anything to a customer without a person. This keeps the pilot safe to run on live work, and it is also how the finished system treats lookups: reading is the agent's job, changing things is a person's decision.
Each lookup must return one of four results: found, confirmed that none exists, found part of it, or could not check. An agent that cannot tell "could not check" from "does not exist" gives confident wrong answers, and a pilot is the place to find that out.
Replay real history before going live
Take a few hundred real cases from the last months, including the long threads and the angry ones, and run them through the agent. Compare each output with what your team actually sent. Test questions written for the pilot hide the real problems. History shows them.
One example from our own replays: a ticket with 25 back-and-forth messages, fed to the model as raw text, produced a reply that contradicted a refund the company had already promised. Distilling the thread into structured facts before the model saw it fixed the contradiction and cut the input from 37,604 tokens to 4,722. No invented test case would have found that.
Then run live, with a person reviewing everything
After the replay, the agent works on incoming cases and your team reviews every output before it goes anywhere. Nothing is sent automatically during the pilot. Reviewers do one of three things with each output: send it as it is, edit it and send it, or discard it. Those three actions are the pilot's raw data, so the review screen must record them.
The three numbers that decide
- Volume handled. How many cases the agent produced a usable output for, out of all cases of that job.
- Sent without edits. The share your team sent unchanged. This shows how much review effort remains after launch.
- Reasons for edits. What people changed, grouped. In a review of 400 edited drafts from a live system, 53 percent of the edits removed promises the agent did not want to make, 22 percent added stock status, and 10 percent replaced "we have no information" with an answer the knowledge base already held. Each group points to one fix.
Fix the largest group, replay the same history, and confirm that the group shrank and nothing else got worse. A pilot that goes through this loop two or three times shows a trend, and a trend is worth more than one good week.
Set the go or no-go rule on day one
Agree before the pilot starts what result means "go". A rule written on the first day, in terms of the three numbers, cannot be bent later by enthusiasm or by disappointment. Write the stop rule as well: the result at which both sides agree the job is not ready for an agent.
What a pilot should end with
- The baseline and the same four numbers after the pilot.
- Volume handled, the unedited share, and the edit reasons with what was fixed.
- A written line between what the AI does directly and what a person approves. See human-in-the-loop AI agents.
- A price. We set the price per completed task on the pilot's numbers, which is why we do not quote one earlier. The reasoning is in AI agent pricing models.
Why 2 to 4 weeks
Two weeks is the shortest period that covers a replay, a live run and one round of fixes. Four weeks covers a second round and a full monthly cycle of the business, such as a promotion or a billing run. A pilot that needs longer than that is a job that was scoped too wide.
The first step is free: describe the job in one line, and within two working days you receive a written plan naming the systems to connect, what the AI handles and what stays with your people.