# How to evaluate an AI agent: acceptance rate | Auspex > Adoption rate says nothing about agent maturity. Count the share of answers that go out with no edits, escalations or guardrail violations. Protocol included. Source: https://auspex.company/en/blog/autonomy-rate-not-adoption-rate/ [Blog](/en/blog/) / Buyer Guide # Don't ask how many agents you have. Ask how many of their answers you let through unchecked By: Andrew Maryasov · Aug 20, 2026 · 12 min read The percentage of companies that have “deployed AI agents” says nothing about whether those agents can be trusted. Two companies with the same line in the report operate differently: in one, an employee rereads nine answers out of ten; in the other, none. A more honest maturity metric is the share of the agent's answers that move on without edits, without escalation to a human and without tripping guardrails — the prohibitions built into the agent. Auspex calls it the acceptance rate, takes a baseline before launch and writes the threshold into the acceptance document, because this is the number that decides whether the process can be scaled and what ongoing support really costs. ![Two gauges: adoption rate is almost full and measures intent, while the share of agent answers accepted without review is low and measures the result](/images/blog/autonomy-rate-not-adoption-rate-en.webp) How many of the agent's answers you let through unchecked “How many agents do you have?” comes up at every meeting about automation. It is a convenient question: it has an answer, the answer fits on a slide and can be compared with last year's. The inconvenient part is that no decision follows from that answer — not about budget, not about hiring, not about whether the agent can be given the next process. ## Why does adoption rate measure intent rather than results? Adoption rate is the share of companies, teams or employees that have started using a tool. The metric comes from SaaS, where it did its job: if a person logs into the CRM every day and does something in it, they are using it, and that is enough to judge the rollout. With agents, that link breaks. An agent whose every answer is reread by an employee before it goes out and an agent that closes requests end to end on its own both count as deployed. In the report they produce the same line. In the budget they don't: the first costs you an employee's salary plus the model bill, the second only the model bill. Worse, adoption can't be compared even within one company. An agent in support and an agent in document processing can have the same coverage and opposite maturity, and no meeting that goes by that number will see it. ## Acceptance rate: what exactly do you count? The acceptance rate is the percentage of the agent's answers that moved on without human intervention. “Moved on” means a concrete action: the email was sent, the CRM field was written, the ticket was closed, the document was handed off for processing. The denominator is every answer the agent produced in this process over the period, including those that never reached an action. The metric has to be worded so that it can be calculated without a meeting. If getting the number means gathering people and agreeing on what counts as an edit, you don't have a metric yet — you have a topic for discussion. ## Which three events should never be lumped into one? Dashboards usually show a single number: “human intervention.” Three different events hide inside it, and two of them mean opposite things. - Edit. The agent produced a result, a person changed it and sent it. This is a quality problem: the agent is close, but not close enough. - Escalation. The agent stopped on its own and handed the case to a person because it falls outside the boundaries you drew for it. That is not a failure; it is exactly what you paid for. - Guardrail violation. The agent did, or tried to do, something it had no right to do. This is a safety problem, and prompt edits won't cure it. An agent with zero escalations is not better than one with ten percent. More often it is worse: it doesn't recognize its own limits and carries through what it should have handed to a person. A zero in escalations is a reason to check whether the handoff rule works at all, not a reason to celebrate at the meeting. Three numbers instead of one: the edit rate, the escalation rate, the violation rate. Folding them into a single “human oversight” figure destroys exactly the information the metric was collected for. ## What did those who already measured this find? GitHub ran a public measurement of the acceptance rate on its own product and disclosed the method. The study by Ziegler and colleagues (2022; 2,631 survey responses, 2,047 of them matched to IDE telemetry) tested which usage metric best predicts a developer's sense of productivity. The share of accepted suggestions won: across the study's cohort of participants it was 27%. The correlation itself is weak (ρ = 0.24) — the metric turned out to be the best of those tested, not strong in its own right. The second study is closer to agents. In 2024, researchers at Sierra AI showed on the τ-bench benchmark that an agent that completed a task successfully once will not necessarily complete it a second time. They proposed counting pass^k — the probability of passing the same task k times in a row. The best agent at the time, built on GPT-4o, succeeded less than 50% of the time on average, and over eight consecutive attempts in the retail scenario it dropped to roughly 25%. The models from that measurement are long outdated, and those percentages shouldn't be repeated as current. What is worth taking away is the construct: a one-off success in a demo is not proof of reliability. The third reference point comes from METR, a nonprofit research organization that evaluates models. In its 2025 work it measures a model's capability by the length of the task the model can complete on its own, and publishes that length for two reliability bars at once: 50% success and 80%. The gap between them is large, and autonomy is discussed against the higher one. What we could not find in public sources are numbers on the share of manual review by function — how many answers are checked by hand in legal and how many in sales. They circulate in overview articles with references to large analyst firms, but there is no primary report behind them that you can open and check. That is why this text contains no specific percentages. ## Why is “no edits” the right bar for an agent and the wrong one for a copilot? Besides the share of accepted suggestions, the authors of the same study measured persistence — whether an accepted suggestion was still unchanged after 30, 120, 300 and 600 seconds. That metric correlated with productivity less well. Developers didn't mind reworking a suggestion as long as it gave them a usable starting point. The difference lies in why a person is in the process. With a copilot the person is there by design: the tool produces a draft, the author finishes it, and an edit is a normal part of the work, not a loss. With an agent that sends the email to the client itself or changes a record in the CRM, the person is in the process out of necessity. Every edit they make means the work was done twice, and you paid for both. Before you measure the acceptance rate, answer one question: is the person in this process checking the agent by design, or because you don't trust it yet? ## How can this metric be gamed? The authors of the same GitHub study left an explicit warning against turning the acceptance rate into the sole quality criterion. An example from the paper: if you cut a suggestion in half and show the two halves one after the other, one acceptance becomes two, and the metric goes up with no benefit to the developer. In agent processes the same thing is even easier to do. Narrow the agent's scope to the easiest requests and the acceptance rate will jump, because the hard ones simply never reach it. Remove the strict guardrails and the violations disappear, because nothing is left to record them. Rewrite the escalation rule so that the agent hands off nothing, and escalations fall to zero. So the number on its own is worth nothing without a denominator and a description of the scope. Alongside the acceptance rate, record how many requests entered the process in total and what share of them reached the agent at all. A metric that rose while coverage fell means narrowing, not maturity. ## Is there a normal acceptance rate? There is no universally good number, and anyone who quotes you an industry standard for the acceptance rate is quoting a fiction. The reference point is set not by the industry but by the cost of an error in the specific process. - High cost of error, low volume — low autonomy is normal here. Keep supervising and don't chase the number: it isn't supposed to grow. - Low cost of error, high volume — low autonomy means you are paying twice, for the agent and for the person who rereads it. - High cost of error, high volume — this is where the metric is most valuable, because every percentage point translates into money, and this is where a test set is worth investing in. The right question is not “how do we raise the acceptance rate” but “does it match the cost of an error in this process.” Sometimes the answer is to leave everything as it is. ## Why is this a metric about money, not technology? Run it on your own numbers. Suppose the agent handles two hundred requests a day, and every answer that wasn't accepted straight away takes a person three minutes to reread. At an acceptance rate of 30%, a hundred and forty answers have to be checked — seven hours every day, almost a full-time position. At 70%, sixty answers and three hours remain. Escalations are counted as time spent in this calculation on purpose: in substance they are correct, but they still take up a person's time. In this arithmetic the model bill is usually a rounding error. It is more visible because it arrives as a single invoice with a big number on it, while the reviewer's salary is spread across the payroll and doesn't look like an AI expense. But that salary is what you are paying. The same logic sits behind the automation payback period, which we covered separately. ## What does the acceptance rate not show? The metric has its limits, and it is worth knowing them before you put it into a contract. It doesn't show the quality of what was accepted. A person may accept without looking, especially in month three, when the agent mostly gets things right. A high acceptance rate without spot checks means only that nobody is looking. The cure is cheap: once a week a person rereads five percent of the already accepted answers, picked at random. It doesn't show what the agent slipped past the monitor. An answer can be formally correct yet obtained in a way that bypasses oversight; the acceptance rate will count that event as a success. And it doesn't replace governance. An agent registry, a separate budget and a named owner for each agent remain mandatory no matter how good the number looks. The acceptance metric is added to that framework; it doesn't exempt you from it. Intercom itself, which built its billing on a similar metric, writes in its own 2026 materials that it is moving from a binary “resolved or not” to assessing the outcome, because success has stopped being binary. In the same piece it gives the average resolution rate for its Fin agent — 76% across more than 12,000 teams (as the page reads as of October 2026). That is a vendor's self-report, not an independent measurement, and it should be read as exactly that. ## How do you take a baseline acceptance rate in two weeks? - Pick one process, not all of them at once. A metric stretched across the whole agent fleet gives you an average from which no decision follows. - Define the unit of measurement: one agent answer is one row in the log. Agree on this in writing before you start. - Log three fields per row: whether a person changed the content, whether there was an escalation, whether a guardrail was triggered. Track formatting and tone with a separate counter. - Don't touch the prompt or the settings for two weeks. Edits made during the measurement destroy the baseline you will later compare the next version against. - Calculate the three rates and look at the distribution by day, not just the average. A dip on particular days usually points to the type of request, not the model. - Add a spot check: a person rereads five percent of the accepted answers. Without this step the number flatters you. Two weeks is the minimum over which a weekly load cycle is visible. If the process has monthly seasonality (period close, reporting), the baseline has to be taken over a longer stretch. ## How do you word the acceptance threshold in a contract? “The agent works” is not an acceptance criterion: it always works, and the only question is how. We have already covered which four blocks are worth writing into an AI agent development contract; the acceptance criterion is the one most often worded in a way that makes it impossible to verify. Wording that makes sense in an SOW — the statement of work attached to the contract — looks roughly like this: on an agreed set of N real requests, the share of answers accepted without substantive edits is at least X percent with zero guardrail violations, and the measurement is carried out on the client's side and on the client's data. The value of X is not taken from an industry benchmark. It is set from your own baseline: first two weeks of measurement on the current process, then a threshold that makes sense here specifically. Separately, it is worth putting in writing that the client assembles the set of test requests — otherwise acceptance turns into a demo whose script was written by the same party that configured the agent. ## What does Auspex do during implementation? Auspex takes the baseline before launch, not after the complaints. Before the agent gets access to production data, the current state of the process is recorded and the set of requests for later acceptance testing is agreed. Escalation is not penalized in this setup: an agent that is punished for handing a difficult case to a person quickly learns not to hand off. And the spot check of accepted answers stays in the process for good, not only for the launch period — it is the only thing that keeps the number from turning into complacency. ## First steps toward your own number - Take one process where an agent is already running and see whether you can pull three numbers out of the logs at all: edits, escalations, violations. If you can't, that is your first task; it is usually a day's work. - Ask whoever rereads the agent's answers how many minutes a day it takes. Multiply by their rate and compare with the model bill. - If you are accepting a contractor's work right now, don't sign off on acceptance on the strength of a demo. Ask for a measurement on your own set of requests, assembled by your side. If you already have an agent running but don't have the three numbers, take a baseline on one process in two weeks: three counters, a threshold from your own data and a spot check of accepted answers. Grow2.ai, our AI division, measures agents as part of a free AI audit; if your questions are broader and start with the process and the CRM underneath the agent, that is a CRM and automation audit at Auspex. Both links are below. Read next - [Grow2.ai — our AI division: agents for your process and a free AI audit](https://grow2.ai/en/ai-audit/) - [What to write into an AI agent development contract](/en/blog/ai-agent-contract-what-to-write/) - [The business that updates itself: levels of autonomy without hiring](/en/blog/self-updating-business/) - [Why some automations stick and others quietly die](/en/blog/why-automations-get-abandoned/) - [CRM automation payback: how to calculate the period](/en/blog/crm-automation-payback/) - [Your AI agent has never been tested in Ukrainian](/en/blog/agent-safety-untested-in-your-language/) - [CRM and automation audit](/en/services/crm-audit/) ## Sources - [Ziegler et al. (2022), Productivity Assessment of Neural Code Completion — arXiv 2205.06537](https://arxiv.org/abs/2205.06537) - [GitHub Research — How GitHub Copilot helps improve developer productivity (14 July 2022)](https://github.blog/news-insights/research/research-how-github-copilot-helps-improve-developer-productivity) - [Sierra AI — τ-Bench: Benchmarking AI agents for the real-world (the pass^k metric, 20 June 2024)](https://sierra.ai/blog/benchmarking-ai-agents) - [METR — Measuring AI Ability to Complete Long Tasks (50% and 80% reliability bars, 19 March 2025)](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) - [Intercom — From resolutions to outcomes: how a vendor moves away from a binary metric](https://www.intercom.com/blog/from-resolutions-to-outcomes-evolving-how-fin-delivers-value/) Andrew Maryasov Founder, Auspex Founder of Auspex. Implementing CRM and automating business processes since 2015 — 1,500+ projects for 1,200+ companies. [amaryasov.expert →](https://amaryasov.expert) ## Frequently asked How is the acceptance rate different from adoption rate? Adoption rate shows how many people or teams have started using the agent. The acceptance rate shows what share of its answers you let through without intervention. The first number grows from the mere fact of launch; the second grows only when the agent can really be trusted. Two companies with the same adoption can have acceptance rates of 10% and 80%. What acceptance rate counts as normal? There is no universal norm, and no industry standard either. The reference point is the cost of an error in the specific process: where an error is expensive, low autonomy is the right decision, not a failure. The threshold is set from your own baseline after two weeks of measurement, not from someone else's benchmark. Should escalation to a human count as an agent failure? No. An escalation means the agent recognized the limit of its competence and passed the case on — exactly the behavior you paid for. The opposite is the warning sign: zero escalations usually means the handoff rule isn't working and the agent is carrying through what it should have handed to a person. Can this metric be gamed? Yes, and quite easily. All it takes is narrowing the agent's scope to the simplest requests, loosening the guardrails or rewriting the escalation rule. There is a fourth way, the quietest one: recalculate the denominator after the fact, throwing out the “atypical” requests. There is only one defense: fix the counting rule in writing before the measurement and don't change it along with the agent version. How long does it take to get a baseline? Two weeks, provided you don't change the prompt or the settings during that time. If the process has monthly seasonality — period close, reporting — the measurement has to run longer so that the baseline captures a full load cycle. We use an off-the-shelf service, not our own agent. Does the metric work? It does, and you need it even more. The vendor shows its aggregate figures across all customers, while what matters to you is your process and your requests. You can keep the three counters on your side regardless of whose agent is inside. Does this metric replace governance and access control? It does not. An agent registry, a separate budget and a named owner for each agent remain mandatory. The acceptance rate is added to that framework as an operational maturity metric and cancels nothing in it. ## Business automation digest 2–3 emails a month — what actually works in CRM and automation. No spam. Unsubscribe in one click. [Free CRM audit →](/en/contact/) ↳ FREE AUDIT 30 min · free · ↓ ## Ready to Get Your Sales Under Control? Free process audit — 30 minutes, and you'll receive: 1,200+ companies since 2015 · 30-day median go-live 01 A map of where you're losing leads 02 Priorities: what to automate first 03 An implementation plan for 2-4 weeks We'll call within 24 hours. The plan is yours, even if we don't work together.