# Automating undocumented processes: the cost | Auspex > One model, the same 70 tasks: 26% without written rules and 99% with them. What this means for automating undocumented processes and what exactly to write down. Source: https://auspex.company/en/blog/undocumented-process-agent-blindspot/ [Blog](/en/blog/) / Field Notes # Automating undocumented processes: the price of what nobody wrote down By: Andrew Maryasov · Aug 25, 2026 · 10 min read Automating undocumented processes doesn't fail because the model is weak. In August 2026 the authors of the DiG-bench benchmark ran a direct test: the same Gemini 3.1 Pro on the same 70 tasks won 18 of 70 when it had to discover the rules on its own, and 69 of 70 when it was simply given a short description of those rules — 26% versus 99%. The difference between failure and a near-perfect result wasn't the model, the agentic harness or the budget; it was whether a text existed. Auspex looks at what carries over to business processes and, above all, what exactly needs to be written down: the description that worked in the experiment is built nothing like a typical procedure document. The pilot runs flawlessly in the demo, in production the agent starts doing strange things, and the team goes looking for a stronger model. A month later the same thing happens with the new model, only at a higher price. It's tempting to put this down to the quality of the model, because that explanation comes with a clear next step. But in August 2026 a benchmark came out that measured a very different factor. ## The experiment that measured the price of an unwritten rule DiG-bench (Discovery in Games) was published on 12 August 2026. The work was led by the research group Thinking About Thinking in collaboration with Oxford, Princeton, MIT, Inria and KAUST; the co-authors include Jürgen Schmidhuber, Joshua Tenenbaum, Thomas Griffiths and Tri Dao. The setup: 70 text games, each with its own transformation rules. The player knows neither the rules nor the win condition; the only way to learn them is interaction — make a move, see what changed, revise the hypothesis. The games are spread across seven tiers of difficulty; the authors released 21 games publicly and kept 49 private for fair measurement. They also point out that these games exist nowhere else on the internet, so a model could not have seen them in training — data contamination simply isn't a question here. The authors ran a direct ablation: they took Gemini 3.1 Pro and gave it the same 70 games, this time with the text of the rules in hand. > Giving Gemini access to the ground-truth rules of the game, specified in concise natural language, raises the win rate from 18/70 to 69/70 games. > > — DiG-bench: Discovery in Games, arXiv:2608.12593 The difference between those two numbers was a short text that, in the first case, simply didn't exist. The authors state the conclusion without softening it: the main difficulty of their benchmark lies precisely in working out the rules. ## What exactly the model was given — and what that description left out The authors characterize the description that lifted the result to 99% very precisely: it summarized the dynamics of the game and the win condition, but contained nothing policy-related — no strategies, no tactics, no move sequences. > These rules summarized the dynamics of the game and the win condition; but they excluded any policy-related information like strategies, tactics or move sequences. > > — DiG-bench: Discovery in Games, arXiv:2608.12593 Nobody explained to the model how to play. They explained what game it was playing and what counts as winning here. When a company is told “describe the process,” it usually writes exactly the part that was removed from the experiment. The result is a set of instructions: call first, then create the deal, then send the proposal, follow up in three days. That is a sequence of moves. The mechanics — what objects exist at all, what states they can be in, which transitions are allowed and which are not — stay unspoken, because inside the team they look so obvious that it feels embarrassing to say them out loud. The mechanics of a process are the answers to questions nobody asks in meetings: - which entities exist (request, customer, deal, return, complaint) and what distinguishes one from another - what states each of them can be in and which transitions between states are allowed - what must never be done, no matter how reasonable it looks in a particular case - which exceptions exist and how they are recognized — not “sometimes it's different” but “if the customer is on the key accounts list, a different rule applies” - where information that isn't in the system comes from and who has the authority to confirm it The success condition is a separate block, and it almost never gets written: - what observable result shows that the step was done correctly - who or what checks this, and by when - how “done badly” differs from “done wrong,” because these are different events with different costs The description that worked in the experiment is closer to the rules of a board game than to a job description. It answers the question “how does the world work here and what counts as winning,” not the question “how do I win.” The difference isn't cosmetic. Instructions describe one route and say nothing about anything outside it, so an agent that leaves the route is left without bearings. A description of the mechanics, by contrast, sets the boundaries of the field: inside them the agent can choose its own path, and by definition it cannot step outside them. ## A better tool won't make up for it The natural reaction to a failed pilot is to look for a stronger harness: another framework, a platform with memory, an agent with a planner. The benchmark's authors tested that hypothesis too. They took Prime Agent — Opus 5 running in the Prime Agent product — which had recently scored 95.5% on the ARC-AGI-3 public games, and ran it on the two hardest tiers of DiG-bench. It showed no improvement over the basic harness with the same Opus 5. The overall conclusion is worded cautiously but unambiguously: existing harnesses offer limited gains in discovery ability specifically. This result shouldn't be stretched too far: the agent does discover some things on its own. Without the rules Gemini still won 18 games, and the strongest model in the set, Opus 5, won 50 of 70. If you count, for each game, the best of all the models tested, together they cover 57 of 70 — but only 9 of 20 on the top two tiers. So implicit rules do get discovered, just not reliably, and the failures land exactly on the hard cases. For implementation this has a very practical consequence. Where the rules aren't written down, you pay not for a license but in the variance of results: the agent will handle ten requests the way it should, and on the eleventh it will do something nobody expected. And it will turn out that the rule it broke really wasn't recorded anywhere — which means that, formally, it never broke it. ## Where the analogy ends The benchmark measures rule discovery through active experimentation in text games. The player there has a sandbox: a bad move costs nothing, and hundreds of attempts are possible. The authors deliberately don't score step efficiency, so as not to penalize exploration. A business process works differently. Here an agent's experiment is a real call to a real customer, a real discount and a real invoice. There is no sandbox, and the cost of a wrong hypothesis is not zero. So the numbers can't be carried over literally: 26% and 99% are results in text games, not metrics for automating a sales department. The analogy works in one direction, and only as an analogy: if a model with an unlimited number of free attempts won 18 games of 70 on its own, then in an environment where every attempt costs money and customer relationships there is even less reason to count on it discovering the rules on its own. ## How to get the mechanics out of people's heads and into text The question “tell me how you do this” gets you instructions — that same sequence of moves. The mechanics can't be extracted that way at all: someone who has run a process for ten years doesn't see their own rules, because to them these aren't rules but simply how the world works. A different kind of question works — questions about boundaries and exceptions: - “What happens if you do it the other way around?” — the answer “well, nobody does it that way” almost always hides an unspoken rule - “When did you last deviate from the instructions, and why?” — this brings out both the exception and the cue it was recognized by - “What mistake does a new hire make in their first week?” — that's a forbidden transition that isn't in any document - “How do you know it's been done well before the result comes in?” — a success criterion that usually exists only as a feeling - “What do you check before you press the button?” — a hidden precondition without which the step makes no sense The document you put together after a conversation like this has three parts: a list of entities and their states, a list of forbidden transitions with reasons, and a success criterion with a description of who checks it and how. Strategies, techniques and “how to do it better” tips don't belong here — those are written separately, and for people. There's a simple way to check whether the description is complete: give it to someone who doesn't know the process and ask them to name a situation in which the document doesn't say what to do. Every such situation is exactly where the agent will start guessing. ## Where the implementation budget actually goes A pilot budget is usually made up of integrations, prompts, testing and a model subscription. The work of extracting implicit rules isn't in it — it's either skipped with “we have everything documented” or discovered after launch, when the price is already a stalled project. This work takes days or weeks, depending on how many exceptions the process has accumulated, and putting it in the plan is more honest than explaining later why the pilot never left the test environment. Auspex puts it first precisely because it is portable: the documented mechanics stay with you regardless of which model you choose a year from now, or whether the one you started with is still on the market at all. This work ties in with the previous part of this analysis. The article “The business that updates itself” was about why growth without hiring runs into the description of the business that lives in employees' heads. DiG-bench adds a price tag to that picture: it shows what the same gap costs when you put an agent, not a person, on the process, and why buying a harness instead of the description doesn't work. ## Where to start this week - open the procedure document for the process you plan to hand to an agent and look for a list of forbidden actions — if all it has is a sequence of steps, you don't have a description of the mechanics - in the same document, find the sentence that answers the question “what shows that the step was done correctly” - ask the person who runs the process when they last deviated from the instructions, and write down the cue that made them decide to If this check shows that the process rests on the memory of two employees, the model isn't the problem — and there is no point in changing it. You can bring that list of gaps to a CRM and automation audit. In one meeting Auspex works out which of your processes is described well enough to hand to an agent and which first needs to be pulled out into text. The service page is in the links below. And a detail people usually notice only on their second project: the description of the mechanics has to live where the agent reads it from, not in a slide deck for management. A document kept apart from the system drifts away from reality in about a quarter, and then you have two sources of truth instead of one. Read next - [The business that updates itself: scaling without hiring doesn't start with agents](/en/blog/self-updating-business/) - [Three signs you shouldn't hand a step to an agent](/en/blog/agent-handoff-decision/) - [Why automation doesn't stick — and which half to switch off](/en/blog/why-automations-get-abandoned/) - [What to write into an AI agent development contract](/en/blog/ai-agent-contract-what-to-write/) - [CRM and automation audit](/en/services/crm-audit/) Sources - [Battleday, Sandbrink, Whittington et al. — DiG-bench: Discovery in Games, arXiv:2608.12593, 12 August 2026 (primary source; all figures come from here)](https://arxiv.org/abs/2608.12593) - [DiG-bench — official benchmark website and leaderboard](https://digbench.ai) - [DiG-bench — technical report and basic harness (GitHub)](https://github.com/discos-research/dig-bench) Andrew Maryasov Founder, Auspex Founder of Auspex. Implementing CRM and automating business processes since 2015 — 1,500+ projects for 1,200+ companies. [amaryasov.expert →](https://amaryasov.expert) ## Frequently asked What is automation of undocumented processes, and why is it harder than ordinary automation? It is the automation of a process in which a large share of the rules isn't recorded anywhere and exists only as the experience of particular employees. The difficulty isn't the number of steps; it's that the agent gets an incomplete description and has to fill in the missing part with guesses. The DiG-bench benchmark measured the contribution of this factor directly: the same model on the same 70 tasks scored 18 wins of 70 without a description of the rules and 69 of 70 with one. Is it true that an AI agent can't figure out an undocumented process on its own? It can, but not reliably. In DiG-bench, Gemini 3.1 Pro won 18 games of 70 without the rules, the strongest model in the set, Opus 5, won 50 of 70, and the best results of all the models tested add up to 57 of 70. On the two hardest tiers, though, that total drops to 9 of 20. The practical conclusion: in simple cases the agent reconstructs the rules, in hard ones it fails, and you can't predict in advance which case will come up in production. What exactly needs to be written down before deploying an AI agent? Two things that a procedure document usually lacks: the mechanics and the success criterion. The mechanics are the list of entities and their states, the allowed and forbidden transitions, the exceptions, each with the cue that identifies it, and the data sources outside the system. The success criterion is the observable result that shows a step was done correctly, plus who checks it. Tellingly, in the DiG-bench experiment Gemini 3.1 Pro was given exactly this kind of description — the dynamics and the win condition — and strategies, tactics and move sequences were deliberately excluded from it. Will a stronger model or a better agent framework help? On the available evidence, no — not if the problem is unwritten rules. The DiG-bench authors tested Prime Agent, built on Opus 5, which scored 95.5% on the ARC-AGI-3 public games, and on the hardest tiers it showed no improvement over the basic harness with the same model. Their conclusion: existing harnesses offer limited gains specifically in the ability to discover new rules. How long does it take to describe a process before automating it? Days or weeks, depending on how many exceptions the process has accumulated and how many people perform it differently. The very first working session with the people who do the work gives an accurate estimate: the number of exceptions that surface in the first hour of conversation is a good predictor of the total scope. What matters here isn't speed but the fact that the result is portable — the documented mechanics remain valid when the model, the platform or the contractor changes. Can the 26% and 99% be applied to your own business? No, and that matters. DiG-bench measures rule discovery through experimentation in text games, where a failed attempt costs nothing and you can make hundreds of them. A business process has no sandbox: an agent's attempt is a real call, discount or invoice. So the benchmark's numbers are an argument about direction, not a forecast of results: if, with an unlimited number of free attempts, a model won only 18 games of 70 on its own, then where every attempt costs money there is even less reason to count on it discovering the rules on its own. ## Business automation digest 2–3 emails a month — what actually works in CRM and automation. No spam. Unsubscribe in one click. [Free CRM audit →](/en/contact/) ↳ FREE AUDIT 30 min · free · ↓ ## Ready to Get Your Sales Under Control? Free process audit — 30 minutes, and you'll receive: 1,200+ companies since 2015 · 30-day median go-live 01 A map of where you're losing leads 02 Priorities: what to automate first 03 An implementation plan for 2-4 weeks We'll call within 24 hours. The plan is yours, even if we don't work together.