The Fix Wasn't the Model. It Was One ADR.

Last week I argued that your agent didn't get dumber. Your process was never written down. Put the decisions in a durable place, version your context, and the model becomes the most replaceable part of your stack. That post ended on a claim I had only half-proven: that a good body of written-down decisions raises the floor for every model and narrows the gap between them.

So here is the experiment, and the short version first.

Now the whole thing, step by step, including the part where that one rule couldn't fix everything, because that part is the honest one.

The Setup

I build day to day with Claude Code. For this test I gave the exact same task to Grok, through its own command-line tool and my SuperGrok account. That's a different vendor's model entirely, not the one I normally reach for.

The feature was small and real. I run an AI-powered pull request review system: when a developer opens a pull request (the standard way you propose a code change on GitHub, GitLab, or Azure DevOps), it reviews the change and posts its feedback automatically. The task here was a new piece of that system. Before the detailed review runs, post one comment at the very top of the pull request that says, in plain English, what the change does, lists the files it touches, and shows which automated checks are about to run. The first thing a reviewer sees.

Both models got the same short brief. It named the internal rules the feature had to follow, but it did not include the text of those rules. Looking them up is part of the job. Each model queries our knowledge base over the OutcomeOps MCP server, pulls the relevant rules, and writes them into its plan before it writes any code. Why copy them in instead of just linking to them? Because a plan that says “follow our list rule” is a plan whose rule gets ignored. A plan that quotes the rule in full is one the model actually follows.

One rule mattered most here, and it comes from a mistake we had already made elsewhere in the system. It is about the one thing language models are famously bad at: they cannot count reliably, and they cannot be trusted to produce a complete list. That isn't a flaw to be patched. It simply isn't what they're for. A language model is built to predict and produce language. It is extraordinary at explaining, in clear English, what a change does and why it matters. Ask it for an exact count, like how many files a pull request changed, and it will hand you a confident number that is often wrong.

Make it concrete. A change touches seven files. Ask the model to summarize it and it might open with “this update changes six files,” or list a file that was never touched, or quietly leave one out. It reads perfectly and it is wrong. The worst kind of wrong, because a number that sounds sure of itself is the one nobody double-checks. Your code, meanwhile, can list those seven files and count them exactly, every single time, in a fraction of a millisecond. One of these two is the right tool for counting, and it isn't the model.

So we split the work by what each side is actually good at. The code assembles the parts that have to be exact: the list of files, the count, the checks about to run. The model writes the part that needs language and judgment: what changed and why it matters. We put that rule in one line:

The model narrates; the renderer counts.

The “renderer” is just the piece of code that builds the final comment. To “narrate” is to explain in words. So the line really means: the model is the writer, the code is the accountant. The writer describes the change; the accountant produces the numbers. Never ask the writer to do the accounting. It will guess, and it will sound certain while it does.

Round One: It Read the Rule and Broke It Anyway

Grok did the lookup correctly. It found the rule and copied that exact sentence, “the model narrates; the renderer counts,” into its own plan. I watched it do it. Then it built the one thing the rule exists to forbid: it let the model write the whole summary, counts and all, and then ran an automatic find-and-replace over the model's text afterward to strip the counts back out.

Read that again. It had the rule in front of it, in its own words, and still shipped the exact approach the rule is there to prevent. Cleaning up after the model is not the same as never letting it make the mess, because the next turn of phrase always slips past the find-and-replace. Grok copied the principle and never carried it into the design.

My code review caught it. It also caught a worse, unrelated bug: the way Grok assembled the file list could have stalled the entire check run on a large change, a cosmetic summary feature taking down the real work. Fixable, but telling.

Here is the part that matters. I did not fix Grok's code. I did not rewrite its prompt. I did not hand it the answer. I asked a different question: why did a capable model, holding the rule, still get it wrong? Because the rule stated the principle but never said, out loud, that the clean-it-up-afterward approach is itself the violation. It told the model what to believe and left just enough room to believe it and still do the wrong thing. That room is not a model defect. It is a gap in the document.

The Intervention: Change the Rule, Not the Model

So I changed the rule myself. I rewrote it to close the gap, and I did three plain things. I said, in so many words, that cleaning up after the model is not allowed. I applied it to every place we show a list, not just this one feature. And I described the approach that is allowed: the model can attach a short note to a row, but the list itself and the count stay the computer's, and any note that points at something not in the real list gets thrown away.

Then I saved it, loaded it back into the knowledge base, and checked that the new wording came back when I searched for it. I didn't touch the model. I didn't touch the prompt or the brief. I didn't hand it example code. The one and only thing I changed was that rule, written down more carefully and made findable so the model would pull the better version the next time it looked.

Then I reset Grok to a clean slate: same brief, no memory of the first attempt. (A field note, since this is a personal blog and not a press release: Grok doesn't save its work as it goes, and I lost its first finished build to a reset before I thought to grab it. I re-ran it from scratch, which is what you want for a fair before-and-after anyway.)

Round Two: It Converged

Same model. Same brief. Better knowledge base. This time Grok built the right design on its own. The model returns prose plus a set of short per-file notes. The code builds the table and the count from the real file list. Notes that point at a file not in that list are dropped. There is no find-and-replace cleaning up counts after the fact, because the model was never asked to produce them in the first place.

That is, almost line for line, the design my primary model built on its own branch. One rewritten rule moved a different company's model off the banned approach and onto the right one. Here is the before and after on the thing the rule governs:

The only thing that changed between round one and round two was one rule. That is the claim from the last post, made concrete: within a band of comparable models, the knowledge is portable, and the lever is what you've written down, not the model.

What the ADR Did Not Fix

This is the section that keeps the rest of the post honest. Tightening one rule converged the thing that rule governs. It did nothing for the things it doesn't.

One shared mistake rode straight through all three builds. Every one of them set the model's maximum response length too low. Claude set it to 700, Grok set 1,200 the first time and 2,000 the second, and the safe floor for this work is around 4,000. The rule I changed was about lists and counts. It said nothing about response length, so that mistake sailed past untouched in every build. Fixing one rule fixes the thing it governs and leaves the rest exactly where it was.

And the correct design introduced a brand-new failure. Because the summary is now built from a structured response, when that response gets cut off by the too-short length limit, there is now a broken, half-finished payload to deal with. Claude's build, when its model call comes up short, quietly falls back to a plain table. Grok's round-two build drops the raw, half-finished mess straight into the comment. The rule told it who owns the count. It never told it what to show when the model call fails, and that judgment, what to do at the edge, is the thing no document handed over. The model fixed the governed design and still showed weaker judgment on the part the document left open.

That is the real shape of the result. A written-down, versioned rule raises the floor and narrows the gap between models, exactly as advertised. It does not turn a model into a senior engineer. The corpus carries the decisions you've written down. The judgment on everything you haven't is still the model's, and across vendors it still varies.

The Problem Isn't the Model. It's the Missing Context Layer.

Strip the specifics and here is what happened. A model produced the wrong result. The fix was not a better prompt, not a different vendor, not a smarter model. The fix was a single decision, written down more carefully and made findable, so the model could pull it when it needed it. Change that one document and a different company's model changed its output to match.

So where was the actual problem? Not in the model. It was in where the knowledge lived. Most of the time, the only place a company's standards live is in the head of the person at the keyboard, the one typing instructions to the AI and hoping the model already knows the house rules. When that's the only copy, every result is a coin flip, and the knowledge walks out the door when that person does.

The fix is an organizational context layer: a single place the decisions live (versioned, searchable, and shared) that every person and every model draws from instead of guessing. I've written about what that layer looks like in what a good organization's intelligence layer looks like. Once it exists, you can change one entry and measure the effect across every model that reads it. You can keep a fixed set of tasks you re-run after a model update, so “it got worse” becomes a measurable diff instead of a rant online. And choosing a model becomes a normal purchasing decision with a known floor, instead of a rewrite.

The model wrote the wrong version. I wrote the right rule. Once that rule existed where the model would read it, the next run produced the right result with nothing else changed. Version your context and the model becomes the most replaceable part of your stack, right up to the edge of what you've written down.

Brian Carpio has spent 28 years in technology and 14 building enterprise cloud infrastructure on AWS, and ships AI in production daily. He writes about context engineering, governance, and the operating models that make AI-assisted development hold up at scale.