Shift Left for AI: Your Agent Didn't Get Dumber

The Week the Model Got "Nerfed"

Every couple of months my feed fills up with the same post. The model got nerfed. It's a complete idiot now. Someone wants a lawsuit, an apology, or a refund. This week it was Opus, and the threads were everywhere.

One person said their coordinator agent stopped coordinating. Another said the task panel was a wall of failed attempts. A consultant said his client deadlines were now impossible and he was looking at a Chinese provider for "stability."

Meanwhile I'm running the same models across four projects: an enterprise context engineering platform, a federated search SaaS, a team puzzle game, and an AI music project where Claude drives ComfyUI on my GPU server to render video. I didn't notice anything. Not because I'm lucky, and not because I'm doing simpler work. Because my process doesn't live inside the model.

What the Complaints Have in Common

Read the angriest post closely and the pattern shows up fast. The author built a custom plugin with hooks around Claude. A coordinator chat spawns sub-chats. It never compacts; instead a hook dumps messages to a file and the next session is supposed to resume from a state.md. It ran 300 tasks through plan, judge, code review, and fix without a glitch. Then one morning it fell apart.

Look at what that system actually depended on:

• The next session noticing it should read state.md. Nothing enforced it.
• The coordinator inferring that a sub-chat finished 30 minutes ago. That's a guess, not a mechanism.
• The model handling passwords and server security work the same way it did last month. That's exactly the kind of behavior a provider tightens on purpose.

Every link in that chain was emergent behavior. It worked because the model happened to do the right thing, consistently, for a while. Shift that behavior a few degrees and the error compounds across every step of a multi-stage pipeline.

I'm not saying regressions never happen. Providers have published postmortems on real quality degradations, and the people who noticed weren't imagining it. But "my workflow broke" and "the model got dumber" are different claims, and most of these posts can't tell them apart. They have no baseline to diff against. Their spec is a feeling about how the model used to act.

Artifacts Over Behavior

My process lives in files the model reads, not in habits I hope it keeps. In OutcomeOps that corpus is patterns, ADRs, coding standards, and instructions for nearly everything I do. It holds 13 years of AWS judgment in a form any model can consume over MCP.

That changes the job I'm handing the model. It's no longer "figure out what Brian would do." It's "apply what Brian already decided." The second job is much easier, and much less sensitive to whatever changed in the latest model update.

The best example is a boring one. I have an ADR about the mistakes LLMs make with Decimal types in DynamoDB. Every model makes that mistake, because the training data is full of code that passes floats to boto3 and breaks. A model can't reason its way out of a gotcha it doesn't know exists. But every model I've tried will follow an ADR that names it. I wrote that fix once, and every model inherits it.

It also makes failures visible. When output drifts from a documented pattern, I can point at the exact line it violated. That's a bug report. "It feels dumber today" is not.

The Plan Directory

The coordinator in that Reddit post had to guess what came next. Mine doesn't, because the plan is written down before any agent starts.

Every piece of work gets a plan/ directory. Inside is an overview.md and a set of chunk files: backend, frontend, tests, infrastructure, whatever the work needs. It might be 5 chunks or 20. The overview spells out which chunks can run in parallel and which have to run in sequence. Each chunk carries everything its agent needs to do the job: the scope, the files, the patterns that apply.

I review every chunk and the overview before anything spawns. Then agents run against their chunks, in parallel where the overview allows. The only human gate is before the fan-out; after that, the overview decides what runs together and what waits.

Two things fall out of this for free:

• No coordination guesswork. The dependency graph is a file, not an inference. Nothing has to notice that another agent finished.
• No scope creep. An agent working from a self-contained chunk has no reason to go refactor code it was never asked to touch. People complain about models wandering off and "improving" things. Mine don't, because the boundary is written down.

Shift Left, One Layer Up

This is just shift left. In DevOps we learned that a bug CI catches after 100 lines is cheap, and the same bug security catches in production is expensive. Reviewing the plan before agents spawn is the same idea applied to AI. The human judgment happens up front where it's cheap, not after 30 minutes of a stuck loop.

The cost curve is actually steeper with agents. A developer who misreads a requirement writes the wrong 100 lines over an afternoon. A flawed plan fanned out to ten parallel agents writes thousands of wrong lines in twenty minutes. Worse, they're internally consistent, so they look fine until integration. Every error in the plan gets multiplied by the fan-out, which makes the review gate before spawn worth more than any CI gate.

The mapping holds across the stack:

What changed is the economics. Writing a thorough spec used to be expensive enough that teams rationalized skipping it. Now the agent drafts the chunks and I only review them. The most expensive part of shifting left became the cheap part.

The Model Becomes Replaceable

Here's the part the "switch to another vendor" crowd misses. When the judgment lives in the corpus, swapping models is a substitution, not a rewrite.

OutcomeOps supports both Claude and GPT, and we proved the swap works. We also tested Grok on Bedrock. It built more or less the same code maps and symbol graphs as the others. When I point Claude Code at those models through Bedrock, they still pull from the same MCP server to find the right Terraform patterns, the right Python standards like Pydantic, and the ADRs about common LLM mistakes.

To be clear about the limits: this holds within a capability band. I'm not going from a frontier model to a small open model on my 5090 and expecting the same results. The model still has to recognize which ADR applies to a new situation and notice when something falls outside the corpus. A good corpus raises the floor and narrows the variance. It doesn't turn a weak model into a strong one.

But across frontier vendors, the knowledge is portable. That's the stability the consultant in that thread thought he was buying by changing providers. He'd get the same problem with a different logo, because his process still lives in somebody else's model.

Stop Debugging the Vendor

The real grievance in those threads is fair: model behavior isn't versioned the way people expect software to be. But the fix isn't a lawsuit or a new provider. It's the same discipline we already apply to code.

• Write the decisions down as ADRs and patterns, not prompts you hope stick.
• Put the plan in files and review it before anything spawns.
• Give every agent a bounded chunk, not a vibe.
• Keep a fixed set of tasks you can rerun after a model update, so "it got worse" becomes a diff instead of a rant.

DevOps taught us to stop depending on the one engineer who knows how the server works. Context engineering is the same lesson one layer up: stop depending on the one model that happens to behave the way you like this month.

Version your context, and the model becomes the most replaceable part of your stack.

Brian Carpio is the founder of OutcomeOps and RetrieveIT.ai. He has spent 28 years in technology and 14 building enterprise cloud infrastructure, and ships AI in production daily.