Why a fifty-year-old principle is the most important design decision you’ll make on your AI stack.
The Dutch Computer Scientist Edsger Dijkstra named separation of concerns in 1974 and the paper he wrote it in was, characteristically, not gentle about it. The idea was simple: each part of a system handles one thing. Not one thing plus a few edge cases you’ll clean up in the next sprint. One thing. And no part should need to know what its neighbors are doing to do its job correctly. The failure mode when you don’t is remarkably consistent across fifty years of software history. Structured programming. Object-oriented design. Service-oriented architecture. Microservices, which are really just separation of concerns applied at the deployment boundary. Every major architectural shift has been, in some sense, the industry rediscovering this same principle after violating it badly enough that the pain became undeniable. A billing service that knows too much about how notifications work will fail in ways a billing service has no business failing. The concerns bleed into each other. The postmortems get long and confusing. Now we’re building agentic AI systems. The principle is the same. The stakes are higher, and the failure modes are stranger.
There’s a version of this you’ve probably seen in a demo. Someone shows a system that takes a user request, breaks it into steps, calls a few tools, retrieves some information, synthesizes a result, and delivers an answer. It looks like magic. The audience nods. Someone in the back asks if it’s production-ready.
It usually isn’t. Not yet. And a big reason why is that the demo version almost always cheats on architecture. Everything is crammed into one agent. One context window. One model that’s simultaneously trying to plan, retrieve, decide, execute, and respond. It works when the task is short and the context is clean. It falls apart when the task is real.
What an Agentic System Actually Is
A properly designed agentic system looks more like a graph than a pipeline. There’s a planning layer that decomposes goals into sub-tasks. A retrieval layer that pulls the right context from memory or a knowledge base. Execution agents that handle individual tool calls or API interactions. A memory layer that tracks intermediate state. An orchestration layer that sequences everything and handles failures when they happen, which they will.
Each of those is a distinct concern. The planning agent shouldn’t be doing retrieval. The retrieval layer shouldn’t be deciding what to do with what it finds. And the execution agent absolutely should not be sitting on a 50,000-token context window packed with full conversation history, every retrieved document, the system prompt, and three prior agent responses. That last one is the mistake that leads to the problem with a name.
Lost in the Middle
Here is something that sounds like it should be obvious but apparently wasn’t, because it took a 2023 paper from Stanford and UC Berkeley to get people to take it seriously: models don’t read long contexts evenly. They attend well to what’s near the beginning and what’s near the end. Everything in the middle gets deprioritized. Not ignored entirely. Just not weighted the way you’d expect, and in ways that are genuinely difficult to detect until something goes wrong.
The researchers called it “lost in the middle.” They tested models on retrieval tasks where the relevant passage was placed at different positions in the context window. Performance followed a U-curve. Strong at the edges, weak in the middle. The longer the context, the worse it got.
Think about what that means for a poorly designed agentic system. An orchestrator that stuffs one agent’s context with the original query, a multi-page system prompt, twelve retrieved documents, the full tool call history, and several prior agent responses is not being thorough. It’s creating a structural guarantee that the model will miss things. The critical fact the whole task depends on is sitting at token 18,000 in a 32,000-token window. That’s exactly where it’s least likely to land. The system was designed to put it in an impossible position.
The Concern Explosion Problem
There’s a version of this mistake that’s easy to rationalize. You give one agent a few extra responsibilities because it’s faster, because the deadline is close, because it mostly works in testing. And it does mostly work, for a while. The demo is clean. The early users don’t notice. Then the context gets longer, the task gets more complex, and the model starts dropping things. Not dramatically. Quietly. The kind of failure that takes three debugging sessions to even locate.
What’s actually happening is cognitive load accumulation. Every concern you stack onto a single agent is another thing competing for the model’s attention in that context window. Intent understanding, retrieval, tool selection, tool execution, result synthesis, response formatting. Each one is fine in isolation. Together, they create a model that loses track of its original goal after several tool calls, hallucinates facts that were technically in the context but not attended to carefully, and cycles through tools it already used because it can’t clearly separate what it’s done from what it still needs to do.
The microservices comparison is almost too on-the-nose. A monolith handling authentication, billing, user data, and notifications in a single service isn’t just a maintenance headache. It fails in correlated ways. A billing bug corrupts user data. A notification spike slows authentication. The concerns bleed. A bloated agent context does exactly the same thing, just faster and with less interpretable stack traces.
Applying the Principle: Agents as Bounded Contexts
The fix isn’t complicated to describe. It’s just harder to do than not doing it, which is why so many teams skip it until they’re staring at a production failure they can’t reproduce.
The practical move is borrowing a concept from Domain-Driven Design: treat each agent as a bounded context. Its own inputs, its own outputs, a well-defined interface to everything outside it. The planning agent doesn’t need to know how the retriever ranks documents. The retriever doesn’t need to know what the planner intends to do with what it finds. They communicate through clean interfaces. They do not share context.
A few patterns fall out of this naturally once you commit to it.
The planner-executor split. Keep goal decomposition separate from task execution. A planning agent breaks the user’s intent into sub-tasks and hands them off with clean specifications. An execution agent handles each sub-task with a tight, focused context: here is the task, here are the tools, here is what you need for this specific step. The planner doesn’t execute. The executor doesn’t plan. This sounds obvious. You’d be surprised how rarely it’s actually enforced.
Context scoping per agent call. Before calling an agent, the orchestrator assembles only the context that agent needs for its specific task. Not the full history. Not every retrieved document. Just the relevant slice. Context windows stay short. The lost-in-the-middle failure mode becomes structurally much less likely. And each agent call becomes independently testable, which matters enormously when something breaks at 2am.
Memory as a separate concern. An agent shouldn’t manage its own memory. It receives pre-retrieved, pre-ranked context from a dedicated memory layer, then reasons over it. Short-term working memory, long-term episodic memory, and the retrieval layer that surfaces relevant facts are each their own layer with their own logic. This is how retrieval-augmented generation works in the single-model case. The same principle scales to the full agent graph.
Observation and action separation. In ReAct-style agents, reading tool outputs and deciding what to call next are logically distinct steps. A lot of implementations conflate them into a single chain of thought that mixes raw tool output with reasoning. Separating them, even just structurally in the prompt, improves reliability. The model reasons over a clean summary, not a raw firehose of JSON.
This is the part people get wrong most often, probably because orchestrators don’t look like agents. They don’t reason. They don’t call tools. They just route. So teams treat them as neutral plumbing and don’t apply the same discipline.
But the orchestrator makes real decisions. Which agent to call. In what order. With what context. Under what failure conditions. What to do when an agent returns something unexpected. Those decisions are a concern, and they belong in one place, cleanly separated from the agents doing the actual work.
The Orchestrator Is a Concern Too
A well-designed orchestrator doesn’t leak its routing logic into agent prompts. It doesn’t ask an execution agent to decide whether to invoke another agent. It owns the sequencing, the retry logic, and the state management. If the orchestrator is getting complicated, that’s where the complexity belongs. Not scattered across eight agent system prompts where you’ll never find it when something goes wrong.
The orchestrator should also be the only component with a global view of task state. Agents see their slice. The orchestrator sees the whole picture. That’s intentional. An agent that knows too much about the overall task will start making decisions based on incomplete information about what other agents are doing. That’s how you get emergent, difficult-to-reproduce behavior that looks random but isn’t.
What This Looks Like in Practice
This is the part people get wrong most often, probably because orchestrators don’t look like agents. They don’t reason. They don’t call tools. They just route. So teams treat them as neutral plumbing and don’t apply the same discipline.
But the orchestrator makes real decisions. Which agent to call. In what order. With what context. Under what failure conditions. What to do when an agent returns something unexpected. Those decisions are a concern, and they belong in one place, cleanly separated from the agents doing the actual work.
A well-designed orchestrator doesn’t leak its routing logic into agent prompts. It doesn’t ask an execution agent to decide whether to invoke another agent. It owns the sequencing, the retry logic, and the state management. If the orchestrator is getting complicated, that’s where the complexity belongs. Not scattered across eight agent system prompts where you’ll never find it when something goes wrong.
The orchestrator should also be the only component with a global view of task state. Agents see their slice. The orchestrator sees the whole picture. That’s intentional. An agent that knows too much about the overall task will start making decisions based on incomplete information about what other agents are doing. That’s how you get emergent, difficult-to-reproduce behavior that looks random but isn’t.
Dijkstra was famously impatient with complexity that engineers had chosen rather than inherited. His argument was that a system’s intellectual difficulty should reflect the difficulty of the problem it solves, not the carelessness of its designers. A system that’s hard to understand because it mixes concerns isn’t complex. It’s merely complicated. He considered that a moral failure, not just a technical one.
Agentic systems are genuinely complex. Multi-step reasoning under uncertainty, with tools, with memory, with partial observability of the world, is a hard problem. It doesn’t need to be made harder by stuffing every concern into a single agent context and hoping the model figures it out.
The principle is fifty years old. That was true of structured programs in 1974. It was true of microservices in 2015. It’s true of the agent graph you’re building right now.
24 years to this day, have passed since Prof Dijkstra is no longer with us. Each time you present an argument about the Separation of Concerns, he’s probably smiling at you from above.
Leave a Reply