The Future You Can't Retrieve

AI can read the future it is given. Not the future human obligation hasn't decided yet. On singularity, pace, and why humans as sensors and outcome-shapers still have a very material role to play.

Anthropic, or more specifically Marina Favaro and Jack Clark, published a piece last month called When AI builds itself, on Anthropic’s progress toward recursive self-improvement. It is one of the more honest things a frontier lab has put out. They show, with internal data rather than press-release numbers, that AI is already speeding up the building of AI. More than 80% of the code merged into their own codebase is now written by Claude. A model went from a 3x speedup on a constrained optimisation task to 52x in a year. The typical engineer ships eight times the code they did in 2024. They are careful, they caveat the lines-of-code metric themselves, and they lay out three futures rather than selling one. Worth reading in full.

I found myself agreeing with almost all of it, and then struggling, near the end, over a single move.

The move is the transfer. The whole engine of the piece runs inside AI research itself, and that is precisely the most legible domain that exists. Clean feedback, crisp metrics, the data sitting on the lab’s own machines, a loop that closes because every part of it can be measured, evaluated and refined. Then, near the end, the claim quietly widens: that a system which can do this “would have skills that would transfer to the rest of science,” and onward into the economy. That is the sentence I struggle with. Not because the skills won’t help elsewhere. They will. But the leap assumes the rest of the world is as legible as a training cluster. Most of it isn’t, and the piece half-knows this. It says so itself: the loop is still incomplete “in exactly the places that matter most: problem selection, metric design, transfer from bounded settings to production.” It even concedes that more intelligence “can’t turn a stranger into an old friend in a weekend.”

I want to take that concession seriously, because I think it is bigger than the authors let it be.

Here is the usual way people frame the gap, and the way I think is wrong. The story goes: the illegible parts of the economy are things the AI doesn’t yet know. The regulator’s intent, the counterparty’s real position, the politics inside a client. Information not yet in the corpus. If that were the whole story, it’s a retrieval problem. And retrieval problems get solved, on a curve not unlike the one in the Anthropic charts.

But in the domains where the stakes are highest, the decisive variable isn’t undiscovered. It’s unmade. A regulation, a deal, a mandate is not a fact waiting to be found. It is an outcome still being produced, slowly, by people acting on each other. A board in San Francisco backs the founder it already trusts over the one with the better deck. A founder who owes an investor an early cheque takes the lower offer. A regulator who is closer to one player than another, consciously or not, writes a slightly different rule. You cannot retrieve a future that human reciprocity hasn’t decided yet.

The human isn’t a better reasoner. The human is in the room.

It is tempting to read all this as “the human knows more,” and to assume the gap closes the moment the model can see what the human sees. So let me give that its due, because it is half true and the honest half matters.

Hand a capable model the full brief, the weighted scenarios, the history, the constraints, and it reasons well. Often better than you. On pure inference under stated information, the human edge is already thin and getting thinner. If this were a contest of working out the answer from a complete picture, I would not be writing this.

It isn’t. And I’ll use a case I was part of, because it is the cleanest version I know.

Some years ago we looked at an early-stage agent network business in West Africa. Growing fast, real signal. The committee split, two and two. What broke the tie was not a number. We had a good read on where the telcos were leaning and driving towards, and that gave a particular argument more weight than it might otherwise have had: that the telcos were about to move into this space, leverage their own agent footprints, and crush any independent network. So we passed. The analysis was clean and, on its own terms, I still think it was reasonable. Every sign pointed that way.

The telcos never got to move. The banks had more weight with the regulator, and that weight held them back from the space far longer than any of us had modelled. Into that gap, the business we passed on outgrew our predictions handsomely. One of its earliest backers, as it happened, was a well-known bank founder, presumably rather closer to the regulator than we were. Did that relationship help drag out the telco decision? I can’t prove it, and I won’t pretend to. But the point doesn’t need the proof. Our mistake wasn’t that we missed a fact, or even that we read the telcos wrong. We read them about right. The fact simply didn’t exist yet. Whether the telcos would be let in was still being decided, slowly, by people with relationships none of us were part of. We treated an unmade outcome as a discoverable one, and the best private signal in the world wouldn’t have fixed that, because the thing that decided it hadn’t happened when we ran the numbers.

So the human in these rooms holds two positions the model does not, and neither is about reasoning.

The first is sensor. The decisive signal often arrives through embeddedness, not through text. The hesitation in a regulator’s phrasing. The fact that the CFO and the CTO of your client have quietly stopped trusting each other. Who actually has the ear of the person deciding, this week, not on the org chart. These do not sit in a dataset because they were never written down, and frequently never will be. A model can only reason over what reaches the prompt. It does not know what it is not being told, and, tellingly, it rarely thinks to ask. The recent work on this is unkind to the optimistic view: models often recognise ambiguity when you point at it directly, yet in normal use they default to answering rather than clarifying, and feeding them retrieved context can make them less likely to ask, not more.

The second position is the one the sensor framing misses, and it is the real point. The human is not only reading the outcome. The human is part of what produces it. Anthropic’s own piece catches the edge of this without quite naming it: one of their engineers mourns that working with Claude is faster but “creates zero debt,” each interaction a lost bid for human collaboration. He is right that something is lost, but debt is only the surface of it. Underneath is standing. The thing built up over years of doing what you said you would: a reputation others rely on, a continuity that means you will still be here next cycle to settle the account, an accountability that makes your word worth taking. That is what a founder draws on when the regulator takes the call. It is what gets the term sheet read first. It is what decides which project survives next quarter, as much as the merits do. An agent with perfect retrieval at the moment of decision sits outside all of it. It did not build the standing, and it cannot feel the live read. So the human edge is not intelligence. It is position. Sensing, and standing.

Which is why governance is the question, not the model.

Follow that one step and the commercial picture turns over.

If the decisive context is relational, partly tacit, and partly produced rather than found, then “which model is smartest” stops being the interesting question. Raw capability is commoditising anyway. The smartest model is becoming a utility. The interesting question is who gets to put a capable model next to the live context, and on what terms.

Note what this is not. It is not that closed systems block the capability. A model does not need to train on your data to reason over it. Retrieval, long context, fine-tuning inside your own perimeter, confidential computing, sovereign deployment, all of it lets a frontier model operate on your most sensitive records without those records ever entering anyone’s training run. Closed data is not a capability barrier. It is a commercial and institutional one. It does not decide whether the work can be done. It decides who is allowed to do it, and who captures the result.

Governance is just the name for controlling that boundary. Who connects the model to the proprietary record, the supervisory relationship, the workflow. Who is accountable for what it does there. Who decides what the system is even pointed at first. I should be precise about the layer, because it cuts two ways. At the very frontier, building the models, compute and data concentration is still a real and durable moat, and I don’t expect that to move soon. The claim is about the layer where the rest of us operate. There, the edge is drifting away from owning the model and toward governing its access to context the model cannot reach on its own.

So where does the value actually go.

Put the pieces together and you get a claim about pace, which is the only kind of claim I am willing to make here.

The recursive loop closes first and runs fastest where the world is legible, instrumented, and owned, which is why it closed inside AI research before anywhere else. It spreads outward at the speed of its slowest component. Anthropic invoke Amdahl’s law in their own piece, about code review becoming the bottleneck once everything upstream sped up. The same law applies to the economy. Overall pace is capped by the parts that don’t speed up, and the parts that don’t speed up are the relationship-laden, obligation-shaped ones. To be clear, I think the bulk of Anthropic’s picture is right. AI will transform most of the economy, deeply. My quarrel is only with the word evenly. Transformation and simultaneity are different claims, and the curves tend to smuggle the second in under the first. Singularity arriving across every domain at once is harder than the charts suggest, not because AI can’t reason or retrieve, the two things it is racing past us on, but because the effect of one decision on the next, carried through people acting on each other, is the part that resists codification.

That splits value capture from value creation, and they don’t sit with the same people. Capture may well concentrate with the frontier-model and compute owners, the utility layer. But a great deal of the value creation sits much closer to the real-time human decision-maker. The one governing what gets into the models and where. The one deciding which calls the system drives outright, in the legible zones, versus which ones it simply helps a human make better, in the relational ones. The market is conceding this in real time. The frontier labs themselves are standing up multi-billion-dollar deployment and services arms, because the model alone does not deploy. Every prior enterprise software cycle spent several dollars on integration for every dollar of licence, and AI looks like it needs more of that work, not less. The model is becoming the easy part. The context, and the relationships around it, are the hard part, and the hard part is where the margin goes.

“Then the agents become the participants.”

The obvious reply to all of this is the one Anthropic would give. Fine, the human is in the room today. But why assume they stay there? Put agents in the room. Let them negotiate, hold the relationships, make the procurement calls, talk to the regulator. The premise dissolves.

Take it seriously, because it is the strongest objection and it is where my argument lives or dies. And my own story is the test case. Could an agent eventually be that bank founder, the one quietly closer to the regulator, accumulating the standing that bent the telco decision? In principle, yes. So what actually has to be true for it to happen, and is that on the same curve as the code charts?

I don’t think it is, and the reason is precise. Participation is not a capability. It is a standing. To hold the kind of credit that moves an outcome, you have to be a party whose goodwill others want and whose word they can rely on next round. That runs on a few things capability alone doesn’t buy. A persistent identity that can’t be quietly reset or forked when it is convenient. Something real at stake that can actually be lost. And, hardest of all, other people deciding over time that you are a party worth trusting in the first place. The bank founder had all three, built over decades. An agent has raw capability and, by default, none of the rest. You can make it more capable overnight. You cannot make it trusted and accountable overnight, because those are conferred by other people slowly, and frequently by people with an active interest in keeping the relationship layer exactly as illegible as it is. The bank founder, I’d wager, had no interest in seeing his influence written down. Few people who hold that kind of standing do.

So the bottleneck doesn’t disappear in an agentic world. It moves, and it moves back toward a human. Someone still has to be the accountable party the regulator’s word is given to, the reputation on the line, the principal the agent acts for. Agents will absolutely become participants in the high-volume, low-trust corners first, the fast ones, the way algorithmic traders are already participants whose actions move the very prices they are predicting. That is real reflexivity, and it is already here. But the slow, high-stakes, relationship-gated decisions, the deal, the mandate, the rule, are gated on standing, and standing is the slowest thing in the system to transfer. This is the gap between recursive intelligence and recursive participation. The first is racing ahead. The second is a social and institutional fact, and those move on human time. I would not bet on them converging on the timeline the curves imply, and the burden of proof sits with whoever says they will.

A note on Africa, where the gap is widest.

I should admit where I’m standing when I say all this. I spend most of my time around African corporates, in both heavily regulated and lightly regulated sectors, which between them make up some of the least legible corners of the economy there is. It is possible I overweight the relationship layer because it is the water I swim in, and that a frontier engineer staring at a clean training loop honestly sees a different world. Fair. But I think the vantage shows something real rather than distorting it.

The usual story about Africa and AI is that the continent is data-poor, so AI won’t land. I think that misreads what is scarce. In most of the markets I work in, the relationship-driven share of any consequential decision is simply larger. Larger, not different in kind. The same mechanism runs in a San Francisco boardroom and a Lagos one; it is just turned up here. Which way a regulator is leaning. Which supplier extends terms to which distributor, and why. Whether a counterparty trusts you enough to clear the deal. Who the real champion is inside a client, and who is quietly working against you. The gap between what a system can see and what actually decides the outcome is widest exactly where the stakes are highest. I’ve argued before that this gap is not only a problem, it is where the opportunity sits, for whoever is honest enough to design around it rather than pretend it isn’t there. The places where the future is most obviously made by people rather than read off the data are the places where the human in the room, and whoever governs the model’s access to that room, hold their value longest.

None of this is a ceiling on AI. It is a claim about order and pace. The legible falls first. The made-by-people future falls last, if it falls at all in the timeframes people are throwing around. And in between is a long and valuable stretch where the best outcomes come from a human who is better prepared, better informed, and still the one holding the relationship that moves the decision.

That is the part I am most confident about. The rest, as always, I would rather argue about than be right about on my own. Push back. Preferably with your own view, and hope we all keep very curious and also participating in shaping where this future takes us.