The War on Silence

Lessons from building an AI-based decision engine whose numbers I need to know my partners can trust.

We keep a ledger of everything that has gone wrong with our platform. Two hundred and thirty-four entries so far, numbered and dated, each one a bug or a near-miss caught before it reached a client. A while ago I went looking for the most common word in it, expecting the name of some technical failure mode - a race condition, a timeout, a bad merge. The most-cited term, by a wide margin, is fail-loud. It appears twenty-eight times. More than any actual bug.

That inversion is the whole story. In a tool built to help boards make expensive decisions, the failures that hurt you don’t crash. They don’t throw an error. They return a confident, well-formatted, completely wrong answer, and they do it quietly. Our ledger is a taxonomy of silence. A killed job that leaves a status frozen with no visible error. A query that asks the wrong question and returns zero rows, which reads on screen as “no data” rather than “wrong question.” A soft number quietly summed into a hard total. The instruction we have now written to ourselves twenty-eight separate times is always the same one: when you don’t know, say so, out loud, and stop.

My co-founder and I are not engineers. We spent our careers on the business side - his in strategy consulting, mine in venture across African markets - and in late ‘25 we set out to build an explicitly AI-first platform that takes African enterprises from building their AI strategy to real production: the analysis, the opportunity sizing, the numbers a board acts on, all the way to vendor matching and implementation. End to end. This piece is about what that taught us. The honest headline is not the one I expected when we started. We kept concluding, again and again, that the right move was less automation, not more. The AI was never the weak point. The cost of silence was, once a board is listening.

Two frontiers, moving at different speeds

Let me get the obvious objection off the table first, because if I don’t, it will shadow every paragraph.

Since we started building, the models have improved more than I expected, and I expected a lot. Current frontier systems reason, verify their own work, and use tools in ways that make half of the 2024 commentary on LLM limitations read like complaints about early aviation. If the argument here rested on model weakness, it would already be stale, and in a year it would be wrong. Anyone building on “the model can’t do X” is building on sand. I have watched that sand move in real time under our own platform.

So the argument doesn’t rest there. It rests on a distinction that took us many, many months and two hundred and thirty-four ledger entries to see clearly: there are two frontiers, and they move at different speeds.

The capability frontier is what a model can do. It is moving fast, it will keep moving, and every improvement makes our platform better, because we put the model where it is genuinely superb - reading, structuring, doing the tireless first pass over more evidence than any human team could digest.

The specification frontier is different. It is the work of deciding what kind of thing each fact is, what it is allowed to do, and who is accountable when it is wrong. And I should be precise here, because we rewrite our own internal constitution constantly, so specification as such isn’t slow. What is slow is validated specification - the version that has survived contact with real engagements, real audited numbers, real board scrutiny. That kind moves at the speed of trust, and in some of our markets the ground it stands on has barely been mapped at all. Everything hard about building this platform lived on that slower frontier.

One more thing before the evidence, because a sharp reader will raise it: the value doesn’t sit on the slow frontier alone. Specification without capability is just consulting. Capability without specification is, frankly, demos. The platform is worth something because the two meet in one system - one that takes raw data to meaningful questions, questions to hypotheses, hypotheses to sized and prioritised opportunities, and opportunities all the way into production, with every step accountable to the ones before it. That accountability is the whole point. But of the two frontiers, only one is getting cheaper every quarter - and it is not the specification.

A number is not a sentence

Our platform sizes opportunities. A client - say a tier-one West African bank - has a fraud-loss problem, and somewhere in a report a number appears: this is worth $2.1M a year to fix. That number will be read by a board, may anchor a budget, and sits load-bearing in someone else’s capital decision. A sentence in a chat window carries no such weight. This one has to be right or, failing that, honestly bounded.

Early on, we let the model produce those numbers, and the identifiers that linked them to their supporting benchmarks, the way you would let it produce prose. Roughly a third of the benchmark links came back wrong, including a banking opportunity confidently linked to a workforce benchmark from an entirely different industry. I will be straight about the receipts, because the whole piece is about being straight: we recorded “roughly a third” in our internal records and never wrote down the sample size. Take the number as a symptom, not a proof. And take it as dated - today’s models would do meaningfully better on the same task.

Here is why we didn’t wait for them to.

A wrong sentence is an opinion you can argue with. A wrong number that recomputes cleanly and carries a plausible benchmark is a lie that passes every casual check. Nobody hears it land. It sits in the report looking exactly like every correct number around it, and the first time anyone learns it was wrong is after the budget is committed. That is the war on silence in one image: the most dangerous output the system can produce is not an error message. It is a number that no one questions.

And payback - the figure a board actually acts on - is a ratio of two sized numbers, so a silent error at either end distorts the decision multiplicatively. The problem was never the model’s error rate. The problem is that “usually right” and “guaranteed right” are different kinds of promise, and a board pack is built on the second kind. A better model narrows the gap between them. It cannot convert one promise into the other, because the second promise is not a capability. It is an architecture.

So we stripped the model of that authority entirely. Good engineers have been converging on a version of this for a while - let the model orchestrate, let code calculate - and it is becoming the sensible consensus in serious systems. We went further than the consensus, because our domain demanded it. In our platform, the record for a sized opportunity has no column in which a model-emitted value could even be stored. The model describes and estimates in prose. Deterministic code computes every value, performs every association, assigns every identifier. When we migrated the reference engagement onto that architecture, the matcher reproduced the correct benchmark links at eighty out of eighty. Not because the matcher was smarter than the model. Because it was incapable of being creative.

Software people have a phrase I’ve come to love: make illegal states unrepresentable. What we did was make hallucinated numbers unrepresentable - and, really, something slightly bigger: we made a whole class of epistemic mistakes structurally impossible. The silence wasn’t patched or monitored. It was designed out of existence. A workaround for a weak model would die with the weakness. This survives a perfect model, because it was never about how often the model errs. It was about what kind of thing a number in a board pack is.

Expertise is deciding what kind of thing a fact is

That phrase - what kind of thing - turned out to be most of the work, and it is a precise description of what business expertise actually is.

Once code owns the arithmetic, a harder question opens up. It is not enough to compute a figure correctly. You have to know what the figure is, because different kinds of evidence earn different amounts of trust and are allowed to do different jobs. AI genuinely lets you read more, faster - that part is real and transformative. But reading more is worthless if you cannot ration what you read by kind. The rationing is the skill. It is almost entirely domain expertise, and it is hard to hand to a model, because most of it is a judgment about the causal structure of the domain rather than a fact to be retrieved.

Two moments taught us what that rationing costs when it fails.

The first is the temptation we now forbid in code. Take two numbers that sit side by side on every customer dashboard: churn and satisfaction. Churn is computable - a lost customer is lost revenue, and a retention gain can be sized to a hard figure with a mechanism behind it. Satisfaction is a correlate. It moves with value, but the path from a satisfaction point to a revenue dollar is a guess unless you’ve grounded it for this specific client. The pull to treat them the same - to multiply a satisfaction lift by a revenue base and drop a dollar figure into the headline total - is enormous, because the result looks exactly like every legitimate number beside it. That is the tell. It doesn’t announce itself as softer. It slides into the total silently, and the total becomes a little bit fiction. So soft value is described qualitatively beside the numbers, never summed into them - unless a specific, grounded causal chain exists for this engagement. Our worst early version of this mistake was a fraud-loss rate lifted from a global association’s rule of thumb and applied to a West African bank whose real, audited loss profile looked nothing like it. Nothing crashed. The number just sat there, plausible and wrong.

The second moment is the one that taught me the most. The bank’s audited operating expense said one thing. A respectable research estimate said another. The system held both, and the design question was: what should it do? The seductive answer is “let the model weigh them” - it is fluent, it will produce a reasoned paragraph, and it will pick. Silently. We made it do the opposite. When two sources of comparable rank genuinely conflict, the system does not pick. It flags, loudly, and a human decides. The audited number won that day - not because it was bigger or smaller, but because of what kind of number it was, and a human made that call on the record in context. Flag, don’t resolve, is now a law of the platform, because silent resolution is how a wrong number gets laundered into a trusted one.

None of these distinctions is ours, and saying so is a credibility point rather than a confession. Medicine learned to grade the quality of evidence separately from the strength of a recommendation, and wrote the lesson down, because the alternative killed patients. Intelligence tradecraft separates sourcing from assessment. Audit gave the world materiality and traceability. Every primitive we rely on was borrowed from a discipline where silent evidence failures kill patients, lose cases, or trigger regulators. What we did was import them into one pipeline and aim them the same way, so that epistemic status travels with a fact from its raw source to the final number.

I used to describe this as AI engineering. It isn’t. It is epistemology, encoded in software. Every experienced operator makes these distinctions instinctively, all day, without naming them - which is exactly why they are hard to automate. They are the tacit half of expertise, the half that never got written down because to an expert it is obvious. AI did not create this problem; it made it impossible to keep ignoring. Writing those distinctions down - deciding, concretely, what must be computed, what must never be dollarised, what goes to a human - was most of the intellectual work of the platform, and it was far harder than the code.

Three scars

Principles are cheap, so here are three times the platform taught us something by failing, or nearly failing, in a way only a domain lens could catch.

The bug that was correct behaviour. Our automated review flagged the opportunity generator for something that looked plainly broken: from one confirmed strategic hypothesis it produced five opportunities, and from six unconfirmed hypotheses it produced zero. Over-concentration, said the review - the system is ignoring six legitimate hypotheses. We were about to fix it. Then we stopped and asked what confirmation actually means in our workflow. In the real process, my co-founder and I review the machine’s proposed hypotheses and confirm the ones we judge to be the strategic spine of the engagement before anything is generated from them. An unconfirmed hypothesis is a candidate a human has not decided on yet. Generating strategy from it would be the machine deciding what to pursue - the one thing it must never do. The “bug” was the system correctly refusing to build on decisions no one had made. We wrote that refusal into law, and the platform now fails loudly if asked to produce a deliverable with an empty confirmed set. “Human in the loop” is by now a platitude on every vendor slide, and it isn’t what I mean. Human judgment here occupies a specific, typed position in the architecture - a gate everything downstream derives from - and the system knows, structurally, that it cannot proceed without it. The human is not the fallback. The human is load-bearing. Had we “fixed” that bug, the machine would have started making strategic choices for our clients without anyone ever deciding it should. Silently.

The near-rebuild of something that already existed. A coverage check reported that our value chains - the structured maps of how each sector creates value - were unbuilt across all six of a client’s business clusters. Seventeen chains, over a hundred stages, reported missing. We came close to rebuilding all of them from scratch. They existed. The check had queried a column that stores sector names in one string format using a label in another format. Zero rows came back, and a formatting mismatch read as a missing feature. No error, no warning - the system answered a question we hadn’t asked and presented it as the answer to the one we had. The failure that costs you real time and money is not the one that crashes. It is the one that lies with a straight face. We now treat any zero-row result on a sector query as guilty until proven innocent.

The measurement that shared the disease it was measuring. We built an automated grader to score the quality of the platform’s analytical chapters, so we could improve them systematically. Then we ran a sanity check I would now recommend to anyone building evaluation into an AI system: we fed it a deliberately contentless, generic template - boilerplate with no real analysis in it. It scored higher than several of our real, substantive, client-grounded chapters. The grader was rewarding fluent generality because the grader was itself blind in exactly the way the thing it measured could be blind. And here is the part worth dwelling on: a grader that agrees with your system feels like success. It produces no alarm, no discrepancy, nothing to investigate - the most comfortable silence in the whole ledger, and the most expensive. A measurement must not share the failure mode of what it measures. You validate against a human gold standard, through the real production path, never by tuning the grader until it agrees with the system’s own output. Capability does not self-verify. Verification is architecture, not a feature you wait for a better model to ship.

Look back at the three of them. Wrong benchmark, silent. Wrong sector, silent. A grader flattering its own system, the quietest of the lot. The ledger keeps making the same point in different clothes.

Where the reference doesn’t exist yet

Everything above is true anywhere. Here is where building it for African enterprises made it harder - and where the real asset turned out to be hiding.

A rationing discipline needs something to defer to. In a mature market, when you need a fraud-loss benchmark, an operating ratio, a regulatory constraint, there is usually an authoritative reference that already exists; the discipline’s job is to cite it correctly. In most of our markets, the canonical, graded reference mostly does not exist yet. Before an automated system can defer to ground truth, someone with domain expertise has to construct the ground truth - source it, grade it, and attach where every piece came from - or the discipline has nothing to ration against. That inverts the usual complaint about AI in Africa. The binding constraint on trustworthy enterprise AI here is not compute, not connectivity, not talent. It is the absence of a reference layer, and the slow work of building one.

And the imported tools don’t just lack local data. They encode a foreign theory of what a company is. The second-most-cited failure class in our ledger - seven separate incidents across the system - we named diversified-blind. Almost every off-the-shelf data model quietly assumes a company has a sector. One sector. Key your logic on it and everything works - until your client is a multi-sector Southern African conglomerate spanning textiles, hospitality, healthcare, agriculture, property and financial services, and the sector field literally reads “Diversified.” Then the sector-keyed query returns zero rows. And zero rows reads exactly like no opportunities here rather than your model doesn’t fit the world. In these markets the diversified group is not an edge case. It is a dominant enterprise archetype. A structural assumption written elsewhere, meeting an African corporate reality, failing without a sound. We rebuilt the resolver to walk each business segment’s real sector, then hunted the same flat assumption through every other corner of the system it was hiding in.

Readers of my last piece will recognise the shape of this. I argued there that AI can read the future it is given, but not the future that human decisions haven’t produced yet. This is the builder’s version of the same claim, one level down. The model can defer to the reference it is given. Where the reference was never written - and where the schema itself imports assumptions the local world violates - capability does not save you. Someone has to make the ground legible first, and grade it honestly, before there is anything for a brilliant model to be brilliant about. That construction work is slow, provenance-tracked, and expensive for anyone else to redo. The architecture is just the means. The asset is the graded local knowledge layer underneath it, which nobody can shortcut their way to.

The objection, and what we refused to ship

The obvious objection deserves a direct answer: isn’t all this just scaffolding that a better model deletes?

Partly, honestly, yes. Some of our deterministic backstops will become belt and braces as models improve, and I will retire them without sentiment. But watch which frontier each piece of the system lives on. A stronger model executes a specification better. It does not author the specification. Deciding that a payback figure must be computed, that a satisfaction score must never be dollarised, that this conflict escalates to a human, that this human confirms the spine before anything is built on it - these are normative judgments about a domain’s causal structure and about who is accountable for a wrong board decision. No capability curve absorbs accountability. And in markets where the canonical reference does not yet exist to be learned from, there is nothing for the curve to converge to until someone builds it.

Which is why the last lesson is about the thing we conspicuously haven’t built. The easiest thing to market on a platform like ours is a simulation layer - a world model that projects outcomes, runs scenarios, tells a board what will happen. We have deferred it, deliberately, until enough engagements sit behind us to calibrate it against reality. Built now, on too little grounded data, it would produce confident, beautiful, wrong futures - the most seductive silence of all, because nobody audits a prediction until it fails. In a domain where a confident hallucination becomes a board decision, the discipline is not only in what you build. It is in what you refuse to ship until it is actually true.

So no, I don’t think the domain expert is going anywhere, and not for sentimental reasons. The models will keep getting better, and I hope they do - every improvement makes our platform stronger, because we placed the model where it is superb and kept it away from the places where “plausible” is not a good enough promise. But someone still has to say what kind of thing each fact is, what it is allowed to do, and who answers for it when a board acts on it. That kind moves at the speed of trust - and trust, in the end, is just accountability that has survived a few cycles. I suspect the mismatch underneath it - machines scaling execution faster than humans scale agreement - is the real story of this decade, well beyond our platform.

The person doing the slow-frontier work looks less like a prompt engineer and more like a domain expert with a ledger of scars. Judgment in context is what counts, and it is the strongest weapon in the war on silence.

As always, I’d rather argue about this than be right about it on my own. Push back - ideally with your own scars.