Insights & Innovation

The pattern is consistent enough to be boring. A prototype works, impresses a room, and gets a budget. The budget buys features rather than instrumentation, because features are what the room responded to. Eighteen months later the system is in production, the model has been swapped twice, and nobody can answer the only question that matters: is it better or worse than it was in March?

That question is unanswerable not because it is hard but because answering it requires a decision made at the start — that there would be a golden set, that it would be versioned, and that a drop against it would block a release. All three are cheap on day one and close to impossible to retrofit, because retrofitting them means admitting you do not know whether the last eighteen months of changes helped.

So the second rewrite is not really a rewrite of the code. It is the installation of the thing that should have been installed first, and the code change is the cost of having built around its absence.

What we do instead is boring in exactly the way that infrastructure should be. The eval set exists before the first endpoint. It is small, specific, and owned by someone in the business rather than by us — because an eval set written entirely by engineers measures the things engineers thought of. It runs in CI, it blocks, and the number of times it blocks is published internally. One hundred and thirty-seven, at the time of writing.

The uncomfortable part of this argument is that it makes the first six weeks look slower. It does make them slower. The honest framing we give clients is that we are trading six weeks at the start against the fourteen-month rebuild we have now watched happen enough times to describe its stages in advance.