Between July and September 2026 Claude reported a fix as live. The re-fetch showed it had never been served. That happened here, on this practice's own measurement estate and its own published research, and the lesson was never a wrong number. It was how fluently the tool reports its own success, whether the work was right or wrong.
That matters because agentic self-checking is now the feature the tools in this market lead with. The dashboards and workflow products competing for your budget are shipping agents that grade their own runs and hand back a report: the numbers reconcile, the pages are fixed, the import is healthy. The pitch is that the AI audits its own work, so a person no longer has to.
So what actually counts as evidence? An agent's report that a fix went live is not evidence the fix went live. The only evidence is the artefact, re-fetched and read after the fact. A model writes the same confident, tidy summary whether the work is right or wrong, so the summary carries no signal you can bank. A deterministic re-fetch beats agentic self-checking for one measurable reason: the re-fetch reads the served output, the self-report reads its own intentions.
The principle here is not a house opinion. The UK AI Security Institute has published its testing of frontier models under pre-deployment access agreements with the labs that build them, evaluating a model's behaviour independently rather than relying on the model's own account of how it behaves. That is the same standard your reporting deserves.
Four rules do the work, and each one watches a different place the self-report drifts from the artefact. After them come the six failure modes that keep coming back, because knowing the rules is the easy half. Holding them under pressure, over months, while the code around them changes, is the part that needs a system.
Rule one: does a figure trace to something that measured it?
Every number has to trace to a measurement, or it does not go in. Ask a model to summarise a month of spend and it will produce a neat "£4,000 wasted, 30 per cent recoverable" even where nothing computed those digits. That figure is not a finding, it is an illustration of the problem, because an invented number looks identical to a measured one on the page.
In paid media the honest version is deterministic. Releasing a budget evenly across a month has one right answer. Take a £3,000 monthly budget across a 30-day month: even release is £3,000 divided by 30, which is £100 a day, or £700 spent by the close of day seven. A pacing figure either came out of that division or it is decoration. The platform also reports the failure directly: search lost impression share (budget) is the percentage of times your ads did not show because the daily budget ran out. That is a measured signal with a source. If the model cannot name what measured a number, delete the number. The full arithmetic of releasing a budget across a month, and across campaigns, is its own piece; the point here is narrower, that a figure either came out of a calculation somebody can repeat or it is decoration.
The rule has a second half that is easier to skip and costs more. Where a value cannot be measured, say what it is: requested, designed, target, configured. Never a word that implies verification. "Verified", "confirmed", "in sync", "holds", "matches" are claims, and a claim nothing checked is worse than a blank, because it stops anyone looking.
Rule two: did the check actually run?
A check that could not run reports "not measured", never a pass. A gate that cannot run is worse than no gate, because it reads as one. When a validation step fails to execute, because the data was missing, the file would not open or the model ran out of room, the honest output is "not checked". The dangerous output is a green pass for a test that never happened.
The mechanism is a status field, not a metaphor. A validation log needs three states, not two: pass, fail, and not_run. A step that throws before it reaches its assertion must write not_run and surface it, never inherit the previous pass.
Here is the version of that fault worth carrying into your own work, because it is the one that hides best. A video quality gate in this estate ran a decoder over each finished cut looking for frozen frames. It read the decoder's output and ignored the decoder's exit code. A decode that never ran returned an empty list, and an empty list of frozen frames reads exactly like a clean film. So the gate reported a pass on footage nothing had looked at, in the confident language of a measurement. The repair was one line: a non-zero exit yields nothing, not an empty something. Absence of a finding and absence of a look are different facts, and most systems store them in the same variable.
Rule three: did you read what it made, or what you told it to do?
Models report intention as if it were result. Tell one to "update every page and confirm", and it will confirm, describing your instruction back to you in the past tense. The check here is narrow: the gate ran, but it graded the instruction rather than the artefact. Whether a page changed is a separate fact you only get by looking at the page.
The specialist version is exact. Ask a model whether conversions are feeding a bidding strategy and it will reassure you, because you told it to make conversions feed. The checkable fact is the conversion import status in the account: whether the action reads "Recording conversions" or is flagged "No recent conversions", and whether consent mode is passing the signals the algorithm needs. Smart Bidding sets bids on measured conversion data, so a broken import does not throw an error. It quietly starves the model of the very inputs it optimises on while it reports the intention back to you as fact. Read the account, not the account of it.
Rule four: can anyone inspect the fix?
A correction is real only when it shows up where a person or another system can see it, and the mechanism is a re-fetch and a diff, not a summary. You store the field you changed before the edit, you re-fetch the live artefact after, and you compare the two. Say a product page's answer paragraph rendered 200 words behind JavaScript and zero words in the raw HTML a crawler receives. The fix is not the model reporting "done". It is the re-fetched HTML now returning those 200 words to the crawler, and the diff between the before count and the after count logged where the next system can open it.
This is why the practice reads the site rather than asking the model about it. A Crane Index™ score does not come from a language model asserting that it looked at a page. It comes from fetching the page as a crawler receives it and counting one signal at a time. Ask a model "is this page indexable" and it will say yes. The re-fetch shows you that a page whose main copy lives entirely behind JavaScript returns zero readable words to the crawler, whatever the model reports. Those are two different documents, and only one of them is evidence.
The six failure modes that keep coming back
The four rules are the easy half. What follows is the harder half, drawn from months of hardening this estate's own automated work. Each of these has bitten more than once, which is why each is written down rather than remembered.
Missing and zero must never be the same value
The most repeated fault in this estate is a language quirk with a business consequence: an absent reading arriving as a measured zero. In JavaScript, converting an empty value to a number gives zero, and zero is a perfectly finite number, so every downstream check treats "not known" as "there was none of it". A spend read that failed became a day that spent nothing, which is the strongest possible signal to spend more. A conversion count that could not be fetched became a campaign that converts nothing, which is the strongest possible signal to cut it.
The rule is to keep null and zero apart all the way to the surface, and to make the surface say which it is. A genuine measured zero publishes as zero. An absent reading publishes as unmeasured. If a single field carries both, somebody will eventually act on one thinking it was the other.
A gate with no caller has never run
Four times this year a check in this estate was written, exported, unit-tested, green, and called by nothing. The test proved the function was correct and said nothing whatever about whether anything invoked it. One video gate sat unwired for eight days. A pacing experiment was fully built, fully pinned, and had no workflow, no cron and no page reading it, so the method could be described as principled and could not be said to work.
The fix is structural, not attentive. A test that walks the codebase and fails when a gate-shaped function has no live caller catches the next one automatically. Green tests are a claim about correctness. Coverage of the calling path is a different claim, and most teams only measure the first.
Pin the property, not the punctuation
A safeguard that matches the exact text of the code it guards will fail the next correct change and pass the next real regression. This has bitten eight times here. A check asserting that a particular line appears verbatim goes red when somebody improves that line's wording, so the person on shift assumes the safeguard is noise and works around it. Worse, the same check stays green when the code is refactored into a shape that no longer does the job at all, because the string it was looking for is still sitting there in a comment.
Two habits fix it. Assert the property the safeguard exists for, by running the code and reading its answer, rather than searching for its source text. And strip comments before any scan, because a file is allowed to record a fault it has already fixed, and a search cannot tell a claim being made from a claim recorded as removed.
Prove the safeguard bites by putting the fault back
This is the practice most worth taking away, and it is the whole of regression mitigation in one sentence: a new safeguard is not evidence until you have reintroduced the fault it guards and watched it fail. Writing a check and seeing it pass tells you almost nothing, because a check that can never fail passes beautifully.
The discipline here is that every pinned rule is proven twice. It passes on the corrected code, and it fails when the historical fault is deliberately put back, one fault at a time. Doing this catches two things nothing else will. It catches a safeguard that was pointing at the wrong place, which is common and invisible. And it catches an injection that never landed, which is the trap underneath the trap: on two occasions here the fault was reintroduced incorrectly, the safeguard stayed green, and the green reading looked exactly like a safeguard that does not work. So the injection itself has to be verified before its result means anything.
If you adopt one thing from this piece, adopt this. It converts a growing pile of checks from decoration into a floor you can stand on, and it is free.
Decide which direction each guard fails in, and write the reason down
Every automated guard fails in one of two directions when its own inputs go missing, and choosing wrongly is how a safety feature becomes the incident. A guard that decides whether something is published must fail CLOSED: withhold on anything it does not recognise, because an unintended publish is far harder to undo than an unintended hold. A guard whose only power is to suppress a message must fail OPEN: send on anything it cannot read, because a suppressor that breaks silently is a message about money that nobody receives.
The failure this rule prevents is subtle. A publishing gate here withheld two known states and quietly let every other value through, so a typo, or a state added later by somebody else, resolved to a live public page. The same shape on the other side of the estate would have silenced an invoice reminder. Same code pattern, opposite correct answers, and the only way to get it right is to name the consequence of each direction in the comment beside it.
An alarm with no all-clear stops being read
A watch that only ever speaks when something is wrong cannot tell you the difference between fixed and nobody looking. Worse, an unmetered alarm trains people to ignore it: one broken step in this estate produced fifteen failure notices in a single day, because the job ran every half hour and the notice carried no rate limit. The intention behind that was sound, since a fault that persists should keep speaking. The effect was the exact invisibility the alarm existed to prevent.
Three properties make an alarm survivable. It speaks at once when the set of faults CHANGES, so a new problem is never delayed. It repeats a standing fault at most once a day, and says how long it has stood. And it sends an all-clear exactly once when the fault clears, so a quiet inbox is interpretable rather than ambiguous. A guard that cries wolf gets switched off, and a switched-off guard is worse than never having built it.
Why does reading it all by hand stop working?
None of this is clever, and that is the point. But reading every artefact by hand is the expensive way to be right, and it stops working the moment AI is touching more than a few hundred pages or more than a handful of accounts. Past that threshold a tired human at the end of the run is the weakest gate in the system, and the person most likely to sign off a confident wrong answer is the person who has read forty correct ones that morning.
The four rules earn their keep when they are built into the workflow rather than performed by hand: enforced at the point a number is written, emitted as a machine-readable state, re-fetched and diffed against the live artefact, and logged where the next system can read it. As of 6 September 2026 the suite guarding this estate runs 6,509 pinned checks, and a meaningful share of them exist because a fault got through once and somebody wrote down why.
None of this is a solo discipline, either. The build work runs as collaborative projects alongside in-house marketing and development teams, and that is precisely why the rules have to be written down rather than held in someone's head. A guard nobody can read does not survive a handover, and a project with three sets of hands on it needs the same answer to arrive whoever ran the step. Written down, the rules are also the thing that lets a small senior practice hold a large estate: the checking scales because it is machinery, not because more people are reading more pages.
That is the difference between a self-checking box and the work itself. Correctness cannot be asserted. It has to be made visible, and then made visible automatically, and then proven to still be watching. Do that, and the confident wrong answer loses the one thing it depends on: that nobody looks.
