We audited our own quality tool and found three checks that never ran

We built a tool that refuses code failing your architecture rules. Then we audited it, and found three checks that had been reporting success without ever running. One of them had never produced a single verdict in its life.

Here is what each one looked like, why nothing caught them, and the shape they all share.

The analyser that never answered

The plugin runs deptrac to verify layer boundaries: that your domain layer does not import your infrastructure. That is the headline feature. It calls the tool, reads its output, and turns findings into a verdict a hook can act on.

It called deptrac with --formatter=compact.

That formatter exists in no version of deptrac. Not 1.x, not 2.x, not 4.x. The call failed every time. The error went to stderr, stderr went to 2>/dev/null, the non-zero exit status hit a || true, and the function returned cleanly with an empty result. An empty result means no violations. No violations means pass.

Every layer violation this plugin has ever reported came from a regular expression scanning import statements. The architecture analysis users believed they were running was silently absent, for months.

The regex is not nothing. It catches the common case. But it does not know what deptrac knows, and the whole point of shelling out to a real static analyser is to get an answer a regex cannot give.

Why nothing caught it. There was a test. It asserted that the code path ran without error. It never asserted that the analyser produced output. A test that checks a function returns is not a test that checks a function works, and the difference only shows up when the failure mode is silence.

The review that returned nothing, one run in two

The plugin has a command that performs an architecture review. It forked into a subagent, and that subagent declared a cap of twenty turns.

When a turn cap lands on a tool call, the agent loop stops there and returns no text. The harness substitutes the literal string Command completed for the missing result and ends the turn. No error. No partial output. No retry.

Twenty-three of that agent's first thirty-eight recorded runs ended that way. A user asking for a code review saw one line of output and a dead prompt, roughly every other time. Nothing in the logs said anything had gone wrong, because from the harness's point of view nothing had: a command completed.

The fix was not raising the cap. An agent that keeps no budget for writing its conclusion fails at any cap. Every capped agent now carries a contract: emit the deliverable as soon as the evidence justifies it, reserve the last third of the budget for writing, name what you did not cover, and never end on a tool call.

The pipeline that inspected nothing

The CI gate scanned src/, because that is where source code lives.

A Laravel application keeps it in app/. A monorepo keeps it in packages/. A Next.js App Router project keeps it in app/ too. On any of those, the scan matched no files, counted zero violations, and exited zero. Green pipeline, zero files inspected.

The run even reported it, in a field nobody was reading: files_scanned: 0. Discovery and scanning are counted separately now, and an empty discovery fails loudly instead of passing quietly.

The shape

Three different subsystems, three different root causes, one defect: a control that reports success when it did not run.

That is worse than a control that reports a false pass, because a false pass is a bug in the logic and someone eventually notices the logic. This is a bug in the reporting, and the reporting is what you look at to decide whether to look further. It consumes attention and returns confidence.

The pattern has a signature, and once you know it you find it everywhere:

• || true at the end of a pipeline, added to stop a flaky step from failing a build
• 2>/dev/null on a call whose stderr is the only place errors appear
• a timeout shorter than the cold start of the thing being timed
• an empty result treated as a clean result
• a test that asserts a code path executes, not that it produces an answer

Each of these is defensible in isolation. Someone added each one for a reason, usually to stop something noisy from blocking work. Together they build a system that cannot tell you it is broken.

Every one of these failures took the same path: a step that could not complete, an error that went nowhere, and a caller that read the absence of a complaint as an answer.

The fork on the right is where the cost lives. An empty result and a clean result are the same value, and only one of them is true.

We did it again while writing this

This is not a story about a past self who was careless. While building the website you are reading, in the same week, the same defect appeared three times.

A test file was written for a business rule, run, and reported PASS (0) FAIL (0). Zero tests executed: the runner's include pattern did not cover the directory. A green result from a suite that ran nothing.

A refactor was validated by comparing a script's output against a reference file captured before the change. Identical, byte for byte. The script could not run at all: its input directory had been removed months earlier during a migration. The comparison was a file against itself.

A production page was checked for headings with curl and reported none, on five pages. curl does not follow redirects by default, and the server redirects to the trailing-slash URL. Five empty redirect pages were measured and read as five broken pages.

Three instruments, all reporting confidently, none of them running. The tool exists because this failure mode is not a lapse in attention. It is what attention feels like from the inside when the instrument is lying.

What to do about it

The only reliable defence is to see the check fail.

Not to review it. Not to reason about whether it would fail. To break the thing it protects, on purpose, and watch it go red. A guard that has never been observed failing is indistinguishable from a guard that cannot fail, and you cannot tell them apart by reading the code, because both look correct.

Every fix in this audit carries a test that fails when the fix is removed, and each of those tests was verified red before being kept. That is a slower way to work. It is also the only way to know the difference between a control and a decoration.

The full record, with the version each was found in, is in the CHANGELOG. We publish those entries because a tool that enforces standards has to be held to them, and because the recurring shape is more useful to you than the individual bugs are embarrassing to us.