Working Is Not a Quality Metric

The system looked healthy.

The happy path worked. The UI rendered. The flow completed. The database writes succeeded. The tests passed. The demo was easy to believe.

That was the symptom, not the diagnosis.

The mistake was treating “working” as a quality metric when it was only evidence that expected behavior occurred under expected conditions.

Why the metric seemed reasonable

Before AI-assisted development, implementation friction was often the bottleneck. Writing the code took long enough that some architectural questions surfaced during the work. The delay did not guarantee good judgment, but it created opportunities to notice assumptions before they multiplied.

AI changes the timing.

A feature that once took days can appear in hours. Boilerplate is handled, common patterns are reproduced, and a test suite for the obvious scenarios arrives with the implementation. The output is not necessarily bad. It is simply produced faster than a human can independently understand every decision inside it.

That makes “it works” unusually persuasive. The visible evidence arrives before the evaluation needed to interpret it.

The failure mode

The failure was not that the tests were false. The tests passed for the behavior they described.

The failure was using those results to answer a larger question.

A unit test can prove that a function returns the expected result for a valid input. An integration test can prove that a well-formed request crosses the expected layers. An end-to-end test can prove that a normal user completes the flow.

None of those tests automatically answers:

• Can a caller forge tenant scope?
• Is authorization missing on a less-traveled path?
• Does a business action leave half-completed state after failure?
• Can a development escape hatch survive in a production route?
• Are two layers making contradictory assumptions about authority?

When generated tests mirror the implementation, they often confirm what the code does rather than challenge whether the system should do it that way. Coverage increases while understanding stays flat.

That is how a codebase becomes convincingly green and architecturally weak.

The diagnosis: judgment moved downstream

The bottleneck moved from implementation to judgment, but the workflow did not move with it.

The agent could write faster than the builder could evaluate. Review remained a final inspection instead of an active process of reconstructing intent, checking boundaries, and asking what would happen when the caller was wrong.

The result was not simply more code. It was more output whose meaning had not yet been independently established.

This is where review becomes more than a search for defects in a diff. It is the place where the system's claimed behavior is compared with its actual responsibilities.

A reviewer should be able to ask why the boundary lives where it does, which assumptions are trusted, what evidence is missing, and where the system is expected to stop. If the only available evidence is “the agent implemented it and the tests are green,” the review loop is already too narrow.

What better evidence looks like

I trust the system more when quality is visible in the structure rather than inferred from the happy path.

Tenant scope comes from authentication, not request parameters. Authorization is enforced by the layers that own policy, not repeated as UI convention. Dangerous operations are constrained by design. Business-critical actions are coherent under partial failure. Tests include forged identity, missing permission, wrong tenant, malformed state, and recovery after interruption.

These checks are less comfortable because they test the system we do not want to be true. They also provide better evidence than another successful screenshot.

The goal is not adversarial testing for its own sake. It is to test the boundary that makes the feature consequential.

The earlier signal

The same failure can usually be detected before the incident if we watch the right signal.

The warning is not necessarily a red test. It is a growing gap between implementation speed and independent evaluation: large diffs accepted without a visible intent record, tests that repeat the implementation's assumptions, review comments limited to naming and formatting, or a team unable to state the system's trust boundary in one sentence.

That gap is judgment debt. It accumulates while the output still looks productive.

Working code is valuable. It tells us that a behavior can occur. It is not permission to infer that the boundaries are sound, the decision is understood, or the system deserves more authority.

The quality question is not whether the code works.

It is what the code has made true, under which conditions, and what evidence would reveal that we got the boundary wrong before a user has to discover it.