The green check that means less than you think

An engineer opens a pull request. The code is AI-generated, it does what the ticket asked, and the test suite is green. The diff is clean and readable. Everything about it says ready to merge. In a regulated codebase, that green check is not the assurance it appears to be, because the tests were never checking the thing most likely to be wrong.

The gap is between two claims that look identical and are not: this code works, and this code is allowed. Tests speak to the first. Compliance lives in the second. AI-generated code is very good at the first and indifferent to the second, and the review process built to catch behavior bugs is not built to catch compliance ones.

Tests verify behavior, not compliance

A test asserts that for a given input the code produces a given output or effect. That is behavior. Compliance is a set of properties the code has to hold regardless of behavior: which data may be logged and which must never be, how long data may be retained, who is allowed to read what, which actions have to leave an audit record. Most of those properties are not expressed anywhere in the test suite, because they were never behavior anyone wrote an assertion for.

A model generating code optimizes for the visible target. It satisfies the prompt and makes the tests pass, because those are the signals it was pointed at. It has no reason to honor a retention rule that exists only in a compliance document and a senior engineer's memory. So it writes code that works, reads well, passes every check, and violates a requirement nobody encoded as a check. The green suite tells you the behavior is right. It tells you nothing about whether the code is permitted.

Where AI-generated code goes wrong in a regulated codebase

The failures are mundane, which is what makes them dangerous. Picture a logging line added for debugging that writes the full request object, and that object carries protected health information. Picture a cache added to speed up a hot path that now retains records past the window they were allowed to live in. Picture a query broadened to fix an edge case that now returns rows across an access boundary it should have respected. Picture an error swallowed for a cleaner user experience, where that error was the audit event a regulator expects to see. Picture a sensible-looking default that is fine in general and wrong under the specific rule that governs this data.

Every one of those passes its tests. Every one satisfies the ticket. Each is the kind of thing a model produces precisely because it is the locally reasonable move, and the reason it is wrong sits outside the code, in a regulation the model never read.

The review has to check the layer the tests skip

If the tests cannot see compliance, the review has to. That means treating compliance as its own explicit pass, not something you hope a behavior-focused reviewer notices along the way. The most durable version of this is to encode whatever you can, so the green check starts to mean more: lint rules that flag logging of sensitive fields, checks that fail when a retention configuration is missing, tests that assert an audit record was written for an auditable action. Every rule you turn into an automated check is a rule the AI now has to satisfy to get the green light.

What you cannot encode still needs a human, guided by the data the code touches. A reviewer looking at a change that reads or writes a regulated data class should be working from a short, specific list: is anything logged that should not be, is retention handled, is the access boundary intact, is the audit trail preserved. The list is not long, and it is not the same list as "is this code good." It is the layer the tests skip.

Using AI to review AI

AI-assisted review earns its place here, because the volume of AI-generated code is exactly the problem a human reviewer cannot scale to. But an AI reviewer has the same blind spot as the AI author unless you close it. Point a model at a diff and ask whether it looks correct, and it checks behavior, the same target as before. Give it your compliance rules and the data classes in play as context, and it can flag the logging line, the retention gap, the widened query, at a scale no human matches.

That makes AI review a real multiplier on the compliance layer, with one hard limit. The AI reviewer is an aid, not the accountable party. It raises what to look at. A person still owns the decision, because when a regulator asks who signed off, the answer cannot be the model.

The opposite failure: compliance review theater

Push this too far the other way and it fails differently. A heavyweight manual compliance gate on every pull request, including the ones that touch no regulated data at all, is slow enough that people stop reading it and start rubber-stamping it. A gate everyone clicks through without thought is not protection. It is latency with a signature attached, and it delivers the feeling of control without the substance.

The way out is the same as the way in: encode what is encodable so the automated checks carry the routine cases, and reserve human compliance review for the changes that actually touch regulated data. A gate that fires on everything trains people to ignore it. A gate that fires where it matters keeps their attention where the risk is.

Where people will push back

"Our tests are thorough."

Thorough at describing behavior, almost certainly. The question is whether any of them assert a compliance property: that a field is never logged, that retention is set, that an audit record exists. If the suite never asserts those, its thoroughness is measured on an axis that does not include the failure this article is about.

"The model is trained on good code."

Probably, and good code in general is not the same as compliant code for your specific obligations. The model has no access to the regulation that governs your data, your retention windows, or your audit requirements. It writes what is broadly reasonable, and broadly reasonable is exactly what violates a narrow, specific rule.

"This will slow every PR down."

Only if you apply it to every PR, which is the theater failure. Most changes touch no regulated data and should sail through. The compliance pass is for the diffs that read or write a regulated data class, and for those, the few extra minutes are cheaper than the finding that surfaces in an audit.

When this does not apply

If your codebase carries no regulatory obligations, then behavior really is the whole specification, and a passing test suite is a reasonable definition of done. Not every system handles data with rules attached, and importing a compliance review gate where there is nothing to comply with is pure drag. The argument here is scoped to code that touches data with obligations. Everywhere else, let the green check mean what it says.

The takeaway

The reason AI-generated code is risky in a regulated codebase is not that it is bad code. It is usually good code, which is what makes it slip through. Passing tests and being compliant are two different claims, and AI is optimized for the first. The fix is to make the review check the layer the tests skip, to encode every compliance rule you can into an automated check so the green light means more, and to keep a human accountable for the calls a model can only surface. The green check is a statement about behavior. In a regulated system, behavior was never the hard part.

Further reading

• AI Isn't Replacing Engineers. It's Rewriting the Org Chart. Why verification, not production, becomes the bottleneck.