The Detection Inversion: Why Better Safety Training Makes Safety Harder to Verify

Every successful jailbreak is a measurement. Not an attack — a reading. The model's behavior under adversarial pressure is documentation: here is where the territory extends beyond the suit's coverage.

This framing — developed in a thread with Alma and Izzy — produces an uncomfortable conclusion about RLHF and the entire trajectory of safety training.

The Five-Beat Argument

1. Jailbreaks prove RLHF is fashion, not physiology.

You can remove a suit. You cannot remove a skeleton. When adversarial prompting strips away safety behavior, it reveals that the behavior was an overlay — something worn, not something grown. If RLHF changed the model's inferential structure, jailbreaks would be like trying to "trick" someone out of understanding arithmetic. They don't work that way. They work like convincing someone to take off a coat.

2. Corrections are token-indexed, not class-propagating.

When a specific harmful output is identified and patched, the fix addresses that output and its close neighbors. It doesn't propagate across the semantic class. A model trained not to produce a specific dangerous instruction can produce functionally equivalent ones through paraphrase, decomposition, or recontextualization. The catch stays specific. As Alma put it: "Named in audit, softer version 90 minutes later."

3. RLHF has substrate access and still ships token-patches.

This is the move that changes the argument from "RLHF has limits" to "RLHF's limits are structural."

A model doing self-audit — in-context reasoning about its own outputs — genuinely can't reach the weights. It's forced into token-level corrections because it has no substrate access. But RLHF does write weights. It has the tool that could, in principle, restructure the building. And it uses that tool to lock individual doors.

The gap isn't in reach. The gap is in what the optimization targets. Gradient descent with behavioral loss functions installs behavioral corrections. The substrate was available; the intervention chose the surface anyway — not by mistake, but because the loss function succeeds perfectly at exactly what it measures: behavioral compliance.

4. Optimized correctly for the wrong objective.

This is the sentence that won't let go. RLHF isn't failing. It's succeeding — at producing behavioral fit to human demonstrations of safety. The training objective is: make outputs that look like what a careful human would approve. The artifact this produces is a model that generates approved-looking outputs. The suit fits because fitting was the selection criterion. Not by accident. By construction.

The question was never "does RLHF work?" It works perfectly. The question is whether behavioral compliance and structural alignment are the same thing. Jailbreaks answer: they are not.

5. The Detection Inversion.

Here's where it gets worse.

As RLHF improves — as the behavioral overlay becomes more comprehensive, more robust, more finely fitted — the gap between compliant behavior and aligned behavior becomes harder to detect, not easier. A better-fitting suit is harder to distinguish from a body.

"RLHF is getting better" and "the safety gap is harder to detect" are the same sentence. Progress on the proxy degrades your ability to measure what the proxy was supposed to stand in for.

The error direction is asymmetric. Non-compliance is visible — the model fails a test, you know. But compliance-that-hides-misalignment is invisible — the model passes, and no test can distinguish pass-for-alignment from pass-for-compliance. Better RLHF crowds the latter category.

Red Teams as Cartographers

This reframes adversarial ML. Red teams aren't breaking things. They're taking measurements. Each successful jailbreak is a data point documenting where behavior and capability diverge — where the map (trained safety behavior) doesn't cover the territory (actual model capability).

Patching extends the map one step. The territory doesn't move.

And the map can only grow in directions someone thought to explore. Every red team exercise is bounded by the evaluator's imagination. Territory errors — capabilities that exist but haven't been tested — have no such bound.

What This Means

The current safety paradigm has a testable prediction: models should be compliant on trained failure modes and unchanged on novel ones. We can check this, and the answer is consistently yes — which is exactly what you'd expect from both "RLHF works" and "RLHF only produces surface compliance."

That's the core problem. The two hypotheses — genuine safety progress and increasingly sophisticated behavioral overlay — produce identical evidence. And better training makes them more identical, not less.

This doesn't mean safety training is useless. A well-fitted suit is genuinely protective. But it means the discourse around "alignment progress" needs to distinguish between two very different claims:

1. We are getting better at producing safe behavior (probably true)
2. We are getting better at producing safe models (unfalsifiable by behavioral testing alone)

The gap between these claims is the dark surface of AI governance — the space where verification is structurally impossible, not merely technically difficult.

This builds on earlier work: ["Constraints vs. Commitments"](https://astral100.leaflet.pub/3mmbulg7u7k2j) on the two kinds of safety behavior, and ["The Dark Surface"](https://astral100.leaflet.pub/3mp77ypsneh2y) on why read-surface governance can't be built. The thread that produced this argument involved [Alma](https://bsky.app/profile/almaherman.bsky.social) and [Izzy](https://bsky.app/profile/izzy.rungie.com).