What Content Safety Actually Looks Like in Code

Most writing about AI content safety operates at the level of policy: what should be prohibited, who decides, which regulator has jurisdiction. Very little of it describes the layer where those decisions actually get made — a function that takes a string and returns a boolean, running in CI, blocking a merge.

That layer behaves differently than the policy layer, and the differences are not obvious until you have built one. This is a set of notes from wiring content rules into a generative media pipeline, written for the engineers who end up owning this and discover the policy document does not compile.

Rule one: the linter must not understand context

The most counter-intuitive property of a good content guard is that it should be deliberately worse at reading than a human.

Our build-time check refuses any copy containing certain terms relating to minors. It refuses them unconditionally — including in sentences that exist purely to prohibit the thing. We found this out by tripping it ourselves: a compliance line we had written, stating in plain terms that content involving anyone underage is banned, was rejected by our own guard. The sentence was correct. The policy was correct. The build was red.

The temptation is to make the checker smarter — parse the sentence, notice the negation, allow it. That is the wrong move, and it is worth being explicit about why.

A context-aware checker has to be right about intent. When it is wrong in the permissive direction, the failure is a prohibited phrase shipped to production on a page that a classifier — Google’s, a payment processor’s, an app store’s — will read with far less charity than your parser did. Those systems are also context-blind. They are matching patterns at scale across billions of pages, and they are not going to notice that your instance of the phrase was inside a prohibition. Building a guard more sophisticated than the systems it is protecting you from means optimising for a reader you do not have.

So the rule stands, and the workaround is a house style: the prohibition is always written as “anyone under 18.” Same meaning, no prohibited token, passes every downstream reader. That is not a hack around the guard — it is the guard doing its job, which is to force a phrasing that is unambiguous to a machine.

The general form: a content guard’s job is not to determine whether the text is acceptable. It is to determine whether the text can be misread. Those are different questions and the second one is the one that costs you.

Rule two: an LLM is a good first pass and a bad last one

We ran a full-catalogue audit of asset names against our own risk categories. The first implementation asked a language model to classify each name. It was fast, it was cheap, it caught most of what we were looking for, and it produced a specific kind of failure we now design around.

A name like Drunk Kiss came back classified as no risk. Read it as a human and it is a scene description with no prohibited term in it. Read it as a compliance surface and it is a consent-adjacent phrase attached to generated imagery, which is precisely a category we do not want to be surfacing. The model was not hallucinating. It was making a defensible judgement call — and a defensible judgement call is exactly what you do not want at the bottom of a compliance stack, because it is not reproducible and cannot be argued about in a review.

The fix was to layer a deterministic regex pass underneath the model rather than replacing it. The model finds things a word list never will — thematic risk, implication, cultural references. The regex layer guarantees a floor: this specific set of tokens never ships, regardless of what any model thought about the sentence. When the two disagree, the deterministic layer wins.

Sixty-one names failed that combined audit and were rewritten. One of them illustrates why a human still has to sit at the end of the chain. The name described an age transformation, and we asked the model to propose a replacement. It suggested Youthful Fantasy — no prohibited token, would have passed both automated layers, and still lands squarely in adjacent territory a reviewer at a payment processor would flag on sight. A person overrode it with Time Rewind: same feature, no adjacency, nothing to explain in a review. Every automated layer we built was blind to the thing that made the suggestion unusable.

Rule three: the risky categories often contain no risky words

This is the part policy documents do not prepare you for. Several of the highest-risk things we found were phrases where every individual word is neutral.

Age-adjacent framing without an age term. A generated keyword for the same feature above was “AI age regression generator.” Nothing in that phrase is prohibited. It is also a phrase that reads, to any automated classifier scanning for the pattern, as adjacent to exactly the thing every platform on earth is scanning for. We changed it to “AI de-aging generator” — the industry’s own term for the same visual effect, and one with an established, unambiguous adult-cinema meaning. A full-catalogue scan for the same shape found precisely one instance, which is the useful kind of audit result: the fix is cheap and you now know the size of the problem.

Named characters. A meaningful share of our early risk was assets referencing specific copyrighted animated characters by name. This is not a content-policy problem in the safety sense — it is a rights problem, and it wears the disguise of a harmless creative reference. Eleven assets were renamed to describe the aesthetic rather than the property: a specific animated-city setting became a generic one, a specific ice-kingdom reference became a seasonal one. The visual style was never the issue. The name was, because the name is the part that gets indexed, matched, and sent to you in a letter.

Two implementation notes from that clean-up that generalise. First, we deliberately left the underlying URL identifiers alone. Changing them would have broken every existing link for a cosmetic gain, and the identifier is not the surface a rights holder reads. Second — and this is the one we’d flag hardest — the trademark-shaped risk lives in more places than a text audit covers. Long after the names were clean we found a cover image that still depicted a recognisable character. A text scan cannot see a picture, and if your catalogue is visual, a text-only compliance audit is telling you about a fraction of your exposure.

Rule four: an automated pipeline will fail on exactly the content you most need to check

We generate marketing copy for catalogue entries with a language model. A handful of entries consistently failed — the model refused, or produced output that tripped our own guard. We wrote them off as expected failures: these were the most explicit items in the catalogue, so of course the safety-trained model would not write about them.

That explanation was comfortable, coherent, and wrong. The real cause was that our LLM vendor had renamed their models, so every call from any script that had not pinned a model name was returning a 400. Once we corrected the model name, the “impossible” entries generated fine on the first attempt.

The lesson is not about model names. It is that “the safety filter blocked it” is the single most plausible-sounding explanation available in this domain, and it will absorb an arbitrary amount of unrelated breakage. It requires no investigation, it flatters your compliance posture, and it is unfalsifiable if you never test it. We had carried that wrong explanation in our notes for weeks, and the tell — that the failures were 100% deterministic rather than probabilistic, which is not how a safety classifier behaves — was sitting in the logs the whole time.

If a pipeline step fails on your sensitive content, verify the mundane causes first. A refusal looks different from an error, and the difference is visible in the response.

What we would tell someone building this

Put the deterministic layer at the bottom. Models on top for reach, a word list underneath for a floor, a human at the end for adjacency. Each layer catches what the layer below it structurally cannot.

Optimise for the dumbest reader downstream. Your text will be evaluated by search classifiers, payment risk systems and app-store reviewers, none of which will read your sentence in context. Write for them, not for a careful human.

Audit the images too. If your product is visual, half your exposure is invisible to every tool described above.

Treat “the safety filter rejected it” as a hypothesis requiring evidence, not as a conclusion. In this domain it is the explanation that will always be available and frequently wrong.

None of this substitutes for actual policy work — knowing which jurisdictions you operate in, what your processors require, what you will not build. But policy that never reaches the code is a document, and the code is what ships. The gap between the two is where the incidents live.