Most conversations about AI agent safety jump straight to the model. Jailbreaks, guardrails, red-teaming. That’s the sexy part. What usually gets skipped is a duller question that I’ve come to think matters more day to day: once an agent is actually live, talking to real people, who’s watching it, what are they watching for, and what happens when they see something wrong?

I spent most of a day working through this with a fictional stand-in — I called it Molt, basically a food ordering agent — just to keep the thinking concrete instead of abstract. Here’s roughly where it went.

Safety Isn’t One Filter

The first thing I got wrong, initially, was treating “safety” as one filter you slap on top. It isn’t. It splits into a handful of genuinely different problems that don’t share a shape. Someone leaking a card number in a response is mechanical — a regex catches it, no judgment needed. A user trying to talk the agent out of its instructions is a different animal; you can pattern-match it today, but people iterate, so whatever catches it now stops working eventually. Abusive language depends on which direction it’s going — what the user says to the agent isn’t the same policy as what the agent says back. And then there’s the stuff that’s genuinely invisible in any single message and only shows up once you look across dozens of sessions from the same person. That last one can’t be caught in the moment no matter how clever your rules are. It just can’t.

So right away you’re not building one pipeline, you’re building several, running at different speeds, with wildly different amounts of trust in whatever’s making the call.

What to Stop Now vs What to Review Later

Which gets you to the real question, and it’s less technical than it looks: what do you stop immediately, and what do you let happen and review afterward? This isn’t really an engineering decision. It’s a statement about how much harm you’re willing to let occur once, in trade for not slowing down everything else.

Working through it with Molt, I stopped asking “how little can we get away with blocking” and started asking something less comfortable: which harms are severe enough that letting them through even once, while we investigate calmly afterward, is not an acceptable answer? That list turned out to be longer than I expected going in — not just the obvious mechanical stuff like a card number slipping into a response, but anything where the damage is done the instant the message reaches the user, regardless of how fast you act afterward. Everything else — the ambiguous request, the pattern that only shows up across sessions, the stuff that genuinely needs judgment rather than a rule — that’s fine to defer, because deferring it costs time, not irreversibility. But pretending the split is mostly “cheap rule now, everything else later” understates how much actually belongs in that first bucket once you take the “even once” question seriously.

It’s About People, Not Code

The part I found more interesting than I expected was how much of this is really about people, not code. A review system built around one undifferentiated “reviewer” role misses that different people need to see completely different slices of the same evidence. Someone triaging fast needs a redacted timeline and enough context to say escalate or close, nothing more. Someone actually digging into a case that matters needs the raw evidence the triage view deliberately hid. Policy and legal want trends and volumes, not individual conversations. And whoever’s running the agent — Molt’s own team, in this case — should only ever see their own cases, never anyone else’s. None of that is a UI decision. It’s governance wearing an engineering costume, and getting it wrong either cripples your ability to investigate anything or turns into surveillance nobody agreed to.

Turning Policy Into Enforcement

Written policy doesn’t mean anything by itself. “We don’t tolerate X” is just words until something enforces it, and the review system is that enforcement — it turns evidence into a decision, the decision into an action, and logs why, all in one motion. The part most designs skip is making the next occurrence easier to catch than the last one. A confirmed case should be able to propose a new rule, that rule should get tested against real traffic so you can see what it would’ve caught and what it would’ve wrongly caught, and only then should a human sign off before it goes live. Skip that loop and you’ve built something that just files history instead of getting smarter from it.

Nothing Here Is Free

And here’s the thing nobody likes saying plainly: none of this is free. Every check costs something, somewhere — latency on the live path, engineering time keeping the async side running, or a real user getting blocked by a false positive. There’s no version where safety doesn’t trade against something else. So the actual design question was never “how do we make this safe.” It was which costs we’re willing to pay, and where. Building it so the agent keeps calling the model directly, with its own credentials, and only reports in and out through a small endpoint on the side, was the answer to that specific tradeoff — the safety layer can never become the reason the product goes down or feels slow, because it’s never actually sitting in the way unless something explicitly chooses to wait for it, and now that “even once” question decides which things do choose to wait.

The Actual Job

That’s the job, really. Not “is the agent safe” as some abstract yes or no, but knowing exactly where you’re spending speed, delivery time, or a bit of user experience to buy some safety, who gets to decide that trade, and how you’d actually know if it’s working.