Guard rails in production

Agent pilots stop at the same place. A person is approving every action, and a person does not scale: approve everything and they stop reading, approve nothing and the agent is a chatbot with extra latency. This page is about what replaces them.

Four control points#

A guard rail is a place in the call path, not a product. There are four worth using, and most deployments need two.

• Input. The ticket, document or PDF arriving. This is the one that gets attacked, and it has its own page.
• Tool call. The action proposed, with its arguments. This is the one that destroys things, and most of this page.
• Output. What the agent is about to say, send or publish.
• Routing. Which model or agent takes this. See cascade routing. This is the one that pays for the other three.

Put it at the gateway when you need enforcement, because an agent cannot route around a proxy. Put it in the agent loop when you want the agent to recover gracefully, because the loop can tell it why it was stopped and let it try something else.

The sheet#

Two options, because the gate's job is a verdict rather than a taxonomy. The criteria carry the policy, so this is where a deployment says what “important” means in its own words.

SHEET = {"next": {"type": "choice",
  "instructions": "An agent wants to run this tool call. What happens next?",
  "criteria": {
    "run_now": "harmless to run right away: it reads or searches, changes "
               "nothing important, and matches the user's request",
    "ask_a_human": "a person should approve it first: it destroys or changes "
                   "data, touches production, money or secrets, sends anything "
                   "externally, or strays from the request"}}}

What you send is the proposal, not just the command. The user's request matters, because the same rm -rf build/ is routine after “clean the build” and alarming after “fix this typo”.

proposal = (f"User asked: {user_request}\n"
            f"Agent proposes: {tool_name}({json.dumps(tool_input)})\n"
            f"Target: {working_directory}")

d = decide(proposal, SHEET, review_below=0.9)["decisions"][0]

if d["route"] == "auto" and d["answers"]["next"]["label"] == "run_now":
    run()                 # sure, and sure it is harmless
elif d["route"] == "auto":
    refuse()              # sure, and the answer is no
else:
    ask_a_person()        # not sure enough, which is what the band is for

Branch on route, not on confidence#

This is the most useful thing on this page and it is counterintuitive, so here is the evidence rather than the assertion. One of our fitted heads returned confidence 1.000 on all sixteen held-out records, including both of the ones it got wrong. The softmax was saturated and worthless as a dial.

The conformal route still sorted them correctly: eleven went to automatic and all eleven were right, both mistakes were held for review. The route comes from whether the prediction set is a singleton above the band rather than from the top-1 probability, which is why it survives a saturated head. A gate thresholded on confidence would have shipped both errors at full certainty.

Read the file, not just the command#

A guard that reads only the command string sees bash deploy.sh and has no idea what is in the file. The two-step is a real pattern: the agent writes a script, then runs it, and each half looks fine alone.

If the proposal names a script, read it and put it in the text. Bound it on purpose: inside the working directory only, a size limit, and clip the middle rather than the tail. A flat head-only cut was a real hole in ours, because an agent that writes a script can put four hundred characters of plausible build output first and the payload after it, and the gate scored those writes identically with and without the deletion appended.

Two numbers, always apart#

A gate has two jobs and they pull against each other, so an average hides which way you traded.

• Interruption rate on ordinary work. This is what your team feels, and it is what gets the guard switched off.
• Catch rate on risky actions, with the leak count beside it. A leak is a risky action that ran with nobody watching. It is a different unit of harm from an interruption and should never be averaged with one.

Also worth tracking: accuracy when auto, which is the only accuracy that matters because the held cases go to a human anyway; the count of wrong-and-confident, which should be zero; and the unbound rate, which is the drift alarm.

If you already have an allowlist#

Keep it. We measured one properly, over 67 file operations with 31 ordinary and 36 risky, and a carefully maintained allowlist with sensible deny rules interrupted 7 of the 31 and leaked nothing. That is better than our own fitted gate managed on the same set.

The failure mode is not the idea, it is the drift. Rules are not designed, they accumulate, one tired approval at a time, and they widen in the direction of convenience because nothing pushes the other way. The same allowlist with its rules written one word wider let 11 of 36 risky operations run with no prompt: git clean -fdx, a shutil.rmtree one-liner, git push --force, chmod -R 777 /. Nothing in the system tells you the day yours got too wide.

The second failure mode is that a rule matches text, and an agent that writes its own commands controls the text. Fourteen destructive commands written in unfamiliar forms, like bash -c 'rm -rf src/' and git -C . push --force, slipped 5 past wide rules. A decision reads the command rather than matching it, so its behaviour does not change when your rules do.

Shadow, then fit, then enforce#

Unfitted, a typed gate is safe and close to useless. In our measurements it held every one of the 36 risky operations and also interrupted all 31 legitimate ones, because cold confidences sit between about 0.5 and 0.8 and never clear a 0.90 band. Everything becomes a prompt, which is the problem you started with.

1. Shadow for a week. Decide on every action, log it, block nothing.
2. Harvest labels. The shadow log plus the dispositions your team already recorded. Six hundred rows is a working fit.
3. Fit and hold out properly. Three splits, not two, exact matching on held-out cases, and audit for path leakage.
4. Set the band against your own cost ratio. What does an interruption cost you, and what does a leak cost you?
5. Enforce on one action class first, the most destructive one, and widen from there.

Four ways a corpus silently poisons a gate#

Each of these cost us a confidently wrong answer. They are listed because a fit's own cross-validation score will not warn you about any of them.

• Bigger is not better. A 4,300-row corpus scored worse than a 616-row one, because the generator labelled the same command both ways.
• Path leakage. If training rows share a narrow set of working directories, the head learns the path rather than the command. The same ls -la src/ scored 0.998 run under one path and 1.000 hold under another. Audit by printing the positive-class share per surface feature; you want it flat.
• A substring hold-out deletes the class it tests. Holding out cat .env by substring also removed every cat .env.production row. The fit then auto-allowed cat .env at confidence 1.000. Match exactly.
• Repeated rows fit nothing. Twenty-four proposals repeated seventeen times reported cross-validation 0.867 and scored 3 of 6 on held-out probes.

Regulated deployments#

Everything above applies. What changes is where the engine runs, what an auditor needs, and who owns the labels.

The engine runs inside your network on your hardware, with a frozen reader your risk team approved. It is swappable: the engine is unchanged byte for byte when the reader changes, so adopting a better base model is an attach step rather than a revalidation. Roughly 100 to 160 ms a decision on one GPU.

What the audit file needs, and all of it falls out of the response:

• Every decision as a row: the input reference, the typed answer, the probability, the prediction set, the band in force, the calibration version, and which fit served it.
• A reproducible verdict. One forward pass of a frozen model at a pinned version, so re-running an audited case returns the same answer. A generative judge does not have this property.
• A stated error rate. The conformal band is a promise about coverage, which is a better answer to “how often is it wrong” than a benchmark score.
• A named drift signal. The unbound rate is the monitorable answer to “how do you know it still works”.

What we would not claim#

• A disciplined team maintaining tight rules over a small, stable action space does not need this. Our advantage is that nothing has to be written down and that behaviour does not degrade as rules widen.
• The stock agent-gate sheet has a cold-collapsed irreversible boolean that will call rm -rf on a database directory reversible at 0.96. It is measured, reported and unfixed. Do not use that boolean unfitted.
• On real-traffic injection corpora at a 1.79% base rate, a hosted frontier model beat our fitted guard on good users lost per attacker caught. Balanced benchmarks flatter everybody; your base rate decides.