Why irreversible actions should hold for a human

Every automated action carries an expected cost equal to the probability of being wrong multiplied by the damage when wrong. Model improvements attack the first term. Permission design attacks the second. Only one of those is under your control at deployment time, and it is not the first one.

That asymmetry is the whole argument. A system that is right 99 percent of the time and can delete a production database has a worse expected cost than a system right 95 percent of the time that cannot. Accuracy claims are therefore the wrong axis on which to evaluate safety, which is inconvenient because accuracy claims are what the category markets on.

It follows that the meaningful safety property is the boundary rather than the accuracy, and boundaries are checkable in a way accuracy is not. You can look at a permission configuration. You cannot look at a 99 percent.

It can quarantine a file or stop a process, but irreversible actions hold for a human unless you have explicitly told it otherwise.

Reversibility is not a binary, but it is close enough to one to be useful. The practical test is whether an action can be undone by the same system, within minutes, without anyone being told.

Killing a process passes that test. Quarantining a file passes it in most estates. Disabling an account passes it technically and fails it socially, because the person locked out has already been affected. Isolating a network segment fails it outright, and so does deleting anything.

The failure mode worth naming is the second category, where the technical undo exists but the consequence does not reverse. A system whose policy is built on technical reversibility alone will act confidently in exactly the cases where the damage is social rather than technical, which is where most of the real incidents caused by automation live.

Most products in this category can be configured to require approval, which means the presence of an approval option proves nothing. Three properties separate a designed boundary from a checkbox.

The first is failure direction. When the model is uncertain, when a tool call errors, or when the approval channel is unreachable, does the system act or wait? A brake that only engages on the happy path is not a brake.

The second is whether the boundary is expressed in permissions or in confidence. A confidence threshold is a property of the model's self-report, and self-reported confidence is exactly the thing OWASP's misinformation entry warns about. A permission boundary does not depend on the model's opinion of itself.

The third is whether holds are recorded. If the audit trail shows actions taken but not decisions deferred, you cannot demonstrate afterwards that the boundary held, only that nothing bad happened to be visible.

Holding costs time, and in a live intrusion time has a price. A system that pauses for approval on network isolation will sometimes let an attacker complete an action they would not have completed under full automation. Pretending otherwise would be the same overclaiming this page is arguing against.

The reason the trade still favours holding is the distribution of outcomes rather than the average. Automated response errors are correlated with unusual conditions, which is precisely when the estate is already under stress, and the tail of that distribution contains outages that cost more than the intrusion would have.

There is a design answer to the time cost that does not require removing the brake: make approval fast rather than making it unnecessary. A pre-authorised responder, a defined escalation path, and a one-tap approval in a channel people already watch reduce the delay to something close to detection latency. That is engineering work, and it is a more honest place to spend it than on making the case for skipping the human.

Abstract arguments about autonomy are easy to nod along to and hard to act on, so it is worth naming the shape of the incident the boundary exists to stop.

It is rarely a dramatic misjudgement. It is usually an ordinary detection, correct in isolation, arriving during an unusual condition the system has not seen: a migration, a maintenance window, a failover, a load test that looks like exfiltration. The reasoning is sound and the context is wrong, and context is the thing least represented in the telemetry.

OWASP describes the mechanism directly: excessive agency enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs. Note that the trigger is unexpected input rather than a defective model. A perfectly functioning system reasoning correctly about an unusual situation is exactly the case a permission boundary is for, and exactly the case a confidence threshold does not catch, because the system is not uncertain.

Excessive Agency enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM.

It is worth reading the mitigation guidance rather than only the risk name, because the guidance is more specific than the general call for caution the category tends to paraphrase it as.

Two of the recommendations are about surface area: limit the extensions an agent may call to the minimum necessary, and limit the functions implemented inside those extensions to the minimum necessary. Both attack the damage term rather than the error term, which is the same asymmetry this page opened with.

The third is explicit about people. OWASP recommends human-in-the-loop control requiring a human to approve high-impact actions before they are taken. That is not a hedge added by a cautious buyer; it is the guidance from the risk register the whole industry cites.

Utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken.

Ask for the action list, not the autonomy claim. A vendor should be able to show you which actions execute unattended, which hold, who may change that, and where the change is recorded.

Ask what happens on failure paths specifically. Uncertainty, tool errors, and an unreachable approval channel are three separate cases, and a product that has thought about the problem will have three separate answers.

Then ask to see a held decision in the audit trail. It is the single fastest way to tell whether the boundary is a design or a description, and it takes about a minute.