By now the diagnosis is consensus. Language models are probabilistic. In a consumer chatbot that is a feature; in a payments flow, an insurance claim, or anything a regulator will eventually read, it is a liability. The standard prescription follows immediately: wrap the probabilistic model in deterministic controls.
The prescription is right and it is not enough, because the hard part was never agreeing that you need determinism. The hard part is seeing where probabilism climbs back across the line you thought you drew. The most common failure in regulated agent systems is not an absent control. It is a control that is itself probabilistic, wearing a deterministic costume.
Three controls that are secretly probabilistic
All three appear in an honest take on Microsoft's Agent Governance Toolkit by Venkat Peri, who works on agentic AI infrastructure for wealth management at Advisor360°. The review flags the weakness in the first two. On the third, the review's wording lands on the wrong side of the line this piece draws, and Peri's own design lands on the right one.
Using an LLM to classify intent, and treating the classification as a safety gate. The toolkit counters goal hijacking with a "semantic intent classifier", and, as Peri points out, that classifier "is itself an LLM call," so it "is also susceptible to adversarial inputs." Put a model in front of the agent and call its verdict the safety gate, and you have built the gate out of a component that is probabilistic and adversarially corruptible. You are fighting probabilism with probabilism. Under exactly the inputs that matter, the crafted and the injected, the guard inherits the weakness it was meant to cover.
Granting capability from a behavioral trust score. The same toolkit scores each agent's behavior from 0 to 1000, and the tier the score falls in decides what the agent may do. Peri calls it an interesting primitive and names its failure: uncalibrated, it "can produce false confidence in well-behaved agents and trigger overly aggressive restrictions on legitimate ones," so it needs domain-specific tuning before anyone relies on it for authorization. Tuning narrows the error, but the number stays what it was: a heuristic computed from behavior the agent itself produces. Once an authorization decision reads it (this agent is trusted enough to do X), that number is in the authority path, and it is wrong precisely when you can least afford it: an agent being manipulated can keep behaving well right up to the action that matters.
Deciding human escalation from the agent's confidence. The model was 0.7 confident, so we didn't surface it to a person. Model confidence is an inferred signal about the model's own state, and here it carries a compliance responsibility: the decision of when a human must be in the loop. The signal that decides whether oversight happens is produced by the thing oversight exists to check. The review says the toolkit "does not have a HITL workflow primitive." Its documentation today describes one: Microsoft's announcement lists "approval workflows with quorum logic", and in the toolkit's tutorial a policy rule declares which actions require approval, and an unanswered request is denied when it times out. Peri's deeper point holds either way: something still has to decide when a person is asked. The review calls that layer the Decision Gateway and describes it as "a probabilistic judgment that takes confidence scores, consequence severity, and context as inputs." Peri's own design is more careful than that sentence. In the Decision Gateway pattern, which actions need a person is declared in the workflow's policy, with email.send and holdings.update under approval: required, before anything runs. In the confidence plane, a run whose accumulated confidence is unrecoverable is terminated and surfaced to a human. Confidence adds a reviewer there, and nothing in either piece lets it remove one. A model's confidence signal should never be able to take a reviewer away.
All three look like controls, and all three fail the same way: under adversarial pressure or distribution shift, the probabilistic signal that was supposed to protect you is the one that bends.
The line isn't AI versus not-AI
So restate the dichotomy correctly, because "deterministic or probabilistic?" is the wrong question. Both belong in the system. The probabilistic component is extraordinary at a specific set of jobs (understanding language, proposing actions, communicating with a person), and you do not want to give those jobs to a state machine.
The real question is which responsibilities each part is allowed to hold, and the durable line runs between responsibilities: communication and proposal may be probabilistic; authority, consequence, and compliance must be deterministic and declared.
The model can propose any action; whether the action executes is decided by a deterministic gate. The model can phrase a friendly redirect; a regulatory refusal is a fixed string set by policy, not composed on the fly. The model can interpret what a user wants; who that user is and what they may touch comes from a signed token and a database. Even "risk" sits on the deterministic side: this action needs a person is a rule a human declares on the action, and a risk score the model infers at runtime is just more probabilism asking to be trusted.
Put the probabilistic component where it does its best work, and keep it away from everything that must not vary.
A test you can run
There is a one-line test for whether a control is real: can the thing you don't trust influence it?
If the model's output can change a control's decision, what you have is a suggestion the model grades itself on. A real control reads only inputs the model cannot fabricate: a cryptographically signed identity, the committed state of a database, a value a human declared in advance. Run your "controls" through that test and the disguised ones fall out of the set immediately. The intent-classifier gate, the trust score and an escalation that confidence can waive all fail it; authorization against a signed token, a policy-fixed refusal string and a confirmation a human declared on the action all pass. A model signal can still make the system more careful (ask again, or refuse in words it did not write) as long as it only ever adds caution and never grants a permission.
A separate evaluator is not the same thing as a model grading its own output, and Peri's own designs use one. In What to ask your agentic AI vendor, compliance and fabrication checks are hard gates ("a single detection of a fabricated claim or compliance violation blocks the output"), and a composite score routes everything else to delivery, curator review, or refusal. The test sorts that design without strain. Blocking, refusing and sending to a curator only ever add caution, so they pass. The path it flags is the other one: a score that releases an output with nobody else looking. For communication, that is a reasonable trade. For anything that carries authority, consequence or a compliance decision, it is a probabilistic signal granting the permission.
Most disguised-probabilism failures survive because nobody asked the control the obvious question.
Why this bites harder in regulated contexts
A probabilistic control that is right 98% of the time is fine for a consumer product. In a regulated context, the 2% is the audit, and the model was usually right is not a defense you want to put in front of a regulator.
Regulation asks for more than accuracy: it asks for controls you can demonstrate. FINRA Rule 3110 requires a supervisory system "reasonably designed to achieve compliance", and in the vendor piece Peri reads it this way: "FINRA Rule 3110 (Supervision) requires demonstrable controls; a capability envelope is one of those, a system prompt is not." Statistical tools have a place in a control environment, and sampling a queue for review is one of them. The control that decides what an agent may do, though, has to answer when someone asks what it will do on the next input. A declared rule answers with the rule and the record of applying it. A probabilistic control can only answer with a rate, however good the rate is, and it's usually right is a statistic where the examiner asked for a demonstration.
The discipline
The instinct to add guardrails to a probabilistic model is right in spirit and dangerous in execution, because guardrails assembled from probabilistic parts inherit the property you were trying to escape. The discipline is narrower and harder than "add controls": decide what each part is allowed to be responsible for, give the probabilistic component the work it is genuinely the best tool for, and let nothing probabilistic hold a responsibility that has to come out the same every time, however well it benchmarks. Drawing that line takes one decision; keeping it from moving as new controls get added is most of the work.
Nicolás Moreno builds Zarel: governed AI operations, where the AI proposes and the contract decides.