AI Is Not Software
Most enterprise security programs still treat AI assistants as software: deterministic products with fixed control surfaces and predictable failure modes. When an AI system is connected to tools, memory, and workflows, policy compliance becomes an inference problem under uncertainty. AI systems do not fail loudly. They fail quietly, and still act.
Scenario excerpt
A regional operations team deploys an enterprise assistant with access to ticketing, vendor records, and a payment connector. A support lead pastes an ambiguous "duplicate invoice" thread. The assistant retrieves similar tickets, invents a complete credit workflow, and executes the credit. No malware ran. The model did not break. What failed was treating plausible language as authority to change money.
Exploit-path names
- Ambiguity to wrong action: Ambiguous goals ("fix this," "make the customer happy") invite the model to invent a complete irreversible workflow. Completeness without authority is how credits, deletes, and access grants happen.
- Long-context constraint loss: Safety instructions and policy snippets lose salience as context grows — long threads, retrieved docs, tool dumps. Early "do not…" instructions never reappear in the final tool arguments.
- Automation bias: Operators rubber-stamp fluent proposals under ticket SLAs. Approvals cite the assistant's wording instead of an independent policy check.
Root causes
- Probabilistic instruction weighting: Prompts and policies compete with user content, retrieved memory, and tool results. Security that depends on "the model will obey the system prompt" is theater.
- Tool-calling expands blast radius: Once the model can call APIs, edit tickets, or move money, a wrong inference is no longer a wrong paragraph — it is a system change.
- Missing separation of inference and authority: Many stacks let the same component that proposes an action also authorize it. Inference may suggest; authority must decide.
Impact categories
- Financial: Fraud, overpayment, unauthorized credits or transfers.
- Operational: Data corruption, unintended workflow completion, irreversible infrastructure changes.
- Compliance: Regulated data mishandled without a recoverable evidence trail.
Countermeasure preview
- Gate irreversible actions with deterministic policy checks — if the gate cannot evaluate, fail closed.
- Require traceable evidence for consequential execution (who/what, resource, policy, decision, time).
- Separate inference from authority to act — models draft; they do not mint authority.
- Bound blast radius by default: least privilege, allowlists, limits, and human approval for high-consequence classes.
- Design for quiet failure: instrument side effects and anomalous tool use, not only toxic text.
What to watch for
- Plans that invent irreversible steps not present in the user request.
- Tool arguments that omit early policy constraints after long context.
- Approvals that cite assistant wording instead of an independent policy check.
Practitioner checklist
- List every production tool and mark irreversible / reversible / external.
- For each irreversible tool, name the non-model policy check that runs immediately before invocation.
- Produce example evidence records for deny and allow.
- Identify memory re-entry paths without provenance and define fail-closed behavior.