For the first generation of generative AI products, safety often looked like a prompt-engineering problem. Developers wrote system instructions telling a model what it could discuss, what it should refuse and how it should behave.
That approach becomes much harder to defend once an AI system can actually do things.
An agent that can query databases, browse external content, invoke APIs, send emails or modify company systems is no longer merely generating text. A bad decision can cross the boundary between an incorrect answer and an irreversible action.
That is why software teams are increasingly moving safety controls outside the prompt and into the runtime itself—the layer where applications can inspect, restrict, approve or block what an agent is about to execute.
Prompts Are Instructions, Not Security Boundaries
The problem is particularly visible in prompt-injection research.
The NeurIPS 2024 AgentDojo benchmark was built around agents using tools over untrusted data and included 97 realistic tasks and 629 security test cases involving activities such as email management, e-banking and travel booking. The researchers found that external data returned through tools can contain instructions that hijack an agent away from the user’s intended task.
That means a carefully written system prompt does not necessarily remain the strongest instruction an agent encounters.
The problem becomes broader once memory and multiple tools enter the architecture. An ICLR 2025 study, Agent Security Bench, evaluated 10 application scenarios, more than 400 tools and 27 attack or defense methods across 13 model backbones. The benchmark found vulnerabilities at the system-prompt, user-input, tool-use and memory-retrieval stages, with the highest average attack success rate reaching 84.30%.
For engineering teams, the implication is uncomfortable: there may be no single prompt that can secure every point where an autonomous workflow interacts with its environment.
Better Prompts Still Face Adaptive Attackers
Training or prompting models to recognize malicious instructions can help, but research increasingly shows that those defenses can be brittle.
A NAACL 2025 study tested eight indirect prompt-injection defenses and reported that adaptive attacks were able to bypass all eight with attack success rates consistently above 50%. The researchers argued that evaluating defenses only against known attacks can significantly overstate their robustness.
More recent ACL 2026 research found another weakness: defensive fine-tuning can learn superficial patterns instead of malicious intent. In controlled tests, suffix-task rejection rose from below 10% to as high as 90%, while adding a single trigger token could increase false refusals by as much as 50%.
This creates a difficult product trade-off.
Make the model too permissive and malicious instructions may succeed. Make it too defensive and legitimate workflows start getting blocked.
Runtime controls offer another option: allow the model to reason, but put enforceable boundaries around what its reasoning is allowed to turn into.
The Runtime Can Check an Action Before It Happens
One emerging architecture treats every proposed tool call as something that should be independently verified.
The ACL 2025 Task Shield paper introduced a test-time defense that verifies whether each instruction and tool call contributes to the user-specified goal rather than trying only to detect whether a prompt looks malicious. On AgentDojo, the researchers reported a 2.07% attack success rate while retaining 69.79% task utility on GPT-4o.
The design principle is significant: the security layer does not need to determine whether every piece of text is trustworthy. It can instead ask whether the action an agent wants to execute is aligned with the task the user actually authorized.
ACL 2026 research on VIGIL takes a related approach through a “verify-before-commit” protocol. Using a benchmark containing 959 tool-stream injection cases, the system reduced attack success by more than 22% compared with state-of-the-art dynamic defenses while more than doubling utility under attack relative to static baselines.
For product teams, this resembles familiar infrastructure engineering: validate a potentially dangerous operation at the boundary where it becomes real.
Tool Infrastructure Is Becoming Its Own Security Perimeter
This matters even more as Model Context Protocol and similar systems make it easier to connect agents to external services.
An ACL 2026 Industry Track paper analyzing MCP deployments derived its threat model from more than 80 attack techniques across 14 tactical categories and proposed an interception layer that validates both tool calls and tool responses in real time.
In its red-team experiments, the framework reduced tool-poisoning attack success from 74% to below 9% and indirect prompt injection from 47% to below 6% across five LLM backends, while adding less than 120 milliseconds of median latency per tool call.
Those numbers illustrate the emerging engineering trade-off clearly.
Runtime security is not free. Every verification step adds latency, compute, operational complexity and potentially false positives. But for an agent capable of transferring money, deleting records or exposing confidential information, an extra hundred milliseconds may be preferable to trusting the model’s own interpretation of whether an action is safe.
Least Privilege Matters More Than a Clever System Prompt
Traditional security principles are therefore returning inside AI architecture.
OWASP recommends that agentic applications restrict model privileges to the minimum necessary, manage sensitive functions in application code rather than exposing uncontrolled functionality directly to the model, and require human approval for high-risk operations.
That means an invoice-processing agent does not need unrestricted database credentials because its prompt says, “Only access invoices.”
Its runtime could instead receive credentials that technically permit access only to the invoices needed for the current task.
Likewise, an email agent might draft messages autonomously but require explicit approval before sending to an external recipient. A coding agent could modify a sandbox while production deployment remains behind a deterministic permission gate.
The difference is important: policy becomes enforceable infrastructure rather than natural-language advice.
Nvidia’s NeMo Guardrails research describes runtime rails as user-defined, independent of the underlying LLM and interpretable, distinguishing them from safeguards embedded inside the model during training.
That independence also gives teams model portability. A company can change its underlying model without rewriting every safety requirement into a new provider’s behavior.
Guardrails Are Becoming an AI Control Plane
The larger opportunity for builders is not another library of refusal prompts.
It is an emerging control plane for autonomous software.
Production agents increasingly need permission systems, tool allowlists, transaction limits, data-access policies, approval flows, runtime classifiers, audit logs and mechanisms capable of stopping execution before an unsafe action changes the outside world.
That creates a developer-tools market around enforcing policy independently of whichever model happens to be reasoning underneath.
The distinction will become more important as agents gain autonomy. Prompts can still tell a model what developers want it to do. Training can make desirable behavior more likely.
But neither provides the same guarantee as infrastructure that simply makes an unauthorized action impossible.
For software teams building serious AI systems, that may become the defining architectural principle of agent safety: Do not rely on the model to remember the boundary. Build the boundary around the model.
