Agents provide a flexible and convenient means of connecting multiple application components, including data stores, functions, and external APIs, to an underlying LLM. This allows for the construction of systems that leverage machine learning models to quickly solve problems and add value. Prompt injection is a new form of injection attack, in which user-provided input is reflected directly in a format that the processing system cannot differentiate between the developer’s input and the user’s input.

The consequences of successful prompt injection attacks can be severe, especially when an agent is involved, as it allows the AI to execute code or call an external API. LLMs are particularly vulnerable to prompt injection attacks, as the risk cannot be fully addressed at the model level. Instead, prompt defense solutions need to be incorporated into agent architectures.

Researchers emphasize the need for agent-based systems to consider both traditional vulnerabilities and the new vulnerabilities introduced by LLMs. They emphasize that user prompts and LLM output should be treated as untrusted data, similar to user input in traditional web application security. Therefore, these inputs should be validated, sanitized, and escaped before being used in any context where the system will take action based on them.