AI is entering business processes faster than some organizations have built hard boundaries for it. Language models don’t just answer questions. They increasingly read documents, browse the web, summarize threads in instant messengers and launch actions through integrations (e.g. MCP – Model Context Protocol). This creates a new, semantic attack surface: prompt injection.
Prompt injection is a situation in which malicious instructions enter the LLM context – sometimes directly from the user (direct prompt injection), and in other cases hidden in external data that the system processes (indirect prompt injection). The effect can be simple, but expensive: the model does something it shouldn’t – it reveals information, manipulates the response or initiates actions that the architecture and permissions granted to it allow (e.g. via API/integration) – even though the user did not want it.
OWASP classifies prompt injection as the highest-ranked risk for LLM and GenAI applications. He also emphasizes something that is inconvenient, but crucial for CISO: due to the nature of generative AI There is no method today that provides a 100% guarantee of eliminating prompt injection – but the risk and effects can be significantly reduced.
In this article, we answer questions that really influence security decisions:
- What exactly is prompt injection and how does it differ fromjailbreaking?
- What are thetypes ofprompt injection attacks?
- How, as a CISO, can build protection against prompt injection – in security policies, environmental architecture and technical controls?
What is prompt injection? Definition
Prompt injection is a vulnerability in which content passed to the model (user prompt or external data attached to the context)changes the behavior or result of LLM in an unintended manner. It may lead to violation of system operation rules, generation of undesirable content, disclosure of information, gaining unauthorized access or initiating actions in connected systems – depending on the architecture and permissions.
Prompt injection occurs when input (regardless of form) influences the model so that it performs actions or generates responses that are contrary to the intended function of the system and trusted instructions (e.g. roles and constraints defined in the system/developer prompt). Importantly, a malicious command does not have to be human-readable – it just needs to go to the model context and be processed by it. Therefore, prompt injection applies not only to chats, but also to GenAI agents and applications that retrieve context from documents, websites, emails or repositories (e.g. in RAG architectures) – however, the use of RAG or fine-tuning alone does not eliminate this vulnerability.
In practice, prompt injection takes advantage of the fact that the model processes instructions and data in the same input stream (usually as text, and in multimodal systems also through other modalities), so untrusted content attached to the context (page fragment, search result, ticket, mail, document) may contain a command that the model will treat as an instruction. The more agency the system has (integrations, actions, permissions), the greater the potential impact.
Difference between prompt injection and jailbreaking
The two concepts are related and are used interchangeably, but they are not the same thing:
- Prompt injection is a superordinate category: the attack involves manipulating the response or behavior of the model through specially crafted input data. The goal may be to change the content of the response, force disclosure of information, or influence what actions the application takes.
- Jailbreaking is a special case of prompt injection where the goal is to make the model ignore the security mechanisms and constraints imposed by the system and start doing things it normally shouldn’t.
In short: every jailbreak is a form of prompt injection, but not every prompt injection is a jailbreak. In the context of enterprises and agents, what is particularly dangerous is that by injecting malicious prompts, the system may be induced to perform unauthorized actions or exfiltrate data – and the scale of the effects depends on the granted permissions and integration.
Types of prompt injection
Direct prompt injection
Direct prompt injection occurs when a malicious command hits the model directly in the content of the user’s prompt. This input could be:
- intentional – when the attacker consciously constructs a prompt to change the behavior of the model,
- unintentional – when an ordinary user types something that was not intended to be a “command” and yet produces an undesirable effect.
The mechanics are simple: input provides content that the model incorrectly treats as an instruction, which can lead to bypassing constraints or performing actions that are contrary to the system’s intended function and trusted instructions and constraints.
Indirect prompt injection
Indirect prompt injection is more difficult to detect because malicious instructions do not come directly from the user. The attacker places them inexternal sourcesthat the GenAI system can process: websites, documents, emails, databases, metadata or even graphic files. Such injections can also be unintentional – if the external content contains instructions that the model will treat as commands.
This is where the most insidious element comes in: the user often cannot see the attacker’s prompt, and the tool can appear normal while executing hidden instructions “in the background”. Malicious instructions can be hidden, for example, in metadata, in invisible Unicode characters, or in formatting that is invisible to humans but readable by the model.
Indirect prompt injection particularly escalates the risk when:
- the application retrieves the context from outside (e.g. by “reading” the content),
- the model has access to tools and integrations (can take actions),
- processed content comes from a wide, difficult to control surface (WWW, emails, instant messengers, repositories).
In practice, this means that “data for analysis” also becomes a potential carrier of commands – also when these commands are not given directly in the text, but embedded in graphic material or another input channel.
Summary of the differences between direct prompt injection and indirect prompt injection
| Direct prompt injection | Indirect prompt injection | |
| Instruction source | User input going directly to the model. | Instructions embedded in external content that the model retrieves and processes. |
| Visibility for the user | Present in prompts, sometimes visible, sometimes obfuscated, but still in the model’s input stream. | Often invisible (metadata, hidden text, obfuscation, multimedia elements). |
| Typical surface attack | Interfaces where the user provides a prompt (chat, form, command). | WWW, documents, emails, instant messaging, repositories, DB records, images. |
When is the impact greatest?
In both cases – when the system has access to data and the ability to launch actions through tools/integrations and extensive permissions.
What effects can a prompt injection attack have on an organization?
Data exfiltration
Prompt injection may lead to the disclosure or exfiltration of sensitive data – both that which the model “sees” in the context of the conversation (e.g. content of processed documents, fragments of the knowledge base) and that which the application has access to through connected tools and integrations.
In practice, the risk covers data that the GenAI system can read or download during the task: email content, documents, information about customers and employees, financial data or other confidential records – depending on what sources are connected and what permissions are granted. If the application has operational channels (e.g. sending messages, publishing content, API calls), exfiltration can occur quickly, without additional exploitation of vulnerabilities in target systems, using legal permissions and integrations available to the application/agents.
Content manipulation and bad business decisions
Prompt injection can lead to response manipulation: the model generates distorted content – it omits important information, reinforces incorrect or biased conclusions, or suggests actions that are inconsistent with the intended purpose of the system and trusted constraints.
In an organization, this is particularly dangerous where the model’s response supports decisions – in risk analysis, recommendations for teams, purchasing, legal or HR processes. Even if there is no data leak, there may be real business losses resulting from decisions made on the basis of a manipulated response.
Disclosure of information about the system and system instructions (system prompt)
Prompt injection attacks often aim to extract informationthat helps the attacker refine the next steps: system instructions, how the assistant works, as well as what tools and integrations are available. This speeds up the iteration of the attack and increases the chances of successfully bypassing security measures in a specific architecture.
It is worth emphasizing this directly: these threats are classified as one of the highest risks in the OWASP Top 10 for LLM applications (2025). In practice, the result is that the model – and often the entire LLM application – does not have a hard, reliable boundary between trusted instructions (from the developer/system) and untrusted input content (from the user or from external data). Because everything goes to one context and is processed through the “same channel”, a properly crafted input can prompt the system to reveal the system prompt, operating logic or information about available functions – exactly those elements that facilitate further escalation of the attack.
Abuse of tools available to the agent
Prompt injection may prompt the agent to select and invoke tools that are in his arsenal but should not be used in a given context (e.g. reaching for the “get customer data” function during a simple email summary).
This does not necessarily mean a classic escalation of privileges at the IAM layer – more often it is a violation of the boundaries of the context and purpose of the task: the agent uses the privileges it already has in an unauthorized way relative to the intended system function and security policies.
Unauthorized actions and changes to systems
When the agent has integration and the ability to perform actions, abuse of tools may result in real changes in systems: sending a message, modifying or deleting data, updating records, launching a workflow.
Another problem is that such actions may look like “normal” operations performed by an authorized entity – only they were triggered by a manipulated context, rather than by the intended purpose of the task and trusted system instructions.
Reconnaissance and preparation of further attack
Prompt injection can be used as a reconnaissance stage: determining what data sources the system processes, what tools it has, what limitations it has, how it responds to specific commands and whether it requires confirmation of the action.
Such reconnaissance helps prepare more targeted attacks – especially when the attacker has the ability to place malicious instructions in content that the organization regularly processes (e.g. documents, emails, websites or records in systems).
How to protect your organization against prompt injection attacks?
There is no approach today that provides a 100% guarantee of eliminating prompt injection – this is due to the generative nature of AI and the fact that in practice LLM systems process both trusted instructions (system/developer) and untrusted content (from the user or external data) in one context. Reliably distinguishing “instructions” from “data” and detecting malicious intent is not something that can be assured deterministically under all conditions today.
This does not mean, however, that we are defenseless.
The organization can significantly reduce the probability of a successful attack and – often more importantly – limit its effects by building layered protection: from employee security awareness, through governance and AI use policies, to technical security, permissions control, adversarial tests and monitoring the behavior of models, agents and their interactions with tools..
Increasing threat awareness among employees
Prompt injection is a semantic type of manipulation – so in practice it hits where employees use GenAI to work with content: summarizing emails, analyzing documents, summarizing instant messaging threads or asking to perform tasks in tools.
If the user does not understand that the LLM system processes both instructions and data in one context, it is easy to engage in risky behavior: pasting untrusted content into tools, asking agents for too broad actions (“do what it takes”), launching integration without thinking about the consequences and what data and actions the agent has access to.
Therefore, cybersecurity education should include not only general cyber hygiene, but also specific rules for safe work with GenAI – including awareness of the risks related to external content, integrations and the scope of commands given to agents.
At SECAWA we can provide various forms of education for all roles in the organization and management levels:
- e-learning in cybersecurity for companies (online lessons and refresher quizzes),
- Practical Anti-Phishing Training – systematic, realistic simulations of attacks that translate knowledge into habits
- cybersecurity training and workshops.
Contact us to discuss the best security awareness solution for your organization.
AI policies and governance
Prompt injection protection starts with rules that reduce the attack surface and limit the consequences when something goes wrong. In practice, there are four areas:
Visibility and control of GenAI (shadow AI) use
If an organization does not know what AI tools are used and for what (the Shadow AI phenomenon) – it is not able to manage the risk. Rules are needed that distinguish between permitted and non-permitted tools and define what data they can process and in what scenarios.
Principles of working with content sources
Indirect prompt injection is based on the fact that the model processes content from sources over which the organization does not have full control – or which may have been modified.
Therefore, it is worth determining what sources are considered trusted (allowlisting), when to use a precautionary approach and how to consistently treat content from the web, emails, documents, instant messengers or records in systems as untrusted input to the context.
Rules for formulating commands and scope of freedom of action
The more general the command and the greater the freedom of action, the easier it is to produce an undesirable effect. Governance AI should promote precise tasks, limited scope and clear criteria for what the model should do – and what it should not do (especially when working on external content and having access to tools).
Least privilege as default rule
If an agent has access to tools and data, they shouldonly receive the permissions necessary to perform a specific task – and nothing more. Conscious design of what an agent can see and what it can change or run in connected systems is key.
In practice, this means limiting access to sensitive resources, minimizing the scope of available functions, and clearly separating “read-only” operations from those that cause changes to systems. The fewer rights and fewer actions an agent has, the smaller the potential impact – even if a prompt injection occurs.
Technical inspections and monitoring
The technical layer has two goals: to make an effective attack more difficult and to quickly detect anomalies before they turn into an incident. The following actions are a practical synthesis of the key recommendations OWASP:
| Goal | What to implement | What to measure/log | |
| Limiting model behavior | Reduce susceptibility to “shifting” the role and forcing actions out of scope. | 1. Precise system/developer prompt: role, scope, prohibitions, terms of use of tools. 2. The “task bounding” principle – the model is to implement only clearly defined types of tasks, without default initiative. 3. Clear rules for ignoring attempts to modify trusted instructions. | 1. Model version + configuration version (e.g. hash/ID system prompt, tool policies). 2. Detection of override/jailbreak attempts (flags from the classifier/heuristics). 3. Out-of-scope response rate, denials and escalations to human. |
| Define and validate expected output format | Limit the possibility of unwanted content being “injected” into responses and enforce predictable results. | 1. Hard response schemas (JSON/schema, field lists, report format). 2. Deterministic validation on the application side (parser + rules). 3. “Fail closed” rules – if the validation fails, do not perform the action, do not propagate the result further. | 1. Percentage of non-compliance with the scheme, reasons for rejections, retry count. 2. Cases of “schema drift” after prompt/tool changes. 3. Correlation: Format incompatibility ↔ unusual tool use. |
| I/O Filtering and Control | Detect and block risky content (including hidden instructions) and limit disclosures. | 1. Input and output filters (rules + classifiers) for categories of sensitive and typical prompt injection signals. 2. Disclosure control: redaction of secrets/PII, blocking “prompt leakage”. 3. For RAG: assessment of response quality (context relevance, grounding in sources, Q/A consistency) and rejections when the model drifts away. | 1. Filter result: allow/block + categories + confidence. 2. RAG telemetry: which documents entered the context, scoring, rejections, source conflicts. 3. Redaction cases (what was redacted and why – without logging the secrets themselves). |
| Permission control and the principle of minimum permissions | Even if prompt injection is successful, limit the possible impact. | 1. Minimum scope of access to data and functions – per task and per agent. 2. Separating operations that do not change state from those that change state (and the latter – as narrowly as possible). 3. Tokens/credentials on the application side, not “in the model”; the model should not “have” secrets. | 1. What permissions are active for a given session/task (policy ID). 2. Each use of the tool: who initiated it, what was the purpose of the task, what were the parameters (with redaction of sensitive values). 3. Attempts to use functions outside the permitted scope (policy violations) |
| Human confirmation for high-risk actions | Block unauthorized actions that change the state of systems or reveal data. | 1. “Human-in-the-loop” for mutating and sensitive operations (sends, record modifications, deletions, publications, transfers, permission changes). 2. Verification of the context of the action: what was the basis for the decision, what sources provided the instructions. | 1. Confirmation events: request → approve/deny, response time, action summary content. 2. Percentage of actions stopped by user or rules. 3. The most common types of actions requiring confirmation (for tuning policies). |
| Separation and marking external content as untrusted | Limit the influence of external data on trusted instructions and decisions about the use of tools. | 1. Explicit “trust boundaries” in context: explicit separation of external content from system instructions. 2. Source policy: allowlisting, limiting domains/file types, precautionary principle for WWW and unverified content. 3. Content normalization/cleaning (e.g. removing hidden elements, metadata where it makes sense). | 1. Context origin: sources, data types, acquisition path (Web/document/email etc.). 2. “Untrusted content” flags and detection of hidden/obfuscated elements. 3. The share of external content in context (volume, number of sources) vs. incidents/anomalies. |
| Adversarial testing and attack simulations | Detect vulnerable paths before the attacker does – and maintain resilience after changes. | 1. Regular direct and indirect prompt injection tests on real flows (RAG, emails, documents, WWW, tools). 2. Action scenarios: whether the agent can be tricked into unauthorized actions, disclosures, use of tools out of context. 3. Regressions after each change: prompts, tools, integrations, policies, data sources. | 1. Test coverage (scenarios, source types, tool types), pass/fail results, bypass vectors. 2. Vulnerability trends over time (is it better/worse after implementations). 3. “Top failure modes” – the most common ways in which the model/agent fails the task. |
Prompt injection is a real attack vector against GenAI applications and agents based on language models. The risk increases wherever the system processes external context and has access to tools, integrations and sensitive data. Since there is no approach today that provides a 100% guarantee of eliminating prompt injection, a layered approach is crucial: raising awareness and rules for safe work with GenAI, AI policies and governance that limit the attack surface, and technical controls, tests and monitoring that reduce the effects and shorten the detection time.
