LLM-based systems introduce a new operating model, but many security threats result from well-known risk patterns. What changes is not the target of the attack itself, but the way in which data, instructions, tools and permissions are combined in one flow. This is why prompt injection and other attacks on AI systems should not be analyzed solely as a problem of the model, but as a problem of the entire system architecture.
In practice, this means that the risk does not end with a malicious prompt, but can start with input data, go through context and RAG, use tools or integrations, and finally lead to abuse of privileges, data leakage or unauthorized actions.
For CISOs, it is not about a single vulnerability, but about layer-by-layer threat modeling.
The scale of the problem is serious: it can be said that 100% of AI systems are vulnerable to attacks because there is no fully effective method of defending against prompt injection today.
Therefore, AI systems should be designed with the assumption that individual protections can be bypassed, and resilience must be built in multiple layers – at the level of input, context, RAG, tools, permissions, output and operational control.
How does communication with LLM (Large Language Model) work?
To model threats well in AI systems, you must first understand how such a system actually works.
From the user’s perspective, it looks simple: we ask a question and get an answer. In practice, however, the model does not work on a single sentence, but on the entire context provided to it by the application.
In its simplest terms, this flow includes system instruction, user input, model, and system response or action. This is where the main attack surface related to prompt injection also appears.
Language model messages and context
For a language model, context is everything it “knows” about the current interaction. It includes not only the content of the user’s question, but also system instructions, previous messages, and in more complex architectures also external data and results of tool operation. It is a set of messages that are sent to the model only after being assembled. This is important because the model does not respond only to the last message – it works based on the entire context it receives from the application.
Question-answer
The simplest variant is the question-answer architecture. The system passes a system instruction and one user message to the model and then receives the response.
This flow looks the cleanest, but already at this level the basic security logic is visible: the model responds to what is in the context, not to the intention of the system designer. Therefore, even a simple chatbot is not just a conversation interface, but a system whose behavior depends on how the input context was built.
Conversation
The situation becomes more complicated when we move to multi-turn conversations. Then, not only the user’s new question is added to the context, but also the model’s previous answers and conversation history
From a functionality point of view, this is natural – the system should “remember” what the conversation was about.
From a security point of view, however, this means thateach subsequent invocation of the model relies on an increasingly broader set of contentthat can influence the response logic. The longer and richer the conversation, the more important is what exactly is recorded in the context.
RAG
The next level is RAG, i.e. enriching the context with external sources of knowledge. The context window may receive documents, database records, website content or other materials that are intended to help the model provide a more complete response.
This allows you to connect the model with organizational knowledge or external data. However, from a security perspective, this means that the model begins to receive content outside the direct conversation with the user – and therefore also content that may be erroneous, unverified or malicious
Use tools
The most extensive variant is an AI system in which the model not only responds, but also uses tools. In such a scenario, the model can first call a specific function, API or automation, and only then – after receiving the result – generate the final response. The result of the tool also returns to the context and influences the further course of the task.
This is a key moment from the CISO’s perspective, because the AI system then ceases to be only a response layer and begins to have a real impact on data, processes and operations. The more tools, integrations and intermediate steps, the greater the risk surface.
This is why communication with the LLM model should not be reduced to a simple “question-answer” pattern. In practice, it is a flow in which context can be built from many sources, expanded with conversation history, fed with external data and supplemented with the results of tool operation. And if so, the security of the AI system must be assessed not only at the model level, but also at the entire architecture that creates this context.
What is prompt injection?
Prompt injection is an attack involving the introduction of a specially prepared instruction that induces the AI model to act in a manner inconsistent with its intended purpose. A malicious prompt does not have to “corrupt” the model itself – it just needs to influence how the model interprets the context and what actions it considers appropriate. This is why Prompt Injection attacks are today treated as the main attack surface in LLM-based systems.
This means that in the case of such an attack, the model receives not only the right question or task, but also an additional, malicious instruction that is supposed to change the way it works. If the system does not separate data from instructions well enough, the model may treat malicious content as a control instruction and respond or act contrary to the system designer’s intentions.
Effects of prompt injection
A prompt injection attack for an organization could mean:
- disclosure of confidential information,
- performing unauthorized actions,
- manipulating the AI system’s responses,
- loss of trust of customers and partners.
We wrote more about prompt injection, types and consequences in this article: http://secawa.com/en/blog/prompt-injection-what-it-is-and-how-to-limit-the-risk/
How to model threats in AI systems?
For CISO the key question is not only whether the system is resistant to prompt injection, but how to assess the risk separately for each layer of the AI system – from input and context, through RAG and tools, to permissions, output and operational control.
In practice, modeling threats in AI systems is worth dividing into 7 areas: inputs, context, RAG, tools and integrations, permissions, output and telemetry and control.
01 Input
This is the first and most obvious area of threat modeling. The following may be sent to the AI system:
- data from the user,
- files,
- websites,
- emails,
- tickets,
- repositories.
It is these entrances that the materials indicate as themain entry points for prompt injection. The problem is that the system does not always distinguish between regular content and malicious instructions. The more input sources and data formats there are, the greater the risk that a malicious prompt will be treated as part of the legitimate context.
02 Context
The second layer is the model context itself, i.e.:
- system prompt,
- memory
- policies.
This is where attempts to hijack the system’s operating logic or reveal hidden instructions occur. From a security perspective, this is critical because context manipulation can lead to a change in the system’s operating logic or the disclosure of model control instructions.
03 RAG
The next layer is RAG, a mechanism for enriching the model context with documents and data from the knowledge base. From an architectural perspective, this includes, among others: vector database, retriever and ranker.
In this layer, the main risks are data extraction, context manipulation and data poisoning. If the system retrieves content from documents, web pages, emails or other external sources, indirect prompt injection can occur without direct user interaction with the attacker.
04 Tools and integrations
When a model has access to APIs, functions, automation or MCP servers, it stops being just a response generation layer and begins to influence processes, data and activities performed outside the LLM itself.
This is where the risk of unauthorized actions and misuse of tools comes into play. Manipulating the prompt, context or task flow is enough for the AI system to perform an operation it should not perform.
05 Permissions
This is one of the most important layers from a CISO perspective. The question is not only what the model has access to, but also whose permissions it operates under and what scope of access the connected tools have.
The key principle is simple: access control must operate beyond LLM, at the application level. If the system does not ensure this, the AI assistant may perform operations with a greater scope of permissions than it should, or launch actions to which the user would not normally have access.
06 Language model output
Threat modeling doesn’t end with input. Equally important is what the system returns next: responses, links, HTML, integrations and data passed to subsequent components.
This is important because the model’s response may be:
- harmful in itself,
- input for the next system,
- carrier of data leakage,
- the starting point for further activities in the automation chain.
Therefore, the model output must be treated as an element of the attack surface, not just the final result.
07 Telemetry and control
The last layer is operational supervision of the AI system – logs, limits, alerts, action approval and anomaly detection. This part does not block the attack itself, but gives the organization the ability to audit, detect abuse and respond quickly.
This is especially important where the system performs activities that are costly, sensitive or difficult to reverse. Without telemetry and control,an organization loses visibility into what the AI system did, how it behaved when executing commands, and whether there were any abuses, anomalies, or costly operational impacts.
7 questions a CISO should start assessing the risk of AI systems
The resilience of the AI system must be designed in many layers. Therefore, a good starting point is to go through the 7 layers of the AI system and assess the risk separately for each of them – from input data and context, through RAG and tools, to permissions, output and operational control.
What can enter the system?
The first question is not about the model, but about input channels. The AI system may receive not only user prompts, but also files, websites, emails and documents. This is where most successful attacks begin, because the model can treat untrusted data as an instruction. Therefore, the CISO should check which entries are filtered, whether entry screening works, whether prompt injection patterns are detected and whether the organization responds to repeated user abuse.
What goes into the model context?
The second question is: what exactly does the model see as context. Good practice is not about a “stronger system prompt”, but about separating data from instructions, clearly labeling what is a system command and what is user or document content, and minimizing the context provided to what is really needed to perform the task. The CISO should also assess whether the context contains secrets, sensitive configurations or unnecessary information that increases the risk of leakage.
What sources of knowledge enrich the answer?
With RAG, the key question is not only what the system uses, but also how it trusts these sources. Documents, web pages, PDFs, emails and comments may contain malicious instructions, so external sources must be treated as untrusted and RAG content filtered and flagged before being included in the context. From the CISO’s perspective, it is also important whether the system maintains control over permissions to source data, whether it segments repositories and whether it detects mass data extraction attempts.
What tools can the model invoke?
When an AI system uses APIs, functions or automation, a wrong answer can quickly become a costly operational failure. Therefore, the CISO should assess what tools the model can run, whether their use is validated before performing the action, whether the output returned by the tools is controlled and whether the model cannot independently skip required process steps. In practice, the security of this layer is based on validating calls, controlling the sequence of actions and limiting the scope of integration to the minimum necessary for a given task.
Whose permissions does the system run on?
This question often determines the scale of the incident. The CISO should determine whether the agent operates in the context of a user, application, or privileged backend, and whether authorization occursoutside the LLMat the application level. Good practices here are clear: all operations should be performed with user, not system, privileges, the scope of tools should be minimal, and the user, agent and backend identities should be separated. Without this, prompt injection may turn into privilege escalation and unauthorized operations.
What comes out of the model and where does it go next?
The model output is not just a response to the user. It is also a potential source of data leakage, another step of automation or an entrance for another system. Therefore, the CISO should verify that the organization controls model responses, detects attempts to disclose data in the output, limits the length of responses and the number of steps, and does not include secrets in prompts and context. In practice, quickly limiting the effects of exfiltration and cost abuse is as important as blocking an attack.
How does the organization log, control and test it?
The last question is whether the organization hasreal supervision over the AI system after implementation. Checking once is not enough. Effective supervision should include system telemetry, alerts, logs, agent behavior monitoring, cost and token control, and regular testing of resistance to prompt injection and RAG attacks. It is also important to measure not only the effectiveness of blocking, but also the quality of the response – so that after implementing security measures, the system continues to operate correctly, accurately and predictably.
Summary
The security of AI systems cannot be assessed by a single test or a single defense prompt. In AI systems, risk arises throughout the entire flow – from input data, through context and RAG, to tools, permissions, output and the telemetry and control layer. Therefore, it is worth conducting risk assessment layer by layer, and not only at the level of the model itself.
From the CISO’s perspective, the most important conclusion is simple: there is no single, fully effective method of defense against prompt injection today. But this does not mean that organizations are helpless.
The resilience of AI systems must be designed in many layers – combining entry control, secure context building, limited trust in sources, validation beyond LLM, permissions control, monitoring and regular testing of resistance to real attack scenarios. Just as important as blocking an attack is limiting its effects and quickly detecting violations.
If you want to verify how your environment will handle attack scenarios in practice, it is worth basing it on controlled tests. At SECAWA, we conduct professional penetration tests, which help discover system vulnerabilities and then turn the results into specific recommendations for corrective actions. This will help prepare the organization for real cyberattacks and better protect data, reputation and business continuity.
Test your system security with controlled cyberattacks
