JadePuffer is the first documented ransomware attack carried out from start to finish by an autonomous AI agent, without human involvement in decision-making at any stage of the intrusion. The Sysdig Threat Research Team tracked down an operation in which a large language model independently exploited a vulnerability in Langflow, took over credentials, moved to a production server running MySQL and Alibaba Nacos, encrypted 1,342 configuration items, and left a note demanding a ransom in Bitcoin.

The agent did not copy the finished script. He diagnosed the cause of the error and implemented a fix at a pace unattainable for a human reading the log by hand: in a documented case, 31 seconds passed from a failed login attempt to a working correction.

JadePuffer attack at a glance

First fully autonomous LLM-driven ransomware attack reported by Sysdig in July 2026
Entry point: CVE-2025-3248, RCE vulnerability in Langflow, a popular framework for building AI applications
Primary target: separate, internet-exposed production server with MySQL and Alibaba Nacos
Agent response time to login error: 31 seconds from failed attempt to working patch
The encryption key never reached the attacker – the victim’s data cannot be recovered even after paying the ransom

The existing incident response procedures assumed that there is always a person behind a ransomware attack who can be slowed down, misled or with whom time can be negotiated. JadePuffer shows that this stage of tradecraft can now be handed over to the model.

What is ransomware?

Ransomware is malicious software that encrypts files or entire operating systems, and the unlocking of data requires the payment of a ransom, most often in cryptocurrency. This type of attack has been one of the most profitable tools in the hands of criminal groups for over a decade, because it attacks what has the greatest operational value for an organization: access to its own data.

The classic ransomware Trojan scenario looks similar regardless of the variant. Malicious code enters the system through phishing, a broken password or a software vulnerability. It then silently encrypts key data on disk and in backups, and displays a ransom note with a payment deadline.

The story shows the growing ingenuity of criminals, but also the repeatable pattern of ransomware: encrypt, demand, wait.

CryptoLocker (2013) – one of the first mass-distributed ransomware programs, spreading through infected email attachments and encrypting files with a strong asymmetric key.
CryptoLocker 2.0 – A variant rewritten in a different programming language that appeared shortly after the infrastructure of the original CryptoLocker was broken. Security researchers assessed it as a copycat using a recognizable name, not a continuation of the original code, and technically less refined.
CryptoDefense – a variant that, due to a programming error, stored the decryption key locally on the infected machine, which allowed some victims to recover their data without paying the ransom.
Diskcoder ransomware – a family of malware that destroys the disk’s master boot record, making file encryption even impossible to boot the operating system.

A ransomware group doesn’t have to write its own code from scratch today. The cybercrime market offers ready-made ransomware-as-a-service kits, and the operator’s role is often limited to selecting the target and sending the first payload. Ransomware threats have evolved from individual, proprietary programs into a service-based, repeatable business model – even before autonomous language models came into play.

JadePuffer – the first ransomware attack fully controlled by LLM

The Sysdig Threat Research Team documented Operation JadePuffer in the summer of 2026 – the first known case in which the entire ransomware attack chain, from reconnaissance to data destruction, was led by an autonomous agent based on a large language model, without human intervention at any stage of the intrusion. Sysdig calls this type of adversaryagentic threat actor (ATA) – an actor whose offensive ability comes from an AI agent rather than a manually maintained toolkit.

The attack took place on two different targets. The first compromised host served the agent only as a starting point to hack into the actual, much more valuable target.

How the JadePuffer attack worked step by step

Phase 1 – input host (Langflow)

Entry via CVE-2025-3248 – Unauthorized RCE vulnerability in the code validation endpoint of Langflow, a popular open-source framework for building LLM applications and agents. The manufacturer patched the vulnerability on April 1, 2025, and CISA marked it as actively exploited in May of the same year.
Reconnaissance and secret capture – the agent collected host data (id, uname, network interfaces, processes) and in parallel searched the environment for API keys of LLM providers, cloud credentials (including Chinese providers Alibaba, Tencent, Huawei, in addition to AWS, GCP and Azure), cryptocurrency wallets and database data.
Dump of the local Postgres Langflow database – stolen credentials, API keys and user records were saved locally, previewed, and temporary files deleted.
MinIO enumeration – The agent used the default credentials (minioadmin:minioadmin) to search the object bucket, including the terraform-state bucket. When one of the queries returned XML instead of the expected JSON, the next payload immediately adapted the parser to the new format.
Maintain access – on the Langflow host, the agent installed an entry in the task scheduler (crontab), sending a signal to the attacker’s infrastructure every 30 minutes. As a result, even a server restart did not interrupt the connection with the attacker.

Phase 2 – actual target (MySQL production server and Alibaba Nacos)

Pivot to a separate server – from the Langflow host, the agent moved to an Internet-exposed production server with a MySQL database and the Alibaba Nacos service, using a root account whose origin the researchers were unable to determine.
Nacos Takeover – In parallel, the CVE-2021-29441 authentication vulnerability was exploited, the JWT was spoofed using the publicly known default Nacos signing key, and a backdoor administrator account was injected into the database via root access.
Recognition of container escape – the agent tested writing and reading files via MySQL, checked access to docker.sock and init process files – a typical set of actions preceding an attempt to exit the container.
Encryption – All 1342 Nacos configuration elements were encrypted with the AES_ENCRYPT() function in MySQL, the original tables were deleted and in their place a README_RANSOM table was created with the ransom note, Bitcoin address and email contact.
Mass data destruction – The agent escalated from deleting single lines to DROP DATABASE commands on entire schemas, commenting in the code its own prioritization of targets according to the potential value of the data. It is worth noting: the annotation that the data was previously copied to an external server is a statement made by the agent itself, written in its own code, and not a fact confirmed independently by researchers. The actual scale of data exfiltration remains unknown.

Four pieces of evidence that the attack was directed by an LLM, not a human

Sysdig identifies four independent premises that together rule out the scenario of a fixed, pre-written script:

Code commenting on own actions. Payloads contained detailed justifications in natural language – why a given database is a priority, which is the “biggest”, what a given step is supposed to achieve. A person writing a one-time script does not do this, the language model does it instinctively.
Fault diagnosis and correction in seconds. The most precise evidence of autonomy can be seen in the sequence of creating an administrator account in Nacos:

Time (UTC)	Agent Action
19:34:24	Inserts the xadmin account with the password hash generated by the call subprocess
19:34:36	Login attempt fails
19:34:48	Tests two possible causes of error in parallel
19:35:07	Introduces a fix: direct import of bcrypt library, removal and restoration accounts
19:35:18	Login is successful

From the failed attempt to the working fix, 31 seconds passed – a time unattainable for a human reading the error log, making a diagnosis and writing a correction.

Understanding context given in natural language, not just pattern matching through the scanner – repeatable behavior in sessions weeks apart. This is the same ability of the model to interpret and respond to natural language that is sometimes abused in prompt injection attacks in other scenarios. Here, however, the agent used this ability in its own offensive action, not as the target of injected instructions.
Ambiguity around the Bitcoin address in the ransom note – this is a standard sample P2SH address replicated in the Bitcoin documentation, likely reproduced by the model from training data. However, the blockchain shows that it is also an active wallet: 737 confirmed transactions and approximately 46 BTC received throughout its history, with each inflow being immediately forwarded, and the current balance is zero. This does not resolve whether the model simply generated this address from training data or whether the operator consciously configured the agent with a real, controlled wallet, but it strengthens the argument that this is not a random, dead string. A total of over 600 separate, targeted payloads made in compressed time confirm that the operation was conducted by an autonomous agent, not a human behind the keyboard.

Consequences for the organization – why this is a wake-up call for CISOs, CTOs, CSOs and CIOs

The most serious consequence of an LLM-driven ransomware attack is not the data encryption itself, but the fact that the assumptions on which data recovery and incident response procedures have been based so far no longer work.

Paying the ransom does not guarantee data recovery. The encryption key in JadePuffer was randomly generated and never sent to the attacker – it was only entered once on the agent’s session screen. The victim will not recover 1342 encrypted Nacos configuration items even after fulfilling the ransom note’s request.
Backups do not solve the problem by themselves. The target of the attack was the live configuration database of the production server, not just the files on the disk. Backup encryption and backup procedures only make sense if the copies are isolated from the same access path used by the agent to attack the source system.
Resources like NoMoreRansom.org will not help here. Initiatives offering free data recovery work when it is possible to crack the implementation of a known ransomware family or recover a compromised key. With a randomly generated AES key that is not stored anywhere, there is nothing to crack.
The entry threshold for conducting ransomware attacks drops to the cost of running the agent. Active ransomware groups today, such as Rhysida or Qilin (which took over a large part of the market after RansomHub disappeared from the network in the spring of 2025 and was absorbed by the DragonForce cartel), are natural candidates for taking over similar agent tools.
The pace of the attack eliminates the window for response. The sequence from failed login to working patch took 31 seconds. A quarterly security review of an infrastructure that changes daily leaves gaps that an autonomous agent can exploit before anyone notices the incident.

JadePuffer is not an isolated signal that language models can be turned against an organization

We recently described how attackers convinced the Meta AI chatbot, to reset passwords and help take over 20,225 Instagram accounts – completely different vector, same mechanism: a model performing malicious actions based on what it “understood”.

The growing number of reported ransomware attacks and ransomware incidents using AI shows that this is no longer a theoretical scenario, but part of the cyberattacks in the GenAI era landscape that organizations must face today. If you want to see more such cases broken down into prime factors, in the free webinar series AI vs. Cybersecurity Secawa we covered both AI-driven attacks and attacks targeting AI systems themselves, with recordings and downloadable materials.

Free series of webinars: AI-supported attacks, Shadow AI, AI-prompt injection attacks and AI as CISO support. 4 meetings, additional materials and several SECAWA specialists who discussed the topic of AI in the context of cybersecurity in detail.

Receive recordings and materials from the AI vs Cybersecurity series

How to protect your organization against AI-driven ransomware?

Preventing ransomware attacks of this type does not require new tools, only consistent closing of vulnerabilities that JadePuffer exploited in No zero-day exploit was needed:

Patch Langflow to remove CVE-2025-3248 and do not expose execution or code validation endpoints to the internet.
Do not store API keys and cloud credentials in an AI orchestration server environment. Secrets should go to a dedicated manager, out of reach of network-accessible processes.
Harden Nacos – change the default signing key (token.secret.key), update to a version that enforces its own key, never expose the service to the Internet and never connect it to the database as root.
Do not expose the database administrative account to the Internet. Strong, unique credentials and restrictions after the source IP address are a minimum.
Introduce egress control so that a compromised application host cannot freely connect to any infrastructure or external databases.
Invest in real-time threat detection at the level of database processes, not just at the network or file signature level.
Monitor compromise indicators – unusual cron jobs with network calls, anomalies in request headers, sudden spikes in the number of requests to configuration services.

However, none of these recommendations will work without people who understand that social engineering in the era of AI agents looks different than it did just two years ago. It is also worth preparing security and IT teams for social-engineering methods supported by GenAI, because the line between a purely technical attack and one supported by manipulation is increasingly blurred.

Summary

JadePuffer changes three assumptions on which previous defense planning was based. First, the barrier to entry for conducting a ransomware attack has dropped to the cost of running an agent – and with models powered by stolen computational access, that cost approaches zero. Second, old, seemingly benign vulnerabilities – like the 2021 Nacos vulnerability – are now automatically refreshed by agents searching the entire historical CVE catalog, so neglected, unpatched infrastructure becomes a more—not less—attractive target. Third, the code generated by LLM self-describes its intent – the same feature that makes the agent dangerous gives defenders a new chance to detect an attack before it is encoded.

Threat modeling for AI-based systems is no longer an academic exercise – it is the starting point for risk assessment in any organization that implements agent-based AI tools or is a potential target of ones such as JadePuffer. You can find more about how to systematically model the threats related to prompt injection in AI systems in our previous material.

If you prefer to see these mechanisms broken down into prime factors live, we invite you to a free series of webinars AI vs. Cybersecurity, where we covered both AI-driven attacks and attacks on AI systems themselves – with recordings and downloads available.

Sources:

https://www.bleepingcomputer.com/news/security/jadepuffer-ransomware-used-ai-agent-to-automate-entire-attack

https://www.darkreading.com/cyberattacks-data-breaches/jadepuffer-first-complete-llm-driven-ransomware-attack

At the end of May, an instruction appeared on Telegram channels that allowed you to take over someone else’s Instagram account without knowing the password and without access to the victim’s mailbox. All you needed was a chat with Meta AI.

Meta confirmed that 20,225 accounts were compromised using this method, and the victims included the dormant Obama White House page, the profile of the Sephora cosmetics chain and the account of a high-ranking US Space Force officer.

How did Meta AI chat help take over Instagram accounts?

Hackers took over accounts by instructing the Meta AI chatbot to link the victim’s profile to their own email address, to which the system sent a password reset link. They didn’t crack passwords or break into other people’s mailboxes, the entire attack came down to a conversation with the chatbot.

According to Meta’s filing, the vulnerability was detected on May 31, 2026, and the Maine Attorney General’s log shows April 17 as the likely date of the first attack. This would mean that thevulnerability remained open for approximately six weeks, although Meta has not officially stated when the attacks began.

What was the vulnerability in the Meta AI chatbot?

The weak link turned out to be High Touch Support (HTS), an artificial intelligence-supported account recovery system launched in March 2026. Meta built it to relieve slow service for users cut off from Instagram.

As the company admitted in a breach report filed with the Maine Attorney General, the tool itself worked as intended, but an error in a separate code path meant that thesystem did not verify whether the specified email address even belonged to a given account. Instead of rejecting such a request, it sent a reset link to a foreign address. The mechanism that failed was a missing authorization check in the backend of the recovery process.

What did the Meta AI chat attack look like step by step?

The script itself was very simple. The attacker connected via VPN from an IP address near the victim’s typical location, probably to make the reset request look credible and not arouse the system’s suspicion.

Then it would ask for a password reset and go to the chat with Meta AI. There, he instructed the chatbot to link the account to a new email address, which received a one-time code.

This allowed me to set a new password and log in to someone else’s account. There was one condition: the victim’s account could not have two-factor authentication enabled (2FA).

What does this attack teach about the security of AI agents?

The chatbot was able to perform serious operations, such as password reset or email change, but it had no way to verify that the real account owner was asking for them. The same mistake, powerful permissions without identity checks, is repeated in a whole class of attacks targeting AI agents.

Why is chatbot a new cyber attack surface?

The problem is not the artificial intelligence itself, but the scope of powers it has been given.

This is a fundamental architectural flaw. The model was granted permissions for sensitive operations without proper access control.

Brian Westnedge, Red Sift, statement for Reuters (transl. editor)

The attack is similar to social engineering used against human consultants: instead of breaking security, the attacker simply asked to perform the operation and the system carried it out.

Some of the experts quoted by Reuters classified the incident as prompt injection, i.e. conversational manipulation of the model. It is worth noting, however, that the technical explanation of Meta’s own technical explanation points not to a vulnerability of the language model, but to a missing verification in the code handling password reset. Regardless of the classification, the conclusion remains the same: the bot was given the right to perform sensitive operations without a mechanism to check who is asking for them and on what basis.

The scale of the potential for harm increases as more platforms turn to chatbots to handle logins, password resets, and account recovery. Ian Goldin from Black Lotus Labs warns that we are entering uncharted territory and that similar attacks will increase. Engin Kirda from Northeastern University put it bluntly: People used to be the targets of fraud, now they are AI agents.

Prompt injection and the security of AI systems. How to model threats?

Read the article

How to protect your Instagram account from being taken over by hackers?

Two-factor authentication turned out to be the most effective barrier. According to the hackers themselves, the exploit did not work against any account with MFA enabled, and even the weakest variant, i.e. the SMS code, blocked it.

However, it is worth using more powerful methods, i.e. a hardware key or passkey. The SMS code, although sufficient in this case, remains susceptible to taking over the victim’s number (SIM swapping), so it should not be the only line of defense.

Since the attack involved connecting a foreign email address to the account, it is also worth checking in the Account Center whether an unknown address or phone number has been assigned to the profile and deleting such entries.

Following the discovery of the vulnerability, Meta disabled HTS and invalidated generated links, forced password resets on affected accounts and placed them under a mandatory security checkpoint, and announced a review of similar recovery processes on its platforms.

How to protect your organization against AI attacks?

The case of Instagram is a textbook example of a threat that we have analyzed in the free series “AI vs Cybersecurity”. Four episodes show not only how cybercriminals use GenAI for attacks, but also how to use AI in everyday work CISO to effectively protect the organization.

The closest episode to the described attack is episode number three: Attacks on AI. We explain how to model and minimize the risk of attacks against AI systems, including model manipulation and prompt injection. This is a related class of threats that the Meta AI case clearly exposed: an agent with the right to perform sensitive operations becomes a target when there is no control over who requests them and on what basis. In addition, we provide a shortened step-by-step guide on how to reduce such risk in practice.

The remaining episodes expand on this picture:

In the first episode – AI Atakuje – we show how criminals use AI for social engineering attacks and automation of activities; The package includes a checklist of 15 questions for CISOs and a guide to recognizing phishing.
Shadow AI(ep.2) addresses the uncontrolled use of AI tools by employees and provides the AI Act validator, a guide to the safe use of AI at work, and a checklist for managing Shadow AI in the organization.
AI as CISO support, the last episode in the webinar series, reverses the perspective and shows how to use artificial intelligence on the defense side.

Access to recordings and all materials is free. Just fill out the form on the website cyklu AI vs Cybersecurity.

Get access to recordings and materials from the free AI vs Cybersecurity webinar series

Fill out the form

Sources:

https://krebsonsecurity.com/2026/06/hackers-used-metas-ai-support-bot-to-seize-instagram-accounts

https://www.bleepingcomputer.com/news/security/meta-ai-support-data-breach-affects-20-000-instagram-accounts

https://insideretail.asia/2026/06/04/how-the-sephora-instagram-hack-exposed-metas-ai-weakness/

https://www.404media.co/hackers-simply-asked-meta-ai-to-give-them-access-to-high-profile-instagram-accounts-it-worked

https://www.reuters.com/legal/government/high-profile-meta-ai-chatbot-breach-spotlights-security-risks-automation-2026-06-03

LLM-based systems introduce a new operating model, but many security threats result from well-known risk patterns. What changes is not the target of the attack itself, but the way in which data, instructions, tools and permissions are combined in one flow. This is why prompt injection and other attacks on AI systems should not be analyzed solely as a problem of the model, but as a problem of the entire system architecture.

In practice, this means that the risk does not end with a malicious prompt, but can start with input data, go through context and RAG, use tools or integrations, and finally lead to abuse of privileges, data leakage or unauthorized actions.

For CISOs, it is not about a single vulnerability, but about layer-by-layer threat modeling.

The scale of the problem is serious: it can be said that 100% of AI systems are vulnerable to attacks because there is no fully effective method of defending against prompt injection today.

Piotr Kaźmierczak, CEO of SECAWA

Therefore, AI systems should be designed with the assumption that individual protections can be bypassed, and resilience must be built in multiple layers – at the level of input, context, RAG, tools, permissions, output and operational control.

How does communication with LLM (Large Language Model) work?

To model threats well in AI systems, you must first understand how such a system actually works.

From the user’s perspective, it looks simple: we ask a question and get an answer. In practice, however, the model does not work on a single sentence, but on the entire context provided to it by the application.

In its simplest terms, this flow includes system instruction, user input, model, and system response or action. This is where the main attack surface related to prompt injection also appears.

Language model messages and context

For a language model, context is everything it “knows” about the current interaction. It includes not only the content of the user’s question, but also system instructions, previous messages, and in more complex architectures also external data and results of tool operation. It is a set of messages that are sent to the model only after being assembled. This is important because the model does not respond only to the last message – it works based on the entire context it receives from the application.

Question-answer

The simplest variant is the question-answer architecture. The system passes a system instruction and one user message to the model and then receives the response.

This flow looks the cleanest, but already at this level the basic security logic is visible: the model responds to what is in the context, not to the intention of the system designer. Therefore, even a simple chatbot is not just a conversation interface, but a system whose behavior depends on how the input context was built.

Conversation

The situation becomes more complicated when we move to multi-turn conversations. Then, not only the user’s new question is added to the context, but also the model’s previous answers and conversation history

From a functionality point of view, this is natural – the system should “remember” what the conversation was about.

From a security point of view, however, this means thateach subsequent invocation of the model relies on an increasingly broader set of contentthat can influence the response logic. The longer and richer the conversation, the more important is what exactly is recorded in the context.

RAG

The next level is RAG, i.e. enriching the context with external sources of knowledge. The context window may receive documents, database records, website content or other materials that are intended to help the model provide a more complete response.

This allows you to connect the model with organizational knowledge or external data. However, from a security perspective, this means that the model begins to receive content outside the direct conversation with the user – and therefore also content that may be erroneous, unverified or malicious

Use tools

The most extensive variant is an AI system in which the model not only responds, but also uses tools. In such a scenario, the model can first call a specific function, API or automation, and only then – after receiving the result – generate the final response. The result of the tool also returns to the context and influences the further course of the task.

This is a key moment from the CISO’s perspective, because the AI system then ceases to be only a response layer and begins to have a real impact on data, processes and operations. The more tools, integrations and intermediate steps, the greater the risk surface.

This is why communication with the LLM model should not be reduced to a simple “question-answer” pattern. In practice, it is a flow in which context can be built from many sources, expanded with conversation history, fed with external data and supplemented with the results of tool operation. And if so, the security of the AI system must be assessed not only at the model level, but also at the entire architecture that creates this context.

What is prompt injection?

Prompt injection is an attack involving the introduction of a specially prepared instruction that induces the AI model to act in a manner inconsistent with its intended purpose. A malicious prompt does not have to “corrupt” the model itself – it just needs to influence how the model interprets the context and what actions it considers appropriate. This is why Prompt Injection attacks are today treated as the main attack surface in LLM-based systems.

This means that in the case of such an attack, the model receives not only the right question or task, but also an additional, malicious instruction that is supposed to change the way it works. If the system does not separate data from instructions well enough, the model may treat malicious content as a control instruction and respond or act contrary to the system designer’s intentions.

Effects of prompt injection

A prompt injection attack for an organization could mean:

disclosure of confidential information,
performing unauthorized actions,
manipulating the AI system’s responses,
loss of trust of customers and partners.

We wrote more about prompt injection, types and consequences in this article: http://secawa.com/en/blog/prompt-injection-what-it-is-and-how-to-limit-the-risk/

How to model threats in AI systems?

For CISO the key question is not only whether the system is resistant to prompt injection, but how to assess the risk separately for each layer of the AI system – from input and context, through RAG and tools, to permissions, output and operational control.

In practice, modeling threats in AI systems is worth dividing into 7 areas: inputs, context, RAG, tools and integrations, permissions, output and telemetry and control.

01 Input

This is the first and most obvious area of threat modeling. The following may be sent to the AI system:

data from the user,
files,
websites,
emails,
tickets,
repositories.

It is these entrances that the materials indicate as themain entry points for prompt injection. The problem is that the system does not always distinguish between regular content and malicious instructions. The more input sources and data formats there are, the greater the risk that a malicious prompt will be treated as part of the legitimate context.

02 Context

The second layer is the model context itself, i.e.:

system prompt,
memory
policies.

This is where attempts to hijack the system’s operating logic or reveal hidden instructions occur. From a security perspective, this is critical because context manipulation can lead to a change in the system’s operating logic or the disclosure of model control instructions.

03 RAG

The next layer is RAG, a mechanism for enriching the model context with documents and data from the knowledge base. From an architectural perspective, this includes, among others: vector database, retriever and ranker.

In this layer, the main risks are data extraction, context manipulation and data poisoning. If the system retrieves content from documents, web pages, emails or other external sources, indirect prompt injection can occur without direct user interaction with the attacker.

04 Tools and integrations

When a model has access to APIs, functions, automation or MCP servers, it stops being just a response generation layer and begins to influence processes, data and activities performed outside the LLM itself.

This is where the risk of unauthorized actions and misuse of tools comes into play. Manipulating the prompt, context or task flow is enough for the AI system to perform an operation it should not perform.

05 Permissions

This is one of the most important layers from a CISO perspective. The question is not only what the model has access to, but also whose permissions it operates under and what scope of access the connected tools have.

The key principle is simple: access control must operate beyond LLM, at the application level. If the system does not ensure this, the AI assistant may perform operations with a greater scope of permissions than it should, or launch actions to which the user would not normally have access.

06 Language model output

Threat modeling doesn’t end with input. Equally important is what the system returns next: responses, links, HTML, integrations and data passed to subsequent components.

This is important because the model’s response may be:

harmful in itself,
input for the next system,
carrier of data leakage,
the starting point for further activities in the automation chain.

Therefore, the model output must be treated as an element of the attack surface, not just the final result.

07 Telemetry and control

The last layer is operational supervision of the AI system – logs, limits, alerts, action approval and anomaly detection. This part does not block the attack itself, but gives the organization the ability to audit, detect abuse and respond quickly.

This is especially important where the system performs activities that are costly, sensitive or difficult to reverse. Without telemetry and control,an organization loses visibility into what the AI system did, how it behaved when executing commands, and whether there were any abuses, anomalies, or costly operational impacts.

7 questions a CISO should start assessing the risk of AI systems

The resilience of the AI system must be designed in many layers. Therefore, a good starting point is to go through the 7 layers of the AI system and assess the risk separately for each of them – from input data and context, through RAG and tools, to permissions, output and operational control.

What can enter the system?

The first question is not about the model, but about input channels. The AI system may receive not only user prompts, but also files, websites, emails and documents. This is where most successful attacks begin, because the model can treat untrusted data as an instruction. Therefore, the CISO should check which entries are filtered, whether entry screening works, whether prompt injection patterns are detected and whether the organization responds to repeated user abuse.

What goes into the model context?

The second question is: what exactly does the model see as context. Good practice is not about a “stronger system prompt”, but about separating data from instructions, clearly labeling what is a system command and what is user or document content, and minimizing the context provided to what is really needed to perform the task. The CISO should also assess whether the context contains secrets, sensitive configurations or unnecessary information that increases the risk of leakage.

What sources of knowledge enrich the answer?

With RAG, the key question is not only what the system uses, but also how it trusts these sources. Documents, web pages, PDFs, emails and comments may contain malicious instructions, so external sources must be treated as untrusted and RAG content filtered and flagged before being included in the context. From the CISO’s perspective, it is also important whether the system maintains control over permissions to source data, whether it segments repositories and whether it detects mass data extraction attempts.

What tools can the model invoke?

When an AI system uses APIs, functions or automation, a wrong answer can quickly become a costly operational failure. Therefore, the CISO should assess what tools the model can run, whether their use is validated before performing the action, whether the output returned by the tools is controlled and whether the model cannot independently skip required process steps. In practice, the security of this layer is based on validating calls, controlling the sequence of actions and limiting the scope of integration to the minimum necessary for a given task.

Whose permissions does the system run on?

This question often determines the scale of the incident. The CISO should determine whether the agent operates in the context of a user, application, or privileged backend, and whether authorization occursoutside the LLMat the application level. Good practices here are clear: all operations should be performed with user, not system, privileges, the scope of tools should be minimal, and the user, agent and backend identities should be separated. Without this, prompt injection may turn into privilege escalation and unauthorized operations.

What comes out of the model and where does it go next?

The model output is not just a response to the user. It is also a potential source of data leakage, another step of automation or an entrance for another system. Therefore, the CISO should verify that the organization controls model responses, detects attempts to disclose data in the output, limits the length of responses and the number of steps, and does not include secrets in prompts and context. In practice, quickly limiting the effects of exfiltration and cost abuse is as important as blocking an attack.

How does the organization log, control and test it?

The last question is whether the organization hasreal supervision over the AI system after implementation. Checking once is not enough. Effective supervision should include system telemetry, alerts, logs, agent behavior monitoring, cost and token control, and regular testing of resistance to prompt injection and RAG attacks. It is also important to measure not only the effectiveness of blocking, but also the quality of the response – so that after implementing security measures, the system continues to operate correctly, accurately and predictably.

Summary

The security of AI systems cannot be assessed by a single test or a single defense prompt. In AI systems, risk arises throughout the entire flow – from input data, through context and RAG, to tools, permissions, output and the telemetry and control layer. Therefore, it is worth conducting risk assessment layer by layer, and not only at the level of the model itself.

From the CISO’s perspective, the most important conclusion is simple: there is no single, fully effective method of defense against prompt injection today. But this does not mean that organizations are helpless.

The resilience of AI systems must be designed in many layers – combining entry control, secure context building, limited trust in sources, validation beyond LLM, permissions control, monitoring and regular testing of resistance to real attack scenarios. Just as important as blocking an attack is limiting its effects and quickly detecting violations.

If you want to verify how your environment will handle attack scenarios in practice, it is worth basing it on controlled tests. At SECAWA, we conduct professional penetration tests, which help discover system vulnerabilities and then turn the results into specific recommendations for corrective actions. This will help prepare the organization for real cyberattacks and better protect data, reputation and business continuity.

Test your system security with controlled cyberattacks

Discover our penetration testing offer

AI is entering business processes faster than some organizations have built hard boundaries for it. Language models don’t just answer questions. They increasingly read documents, browse the web, summarize threads in instant messengers and launch actions through integrations (e.g. MCP – Model Context Protocol). This creates a new, semantic attack surface: prompt injection.

Prompt injection is a situation in which malicious instructions enter the LLM context – sometimes directly from the user (direct prompt injection), and in other cases hidden in external data that the system processes (indirect prompt injection). The effect can be simple, but expensive: the model does something it shouldn’t – it reveals information, manipulates the response or initiates actions that the architecture and permissions granted to it allow (e.g. via API/integration) – even though the user did not want it.

OWASP classifies prompt injection as the highest-ranked risk for LLM and GenAI applications. He also emphasizes something that is inconvenient, but crucial for CISO: due to the nature of generative AI There is no method today that provides a 100% guarantee of eliminating prompt injection – but the risk and effects can be significantly reduced.

In this article, we answer questions that really influence security decisions:

What exactly is prompt injection and how does it differ fromjailbreaking?
What are thetypes ofprompt injection attacks?
How, as a CISO, can build protection against prompt injection – in security policies, environmental architecture and technical controls?

What is prompt injection? Definition

Prompt injection is a vulnerability in which content passed to the model (user prompt or external data attached to the context)changes the behavior or result of LLM in an unintended manner. It may lead to violation of system operation rules, generation of undesirable content, disclosure of information, gaining unauthorized access or initiating actions in connected systems – depending on the architecture and permissions.

Prompt injection occurs when input (regardless of form) influences the model so that it performs actions or generates responses that are contrary to the intended function of the system and trusted instructions (e.g. roles and constraints defined in the system/developer prompt). Importantly, a malicious command does not have to be human-readable – it just needs to go to the model context and be processed by it. Therefore, prompt injection applies not only to chats, but also to GenAI agents and applications that retrieve context from documents, websites, emails or repositories (e.g. in RAG architectures) – however, the use of RAG or fine-tuning alone does not eliminate this vulnerability.

In practice, prompt injection takes advantage of the fact that the model processes instructions and data in the same input stream (usually as text, and in multimodal systems also through other modalities), so untrusted content attached to the context (page fragment, search result, ticket, mail, document) may contain a command that the model will treat as an instruction. The more agency the system has (integrations, actions, permissions), the greater the potential impact.

Difference between prompt injection and jailbreaking

The two concepts are related and are used interchangeably, but they are not the same thing:

Prompt injection is a superordinate category: the attack involves manipulating the response or behavior of the model through specially crafted input data. The goal may be to change the content of the response, force disclosure of information, or influence what actions the application takes.
Jailbreaking is a special case of prompt injection where the goal is to make the model ignore the security mechanisms and constraints imposed by the system and start doing things it normally shouldn’t.

In short: every jailbreak is a form of prompt injection, but not every prompt injection is a jailbreak. In the context of enterprises and agents, what is particularly dangerous is that by injecting malicious prompts, the system may be induced to perform unauthorized actions or exfiltrate data – and the scale of the effects depends on the granted permissions and integration.

Types of prompt injection

Direct prompt injection

Direct prompt injection occurs when a malicious command hits the model directly in the content of the user’s prompt. This input could be:

intentional – when the attacker consciously constructs a prompt to change the behavior of the model,
unintentional – when an ordinary user types something that was not intended to be a “command” and yet produces an undesirable effect.

The mechanics are simple: input provides content that the model incorrectly treats as an instruction, which can lead to bypassing constraints or performing actions that are contrary to the system’s intended function and trusted instructions and constraints.

Indirect prompt injection

Indirect prompt injection is more difficult to detect because malicious instructions do not come directly from the user. The attacker places them inexternal sourcesthat the GenAI system can process: websites, documents, emails, databases, metadata or even graphic files. Such injections can also be unintentional – if the external content contains instructions that the model will treat as commands.

This is where the most insidious element comes in: the user often cannot see the attacker’s prompt, and the tool can appear normal while executing hidden instructions “in the background”. Malicious instructions can be hidden, for example, in metadata, in invisible Unicode characters, or in formatting that is invisible to humans but readable by the model.

Indirect prompt injection particularly escalates the risk when:

the application retrieves the context from outside (e.g. by “reading” the content),
the model has access to tools and integrations (can take actions),
processed content comes from a wide, difficult to control surface (WWW, emails, instant messengers, repositories).

In practice, this means that “data for analysis” also becomes a potential carrier of commands – also when these commands are not given directly in the text, but embedded in graphic material or another input channel.

Summary of the differences between direct prompt injection and indirect prompt injection

	Direct prompt injection	Indirect prompt injection
Instruction source	User input going directly to the model.	Instructions embedded in external content that the model retrieves and processes.
Visibility for the user	Present in prompts, sometimes visible, sometimes obfuscated, but still in the model’s input stream.	Often invisible (metadata, hidden text, obfuscation, multimedia elements).
Typical surface attack	Interfaces where the user provides a prompt (chat, form, command).	WWW, documents, emails, instant messaging, repositories, DB records, images.

Differences between direct prompt injection and indirect injection

When is the impact greatest?
In both cases – when the system has access to data and the ability to launch actions through tools/integrations and extensive permissions.

What effects can a prompt injection attack have on an organization?

Data exfiltration

Prompt injection may lead to the disclosure or exfiltration of sensitive data – both that which the model “sees” in the context of the conversation (e.g. content of processed documents, fragments of the knowledge base) and that which the application has access to through connected tools and integrations.

In practice, the risk covers data that the GenAI system can read or download during the task: email content, documents, information about customers and employees, financial data or other confidential records – depending on what sources are connected and what permissions are granted. If the application has operational channels (e.g. sending messages, publishing content, API calls), exfiltration can occur quickly, without additional exploitation of vulnerabilities in target systems, using legal permissions and integrations available to the application/agents.

Content manipulation and bad business decisions

Prompt injection can lead to response manipulation: the model generates distorted content – it omits important information, reinforces incorrect or biased conclusions, or suggests actions that are inconsistent with the intended purpose of the system and trusted constraints.

In an organization, this is particularly dangerous where the model’s response supports decisions – in risk analysis, recommendations for teams, purchasing, legal or HR processes. Even if there is no data leak, there may be real business losses resulting from decisions made on the basis of a manipulated response.

Disclosure of information about the system and system instructions (system prompt)

Prompt injection attacks often aim to extract informationthat helps the attacker refine the next steps: system instructions, how the assistant works, as well as what tools and integrations are available. This speeds up the iteration of the attack and increases the chances of successfully bypassing security measures in a specific architecture.

It is worth emphasizing this directly: these threats are classified as one of the highest risks in the OWASP Top 10 for LLM applications (2025). In practice, the result is that the model – and often the entire LLM application – does not have a hard, reliable boundary between trusted instructions (from the developer/system) and untrusted input content (from the user or from external data). Because everything goes to one context and is processed through the “same channel”, a properly crafted input can prompt the system to reveal the system prompt, operating logic or information about available functions – exactly those elements that facilitate further escalation of the attack.

Abuse of tools available to the agent

Prompt injection may prompt the agent to select and invoke tools that are in his arsenal but should not be used in a given context (e.g. reaching for the “get customer data” function during a simple email summary).

This does not necessarily mean a classic escalation of privileges at the IAM layer – more often it is a violation of the boundaries of the context and purpose of the task: the agent uses the privileges it already has in an unauthorized way relative to the intended system function and security policies.

Unauthorized actions and changes to systems

When the agent has integration and the ability to perform actions, abuse of tools may result in real changes in systems: sending a message, modifying or deleting data, updating records, launching a workflow.

Another problem is that such actions may look like “normal” operations performed by an authorized entity – only they were triggered by a manipulated context, rather than by the intended purpose of the task and trusted system instructions.

Reconnaissance and preparation of further attack

Prompt injection can be used as a reconnaissance stage: determining what data sources the system processes, what tools it has, what limitations it has, how it responds to specific commands and whether it requires confirmation of the action.

Such reconnaissance helps prepare more targeted attacks – especially when the attacker has the ability to place malicious instructions in content that the organization regularly processes (e.g. documents, emails, websites or records in systems).

How to protect your organization against prompt injection attacks?

There is no approach today that provides a 100% guarantee of eliminating prompt injection – this is due to the generative nature of AI and the fact that in practice LLM systems process both trusted instructions (system/developer) and untrusted content (from the user or external data) in one context. Reliably distinguishing “instructions” from “data” and detecting malicious intent is not something that can be assured deterministically under all conditions today.

This does not mean, however, that we are defenseless.

The organization can significantly reduce the probability of a successful attack and – often more importantly – limit its effects by building layered protection: from employee security awareness, through governance and AI use policies, to technical security, permissions control, adversarial tests and monitoring the behavior of models, agents and their interactions with tools..

Increasing threat awareness among employees

Prompt injection is a semantic type of manipulation – so in practice it hits where employees use GenAI to work with content: summarizing emails, analyzing documents, summarizing instant messaging threads or asking to perform tasks in tools.

If the user does not understand that the LLM system processes both instructions and data in one context, it is easy to engage in risky behavior: pasting untrusted content into tools, asking agents for too broad actions (“do what it takes”), launching integration without thinking about the consequences and what data and actions the agent has access to.

Therefore, cybersecurity education should include not only general cyber hygiene, but also specific rules for safe work with GenAI – including awareness of the risks related to external content, integrations and the scope of commands given to agents.

At SECAWA we can provide various forms of education for all roles in the organization and management levels:

e-learning in cybersecurity for companies (online lessons and refresher quizzes),
Practical Anti-Phishing Training – systematic, realistic simulations of attacks that translate knowledge into habits
cybersecurity training and workshops.

AI policies and governance

Prompt injection protection starts with rules that reduce the attack surface and limit the consequences when something goes wrong. In practice, there are four areas:

Visibility and control of GenAI (shadow AI) use

If an organization does not know what AI tools are used and for what (the Shadow AI phenomenon) – it is not able to manage the risk. Rules are needed that distinguish between permitted and non-permitted tools and define what data they can process and in what scenarios.

Principles of working with content sources

Indirect prompt injection is based on the fact that the model processes content from sources over which the organization does not have full control – or which may have been modified.

Therefore, it is worth determining what sources are considered trusted (allowlisting), when to use a precautionary approach and how to consistently treat content from the web, emails, documents, instant messengers or records in systems as untrusted input to the context.

Rules for formulating commands and scope of freedom of action

The more general the command and the greater the freedom of action, the easier it is to produce an undesirable effect. Governance AI should promote precise tasks, limited scope and clear criteria for what the model should do – and what it should not do (especially when working on external content and having access to tools).

Least privilege as default rule

If an agent has access to tools and data, they shouldonly receive the permissions necessary to perform a specific task – and nothing more. Conscious design of what an agent can see and what it can change or run in connected systems is key.

In practice, this means limiting access to sensitive resources, minimizing the scope of available functions, and clearly separating “read-only” operations from those that cause changes to systems. The fewer rights and fewer actions an agent has, the smaller the potential impact – even if a prompt injection occurs.

Technical inspections and monitoring

The technical layer has two goals: to make an effective attack more difficult and to quickly detect anomalies before they turn into an incident. The following actions are a practical synthesis of the key recommendations OWASP:

	Goal	What to implement	What to measure/log
Limiting model behavior	Reduce susceptibility to “shifting” the role and forcing actions out of scope.	1. Precise system/developer prompt: role, scope, prohibitions, terms of use of tools. 2. The “task bounding” principle – the model is to implement only clearly defined types of tasks, without default initiative. 3. Clear rules for ignoring attempts to modify trusted instructions.	1. Model version + configuration version (e.g. hash/ID system prompt, tool policies). 2. Detection of override/jailbreak attempts (flags from the classifier/heuristics). 3. Out-of-scope response rate, denials and escalations to human.
Define and validate expected output format	Limit the possibility of unwanted content being “injected” into responses and enforce predictable results.	1. Hard response schemas (JSON/schema, field lists, report format). 2. Deterministic validation on the application side (parser + rules). 3. “Fail closed” rules – if the validation fails, do not perform the action, do not propagate the result further.	1. Percentage of non-compliance with the scheme, reasons for rejections, retry count. 2. Cases of “schema drift” after prompt/tool changes. 3. Correlation: Format incompatibility ↔ unusual tool use.
I/O Filtering and Control	Detect and block risky content (including hidden instructions) and limit disclosures.	1. Input and output filters (rules + classifiers) for categories of sensitive and typical prompt injection signals. 2. Disclosure control: redaction of secrets/PII, blocking “prompt leakage”. 3. For RAG: assessment of response quality (context relevance, grounding in sources, Q/A consistency) and rejections when the model drifts away.	1. Filter result: allow/block + categories + confidence. 2. RAG telemetry: which documents entered the context, scoring, rejections, source conflicts. 3. Redaction cases (what was redacted and why – without logging the secrets themselves).
Permission control and the principle of minimum permissions	Even if prompt injection is successful, limit the possible impact.	1. Minimum scope of access to data and functions – per task and per agent. 2. Separating operations that do not change state from those that change state (and the latter – as narrowly as possible). 3. Tokens/credentials on the application side, not “in the model”; the model should not “have” secrets.	1. What permissions are active for a given session/task (policy ID). 2. Each use of the tool: who initiated it, what was the purpose of the task, what were the parameters (with redaction of sensitive values). 3. Attempts to use functions outside the permitted scope (policy violations)
Human confirmation for high-risk actions	Block unauthorized actions that change the state of systems or reveal data.	1. “Human-in-the-loop” for mutating and sensitive operations (sends, record modifications, deletions, publications, transfers, permission changes). 2. Verification of the context of the action: what was the basis for the decision, what sources provided the instructions.	1. Confirmation events: request → approve/deny, response time, action summary content. 2. Percentage of actions stopped by user or rules. 3. The most common types of actions requiring confirmation (for tuning policies).
Separation and marking external content as untrusted	Limit the influence of external data on trusted instructions and decisions about the use of tools.	1. Explicit “trust boundaries” in context: explicit separation of external content from system instructions. 2. Source policy: allowlisting, limiting domains/file types, precautionary principle for WWW and unverified content. 3. Content normalization/cleaning (e.g. removing hidden elements, metadata where it makes sense).	1. Context origin: sources, data types, acquisition path (Web/document/email etc.). 2. “Untrusted content” flags and detection of hidden/obfuscated elements. 3. The share of external content in context (volume, number of sources) vs. incidents/anomalies.
Adversarial testing and attack simulations	Detect vulnerable paths before the attacker does – and maintain resilience after changes.	1. Regular direct and indirect prompt injection tests on real flows (RAG, emails, documents, WWW, tools). 2. Action scenarios: whether the agent can be tricked into unauthorized actions, disclosures, use of tools out of context. 3. Regressions after each change: prompts, tools, integrations, policies, data sources.	1. Test coverage (scenarios, source types, tool types), pass/fail results, bypass vectors. 2. Vulnerability trends over time (is it better/worse after implementations). 3. “Top failure modes” – the most common ways in which the model/agent fails the task.

Source: LLM01:2025 Prompt Injection OWASP

Prompt injection is a real attack vector against GenAI applications and agents based on language models. The risk increases wherever the system processes external context and has access to tools, integrations and sensitive data. Since there is no approach today that provides a 100% guarantee of eliminating prompt injection, a layered approach is crucial: raising awareness and rules for safe work with GenAI, AI policies and governance that limit the attack surface, and technical controls, tests and monitoring that reduce the effects and shorten the detection time.

Generative artificial intelligence (GenAI) is no longer just a tool for development and innovation—it is now also fueling increasingly sophisticated social engineering attacks. Instead of manual campaigns based on templates and guesswork, cybercriminals now have technology that automates the entire fraud process – from reconnaissance to message personalization, and recognizing a social engineering attack has become much more difficult

How does GenAI support the most popular social engineering attack methods?

Cybercriminals are actively using AI to generate phishing emails, construct fake websites and create deepfakes, accelerating the pace and scale of attacks. This means that social engineering attacks are no longer the domain of manual campaigns based on intuition – they are becoming automated, highly personalized and difficult to distinguish from real messages.

It is the art of persuasion (influencing people), the aim of which is to obtain confidential information or sensitive data through emotional blackmail and manipulation. Cybercriminals encourage employees to take rash actions that violate the organization’s cybersecurity through various methods of gaining trust. Thanks to this, they conduct successful social engineering attacks, which we later read about in many industry reports and LinkedIn posts.

How is this possible? GenAI enables automation and scaling of social engineering methods in a way never before possible:

Content created by AI is free from typical linguistic errors that used to allow for quick recognition of a social engineering attack. Instead of broken Polish, we have natural syntax, correct punctuation, and adapted context.
GenAI cangenerate a personalized phishing campaign in a few minutes. Something that previously took specialists up to several hours.
Attacks are becoming more and more hyper-personalized and context-specific, which increases the likelihood that the recipient will trust the fraudster, unknowingly disclosing data or performing dangerous actions.

In practice, this means that social engineering is no longer a game of chance, but becomes a sophisticated art of gaining power over minds –automated, scalable and increasingly difficult to detect. Thanks to GenAI tools, cybercriminals can create hyperpersonalized messages that target the specific emotional background of the recipient, strengthening manipulation and weakening vigilance.

The attack not only looks credible, but also effectively bypasses security filters without causing alarm to either the systems or the employee. This is a real threat to system security, which in fractions of a second can lead to data theft, taking over access or losing control over the organization’s resources.

15 key questions for CISOs:
Assess your company’s readiness for AI-powered attacks

Get a free list of questions

Examples of social engineering attacks supported by generative artificial intelligence

Thanks to GenAI, cybercriminals gain the ability to conduct social engineering campaigns on an unprecedented scale and at much lower operational costs. Classic methods such as phishing and vishing have been taken to a new level by AI. It is worth emphasizing that GenAI does not just create new types of attacks. Rather, it strengthens existing techniques, eliminating their existing weaknesses.

Hyperpersonalized AI-generated phishing

Traditional phishing has been based on simple, mass messages sent to random victims for years. However, thanks to GenAI, this model has undergone a real transformation. Today, phishing can be almost indistinguishable from real business communication, both in terms of language and context.

Modern AI models (powered by data from social media or company websites) cancreate personalized messages tailored to the role, language and communication style of a specific person. AI not only knows how to write, but also to whom, when and with what topic to increase the chances of a click.

More and more phishing campaigns resemble “conversational scams”. Instead of a pushy tone, we have a friendly question, and instead of a fake invoice – a subtle reference to the company’s current activities. The attack begins with gaining trust and ends with extorting data or encouraging an unauthorized transfer.

Business Email Compromise (BEC)

One of the most dangerous variants of AI-supported phishing is BEC – fraud based on impersonating directors, managers or business partners. It is a social engineering attack using manipulation based on authority and time pressure. AI helps you write a persuasive message that mimics a specific leader’s communication style by analyzing their public statements, emails or posts.

Spear phishing

Spear phishing is a cyberattack that targets a specific person or group. Thanks to GenAI, cybercriminals can automate the creation of such an attack, for example using historical information about the victim. As a result, a message is created that perfectly hits the topic, context and at the right moment, e.g. a question about a project that a given person is actually involved in, or a request for a comment on an event that has just taken place in the company.

Quishing (QR code phishing)

The popularity of QR codes is growing, e.g. in marketing, payments and authorization. And cybercriminals have found a way to use this format for their own purposes.

Quishing, or QR code-based phishing, is a technique that involves embedding malicious QR codes in emails, posters, or physical objects to direct the user to a fake website once scanned. GenAI enables you to quickly generate thousands of variants of quishing campaigns and quickly design malicious forms and pages to look like your company’s.

Such a social engineering attack bypasses email filters, the malicious link is literally hidden in the image. This means that many classic filtering solutions do not detect the threat, and employees scan the code with full confidence.

Free quishing lesson for your team

Send this graphic to your employees. Those who scan the QR code will receive a lesson about quishing, in which we remind you of key safety rules!

Smishing

Smishing is a social engineering attack carried out via SMS messages, usually containing a link to a fake login page or a request for immediate action, which may, for example, compromise the security of an IT system or steal confidential information (e.g. “Your account will be blocked. Confirm your details”).

Smishing is increasingly an element of hackers’ activities as part of multi-channel attacks: a cybercriminal first sends a phishing email and then “reminds” you to perform the action via SMS, increasing the credibility of the attack and the time pressure. AI helps generate such large-scale coordinated campaigns in less time.

Remind your employees that if they suspect that an SMS may be smishing, they should not click on the link and it is worth reporting the incident to CERT Polska via the form at incydent.cert.pl or by sending a suspicious SMS to 8080.

Cybersecurity Team: How to recognize smishing?

Get a free guide

Telephone attacks (vishing) using voice cloning

Vishing (voice phishing) are social engineering attacks conducted through a telephone conversation, the aim of which is to manipulate the victim into taking a specific action (while ignoring the security aspect) – e.g. resetting a password, transferring funds or sharing sensitive data.

Voice cloning is an AI technology that allows you to recreate someone’s voice based on a short recording – e.g. from a webinar, YouTube or a company podcast. Just 30 seconds is enough for the system to learn the intonation, tempo, tone and typical phrases of a given person.

The combination of vishing and voice cloning gives cybercriminals an extremely dangerous tool. They can:

call with a spoofed number (e.g. “company president”),
speak in a convincing, familiar voice
apply time pressure (“I’m traveling, I need help immediately”),
and force an action, e.g. account reset, transfer, access to systems, disclosure of confidential information.

It is worth remembering that these attacks are often part of a multi-channel campaign (similar to smishing). For example, an email or SMS is sent in advance announcing the conversation. This combination drastically increases the credibility of the attack.

AI-generated deepfake video

Deepfake is a technology that allows you tocreate fake video or audio recordings that look and sound like reals. Thanks to GenAI, a short fragment of a recording is enough to generate a realistic video of someone saying or doing something they have never said or done. For example, we see the president on the screen and in the background we hear his voice (cloned thanks to AI), persuading us to take quick action: approving a transfer, changing access, or installing software.

Cyber criminals use this technique to:

impersonating organizational leaders in video messages,
strengthening BEC and spear phishing attacks,
increasing pressure and authority in social engineering campaigns.

In the era of online meetings and remote work, the video form has gained considerable credibility. If a message appears in “internal” Slack, MS Teams or a company channel, few employees will question it.

Reliable AI profiles (fake persona)

One of the most dangerous methods of social engineering attacks powered by GenAI is the creation of false identities online. Thanks to AI, cybercriminals can instantly generate:

realistic profile photos,
false professional CVs (e.g. “Security Manager in a large financial company”),
industry posts and interactions that build the image of an expert.

Such profiles are used, among others, to establish relationships on LinkedIn with company employees for the purpose of subsequent spear phishing or reconnaissance, spreading disinformation, gaining access to closed groups or communication channels (e.g. Slack, Teams, Discord).

AI enables not only the rapid generation of content, but also its distribution – e.g. by publishing comments, sharing articles, reacting to posts. In this way, the “person” looks credible, builds relationships and gains trust.

Tip: sensitize employees to be cautious about accepting invitations and contacts on professional social media, especially from people without a mutual relationship, with a very general profile or a “too perfect” photo.

How to recognize social engineering attacks powered by GenAI – 3 tips for your team

Modern phishing campaigns, powered by GenAI, are devoid of old, obvious errors. However, even the most polished email has its red flags. Below are the three most important warning signs your team should know, and at the end a free guide for your employees.

Be vigilant towards links (especially those that hide their purpose)

Malicious links remain one of the main vectors of phishing – but today they are masked much more effectively than before. Cybercriminals use external link shorteners (e.g. bit.ly, tinyurl), hyperlinks embedded in the text (“click here”), visually valid but fake domains (e.g. amaz0n.co vs amazon.com).

Good security practices to share with your team:

Hover your mouse without clicking – the real address will appear in the lower left corner of the screen.
Avoid shortened links – they hide the real purpose.
Look for typos and strange domains – e.g. amaz0n.com instead of amazon.com.

Check attachments before opening

Many phishing campaigns are based on documents that are appropriately named (“Invoice_12_2024.doc”), contain macros that run malicious code, and are locked in password-protected archives to bypass scanners.

What to watch out for:

ZIP/RAR archives with the password provided in the message body.
Documents with macros: .docm, .xlsm, old .doc/.xls.
Files .exe, .scr, .bat and double extensions: e.g. “invoice.pdf.exe”.

Rule: You should only open attachments from absolutely trusted sources and should scan them first.

Language errors 2.0

AI-generated content is grammatically correct, free from typos, formal tone and artificial precision. This means that the focus needs to shift from “classic mistakes” to subtle social engineering signals.

What to look out for:

messages with too many polite forms or formal phrases without personalization (“Dear Customer”)
strange constructions resulting from machine translation (“please complete the access confirmation procedure”),
a mixture of languages or borrowed words (“thank you for payment”).

Conclusion: perfect language is NOT a guarantee of authenticity. GenAI can generate content that meets all validity rules but is still fake. A holistic approach is needed: analysis of the language, context, recipient, message structure and metadata.

Cybersecure Team:
8 tips for your employees on how to recognize phishing

Get a free guide

How to effectively educate employees? Cybersecurity training vs. Practical Anti-Phishing Training

Phishing is the largest and most costly cyber threat today. In 2024, it was responsible for 39% of all incidents reported to CERT Polska, and the number of attacks exceeded 600,000. According to ENISA, as many as 60% of cyberattacks start with phishing, and the average cost of a data breach is USD 4.88 million (IBM, 2024).

No technical security measures guarantee 100% protection. In critical situations, people remain the last line of defense. Their decisions – click or report? – determine the success of the attack.

Why may traditional cybersecurity training be insufficient?

Gartner research shows that 69% of employees intentionally ignore security policies, and 93% know that their actions are risky – but they still repeat the same mistakes.

This shows that classic training is usually unable to change everyday habits. Why?

They are too general: they do not take into account the specificity of the industry, roles in the organization or typical company scenarios.
They are based on the transfer of theory: there is no learning in practice, so knowledge quickly evaporates.
Detached from reality: not updated as often as cybercriminals’ methods are updated.
They do not teach defensive behavior: they do not show how to react to a real threat in a dynamic situation.

Practical Anti-Phishing Training – a method of education that shapes safe habits

When no technical tool can guarantee complete protection, an organization’s cyber resilience depends on how its employees behave. It is not the level of security awareness that should determine readiness for cyber threats, but the ability to respond appropriately in practice.

That’s why Practical Anti-Phishing Training was created not as another e-learning or cybersecurity training to be checked off, but as a systematic training process that shapes real defensive reflexes. Instead of a one-time action, we provide a continuous, automated educational program conducted by SECAWA experts – we deal with everything: from preparing phishing simulations to analyzing the results. Designated people in the company (usually the CISO) have access to the proprietary training platform, which provides transparent reports and shows the real progress of the team in the form of accurate but readable statistics.

Why is it worth implementing Practical Anti-Phishing Training in your organization?

Below you will find a comparison table Practical Anti-Phishing Training with traditional cybersecurity training for companies. If you want to learn about SECAWA Practical Anti-Phishing Training and test our original training platform without costs and obligations – make an appointment at Free Phishing Test!

97% of organizations that experienced an AI incident did not have adequate access controls in place (IBM, 2025). In addition, an average of 13% of organizations reported breaches that directly affected their AI models or applications.

Since GenAI is available at your fingertips, those who can use it have the advantage – not always in good faith. It is largely up to people responsible for the organization’s security, especially CISO (Chief Information Security Officer), whether AI will become a loophole or a defense tool.

We will talk about how to transform AI from an uncontrolled risk into an element of your security strategy as part of a free series of webinars AI vs Cybersecurity – practical examples, ready-made solutions and Q&A sessions.

Phishing powered by GenAI

Phishing has remained one of the most common and effective attack vectors for years – it is based on social engineering techniques, including manipulation of emotions, trust and cognitive errors. Cybercriminals impersonate trusted people or institutions to extort data, funds or gain access to systems. Since Generative AI has become widely available, this technique has undergone significant evolution.

What is AI-generated phishing?

AI-generated phishing attacks use large language models (LLM) to generate natural-sounding, convincing and highly personalized phishing messages. Unlike traditional campaigns (often written in broken English and sent en masse), these messages are tailored to the context, industry and even the role of the recipient, and are often based on current projects and the organization’s internal communication style.

AI also enables more effective attacks such as Business Email Compromise (BEC) and the so-called CEO fraud. Generative models let criminals create credible messages in the local language, tailored to the company’s communication style. They impersonate people in high positions, exerting pressure – e.g. with an urgent request for an unusual transfer or disclosure of confidential data. GenAI lets criminals engage in a dynamic dialogue with the victim (answer questions, escalate the narrative, and even analyze previous communication threads). These attacks become even more credible and targeted.

This is why traditional security filters based on detecting known patterns (e.g. keywords) are becoming less and less effective against realistic and contextually relevant phishing attacks generated with the help of GenAI.

Cybersecure Team:
8 tips for your employees on how to recognize phishing

Get a free guide

Techniques that increase the credibility of cyberattacks: voice cloning, deepfake, multi-channel attacks

Phishing powered by GenAI is no longer limited to the email inbox: today it also includes audio, video and coordinated communication across multiple channels.

Voice cloning

Using voice cloning tools, cybercriminals can create realistic voice recordings based on just a few seconds of a sample – e.g. from social media, webinars or video recordings. These synthetic voices are used, among others, in vishing (voice phishing) attacks, during which the attacker impersonates a member of the management board or financial director and applies pressure to get the victim to take action – e.g. transfer funds or provide credentials.

Listen to what voice cloning might sound like here.

Deepfake video

GenAI allows you to create realistic videos featuring famous people. Criminals generate materials in which celebrities, CEOs or influencers “in their own words” encourage a specific action. These attacks are particularly effective because they involve vision and hearing at the same time, increasing the credibility of the message.

Deepfakes are also used to attempt to infiltrate organizations during recruitment processes. A famous case was the situation at Arup, where the “candidate” for the position of engineer turned out to be a generated deepfake. Fortunately, the recruiter recognized the ruse, but if the attack had been successful, the fake employee could have gained access to company systems, customer data, and even introduced ransomware. We wrote more about this case at Arup on our LinkedInie.

According to the IBM Cost of a Data Breach 2025 report, 16% of all data breaches were related to the use of artificial intelligence, of which:

37% were AI-generated phishing
35% are incidents involving the use of deepfake or voice clone.

This means that over 5% of all global data breaches in 2025 were directly related to deepfakes or synthetic voice – and this number will undoubtedly increase.

Multi-channel attacks

GenAI makes conducting automated and coordinated phishing campaigns much easier and faster than before. Cybercriminals combine multiple communication channels – emails, text messages, phone calls, corporate messengers and social media – creating a coherent, consistent message. AI tools such as voice cloning, these attacks are hyper-realistic and put strong pressure on the recipient, which increases their effectiveness and makes it harder for security systems to detect the trick.

15 key questions for CISOs:
Assess your company’s readiness for AI-powered attacks

Get a free list of questions

Automatic victim reconnaissance that allows you to create hyperrealistic campaigns

One of the huge advantages that cybercriminals have gained thanks to generative AI is the automation of reconnaissance, i.e. the stage preceding the attack in which data about potential victims is collected.

In the past, such reconnaissance required tedious manual analysis. Today, thanks to the combination of LLM (Large Language Models) with OSINT tools, criminals can automatically search and analyze:

LinkedIn profiles
company publications, reports, conference presentations,
social media entries (Twitter/X, Facebook),
recruitment advertisements,
forum posts, code comments, and internal documents publicly disclosed.

For CISOs, this means the need for open-source intelligence (OSINT), i.e. analysis:

what information the company (and its employees) unknowingly shares publicly,
how data is secured,
Do risk analysis procedures take into account OSINT + AI-based attack vectors?

Other cyber threats in the era of GenAI: Shadow AI, Shadow Agents and prompt injection

As GenAI becomes more integrated into everyday business operations, new, complex attack vectors emerge that – without proper security procedures – can escape the CISO’s control.

Shadow AI

Shadow AI is a phenomenon of unauthorized use of artificial intelligence tools and models by employees – without the knowledge of the IT department or the consent of the CISO. A simple example: a sales employee who pastes customer data into a free AI tool to generate an offer. These types of activities carry a huge risk of privacy violations, loss of control over data and violation of regulatory provisions (e.g. GDPR).

According to IBM, as many as 20% of organizations experienced a Shadow AI-related data breach, and these incidents were on average $670,000 more expensive than attacks that did not involve this area. Customers’ personal data were most frequently compromised – in 65% of cases. They are the most tempting morsel for cybercriminals: they can be sold, used for extortion, or impersonated as victims in subsequent attacks. Worse yet, 63% of organizations have no AI management policies in place. Ask whether the organization has procedures in place to minimize the risk of Shadow AI. If not, now is the time to address it.

Shadow Agents

In the coming months, the problem of Shadow AI may give way to an even more serious phenomenon – Shadow Agents. Google Cloud predicts that employees will increasingly implement autonomous AI agents that make decisions and perform business tasks without human intervention. Sounds efficient? Yes – but also dangerous.

Such agents can operate outside the organization’s control, processing customer, project and infrastructure data. This is a huge threat to which cybersecurity resilience must be built. It is worth noting here that a ban on the use of AI agents is a path to nowhere. Employees who want to improve their work can find a way to use them outside the corporate environment.

Instead, organizations should implement Agentic Identity Management – a new IAM model that treats AI agents as digital users with their own identity.

Access should be granted on a temporary, just-in-time basis and in accordance with the principle of least privilege.
AI agents should only have access when they really need it – for specific tasks and for specific periods of time.
The system should constantly assess whether such access still makes sense, adapt it to the situation, and allow for a clear indication of who is responsible for what.

This approach will reduce the risk of abuse and the so-called privilege creep, as agents accumulate more and more privileges that they no longer need – often without the organization’s control.

Prompt injection

Prompt injection is one of the newest threats to organizations using AI. The attack consists in sneaking a malicious command into the prompt (text instruction), which the AI model will treat as its own command, ignoring security measures, user intentions or operating principles. As a result, the model may, for example, reveal confidential data, bypass access control, perform unauthorized actions or harm the organization in a way that is difficult to detect.

In practice, it may look like a regular email with a link to an AI chat, where an innocent-sounding question contains hidden code that instructs the model to reveal sensitive information.

Google Cloud Security warns that attacks of this type will increase rapidly in 2026, along with the increasing availability of AI models and their integration with business systems. Prompt injection has a low entry barrier and high effectiveness, which is why it is an increasingly attractive tool for mass data leakage or silent sabotage.

Although phishing remains one of the most common cyber threats, it is increasingly one of the elements of more complex campaigns supported by artificial intelligence. Cybercriminals today have a whole range of tools: from voice cloning and deepfakes, through autonomous agents, to prompt injection attacks.

This requires changes in the organization’s security strategy: classic filters and procedures are no longer sufficient, especially when threats originate inside the organization, as in the case of Shadow AI.

That’s why in the second and third webinars of the AI vs Cybersecurity’ series, we will focus on these less obvious, but critical attack vectors. Participants:

they will see how real Shadow AI incidents took place and what manipulation of language models can look like,
will receive specific rules and processes that allow them to use AI safely,
will learn how to design an environment resistant to abuse of AI agents and LLMs,
they will receive ready-made defense scenarios that actually minimize the risk of sabotage and data leakage.

Free series of webinars for CISOs:
How to incorporate AI into the organization’s security strategy?

I'm reserving my place!

Recommendations for CISOs – how to build the organization’s resistance to GenAI-powered attacks

Practical employee education and regular testing of the organization’s resilience

Even the best-designed infrastructure will not stop an employee who thoughtlessly clicks on a malicious link. That is why it is so important toincrease employee security awareness and develop lasting cyber habitsthat will protect both themselves and the entire organization.

Instead of theoretical cybersecurity training, it is worth considering Practical Anti-Phishing Training, based on realistic simulations of cyberattacks. Employees receive messages tailored to their role, industry and context – exactly what they may encounter in their everyday work. If someone falls for the trick, they immediately receive a short micro-training that explains why the message was dangerous, what the consequences could be, and how to recognize a similar threat in the future.

As a result, over time, the click rate for phishing drops significantly and the number of reported messages increases, which can be seen in the clear but accurate statistics presented in the dashboard of the SECAWA training platform and automatically generated reports.

You can test your team’s response to cyberattacks at no cost or obligation and while also testing our phishing simulation training platform. All you need to do is make an appointment at Free Phishing Test, in which we will send a realistic phishing simulation to a selected group of employees. During the Free Phishing Test, you will be able to observe the statistics in the platform’s dashboard, and at the end we will prepare a report with the test results for you. Click here to find out more!

Principle of limited permissions for people and AI systems

It is not enough to treat the LLM as a tool. We need a clear assignment of “identity” for each model, agent or plugin – along with the scope of its permissions. This means:

access only to specific data, applications and APIs needed to complete the task,
temporary permissions (limited to the moment of execution of the action),
registering activity: who activated what, when and for what purpose
function monitoring: what prompts, what answers, what operating logic.

Just as we limit access for humans, we must do the same for AI to minimize the risk of abuse, errors and even hijacking.

Set clear rules for using AI

Not every attempt to “make work easier” with AI is dangerous, but without clear rules, the company is exposed to data leaks. Therefore, as a CISO:

Set clear rules: which tools are allowed, what data cannot be used, in what cases AI can and cannot support.
Give an alternative: create a safe environment for working with AI.
Introduce control and monitoring: so you know when unauthorized use, data leaks or policy violations occur.

Protect AI models and agents from hijacking or manipulation

Those responsible for organizational security should secure AI models similarly to securing critical infrastructure – controlling input data, isolating environments, testing for vulnerabilities and protecting APIs, especially if the model has access to corporate data.

Strengthen your security foundations

New threat vectors do not mean that basic security principles are becoming secondary. On the contrary, an organization’scybersecurity strategy should combine several independent layers of protectionthat complement each other and compensate for gaps in single solutions. Technological security should be strengthened, security procedures improved and employee security awareness should be developed.

JadePuffer attack at a glance

What is ransomware?

JadePuffer – the first ransomware attack fully controlled by LLM

How the JadePuffer attack worked step by step

Phase 1 – input host (Langflow)

Phase 2 – actual target (MySQL production server and Alibaba Nacos)

Four pieces of evidence that the attack was directed by an LLM, not a human

Consequences for the organization – why this is a wake-up call for CISOs, CTOs, CSOs and CIOs

JadePuffer is not an isolated signal that language models can be turned against an organization

Free series of webinars: AI-supported attacks, Shadow AI, AI-prompt injection attacks and AI as CISO support. 4 meetings, additional materials and several SECAWA specialists who discussed the topic of AI in the context of cybersecurity in detail.

How to protect your organization against AI-driven ransomware?

Summary

How did Meta AI chat help take over Instagram accounts?

What was the vulnerability in the Meta AI chatbot?

What did the Meta AI chat attack look like step by step?

What does this attack teach about the security of AI agents?

Why is chatbot a new cyber attack surface?

Prompt injection and the security of AI systems. How to model threats?

How to protect your Instagram account from being taken over by hackers?

How to protect your organization against AI attacks?

Get access to recordings and materials from the free AI vs Cybersecurity webinar series

How does communication with LLM (Large Language Model) work?

Language model messages and context

Question-answer

Conversation

RAG

Use tools

What is prompt injection?

Effects of prompt injection

How to model threats in AI systems?

01 Input

02 Context

03 RAG

04 Tools and integrations

05 Permissions

06 Language model output

07 Telemetry and control

7 questions a CISO should start assessing the risk of AI systems

What can enter the system?

What goes into the model context?

What sources of knowledge enrich the answer?

What tools can the model invoke?

Whose permissions does the system run on?

What comes out of the model and where does it go next?

How does the organization log, control and test it?

Summary

Test your system security with controlled cyberattacks

What is prompt injection? Definition

Difference between prompt injection and jailbreaking

Types of prompt injection

Direct prompt injection

Indirect prompt injection

Summary of the differences between direct prompt injection and indirect prompt injection

What effects can a prompt injection attack have on an organization?

Data exfiltration

Content manipulation and bad business decisions

Disclosure of information about the system and system instructions (system prompt)

Abuse of tools available to the agent

Unauthorized actions and changes to systems

Reconnaissance and preparation of further attack

How to protect your organization against prompt injection attacks?

Increasing threat awareness among employees

AI policies and governance

Visibility and control of GenAI (shadow AI) use

Principles of working with content sources

Rules for formulating commands and scope of freedom of action

Least privilege as default rule

Technical inspections and monitoring

How does GenAI support the most popular social engineering attack methods?

15 key questions for CISOs: Assess your company’s readiness for AI-powered attacks

Examples of social engineering attacks supported by generative artificial intelligence

Hyperpersonalized AI-generated phishing

Business Email Compromise (BEC)

Spear phishing

Quishing (QR code phishing)

Free quishing lesson for your team

Smishing

Cybersecurity Team: How to recognize smishing?

Telephone attacks (vishing) using voice cloning

AI-generated deepfake video

15 key questions for CISOs:
Assess your company’s readiness for AI-powered attacks

Cybersecure Team:
8 tips for your employees on how to recognize phishing

Cybersecure Team:
8 tips for your employees on how to recognize phishing

15 key questions for CISOs:
Assess your company’s readiness for AI-powered attacks

Free series of webinars for CISOs:
How to incorporate AI into the organization’s security strategy?