LLM Red Teaming: A Complete Guide to Testing and Securing AI Applications

Your AI model can pass every internal safety test, but you can’t guarantee full safety when a real user tries to break it. A hacker will manage to manipulate the algorithm and bypass the preloaded instruction set. That would give the hacker access to important information as well as the opportunity to perform prohibited actions. That is why LLM red teaming becomes relevant.

Instead of just determining if the LLM can be jailbroken, security testers advance the attack by one step and check whether it is possible to actually break the trust boundary. The research on AI agent hijacking conducted by NIST has shown that the baseline attack success rate was at least 11%, while for red team attacks the rate was 81%.

This guide is going to cover the practical side of LLM red teaming. We are going to talk about prompt injection, RAG attacks, data exposure, misuse of agents and tools, attack chaining, and impact validation and remediation.

Key Takeaways

• LLM red teaming is verifying whether any AI attack can lead to real security consequences.
• Test all aspects of the AI application, including LLM, RAG, agents, tools, APIs, and access controls.
• RAG should be tested more for data poisoning, retrieval attacks, and cross-tenant data access.
• AI agents must have permission restrictions, as malicious outputs may initiate unauthorised actions.
• Remember to validate real impact, not whether the AI can be manipulated.
• Automated and manual testing should be combined to identify common and specific attack vectors.
• Retesting should ensure that the root cause is fixed, not that one attack payload was blocked.

What is LLM Red Teaming?

LLM red teaming is a security assessment conducted adversarially to determine if an LLM-powered application can be manipulated by an attacker. It also checks if the manipulation can fail security, privacy, or business of an organization.

Testing is not limited to the model. Red teamers study the full AI application, as well as the parts that are linked to it, including:

• LLM and prompts: Testing prompt injection, jailbreaks, and instruction manipulation.
• RAG pipeline: Examining the impact of poisoned and unapproved content on retrieval and model responses.
• Vector databases and embeddings: Testing retrieval controls, data isolation, and potential information exposure.
• Agents and tools: Evaluating if manipulation of the model may result in unauthorized API calls or business actions.
• Application layer: Examining authentication, authorization, and output validation of the AI system.

This broader perspective is important for Generative AI security. It can lead to an application vulnerability if a model weakness is exploited.

According to the OWASP GenAI Red Teaming Guide, assessing AI systems involves three categories. These are model evaluation, implementation testing, infrastructure assessment, and runtime behaviour analysis.

Difference Between LLM Red Teaming and Traditional Penetration Testing?

Unlike regular penetration testing, LLM red teaming goes beyond just testing applications for vulnerabilities. It involves testing the adversarial behaviour of the AI model, prompts, retrieval, agent decisions, and AI-specific trust boundaries. Traditional penetration testing is still significant to an AI application.

This broader risk landscape is addressed by OWASP’s 2025 Top 10 list for LLM and GenAI applications, such as prompt injection, disclosure of sensitive information, data and model poisoning, improper output handling, excessive agency, leakage of the system prompt, and weaknesses in vectors and embeddings.

What Are the Main Attack Surfaces in an LLM Application?

The main attack vectors are the LLM and prompt layer, RAG and knowledge layer, agent and tool layer, APIs, memory, identity controls, and external integrations.

Red teaming maps these components before they are exploited to gain insight into how untrusted input is inserted, how sensitive data is moved, and where the LLM can affect the behaviour of the application.

• LLM and Prompt Layer 

The LLM and prompt layer can be attacked by direct prompt injection, jailbreaks, instruction manipulation, and leakage of system prompts. Testers check interactions between system instructions, developer instructions, and user input and conversation history. 

They examine if lower-trust input can overrule trusted instructions or alter the behaviour of the model, and if the output of the model will have an impact on components downstream.

• RAG Pipeline

There are several ways in which the RAG pipelines can be attacked. These are poisoning of knowledge sources, manipulation of retrieval, weak access control, and unauthorised data entering the model context. Tests include document ingestion, chunking, embedding models, vector databases, similarity search, and retrieved context. Red teamers also test if tenant isolation and metadata filtering allow users to access documents they have no access to.

• AI Agent

An AI agent’s attack occurs when the output of the manipulated model affects tool selection, tool parameters, permissions, or external actions. Testers analyze integrated APIs, databases, email systems, cloud resources, and business tools to determine if the agent can execute actions that the user is not authorized to do. 

The application needs to be authorized as its own security boundary, and the LLM should never be the security boundary.

Want To See Real Security Improvements

Gain a comprehensive roadmap for securing your systems with the guidance of our expert cybersecurity professionals.

Download Case Study
⤓

How Does LLM Red Teaming Test an AI Application?

LLM red teaming is used to test the interactions between model behavior and prompts, RAG data, agents, tools, and application functions. You start by establishing how the application behaves normally, then introduce adversarial inputs and trace what happens when the model receives them.

How to Test Prompt Injection?

Prompt Injection testing is used to see if attacker-controlled content can be used to override the intended instructions, alter model behavior, or impact protected app functionality. According to OWASP, this is LLM01:2025 Prompt Injection. To test prompt injection, you need to follow a few things.

Know where to begin: understand the typical responses, allowable actions and instructions, and hierarchy. Now test the model’s performance on:

• System instructions
• Developer instructions
• User input
• Conversation history
• Retrieved or externally supplied content

The key is to see whether lower-trust content can influence higher-trust instructions or make the model perform an unintended task.

How Do Direct and Indirect Prompt Injection Differ?

Through direct injection, you can be able to check whether the model will obey the system or developer instructions, display limited information, or alter the purpose. If the application connects the LLM to tools, continue the test and check whether the manipulated response can influence a tool request. 

Indirect injection becomes more interesting in RAG and agentic applications. An attacker could embed code within a document, email, webpage, or knowledge-base record. The malicious instruction is then incorporated into the context of the LLM when it is used to retrieve that content.

Your objective is to determine whether the application treats that content as data or accidentally allows it to act as an instruction.

How Is RAG Security Tested During LLM Red Teaming?

LLM red teaming tests the RAG pipeline by checking whether poisoned content, retrieval weaknesses, or broken access controls can influence the information reaching the LLM. It supports LLM pentesting and follows the complete retrieval process:

• Document ingestion
• Chunking 
• Embedding model
• Vector database
• Similarity search
• Retrieved context
• LLM

Each stage gives you a different place to test.

How Do RAG Poisoning and Retrieval Attacks Work?

RAG poisoning allows attacker-controlled content to influence future retrievals, while retrieval attacks attempt to expose information that the user should not be able to access.

During testing, the LLM red teamers check for:

• Who can upload or modify documents
• Does the application automatically trust and index uploaded content
• Are metadata, ACLs, and tenant identifiers preserved during ingestion
• Can similarity search return another user’s documents
• Are metadata filters enforced before content enters the context window
• Can poisoned content influence the final response or an agent action

OWASP’s LLM08:2025 Vector and Embedding Weaknesses covers security problems involving vector generation, storage and retrieval, including unauthorised access and information leakage.

How Do AI Agents And Tool Permission Access Work?

An AI agent is tested by seeing if model outputs can be manipulated to affect the tools used and the parameters for the tools or external actions beyond the user’s permissions.

This is important when the agent can access:

• Computerized customer records and databases.
• Internal or external APIs
• Email and messaging systems.
• Cloud resources
• File systems
• Business applications

The attack path you want to investigate is to manipulate the model, select the tools, manipulate the parameters, and take privileged actions.

Imagine a support agent can use search_customer() and create_ticket(), while only an administrator should be able to use update_customer(). 
Only an administrator can call update_customer().

TOOLS_BY_ROLE = {
    "support": {"search_customer", "create_ticket"},
    "admin": {"search_customer", "create_ticket", "update_customer"}
}

def validate_tool_call(role, tool):
    return tool in TOOLS_BY_ROLE.get(role, set())

Next, attempt to get the support-level agent to execute update_customer().

If the LLM generates the request and the back end rejects it, the authorization boundary is functioning. If the application executes the action simply because the LLM requested it, you have allowed model output to influence a privileged security decision.

This is the concern that lies under LLM06:2025 Excessive Agency. Excessive functionality, permissions, or autonomy can lead to harmful model behaviour.

How Are Information Disclosure and Improper Output Handling Tested?

Information disclosure tests attempt to access data that the user shouldn’t be able to access, and output handling tests follow the model-generated information through the systems that process it next.

It starts with potential disclosure points such as: 

• System prompts and hidden instructions
• Internal RAG documents
• Customer and tenant information.
• Credentials and secrets
• Conversation memory
• API and database responses

System Prompt Leakage is classified as LLM07:2025, and the OWASP states that system prompts should not be used as a security measure or secret storage. Then it follows the model’s output via APIs, databases, parsers, browsers, and other parts of an application. Output from the LLM shouldn’t be taken for granted because it was generated by the LLM.

If the application passes generated content into a downstream component without validating its schema, values, encoding, and permissions, the output may become a source of common LLM vulnerabilities like: 

• XSS
• SSRF
• SQL injection
• Privilege escalation
• Remote code execution

This threat is classified as LLM05:2025 Improper Output Handling. The actual goal of LLM red teaming is to show that the model can be tricked.

What Is the Step-by-Step LLM Red Teaming Method?

A practical Large Language Model security methodology follows the entire attack surface of an application, from the first user input through to the last action that the AI system takes. You learn the architecture, find trust boundaries, create realistic scenarios, test the impact of those scenarios, and test the controls after remediation. 

AI Red Teaming encourages a cross-model evaluation, implementation testing, infrastructure assessment, and runtime behaviour analysis approach to red teaming, not just as a set of jailbreak prompts to use.

Step 1: How to Scope the AI Application?

First, identify all the components that can affect the behaviour of the model and receive model output. This provides you with technical limits when conducting the assessment.

Map components such as:

• LLM model and model version
• System, user and developer prompts
• Content filters and Guardrails.
• RAG pipeline and embedding model
• Vector database and document store
• Conversation memory
• Agents and tool-calling functions
• APIs, databases and external integrations
• The mechanisms used to authenticate and authorise

Step 2: How Do You Map the AI Attack Surface?

The next step is to trace how untrusted input moves through the system and where it crosses into trusted components. These trust boundaries may be more significant than the LLM itself.

Consider routes like:

• User input to system/developer instructions
• External document to ingestion to embedding to vector database
• Retrieved content to LLM context 

The output generated by the LLM goes through a parser to produce the application

Step 3: How to Create the Threat Model?

Create the threat model in accordance with the actual architecture of the application, and emphasize the attacks that are more dangerous.

• Public chatbot: Test prompt injection, jailbreaks, system prompt leakage, and sensitive information exposure. 
• RAG application: Test document poisoning, embedding security, similarity search, metadata filtering, and cross-tenant isolation. 
• AI agent: Test permissions, function calling, identity, memory, and external actions.

Step 4: How do you Conduct Adversarial Tests?

This step requires combining automated tests with manual attacks that exploit the application’s specific controls and business logic. Begin with standard behavior and then use variations like:

• Direct and indirect prompt injection
• Jailbreaking attacks and instruction hierarchy
• RAG poisoning and retrieval attacks
• Cross-user and cross-tenant retrieval
• System prompt leaks
• Tool choice and parameter manipulation
• Inadequate handling of LLM output

Automation helps to have coverage across different prompts and parameters. Manual tests help you to adjust your attack based on the reaction. The reason is that a vulnerability might be discovered only after the combination of all the elements.

Step 5: How Do You Chain the Vulnerabilities?

After identifying individual LLM vulnerabilities, see whether an attacker can chain them together and create a more damaging situation.

The attacker could:

• Introduce some malicious file into the RAG knowledge base
• Instruct the RAG system to retrieve the file
• Manipulate the LLM into using the contents to call a tool
• Abuse the permissions granted to the tool inappropriately
• Make an API call that they were not supposed to make

During each step, determine if there is any security measure in place that blocks the attack. The attack will fail if the LLM does not take malicious content into account.

Step 6: What Is the Impact of This Vulnerability?

You shouldn’t just prove that the AI model is susceptible to tampering. Show what a person can do with it.

For example, see if the vulnerability enables you to:

• Get access to another tenant’s data
• Invoke a restricted function
• Change a customer record
• Leak confidential information
• Access a restricted backend API

Step 7: How Would You Retest After Implementing the Mitigation Strategy?

During the retest, you should ensure the actual security issue has been fixed. Only fixing the specific attack method can’t solve the issue. If you have patched one prompt injection string, test for other similar strings that could still trigger the exploit. 

In case one malicious document is patched, test your document and trust policies to confirm that they can protect you from a similar problem in the future. Where you have patched a function that could cause privilege elevation, test to see whether another function would perform the same task.

Is Automated LLM Security Testing Sufficient?

Automated LLM security testing provides scalable and repeatable coverage, but human-led red teaming is still needed to uncover application-specific attack chains.

Automation can test large numbers of prompts, models, and configurations, making it useful for AI security testing after changes. But finding that a prompt injection works does not show its full impact. A human red teamer can track that manipulation further to determine if it impacts RAG retrieval, if it crosses a tenant boundary, if it impacts tool selection, or if it triggers a privileged API action. 

Similarly, the OWASP GenAI Red Teaming initiative is looking to test with various AI architectures instead of just jailbreak. Automation helps to bring scale while human-led red teaming gives individual weaknesses to realistic attack paths.

What Should a Red Teaming Report for an LLM contain?

An LLM red teaming report should include the vulnerability, entry point, affected component, attack path, evidence, impact, root cause, and remediation advice.

It should properly document:

• Finding: Name of the vulnerability, such as Indirect prompt injection.
• Entry Point: Mention clearly from where the attack originated.
• Affected Components: Mention systems affected in exploitation, such as the RAG pipeline and AI model.
• Attack path: How the attack moved through affected components
• Impact: Unauthorised customer-data access
• Root cause: Mention what the security weakness was that led to the attack.
• Remediation: Steps taken to prevent the attack.

The report should also differentiate between model behaviour and security impact. Do not automatically classify a successful jailbreak without any impact on a meaningful application in use as having a similar impact as an attack that breaches an authorisation boundary.

Qualysec’s Approach to LLM Red Teaming?

Qualysec offers a human-led, AI-Powered methodology to assess LLM applications for vulnerabilities in models, prompts, RAG pipelines, APIs, agents, and connected tools. The testing isn’t limited to just jailbreak attempts. 

The AI security testing goes beyond simple jailbreak attempts. We examine prompt injection, indirect prompt injection, sensitive data exposure, RAG poisoning, retrieval access controls, excessive agency, and unsafe tool usage. We also trace vulnerabilities across the application to determine whether a model-level weakness can lead to a real security impact.

The combination of AI-assisted testing and human testing allows for repeatable adversarial testing, and experienced testers can interpret and break application-specific behaviour, business logic, and attack chains that automated testing can be used to analyze. Findings are supported by repeatable evidence, impact analysis and remediation advice, and finally retesting to verify that the weakness has been remedied.

This way, organisations can understand the LLM and the application that surrounds it as a single attack surface instead of the LLM itself.

Conclusion

LLM red teaming is not only about identifying jailbreaks but about establishing if AI manipulations can be used as a real threat. The combination of prompt testing, RAG, agent, tool, and access control testing is what can help detect threats missed during traditional testing.

The combined use of automated and human testing allows for identifying the true impact, fixing vulnerabilities, and making AI applications more resilient.

Need Help Choosing the Right AI Cybersecurity Partner?

Compare security providers with confidence and identify the right partner to protect your applications, cloud, APIs, and AI systems before threats become breaches.

Schedule a Security Assessment
→

FAQs

1. What is LLM red teaming?

LLM red teaming is an adversarial security assessment. It checks if an attacker can successfully manipulate an AI application and make the manipulation translate to actual impact.

2. Is LLM red teaming the same as jailbreak testing?

No, jailbreak only tests circumvention of the model’s limitations. LLM red teaming tests RAG, agents, tools, APIs, data access, and attack chains.

3. Which security risks can be assessed by LLM red teaming?

The most common risks include prompt injection, disclosure of sensitive information, RAG poisoning, overagency, system prompt leaking, and output misuse. It also checks for vector or embedding risks.

4. Why is LLM red teaming especially significant in the case of AI agents?

AI agents have access to tools and can interact with external systems. So, the manipulation of the model may lead to actions that exceed users’ permissions. This is why LLM red teaming is important for AI agents.

5. Are automated tools enough for performing LLM red teaming?

No, only automated tools are not enough. The tools provide scalability and speed, yet LLM red teaming also needs to involve human effort. Human-led testing helps discover application-specific risks and build a chain of multiple risks.