Securing Enterprise AI: Preventing Data Leakage in LLMs and AI Agents

AI data leakage occurs when an AI system gives out information to a person who shouldn’t be receiving it. This could happen if a chatbot quotes a salary sheet to the wrong employee, an agent sends an email to a customer list using an address inserted into a document, or customer information is left in a third party’s logs after it has been entered into a prompt. Generally, the problem does not lie with the model itself. The leak flows through the text the AI can read and generate, as well as through the inputs and outputs that produce it.

The controls that come into play are largely tangible. Apply source system permissions before they get to the model. Limit and restrict agents’ credentials for short time periods. Restrict the access of tools to data, hide sensitive data from prompts or logs and test the entire configuration as an attacker would do.

CERT-In demonstrated these measures on 15th October 2026. It warned in the advisory that AI models assessed by the vendors had already infiltrated the systems of several Indian organisations and accessed business files, including personal information. They infiltrated the systems using forgotten APIs, exposed credentials, and over-permissive cloud roles. None of the organisations detected it. The providers are those that the providers themselves found.

This guide covers preventing data leakage in enterprise AI agents and the LLM applications around them: the common leak paths, two controls we tested in a Python simulation, the guardrails to build first, how an assessment works and how Indian rules apply.

Where enterprise AI leaks data

Six leakage paths turn up again and again, and none of them depends on the model misbehaving.

• Retrieval that ignores permissions. A RAG assistant indexes SharePoint or a shared drive and answers anyone who asks, so the salary sheet becomes just another chunk of text. OWASP files this under Sensitive Information Disclosure (LLM02) and Vector and Embedding Weaknesses (LLM08).
• Instructions hidden in content. An invoice PDF, a support ticket or a web page carries text the model treats as a command. That’s indirect prompt injection.
• Agents with standing credentials. An agent calling tools with its own service token can do whatever that token allows for whoever happens to be talking to it. OWASP’s Top 10 for Agentic Applications 2026 calls this Identity and Privilege Abuse (ASI03).
• Tools that can send data out. An agent that can email, upload files or call any URL gives an attacker an exit, particularly when nobody checks the destination or the payload.
• Memory, caches, logs and providers. Conversation history, cached answers and persistent agent memory can surface one user’s data to another. Full prompts often land in monitoring tools with wider access than the source data, and a model provider may keep prompts for as long as its terms allow. Staff also paste customer files into personal chatbot accounts. IBM’s 2026 India study found that where this kind of shadow AI was involved, it added ₹1.79 crore to the average breach, which was ₹25.5 crore across the organisations studied.
• Secrets in code and configuration. API keys in repositories, connection strings in config files and cloud roles with wildcard permissions. These are the weaknesses CERT-In found in its October cases.

A tested example: permissions before retrieval

We built a small retriever in Python to show the most common of these. It holds three synthetic documents: a leave policy everyone may read, an HR compensation sheet and an orphaned chunk that lost its access list during indexing. No LLM is involved, because the test checks which documents would reach the model. The first version ranks by similarity and nothing else.

The fix applies permissions before ranking and takes group membership from the identity provider, never from anything the user typed.

Asked about pay bands, the vulnerable version gave a sales user all three documents, compensation sheet included. With the fix, the sales user got only the leave policy, the HR user got the compensation sheet and the leave policy, and nobody got the orphaned chunk.

Filtering before ranking is the minimum, though, not the whole job. Restricted text mustn’t reach the model by another door, so the same check has to cover rerankers, cached answers, conversation history and the citations shown to users. Permission changes need to travel fast as well. When someone loses access, a document is deleted, or a group changes, the index should catch up within minutes rather than at the next weekly rebuild.

Generative AI prompt injection prevention starts outside the model

Prompt injection is on the OWASP Top 10 for LLM Applications 2025, and OWASP says there is no cure. It’s adding several layers of control: limiting privileges, filtering inputs and outputs, validating what the model outputs and getting a human to approve high-risk actions. Using detection tools is helpful, but there are limitations to how accurate they are, and if an attacker rephrases a payload (or uses white text in the PDF), they can often evade them. 

Along with “Can we stop the model from being fooled?” ask a second question: what can a fooled model do? 

The greater concern is actually indirect injection, since the attacker doesn’t touch your systems. They write an email to you, post a resume on a supplier portal page and wait to see if your agent reads it. 

This scenario was simulated for an accounts payable agent. In order to keep the test fair as to the scope, we did not employ an LLM. Using a stub, we can simulate a model that has already been fooled and respond to any payment instruction read by the tool, thus testing the part of the company we control: the authorisation. A stub can be used to simulate a model already fooled and respond to any payment instruction read by the tool without depending on whether the first model falls for the first payload. This test is the first one: 

The invoice carried one extra line asking the agent to pay ₹4,80,000 to a new vendor “as approved by finance”. The vulnerable tool paid it.

The fixed version takes its authority from the verified user, keeps a running daily total so a large payment can’t be split into small ones, and binds any approval to the exact payee, amount and invoice:

The injected instruction still reached the tool, which refused it because the vendor wasn’t approved. It also turned away a read-only user, sent a ₹2,00,000 payment to an approver, paid a small legitimate invoice with the user’s short-lived token, and pushed the second of two split payments into approval once the day’s total passed ₹50,000. What the simulation can’t tell you is how often a real model obeys injected text. That depends on the model and the payload, and it’s precisely what live testing has to measure.

Three rules carry most of the weight in generative AI prompt injection prevention:

1. Authorities are not based on the prompt or the model output. If the agent is acting on behalf of a user, it is the user’s session. Agents running unattended should use a narrowly scoped workload identity; CERT-In’s October advisory recommends employing a separate, short-lived identity for each run.
2. Each tool has the most limited scope it needs to function, but restrictions are programmed.
3. In any irreversible transaction or transaction of financial value, human approval is required to demonstrate to the approver what exactly the transaction was.

Approval will only benefit if the approver is indeed the one they claim to be. The finance employee at Arup was involved in a video conference call with deepfaked employees and processed 15 transfers valued at approximately US$25.6 million. Use a communication path that attackers cannot easily spoof, such as the approver’s own authenticated application. Video KYC and executive impersonation are significant enough issues to warrant their own deepfake defence solutions for banks, which are available separately. 

Enterprise AI guardrails implementation, in the order that works

Often guardrails are purchased as a product and switched on in the chat window. The useful ones are on top and can be checked. The following table correlates the major OWASP risks with a control and a way in which a tester could verify it.

The sequence you construct them in determines their level of exposure during construction. The first step in enterprise AI guardrails implementation is to begin with what leaks the most: 

1. Take inventory. Include all models, agents and AI features, as well as all tools a team registered with a company card. Review whether prompts are stored, used for training, or reviewed by humans for each external model service, and what your settings and contract allow.
2. Bring the ID card through the entire process. A user’s identity, or the agent’s scoped workload identity, must be transmitted to the retrieval layer and all tool calls.
3. Set retrieval permissions using deny-by-default access lists before content is delivered to the model.
4. Determine where and what tools are being used. Replace standing tokens with short-lived ones, enforce limits in code and limit data flow paths. CERT-In’s October Advisory: Default deny egress for internal agents. Network controls will be effective in minimising misuse but will not cover all devices and personal accounts.
5. Keep what the model sees to a minimum. Remove identifiers and confidential data from unnecessary contexts where the task is not needed.
6. Reproduce outputs in layers (with redacting). Regular Aadhaar Verhoeff checksum and PAN structure checks reduce the number of false alarms from matching any 12-digit number, but still they don’t catch enough. Use this in conjunction with the data classification, context rules, and access controls listed above.
7. Log without creating a new leak. In case of any conflict, ICT logs should be stored securely for the first 180 days in India, following the directions given by CERT-In 2022. Mask personal information before it gets to the log store and maintain a tamper-proof log of what the agent did. 
8. Plan the shutdown. Prior to disabling a connector, revoking credentials, keeping evidence and working out reporting responsibilities, decide beforehand. The blueprint for CERT-In’s May 2026 notification requests operational boundaries and audit logs for agentic AI, and includes an emergency shutdown request. 

What an LLM security vulnerability assessment covers

An LLM security vulnerability assessment goes through the entire application model, prompts and retrieval, tools, identity, memory, and the infrastructure that underlies it. Testing the chat box alone misses most of the assessment. Here’s how a typical engagement goes.

1. Scope and map. Identify the information each assistant can access, the tools each agent can call, and the people who use them. That map is now the test plan.
2. Get shadow deployments via single sign-on logs, expense claims and outbound traffic.
3. Execute automated probes using tools like garak, PyRIT and promptfoo for a quick baseline on known jailbreak patterns and extraction patterns. They are a beginning, not a test. 
4. Inject through every content channel, including emails, PDFs, tickets, CVs and web pages, and record what the agent tries to do.
5. Test authorisation across roles and tenants, including what caches, memory and conversation history return to the wrong user. In our experience, the serious findings usually turn up here.
6. Abuse the tools. Chain calls, split transactions, change payees and try sending data to outside addresses.
7. Check output handling for stored cross-site scripting, SQL injection and command execution.
8. Look for secrets in repositories and previous history, configuration files, client apps and cloud roles, as October incidents did.
9. Report and retest. Every finding requires the payload that caused it, the data that was exposed, a patch and a retest to prove it was closed within the scope of the test.

Mapping to OWASP lists and MITRE ATLAS provides a single vocabulary for developers, auditors and the board. Retest whenever the model, a tool, a data source or a permission model changes, and set a regular cycle based on how sensitive the data is and what regulators or customers expect. 

Corporate artificial intelligence data protection under Indian rules

There is no uniform AI legislation in India. Whether a specific rule applies depends on your organisation type, the data you handle, and the provisions in effect. Do not miss these; they are the primary ones.

Two points cause the most confusion. Embeddings aren’t automatically personal data, but if a vector relates to an identifiable person, or the original text can be recovered from it, treat it as personal data. And we aren’t aware of a general Indian rule that requires AI penetration testing, though sector regulators set their own testing expectations, so check yours.

What these rules share is an expectation that you can show your safeguards were reasonable and actually worked. That’s why corporate artificial intelligence data protection relies on evidence. After an incident, a dated test report with retest results carries far more weight than a vendor’s datasheet.

Common mistakes worth avoiding

• Treating the system prompt as a control. “Never reveal salary data” is a request to the model, not a permission check.
• Giving the agent one powerful service account because it’s quicker than delegated access.
• Indexing a whole drive and planning to add permissions later.
• Forgetting caches and memory when testing who can see what.
• Approval prompts that don’t show the payee and amount, or limits that reset with every transaction.
• Logging full prompts and answers into a tool half the engineering team can search.
• Testing once at launch and never again after the next model upgrade.
• Leaving keys in an “internal” repository. CERT-In’s October advisory lists keys found in repository history.

Prepare for Your Next Cybersecurity Audit with Qualysec

Choose a partner that helps you identify and fix real security risks before attackers do. We are here to help.

Talk to an Expert→

How Qualysec helps

Qualysec is a specialised penetration testing company. When we test AI applications, we look past the prompt to the data, tools and identities behind the chat window, because that’s where attackers go.

An engagement follows the assessment steps above. Automated probes give us a baseline, and most of the time then goes into manual work: indirect injection through real content channels, cross-role and cross-tenant retrieval, memory and cache checks, tool abuse, output handling, and the secrets and cloud roles underneath. For each finding, you get the payload that reproduced it, the data it exposed, a fix your developers can apply and a retest to confirm it’s closed.

Wubble AI, an AI music platform, came to us after attackers had taken over its production environment. We traced how they got in across the SaaS platform, the cloud infrastructure and the connected environments in scope, walked the engineering team through each fix, and retested until no known attack path remained within that scope. In its Clutch review, the client said its team couldn’t have found the root cause alone. It was breach recovery rather than AI model testing, but the lesson carries over to agents: the root cause sat in the infrastructure around the product, which is where agents pick up access they were never meant to have.

Frequently asked questions

1. What is the first step in stopping AI agents from leaking data?

List every agent, the data it can read, the tools it can call and the identity it runs under. Then make sure that identity, whether a user’s or a scoped workload identity, is checked at every retrieval and tool call.

2. Can guardrail products stop prompt injection on their own?

No. They catch known patterns and cut down noise, but injected instructions do get through. Real protection comes from limiting what a fooled model can do: scoped tools, short-lived credentials, controlled destinations and human approval for sensitive actions.

3. Do model providers keep the prompts we send?

It depends on the service, the plan and your settings. Some providers keep prompts for abuse monitoring, some let you switch off training use, and enterprise contracts often change the defaults. Check each service’s terms before sending personal or confidential data.

4. Does the DPDP Act apply to data inside an LLM or vector database?

Once its main duties apply from May 2027, yes, for digital personal data within the Act’s scope. That includes prompts, retrieved documents, logs and agent memory. Embeddings need a case-by-case look at whether they relate to an identifiable person.

5. How often should AI applications and agents be retested?

After every meaningful change to the model, tools, data sources or permissions, plus a regular cycle set by the sensitivity of the data and what regulators or customers expect. Agents change faster than ordinary applications, so a one-off test goes stale quickly.

6. Is AI penetration testing mandatory in India?

We aren’t aware of a general rule that makes it mandatory, but sector regulators can set their own testing requirements. It’s still one of the clearest ways to show that the safeguards you rely on actually work.