1. Introduction: Your Model Remembers More Than You Think
In this guide, AI data security means keeping the data that trains and feeds a model from leaking, either through the systems around the model or through the model’s own answers. Most programmes miss this second route. Models learn statistical patterns, but they can also memorise parts of their training data, especially rare or duplicated records, and a patient run of ordinary queries can bring some of this memorised data back out.
Many of the controls that do the majority of the work are well known. Do not be more detailed than the product requires. Reduce memorisation using regularisation or differential privacy. Minimise and duplicate training data, restrict access to the model, carefully safeguard the model checkpoints as well as the original training data, and test leakage using actual attacks before release.
To illustrate how easy it is to get it wrong, we trained a model that makes loan decisions for 600 synthetic applicants and then attacked it in the same way an outsider with API access might. With a threshold of only 0.5 (which is the same as guessing), simply looking at the model’s confidence in the right answer allowed it to clearly distinguish between the training set and the unseen set, achieving an AUC of 0.955. All the requests were legitimate, and there would have been nothing special in the logs.
There are also attacks on models now appearing in breach data! Of the 1,000 organisations surveyed, about one-fifth experienced an AI breach, with the most costly type of AI breach being model inversion, which averaged US$6.07 million, in IBM’s 2026 study.
We describe the attacks, how to test for them, worthwhile defences, and the rules. Prompt injection and AI agents were covered in our dedicated guide to securing enterprise AI.
2. Securing AI Data or Using AI for Security? Two Meanings Behind One Search Term
The same search term is used for two different positions. The first, which is the focus of this guide, is data protection within AI systems: training sets, feature stores, model weights, embeddings and predictions that a model returns.
The second uses machine learning to protect data elsewhere by classifying sensitive files, spotting unusual access and speeding up investigations. That’s AI in data security, a tooling decision with its own questions.
They come together at a single location. A classifier trained with personal data has been trained with personal data, which means that it has the following leakage risks. When you are purchasing data security AI for a security team, ask the vendor questions about the model it has been trained on and what it returns for callers, and incorporate that into your testing.
3. Where Machine Learning Data Security Breaks: The Lifecycle Attack Surface
Machine learning data security is more challenging than regular application security, in part because data-sensitive material changes form as it moves. Features turn to rows, rows turn to features, features turn to gradients, and gradients turn to weights. The more you move upstream in that chain, the less the personal information will be easy to see. Encryption and access control are important at all stages, but they do not prevent a deployed model from displaying what it has learned.
Five places hold most of the exposure.
- Training pipelines and feature stores. Data is loaded from source systems to a lake, a feature store and a training job. Each copy is as fragile as the original copy, but the original rules may not be retained.
- Checkpoints, experiment trackers and prediction logs. A checkpoint is intellectual property and, for an attacker, a much more solid base to attack from than an API since it enables white-box attacks. Sample rows are sometimes stored in trackers, and prediction logs contain all the input from users.
- Inference endpoints. Information is included in confidence scores, ranked classes, timing and error messages. The most common form of over-disclosure is a full probability vector returned in a product that displays only an “approved” or “declined” status.
- Embeddings and gradients. A vector of numbers looks like an anonymous vector, but an embedding can be inverted at times to retrieve text or traits, and gradients that are shared while training can leak the examples that will be embedded.
- Federated and partner training. The raw data remains with each party, but the updates that are shared are encrypted with a signal, and few contracts state who is responsible for what the combined model remembers.
Data security in AI requires testing that must traverse the whole data journey from source to prediction, as no single control addresses all five.
4. Model Inference Attacks: How Attackers Pull Data Out of a Trained Model
An inference attack is an attack in which a model learns a property it was not designed to reveal. AI security vulnerabilities will never be found in a code scan, but they can occur even if the predictions are correct. An attacker’s access determines what they can do. The black-box attackers can only receive scores or labels as API responses. The weights and a checkpoint (or training gradients) are the things that the white-box attackers have, and they get much further.
| Attack | Typical access | What the attacker learns |
|---|---|---|
| Membership inference | Black-box scores or labels, plus a candidate record | Whether that record was likely in the training set |
| Model inversion and reconstruction | Scores and many queries, or white-box access | Representative inputs or sensitive features |
| Attribute and property inference | Many queries | Traits of an individual or of the dataset as a whole |
| Training data extraction | Prompts to a generative model | Memorised text |
| Model extraction | High query volume | A working copy of the model |
Membership inference is a question of whether a given record was used in training. This is a probability, not a proof; it is accompanied by false positives, and it is only significant if the attacker has the correct record in his possession and can connect it to a person. In this case, a positive prediction is a significant revelation even if the training sample is sensitive, as in the case of borrowers who defaulted or patients with a diagnosis of a specific disease. Shokri and coworkers showed the attack using shadow models, while Yeom and coworkers associated it with overfitting.
The purpose of inversion in models is different. Rather than testing for one record, the attacker makes adjustments until the confidence is raised, which is because they try to figure out what a typical record in a class might look like, or they may guess at a sensitive feature.
Training data extraction impacts generative models. By prompting a large language model, Carlini and colleagues were able to extract the exact text that was used to train the model, including personal information. Fine-tuning on repeated lines in training is the easiest way to get them, and it is the safest time to fine-tune on the duplicated contracts or support tickets.
Model extraction copies the model itself. Tramèr and colleagues rebuilt working copies of commercially hosted models through their prediction APIs, and a copy lets an attacker test evasion tricks offline.
NIST’s AI 100-2e2025 taxonomy groups data reconstruction, membership inference and property inference as data privacy attacks and treats model extraction as a model privacy attack.
5. AI Data Security Risks Mapped to Business Impact
The AI data security risks to address first are those where being in the training data reveals something sensitive. These three cases are illustrations, not client stories.
- A lending model. Someone who submits a person’s details and reads the confidence score might infer, with some probability, that the person applied for credit, perhaps unsuccessfully. No name appears in any log, yet a sensitive inference has reached a third party.
- A health model. If a triage model is trained only on patients with one condition, a confident membership result points towards a diagnosis, which is special category data under the GDPR.
- A fraud model. An attacker sends a payments risk-scoring API thousands of queries, trains a copy and refines fraud patterns offline until they pass.
To decide where to start, score each model from one to three on the sensitivity of its training data, how widely it can be queried, and how much each response reveals, then multiply. A public health model returning probabilities scores 27, while an internal forecasting tool returning one number scores 1. It’s rough, but it gives security and data science teams one list to work from.
6. Why a Standard Penetration Test Misses Inference Attacks
Picture an inference endpoint that passes every item on a conventional test plan. Authentication holds, input validation works, and TLS is current. Behind it sits an overfitted model returning full probability vectors. The report is accurate, and the endpoint still leaks, because nobody asked the model anything about its training data.
Scanners miss it as well. An inference attack is a stream of ordinary, well-formed requests, so the only signal is in the volume and pattern of the traffic.
An AI-aware test adds four things to a normal engagement:
- an inventory of models and where their training data came from
- measured attacks, such as membership and extraction tests, with numbers attached
- a check of what each response discloses against what the product needs
- findings mapped to frameworks and assigned to one accountable owner
7. How to Run a Penetration Test on an AI Model
The approach is like any effective engagement: scope, attack, measure, fix and retest.
- Scope models, data lineages, endpoints, and query budgets. Enumerate each production model, the training data behind it, who can invoke it, how often, and the output of each endpoint. Since these tests can handle thousands of requests, decide on a query budget in advance and test the rate limits rather than getting around them.
- Try membership inference with a confidence or loss threshold. In the absence of known members, calibrate the attack with a shadow model trained on similar data.
- Test raw outputs for inversion risk by sending one record with minor tweaks and determining whether the scores shift smoothly enough to help an attacker.
- Compare the decisions of trained copies at increasing query budgets with the original measure extraction cost.
- Untrained plant canary records. A canary is a made-up record with an improbable combination of fields. If the trained model is much more confident in it than similar records it has never seen, then it is memorising rare rows. The outcome is converted into a quantifiable exposure score by Carlini and associates’ secret sharer research.
- Map each finding to MITRE ATLAS (AML.T0024), the draft OWASP Machine Learning Security Top 10 (ML03 model inversion, ML04 membership inference, ML05 model theft) and the NIST AI RMF.
A tested membership inference example
Here is how we tested. We used Python 3 and scikit-learn 1.9.1 on 1,200 synthetic rows with 20 features and deliberately noisy labels. Half were used to train a random forest, and half were held back, and the attacker sees only the confidence score for the correct answer. Scoring known members and non-members is the standard way to measure leakage, and it’s generous to the attacker, who in practice needs a shadow model to pick a threshold.
The model scored 1.000 on its training rows and 0.787 on unseen ones, and a confidence threshold turned that gap into an attack AUC of 0.955. AUC summarises the whole population, so we also checked how many members the attack could name while keeping false positives at 1%. The answer was 25.3%, roughly one member in four.
The fix has two parts: stop the model from memorising and stop the API from saying so much. This snippet continues the script above.
Reducing tree depth and leaf size reduces the attack AUC to 0.619 and the percentage of tree members named at a 1% false positive rate to 3.0%. The AUC when using only a label was 0.574. Both patches are paid. The unseen data accuracy dropped from 0.787 to 0.728, and the unpruned model had an accuracy of 0.607 when presented with labels only, consistent with research on label-only membership inference.
The pattern is reproduced across the five random seeds with attack AUCs of 0.934 to 0.961 for the unpruned model and 0.558 to 0.619 for the regularised model. While a smaller train-test gap is good, it is not necessarily a complete privacy test; a model that generalises well can still memorise rare records. They are based on one type of model and synthetic tabular data; they demonstrate the technique and the direction of each change rather than the amount of leakage in your model or whether the data could be used to identify a real person.
Probabilities versus labels at the API
The code change is small. This is an illustrative pattern rather than tested code:
The impact of the model theft is greater. When tested on probability output in our extraction test, a copy trained with probability had matched the decisions made by the original 89.7% of the time at 50 queries. The accuracy is 88.9% with just labels. The probability-based copy turned out to be a straightforward linear model, hence the plateauing at approximately 93%, and the level of agreement was assessed with synthetic inputs. These numbers should not be read as real-world theft costs but as a comparative number of two output styles.
8. Defences That Reduce Inference Leakage, and What Each One Costs
Each of these types of defence has a cost, whether it’s in accuracy, speed, helpfulness or money. Good AI privacy and security practice is about making those choices consciously and keeping a record of the reasons for them.
| Defence | What it reduces | What it costs |
|---|---|---|
| Labels, rounded scores or bands instead of raw probabilities | Membership signal, inversion, cheap extraction | Downstream systems lose scores they may rank on |
| Differential privacy in training, such as DP-SGD | Each record’s influence, with a measurable bound | Lower accuracy, slower training and more tuning; small datasets suffer most |
| Regularisation and early stopping | Overfitting, a major driver of membership leakage | Some accuracy, about six points in our test |
| Rate limits and query monitoring | Extraction and shadow-model training | Engineering time; friction for heavy but honest users |
| Minimisation, deduplication, tokenisation and synthetic data | What the model can memorise in the first place | Feature work; synthetic data can leak too, so test it |
| Output filters on generative models | Verbatim regurgitation of sensitive strings | Tuning effort and some false blocks |
| Access control on checkpoints, trackers and prediction logs | White-box attacks and copies of training samples | Process overhead for data science teams |
| Machine unlearning or retraining | One record’s influence after an erasure request | Exact retraining is expensive; approximate methods give weaker assurance. |
If you are using a model fine-tuning service, ask them whether they keep your training data or use it for other purposes. You choose what to do before the first request. Retrain now, retrain next cycle, test the model to prove it no longer represents that person. Establish a protocol for incident response as well: Identify personnel who can modify an endpoint to label-only, annul keys, or revert a model.
9. Compliance Requirements for Models Trained on Personal Data
There’s no law that explicitly states “test your model for membership,” but there are now several laws that require you to prove that a model trained on personal data does not reveal that data. Which of these apply will depend on where you operate, what data you hold and what is in force.
| Instrument | Status | What it means for model data |
|---|---|---|
| GDPR with EDPB Opinion 28/2024 | GDPR in force; opinion adopted 17 December 2024 | A model trained on personal data is anonymous only if extracting that data, directly or through queries, is insignificantly likely; authorities assess this case by case and may weigh tests against membership inference, inversion and reconstruction. |
| EDPB draft anonymisation guidelines | Adopted for consultation on 8 July 2026; consultation open until 30 October 2026 | Proposes testing anonymity against record isolation, linkage and inference |
| EU AI Act, Article 15 | High-risk obligations apply from 2 December 2027 (Annex III) and 2 August 2028 (Annex I) under Regulation (EU) 2026/1744 | High-risk systems need measures against confidentiality attacks and model flaws |
| India’s DPDP Act and DPDP Rules 2025 | Most duties on data fiduciaries apply from 13 May 2027 | Reasonable security safeguards, breach notification and erasure rights for digital personal data used in training, subject to the Act’s exemptions |
| CERT-In AIBOM guidelines, v2.0 | Technical guidelines issued 9 July 2025 | An AI bill of materials recording models, versions, data sources and datasets, which also helps trace training data for erasure |
| ISO 42001:2023 | Certifiable standard | An AI management system that procurement teams often ask for |
The EDPB opinion deserves a close read, because it ties a model’s legal status to how well it resists the attacks in section 4. Measured results like ours become evidence in that assessment. Whether a successful membership attack is itself a notifiable breach hasn’t been settled, so take legal advice on your own case.
10. The AI Model Testing Checklist
Before training
- Every dataset has a named owner, a purpose and a retention rule
- Unneeded fields removed, identifiers tokenised, duplicates dropped
- Feature store access reviewed
- Canary records planted and logged
Before deployment
- Train-test accuracy gap measured
- Membership tested (AUC and true positive rate at a low false positive rate) and extraction tested
- API returns no more than the product needs
- Rate limits and query alerts switched on
- Differential privacy decision written down
- Checkpoints, trackers and prediction logs access-restricted
In production
- Query logs kept as long as your regulators require
- Alerts on systematic query sweeps
- A retest after every retrain or data change
- An incident plan that covers disclosure through a model
Evidence that auditors and regulators will want to retain includes test scripts and seeds, measured results, framework mapping, a record of each trade-off and who approved it, and retest results, all linked to the model release that they approved.
11. How Qualysec Supports AI Model Testing
Qualysec is a company that specialises in penetration testing. We test the model and its data as well as the application around them for standard artificial intelligence systems.
The engagements start with scoping as detailed in Section 7. Then our testers run measured membership, inversion and extraction attacks, along with the usual platform checks, cloud and access to training data, checkpoints and logs. Each finding includes the script and seed used to reproduce it, what it exposes, the confidence of the result and a mapping to NIST, MITRE ATLAS, OWASP and the rules you report against. We retest each fix, and the retest figures are dated evidence for your auditors.
One example of a public AI platform is Wubble AI, an AI music platform that attackers attacked and hijacked its production environment. We traced them through the in-scope platform and cloud environments to understand how they got in. We walked the engineers through each fix and re-tested until there were no known attack paths left in that scope. The discipline is the same: find the root cause, prove the fix, and keep the evidence. This was not a model inference test but a breach recovery test.
12. Conclusion and Next Steps
One would expect the predictions of a model to carry some information. The risk is the extra information they carry about the people in the training data. AI data security what each end says, send that, and treat anything else as a disclosure you must justify.
Try it out before someone else does. If you want to know whether a confidence threshold can separate your training records from unseen ones, hear it from your tester, not a regulator. Fix the highest-scoring exposure and retest after every retrain.
Primary call. A model can pass every API test and still tell an attacker who was in its training data. Book a scoping call with our team to find out whether yours does, before it goes live.
Secondary call. Want to see how findings are written first? Review a sample penetration testing report, with reproduction steps and fixes written for the engineer who has to ship them.
13. Frequently Asked Questions
1. How is AI used in data security?
Detection and classification: Identify sensitive files, detect anomalous access, prioritise alerts and accelerate investigations. These tools are models that have been trained on sensitive data, so you should test them for leakage like you would any other model.
2. Can a model trained on personal data be anonymous?
Sometimes, but you can’t count on that. According to the EDPB’s 2024 opinion, it relies on whether personal data can be extracted from the model, either directly or by means of queries, which must be assessed on a case-by-case basis. That evidence includes measured results of attacks.
3. How to protect data when using AI?
Less data, deduplicate it, restrict access to the model, return only what the product needs, and test for leakage before you ship. Launch. Monitor usage. Retest after every retrain.
4. How can weak security around AI models lead to sensitive data leakage?
The outputs of a model trained on sensitive data can leak some of it. Attackers use confidence scores to guess who was in the training set, reconstruct typical inputs, extract memorised text or copy the model, and make every request look legit.
5. Which compliance requirements apply to AI systems and their data?
In Europe, we have the GDPR, the EDPB guidelines on AI models and, for high-risk systems, the EU AI Act. In India, the DPDP Act will apply the main duties in May 2027, with CERT-In’s AIBOM guidelines as guidance. Procurement often mentions ISO/IEC 42001 and NIST AI RMF.
6. What are the main risks to data inside AI systems?
The principal risks to data within standard artificial intelligence systems include membership and attribute inference, model inversion, training data extraction, model theft, poisoned data, exposed pipelines or checkpoints, and the regulatory and trust consequences of a leak.
