What Is Data Poisoning? How It Attacks AI Training and RAG Data
What is data poisoning?
Data poisoning is an attack in which someone deliberately slips corrupted, misleading, or booby-trapped records into the data an AI system learns from or reads, so the model behaves the way the attacker wants. The model code stays untouched. Instead, what changes is the knowledge behind it.
Think of a chef whose recipe book has been quietly edited. The chef cooks carefully and follows every instruction, yet the dish still tastes wrong. So the fault sits in the source, not in the cook.
AI models work the same way. For example, they find patterns in the data they see. Then, if a malicious record enters that data, the model absorbs it and repeats it with full confidence.
In this guide we answer what is data poisoning as a single term. First we look at how it works, then at its types, then at how it differs from lookalike terms. After that we cover defenses and finish with a practical checklist for your business.
Why is data an attack surface for AI systems?
In traditional software, code decides behavior. Security teams therefore focus on code, servers, and networks. In machine learning, a large share of behavior comes from data. So someone who can change the data can influence the system without writing a single line of code.
As a result, a new risk area opens. Models often learn from wide and messy sources: public web pages, open datasets, user reviews, and internal documents. Also, each of these is an entry point you do not fully control.
The data pipeline is also long. Collection, cleaning, labeling, training, and deployment each offer a moment where someone can interfere. In practice, the weakest link is the one an attacker will pick.
Security frameworks treat the topic seriously for this reason. OWASP lists data and model poisoning as its own risk in its list for large language model applications. The goal is not always to break the model. Instead, the goal can be to plant a hidden behavior.
How does data poisoning work?
Conceptually, the process has three steps, first access, then blending, then effect. In the first step, the attacker gains access to a data source that the model or system will read. Sometimes that access is direct. Often, however, it is indirect, through an open source that anyone can contribute to.
In the second step, the corrupted records blend into the pipeline. They usually look ordinary, so a quick visual check will not catch them. Effective poisoning is quiet, and the system keeps running.
In the third step, the model learns from those records, or the system pulls them into its answers. Behavior shifts as a result. The shift can show up as a general drop in quality. Or it can appear only on one topic or under one condition, which is harder to see.
- The attacker finds a way to contribute to a data source or alter it.
- Next, the corrupted records pass through collection and cleaning unnoticed.
- Then the model trains on them, or the system retrieves them at answer time.
- Finally, the effect surfaces only when users notice strange answers or an evaluation catches it.
For safety reasons, this article deliberately gives no attack recipe. Our aim is to help you understand the process so you can build your defense around it.
What types of data poisoning exist?
You can split data poisoning into three main groups, based on which stage the attacker targets. This split also tells you where to place your defenses.
- Training data poisoning: Corrupted records enter the pretraining or training data.
- Fine-tuning poisoning: Corrupted records enter the dataset you use to adapt an existing model.
- Knowledge base poisoning: A misleading document lands in the store a model consults when it answers (RAG).
The first two types go inside the model. Also, the poison settles into the weights, and undoing it usually means retraining. In contrast, the third type never touches the model. It only damages the source the model reads, so cleanup is easier, although detection is not always easy.
You can also classify attacks by purpose. For example, some aim to lower overall accuracy. Others aim for a wrong answer on one topic or for a hidden behavior. The US National Institute of Standards and Technology (NIST) treats poisoning as its own attack class in its adversarial machine learning taxonomy.
What do backdoor and trigger mean in data poisoning?
A backdoor means the model behaves normally under ordinary conditions and acts differently only when it sees a specific signal. That signal is called a trigger. For instance, a trigger can be a word, a pattern, or a data feature. In other words, it works like a hidden rule.
The danger is that the model looks fine in standard tests. Because the test data contains no trigger, the model answers as expected. Therefore a high overall accuracy score is a poor proof of safety.
An analogy helps. A building passes its security inspection, with locked doors and working cameras. Yet someone left a hidden side door during construction, and only the person who knows about it can walk in. In short, a backdoor is that hidden path inside a model.
To stay on the safe side, then, we do not explain how triggers are designed. What matters is that you know the risk exists and that you question where your models come from. This applies especially to ready-made models you download from outside.
What is RAG document poisoning, and why does it matter so much?
RAG (retrieval augmented generation) means the model searches a document store before it answers and adds what it finds to its reply. We covered the basics in our guide to RAG. So here we focus only on the security side.
RAG document poisoning means a misleading or corrupted document gets into that store. When the search finds it, the model trusts the content and works it into the answer. The model never changes. Only the material it reads has been damaged.
This type matters because the barrier to entry is low. Corrupting training data may require scale. By contrast, adding one document to a shared folder, an open wiki page, or a user upload area is far easier.
That document might also carry a prompt injection, meaning text that looks like an instruction to the model. This is a separate risk. We compare it in the table below and cover it in our prompt injection guide.
How documents are split and searched also matters. For the related basics, read about chunking and embeddings.
How does fine-tuning and user feedback poisoning happen?
Fine-tuning adapts a ready-made model to a task with your own data. Also, you can find the basics in our fine-tuning and LoRA guide. The dataset is usually small, so each individual record carries more weight.
The risk grows when a product learns from user content. For example, picture a chat app where users give thumbs up or corrections. If you collect that feedback to improve the model, someone could steer that channel on purpose.
Here, the problem is the assumption that feedback is trustworthy. Without a filter that separates real users from bad actors, every rating and every correction becomes a potential data source. Human review is therefore a valuable safeguard in automatic learning loops.
Also, another source is data produced by other models. If you use synthetic data, you should question the quality and trustworthiness of its source as well. We cover that topic in our synthetic data guide.
What is data poisoning compared with prompt injection and model collapse?
People often mix up these three terms because all of them involve data problems that damage AI. Yet timing, intent, and remedies differ. The table summarizes the difference.
| Feature | Data poisoning | Prompt injection | Model collapse |
|---|---|---|---|
| When it happens | During training, fine-tuning, or knowledge base preparation | While the model runs, through the prompt or content it reads | When models are retrained repeatedly on each other's output |
| Is there intent? | Yes, the attacker acts on purpose | Yes, the attacker acts on purpose | No, it is usually an unwanted side effect |
| What it targets | The data the model learns from or reads | The instruction flow at that moment | The diversity and quality of the data |
| Duration of effect | Can be lasting, may need retraining | Usually limited to that session | Builds up across generations |
| Main defense | Data provenance, access control, anomaly detection | Permission limits, input and output checks | Keeping real data in the mix, separating sources |
The takeaway is simple. If someone asks what is data poisoning in one line, say that it is a problem of what the model learned or reads. Data poisoning is a problem of what the model learned. In contrast, prompt injection is a problem of what the model obeys right now. Finally, model collapse is a problem of a model imitating itself. We explain collapse in a separate article.
The two attacks can also feed each other. A poisoned document that carries a hidden instruction creates both effects at once. For that reason, do not leave your defense to a single layer.
Who is at risk from data poisoning?
Not every AI user carries the same risk, because risk depends on where the data comes from and who can touch it. The following groups deserve closer attention.
- Teams that collect data from outside sources: Anyone training on web crawls or public datasets cannot fully inspect the source.
- Products that fine-tune on user content: Products that feed reviews, feedback, or chat logs back into training.
- Systems with open knowledge bases: RAG systems connected to wikis, forums, or shared document stores that many people can edit.
- Teams that download ready-made models: Anyone who puts unverified models straight into production.
By contrast, a small system that only uses approved and versioned documents under your control has lower risk. Still, the risk is not zero. Also, internal mistakes and wrong version uploads can cause similar damage.
In short, ask yourself who stands at the door of your data, and whether you really control that door. If the answer is unclear, the defense sections below are your priority.
What does data poisoning look like in a real business?
The following is an example scenario, not a real customer or incident. Imagine a store with a support assistant that reads return policies from a shared document folder.
However, anyone can upload files to that folder. One day a document appears that describes the return terms incorrectly. Then the assistant finds it during search and starts telling customers the wrong return window. The model is healthy, but the answers are wrong.
In this scenario there may be no attacker at all. For instance, a careless employee could cause the same result. But if a bad actor uses the same path on purpose, the effect becomes more targeted and harder to spot.
As a result, the business risks customer trust and possibly legal exposure. Teams that build a company knowledge assistant should therefore plan content governance from day one. For such a setup, take a look at our RAG development solution.
What are the signs of poisoning, and how do you notice them?
Poisoning is often quiet, yet some signals should raise suspicion. No single sign is proof. When several appear together, an investigation is justified.
- The model keeps giving the same wrong answer on one topic.
- Answer quality changes noticeably right after a data update.
- The same question gets very different levels of certainty depending on its wording.
- The knowledge base contains new documents of unclear origin that are retrieved very often.
- User complaints cluster around one topic or product.
So you need monitoring to catch these signals. For example, you can log which documents appear in how many answers. Then, if one document suddenly dominates, find out why.
Regular evaluation sets also help you measure the effect of data changes. That way you can compare the state before and after an update. For the evaluation approach, read our LLM evals guide.
Why is data provenance the foundation of defense?
Data provenance is the trail that shows where a record came from, who added it, when, and which transformations it went through. In other words, you cannot trust data when you do not know its source. This simple principle is the first layer of defense.
OWASP also lists tracking data provenance, using trusted sources, and versioning datasets among its core mitigations. The logic is clear. So when a problem appears, you must be able to find which records it touched.
- Record the owner and trust level of every data source.
- Then version your datasets and note which version each training run used.
- Also keep outside data in a separate area and validate it before use.
- Store a fingerprint (hash) of records so later changes become visible.
These records help beyond security, too. When a model starts to behave oddly, you can trace back through them. You then isolate the faulty version quickly and roll back to the previous one.
How do you set up access control and an approval process?
The cheapest barrier against poisoning is limiting who can write to your data. Also, the principle of least privilege applies here. The fewer people who can add documents to a knowledge base, the smaller the attack surface.
Two layers work well in practice. The first is role based write access, so only responsible people can add content. The second is an approval step that every addition must pass.
- Limit write access to the knowledge base to a small, known group.
- Require a second person to approve new documents before they go live.
- Also keep user uploads in a separate area and never merge them directly into the main knowledge base.
- Log every change together with who made it.
These measures work especially well for RAG systems, because the poison enters a document. The poison enters the document, not the model, so classic access control applies right away. For layers that limit model output, see our guide to AI guardrails.
How do anomaly detection, evaluation, and red teaming help against poisoning?
Prevention is never perfect, so you also need a detection layer. First, anomaly detection flags unusual records in a dataset. For instance, you review records that differ sharply from the rest, repeat too often, or have an unclear source.
On the evaluation side, two ideas combine. First, run a fixed test set after every data change. Second, make sure that set covers sensitive topics and not only overall accuracy.
Red teaming is an authorized, defense oriented testing approach. In practice, a team probes the system as an attacker would and finds weak points before real attackers do. We explain it in our AI red teaming guide.
In short, the three methods complement each other. Anomaly detection works at the entrance, evaluation works at the exit, and red teaming looks at the whole system. None is enough alone, but together they cut the gaps noticeably.
Why is poisoned data so hard to tell apart?
The most uncomfortable part of the question what is data poisoning is that an attack often resembles ordinary data. For example, a corrupted record can look like a typo or an outdated fact. Even a human reviewer may struggle to tell the difference.
The second difficulty is scale. A large dataset can hold millions of records, and nobody can read them all. Therefore sampling and automated checks help, but they are not perfect.
The third difficulty is that the effect is scattered. The model works well overall and then fails only on a narrow topic. If your test set misses that topic, the problem stays invisible.
Finally, bad data does not always come from attackers. For instance, old documents, conflicting versions, and wrong labels cause similar damage. Good data hygiene therefore protects you against deliberate and accidental problems at the same time.
Do ready-made models and datasets carry supply chain risk?
Yes, they do. If adding an unverified library to a software project is risky, using an unverified model or dataset is just as risky. Therefore we call this AI supply chain risk.
Platforms that share model files and datasets are very valuable. Still, in an environment where anyone can contribute, it falls on you to confirm that a package contains what it claims. Reading the verification advice in the provider's official documentation is a good habit.
- Take models and datasets only from known sources with clear ownership.
- Then compare the fingerprint of a downloaded file with the one the publisher provides.
- Evaluate a new model in a separate test area before moving it to production.
- Also record license and usage terms.
Thus this gives you a solid record for both security and compliance. To plan your security layers step by step, see our AI integration solution.
How do you separate data poisoning from ordinary quality problems?
Not every bad answer is an attack. In fact, most of the time the cause is hallucination, missing data, or an outdated document. To avoid false alarms, rule out the simple explanations first.
- Can you reproduce the error? Because a consistent error differs from a random hallucination.
- Did the error start after a specific data update?
- Can you trace the wrong answer back to a document or record?
- Do you know who added that document and when?
If you can answer these questions clearly, you can also tell whether the event was accidental or deliberate. If you cannot, then the real gap is your monitoring setup. Remember that hallucination is a separate phenomenon. Details are in our AI hallucination guide.
Who should own data security on your team?
The technical answer to what is data poisoning is not enough, so ownership matters too. You also need to know who owns the response. In many organizations the team that prepares data, the team that builds the model, and the team that runs the product are different people. When trouble strikes, everyone assumes someone else is responsible.
Defining three roles removes that ambiguity. First, the data owner is responsible for the content and freshness of each source. Second, the model owner approves what goes into training or the knowledge base. Finally, the security lead manages access rules, monitoring, and the incident process.
In small teams, however, one person may hold all three roles. What matters is that the roles are written down and that an independent second person approves changes. That way one mistake or one bad actor cannot corrupt the pipeline alone.
Finally, do not neglect training. Also, when employees know which documents they may upload and where, the most common door for poisoning closes. For team training, see our corporate AI training page.
What should you do if you suspect poisoning?
When suspicion arises, a clear order of steps beats panic. The following steps are a general incident response flow. So adapt them to your own system.
- First, identify the suspicious data source and the related update.
- If needed, switch off the affected part of the knowledge base for now.
- Then roll back to the last known healthy data version.
- Next, examine the problem records and find out how they got in.
- Finally, update access and approval rules to close that gap.
- If retraining is needed, start from cleaned data and run your evaluation set.
This process is faster for RAG systems, because you only remove the document. With training or fine-tuning poisoning, the model weights may be affected and retraining may be necessary.
One important note: if the incident touches personal data or customer information, you may have legal duties. This article is not legal advice. Therefore, in such a case, consult a qualified lawyer.
What are the limits and common misunderstandings about data poisoning?
A few misunderstandings are common. Clearing them up therefore reduces both unnecessary fear and unnecessary comfort.
- "Big company models cannot be poisoned." No model is immune, and risk depends on the nature of the data source.
- "High accuracy means safe." A backdoored model can look excellent in standard tests.
- "Only training data is risky." Knowledge bases and feedback loops are equally important surfaces.
- "One tool solves it." Defense comes in layers, and the root cause is often process.
The NIST taxonomy also stresses that current defenses do not offer complete guarantees and that open research questions remain. Your goal is therefore to reduce risk and to notice problems, not to reach zero risk.
This field also moves fast. Also, we recommend checking the current state of methods and tools in the official documents of standards bodies and providers.
What is a practical checklist for a business asking what is data poisoning?
The list below can serve as a starting point for a small or midsize team. However, you do not need to finish it in a day. Set your priorities according to your own risk.
- List in writing which sources feed your model and knowledge base.
- Review who can write to your data and remove unnecessary permissions.
- Tie every new document and data addition to an approval step.
- Verify the source of every model and dataset you download.
- Run a fixed evaluation set after each data update.
- Track how often each document appears in answers.
- Set up a backup routine that lets you roll back to a healthy data version.
- Plan an authorized red teaming exercise at least once a year.
The first three items are the cheapest and have the highest impact. So for most teams, starting there makes sense.
If you want to set up AI safely in your organization, you can reach us through our AI consulting page. For the broader picture, see our AI and automation services.
Where can you learn more about data poisoning?
The most reliable starting points are the official publications of standards bodies and security communities. Also, the framework in this article rests on them.
- OWASP Top 10 for LLM Applications: The risk list for large language model applications, with data and model poisoning as its own item.
- NIST adversarial machine learning taxonomy: The official publication that classifies attack types, including poisoning, and defines the terms.
These sources get updated. Instead of relying on a specific version or number, open the publication and read its current form. The information here is general education and does not replace security or legal advice.
To sum up, what is data poisoning? It is a quiet attack on what an AI system learns or reads. Instead, you build your defense with provenance, access control, monitoring, and regular testing. Do not open the door of your system to data whose source you do not know.
