What Is Chunking? How to Split Documents for RAG
What is chunking?
Chunking is the process of splitting a long document into smaller, self-contained pieces (chunks) that an AI system can embed and search. Also, each piece is indexed on its own. When a user asks a question, only the most relevant pieces reach the model, so it never has to read the whole archive.
The word comes from English "chunk," meaning a manageable block. Ask a dozen engineers what is chunking, and you will hear this same answer in many forms. Some teams say "document splitting" or "text segmentation" instead. The idea is the same. Still, the name matters less than the rule behind it.
Chunking is not a product you can buy. It is the first and most neglected step of a retrieval pipeline. A badly split document cannot be rescued by any model that comes after it.
So what is chunking in business terms? It decides how findable your company documents are for an AI assistant. In addition, the same model, the same archive and the same question can produce very different answers once the splitting rule changes.
What is chunking in plain terms? A simple analogy
Picture a thick encyclopedia. When you have a question, you do not pile every volume on your desk. Also, you check the index, find the right entry and read only those pages.
Chunking creates the entries of that encyclopedia for your documents. For example, if an entry is too short, the context breaks. Also, if it is too long, the sentence you need disappears in the crowd.
Think of a good editor. Before cutting a topic apart, the editor looks at headings, paragraphs and tables. Nothing gets sliced in the middle of a sentence. That matters because a half sentence tells the model very little.
In an AI system, a splitting rule plays the editor. You pick the rule. That choice directly shapes how accurate your search will be.
For example, two teams may work with the same documents. The first team cuts at page breaks and the second splits by headings. The second system can give more consistent answers for that reason alone.
Why do we split documents into chunks?
There are two practical reasons. First, embedding models turn each input into a single list of numbers, called a vector. In practice, if you squeeze a very long text into one vector, the details blur together. We covered the basics in our embeddings guide.
Second, language models have a limit on how much text they can take at once. Even when that limit grows, stuffing everything in raises cost and noise. Our context window article explains the limit in detail.
A third reason is search precision. A user question usually matches one paragraph of a document, not the whole file. Small, focused pieces let the search engine land on exactly that paragraph.
Less text also makes answers easier to trace to a source. Research on long inputs suggests that models can use information in the middle of a long prompt less reliably. You can read the findings in the Lost in the Middle paper.
Example scenario: a furniture company keeps shipping, returns and assembly rules in one long document. A customer asks only about assembly fees. For example, if the whole document goes to the model, it reads three topics and may mix up the rules. With three separate chunks, only the assembly section comes back.
Where does chunking sit in a RAG pipeline?
RAG, or retrieval augmented generation, finds documents and hands them to the model as evidence. The method was introduced in the paper by Lewis and colleagues. In addition, the flow looks roughly like this.
- Collect your documents and convert them to text.
- Split the text into chunks. This is where chunking happens.
- Turn each chunk into a vector with an embedding model and store it in a vector database.
- When a question arrives, embed the question too and fetch the nearest chunks.
- If needed, filter the results with a reranker.
- Give the selected chunks to the model and let it write the answer.
As you can see, chunking defines the input of retrieval. Every later step works with these pieces. Also, if the chunks are poor, the search and the answer are poor as well.
Do not skip one detail. You usually split once, but you live with the result in every single query. In practice, if you change the rule later, you must embed the whole archive again. So a little time spent early saves a heavy reprocessing job later.
Each step also has its own job. Chunking prepares the information for search, embedding converts meaning to numbers, and retrieval finds the closest match. A mistake in one step cannot be fixed by the others.
How does fixed-size chunking work?
Fixed-size splitting is the simplest method. You cut the text when it reaches a set length, then start a new chunk. You can measure length in characters or in tokens. Our token guide explains tokens.
The advantages are speed and predictability. Every chunk has about the same size, so you can estimate cost and storage in advance.
The risk is that it ignores meaning. A cut may land in the middle of a sentence or even a table row. Read on its own, that chunk is incomplete. Because search returns chunks one by one, such a gap goes unnoticed until a user complains.
Example scenario: a paragraph about returns is split in two. The first chunk says "you can return items." The second carries the condition "only if unopened." If search returns just the first chunk, the model sounds confident and is wrong.
Here is one tip. In practice, when you cut by length, move the boundary to the nearest sentence end. Many frameworks offer "recursive" splitters that do exactly this: they try paragraphs first, then sentences, then words.
Fixed size is a fast starting point. That said, it rarely stays the only method. For a serious archive, however, it rarely stays the only method.
What does overlapping chunking do?
Overlap means leaving a shared piece of text at the border of two neighboring chunks. The second chunk starts with the last sentences of the first. That way the information at the cut appears in both.
The goal is to reduce broken context. Put simply, a repeated sentence is cheaper than a lost condition. In the previous example, the condition "only if unopened" sits in both chunks. Whichever chunk search returns, the answer stays complete.
Overlap has a price, though. Also, the same text gets indexed more than once. Storage grows and search results can repeat themselves.
How much overlap you need depends on the document. Do not copy numbers from a provider default. Test with your own files instead. The OpenAI file search guide, for instance, treats chunk size and overlap as adjustable options. Check the current defaults in the official documentation.
When does heading-based chunking work better?
Heading-based splitting follows the document's own structure. In addition, the text between one section heading and the next becomes a chunk. For example, if a section is too long, you go down to subheadings or paragraphs.
This method shines with structured documents. It works because the author has already done the hard thinking about topic boundaries. User manuals, policy texts, contracts and help center articles are good examples. The author already divided the topic with headings.
LangChain and LlamaIndex offer ready components for this. Format-aware splitters use Markdown headings or HTML tags as boundaries. You can browse the options in the LangChain splitter documentation.
The weakness is that it fails on documents without headings or with messy structure. And when section sizes vary widely, some chunks grow too large while others stay tiny. In that case, combine heading-based splitting with a length limit.
For example, if a help article has three paragraphs under "Shipping," all three become one chunk. Also, if ten paragraphs sit under the same heading, you cut the section at paragraph borders into two or three chunks. In both cases, write the heading at the start of each chunk.
What is semantic chunking and when is it worth it?
Semantic chunking picks boundaries by meaning rather than by length. You compare the embeddings of consecutive sentences. In practice, when the topic shifts clearly, you start a new chunk.
The benefit is that each chunk stays on a single topic. It can help with loosely structured, flowing text such as meeting transcripts or long emails.
The cost is extra computation. Also, you need an embedding for every sentence, which adds time and spend. You also have to tune a threshold, and the right value changes from document to document.
We do not make it our first choice. We start with a heading-based and overlapping baseline. Then, if measurement shows that semantic splitting makes a difference, we switch.
LlamaIndex documentation also presents several parsers under one abstraction. See the node parser page for details.
What is parent-child (hierarchical) chunking?
Small chunks are precise for search because they focus on one topic. Large chunks are better for answering, because they carry the surrounding context. Hierarchical chunking combines both.
First, split the document into large parent sections. Then split each parent into small child chunks. In addition, you search over the children. In practice, when one matches, you give the model the parent section it belongs to.
This way you gain precision and context together. Frameworks such as LlamaIndex provide ready parsers for this pattern.
Know the risks as well. In practice, if the parent is huge, the noise comes back. For example, if several children point to the same parent, the model may receive the same text again and again. You also have to store the link between child and parent.
We usually suggest it for long manuals and contracts. For short FAQ entries it only adds complexity.
How do you choose the right chunk size?
Chunk size is the most debated setting in chunking. There is no magic number. Instead of guessing, treat the size as a setting you tune. For that reason we give no figure in this article, and we do it on purpose.
Think of it as a balance. Also, chunks that are too small break the context. A sentence alone means little, and retrieval precision drops. In addition, chunks that are too large add noise. They carry unrelated information and distract the model.
The right size depends on three things.
- Document type: short FAQ items and long contracts do not fit the same size.
- Question type: short fact questions want small chunks, while explanatory questions want larger ones.
- Embedding model limit: check the accepted input length in the provider's documentation.
The practical way is to try several sizes against the same question set. Pick the one that returns the right chunk most often. The answer is always the same: find it by testing.
Example scenario: a dental clinic wants to archive treatment pages and booking rules. The booking rules are short items, so small chunks work well. The treatment descriptions are long narratives, so they want larger chunks. Using two sizes for two document types can make sense here.
While choosing, check whether a chunk is understandable when read alone. Also, if a person can read it and grasp the topic, the model can too. This simple test works for many teams.
Why should you attach metadata to every chunk?
A chunk on its own loses its context. Which document does "the period stated in Article 4" come from? Metadata answers that question.
Metadata is extra information attached to the chunk. Also, it costs very little to store. It usually includes the following.
- Document title and section heading
- Source file name or address
- Page number
- Publication or update date
- Document type, language and version
- Access permission (who may see it)
It gives you three benefits. First, you can filter the search: only the current version, or only the documents of one department. Second, you can cite the source in the answer. Third, you avoid showing restricted documents to the wrong people.
Adding the heading to the chunk text is a strong trick. In practice, if a chunk starts with "Shipping time," the embedding carries that context too. In LlamaIndex, the metadata of the parent document passes down to child nodes, as the node parser page above describes.
Example scenario: an HR handbook has general rules and manager-only sections. For example, if you tag each chunk as "everyone" or "manager," search only covers chunks that match the user's permission. Apply the tag during the search, not after it.
How do tables and PDFs break chunking?
PDF is a format designed for human eyes. The text order inside the file may not match the reading order. Two-column pages can come out with the columns mixed up.
Headers and footers repeat on every page. Also, if they end up inside chunks, they pollute the search. Clean them out during extraction.
Tables are even harder. First, the row structure matters, and second, the header gives every number its meaning. In practice, if you cut a row in the middle, a number gets separated from its header. A value like "120" forgets which column it belongs to. The fix is to treat a table as an unsplittable unit. For example, if the table is big, repeat the column headers in each chunk.
Scanned documents need optical character recognition (OCR) first. An OCR error is copied straight into the chunk. For important files, sample the extracted text and check it by eye.
Example scenario: an accounting office scans old invoice templates. Also, if OCR reads an "8" as a "3," the chunk carries a wrong amount. Search treats it as correct and the model repeats the error with confidence. Never skip human review for critical documents.
If you want help with this work, see our AI document processing solution.
How do the main chunking strategies compare?
The table below puts the four core strategies side by side. These are general tendencies, and results vary with your documents.
| Strategy | Advantage | Risk | Best fit |
|---|---|---|---|
| Fixed size | Simple, fast, predictable cost | Cuts sentences and tables in the middle | First trial, plain text |
| Overlapping | Keeps context at the cut point | Repeated content, more storage | Flowing narrative text |
| Heading and section based | Follows the document's logic | Fails on messy documents | Manuals, policies, contracts |
| Semantic | Produces single-topic chunks | Extra compute, threshold tuning | Unstructured long text, transcripts |
The table shows that there is no single winner. Most production systems mix strategies. For example, they split by heading first, cut large sections by length, and add a small overlap at the cut.
How does chunking change by document type?
One rule does not fit a whole archive. Also, you can choose a starting point by document type. These are field-based pointers, not guarantees.
- Short FAQ items: keep each question and answer pair as one chunk, with no splitting.
- User manuals: split by heading and cut long sections at paragraph borders.
- Contracts and policies: add the clause number and heading to every chunk and consider hierarchical splitting.
- Product catalogs: make each product one chunk and write the product name into the metadata.
- Support conversations: keep conversation turns together, because a single message often means little.
- Meeting transcripts: use semantic splitting or speaker changes as boundaries.
Example scenario: an online store loads both product pages and its return policy into the same vector database. In practice, if products are single chunks and the policy is split by headings, the two document types can be searched without hurting each other.
Write the document type into the metadata too. That lets you build filters such as "search only in policies."
Which terms get confused with chunking?
Many readers who ask what is chunking also ask how it differs from neighboring concepts. In addition, the table below sums up the differences. Each term has its own article.
| Term | What it does | Link to chunking |
|---|---|---|
| Chunking | Splits a document into pieces | Sets the size of the search unit |
| Embedding | Turns text into a vector that represents meaning | Chunks become its input |
| Reranker | Reorders the retrieved chunks | Good chunks make ranking easier |
| Context window | The limit of what a model reads at once | Chunks must fit inside it |
| Token | The model's unit of text | Chunk size is often measured in tokens |
Chunking answers "how much and from where." A reranker answers "which one is more relevant." We described the broader job of managing model input in our context engineering article.
How do chunking, cost and updates affect each other?
So what is chunking going to cost you? The splitting decision shapes not only quality but also running cost. As the chunk count grows, embedding work and storage grow with it. Overlap pushes that count up even further.
Updates matter too. Suppose one paragraph of a document changes. For example, if chunks are arbitrary, you may need to reprocess the whole file. With heading-based splitting, you refresh only the chunks of the changed section.
For that reason, give every chunk a stable ID. Also, if you derive the ID from the document name and section heading, replacing old versions with new ones becomes easy.
Do not forget deleted documents. Otherwise stale chunks keep answering questions for a source that no longer exists. In practice, when you remove a source file, remove its chunks from the index as well. Otherwise the model may cite a rule that no longer exists.
For cost, check current prices on the provider's official page. Embedding cost usually follows the token count, so needless repetition shows up directly on the bill.
Can ready-made tools do the chunking for us?
Partly. Many providers split files automatically when you upload them. That is great for a quick prototype. Still, default settings do not know your documents.
The OpenAI file search documentation, for example, offers an automatic strategy. It also lets you set chunk size and overlap by hand. Check which settings exist and what their defaults are in the official documentation, because they can change.
LangChain and LlamaIndex give you more control. You choose the splitter, the separator and the metadata flow yourself. In return, you maintain the code and the settings.
Our practical advice: start with the ready-made setting, then measure the results. In practice, if you see problems, define the rule by hand. We do not call any tool the best, because the right choice depends on your documents and your team's skills.
How do you measure chunking quality?
Do not choose chunking because it looks good. Measure it. Also, you need a list of questions that resembles real user questions, plus a note on which document holds each correct answer.
Then ask: does the right chunk appear among the top results? This is usually called retrieval accuracy. The second question is whether the model answers correctly with the retrieved chunks.
Change only one setting at a time. For example, if you change chunk size, keep overlap fixed. Otherwise you cannot tell where a difference came from.
For evaluation methods, see our LLM evals article. Also sample real user questions regularly, because the document archive changes over time.
Failure analysis helps as well. Then you see patterns instead of single cases. Open the questions that got wrong answers one by one and see why the right chunk did not come back. Usually the cause is one of two things: the chunk was too small and had no context, or it was too large and unrelated text drowned out the key sentence. That finding tells you which way to adjust.
Some wrong answers come from the model making things up. We covered that in our hallucination article. Sometimes, however, the real culprit is a chunk that was missing, not the model.
Why do access control and privacy matter in chunking?
The security side of chunking is often overlooked. Once you split a document, every chunk becomes an independent record. That record does not carry the source document's access rules on its own.
If you do not write the rule onto the chunk, an unauthorized user could see a piece of a manager-only document. Add the permission to the metadata and filter every search by it.
Personal data is a separate topic. Because chunks are copies, deletion requests need extra care. In addition, chunks live in the vector database as another copy. Also, if a person asks for their data to be deleted, you must find and delete those copies too. Storing a source ID on each chunk makes that job easier.
This is not legal advice. For archives that hold personal data, clarify your obligations with your own legal counsel.
What are the most common chunking mistakes?
These are the mistakes we see most often in the field.
- Splitting everything with one setting: a contract and an FAQ do not follow the same rule.
- Copying a number: applying a chunk size from another article without testing it.
- Skipping cleanup: headers, footers and menu text stay inside the chunks.
- Leaving out metadata: without source, date and permission, a chunk is an orphan.
- Cutting a table by rows: numbers without headers mean nothing.
- Never measuring: judging chunking only by reading a few model answers.
There is one more risk: security. Documents may contain hidden instruction text, and it can influence the model once a chunk reaches it. See our prompt injection article for that risk.
Most of these mistakes do not show up in the first week. The system works on demo questions and breaks when real users arrive. For that reason, test with question types you have not seen before going live.
What is chunking's checklist for a company document archive?
Now you know the answer to what is chunking, so follow this list in order when you prepare your own archive.
- Separate your document types: manuals, contracts, FAQs, table-heavy reports, transcripts.
- Choose a splitting rule for each type. Start heading-based for structured files and overlapping for flowing text.
- Clean the text extraction: remove headers, footers and page numbers.
- Treat tables as unsplittable units and repeat their headers.
- Add title, source, version and permission to every chunk.
- Check the input limit of your embedding model in the provider documentation.
- Prepare a realistic question list and try several sizes against it.
- Test that an unauthorized document never leaks into an answer.
- When a document changes, re-index only the affected chunks.
- Re-measure with sample questions at regular intervals.
The order matters. First you learn the documents, then you choose the rule. You first get to know the documents, then choose the rule, and measure last. Starting to measure before you pick a rule wastes time. Also keep the first version small: begin with ten documents instead of hundreds, see the result, then scale up.
You can use our word counter to get a quick feel for chunk length. It only gives a rough idea, since the exact measure is the token count.
When should you get outside help with chunking?
Splitting a small set of manuals and trying it yourself is reasonable. As the document count grows and the types vary, the work turns into engineering. Scanned PDFs, permission rules and constantly changing archives take the most effort.
We, Talha Aslan and our team, start these projects by reviewing your documents. Then we build a small trial set, measure the results and scale up. For company knowledge needs, you can look at our RAG development page.
Note: This article is general information and not legal advice. Before you give documents with personal data to an AI system, consult your own legal counsel. Always check providers' current limits and default settings in their official documentation.
What is chunking in short?
In short, chunking means splitting a document into meaningful pieces that suit search. Pieces that are too small break the context. Pieces that are too large add noise.
You can begin with simple fixed-size splitting. Then move to heading-based and overlapping methods, add metadata and protect your tables. Try semantic splitting only when measurement calls for it.
The most important rule is this: do not borrow a number from someone else, and test with your own documents. After all, the right chunking depends on the kind of document and the kind of question.
