Chunking Documents

Retrieval works on chunks

A whole 40-page handbook is far too broad and unfocused a unit to hand to an embedding model and expect a useful, specific search match - its single vector would end up representing an average of dozens of unrelated topics. Chunking solves this by splitting a long document into smaller, more focused passages before computing embeddings, so each chunk's vector represents one coherent idea rather than a blur of everything in the document.

Getting the chunk size right takes some judgment, and it's easy to go wrong in either direction. Chunks that are too large still mix several topics together, diluting the vector and hurting retrieval precision. Chunks that are too tiny lose the surrounding explanation that gives a single sentence its meaning - a lone sentence like "set it to 30 seconds" is useless without the paragraph explaining what setting it refers to. A small amount of overlap - deliberately repeating a little text at the boundary between neighboring chunks - can help preserve a sentence that would otherwise get awkwardly split in half, though it does mean the same content can now show up in more than one retrieved result.

Where possible, prefer splitting along structure the document already provides - headings, paragraph breaks, list items - rather than a blind fixed character count that might cut a sentence in half. And always keep track of where each chunk came from: its source file path, the heading it fell under, its position (chunk index) within the document, and the document's version, all stored as metadata alongside the chunk.

Analogy: Searching a book by returning the entire book back to someone isn't useful - they still have to find the relevant part themselves. Returning one isolated sentence, ripped out of context, can be actively misleading. A well-sized chunk is the middle ground: it gives the actual answer and enough surrounding explanation to understand it correctly.