Review Chunking

Chunking is the process of dividing a document, dataset, or stream into smaller units for storage, search, retrieval, or processing.

Chunking divides a document, dataset, or stream into smaller units that a system can store, search, retrieve, or process. The boundaries may follow length, structure, meaning, time, or a combination of these.

Chunk boundaries determine what context stays together. Poor chunks can split evidence from its qualifier, create noisy retrieval, or exceed a model’s context limits.

A policy manual is divided by section headings, with a small overlap between adjacent sections, before each chunk is embedded for retrieval.

Chunking is not simply cutting text into equal character counts. Fixed-size chunks are one method, and they may damage meaning when structure matters.

Chunk where the meaning can travel together.

* [[retrieval-augmented-generation|Retrieval-Augmented Generation]] * [[embedding|Embedding]] * [[context-window|Context Window]] * [[search-retrieval|Search / Retrieval]]

* [Patrick Lewis and colleagues, “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”](https://arxiv.org/abs/2005.11401) — shows retrieval over indexed passages as external model context.

* Compare [[retrieval-augmented-generation|Retrieval-Augmented Generation]], [[embedding|Embedding]], [[context-window|Context Window]], and [[search-retrieval|Search / Retrieval]] to place Chunking in the surrounding system. * Follow the cited source material above for the method, standard, or research context behind this draft.