Chunk models

A chat model that cuts a document by topic, or writes a note placing each chunk. See Chunking.

vectrixdb.chunk_models

Chunking that asks a model: where each topic starts, what each chunk is about, and every token read in its document.

Three ways of cutting and embedding that the plain strategies cannot do on their own, each given to :meth:vectrixdb.Vectrix.add_document:

llm_cutter(chat) For chunk="llm". The model reads the paragraphs in order, numbered, a window of them at a time, and says which start a new topic; each topic is one chunk, split to chunk_size where it runs longer. A reply that is not the numbers asked for cuts that window by size alone.

context_writer(chat) For context_with=. For every chunk the model reads the document around it and writes a sentence or two placing it there: the document's subject, the section, what it refers to. The note goes in front of the chunk for the embedder only, the way embed_heading puts the heading path there, and is kept as _vx_context. One call a chunk, so it costs what the collection's size says.

late=True Late chunking. The document is read by the embedding model token by token, in windows as long as the model reads, overlapping by a quarter, and each chunk's vector is the mean of its own tokens, each read with the words around it. A question is embedded the usual way, which is the mean of its tokens only for a model that pools the mean: such a model, or one with embed_tokens, is needed. The bundled bge-small pools its first token, so it cannot.

chat is a :class:vectrixdb.evaluation.ChatWriter, or any callable from chat messages to the model's text. What is sent is marked as data, with anything that could pass for a chat template's control tokens made inert, so a document cannot give the model instructions. A model that refuses, or goes quiet three times running, stops with WriterUnavailable.

context_writer(chat, *, around=8000)

What context_with= asks for each chunk: chat reads about around characters of the document and places it.

The document's opening goes with it when the window starts later, since that is where a title and a subject usually are.

late_ready(model)

None when this model can embed late; otherwise why it cannot, in words.

late_vectors(model, text, spans)

One vector a chunk, from the document read whole: the mean of the chunk's own tokens, normalised.

spans are the chunks' (start, end) offsets in text. The text is cut at every chunk's edges into pieces, read in order, and each chunk is the pieces inside it, so chunks that overlap share tokens. A chunk with no token of its own is embedded alone.

llm_cutter(chat, *, window=12000)

What chunk="llm" asks where each topic starts: chat reads window characters of paragraphs at a time.

The same paragraphs asked about twice are answered from what the model said the first time, so building the same document at several sizes asks once.