Ingest

Reading files into text, and cutting text into chunks. See Ingest documents and Chunking.

vectrixdb.ingest

Ingestion: loaders, chunkers, near-duplicate detection and the embedding cache.

Retrieval quality is decided here, before any vector exists. A chunk that cuts a sentence in half embeds as noise; a section returned whole reads as an answer; the same paragraph indexed twice pushes a better one off the page.

The pieces, each usable on its own:

• :func:load turns a file (PDF, DOCX, PPTX, XLSX, CSV, HTML, Markdown, text) into a :class:LoadedDocument with page and heading positions, and hands any suffix somebody registered an extractor for to that extractor (see :mod:vectrixdb.extract).
• :func:chunk splits text by one of four strategies and returns :class:Chunk objects that know their offsets, page and heading.
• :class:NearDuplicateIndex is MinHash over word shingles, persisted beside the collection, so add(dedupe=0.9) can skip a paragraph it already has.
• :class:EmbeddingCache keys vectors on content and model, so re-ingesting a folder embeds only what changed.
• :class:ParentStore keeps the enclosing sections that parent-child retrieval returns in place of the small chunk that matched.

Vectrix.add_document and Vectrix.search(parents=True) wire them together; nothing here imports the model, so the module is cheap to load.

EXTRACTED_MARKER = 'extracted' module-attribute

STRATEGIES = ('recursive', 'sentence', 'semantic', 'markdown', 'fixed', 'llm') module-attribute

Chunk dataclass

A piece of a document that knows where it came from.

LoadedDocument dataclass

Text with the positions that give chunks their page and heading.

page_label(page)

The number printed on page, when the file says it is not page itself.

figure_at(offset)

The figure whose line starts exactly here, if there is one.

to_markdown(extra=None)

The document as one Markdown file that reads back exactly.

Front matter carries what plain Markdown cannot: where the pages break and the number printed on each, the headings as they were found, the figures and the metadata. extra adds keys for a person reading the file, a document id, where it came from, who extracted it; they are written first and ignored on the way back in. The vectrixdb: extracted line is what stops this file ever being read as a new original.

from_markdown(text) classmethod

Markdown, with or without front matter, as a document.

A file this library wrote comes back as it was, offsets and all, and is not normalised a second time. Any other Markdown is read the ordinary way, and its front matter, doc_id, audience, client_id, becomes the document's metadata and so every chunk's: that is where the fields a policy decides by can come from.

heading_path_at(offset)

Every heading the offset sits under, outermost first.

with_page_markers()

The text with --- Page N --- lines, for build_tree_from_pdf.

PreparedDocument dataclass

Everything about a document that is decided before it is written: its chunks, their ids and their metadata, the sections parent retrieval returns, its version and how it scored.

NearDuplicateIndex

MinHash signatures with LSH bands, saved as one JSON file.

Jaccard similarity of word 3-shingles, estimated from permutations hashes. Banding (bands bands of permutations // bands rows) makes the candidate lookup a dict hit rather than a scan, and the estimate is then checked against every candidate. Sized for a single node: a hundred thousand signatures of 128 hashes is a few tens of megabytes on disk.

query(text, threshold=0.9, signature=None)

The most similar stored document at or above threshold, or None.

EmbeddingCache

Vectors keyed on (model, content hash) in one SQLite file.

Re-ingesting a folder where three files changed embeds three files. The key includes the model name, so switching models never serves a stale vector, and the dimension is stored with each row for safety.

ParentStore

The enclosing sections that small chunks point back to.

Parents are never searched; they are returned. Three places they can live, and which one is right depends on who reads them:

• a SQLite file beside the collection, the default, so they survive restarts and travel with export();
• Cosmos DB, given cosmos://<account>.documents.azure.com/<database>/<container>, when more than one process reads them. That is not a nicety on a deployment that scales out: parents written by the instance that ingested a document are invisible to the instance serving the search if they sit on a local disk, and the symptom is parent-child retrieval quietly returning nothing rather than an error anywhere;
• in memory, when there is nowhere given, which is fine for one process that does not restart and wrong for anything else.

The SQLite one makes its own folder. It used not to, and a collection opened on a remote backend never built that folder itself, so the first document into a fresh deployment died on unable to open database file from three frames inside an unrelated-looking call.

where property

Where the parents are, for a log line on a cold start.

chunk(text, strategy='recursive', size=1000, overlap=200, *, embed=None, threshold=0.1, cut_with=None)

Split text into chunks by strategy, each with offsets, page and heading.

Strategies:

• recursive: split on paragraphs, then lines, then sentences, then words, merging pieces up to size characters with overlap characters carried between neighbours. The general-purpose default.
• sentence: whole sentences packed up to size, overlapping by the last sentence of the previous chunk. Never cuts mid-sentence.
• semantic: sentences grouped where they are about the same thing. Needs embed. Every sentence is embedded, and the threshold fraction of boundaries with the lowest similarity between neighbours start a new chunk (0.1 splits at the ten percent sharpest topic changes); size is the hard cap. Costs one embedding per sentence.
• markdown: every heading starts a new chunk and the heading text rides along on each chunk under it; long sections fall back to recursive within the section.
• fixed: every size characters, overlap carried over, with no regard for paragraphs or sentences; only a word is never cut, the window ending at the space before it.
• llm: a model reads the paragraphs in order and says where a new topic starts, and each topic is a chunk, split to size where it runs longer. Needs cut_with, a callable from the text and its paragraphs' (start, end) offsets to the paragraph numbers that start a chunk; :func:vectrixdb.chunk_models.llm_cutter makes one from a chat model.

Pass a :class:LoadedDocument and every chunk carries its page and heading; pass a string and only offsets are known.

load(source, kind=None, extractors=None, images=False, ocr=None, page_reader=None, every_page=False)

Read a file into text with page and heading positions.

kind overrides the extension: one of pdf, docx, doc, pptx, xlsx, csv, html, markdown, text, image, audio, video. The last three are read by the local engines in :mod:vectrixdb.extract.engines and need an extra each. PDF needs pypdf (or PyMuPDF as fitz), DOCX needs python-docx; the others use the standard library. A missing parser is a :class:DependencyError naming the pip install, not an ImportError from three frames down.

extractors is a mapping of suffix to callable, an :class:~vectrixdb.extract.ExtractorRegistry or an :class:~vectrixdb.extract.HttpExtractor. A suffix somebody registered an extractor for goes to that extractor, including a suffix the built-in readers know; passing kind asks for a built-in reader by name and skips them.

images=True asks for the pictures in a PDF, a Word document, a PowerPoint deck and an Excel workbook as well. Each becomes a figure where it sits, with its bytes in doc.images: at the end of its page in a PDF, p4-fig1; after its paragraph in Word, fig3; at the end of its slide in PowerPoint, s2-fig1; after its sheet's rows in Excel, sheet2-fig1. No caption yet, which is why this is for a caller that has something to describe them with.

ocr reads the pages of a PDF that have no text layer: a callable of one page drawn as a PNG, returning its lines in the order a person reads them, which is what :func:~vectrixdb.extract.layout.reading_order is for. Only pages under min_chars are drawn and only those are sent, so a report that is text throughout costs nothing and never calls it; a page with no ink is not sent either, because a reader shown an empty page invents text for it. This is the one way to have both halves of a PDF: registering an extractor for .pdf replaces this reader and takes the figures with it, and ocr leaves the figures where they were.

page_reader reads a PDF's pages by sight, the way a person does: a :class:~vectrixdb.extract.page_reader.PageReader, given a picture of each page the rules are unsure of, a chart, big figures set apart from their labels, a page whose file draws its parts out of order, with the page's own words, and answering the page in Markdown. Every number it writes is held to the page: one the page does not print is taken out, and a reading with too many is not taken. every_page gives it every page with words on it rather than the unsure ones alone.

load_bytes(data, name, kind=None, extractors=None, source=None, images=False, ocr=None, page_reader=None, every_page=False)

:func:load for bytes that are not a file here: an object fetched from a bucket, an upload. name picks the reader by its suffix and is what the document is called; source is where it came from, a URI.

An extractor is handed the bytes as they are. A built-in reader wants a file, so the bytes go to a temporary one that is removed afterwards.

images keeps the pictures inside the document, which is what a describer and an image embedder need. It costs nothing to leave off and a decode of every picture to turn on, so it is off unless asked for. Without it a PDF, a Word document, a deck or a workbook has no figures at all, not merely figures without descriptions: a document fetched from a bucket used to lose them in silence, whatever the collection was opened with.

ocr reads a PDF's pages that have no text layer, and page_reader the pages the rules are unsure of, by sight; see :func:load.

markdown_document(text, pages=None, headings=None, metadata=None, figures=None)

Markdown as a document: images become figure lines, pipe tables become rows, and any page, heading or figure offsets given move with the text.

This is what a .md file, a string handed to add_document and an extractor's reply all go through, so a table or a figure means the same thing wherever the Markdown came from.

normalise_figures(text)

Markdown images as figure lines on their own paragraph, each with the description from the blockquote under it, when there is one.

normalise_tables(text)

Markdown pipe tables as one line per row, Header: value; Header: value.

The same rendering a spreadsheet row gets, for the same reason: a row of pipes is noise to a sentence embedding, and Region: EMEA; Revenue: 1200 reads like a sentence and keeps every cell an exact keyword. Tables inside a code fence are left alone.

rows_to_lines(rows)

Tabular rows as one line each, Header: value; Header: value.

The first row is the header when every filled cell in it is text. A header turns a row into something a sentence embedding can read and a keyword search can hit: Region: EMEA; Revenue: 1200 rather than EMEA 1200. Empty cells are left out, so a sparse row says what it holds and nothing about what it does not.

split_front_matter(text)

(front matter, body). No front matter is an empty dict.

Reads what people write at the top of a Markdown file: key: value lines, values in JSON or bare, [a, b] lists and block lists of - item lines. Anything nested deeper is left out rather than guessed at. The body starts after the closing rule.

prepare_document(doc, doc_id, *, chunk='recursive', chunk_size=1000, overlap=200, parent_size=None, metadata=None, embed_heading=False, threshold=0.1, embed=None, quality_threshold=None, cut_with=None, context_with=None)

Cut a document and work out what every chunk carries. None when there is nothing to cut.

Nothing is written and nothing is embedded, apart from what semantic chunking asks embed for. Vectrix.add_document and the REST server's documents route both call this, so a chunk means the same thing whichever way it came in: the same ids, the same citation, the same lineage stamps, the same figure links.

cut_with is what chunk="llm" asks where each topic starts. context_with(document, chunk, start, end) writes a note that places a chunk in its document, which goes in front of the chunk for the embedder, and is kept as _vx_context; the stored text is the chunk as written. :mod:vectrixdb.chunk_models makes both from a chat model.

texts_to_embed(texts, metadata)

The strings the dense model sees: a chunk's text, with whatever context its metadata says belongs in front of it or after it. The stored text never carries that context.

describe_figures(doc, describer, *, name='', skip_decorative=True, repeated_on=3)

Put a description under every figure whose image is at hand.

describer takes the image bytes and a context, caption, name, page, before and after (the text either side), and returns the description as a string, or a mapping with any of caption, description, table (rows, for a picture of a table, rendered the way a spreadsheet row is) and decorative. None leaves the figure as it is. A figure that already has a description is not described again, which is what makes this free on a kept document.

Decorative images are taken out of the text altogether: small ones, long thin ones, and an image that turns up repeated_on times or more, which is a letterhead and not a figure.

image_size(data)

(width, height) of a PNG, JPEG, GIF, BMP or WebP, read from its header with nothing but struct; None for anything else.

is_decorative(data, min_bytes=1500, min_side=48, max_aspect=8.0)

A logo, an icon, a rule, a signature: too small to be a figure, or far too long and thin. Costs nothing to tell, which matters because the alternative is paying a vision model to describe a letterhead.