Evaluation

Golden questions, and every setup scored on them. See Measure retrieval and Evaluate every setup.

vectrixdb.evaluation

Measure retrieval, and answers, against questions whose answers are known.

Two layers, and they answer different questions.

Retrieval: for each question, did the documents that hold the answer come back, and how high? Recall at k, MRR and nDCG over the ids a person marked as right. It is cheap, it needs no model, and it is the number to watch while changing a chunker or an embedder, because it moves when retrieval moves and for no other reason.

Answers: given the answer a model wrote, how good is it? That needs a judge, which is a callable you bring, and it is scored on the four names Amazon Bedrock's managed RAG evaluation uses, so the two can be read side by side: correctness, completeness, faithfulness, helpfulness.

Questions come from your own JSONL, or from the prompt dataset a Bedrock evaluation job takes (conversationTurns with a prompt and reference responses). A Bedrock file carries reference answers and no ids, so retrieval cannot be scored from it until somebody says which documents are right: :func:suggest_expected proposes them for a person to check, and proposes only. The file can be local, s3://bucket/key, or a Blob address, and :func:golden_template writes one to fill in, a row per document it samples, with the answer's document already named.

Setups: which way of searching should a collection use? :func:evaluate searches every golden question with every setup the collections can do, every engine, every set of vectors and every method, times each search, ranks the setups and names three: finds_the_most, best_for_balance (the fastest within 4 points of it) and best_for_time (the fastest within 8). It saves the run where the Evaluate pages read it, and switches nothing: a person chooses.

Two things a golden file also settles. :func:answer_cutoff measures the relevance below which the top result is more likely a near miss than the answer, which is the number a program needs before it declines to answer. :func:sweep compares what :func:evaluate cannot, because it is decided at ingestion: the chunker, the chunk size and whether the headings were embedded. It builds the index again for each, in a scratch folder.

Chunking techniques. :func:compare_chunking asks what :func:sweep cannot: with the same characters handed to the model, which way of cutting gets the most questions answered right? Structure-aware, parent-child, recursive, sentence, semantic and fixed-size, each at three sizes, headings embedded and not; with a model, LLM-based cutting too, and each technique with a note from the model in front of every chunk; and late chunking where the embedding model can. Each technique at its best, compared question by question, a gap that could be luck called a tie. It saves the run where the Chunking tab reads it.

Written questions. :func:write_golden drafts a golden file with a model, the way DeepEval's synthesizer does, run on the collection's own chunks: passages spread across the sections and scored by a critic, questions of several kinds, each quote found on its page, copied wording and repeats turned down, and every row a draft for a person to check.

ADDS = {'headings': 'h', 'none': 'n', 'context': 'c', 'late': 'l'} module-attribute

ANSWER_METRICS = ('correctness', 'completeness', 'faithfulness', 'helpfulness') module-attribute

GOLDEN_SCHEMA = {'$schema': 'https://json-schema.org/draft/2020-12/schema', 'title': 'VectrixDB golden question', 'description': 'One line of a golden file: a question, where the answer is, and the answer a person would accept. A row golden_template wrote and nobody has filled in yet has an empty question and a hint.', 'type': 'object', 'required': ['question', 'expected'], 'additionalProperties': False, 'properties': {'id': {'type': ['string', 'null'], 'minLength': 1, 'description': "The question's name in reports. Numbered by its line when left out."}, 'question': {'type': 'string', 'description': 'What a person would ask. Empty only on a template row, which has a hint.'}, 'expected': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string', 'minLength': 1, 'pattern': _EXPECTED}, 'description': "Where the answer is: a document by its id, td/report.pdf, or one page of it, td/report.pdf#page=41, the page's place in the file as its citation gives it."}, 'reference': {'type': 'string', 'description': 'The answer a person would accept, for scoring written answers.'}, 'evidence': {'type': 'array', 'items': {'type': 'string', 'minLength': 1}, 'description': 'The words on the expected pages that answer the question, exactly as they are there, one quote an item: what the golden writer checked each question against. A run can then ask whether these words came back, not only a piece of the right page. A quote of fewer than three words is not used.'}, 'hint': {'type': 'string', 'description': 'Where the answer is, for a person: on a template row, how the document or page starts, for whoever writes the question; on a written draft, the page and the words that answer it, for whoever checks it.'}, 'draft': {'type': 'boolean', 'description': 'True while a person has not checked a row that a model or suggest_expected wrote.'}}, 'if': {'properties': {'question': {'pattern': '^\\s*$'}}, 'required': ['question']}, 'then': {'required': ['hint']}} module-attribute

PICKS = ('finds_the_most', 'best_for_balance', 'best_for_time') module-attribute

TECHNIQUES = {'markdown': {'name': 'Structure-aware', 'chunk': 'markdown', 'parents': False, 'cuts': 'At headings, then long sections'}, 'parent': {'name': 'Parent-child', 'chunk': 'markdown', 'parents': True, 'cuts': 'Small chunks find, their sections are handed over'}, 'recursive': {'name': 'Recursive', 'chunk': 'recursive', 'parents': False, 'cuts': 'Paragraphs, then lines, sentences and words, until a piece fits'}, 'sentence': {'name': 'Sentence', 'chunk': 'sentence', 'parents': False, 'cuts': 'Whole sentences, up to the size'}, 'semantic': {'name': 'Semantic', 'chunk': 'semantic', 'parents': False, 'cuts': 'A new chunk where the meaning shifts'}, 'fixed': {'name': 'Fixed-size', 'chunk': 'fixed', 'parents': False, 'cuts': 'Every so many characters, at a space'}, 'llm': {'name': 'LLM-based', 'chunk': 'llm', 'parents': False, 'cuts': 'A model says where each topic starts', 'needs': 'cut_with'}} module-attribute

ChatWriter

Bases: ChatRoute

A chat model that writes golden questions, and by default judges them too.

Any service with an OpenAI style chat completions route: Azure OpenAI, a model on Azure AI Foundry such as Phi-4-mini, OpenAI, or a server on your own machine such as Ollama. It is called with chat messages and answers with the model's text, or None when the service gave none, and :attr:failure then says what the service said. :meth:from_environment reads AZURE_OPENAI_WRITER_DEPLOYMENT with AZURE_OPENAI_ENDPOINT, or VECTRIXDB_WRITER_URL with VECTRIXDB_WRITER_KEY, VECTRIXDB_WRITER_MODEL and VECTRIXDB_WRITER_KEY_HEADER.

ChunkingStore

Bases: ReportStore

Where chunking runs are kept, beside the retrieval runs: chunking/runs/<id>/report.json.

The same store as :class:~vectrixdb.evaluation.ReportStore, a folder, a bucket or a container, whose retrieval runs are in retrieval/runs/, and the same golden files beside them, kept once a version.

Golden dataclass

A golden file as read: its questions, where it came from, and its fingerprint.

sha256 is of the bytes, so two saves of the same file are the same golden data and a function watching the file can tell a real change from a save that changed nothing. unfilled counts template rows nobody has written a question for yet; drafts counts questions a writer drafted and a person has not yet marked as checked. raw is the bytes as read, which a saved run keeps once a version so its questions can be downloaded later; questions made in memory have none.

GoldenCheck dataclass

What :func:check_golden found. ok when nothing stands in the way of a run.

rows counts the lines with anything on them; ready the questions that can be scored, unfilled the template rows still waiting for a question and drafts the questions nobody has checked. golden is the file as :func:read_golden reads it, there only when it is ok.

summary()

The problems, then what is worth knowing, one a line, as a person reads them.

GoldenProblem dataclass

One thing wrong with a golden file, or worth knowing about it, and the line it is on.

GoldenWriting dataclass

What :func:write_golden wrote, and what it passed over on the way.

rows are the golden file's rows; kinds counts them by kind and evolved how many were made harder. asked is the model calls made and reused the answers taken from the cache instead. passed_over counts the passages never asked about, by reason, and turned_down the questions asked for again, by reason; set_aside is how many places never gave a question that passed. unjudged counts questions the critic gave no verdict on, kept on the checks alone.

summary()

What was written and what was not, a few lines a person reads.

MissingDocumentsWarning

Bases: UserWarning

Golden questions expect a document a collection does not hold, so no setup can find them.

Question dataclass

One question, the answer a person would accept, the ids of the documents, or chunks, that hold it, and the words there that answer it.

evidence is those words, exactly as they are on the page: the golden writer keeps the quotes each question was checked against. With them a run can ask whether the words that answer came back, not only a piece of the right page.

ReportStore

Where runs are kept: one file a run, retrieval/runs/<id>/report.json.

The report holds everything: every setup's numbers, the three picks and the rules they were chosen by, so nothing is written beside it. The one other thing kept is the golden file itself, once a version and not once a run, as golden_dataset/<sha256>.jsonl, beside the golden dataset when that is kept in the same store: the report names the questions by their fingerprint and never holds their text. Chunking runs are kept in the same store under chunking/runs/, by :class:~vectrixdb.evaluation.ChunkingStore.

A run saved before 2.2 is at runs/<id>/report.json, and a golden file kept before at golden/<sha256>.jsonl. Both are still read, and nothing new is written to either.

files is anything with read, write, exists and list, the same small interface the document store uses, so a folder (:class:vectrixdb.documents.LocalFiles), a bucket (:class:~vectrixdb.documents.S3Files) or a container (:class:~vectrixdb.documents.BlobFiles) all work. A path is a folder.

save(report, golden=None)

Write a run's report and return its id.

golden is the golden file's bytes as they were read. They are kept once per version, before the report, so a run's own questions can be downloaded after the file has changed, and only when they are the bytes the report's fingerprint names.

golden(sha256)

The golden file a run used, by its fingerprint; KeyError when it was not kept.

has_golden(sha256)

Whether this version of the golden file was kept. A kept one never goes, so a yes is remembered.

ids()

Every run that has a report, newest first.

get(run)

One run's report. "latest" is the newest; a missing run raises KeyError.

history(limit=20, offset=0)

Newest first: each run's id, time, golden file and every setup's top 10 count.

Small on purpose: it is what "run by run" is drawn from, and what the list of runs to open shows. offset skips the newest, for the next page. Each run's report is read once per process and what the line needs is kept, because a saved run never changes.

Target dataclass

One handle to evaluate, and the name people will read for it.

name defaults to the engine's own name ("VectrixDB", "Azure AI Search"); give one when two targets share an engine, two indexes built with different settings, say. collection is the name the server knows this collection by, so the Evaluate page can open it in Search; it defaults to the handle's own name. models relabels a vector: {"vectrixdb": {"label": "Hugging Face", "name": "bge-large-en-v1.5", "kind": "hf"}}.

WriterUnavailable

Bases: RuntimeError

The model stopped answering, or never did: a key it refuses, a deployment that is not there, a server that is down.

answer_cutoff(db, questions, *, unanswerable=(), by='doc', measure='similarity', **search)

The relevance below which the top result is more likely a near miss than the answer.

Every labelled question is searched, and its top result is either the answer, one of the documents the question expects, or a near miss: the best the collection had, and wrong. unanswerable is questions a person knows the documents do not answer; whatever comes top for one is a near miss by definition, and they are what makes the cut-off honest about questions from outside the documents, so give some.

measure is the attribute of a result that is read: similarity, the cosine alone, which means the same in every mode, or relevance, which is the reranker's verdict where one ran. search goes to db.search, and the cut-off belongs to exactly that way of searching.

The reply holds cutoff: the value that best separates the two, by the share of answers kept plus the share of near misses declined. Beside it, separation, how often an answer outscores a near miss, where 0.5 is a coin and nothing under about 0.75 is worth acting on; table, the trade at each candidate, to choose a stricter or a looser one from; and at_cutoff. With fewer than five answers or five near misses cutoff is None and reason says which: a number from three examples is not a measurement.

answer_report(questions, answers, judge, metrics=ANSWER_METRICS)

Score written answers with a judge you bring.

answers maps a question's id to the answer text, or to a mapping with answer and sources. judge(metric=, question=, reference=, answer=, sources=) returns a number from 0 to 1; anything outside that is an error, because a judge that returns 7 has not understood the question and averaging it in would hide that. A question with no answer is counted and not scored.

answer_with(chat)

An answer for :func:compare_chunking: the chat model answers from the passages alone.

chat is a :class:~vectrixdb.evaluation.ChatWriter, or any callable from chat messages to the model's text. A model that refuses, or goes quiet three times running, stops the run with WriterUnavailable.

build_report(results, golden, *, question_ids=(), targets=None, missing=None, balance=4, time_points=8, created=None)

One run, as the pages and the Function App read it.

golden describes the questions: source, sha256, questions, labelled and unfilled (template rows nobody has written a question for yet). targets describes what was searched, per engine, so a later run can tell whether the index changed between them. missing is what :func:vectrixdb.evaluation.missing_documents found: by target, the expected documents it does not hold.

check_golden(source, db=None, *, fetcher=None)

Every way a golden file falls short of :data:GOLDEN_SCHEMA, found before a run rather than in one.

Every line is checked, so a file with five mistakes is put right in one go: a line that is not JSON, a field the schema does not have (a typo like expeted would otherwise drop the row from the scores without a word), a question or expected missing or empty, a value of the wrong kind, a page number that is not one. Then the file as a whole: an id used twice, the same question twice, and whether any question is ready. Those are errors, and a run should not start with any.

warnings do not stop a run: template rows still waiting for a question, drafts nobody has checked, and, given db, the documents the collection does not hold, which no setup can find (see :func:missing_documents). A Bedrock prompt dataset names no documents at all, so it cannot be scored for retrieval until it is labelled, and the check says so. source is what :func:read_golden takes, and fetcher too.

No file there at all is not an error to read: it is missing, with one problem that says how to get questions, whatever holds the file. A store's own "not found" (a page of XML from Blob storage) is not put in front of a person.

choose(setups, balance=4, time_points=8)

The three picks, by key.

finds_the_most finds the right answer in the top 10 for the most questions; a tie goes to the one with more right answers first, then to the faster. best_for_balance is the fastest setup within balance percentage points of that, and best_for_time the fastest within time_points. Within is inclusive: at 4 points, 92% counts when the most is 96%. All three can be the same setup, and when nothing ran they are all None.

chunking_build(key)

The build a key names: markdown-1000-h is Structure-aware at 1,000 characters, its headings in front of each chunk.

A key is <technique>-<size>-<letter>: a technique of :data:TECHNIQUES, a size of at least 50 characters, and what goes in front of each chunk, h its headings, n nothing, c a model's note, l embedded late. A size a run does not try is a build too, so a person can name one. ValueError for anything else, saying what a key is.

chunking_choice(results, current=None, *, alpha=ALPHA)

Which build a run picks, given the one in use: {"pick": key, "switch": bool, "why": words}.

The run's best build, ranked as :func:chunking_report ranks them. The one in use stays when it was built too and the questions only one of the two got right split the way luck could, because every switch cuts and embeds every document again. results are the builds as they were made, with their outcomes; a report keeps none, so it cannot decide this.

chunking_options(build, *, cut_with=None, context_with=None)

What add_document and rechunk are given to cut a document the way a build does.

build is a key or a plan's entry. LLM-based needs cut_with, and a build with the model's note context_with; :mod:vectrixdb.chunk_models makes both from a chat model. ValueError when one is needed and not given. Every choice a build makes is given, late too, so rechunk(), which keeps what it is not given, keeps nothing of the cut a document had.

chunking_plan(techniques=None, *, sizes=SIZES, child_sizes=CHILD_SIZES, headings=(True, False), context=False, late=False)

Every build a run makes: each technique at each size, headings embedded and not.

A build's key names it, markdown-1000-h for Structure-aware at a thousand characters with its headings embedded. The overlap is a fifth of the size. Parent-child is cut at child_sizes, because its chunks only find and its sections are what the model reads. techniques left out is every one that needs no model. context adds a build of each technique at its middle size with a model's note in front of every chunk, -c, and late one embedded late, -l.

chunking_report(results, golden, *, budget=BUDGET, search=None, alpha=ALPHA, created=None, skipped=None)

One run, as the Chunking tab reads it: every build, every technique at its best, and the pick.

skipped says what the run left out and why: a technique that needs a model nobody gave, late chunking with a model that cannot.

scored is answered when a model answered and a judge checked, else found, and every count on the page is of that. A build ranks above another on it, and on fewer chunks when they are equal. Each technique is its best build, and against holds, for each other technique, the questions only one of the two got right and whether a split like that could be luck. tie marks a technique that could be as good as the best. The questions' outcomes stay out of the report: it holds counts, never which question was missed.

chunking_store(where)

Where chunking runs are saved and read: the places :func:~vectrixdb.evaluation.report_store takes.

compare_chunking(documents, golden, *, techniques=None, sizes=SIZES, child_sizes=CHILD_SIZES, headings=(True, False), budget=BUDGET, search=None, chat=None, answer=None, judge=None, save_to=None, fetcher=None, open_with=None, workdir=None, workers=8, alpha=ALPHA, progress=None, cut_with=None, context_with=None, late=None)

Compare the chunking techniques on the golden questions and pick the one that answers the most.

documents is what the collections hold, as originals or as the Markdown the library kept: {doc_id: path or text} for one collection, {collection: {doc_id: ...}} for several, or a list of paths. Kept Markdown keeps its pages, so a question labelled with one page is scored on that page. golden is a path, an s3:// or Blob address, a :class:~vectrixdb.evaluation.Golden, or a list of questions.

chat is a model to answer with and to judge with; answer and judge pass your own instead, and either way both are needed. Without them the run counts what was found within budget characters. The same model cuts the LLM-based technique and writes a note for every chunk of a build at each technique's middle size, unless cut_with and context_with give others, from :mod:vectrixdb.chunk_models; without a model those are left out and the report says so. late left out adds a build embedded late where the model can. Every build of :func:plan_chunking is made one after another, so their times compare with each other. With save_to the run is saved where the server's Chunking tab reads it, beside the evaluation runs. Nothing is switched: the pick is for a person.

describe_target(target)

What a report says about one target: its engine, what it holds, how it was opened.

Two runs whose targets read the same were searching the same index, so a change here between runs is drawn on the run-by-run line.

evaluate(targets, golden, *, save_to=None, only=None, skip=(), balance=4, time_points=8, by='doc', fetcher=None, progress=None)

Search every golden question with every setup, rank the setups, and pick three.

targets is a collection handle, a list of them, a mapping from a name to one, or :class:Target objects: the same documents on each engine you want compared, VectrixDB here and an Azure AI Search index, say. A handle offers what it was opened for, so open local collections with mode="ultimate" to include every method. golden is a path, an s3:// or Blob address, a :class:Golden, or a list of questions.

The report ranks every setup by how many questions found the right answer in the top 10, then by how many found it first, then by speed, and names finds_the_most, best_for_balance (the fastest within balance points of the most) and best_for_time (the fastest within time_points). With save_to, a folder, an s3:// or Blob address or a store, the run is saved there, which is what the server's Evaluate pages read. only limits the run to these setup keys. skip leaves out every setup of these methods, ("hybrid_reranked",) say, for a reranker that runs on a host too small for it: on Azure Flex Consumption the cross-encoder took 13 to 19 seconds a search and found less than the same search without it. by="evidence" counts a result right only when it is a chunk that holds the words the golden row quotes as answering, where the row has them, rather than any chunk of the right page: see :func:retrieval_report. Nothing is switched: the picks are for a person to choose between.

Before any searching, every expected document is looked up on every target (see :func:missing_documents). One a target does not hold is warned about and kept in the report's missing; the questions stay in, so the scores say what searching that index really finds.

golden_template(db, path=None, n=50, *, seed=0, writer=None, by='doc')

Rows of a golden file to fill in: one sampled document each, its id already in expected.

A person reads the hint, the start of the document, and writes a question that document answers, and a reference answer if answers are to be scored too. Rows left empty are skipped when the file is read, so it can be filled a few at a time and evaluated as it goes.

writer is a callable you bring, a model call say, from a document's text to a question. What it writes is marked "draft": true for a person to check and unmark; a question a model wrote about a document it just read is easier than the questions people ask.

The sample is n documents chosen with seed, so the same seed on the same collection gives the same rows. Written as JSONL to path when one is given, and returned either way. db may be a list of handles, several collections sampled together into one golden file.

by="page" gives a row a page instead, expected naming the page, report.pdf#page=41, and hint saying which page it is, what it is printed as, the heading it sits under and how it starts. A report of two hundred pages is one document, so every setup finds "the document" for every question about it; the page is what tells them apart. The n pages are spread evenly through the collection's pages from a start seed picks, so every part of a long report is asked about, and pages with next to no text on them, covers and dividers, are left out. A document with no pages to give, a recording or a picture, keeps its one row. Pages come from the kept Markdown, so the collection keeps its documents (keep_source).

handed_over(db, question, build, *, budget=BUDGET, search=None)

What a question is handed from a collection cut the way build does, as a run measured it: [(result, text), ...].

Enough results are asked for to fill budget characters at the build's size, a Parent-child build's sections rather than its chunks, and they are handed over best first, the last cut to fit. search is the rest of what search() is given, the way a retrieval run picked say; left out, it is the way a run searches every build, hybrid with no reranker. A host that answers from the build a run picked hands its model this, and so reads what was measured.

judge_with(chat)

A judge for :func:compare_chunking: the chat model says whether an answer matches the golden one.

late_possible(open_with=None)

None when the scratch collections a run opens can embed late; otherwise why not, in words.

load_bedrock_jsonl(path, fetcher=None)

Questions from a Bedrock RAG evaluation prompt dataset.

One line is one conversation; every turn with a prompt becomes a question, and its reference responses become the reference answer. The file has no document ids, so expected is empty until it is filled.

load_questions(path, fetcher=None)

Questions from this library's JSONL: question, reference and expected per line. A Bedrock file is recognised and read as one.

path is a file, s3://bucket/key or a Blob address; see :func:read_golden, which also says how many template rows are still waiting for a question. Those rows are left out here.

missing_documents(targets, questions, *, by='doc')

The expected documents each target does not hold, and the questions that leaves unfindable.

A document removed, renamed or never added is a miss on every setup: it lowers every score and says nothing about the setups, so this runs before any searching and warns with :class:MissingDocumentsWarning. Each expected id is looked up by its own id and its first chunks' (id:0 to id:2), in the collection and then in its store, which is how a handle opened only to search an index another process filled finds them, and last by the _vx_doc its chunks carry.

targets takes what :func:evaluate takes. The answer is what a report keeps, by target name, and names only the targets with something missing: documents, every expected id not found, and questions, the questions none of whose documents was found. A target the check could not run on has error instead. Empty when everything is there.

stale_evidence(targets, questions)

The questions whose evidence no chunk of their documents holds any more, by target name.

Evidence is written from the chunks as they were. A document read again since, with a better reader or other chunking, may word its pages differently, and a quote no chunk holds counts every setup wrong on that question however well it searched. This looks each question's quotes up in the chunks of its expected documents, in the collection's own record of what it indexed, and names the questions none of whose usable quotes is there. A document the target does not hold at all is :func:missing_documents' to report, and is left out here; a target whose chunks cannot be read is left out too. Empty when every quote is still on its pages.

plan_chunking(*, chat=None, cut_with=None, context_with=None, late=None, open_with=None, techniques=None, sizes=SIZES, child_sizes=CHILD_SIZES, headings=(True, False))

The builds a run can make with what it was given; what it leaves out, and why; and the two models it cuts and notes with.

With chat, the model cuts for LLM-based and writes the notes, unless cut_with or context_with give others. Without one, both are left out and said to be. late left out is late chunking where the scratch collections' model can, and said not to be where it cannot; True asks for it whatever, and False leaves it out.

read_golden(source, fetcher=None)

Read a golden file from a path, s3://bucket/key or a Blob address.

Our JSONL and Bedrock's prompt dataset are both read. Rows that :func:golden_template wrote and nobody has filled in are left out and counted, so a file half written can still be evaluated for the half that is.

report_store(where)

Where runs are saved and read: a folder, s3://bucket/prefix, a Blob address, or a store.

A Blob address is https://<account>.blob.core.windows.net/<container>/<prefix>. Anything with read, write and list is used as it is, which is how a store around a client you built is passed.

retrieval_report(db, questions, k=(1, 3, 5, 10), by='doc', **search)

Recall at k, MRR and nDCG at the largest k, over the labelled questions.

by="doc" counts a hit when any chunk of an expected document comes back, which is the right grain when labels name documents, and when an entry names one page of one, report.pdf#page=41, a chunk that covers that page; the results are then ranked by the pages they cover rather than by document. by="chunk" wants the chunk ids themselves. by="evidence" ranks chunks, and counts one that holds the words the golden row quotes as answering, so the right page is not enough where the page is long; a question with no evidence is counted by its documents and pages. search is passed to db.search, so the same questions can be asked with mode=, vectors=, rerank= and the rest, and the reports compared. Questions with no expected are counted and left out; a report over none of them says so rather than reporting a perfect score.

run_chunking_build(documents, questions, build, *, budget=BUDGET, search=None, answer=None, judge=None, open_with=None, workdir=None, workers=8, cut_with=None, context_with=None)

One build of :func:chunking_plan: the documents cut its way, every question asked, what came back scored.

An LLM-based build needs cut_with, and a build with the model's note context_with: :mod:vectrixdb.chunk_models makes both.

Each collection in documents is built in a scratch folder of its own and asked only the questions whose documents it holds. For each question the results are handed over best first until budget characters, and found counts the questions whose expected page was among them. With answer and judge, the model answers from those characters alone and the judge checks the answer against the golden one; a question with no golden answer is then left out, because there is nothing to check it against.

The reply carries every question's outcome, outcomes, so two builds can be compared question by question, which is what :func:chunking_report needs. The scratch folders are removed after.

run_setup(target, setup, questions, *, by='doc', depth=DEPTH)

Search every labelled question with one setup and time each search.

Returns the setup with ranks, the position of the first right result for each question within depth results as they come back (a document that fills three places takes three), and times_ms, the wall time of each search() call, query embedding and reranking included. One untimed search first loads the models, and the search cache is kept out of it. A setup whose first search fails, or whose graph turns out to be missing, comes back with error and no numbers; so does one where a question fails twice.

save_questions(questions, path)

search_of(way)

The search() arguments for one way of searching, named the way a person pins one.

<method> or <method>.<vectors>: a method of :data:METHODS, and for any method but keyword search, which vectors, as vectors= names them, both for all of them fused. hybrid_semantic.both is {"mode": "hybrid", "rerank": "semantic", "vectors": "both"}. No collection is named, so one pin reads the same in every collection. ValueError for a method it does not know, or vectors given to keyword search, which reads none.

setups_of(target)

Every setup this target can be searched with, as plain data.

Each is a dict a report keeps as it is: key, the target it belongs to, the engine in full and engine_short, the method and its method_label, the ranker in words, the models its vectors come from, and search, the keyword arguments that make db.search run it. Keyword search, with Azure's semantic ranker or without, is one setup whatever the vectors, since it reads none of them.

suggest_expected(db, questions, by='doc', top=1, **search)

For each question, the ids its reference answer retrieves.

A starting point for labelling and nothing more. Searching with the answer finds the document that says it far more reliably than searching with the question, which is why this is a fair way to propose a label and an unfair way to score one. Nothing is written to the questions: a person reads the suggestions and keeps the ones that are right.

sweep(documents, questions, *, chunk=('recursive',), chunk_size=(1000,), overlap=(200,), embed_heading=(False,), search=({'mode': 'dense'},), by='doc', k=(1, 3, 5, 10), open_with=None, workdir=None, progress=None)

Build the index every way asked, ask the golden questions of each, and rank them.

documents is the originals: paths, or {doc_id: path} when the golden file's expected names ids the files would not get by themselves. Every combination of chunk, chunk_size, overlap and embed_heading is one build in a scratch folder, removed after; an overlap that is not smaller than the size is left out. Each build is asked every question once for each entry of search, the keyword arguments of db.search: {"mode": "dense"}, {"mode": "hybrid", "rerank": False}. open_with goes to Vectrix, for a dense_model or a text_language; the collection is opened in the highest mode any search asks for.

Labels have to survive a change of chunker, so they name documents: by="doc" is the default and the only grain that means the same in two builds. The reply's rows are ranked by nDCG, then MRR, then by time, each with its recall at k, the chunks it made, the seconds the build took and the median search in milliseconds; best is the first. Builds run one after another on one machine, so the times compare with each other and with nothing else.

sweep_markdown(report)

The sweep as a table, best first, for a pull request or a page.

write_golden(db, path=None, n=50, *, writer=None, critic=None, examples=(), scenario='', task='', mix=None, evolve=1, threshold=0.7, similarity=0.5, tries=3, seed=0, cache=None, progress=None)

Draft n golden questions from what the collection holds, each checked, for a person to check again.

db is a handle, or a list of them for one golden file over several collections. writer is a :class:ChatWriter, or any callable from chat messages to the model's text; left out, it is the one the settings name. critic judges passages and questions and is the writer unless given. The steps are in :mod:vectrixdb.evaluation's account of the writer: passages spread across sections and scored, a question of each kind in mix, checks no model makes, the critic at threshold, and evolve rewrites. similarity is how alike a second place must be, by the collection's own vectors. examples are a few questions people really ask, whose style is copied; scenario and task say who asks and why.

Written as JSONL to path when one is given, and never over a file that is already there, which may hold questions somebody checked. cache is a file every answer is kept in as it comes, so a run stopped halfway is taken up where it stopped, and one run again costs nothing. A model that refuses, or stops answering, raises :class:WriterUnavailable at once rather than after every question. The same seed on the same collection, with the same answers, gives the same rows.