Extraction quality

Whether a text is the output of an extraction that worked. See Extraction quality.

vectrixdb.quality

Is this text the output of an extraction that worked?

A scanned PDF that OCRs badly produces chunks that pass every schema check and destroy retrieval: the fields are all there, the text is nonsense, and nothing downstream can tell. This is closer to signal reliability than to validation, and the roadmap held it back until it could be measured against a labelled set rather than shipped as a heuristic nobody trusts.

So it is measured. :func:extraction_quality scores a text on five signals that OCR failure moves and clean prose does not, and the default threshold comes from :func:calibrate run over a labelled set: real prose from this repository's own documentation, and the same prose degraded by a simulator of the substitutions, joins and glyph noise OCR actually produces, at a range of noise levels. scripts/quality_eval.py rebuilds that set and prints the separation; the numbers it printed when the threshold was set are in the docstring of :data:DEFAULT_THRESHOLD.

What it is not: a language model, a dictionary, or a judgement about meaning. A page of clean text about nothing scores well. What it catches is the failure that reads like a permissions problem or a ranking problem when it is an ingestion problem, and it says so at the write, where the stack trace points at the pipeline.

DEFAULT_THRESHOLD = 0.78 module-attribute

ExtractionQuality dataclass

The score, the signals it came from, and the verdict at a threshold.

score is in [0, 1], higher is cleaner. signals are the five components, each in [0, 1], so a low score can be read: a text that is all symbols fails plain_characters; one that is words nobody has seen fails known_words; one cut into fragments fails whole_words.

Calibration dataclass

What a labelled set says about a threshold.

extraction_quality(text, threshold=DEFAULT_THRESHOLD)

Score a text's prose on five signals OCR failure moves and clean prose does not, scaled by a sixth.

A table's rows are not prose. Region: EMEA; Revenue: 1200 has more figures than words, repeats its column names on every line and uses no common words, which is everything a failed OCR pass looks like, and a clean spreadsheet scored 0.20 against a line of 0.78. So the lines a reader writes for rows, and its figure lines, are set aside, and what is left is judged. A text that is nearly all rows has too little prose to judge, and is usable, as a two-word heading is.

calibrate(labelled, threshold=None)

Measure the detector against (text, is_clean) pairs.

With threshold given, reports precision and recall at it. Without, picks the threshold that maximises balanced accuracy over the set, which is how :data:DEFAULT_THRESHOLD was chosen. Precision and recall are for the clean class: precision is how often "usable" is right, recall is how much clean text is kept.

degrade(text, level, seed=0)

Put text through a simulation of a bad OCR pass.

level is the share of characters touched, from 0 (untouched) to 1. Each touched character is swapped for a shape-alike, dropped, joined to its neighbour by losing the space, or replaced with glyph noise, in the proportions those failures show up in real output. Deterministic for a given seed, so a labelled set can be rebuilt.