Extract¶
Extractors for what is not text yet: services, OCR, speech, video. See Extract, keep, index.
vectrixdb.extract
¶
Extraction: whoever is best placed turns a file's bytes into text.
The library reads what it can read offline: PDF, DOCX, PPTX, XLSX, CSV, HTML, Markdown and text. A scanned page, an hour of audio or a format nobody here has heard of needs somebody else, and that somebody is a callable:
def ocr_pdf(data: bytes, name: str) -> LoadedDocument: ...
extract.register(".pdf", ocr_pdf) # for the process
Vectrix("docs", extractors={".pdf": ocr_pdf}) # for one collection
Vectrix("docs", extractors=HttpExtractor(url, routes={".pdf": "/extract/pdf"}))
An extractor takes the bytes and the file's name and returns a
:class:~vectrixdb.ingest.LoadedDocument, a string of Markdown, or a mapping
with text and optionally pages, headings and metadata. It may
take a third argument, source, to be told where the bytes came from. A
suffix nobody registered falls through to the built-in readers, so
registering one extractor changes one format and nothing else.
What this module does not do: decide what a document is entitled to. An extractor returns text and positions. The fields a policy decides by come from the ingestion, never from a service that was only asked to read a file.
Extractor = Callable[..., Any]
module-attribute
¶
ExtractionError
¶
Bases: VectrixError
Turning a file into text failed, and nothing was written.
One type for every way that goes wrong, an extractor that raised, an
endpoint that answered 500, a reply whose page offsets run past its own
text, so the ingestion worker and the REST route catch this and nothing
else. route and status are set when an endpoint was involved.
ExtractorRegistry
¶
Suffix to extractor, with a parent to fall back on.
A collection's registry sits on top of the process-wide one, so
extractors={".pdf": fn} overrides PDF for that collection and still
sees whatever register() added for everything else. "*" matches
any suffix nothing more specific claimed.
extract(data, name, source=None)
¶
Run the extractor for name, or return None when there is none.
HttpDescriber
¶
A figure described by an endpoint you run.
What Vectrix(describe_figures=...) wants is a callable of the image
bytes and a context; this is one that posts the image to url. With
body="raw" the image is the request body, its name goes in
filename_header and the context, the caption, the page and the text
either side, goes percent-encoded in X-Context. With the default,
multipart, the image is the file field and the context a context
field of JSON.
The reply is plain text, the description, or JSON with any of
caption, description, table (rows) and decorative.
HttpExtractor
¶
Extraction by an endpoint you run.
routes maps a suffix to a path on base_url; a route may carry a
query string. The file goes up as a multipart field, or with
body="raw" as the request body with its name in filename_header,
which is how a service that reads await request.body() wants it.
The reply is plain text, read as Markdown, or JSON::
{"text": "...", "pages": [[0, 1], [1842, 2]],
"headings": [[0, "Late fees", 1]], "metadata": {"ocr": true}}
Plain text gives a document cited by name; pages is what makes
scan.pdf#page=4 possible. Anything but a 2xx raises
:class:ExtractionError naming the route and the status.
The bytes of every file with a routed suffix are sent to base_url,
so it is an address the host chose. Only http and https are
accepted. headers is where a key for the service goes, if it has one.
With tracing on, the request carries a traceparent header, so a
service that traces, VectrixDB's own among them, reads the file in the
caller's trace.
mask asks the service to mask identifiers as it reads, the way
VectrixDB's own extraction service does with ?mask=1: True for
the identifiers, "all" for everything its engine knows, or the types,
a list or a comma-separated string. What the service reports it masked
comes back in the document's metadata as masking, counts and a risk
score, so what is kept says what was taken out of it.
from_environment(env=None, *, routes=None)
classmethod
¶
The service VECTRIXDB_EXTRACTOR_URL names, or None when it names none.
VECTRIXDB_EXTRACTOR_KEY, or the file VECTRIXDB_EXTRACTOR_KEY_FILE
names, as a container's secrets are mounted, goes in the header
VECTRIXDB_EXTRACTOR_KEY_HEADER names, x-api-key by default;
VECTRIXDB_EXTRACTOR_BODY is raw
(the default) or multipart; VECTRIXDB_EXTRACTOR_TIMEOUT is in
seconds, 300 by default. VECTRIXDB_EXTRACTOR_ROUTES, a JSON mapping
of suffix to route, wins over routes, which is what the caller
routes by when the setting is not there: for VectrixDB's own extraction
service, :func:vectrixdb.api.extraction.extraction_routes.
VECTRIXDB_EXTRACTOR_MASK asks the service to mask as it reads:
1 for the identifiers, all, or the types, comma separated.
coerce(result, name='')
¶
Whatever an extractor returned, as a :class:LoadedDocument.
A string is read as Markdown: its headings are found, its images become
figure lines and its pipe tables become rows, exactly as a .md file
is read. A mapping is the structured reply, where the extractor knows
the page breaks; its offsets are checked, because an offset past the end
of the text would give every chunk after it the wrong page in silence.
load_url(url, *, extractors=None, headers=None, timeout=30.0, transport=None, images=False)
¶
A document fetched by its address and read like a file.
What it is called decides who reads it: report.pdf at the end of the
address goes to the PDF reader or to an extractor registered for
.pdf. An address with no suffix that answers with HTML is read as a
page. Only http and https. This fetches whatever address it is
given, so it is for addresses the host chose, not ones a request named.
The address is fetched whole and kept without its credentials: a signed
link's signature, a token or a key never reaches the document's
source, its citations or an error message. A bot check sent in place
of the page, Cloudflare's "Just a moment..." or a CAPTCHA, raises
:class:ExtractionError naming the site and what it sent, rather than
being read as the page.
register(suffixes, extractor)
¶
Use extractor for these suffixes everywhere in this process.
registered()
¶
The suffixes somebody registered. Built-in readers are not listed.
resolve(extractors)
¶
A registry from what a caller passed: nothing, a mapping, a registry,
or one object that serves several suffixes, as :class:HttpExtractor does.