Sources¶
Feeds and pages a collection keeps up with. See Keep a collection in step with feeds and pages.
vectrixdb.sources
¶
Feeds and pages a collection keeps up with: read again on a schedule, and only what changed written again.
db = Vectrix("news")
db.sources.add("https://example.com/feed.xml", every="6h") # RSS, Atom or JSON Feed
db.sources.add("https://example.com/pricing", every="1d") # a page, or a PDF at an address
report = db.sources.refresh() # the sources that are due
print(report) # 2 sources: 12 added, 1 updated, 40 unchanged, 0 removed, 0 failed
A feed gives one document an entry, with its title, link, author and dates on
every chunk. Feed(address, articles=True) reads the article each entry
links to rather than the feed's summary of it. A podcast's episodes are
transcribed when the collection has an audio engine registered for their
files, and are their show notes otherwise. A page gives one document, its
main text, so a change to the menu around it is not a change.
Only what changed is written again. A feed or page that answers 304 to the
ETag and date it gave last time costs one request; an entry whose text is the
same is left alone; one that changed replaces its document in place, under the
same id, so nothing is indexed twice and lineage still walks back to the write.
An entry that drops out of a feed keeps its documents. A page that answers 404
or 410 is marked gone, and its chunks are removed only when the source was
added with delete_when_gone=True.
Every fetch goes through :mod:vectrixdb._fetch: http and https only, never a
private, loopback or metadata address however a name resolves, every redirect
checked, size and time capped, robots.txt obeyed, a second at least between
two requests to one host, and a bot check refused rather than indexed. An
address with a password, a token or a signature written into it is refused
when it is added, because it would be kept where every operator can list it:
write ${NAME} in its place and the value is read from the environment each
time the source is fetched, and never shown or stored.
A licensed news feed, an internal system or anything else is a :class:Source
of your own: subclass it, give it a kind, and :func:register_source it,
or declare it under the vectrixdb.sources entry point, so a scheduled
refresh in another process can build it again from what was kept.
Two refreshes never work on one source at once, here or on another server: a
refresh holds a lease on each source while it reads it, and one that died lets
go when the lease runs out. Sources are kept beside the collection records when
the database has a store for them (VECTRIXDB_COLLECTION_STORE: Cosmos DB,
PostgreSQL, DynamoDB), so every server sees the same list, and in the
database's own file otherwise.
AUDIO_MAX_BYTES = 300 * 1024 * 1024
module-attribute
¶
DEFAULT_EVERY = 6 * 3600.0
module-attribute
¶
MAX_ITEMS = 50
module-attribute
¶
MIN_EVERY = 5 * 60.0
module-attribute
¶
Feed
¶
Bases: Source
An RSS 2.0, RSS 1.0, Atom or JSON Feed: one document an entry.
articles=True reads the article an entry links to, through the same
checks as the feed, and indexes that rather than the feed's summary; a
site that refuses it, by robots.txt or a bot check, gets the feed's own
text and a note in the report. transcribe turns a podcast's episodes
into transcripts when the collection has an audio engine registered for
their files; without one, or with transcribe=False, an episode is its
show notes. Nothing is downloaded to find out.
Item
dataclass
¶
One document a source gives: a feed's entry, a page, an article from a licensed feed.
key is the item's id within its source, the same every time it is
read. Give its text as text (Markdown), document (a
:class:~vectrixdb.ingest.LoadedDocument), or read, a function
called only when the item is new or changed: that is how an item that
costs something to read, an article to fetch, an episode to transcribe,
is not read again for nothing. fingerprint is what says it changed,
anything that does when it does, a revision number, an updated date; left
out, the item is read every time and its text compared.
gone says it no longer exists, a page that answers 410: its
documents stay unless its source deletes what is gone.
Page
¶
Bases: Source
One address, read again on a schedule: a web page's main text, or a file such as a PDF.
It is written again only when its text changed. A page that answers 404
or 410 after it was indexed is reported gone and its chunks kept;
delete_when_gone=True removes them instead.
RefreshReport
dataclass
¶
What one refresh did: each source, and the totals.
Source
¶
What a collection keeps up with. Subclass it for a source the built-ins do not read.
A subclass names its kind, reads in :meth:read, and says in
:meth:settings what besides its address builds it again; the default
:meth:from_settings passes them back as keyword arguments. Keep
secrets out of both: a licensed feed's key belongs in the environment,
read when :meth:read runs, or in the address as ${NAME}, which
context.address has filled in and self.address does not. Then
:func:register_source it::
class Wire(Source):
kind = "wire"
def read(self, context):
for story in wire_client(os.environ["WIRE_KEY"]).latest(desk=self.address):
yield Item(key=story.id, fingerprint=story.revision, title=story.headline,
link=story.url, published=story.time, read=lambda s=story: s.body_markdown())
register_source("wire", Wire)
db.sources.add(Wire("energy"), every="1h")
SourceContext
¶
What a source reads with: the guarded fetch, what it kept last time, and the collection's readers.
address is the source's own address to fetch, any ${NAME} in it
read from the environment, and shown is the same address as it may
be shown. state is a small dict kept between refreshes, an ETag or
a cursor; it is saved when the source has been read without an error,
and an ETag and a date only once every item it gave has been written.
get(url, headers=None, *, max_bytes=None, timeout=None)
¶
What url answers, through every check of :class:vectrixdb._fetch.Fetcher, whatever the status.
max_bytes and timeout change the caps for this one request.
known(key)
¶
The fingerprint recorded when the item with this key was last written, or None.
has(key)
¶
Whether an item with this key has been written before.
reads(name)
¶
Whether the collection has an extractor of its own for a file of this name: an audio engine for .mp3.
read_bytes(data, name, *, source)
¶
Bytes read into a document the way the collection reads a file of this name.
note(text)
¶
A line for the refresh's report: something worth knowing that is not a failure.
scrub(text)
¶
text with every value the source's address reads from the environment taken back out, as ${NAME}.
SourceInfo
dataclass
¶
One source as it is kept: never a secret.
SourceOutcome
dataclass
¶
What one refresh did with one source.
Sources
¶
The feeds and pages one collection keeps up with. db.sources on a :class:~vectrixdb.easy.Vectrix.
records is where they are kept (see :func:Sources.of), writer
what writes their documents. fetcher is a
:class:vectrixdb._fetch.Fetcher, made from the environment for each
refresh when left out: VECTRIXDB_SOURCES_HOSTS limits the hosts,
VECTRIXDB_SOURCES_INTERNAL_HOSTS names intranet hosts allowed to be
private, and HTTPS_PROXY with NO_PROXY are read as everywhere.
of(db, **given)
classmethod
¶
The sources of a :class:~vectrixdb.easy.Vectrix: kept where its database keeps them, written with add_document.
add(source, every=DEFAULT_EVERY, *, kind=None, by=None, **options)
¶
Keep this collection up with source: an address, or a :class:Feed, :class:Page or a Source of your own.
An address is fetched once to tell a feed from a page, unless
kind says which; options go to that class, articles=True
to a feed, delete_when_gone=True to a page. Adding an address
that is already here changes how often it is read and its options,
and keeps what it has written. Nothing is written until
:meth:refresh.
list()
¶
Every source of this collection, in the order they were added.
get(source)
¶
One source, by its id, its address or the source itself; None when it is not here.
remove(source, *, delete_documents=False)
¶
Stop keeping up with a source. Whether there was one.
Its documents stay, unless delete_documents; then every document
it wrote goes, except one another source of this collection wrote too.
reset(source=None)
¶
Forget what a source has written, or every source's, so the next refresh writes it all again.
For a collection cut again in a new way, or emptied by hand. The documents stay where they are until the refresh replaces them. Returns how many sources were reset.
refresh(force=False, *, only=None, max_items=MAX_ITEMS)
¶
Read every source that is due, or every one with force, and write only what changed.
only refreshes one source, due or not. max_items is the most
entries written for one source this time; the rest wait for the next
refresh. Never raises for a source that fails: the report says which,
and why, and the next refresh tries again.
every_text(seconds)
¶
Seconds the short way: 21600 is 6h, 5400 is 1h30m.
parse_every(every)
¶
Seconds between two reads of a source, from "6h", "1d", "30m", "1h30m", a number or a timedelta.
register_source(kind, cls)
¶
Make a kind of source of your own known here; returns a function that forgets it.
A package does the same with an entry point in the vectrixdb.sources
group, which is what a scheduled refresh in another process finds.