Masking¶
Finding and replacing identifiers before a document is indexed. See Masking.
vectrixdb.masking
¶
Masking of identifiers in text that people are shown, and in documents before they are indexed.
Two places, one package. On the way out of a server, :func:mask_text and
:func:mask_value hide email addresses, phone numbers and card numbers in a
reply to a person or a guest, by their shape, in this process, in microseconds:
a script with an API key is sent the text as stored, because a pipeline needs
it. On the way in, :func:mask reads a whole document with the engine the
deployment chose, so what is indexed never held the identifier at all.
>>> from vectrixdb.masking import mask_text, mask
>>> mask_text("Call Ada on +1 416 555 0199 or ada@example.com, card 4111 1111 1111 1111.")
'Call Ada on +• ••• ••• 0199 or a•••@example.com, card •••• •••• •••• 1111.'
>>> mask("SIN 046-454-286, key AKIAIOSFODNN7EXAMPLE").text
'SIN [SIN], key [API_KEY]'
Four engines answer :func:mask, one call in front of them: the patterns,
always there; Presidio, in this process, pip install 'vectrixdb[masking]';
Azure AI Language; Amazon Comprehend. VECTRIXDB_MASKING_ENGINE names one,
or auto picks Azure AI Language when AZURE_LANGUAGE_ENDPOINT is set,
else Presidio when it is installed, else the patterns. Comprehend is used when
named, never picked, so a region set for Bedrock never starts billing another
service. Whatever ran, the patterns run after it. VECTRIXDB_MASKING_LANGUAGES
says which languages the documents are in, en,fr unless you say, which is
what Presidio loads a model for and what an engine is asked to cover.
Masking lowers casual exposure. It is not a guarantee, and the documentation says so wherever it is offered: a thing no engine knows is shown as stored, and a search still finds a chunk by a number the index holds as written, which is why masking at ingestion is the stronger of the two.
DEFAULT_TYPES = ('email', 'phone', 'credit_card', 'ssn', 'sin', 'iban', 'bank_account', 'api_key', 'connection_string')
module-attribute
¶
ENGINES = ('auto', 'regex', 'presidio', 'language', 'comprehend')
module-attribute
¶
MASK = '•'
module-attribute
¶
TYPES = ('name', 'email', 'phone', 'credit_card', 'ssn', 'sin', 'iban', 'bank_account', 'passport', 'drivers_license', 'ip_address', 'address', 'date_of_birth', 'date', 'api_key', 'connection_string', 'internal_host')
module-attribute
¶
WEIGHTS = {'name': 0.5, 'email': 0.6, 'phone': 0.5, 'credit_card': 0.9, 'ssn': 0.95, 'sin': 0.95, 'iban': 0.85, 'bank_account': 0.85, 'passport': 0.8, 'drivers_license': 0.7, 'ip_address': 0.3, 'address': 0.6, 'date_of_birth': 0.5, 'date': 0.1, 'api_key': 0.95, 'connection_string': 0.95, 'internal_host': 0.4}
module-attribute
¶
ComprehendEngine
¶
Amazon Comprehend's PII detection, one call a document, through boto3.
Found
dataclass
¶
One identifier in a text: what it is, where it is, and who found it.
LanguageEngine
¶
Azure AI Language's PII task, one call a document, in the language the document is in.
Masked
dataclass
¶
What :func:mask made of a text: the text, what was found, how risky it was, and who looked.
PresidioEngine
¶
Microsoft Presidio, in this process, one spaCy model a language.
RegexEngine
¶
Identifiers by their shape. Always available, no model, no service.
describe_engine(engine, languages=None)
¶
The engine as a health reply says it: its name, which of the languages it covers, and that the patterns run last.
download_models(languages, downloader=None)
¶
Fetch the spaCy model Presidio reads each language with. Returns the model names fetched.
engine_from_env(env=None)
¶
The engine the deployment chose, VECTRIXDB_MASKING_ENGINE, built from its settings.
auto, the default, picks Azure AI Language when its endpoint is set,
else Presidio when it is installed, else the patterns. A named engine
whose settings or package are missing is a ConfigurationError or a
DependencyError here, at start, rather than a surprise at the first file.
languages_from_env(env=None)
¶
The languages the documents are in, VECTRIXDB_MASKING_LANGUAGES, en,fr unless said.
mask(text, *, types=None, engine=None, language=None, env=None)
¶
text read by the engine and then by the patterns, with what was found masked.
types is what to hide: nothing said means the identifiers, not names;
"all" means everything an engine knows. engine is one of the
engine objects, or None for the deployment's, from the environment.
language is the text's, en or fr, which Presidio and Comprehend
need to know and Azure AI Language will detect if not told.
mask_text(text)
¶
text with its email addresses, phone numbers and card numbers masked where they stand, by their shape.
mask_value(value, keep=lambda key: False, _key=None)
¶
Every string inside value masked: a result, its metadata, a list of highlights.
keep names the keys whose values are left as they are, whole subtrees
included: an id a caller sends back to open a chunk has to stay the id.
types_of(given)
¶
The types to mask, from what a record or a request says: nothing, all, a list, or a comma-separated string.