privyscope

Multilingual PII detection & masking engine — the language-neutral core of the privyscope project.

🌐 Showcase & Interactive Docs

This package contains the engine (a two-stage hybrid pipeline: a regex filter plus an optional ONNX BIOES/Viterbi NER stage), the privyscope console command, and the plugin machinery that discovers installed language packs. It ships no language data on its own — install at least one language pack to do real work:

pip install privyscope         # Basic Package

pip install privyscope-ko      # Korean   (pulls in this core automatically)
pip install privyscope-en      # English
pip install privyscope-ja      # Japanese
pip install privyscope-zh-hans # Simplified Chinese
pip install privyscope-zh-hant # Traditional Chinese

Installing a language pack pulls in this core as a dependency, so you normally never pip install privyscope directly.

Why privyscope?

Compared to traditional NER systems from 3-4 years ago, privyscope delivers significantly higher precision and robustness by employing several modern architectural advancements:

1. BIOES Tagging: Unlike the traditional BIO scheme, we use the BIOES (Begin, Inside, Outside, End, Single) scheme. This allows the model to explicitly learn span boundaries and single-token entities, which is critical for CJK languages where particles often follow PII entities without spaces.
2. Constrained Viterbi Decoding: Instead of making independent per-token decisions, we use a Viterbi decoder with learned transition biases. This ensures that the output sequence is always grammatically valid (e.g., an "Inside" tag can never follow an "Outside" tag), preventing fragmented or "broken" masking results.
3. Domain-Adaptive MLM: Before fine-tuning on PII extraction, we perform additional Masked Language Modeling (MLM) on a large-scale synthetic PII corpus. This "domain adaptation" phase allows the model to internalize the structural characteristics of PII (like API keys, ID numbers, and complex addresses) that are rarely seen in general-purpose datasets like Wikipedia.
4. LLM-Augmented Synthesis: We leverage LLMs and Faker to generate hundreds of thousands of high-fidelity synthetic PII sentences across various registers (formal, conversational, etc.). This massive increase in training variety ensures the model remains robust against novel contexts and diverse writing styles.

One command, any combination of languages

The privyscope command lives only here in the core, so co-installed language packs never collide over it (each pack only adds its own privyscope_<lang> data package).

# one language installed → it's used automatically
privyscope redact "홍길동 010-1234-5678"

# several installed → auto-detected per text, or forced with --lang
privyscope redact "John Smith 555-123-4567"          # → English
privyscope redact --lang ko "Call 010-1234-5678"     # force Korean
privyscope --version                                  # lists installed languages

In action

privyscope redact highlights every detected span in the source text, prints the redacted output with each entity replaced by its label, and reports the span counts by label — across languages, from the same command.

English

Korean

Japanese

Chinese (Simplified)

Chinese (Traditional)

Accuracy

Each language pack is scored by running the full pipeline (regex + NER) over a held-out validation set disjoint from training (n = 1,000, seed = 0 shuffle), using typed/strict scoring (label and exact offsets must match) across all 8 base entities (PER, LOC, DATE, EMAIL, ID_NUM, BANK, PHONE, SECRET) and all 4 registers (formal, conversational, edge_case, spoken). These numbers are therefore a conservative lower bound — the validation sets deliberately mix spoken, typo, and space-collapsed registers, so clean production text scores higher.

Language	Pack	micro-F1	n	Strong entities (≥ 0.98)	Known gap
Japanese	privyscope-ja	0.996	1,000	PER, LOC, DATE, EMAIL, ID_NUM, PHONE, SECRET	BANK 0.94
Chinese (Simplified)	privyscope-zh-hans	0.984	1,000	PER, LOC, DATE, BANK, SECRET	PHONE recall 0.93, ID_NUM recall 0.94, EMAIL 0.95
Chinese (Traditional)	privyscope-zh-hant	0.980	1,000	PER, LOC, DATE, BANK, SECRET	ID_NUM precision 0.87, EMAIL 0.95
English	privyscope-en	0.945	1,000	DATE, EMAIL, BANK, PHONE	ID_NUM recall 0.77, LOC 0.92, PER 0.94
Korean	privyscope-ko	0.926	1,000	PER, DATE, EMAIL, PHONE	LOC 0.49, ID_NUM recall 0.89

Per-entity strict F1 and the exact reproduction command (privyscope eval --lang <code> your_val.jsonl) are in each pack's README. The largest open gaps are ko LOC (multi-unit apartment/building tails split by the NER model), en ID_NUM recall, and zh-Hant ID_NUM precision.

Python API

from privyscope import Privyscope

# one language (explicit, or the sole one installed)
engine = Privyscope.from_pretrained(lang="ko")
engine.redact("홍길동의 전화번호는 010-1234-5678").masked_text   # "<PER>의 전화번호는 <PHONE>"

# multiple languages → route each text automatically
auto = Privyscope.auto()
auto.redact("John Smith 555-123-4567")    # English engine
auto.redact("홍길동 010-1234-5678")        # Korean engine

See docs/ (in the repo) for the CLI reference, offline usage, the output schema, fine-tuning, and the multilingual use-case guide.

Stage 1: the regex filter

Stage-1 rules are compiled from the pii-pattern-engine ruleset, which each language pack vendors as a submodule and compiles into its regex_rules.yaml at build time.

Beyond a pattern, a rule may name a verification function — a checksum or dictionary validator that a match must pass before it becomes a span:

- id: rrn_01
  label: ID_NUM
  pattern: '(?<![A-Za-z0-9])(?:[0-9]{2}[01][0-9][0-3][0-9]-?[1-4][0-9]{6})(?![A-Za-z0-9])'
  priority: 100
  verify: kr_rrn_valid      # must pass, or the match is discarded

This is what keeps high-recall patterns usable: 900101-1234568 has a valid RRN checksum and is redacted, while 900101-1234567 does not and is left alone. The validators are vendored into privyscope/_core/_vendor/ by scripts/vendor_verification.py (run automatically by setup.py), and resolved by name in privyscope._core.verification.

Verification fails open: an unknown function name, or one that raises, keeps the match. For a redaction engine, over-redacting is safer than silently dropping PII.

Writing a language pack

A language pack is tiny: a package that ships two YAML files (regex_rules.yaml, entity_config.yaml) and registers a LanguagePlugin under the privyscope.languages entry-point group.

# privyscope_xx/__init__.py
from privyscope import Privyscope, LanguagePlugin, __version__

LANG = LanguagePlugin(
    code="xx", display_name="Example", default_repo="org/privyscope-xx",
    package="privyscope_xx", scripts=("Latin",), default_base_model="roberta-base",
)

# pyproject.toml
[project]
dependencies = ["privyscope>=0.1.0"]

[project.entry-points."privyscope.languages"]
xx = "privyscope_xx:LANG"

License

Apache-2.0. See LICENSE.

---