Pathology Extraction & Normalization

Prototype for extracting pathology mentions from real and multilingual anonymized HCP prompts (pre-filtered to the Endocrinology therapeutic area) and normalizing them to a standard ontology in English. This project is the response to a two-part challenge:

• Part 1 — Research brief: choose and justify an ontology standard as well as a classification approach to replace the baseline. → docs/research_brief.md
• Part 2 — Implementation: create a working extraction and normalization pipeline over the provided sample and generate a sample output file. → docs/implementation_brief.md

The baseline being replaced: a fixed top-20-per-therapeutic-area list, one zero-shot LLM call per prompt, en-UK output — with four known limitations: coverage, granularity, robustness and scalability.

Overall view: Preprocessing, EDA and the 1,919-concept SNOMED/UMLS dictionary are implemented and verified on the full 10k sample (eda/). The Part 2 model/ cascade runs all three nets: Net 1 (name-match) + Net 2 (SapBERT-XLM-R embeddings) over the full 10k prompts (uv run python -m model.pipeline → ~20.5% labelled with no LLM call: Net 1 covers 17.8% and Net 2 covers 2.7%), plus Net 3 — a local LLM (Qwen2.5-3B) that resolves the deferred remainder - gated OFF by default (--use-llm to demo). Label-free metrics (model.evaluate) and the deliverable model/README.md complete it. Full design rationale: docs/implementation_brief.md.

---

Project structure

.
├── README.md                 # this file
├── pyproject.toml            # project metadata + dependencies (managed by uv)
├── uv.lock                   # exact, hashed dependency lockfile (committed)
├── .python-version           # pins the interpreter (3.12) for uv
├── .gitignore                # excludes confidential data, outputs, .venv, secrets
├── .env.example              # template for the secrets you must provide (UMLS_API_KEY)
├── config.py                 # paths, seed, constants — single source of truth
├── text_utils.py             # shared, dependency-free normalize_text (preprocessing + model)
│
├── data/                     # (gitignored — confidential)
│   ├── raw/                  # provided JSONL — immutable
│   └── processed/            # generated: prompts_enriched.jsonl, endocrine_dictionary.json
│
├── eda/                      # data understanding + label-space construction
│   ├── preprocess_prompts.py # builds the enriched dataset → data/processed/
│   ├── analyze_prompts.py    # EDA stats + plots → outputs/figures/
│   ├── build_dictionary.py   # builds the SNOMED endocrine dictionary (UMLS) → data/processed/
│   ├── analyze_dictionary.py # characterizes the dictionary (size, coverage) → outputs/figures/
│   └── README.md             # how to run the EDA stage (run order + outputs)
│
├── model/                    # Part 2 — the 3-net extraction cascade
│   ├── dictionary_index.py   # label space: name index + roll-up + generalize (shared backbone)
│   ├── net1_namematch.py     # Net 1 — deterministic multilingual name match
│   ├── net2_embed.py         # Net 2 — SapBERT-XLM-R embedding meaning-match (cached)
│   ├── net3_llm.py           # Net 3 — local LLM (Qwen2.5-3B) constrained pick; gated off
│   ├── output.py             # per-message prediction record (schema + JSONL writer)
│   ├── pipeline.py           # CLI orchestrator → outputs/predictions_sample.jsonl
│   ├── evaluate.py           # label-free metrics + spot-check + funnel figure
│   └── README.md             # Part 2 deliverable README (how to run + headline results)
│
├── outputs/                  # (gitignored) figures + predictions_sample.jsonl
│   └── figures/
│
└── docs/
    ├── research_brief.md        # Part 1 deliverable
    ├── implementation_brief.md  # Part 2 design decisions
    └── research_notes.md        # working notebook (EDA findings + decisions)

---

Environment & setup

The project uses uv with a uv-managed Python 3.12.

# one-time: install uv (https://docs.astral.sh/uv/getting-started/installation/)
curl -LsSf https://astral.sh/uv/install.sh | sh

# create the environment from the lockfile
uv sync

# create your secrets file — the dictionary build needs a free UMLS API key
cp .env.example .env   # then edit .env and set UMLS_API_KEY (sign up: https://uts.nlm.nih.gov/uts/signup-login)

# run the EDA / setup stage
uv run python -m eda.preprocess_prompts
uv run python -m eda.analyze_prompts
uv run python -m eda.build_dictionary   # needs UMLS_API_KEY in .env
uv run python -m eda.analyze_dictionary

# run the Part 2 extraction pipeline (Net 1 + Net 2 by default; Net 3 is opt-in)
uv run python -m model.pipeline                    # full 10k → outputs/predictions_sample.jsonl
uv run python -m model.pipeline --sample-size 300  # quick deterministic sample
uv run python -m model.pipeline --no-net2          # Net 1 spine only (no ML deps needed)
uv run python -m model.pipeline --use-llm --llm-budget 5   # + Net 3 (local LLM) on a few deferred prompts

# run evaluation on the extraction pipeline results
uv run python -m model.evaluate

Dependency management

pyproject.toml (abstract deps) + uv.lock (exact, hashed, cross-platform) are the canonical reproducibility artifacts and are committed. There is intentionally no requirements.txt — uv.lock supersedes it. If a non-uv reviewer needs a pip-installable list, generate one on demand:

uv export --format requirements-txt > requirements.txt

---

Data & confidentiality

The provided data is confidential real-world HCP data (EU countries → GDPR special-category health data). Accordingly:

• data/ and *.pdf are gitignored and must never be committed.
• data/raw/ is treated as immutable — preprocessing never writes back to it.
• This confidentiality constraint also informs design (e.g. preferring local processing before routing prompts to any third-party LLM API). See the research brief.

Enriched dataset schema

preprocess_prompts.py produces one record per message in data/processed/prompts_enriched.jsonl:

Field	Source	Notes
chat_message_id, country, prompt	original	preserved as-is; prompt is the raw text (case + accents kept) for neural models
conversation_id, timestamp	parsed from chat_message_id (<epoch_ms>_<conversation_id>)	timestamp is a datetime; enables thread reconstruction
turn_index, thread_size	derived	position within the conversation
char_len	derived	length of the prompt
language, language_confidence, language_margin	detected per conversation (concatenated turns), broadcast to its messages	margin = top-1 vs top-2 probability gap
prompt_norm	normalized	lowercase + accent-fold + whitespace-collapse, for matching / dedup

---

EDA plan (the analyses in eda/analyze_prompts.py)

#	Analysis	Purpose
A. Volume, shape & quality
A1	Exact duplicate count (on prompt_norm)	redundancy / caching opportunity
A2	Prompt length histogram (chars)	how short / telegraphic inputs are
A3	Prompts per country (bar)	geographic mix & implied language skew
A4	Prompts per day (hist)	confirm sampling window / bursts
B. Thread structure
B5	Turns per conversation (hist)	how multi-turn the data is
B6	Joined-thread length (hist)	text available per conversation
B7	Per-country distribution after joining	country mix at conversation level
C. Language
C8	Detected-language distribution	the real language mix
C9	Country × detected-language cross-tab	quantify country ≠ language mismatch
C10	Per-country language match rate	how often language detection agrees with the country prior
D. Context dependency (simplified)
D11	Share of follow-up (non-first) turns	ceiling on potential context-dependence
D12	Length by turn position	do threads decay into terse, context-dependent fragments?
E. Threading validation
E13–15	Monotonic timestamps + gaps; single-country/language per thread; suffix sanity	confirm the id suffix is a clean session id before relying on threading

Deferred to a post-ontology pass (depends on having a concept vocabulary): pathology/drug/lab content profiling, abbreviation→concept mapping, pathology-shift within threads, the "turn inherits its pathology from context" measure, and broader noise profiling.

---

Key design decisions

• Ontology: SNOMED CT as the label space, accessed via UMLS, normalized with a three-net pipeline (name-match → embedding → LLM). See docs/research_brief.md.
• Two text views, prompt (raw) + prompt_norm — normalize for symbolic matching, preserve the raw text for neural models (case and accents carry signal, especially in FR/PT).
• Language detected per conversation with langdetect (pinned seed) — concatenated turns are far more reliable than a single short prompt and the result is broadcast to each message. fasttext lid.176 or lingua (with a language allowlist) are drop-in upgrades if per-turn precision is needed.
• Net 2 — meaning match: the cross-lingual biomedical entity-linker SapBERT-XLM-R, because the task is concept normalization to UMLS/SNOMED (its training objective) and it covers all four languages in one space. Accepted matches roll up to the condition family, since whole-prompt embeddings are unreliable at the specific level.
• Net 3 — the hard remainder: a small local LLM (Qwen2.5-3B-Instruct), run offline and gated off by default — it only picks from Net 2's shortlist (no hallucinated labels) and nothing leaves the machine. A larger or hosted model is a drop-in for more accuracy.
• Full decision log — the honest finding behind each choice — is in docs/implementation_brief.md §9.

---

Future work (see the briefs' §8)

• Disease-span extraction
• Drug/lab → condition map
• A hand-labelled gold set + LLM-judge
• ICD-10 crosswalks