interscript · ronaldtse · Aug 30, 2026 · Aug 7, 2026 · Aug 7, 2026 · Aug 7, 2026
diff --git a/TODO.improve-models/01-urdu-byt5-d1.md b/TODO.improve-models/01-urdu-byt5-d1.md
@@ -0,0 +1,31 @@
+# 01 — Urdu diacritization d1: ByT5-base + cross-lingual init
+
+## Why
+Urdu is our weakest shipped model (14.77% CER, urdu_diacrit/run-001,
+custom char encoder trained on 635K cross-lingually machine-labeled
+lines). Arabic — same script family, same task shape — sits at 2.68 DER
+on ByT5-base with paragraph context. The gap is architecture + teacher
+vintage, not task difficulty.
+
+## Plan
+1. Data: existing corpus on volume `urdu-diacrit-datasets`:
+   - `urdu-diacritized/{train,val,test}.txt` (635K machine-labeled —
+     WEAK labels, teacher-poison rule applies: treat as stage-1 only)
+   - `urdu-diacrit/*.jsonl` (HF G2P-derived pairs from WO #306)
+2. `train_urdu_d1.py` (rababa):
+   - Init: Arabic r5 teacher `/checkpoints/rababa_arabic_byt5/
+     run-005-context/best` (ByT5-base) — cross-lingual init gives the
+     shared-abjad prior instead of starting cold.
+   - Stage 1: 1 epoch over the weak 635K (line units, byte tokenizer).
+   - Stage 2: none yet (no gold corpus found — UDD has no verifiable
+     public URL; do NOT fabricate one). The run's own test split is
+     machine-labeled too, so the eval is comparative, not absolute.
+3. Eval: greedy CER via editdistance on `urdu-diacritized/test.txt`,
+   identical protocol to the 14.77 number + word-level accuracy.
+   Target: clearly under 14.77 CER (architecture+init upgrade).
+4. Launch detached under supervisor app `rababa-urdu-d1`
+   (EVAL_DONE guard, checkpoint-resume, volume commits).
+
+## Guards
+- No LLM teacher. No RL. Weak corpus is from our Arabic model only.
+- Parallel to r6 (separate Modal app + GPU).
diff --git a/TODO.improve-models/02-arabic-r7-news-domain.md b/TODO.improve-models/02-arabic-r7-news-domain.md
@@ -0,0 +1,46 @@
+# 02 — Arabic r7: news-domain adaptation (OOD repair)
+
+## Why
+r5 paragraph-context specialized to the SadeedDiac domain and trades
+~0.5 DER out-of-domain (WikiNews-2024 multi-ref: r5 20.52/12.72 vs
+r3 19.99/12.60). If r6 verifies, case endings improve too — but the
+OOD gap needs domain data, not morphology.
+
+## Plan
+1. `label_arabic_news.py` (rababa) — runs NOW, parallel to r6:
+   - Fetch unlabeled Arabic news from HF
+     (`khalidalt/ultimate_arabic_news`, fallback
+     `Abdelkareem/arabic-bbc-news`), clean (Arabic-letter fraction,
+     length, dedupe).
+   - Pseudo-label with r5 (windowed 1400B zero-skip, greedy, same
+     harness as eval) → volume `/datasets/arabic-news-r5/`.
+   - Add GOLD news: `/datasets/wikinews/WikiNews_2014.txt.diac`
+     (gold-diacritized, different year/documents than the 2024 probe).
+   - NEVER touch `WikiNews_2024*` — that is the OOD probe.
+2. `train_arabic_r7.py` — launch AFTER r6 verdict + labels exist:
+   - INIT = r6 best if r6 verifies better on SadeedDiac, else r5.
+   - Mix: cached r5-units (replay, protects ID) + news units
+     (2014 gold upweighted + r5-pseudo modern news), news ≈ 15-20%
+     of steps. r5-proven batch/accum, A100-80GB.
+   - Gates: SadeedDiac-25 windowed zero-skip must not regress beyond
+     +0.1 DER of the init model; WikiNews-2024 multi-ref must improve.
+3. Fold verdict into docs/RESULTS.md + DISTILL-SOURCE-PROMPT.md only
+   when both gates pass.
+
+## Guards
+- Domain adaptation via pseudo-labels is self-training — the replay
+  majority + gold 2014 news keeps ID anchored; the ID gate is the
+  hard stop against entrenchment.
+- No RL, no LLM labels.
+
+## DEFERRED (2026-08-22)
+
+r6's verified OOD sweep (WikiNews-2024 full 19.82/12.46 — beats r3
+AND r5) absorbed this workstream's purpose: there is no OOD deficit
+left to repair. r7 as designed costs 20h of A100-80GB for marginal
+OOD gains with ID-regression risk, and its concurrent A100 footprint
+is exactly what triggers Modal workspace evictions. News labeling
+stopped at 5,600/13,987 windows — all committed and resumable on the
+volume (label_progress.jsonl), and train_arabic_r7.py is ready with
+--init-run run-006-morph. Reopen ONLY if a news-heavy client use
+case emerges or Arabic OOD regresses in the wild.
diff --git a/TODO.improve-models/03-rag-homograph-probe.md b/TODO.improve-models/03-rag-homograph-probe.md
@@ -0,0 +1,32 @@
+# 03 — RAG homograph disambiguation probe (Persian first)
+
+## Why
+Persian v1 sits at 77.34% SentenceBench homograph (ezafe-normalized) —
+above published Homo-GE2PE (76.89) but flat: RL was negative, the
+mapped-representation line is closed. Retrieval context is the one
+untried lever, and homograph resolution is precisely a context problem.
+
+## Plan
+1. `eval_persian_rag_probe.py` (rababa-farsi) — INFERENCE ONLY, no
+   retraining for the probe:
+   - Index HomoRich TRAIN homograph sentences (TF-IDF char n-gram
+     retrieval over sentence contexts, pure numpy/sklearn-free).
+   - For each SentenceBench test sentence, retrieve top-k (k=3)
+     contextually nearest TRAIN sentences containing the same
+     homograph token, format as few-shot prefix:
+     `<train-sentence> => <diacritized/phonemized> ;` repeated, then
+     the test sentence. ByT5-small handles prefix context in bytes.
+   - Baseline vs RAG on the SAME harness (`eval_sentencebench.py`
+     protocol, ezafe-normalized homograph accuracy + exact match).
+2. Decision rule:
+   - RAG ≥ +1.5pp homograph → invest: cache retrieval index, then a
+     fine-tune WITH retrieved prefixes (train-time consistency).
+   - +0.5..1.5pp → cheap inference-time add-on only, document.
+   - < +0.5pp → close the lever, record negative in docs/RESULTS.md.
+3. If Persian moves, port the probe to Hebrew (Nakdimon homographs)
+   and Arabic (SadeedDiac residual analysis).
+
+## Guards
+- Retrieval from TRAIN splits only — zero test contamination; assert
+  no test sentence appears in the index.
+- Teacher stays v1 (RELEASE-FROZEN); probe never modifies it.
diff --git a/TODO.improve-models/04-hebrew-s46-scaled-weak.md b/TODO.improve-models/04-hebrew-s46-scaled-weak.md
@@ -0,0 +1,31 @@
+# 04 — Hebrew s46: scale + diversify the weak stage
+
+## Why
+s45 proved the curriculum (+0.88 DER over gold-only: 16.58 vs 17.46)
+but its weak stage was knesset only (1.5M lines, single domain —
+parliamentary transcripts). hewiki (80K Hebrew-Wikipedia lines) is a
+second, encyclopedic domain sitting unlabeled on the volume. Same
+lever, turned up: more + more-diverse weak data before the identical
+gold FT.
+
+## Plan
+1. `label_hewiki_full.py` (rababa) — runs NOW on A10G:
+   - Full-scale Dicta labeling of `/datasets/hewiki/train.txt`
+     (80K lines) via the batch recipe from `batch_distill_hewiki.py`
+     (DictaBERT predict batches), nikud-only targets (strip teamim),
+     length/Hebrew-fraction filters, 40-char window decontam vs
+     Nakdimon test.
+   - Output: `/datasets/hebrew-hewiki-dicta/{train,val}.txt`.
+2. `train_hebrew_s46.py` — s45 VERBATIM with one change:
+   - Stage 1 weak = knesset 1.5M + hewiki-dicta (all of it,
+     ~80K lines ≈ 5% of weak steps — a domain garnish, not a pivot).
+   - Stage 2 = s43 gold recipe unchanged (hebrew-v4 jsonl, 3 ep,
+     batch 8, LR 3e-4, warmup 500).
+   - Eval: beam-4 DER, identical protocol/harness as s45's 16.58.
+3. Gates: must beat s45's 16.58 to replace it; otherwise record as
+   flat and keep s45 (weak-stage ceiling reached).
+
+## Guards
+- Dicta labels are weak only — the gold stage corrects (no-teacher-
+  poison, the exact s45-validated pattern).
+- Zero Nakdimon-test contamination (window decontam).
diff --git a/TODO.improve-models/05-khmer-g2p-byt5.md b/TODO.improve-models/05-khmer-g2p-byt5.md
@@ -0,0 +1,25 @@
+# 05 — Khmer G2P v1: ByT5-small on the 17.9K UNGEGN word pairs
+
+## Why
+The only Khmer artifact is the legacy crystalseq transformer
+(net-500-epochs.pth, loss 0.143, no held-out eval) plus its fp16 IMF
+zip. Every other language in the zoo runs a modern recipe with a
+measured number. Khmer is the breadth gap.
+
+## Plan
+1. Data: secryst-datasets:/data-khmer-translit/{data_kh,data_rom}.csv
+   — 17,911 aligned word pairs (validated: no dups, p95 word = 36B,
+   1 empty rom line filtered). Split 80/10/10 seeded.
+2. `train_khmer_g2p.py` (secryst-train repo):
+   - google/byt5-small, word-level src→tgt, batch 64, LR 3e-4,
+     30 epochs, best-on-val-loss checkpointing (small-data recipe).
+   - Eval: held-out test — word accuracy (exact match) + CER via
+     editdistance. This is the FIRST measured Khmer number; the
+     crystalseq model gets the same eval for the comparison table.
+3. Export to IMF v1 when the number lands (parts contract, byte+3
+   tokenizer) — replaces the fp16 zip of the legacy model.
+
+## Guards
+- Word-level G2P (not diacritization): no LLM-teacher concerns, but
+  the no-RL/no-LLM standing rules apply anyway.
+- Keep the legacy crystalseq artifacts untouched (source files).
diff --git a/TODO.improve-models/06-urdu-beam-and-d2.md b/TODO.improve-models/06-urdu-beam-and-d2.md
@@ -0,0 +1,21 @@
+# 06 — Urdu d1 follow-up: beam-4 eval, then d2 decision
+
+## Why
+d1 (CER 6.40%, word_acc 47.43%) was evaluated GREEDY only. The Hebrew
+s45 experience: beam-4 at inference is worth double-digit DER points
+on ByT5 diacritizers. Before spending a d2 training run, collect the
+free win and re-read the saturation point.
+
+## Plan
+1. `eval_urdu_d1_beam.py` (rababa): beam-4 vs greedy on the identical
+   test protocol (urdu-diacrit/test.jsonl, CER + word_acc, n=11,714).
+2. Decision after the number:
+   - If beam-4 word_acc still < ~55%: d2 with a second epoch at lower
+     LR (1e-5) from run-001-d1 — cheap, single A100 pass.
+   - If beam-4 lifts word_acc >= ~55%: declare d1 final for the weak
+     corpus; the next real gain requires gold Urdu data (none
+     verifiable — do not chase it).
+
+## Guards
+- Same test set, same alignment protocol as the d1 verdict so the
+  numbers are comparable line-for-line.
diff --git a/TODO.public b/TODO.public
@@ -0,0 +1,68 @@
+# TODO.public — remaining work after the 2026-08-22 audit
+
+Verdicted and frozen: Arabic r6 (2.5793/1.5317 ID, 19.82/12.46 OOD),
+Thai 1.73% PER, Persian v1 (77.34 homograph), Hebrew s45 16.58 (s46
+verdict pending), Urdu d2 5.77 CER (within-corpus), Khmer v1 59%
+word-acc. No further teacher training on Arabic/Thai/Persian — the
+levers below are the only funded work.
+
+## 1. Khmer: reframe as vowel-restoration (EXPANSION, not marginal)
+The UNGEGN rule map (secryst/data-khmer-translit/ungegn-khmer-system.yaml)
+handles FULL orthography; the real-world problem is Khmer's unwritten
+inherent vowels and reduced homographic orthography, which rules
+cannot restore. Deliverable: a model that romanizes INCOMPLETE input.
+- [ ] Rules-vs-ML audit on the 895-word split (full orthography):
+      run the yaml map, publish both numbers honestly.
+- [x] Vowel-strip augmentation: strip vowel signs/coeng from src,
+      keep full tgt; train Khmer v2 on (full→full) + (stripped→full).
+      This is a new CAPABILITY (restoration), measured by a
+      stripped-input benchmark where rules structurally cannot run.
+      DONE (run-002-restore): stripped 19.22% word_acc / 39.41 CER
+      (rules 0.0%); full 53.18/28.27 (−5.8pp vs v1 58.99).
+- [ ] Ship decision: map for full orthography, v2 for reduced.
+      Recommended exactly that; optional parked lever = v1-init
+      two-stage curriculum to reclaim the 5.8pp.
+
+## 2. Urdu: anchor the claim (zero GPU)
+- [ ] Find the official UDD (gold-standard Urdu diacritization)
+      source — the paper's own release, not a guessed URL. If found:
+      evaluate d2 on it, publish the comparable number. If not
+      found: label the claim "within-corpus, no gold available"
+      everywhere it appears.
+
+## 3. Hebrew s47: transplant the r6 morph template — CLOSED (negative)
+Hebrew's 16.58 residual is plausibly morphology (nikud encodes
+gender/number/agreement) — the same shape as Arabic's iʿrāb residual
+that r6's aux-task fixed. The template is validated; this is its
+first cross-language transplant.
+- [x] Ground a Hebrew morph labeler (Dicta morph disambiguator /
+      HebPipe-class POS+morph) — must run offline-ish on Modal.
+      DONE: dictabert-morph (probe + labels verified).
+- [x] Label knesset + hewiki lines with morph tags (A10G, ~3h).
+      DONE: 200K knesset lines → /datasets/hebrew-morph/train.jsonl
+      (dictabert-morph, segmented src + per-token tags, 100% keep
+      after prefix-split fix; hewiki morph not needed for s47 v1).
+- [x] train_hebrew_s47.py: s45 recipe + TAG-prefixed morph stream
+      (init from s46-best if it beats s45, else s45), aux up x4.
+- [x] Gate: beat s45/s46 DER on the identical beam-4 harness.
+      RESULT (2026-08-23): s47 16.53 vs s46 16.43 — FAILED.
+      r6's aux-task win is not template-portable; Arabic's residual
+      was iʿrāb-shaped, Hebrew's is not. s46 stays canonical; line
+      closed (no s48).
+
+## 4. Arabic r6 beam-4 probe — CLOSED (negative)
+All Arabic numbers are greedy; Hebrew gained 12 DER points from
+beam. If beam helps r6, ship beam — free quality, zero training.
+DONE (2026-08-23): beam-4 Total DER 2.5588 vs greedy 2.5793, Morph
+1.5379 vs 1.5317 — noise both ways. Greedy stays (4x cheaper).
+
+## 5. Delivery (agent-side per manifest; our job: keep teachers stable)
+- [ ] IMF v1 export + index registration: urdu-diac d2, khmer
+      decision after §1, r6 teacher refresh for the agent's
+      ara-diac re-distillation.
+- [ ] secryst.org models page refresh with the verdict table.
+- [ ] Close CI: interscript-ruby #764/#763, maps #181.
+
+## Non-goals (money discipline)
+No r8, no more Thai/Persian, no LLM-teacher experiments (#313
+parked), no Khmer training beyond §1's restoration framing.