Skip to content

fix: seven ISC pipeline conformance bugs — the corpus sweep runs end-to-end - #779

Merged
ronaldtse merged 10 commits into
mainfrom
fix/any-list-items-leak
Sep 30, 2026
Merged

ronaldtse merged 10 commits into
mainfrom
fix/any-list-items-leak

Conversation

@ronaldtse

@ronaldtse ronaldtse commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Closes the ISC corpus sweep enablement and lands the first tranche of conformance fixes it exposed.

The leak that failed iso-mal

Items::Set.from_strings(arr.map(&:to_s)) stringified every any([...]) list entry. A primitive entry — boundary inside any([boundary, "‌", "‍"]) — stringified to its object inspect, and the composed constraint became a garbage character class, so iso-mal-Mlym-Latn-15919-2001's boundary-anchored rules never fired. Lists now keep their Items; the adapter unwraps each entry kind. iso-mal passes 11/11 through the Interpreter and Compiler::Ruby with the stock ASCII \b boundary — the TS-boundary-divergence theory from the earlier analysis was a red herring caused by measuring against the wrong corpus.

The sweep was measuring the wrong corpus

Interscript.maps() enumerated the installed gem's frozen 2.4.x .imp corpus, not the maps checkout. The ISC_SWEEP gate now enumerates through the load path (matching locate()) and honors INTERSCRIPT_MAPS_PATH; before that, the "ISC sweep" ran 287 legacy .imp maps and the "six failures" were that corpus's iso-mal, unfixable from this repo.

Three conformance bugs the real sweep exposed

  1. Ranges exploded into String#succ sequences. Expanding Items::Range to an array of single chars iterated Ruby's string succ ladder (a, b, … z, aa, ab …), which never reaches non-ASCII codepoints — any("a".."�") matched nothing beyond ASCII. This silently disabled every corpus map's word-capitalization post-rule (sub any("a".."�"), upcase, before: boundary): 73% of the 391 baseline failures were case-only (īş vs Īş). Ranges now stay native Any(Range); all three runtimes compile that to the codepoint class [lo-hi].
  2. Escape sequences silently vanished. The Normalizer folded :string captures itself and read only the :char key, so \", \\, \n, \t and \uXXXX fragments decoded to nothing (bas-rus "pod"ezd", gost's hard-sign-to-quote rules). The Transform already had the correct fold; both paths now share Transform.decode_string_parts.
  3. Every transliterate() call re-parsed the map. Two bypasses — the .isc branch of DSL.parse and Compiler.call's direct parse_isc — re-ran the full Parslet parse per call. A 700 KB map costs ~30 s per parse (the legacy .imp path parses the same map in 0.17 s), which blew the 5 s per-example test timeout. Both entry points now route through the one cached parse.
  4. Bare alias refs inside any(...) leaked a debug string. any(greek) parsed as an :alias_ref no transform rule matched, degrading to the literal string [:alias_ref, "greek"] inside constraint regexps — every context-gated rule (Greek velar assimilation γ→n, ta-marbuta contexts) silently never fired. Alias refs now transform, and single-alias any() args keep set semantics matching the legacy runtime's Any(nil) for imported aliases (resolving them to the charset string would bake a 1000-char literal into a lookbehind that never matches).
  5. ISC documents weren't addressable like .imp maps. Two parity gaps, both fatal to Compiler::Ruby: Document#name now derives from the file name (registering under the system code made every map whose code differs from its file name resolve to a Hash-default empty entry — stages[:main].call crashed; this was 3106 of the sweep's failures), and every alias/stage now carries doc_name, which the compilers emit as Maps.transliterate(stage.doc_name, ...) — without it, aliased-dependency runs (un-ell runs elot, 131 maps) compiled to transliterate(nil, s, :main).

Sweep status: opt-in, not a CI gate

The sweep runs end-to-end on all 289 ISC maps. It still measures a conformance tail, so ISC_SWEEP=1 stays env-gated for local tracking:

ISC_SWEEP=1 INTERSCRIPT_MAPS_PATH=../maps/maps \
  rspec spec/interscript_spec.rb

Flip it into the rake workflow once the tail reaches zero. The non-sweep suite is green with CI-equivalent flags.

rule(list:) built Items::Set.from_strings(arr.map(&:to_s)); stringifying
an any([...]) list baked object inspects into the char set, so maps
whose sets contained primitives, alias refs or multi-char entries
compiled a garbage class (iso-mal's "പ്പം ഹ" cases). Keep the Items as
they are — convert_set already normalizes each entry kind.

With this, the env-gated ISC corpus sweep runs end-to-end on the 289-map
corpus. It is NOT green yet: the sweep now measures a real conformance
gap (case/capitalization post-rules, escape decoding, cluster ordering)
that the following commits close.
The .isc branch bypassed @cache, so every transliterate() call re-ran
the full Parslet parse — a 700 KB map costs ~30 s per call and blew the
5 s per-example test timeout. The .imp path already caches; parity.
Expanding Items::Range into an Array of single-char Strings iterated
Ruby String#succ (a, b, … z, aa, ab …), which never reaches non-ASCII
codepoints — ranges like any("a".."\uFFFF") matched no non-ASCII
letters, silently dropping the corpus's word-capitalization post-rules
(73% of the sweep failures were case-only). Both runtimes compile
Any(Range) to a codepoint character class.
The Normalizer folded :string captures itself and only read the :char
key — escape fragments ({dquote:}, {backslash:}, {newline:}, {unicode:})
have no :char key and every escaped character silently vanished
(bas-rus "pod\"ezd" tests, gost's hard-sign-to-quote rules, \uXXXX
decoding). The Transform already had the correct fold; share it via
Transform.decode_string_parts.
The sweep is now runnable end-to-end (parse cache, native ranges,
escape decoding) but still measures a real conformance gap across the
corpus — enabling it in CI would go red. It stays env-gated for local
tracking; flip it on once it reaches zero.
Compiler.call called parse_isc directly for .isc paths, re-running a
~30 s Parslet parse on every compiler instantiation — the spec's shared
compile cache never got a hit on the first call per compiler, so the
first example of every large map timed out. DSL.parse locates,
dispatches and caches; one entry point, one cache.
Bare identifiers inside any(...) parsed as :alias_ref, which no
transform rule matched — the hash degraded downstream into the debug
string "[:alias_ref, \"greek\"]" baked into constraint regexps, so
every context-gated rule silently never fired (Greek velar
assimilation γ→n, ta-marbuta contexts). Two parts:

- transform rules for :alias_ref (list entries and single any() args)
- single-alias any() args stay a Set; imported aliases compile to the
  vacuous fragment exactly as the legacy runtime's Any(nil) does —
  resolving them to the charset string would bake a 1000-char literal
  into a lookbehind that never matches
Two parity gaps with the legacy DSL, both fatal to Compiler::Ruby:

- Document#name now derives from the file name (locate() and
  Maps.transliterate address maps that way). Registering under the
  system code made every map whose code differs from its file name
  resolve to a Hash-default empty entry — stages[:main].call crashed.
  3106 of the sweep's 3175 failures were this crash.

- Every alias and stage now carries doc_name. The Ruby/JS compilers
  emit run directives as Maps.transliterate(stage.doc_name, ...);
  without it, aliased-dependency runs (un-ell runs elot, 131 maps)
  compiled to transliterate(nil, s, :main).
@ronaldtse ronaldtse changed the title fix: five ISC pipeline conformance bugs — the corpus sweep runs end-to-end fix: seven ISC pipeline conformance bugs — the corpus sweep runs end-to-end Sep 30, 2026
@ronaldtse
ronaldtse merged commit 7cc476d into main Sep 30, 2026
4 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant