fix: seven ISC pipeline conformance bugs — the corpus sweep runs end-to-end - #779
Merged
Merged
Conversation
rule(list:) built Items::Set.from_strings(arr.map(&:to_s)); stringifying an any([...]) list baked object inspects into the char set, so maps whose sets contained primitives, alias refs or multi-char entries compiled a garbage class (iso-mal's "പ്പം ഹ" cases). Keep the Items as they are — convert_set already normalizes each entry kind. With this, the env-gated ISC corpus sweep runs end-to-end on the 289-map corpus. It is NOT green yet: the sweep now measures a real conformance gap (case/capitalization post-rules, escape decoding, cluster ordering) that the following commits close.
The .isc branch bypassed @cache, so every transliterate() call re-ran the full Parslet parse — a 700 KB map costs ~30 s per call and blew the 5 s per-example test timeout. The .imp path already caches; parity.
Expanding Items::Range into an Array of single-char Strings iterated
Ruby String#succ (a, b, … z, aa, ab …), which never reaches non-ASCII
codepoints — ranges like any("a".."\uFFFF") matched no non-ASCII
letters, silently dropping the corpus's word-capitalization post-rules
(73% of the sweep failures were case-only). Both runtimes compile
Any(Range) to a codepoint character class.
The Normalizer folded :string captures itself and only read the :char
key — escape fragments ({dquote:}, {backslash:}, {newline:}, {unicode:})
have no :char key and every escaped character silently vanished
(bas-rus "pod\"ezd" tests, gost's hard-sign-to-quote rules, \uXXXX
decoding). The Transform already had the correct fold; share it via
Transform.decode_string_parts.
The sweep is now runnable end-to-end (parse cache, native ranges, escape decoding) but still measures a real conformance gap across the corpus — enabling it in CI would go red. It stays env-gated for local tracking; flip it on once it reaches zero.
Compiler.call called parse_isc directly for .isc paths, re-running a ~30 s Parslet parse on every compiler instantiation — the spec's shared compile cache never got a hit on the first call per compiler, so the first example of every large map timed out. DSL.parse locates, dispatches and caches; one entry point, one cache.
Bare identifiers inside any(...) parsed as :alias_ref, which no transform rule matched — the hash degraded downstream into the debug string "[:alias_ref, \"greek\"]" baked into constraint regexps, so every context-gated rule silently never fired (Greek velar assimilation γ→n, ta-marbuta contexts). Two parts: - transform rules for :alias_ref (list entries and single any() args) - single-alias any() args stay a Set; imported aliases compile to the vacuous fragment exactly as the legacy runtime's Any(nil) does — resolving them to the charset string would bake a 1000-char literal into a lookbehind that never matches
Two parity gaps with the legacy DSL, both fatal to Compiler::Ruby: - Document#name now derives from the file name (locate() and Maps.transliterate address maps that way). Registering under the system code made every map whose code differs from its file name resolve to a Hash-default empty entry — stages[:main].call crashed. 3106 of the sweep's 3175 failures were this crash. - Every alias and stage now carries doc_name. The Ruby/JS compilers emit run directives as Maps.transliterate(stage.doc_name, ...); without it, aliased-dependency runs (un-ell runs elot, 131 maps) compiled to transliterate(nil, s, :main).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the ISC corpus sweep enablement and lands the first tranche of conformance fixes it exposed.
The leak that failed iso-mal
Items::Set.from_strings(arr.map(&:to_s))stringified everyany([...])list entry. A primitive entry —boundaryinsideany([boundary, "", ""])— stringified to its object inspect, and the composed constraint became a garbage character class, soiso-mal-Mlym-Latn-15919-2001's boundary-anchored rules never fired. Lists now keep their Items; the adapter unwraps each entry kind. iso-mal passes 11/11 through the Interpreter and Compiler::Ruby with the stock ASCII\bboundary — the TS-boundary-divergence theory from the earlier analysis was a red herring caused by measuring against the wrong corpus.The sweep was measuring the wrong corpus
Interscript.maps()enumerated the installed gem's frozen 2.4.x.impcorpus, not the maps checkout. TheISC_SWEEPgate now enumerates through the load path (matchinglocate()) and honorsINTERSCRIPT_MAPS_PATH; before that, the "ISC sweep" ran 287 legacy.impmaps and the "six failures" were that corpus's iso-mal, unfixable from this repo.Three conformance bugs the real sweep exposed
Items::Rangeto an array of single chars iterated Ruby's string succ ladder (a, b, … z, aa, ab …), which never reaches non-ASCII codepoints —any("a".."�")matched nothing beyond ASCII. This silently disabled every corpus map's word-capitalization post-rule (sub any("a".."�"), upcase, before: boundary): 73% of the 391 baseline failures were case-only (īşvsĪş). Ranges now stay nativeAny(Range); all three runtimes compile that to the codepoint class[lo-hi].:stringcaptures itself and read only the:charkey, so\",\\,\n,\tand\uXXXXfragments decoded to nothing (bas-rus"pod"ezd", gost's hard-sign-to-quote rules). The Transform already had the correct fold; both paths now shareTransform.decode_string_parts..iscbranch ofDSL.parseandCompiler.call's directparse_isc— re-ran the full Parslet parse per call. A 700 KB map costs ~30 s per parse (the legacy.imppath parses the same map in 0.17 s), which blew the 5 s per-example test timeout. Both entry points now route through the one cached parse.any(greek)parsed as an:alias_refno transform rule matched, degrading to the literal string[:alias_ref, "greek"]inside constraint regexps — every context-gated rule (Greek velar assimilation γ→n, ta-marbuta contexts) silently never fired. Alias refs now transform, and single-aliasany()args keep set semantics matching the legacy runtime'sAny(nil)for imported aliases (resolving them to the charset string would bake a 1000-char literal into a lookbehind that never matches).Compiler::Ruby:Document#namenow derives from the file name (registering under the system code made every map whose code differs from its file name resolve to a Hash-default empty entry —stages[:main].callcrashed; this was 3106 of the sweep's failures), and every alias/stage now carriesdoc_name, which the compilers emit asMaps.transliterate(stage.doc_name, ...)— without it, aliased-dependency runs (un-ell runs elot, 131 maps) compiled totransliterate(nil, s, :main).Sweep status: opt-in, not a CI gate
The sweep runs end-to-end on all 289 ISC maps. It still measures a conformance tail, so
ISC_SWEEP=1stays env-gated for local tracking:Flip it into the rake workflow once the tail reaches zero. The non-sweep suite is green with CI-equivalent flags.