fix: compile not_before/not_after as guards; full expressions in any lists - #12
Merged
Merged
Conversation
…lists Ruby's interpreter compiles not_before to a negative lookbehind and not_after to a negative lookahead (build_regexp). The Python engine counted them in the selection key but never matched against them, so rules like alalc-ara's hamza+fatḥa -> 'a' fired where Ruby declines them: mas'alah came out masalah, ta'ālīf came out tālīf. Three coupled fixes: - isc.py renders not_before/not_after constraints into parallel subs (they were dropped at rendering) and engine.py compiles them; the negative lookbehind distributes over top-level alternatives because Python re requires fixed-width lookbehinds where Onigmo does not. - any([...]) alternatives are full expressions, not quoted strings only. The list tokenizer kept just the quoted strings, silently dropping bare atoms (boundary) and alias references; alternatives now tokenize recursively (regex, literal, max_length). - _library_aliases substitutes alias references defined earlier in the same .iml file (var-kor: jamo = any([jamo_leading_cons, ...])), which the dropped-alternatives path used to hide. alalc-ara: 94/94. un-bul (imports var-kor) and bgnpcgn-deu load and pass again with the previously-dropped alternatives now resolved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #11. Three coupled conformance fixes, all measured against the Ruby interpreter:
1. not_before/not_after are match constraints, not just sort-key weights. Ruby's
build_regexpcompiles them as(?<!...)/(?!...). The Python engine counted them towardmax_lengthparity but never matched against them, so alalc-ara'sfrom "أ"+fatḥa → "a"(not followed by ة/ل) fired where Ruby declines it: mas’alah → masalah, ta’ālīf → tālīf, ma’āthir → māthir, faj’atan → fajatan. isc.py now renders them into parallel subs (they were dropped at rendering) and engine.py compiles them — the negative lookbehind distributes over top-level alternatives ((?<!A|B)≡(?<!A)(?<!B)) because Pythonrerequires fixed-width lookbehinds where Onigmo doesn't.2.
any([...])alternatives are full expressions. The list tokenizer kept only quoted strings, silently dropping bare atoms (any([boundary + "ال", boundary])degraded toany(["ال"])) — after fix 1 gave the guarded rules their real keys, alalc-ara word-initial آ then wrongly took the medial "’ā" rule. Alternatives now tokenize recursively in regex/literal/max_length, with nestedany([any(...)])(var-kor jamo) supported.3. Library alias references resolve at harvest.
def_alias jamo, any([jamo_leading_cons, ...])was stored verbatim; the dropped-alternatives path hid the unresolved names. References to aliases defined earlier in the same .iml are now substituted (string literals untouched).Result: alalc-ara 94/94; un-bul-Cyrl-Latn-1977 (imports var-kor) and bgnpcgn-deu full parity restored; suite 39 passed / 1 xpassed.