Repository navigation
Conversation
widenUnicodeClasses converted single \u escapes to UTF-8, but left ranges such as U+0400 to U+04FF unchanged. Boost cannot compile a class with such a range and throws an exception. tokenize catches it and stops, so the rest of the text is returned without highlighting. In a negated class the same ranges were dropped instead, so the class did not exclude those characters. Each range is now converted into the UTF-8 byte sequences that cover it. Surrogates are skipped, because valid UTF-8 never contains them. A negated class now consumes a whole UTF-8 character instead of a single byte, so it can no longer match in the middle of a character. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
BSL was excluded in generate.js because Boost could not compile its patterns. The previous commit fixes that. One more problem remained. BSL keywords use the i flag, but neither Boost, which compares bytes, nor java.util.regex without UNICODE_CASE ignores the case of Cyrillic letters. Only lower-case keywords such as "если" were found, not "Если". generate.js now replaces each non-ASCII letter in a pattern with the i flag by a class with both of its cases: "если" becomes "[еЕ][сС][лЛ][иИ]". If such a letter is inside a character class, the generator stops with an error, because no grammar needs that yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
FrayxRulez
left a comment
There was a problem hiding this comment.
Thanks, the approach is right. Splitting ranges into UTF-8 sequences follows the standard algorithm (it matches Rust's Utf8Sequences step by step), and case folding belongs in generate.js rather than the engine. Two things before merge, one in the code and one in the description.
Should fix
1. A negated class can still match part of a character. In appendClass, the negated rewrite is
(?:(?!…)[\x00-\x7F\xC0-\xFF][\x80-\xBF]{0,3})
The {0,3} can backtrack and give up continuation bytes. If what follows can match a continuation byte (., [\s\S], \W, or a narrow class such as [^"]), a match can end in the middle of a character, which is what this commit sets out to prevent. No pattern in the current table triggers it (I traced the YAML scalar and the BSL keyword and number patterns; after a partial byte both fail), but the next grammar may. Making it atomic closes it:
(?>[\x00-\x7F\xC0-\xFF][\x80-\xBF]{0,3})
or one branch per encoding length. Boost takes (?>, and java.util.regex never sees this rewrite. A test with a negated class followed by [\s\S] would cover it.
2. "Only one existing pattern changes" is not quite true. OpenQASM's number pattern has a literal µs under /i, and foldCase turns µ into [µΜ], since 'µ'.toUpperCase() is Greek capital Mu. That change is harmless, but it shows foldCase is not what JavaScript's /i does: without the u flag two characters are equal when their upper cases are, so /µ/i matches µ, Μ and μ (U+03BC), and /σ/i matches ς too. Building the class from every character whose upper case equals c's, plus c itself, would match Prism, and including c also keeps titlecase letters such as Dž and the Kelvin sign, which the current class drops. Cyrillic is unaffected, so this does not block BSL.
Already wrong before this PR, in code it touches
3. A range whose end is not a \u escape is read as one character. ClassMember cannot tell a lone member from a range such as -\xFF: both have high == -1, so the member.high < 0 branch keeps only U+00A0. The PR adds an else for the mirrored case (start not \u) but not this one. Relatedly, an \xHH member at or above 0x80 inside a converted class is still a raw byte: the YAML class has \x7f-\x84\x86-\x9f, which is now tested only against a character's first byte, so U+0080–U+009F are no longer excluded the way java.util.regex excludes them. Reading \xHH as a code point fixes both, and is what would let the � → \xFF replacement in sanitize go.
4. On Windows desktop builds, Boost's icase folds bytes by the ANSI code page. Boost.Regex picks w32_regex_traits on Win32 unless told otherwise (only Windows Runtime targets default to the C++ locale), and those build their lower-case table with LCMapStringA for the user's locale. Under CP1251, byte 0xB2 lowers to 0xB3 and 0xA8 to 0xB8, so in UTF-8 в (D0 B2) equals г (D0 B3) and и (D0 B8) equals Ш (D0 A8) in the folded BSL keywords. Linux CI uses the "C" locale and cannot see it. It is a consumer build setting (BOOST_REGEX_USE_CPP_LOCALE) rather than something this PR has to change, but BSL is the first grammar that depends on it.
Minor
appendCodePointscan be two statements:if (low < 0xD800)add[low, min(high, 0xD7FF)];if (high > 0xDFFF)add[max(low, 0xE000), high].- The comments could be shorter, the paragraph above
foldCaseand the four lines on the negated rewrite in particular.
The new checks are good: ranges across encoding-length boundaries, surrogate skipping, and a negated class consuming a whole four-byte character.
The negated class took a whole character as a lead byte followed by up to three continuation bytes. The quantifier could backtrack and give up continuation bytes, so when what followed could match one, such as [\s\S], a match could end in the middle of a character. Match one branch per UTF-8 encoding length instead. Each branch has a fixed length, so there is nothing to give up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
foldCase spelled out a letter as its lower and upper case. Without the u flag, JavaScript's /i treats two characters as equal when their canonical upper cases are, so /µ/i also matches Greek small mu, /σ/i matches final sigma, and a titlecase letter such as Dž matches itself. The old class dropped those, and it added k for the Kelvin sign, which /i does not match. Build the class from every UTF-16 code unit with the same canonical form, as ECMAScript's Canonicalize defines it. The only existing pattern this changes outside BSL is OpenQASM's duration unit µs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
Boost reads \xHH as a byte, where JavaScript and java.util.regex read the code point. Inside a converted class this kept \x85 from excluding U+0085, and a range whose end was an \xHH escape was read as its start alone, with the hex digits taken as letters: a class from U+00E0 to \xff became "U+00E0 or f". To make Prism's ranges from \xA0 to U+FFFF compile at all, sanitize rewrote the U+FFFF escape as \xFF. Boost then matched raw bytes, so a JavaScript function name in Cyrillic was cut at its first continuation byte, and java.util.regex matched only up to U+00FF. Read \xHH as a code point wherever \u escapes are read, give ClassMember an explicit range flag, keep the ASCII part of a range in the class where the i flag applies to it, and remove the replacement. A range over every surrogate also takes the characters above U+FFFF, as JavaScript takes both halves of each pair. Tokenizing one text with Cyrillic, accented and astral characters in 279 languages, 257 now match Prism against 246 before, and none that matched before differ. Tokenizing is 10-25% slower, measured on 100 KB of JavaScript and on 125 KB of mostly Cyrillic text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
The four branches that skip the surrogates are the two halves of the range on either side of them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
With icase, Boost folds each byte by the locale it compiles a pattern
under. On Windows that is the ANSI code page, and in CP1251 the bytes
above 0x7F are letters with a case. The UTF-8 bytes this library writes
were folded as if they were those letters: D0 B2 ("в") matched D0 B3
("г"), and the byte range 80-84 in U+0400 to U+0484 became 90-84,
which Boost rejects. All three case-insensitive BSL patterns failed to
compile under CP1251, and tokenize stopped highlighting there.
Write the UTF-8 bytes inside (?-i:) and keep the ASCII members of a
class outside it, where icase still applies. The case of a letter above
ASCII is already spelled out by foldCase in generate.js.
check-unicode takes a locale to run these checks under, and the
workflow builds ru_RU.CP1251 with localedef for it. This stands in for
Windows, where Boost uses w32_regex_traits rather than
cpp_regex_traits; it was not run on Windows itself.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
|
Thanks for the careful review @FrayxRulez 1. 2. 3. Removing the replacement changes more than YAML, so I compared one text with Cyrillic, accented and astral characters across 279 languages against Prism itself. 257 now match Prism, against 246 before. The 11 that changed are the JavaScript family, where a Cyrillic function name used to be cut at its first continuation byte, and nothing that matched before broke. It did show one regression on the way: 4. Rather than depend on how the consumer builds Boost, the commit writes the generated UTF-8 bytes inside Minor: |
Summary
BSL was listed as unsupported in
generate.js, because Boost could not compile its patterns. This PR fixes that and the problems around it that review turned up. Each fix is its own commit.1.
fix: support ranges of non-ASCII characters in character classesBSL bounds its words with a class like
[\w+ U+0400–U+0484, U+0487–U+052F, U+1D2B, U+1D78, U+2DE0–U+2DFF, U+A640–U+A69F, U+FE2E, U+FE2F].widenUnicodeClassesconverted single\uescapes to UTF-8 but left ranges unchanged, and Boost rejects such a class with "Invalid character range".tokenizecaught the exception and returned the rest of the text without highlighting. Each range is now converted into the UTF-8 byte sequences that cover it, for example U+0400–U+04FF into[\xD0-\xD3][\x80-\xBF], and a negated class consumes a whole UTF-8 character rather than one byte.2.
feat: support BSL (1C:Enterprise) and OneScriptBSL keywords use the
iflag, but neither Boost, which compares bytes, nor java.util.regex withoutUNICODE_CASEignores the case of Cyrillic letters.generate.jsnow spells out each non-ASCII letter of a case-insensitive pattern as a class (foldCase), andbslis removed fromunsupported. Theoscriptalias and its "OneScript" title come from Prism's metadata.3.
fix: keep a negated class from matching part of a characterThe whole-character match in a negated class could backtrack and give up continuation bytes, so a match could end inside a character when what followed could match one, such as
[\s\S]. It now has one branch per encoding length.4.
fix: fold case in generate.js the way JavaScript's /i doesfoldCaseused a letter's lower and upper case. Without theuflag, JavaScript's/itreats characters as equal when their canonical upper cases are (ECMAScript'sCanonicalize), so/µ/ialso matches Greek small mu,/σ/imatches final sigma, andDžmatches itself. The class is now built from every UTF-16 code unit with the same canonical form. That also keepsſandıto themselves, as JavaScript does, rather than addingS sorI i.5.
fix: read \xHH as a code point and drop the U+FFFF replacementBoost reads
\xHHas a byte, where JavaScript and java.util.regex read the code point. Inside a converted class this kept\x85from excluding U+0085, and a range ending in an\xHHescape was read as its start alone, with the hex digits taken as letters. To make Prism's ranges from\xA0to U+FFFF compile at all,sanitizerewrote the U+FFFF escape as\xFF, so Boost matched raw bytes and a Cyrillic function name in JavaScript was cut at its first continuation byte.\xHHis now a code point wherever\uescapes are read,ClassMemberhas an explicit range flag, and the replacement is gone. A range over every surrogate also takes the characters above U+FFFF, as JavaScript takes both halves of each pair.6.
refactor: write appendCodePoints as two statements7.
fix: keep icase from folding the UTF-8 bytes of a patternWith
icase, Boost folds each byte by the locale a pattern is compiled under. On Windows that is the ANSI code page (w32_regex_traits), and in CP1251 the bytes above 0x7F are letters with a case. The UTF-8 bytes were folded as if they were those letters: D0 B2 (в) matched D0 B3 (г), and the byte range80-84from U+0400–U+0484 became90-84, which Boost rejects, so all three case-insensitive BSL patterns failed to compile. The UTF-8 bytes are now written inside(?-i:…), and the ASCII members of a class stay outside it, whereicasestill applies. This commit is last so it can be dropped on its own if a consumer-side setting is preferred.Impact on other grammars
µsunder/inow also accepts Greek mu and capital Mu, as Prism does.Testing
Run on a freshly generated
grammars.dat, the same steps as thecheck-grammarsworkflow:check-escapes.jspasses.tools/corpus.txt.check-unicode: 0 failures. Each commit adds the checks that failed before it: code-point ranges in classes, a negated class followed by[\s\S],µsin OpenQASM with all three mus,\xHHat either end of a range, a Cyrillic function name in JavaScript, astral characters in\x80–U+FFFF, BSL keywords in mixed case and word boundaries next to Cyrillic letters.check-unicodenow takes an optional locale, and the workflow buildsru_RU.CP1251withlocaledefand runs it under that. Under CP1251, all 3990 patterns compile and BSL tokens are the same as in the "C" locale. This usescpp_regex_traitswith the CP1251 locale as a stand-in for Windows; I have not run it on Windows itself.With java.util.regex on UTF-16 strings, the folded BSL keyword pattern matches
Процедура,ЕСЛИandТогдаand leavesПеременнаяЕслиalone, and OpenQASM's number pattern matches5μs.grammars.datis not part of this PR. It is rebuilt on master.Telegram for Android
This PR alone will not bring BSL to the Android client. The client highlights code with libprisma through JNI and takes only part of this repository from its submodule:
TMessagesProj/jni/third_party/libprismais pinned to94640c6, the commit this PR is based on.TMessagesProj/jni/prisma/CMakeLists.txtbuildsUnicodeEscapes.h,LanguageTree.cppandTokenList.cppfrom the submodule and replacesHighlight.*andSyntaxHighlighter.*with its own. The changes toUnicodeEscapes.hreach the client once the submodule is updated.TMessagesProj/jni/prisma/generate.js, which differs from this repository's only in how it joins aliases (alias.join(' ')instead ofalias[0]).generate_prisma_grammars.shruns that copy, so thegenerate.jschanges here (foldCase,bslout ofunsupported, the U+FFFF replacement removed) have to be applied to it as well beforeassets/grammars.datis regenerated.With the first two commits applied that way to Telegram-Android 13.2.2 (7303), a standalone arm64-v8a build highlights a
```bslblock on a phone: keywords in any case, strings, numbers, operators and punctuation, the same tokens Prism's BSL grammar produces. The later commits have not been tried in the client.Closes #3