Skip to content

feat: support BSL (1C:Enterprise) and OneScript - #63

Open
nixel2007 wants to merge 7 commits into
TelegramMessenger:masterfrom
nixel2007:claude/quirky-ramanujan-kmzkdx
Open

nixel2007 wants to merge 7 commits into
TelegramMessenger:masterfrom
nixel2007:claude/quirky-ramanujan-kmzkdx

Conversation

@nixel2007

@nixel2007 nixel2007 commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

BSL was listed as unsupported in generate.js, because Boost could not compile its patterns. This PR fixes that and the problems around it that review turned up. Each fix is its own commit.

1. fix: support ranges of non-ASCII characters in character classes

BSL bounds its words with a class like [\w + U+0400–U+0484, U+0487–U+052F, U+1D2B, U+1D78, U+2DE0–U+2DFF, U+A640–U+A69F, U+FE2E, U+FE2F ]. widenUnicodeClasses converted single \u escapes to UTF-8 but left ranges unchanged, and Boost rejects such a class with "Invalid character range". tokenize caught the exception and returned the rest of the text without highlighting. Each range is now converted into the UTF-8 byte sequences that cover it, for example U+0400–U+04FF into [\xD0-\xD3][\x80-\xBF], and a negated class consumes a whole UTF-8 character rather than one byte.

2. feat: support BSL (1C:Enterprise) and OneScript

BSL keywords use the i flag, but neither Boost, which compares bytes, nor java.util.regex without UNICODE_CASE ignores the case of Cyrillic letters. generate.js now spells out each non-ASCII letter of a case-insensitive pattern as a class (foldCase), and bsl is removed from unsupported. The oscript alias and its "OneScript" title come from Prism's metadata.

3. fix: keep a negated class from matching part of a character

The whole-character match in a negated class could backtrack and give up continuation bytes, so a match could end inside a character when what followed could match one, such as [\s\S]. It now has one branch per encoding length.

4. fix: fold case in generate.js the way JavaScript's /i does

foldCase used a letter's lower and upper case. Without the u flag, JavaScript's /i treats characters as equal when their canonical upper cases are (ECMAScript's Canonicalize), so /µ/i also matches Greek small mu, /σ/i matches final sigma, and Dž matches itself. The class is now built from every UTF-16 code unit with the same canonical form. That also keeps ſ and ı to themselves, as JavaScript does, rather than adding S s or I i.

5. fix: read \xHH as a code point and drop the U+FFFF replacement

Boost reads \xHH as a byte, where JavaScript and java.util.regex read the code point. Inside a converted class this kept \x85 from excluding U+0085, and a range ending in an \xHH escape was read as its start alone, with the hex digits taken as letters. To make Prism's ranges from \xA0 to U+FFFF compile at all, sanitize rewrote the U+FFFF escape as \xFF, so Boost matched raw bytes and a Cyrillic function name in JavaScript was cut at its first continuation byte. \xHH is now a code point wherever \u escapes are read, ClassMember has an explicit range flag, and the replacement is gone. A range over every surrogate also takes the characters above U+FFFF, as JavaScript takes both halves of each pair.

6. refactor: write appendCodePoints as two statements

7. fix: keep icase from folding the UTF-8 bytes of a pattern

With icase, Boost folds each byte by the locale a pattern is compiled under. On Windows that is the ANSI code page (w32_regex_traits), and in CP1251 the bytes above 0x7F are letters with a case. The UTF-8 bytes were folded as if they were those letters: D0 B2 (в) matched D0 B3 (г), and the byte range 80-84 from U+0400–U+0484 became 90-84, which Boost rejects, so all three case-insensitive BSL patterns failed to compile. The UTF-8 bytes are now written inside (?-i:…), and the ASCII members of a class stay outside it, where icase still applies. This commit is last so it can be dropped on its own if a consumer-side setting is preferred.

Impact on other grammars

  • JavaScript family. With commit 5, names containing non-ASCII letters match the way Prism matches them. Tokenizing one text with Cyrillic, accented and astral characters in 279 languages, 257 now produce the same tokens as Prism, against 246 before; the 11 that changed are javascript, typescript, jsx, tsx, flow, n4js, coffeescript, actionscript, ejs, jsonp and mongodb. No language that matched Prism before differs now.
  • OpenQASM. The duration unit µs under /i now also accepts Greek mu and capital Mu, as Prism does.
  • YAML. The plain scalar's negated class excludes U+0080–U+009F and characters above U+FFFF again, as JavaScript does, and no longer stops in the middle of a multi-byte letter.
  • Speed. Commit 5 makes tokenizing 10–25% slower, measured on 100 KB of JavaScript and 125 KB of mostly Cyrillic text in seven languages. Commit 7 does not change it.

Testing

Run on a freshly generated grammars.dat, the same steps as the check-grammars workflow:

  • check-escapes.js passes.
  • Boost and java.util.regex both compile 3990 of 3990 patterns and agree on every pattern over tools/corpus.txt.
  • check-unicode: 0 failures. Each commit adds the checks that failed before it: code-point ranges in classes, a negated class followed by [\s\S], µs in OpenQASM with all three mus, \xHH at either end of a range, a Cyrillic function name in JavaScript, astral characters in \x80–U+FFFF, BSL keywords in mixed case and word boundaries next to Cyrillic letters.
  • check-unicode now takes an optional locale, and the workflow builds ru_RU.CP1251 with localedef and runs it under that. Under CP1251, all 3990 patterns compile and BSL tokens are the same as in the "C" locale. This uses cpp_regex_traits with the CP1251 locale as a stand-in for Windows; I have not run it on Windows itself.

With java.util.regex on UTF-16 strings, the folded BSL keyword pattern matches Процедура, ЕСЛИ and Тогда and leaves ПеременнаяЕсли alone, and OpenQASM's number pattern matches 5μs.

grammars.dat is not part of this PR. It is rebuilt on master.

Telegram for Android

This PR alone will not bring BSL to the Android client. The client highlights code with libprisma through JNI and takes only part of this repository from its submodule:

  • TMessagesProj/jni/third_party/libprisma is pinned to 94640c6, the commit this PR is based on.
  • TMessagesProj/jni/prisma/CMakeLists.txt builds UnicodeEscapes.h, LanguageTree.cpp and TokenList.cpp from the submodule and replaces Highlight.* and SyntaxHighlighter.* with its own. The changes to UnicodeEscapes.h reach the client once the submodule is updated.
  • The client has its own copy of the generator, TMessagesProj/jni/prisma/generate.js, which differs from this repository's only in how it joins aliases (alias.join(' ') instead of alias[0]). generate_prisma_grammars.sh runs that copy, so the generate.js changes here (foldCase, bsl out of unsupported, the U+FFFF replacement removed) have to be applied to it as well before assets/grammars.dat is regenerated.

With the first two commits applied that way to Telegram-Android 13.2.2 (7303), a standalone arm64-v8a build highlights a ```bsl block on a phone: keywords in any case, strings, numbers, operators and punctuation, the same tokens Prism's BSL grammar produces. The later commits have not been tried in the client.

Closes #3

widenUnicodeClasses converted single \u escapes to UTF-8, but left
ranges such as U+0400 to U+04FF unchanged. Boost cannot compile a
class with such a range and throws an exception. tokenize catches it
and stops, so the rest of the text is returned without highlighting.
In a negated class the same ranges were dropped instead, so the class
did not exclude those characters.

Each range is now converted into the UTF-8 byte sequences that cover
it. Surrogates are skipped, because valid UTF-8 never contains them.
A negated class now consumes a whole UTF-8 character instead of a
single byte, so it can no longer match in the middle of a character.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
BSL was excluded in generate.js because Boost could not compile its
patterns. The previous commit fixes that.

One more problem remained. BSL keywords use the i flag, but neither
Boost, which compares bytes, nor java.util.regex without UNICODE_CASE
ignores the case of Cyrillic letters. Only lower-case keywords such as
"если" were found, not "Если".

generate.js now replaces each non-ASCII letter in a pattern with the i
flag by a class with both of its cases: "если" becomes
"[еЕ][сС][лЛ][иИ]". If such a letter is inside a character class, the
generator stops with an error, because no grammar needs that yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm

@FrayxRulez FrayxRulez left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the approach is right. Splitting ranges into UTF-8 sequences follows the standard algorithm (it matches Rust's Utf8Sequences step by step), and case folding belongs in generate.js rather than the engine. Two things before merge, one in the code and one in the description.

Should fix

1. A negated class can still match part of a character. In appendClass, the negated rewrite is

(?:(?!…)[\x00-\x7F\xC0-\xFF][\x80-\xBF]{0,3})

The {0,3} can backtrack and give up continuation bytes. If what follows can match a continuation byte (., [\s\S], \W, or a narrow class such as [^"]), a match can end in the middle of a character, which is what this commit sets out to prevent. No pattern in the current table triggers it (I traced the YAML scalar and the BSL keyword and number patterns; after a partial byte both fail), but the next grammar may. Making it atomic closes it:

(?>[\x00-\x7F\xC0-\xFF][\x80-\xBF]{0,3})

or one branch per encoding length. Boost takes (?>, and java.util.regex never sees this rewrite. A test with a negated class followed by [\s\S] would cover it.

2. "Only one existing pattern changes" is not quite true. OpenQASM's number pattern has a literal µs under /i, and foldCase turns µ into [µΜ], since 'µ'.toUpperCase() is Greek capital Mu. That change is harmless, but it shows foldCase is not what JavaScript's /i does: without the u flag two characters are equal when their upper cases are, so /µ/i matches µ, Μ and μ (U+03BC), and /σ/i matches ς too. Building the class from every character whose upper case equals c's, plus c itself, would match Prism, and including c also keeps titlecase letters such as Dž and the Kelvin sign, which the current class drops. Cyrillic is unaffected, so this does not block BSL.

Already wrong before this PR, in code it touches

3. A range whose end is not a \u escape is read as one character. ClassMember cannot tell a lone member from a range such as  -\xFF: both have high == -1, so the member.high < 0 branch keeps only U+00A0. The PR adds an else for the mirrored case (start not \u) but not this one. Relatedly, an \xHH member at or above 0x80 inside a converted class is still a raw byte: the YAML class has \x7f-\x84\x86-\x9f, which is now tested only against a character's first byte, so U+0080–U+009F are no longer excluded the way java.util.regex excludes them. Reading \xHH as a code point fixes both, and is what would let the � → \xFF replacement in sanitize go.

4. On Windows desktop builds, Boost's icase folds bytes by the ANSI code page. Boost.Regex picks w32_regex_traits on Win32 unless told otherwise (only Windows Runtime targets default to the C++ locale), and those build their lower-case table with LCMapStringA for the user's locale. Under CP1251, byte 0xB2 lowers to 0xB3 and 0xA8 to 0xB8, so in UTF-8 в (D0 B2) equals г (D0 B3) and и (D0 B8) equals Ш (D0 A8) in the folded BSL keywords. Linux CI uses the "C" locale and cannot see it. It is a consumer build setting (BOOST_REGEX_USE_CPP_LOCALE) rather than something this PR has to change, but BSL is the first grammar that depends on it.

Minor

  • appendCodePoints can be two statements: if (low < 0xD800) add [low, min(high, 0xD7FF)]; if (high > 0xDFFF) add [max(low, 0xE000), high].
  • The comments could be shorter, the paragraph above foldCase and the four lines on the negated rewrite in particular.

The new checks are good: ranges across encoding-length boundaries, surrogate skipping, and a negated class consuming a whole four-byte character.

The negated class took a whole character as a lead byte followed by up
to three continuation bytes. The quantifier could backtrack and give up
continuation bytes, so when what followed could match one, such as
[\s\S], a match could end in the middle of a character.

Match one branch per UTF-8 encoding length instead. Each branch has a
fixed length, so there is nothing to give up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
foldCase spelled out a letter as its lower and upper case. Without the
u flag, JavaScript's /i treats two characters as equal when their
canonical upper cases are, so /µ/i also matches Greek small mu, /σ/i
matches final sigma, and a titlecase letter such as Dž matches itself.
The old class dropped those, and it added k for the Kelvin sign, which
/i does not match.

Build the class from every UTF-16 code unit with the same canonical
form, as ECMAScript's Canonicalize defines it. The only existing
pattern this changes outside BSL is OpenQASM's duration unit µs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
Boost reads \xHH as a byte, where JavaScript and java.util.regex read
the code point. Inside a converted class this kept \x85 from excluding
U+0085, and a range whose end was an \xHH escape was read as its start
alone, with the hex digits taken as letters: a class from U+00E0 to
\xff became "U+00E0 or f".

To make Prism's ranges from \xA0 to U+FFFF compile at all, sanitize
rewrote the U+FFFF escape as \xFF. Boost then matched raw bytes, so a
JavaScript function name in Cyrillic was cut at its first continuation
byte, and java.util.regex matched only up to U+00FF.

Read \xHH as a code point wherever \u escapes are read, give
ClassMember an explicit range flag, keep the ASCII part of a range in
the class where the i flag applies to it, and remove the replacement.
A range over every surrogate also takes the characters above U+FFFF,
as JavaScript takes both halves of each pair.

Tokenizing one text with Cyrillic, accented and astral characters in
279 languages, 257 now match Prism against 246 before, and none that
matched before differ. Tokenizing is 10-25% slower, measured on 100 KB
of JavaScript and on 125 KB of mostly Cyrillic text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
The four branches that skip the surrogates are the two halves of the
range on either side of them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
With icase, Boost folds each byte by the locale it compiles a pattern
under. On Windows that is the ANSI code page, and in CP1251 the bytes
above 0x7F are letters with a case. The UTF-8 bytes this library writes
were folded as if they were those letters: D0 B2 ("в") matched D0 B3
("г"), and the byte range 80-84 in U+0400 to U+0484 became 90-84,
which Boost rejects. All three case-insensitive BSL patterns failed to
compile under CP1251, and tokenize stopped highlighting there.

Write the UTF-8 bytes inside (?-i:) and keep the ASCII members of a
class outside it, where icase still applies. The case of a letter above
ASCII is already spelled out by foldCase in generate.js.

check-unicode takes a locale to run these checks under, and the
workflow builds ru_RU.CP1251 with localedef for it. This stands in for
Windows, where Boost uses w32_regex_traits rather than
cpp_regex_traits; it was not run on Windows itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013dHLrzWVjUm5n5yTmAfmWm
@nixel2007

Copy link
Copy Markdown
Author

Thanks for the careful review @FrayxRulez
All four points held up, and I pushed a fix for each as its own commit, so any of them can go if you disagree.

1. 21a3de5: the negated class now takes one branch per encoding length, so there is nothing to give up. There's a test with [\s\S] after it, on a two-byte and a four-byte character.

2. cfd4418: you were right about µ and σ, and the description is fixed. One thing I changed from your suggestion: "same upper case, plus c" adds S s to ſ and I i to ı, but /ſ/i and /ı/i only match themselves, because Canonicalize keeps a non-ASCII character whose upper case is ASCII as it is. So the class is built from Canonicalize and checked against Node's /i over the whole BMP. The Kelvin sign wasn't dropped, by the way: the old class added a k to it, which /i doesn't match.

3. c10cb03: \xHH is read as a code point wherever \u is, ClassMember has an explicit range flag, and the U+FFFF → \xFF replacement in sanitize is gone. The range case was a bit worse than one character: with \xff read as a two-character escape, the ff became letters of the class.

Removing the replacement changes more than YAML, so I compared one text with Cyrillic, accented and astral characters across 279 languages against Prism itself. 257 now match Prism, against 246 before. The 11 that changed are the JavaScript family, where a Cyrillic function name used to be cut at its first continuation byte, and nothing that matched before broke. It did show one regression on the way: dot's \x80–U+FFFF stopped matching an emoji, since in JavaScript that range takes both halves of a pair. A range over every surrogate now takes the characters above U+FFFF too. The cost is speed: tokenizing is 10–25% slower on 100 KB of JavaScript and 125 KB of mostly Cyrillic text.

4. 5c1a771, the last commit. This one goes further than wrong folds. I modelled it on Linux with a localedef-built ru_RU.CP1251 as the global locale, so cpp_regex_traits rather than w32_regex_traits. With icase, Boost also translates the ends of the byte range [\x80-\x84] that U+0400–U+0484 produces: \x80 lowers to \x90, the range comes out reversed, and all three case-insensitive BSL patterns throw. 3987 of 3990 patterns compiled, against 3981 of 3981 before this PR.

Rather than depend on how the consumer builds Boost, the commit writes the generated UTF-8 bytes inside (?-i:…) and keeps the ASCII members of a class outside. Non-ASCII case is already spelled out by foldCase. Under CP1251 all 3990 patterns now compile and BSL tokens match the "C" locale. The workflow builds the locale with localedef and runs check-unicode under it. I haven't run it on real Windows, and if you'd rather keep this a consumer-side setting, dropping that commit leaves everything else intact.

Minor: 7245a6f makes appendCodePoints two statements, and the comments over foldCase and the negated rewrite are shorter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[question] Why bsl language is not supported?

3 participants