diff --git a/Apps/Maps/CHANGELOG.md b/Apps/Maps/CHANGELOG.md index 5b622f3..0abef58 100644 --- a/Apps/Maps/CHANGELOG.md +++ b/Apps/Maps/CHANGELOG.md @@ -1,5 +1,50 @@ # Changelog +## 0.5.0 + +Offline search of places, streets, addresses and kinds of places in the downloaded maps. + +- Menu → Search, / or F: results while typing from 3 letters, nearest first, with type, distance and direction; + addresses as "street number" in either order; diacritics and Greek/Cyrillic in Latin letters don't matter; a + shorter beginning is tried when nothing matches. Nearby by category, the last 10 chosen results. +- A chosen result is shown on the map with a mark and its sheet with Go and Save as waypoint; streets fit the screen. +- House numbers get the nearest street within 50 m, up to 100 m marked as estimated; village addresses in regions + where most numbers are plain. +- The search index per region (`maps//search.idx`, about 1.5 MB for Bratislava) is made on the device in + idle time, the region on screen first, and again after the region's tiles are updated; big regions get one index + per block of 15 x 15 tiles. Bratislava: about 5 minutes on a T-Deck, the same file the PC tool makes. On the T-Deck + a keystroke takes 1-150 ms (Bratislava). +- House numbers also take named paths and pedestrian ways as their street (88 % right in Bratislava against OSM + addresses, 87 % without). +- Results show the town or village they are in ("Street · Holice"), and the city district inside a city or town + ("Address · Ružinov, Bratislava"), so the same street name in two places can be told apart. Estimated from the + places in the map data; works with indexes already on the card. + +**Fixes** +- While a region downloads, the tiles on screen get their POIs and house numbers after a second without input, + instead of after the whole download (they were not prepared while any download ran). The tile cache makes room + for them, and a tile on screen that the download moves into the new region stays first in line. +- Letters the UI font lacks (such as š in "Spišská Nová Ves") are drawn as their base letter instead of being left + out. + +## 0.4.1 + +Leaving the route while navigating is noticed sooner when the GPS is good, and not by mistake when it is poor. + +- The off-route limit follows the GPS precision: the median accuracy of the last 10 s of fixes (a single bad fix + doesn't count) gives a limit of 1.5 x accuracy + 5 m, between 15 m and 60 m (was 40 m, or accuracy + 25 m). +- Three positions in a row beyond the limit are off route (was 10 s). With poor GPS a few more are needed (25 m + accuracy: 8), because GPS noise of that size alone crosses the limit now and then on a long walk. +- Going into another street is noticed earlier: moving faster than 1 m/s more than 60 degrees away from the route's + direction, or being clearly nearer to a side street than to the route (with good GPS), counts from two thirds of + the limit. +- The first new route is searched at once, later ones at most every 10 s (was 20 s); a new route settles for two + positions before it is checked. +- Worse than 50 m accuracy: "Weak GPS" on the navigation bar, the route is followed without recalculating. +- Fixes without an accuracy count as 30 m (Tactility gives HDOP x 5 m when the GPS reports HDOP). +- For testing: the "replay" debug command can add GPS noise and a set accuracy, "offroute" changes the rule's + values on the device, and the serial log has one line per position while navigating. + ## 0.4.0 Offline navigation: routes for walking, cycling and driving, found on the device from the downloaded maps, with a diff --git a/Apps/Maps/README.md b/Apps/Maps/README.md index 8213038..64bf64e 100644 --- a/Apps/Maps/README.md +++ b/Apps/Maps/README.md @@ -4,7 +4,7 @@ Offline vector maps for [Tactility](https://github.com/TactilityProject/Tactilit downloads an area from [OpenFreeMap](https://openfreemap.org/) by itself and draws it fullscreen, with anti-aliased roads, water, parks and buildings, from tiles on the SD card. -Version 0.4.0, tested on a LilyGO T-Deck (ESP32-S3, 8 MB PSRAM); it also runs on ESP32-P4. Changes are listed in +Version 0.5.0, tested on a LilyGO T-Deck (ESP32-S3, 8 MB PSRAM); it also runs on ESP32-P4. Changes are listed in [CHANGELOG.md](CHANGELOG.md). ## Install @@ -132,8 +132,13 @@ S, Esc). - **Start** follows the route: the map turns with your direction (heading up), your position sits low on the screen so you see further ahead, and a bar at the top shows the next turn with an arrow and the street, e.g. "In 120 m turn left onto Štefánikova", and below it the distance and time left. The X at the right end of the bar (or Esc) - stops navigating; the menu button is below the bar. The route ahead is blue, the part you passed grey. More than 40 m off the route for 10 seconds: - "Off route — recalculating", and a new route from where you are (at most every 20 s). Within about 25 m of the + stops navigating; the menu button is below the bar. The route ahead is blue, the part you passed grey. +- **Leaving the route** is noticed as soon as the GPS allows: the limit follows the GPS precision of the last + 10 seconds (about 15 m with a good fix, about 40 m between tall buildings), and three positions in a row beyond + it, or going into another street, mean "Off route — recalculating" (with poor GPS a few more positions are + needed, so it doesn't happen by mistake). The first new route comes at once, later ones at most every 10 s. + When the GPS is very poor (worse than about 50 m) the bar says "Weak GPS" and the app keeps following the route + without recalculating. Within about 25 m of the destination: "You have arrived"; following stops and the line stays. - **Start and destination** snap to the nearest road the mode may use within 200 m (500 m for a car: the last bit is on foot), leaving out small pieces of road that lead nowhere for that mode, such as a service road inside a @@ -148,6 +153,29 @@ S, Esc). tiles in idle time (the amber dot). Without a road within 200 m of the start or the destination, or without a connection between them in the downloaded map, the app says so. +**Search (0.5.0).** Menu → Search, or / or F on the keyboard, opens the search sheet over the map: +type a place, a street, an address ("Štefánikova 4", "4 Štefánikova", "Hlavná 12/A") or a kind of place ("cafe", +"pharmacy"). Results come from 3 letters on, while you type, without a network: the nearest first, each with its type +in plain words ("Café", "Street", "Address"), the distance and a small arrow pointing to it (from your position, or +from the map centre, which the sheet says once). Diacritics don't matter ("stefanikova" finds "Štefánikova"), and +Greek and Cyrillic names are found in Latin letters too ("kyiv"). Nothing found: the sheet shows what the longest +matching beginning of your text finds ("Showing results for 'stefan'"). With an empty field the sheet offers "Nearby +by category" (food, café, shops, supermarket, pharmacy, toilets, stops, parking, fuel, bank/ATM, hospital, viewpoint) +and your last 10 chosen results. Every result names the town or village it is in, and the city district inside a city +("Street · Holice", "Supermarket · Petržalka, Bratislava"), estimated from the nearest places on the map, a bigger +place counting from further away. Choosing a result moves the map there with a red mark and opens its sheet with Go, +Save as waypoint and Close; a street is shown whole. Roll or W/S move through the list, Enter opens, Esc (or +Backspace on an empty field) closes; on touch boards the on-screen keyboard appears. +House numbers in the map data carry no street name: the app gives each one the nearest named street within 50 m +(88 % right in Bratislava against real OSM addresses), or one within 100 m marked "Street estimated from the map". In +regions where most addresses are "village + number" (much of Slovakia), plain numbers are also found under the +nearest village ("Village estimated from the map"; 78 % right on Žitný ostrov). Always check the place on the map before you +go. The search index (about 1.5 MB for a city) is made on the device for each downloaded region, once the map has +been left alone for 20 s and no download runs; the work dot then says "Building search index". Bratislava (99 tiles) +takes about 5 minutes on a T-Deck; it is made again when the region's tiles are updated. Regions larger than 15 x 15 +z14 tiles get one index per block of 15 x 15 tiles. Until a region's index is ready, search doesn't find anything in +it. The PC tool `host/search_build --tiles` makes the same file. + **Menu:** Download map, Downloaded maps, Waypoints, Add waypoint, Record track, GPX files, Location, Settings, About and licences, Exit. W/S, trackball or finger to move; Enter, click or tap to choose; Esc, Backspace or tap outside to close. diff --git a/Apps/Maps/main/CMakeLists.txt b/Apps/Maps/main/CMakeLists.txt index b11441b..be906b2 100644 --- a/Apps/Maps/main/CMakeLists.txt +++ b/Apps/Maps/main/CMakeLists.txt @@ -6,3 +6,7 @@ tactility_component_register( REQUIRES graphics-module binfont-module location-module http-module esp_http_client esp_rom mbedtls ) target_compile_options(${COMPONENT_LIB} PRIVATE -O2 -Wno-format-truncation) +# The search index must match the PC's byte for byte (M6): no fused multiply-add in its float maths (MSVC on the PC +# does not fuse; GCC would, and one house at the 50 m limit then went to another street) +file(GLOB SEARCH_SOURCES Source/search/*.cpp) +set_source_files_properties(${SEARCH_SOURCES} PROPERTIES COMPILE_OPTIONS "-ffp-contract=off") diff --git a/Apps/Maps/main/Source/app/BinFontText.cpp b/Apps/Maps/main/Source/app/BinFontText.cpp index 667d684..9e7a0db 100644 --- a/Apps/Maps/main/Source/app/BinFontText.cpp +++ b/Apps/Maps/main/Source/app/BinFontText.cpp @@ -2,6 +2,7 @@ #include "BinFontText.h" #include "../platform/Platform.h" +#include "../search/TextFold.h" #include #include @@ -9,6 +10,7 @@ #include #include +#include #include #include @@ -36,14 +38,21 @@ uint32_t nextCodepoint(const char*& text) { if (first < 0x80) { return first; } - if ((first & 0xE0) == 0xC0 && *text != '\0') { - return ((first & 0x1Fu) << 6) | (static_cast(*text++) & 0x3Fu); + // 2, 3 or 4 bytes; a broken sequence is skipped and shown as '?' + const int extra = (first & 0xE0) == 0xC0 ? 1 : (first & 0xF0) == 0xE0 ? 2 : (first & 0xF8) == 0xF0 ? 3 : 0; + uint32_t codepoint = first & (0x3Fu >> extra); + int taken = 0; + while (taken < extra && (static_cast(*text) & 0xC0) == 0x80) { + codepoint = (codepoint << 6) | (static_cast(*text++) & 0x3Fu); + taken++; } - // Longer sequences are skipped and shown as '?' - while ((static_cast(*text) & 0xC0) == 0x80) { - text++; + if (extra == 0 || taken < extra) { + while ((static_cast(*text) & 0xC0) == 0x80) { + text++; + } + return '?'; } - return '?'; + return codepoint; } } // namespace @@ -116,10 +125,65 @@ void BinFontText::close() { owned = false; } +namespace { + +/** Upper case Latin letters beyond Latin-1 (Latin Extended-A and -B pairs, Latin Extended Additional). */ +bool isUpperLatin(uint32_t c) { + if (c >= 0xC0 && c <= 0xDE) { + return c != 0xD7; + } + if ((c >= 0x100 && c <= 0x137) || (c >= 0x14A && c <= 0x177) || (c >= 0x1E00 && c <= 0x1EFF) || (c >= 0x218 && c <= 0x21B)) { + return (c & 1) == 0; + } + if ((c >= 0x139 && c <= 0x148) || (c >= 0x179 && c <= 0x17E)) { + return (c & 1) == 1; + } + return c == 0x178 || c == 0x181 || c == 0x186 || c == 0x189 || c == 0x18A || c == 0x191 || c == 0x198 || c == 0x1A0 || c == 0x1AF; +} + +} // namespace + +const char* BinFontText::withFallback(const char* text) { + // Only rewritten when a glyph is missing + bool missing = false; + for (const char* p = text; *p != '\0' && !missing;) { + BinFontGlyph glyph; + missing = !binfont_get_glyph(font, nextCodepoint(p), &glyph); + } + if (!missing) { + return text; + } + size_t used = 0; + const char* p = text; + while (*p != '\0' && used + 8 < sizeof(fallbackText)) { + const char* start = p; + const uint32_t codepoint = nextCodepoint(p); + BinFontGlyph glyph; + if (binfont_get_glyph(font, codepoint, &glyph)) { + while (start < p && used + 1 < sizeof(fallbackText)) { + fallbackText[used++] = *start++; + } + continue; + } + // The search's fold: diacritics off, lower case ("Š" -> "s"), then the case back + char one[8] = {}; + memcpy(one, start, std::min(static_cast(p - start), sizeof(one) - 1)); + char folded[16]; + foldLatin(one, folded, sizeof(folded)); + for (size_t i = 0; folded[i] != '\0' && used + 1 < sizeof(fallbackText); i++) { + const char c = folded[i]; + fallbackText[used++] = isUpperLatin(codepoint) && c >= 'a' && c <= 'z' ? static_cast(c - 'a' + 'A') : c; + } + } + fallbackText[used] = '\0'; + return fallbackText; +} + int BinFontText::textWidth(const char* text) { if (font == nullptr) { return 0; } + text = withFallback(text); // The same advances and kerning binfont_draw_text_rgb565() applies int32_t widthX16 = 0; uint32_t previous = 0; @@ -153,7 +217,7 @@ void BinFontText::drawText(Canvas& canvas, int x, int y, const char* text, uint1 return; } } - binfont_draw_text_rgb565(target, x, y + ascentPixels, font, text, color, PIXEL_BUFFER_CONVERSION_AVERAGE); + binfont_draw_text_rgb565(target, x, y + ascentPixels, font, withFallback(text), color, PIXEL_BUFFER_CONVERSION_AVERAGE); } bool BinFontText::glyph(uint32_t codepoint, LabelGlyph& out, bool withMask) { diff --git a/Apps/Maps/main/Source/app/BinFontText.h b/Apps/Maps/main/Source/app/BinFontText.h index 627a24f..290b483 100644 --- a/Apps/Maps/main/Source/app/BinFontText.h +++ b/Apps/Maps/main/Source/app/BinFontText.h @@ -42,6 +42,12 @@ class BinFontText final : public Font, public LabelFont { void drawText(Canvas& canvas, int x, int y, const char* text, uint16_t color) override; private: + /** + * The text to draw: letters the font has no glyph for (Tactility's UI font covers ASCII and Latin-1, so "Spisska" + * would lose its s with caron) become their base letter, keeping upper case. The text itself when nothing is missing. + */ + const char* withFallback(const char* text); + char fallbackText[256] {}; BinFont* font = nullptr; /** Borrowed fonts belong to Tactility and are not closed. */ bool owned = false; diff --git a/Apps/Maps/main/Source/app/Debug.cpp b/Apps/Maps/main/Source/app/Debug.cpp index b07cf8e..f7c5ab2 100644 --- a/Apps/Maps/main/Source/app/Debug.cpp +++ b/Apps/Maps/main/Source/app/Debug.cpp @@ -113,6 +113,12 @@ DebugScript::Step DebugScript::next(uint32_t now) { step.key = 10; } else if (strcmp(argument, "bs") == 0) { step.key = '\b'; + } else if (strcmp(argument, "space") == 0) { + step.key = ' '; + } else if (strcmp(argument, "up") == 0) { + step.key = 0x2191; + } else if (strcmp(argument, "down") == 0) { + step.key = 0x2193; } else { step.key = static_cast(argument[0]); } @@ -163,10 +169,32 @@ DebugScript::Step DebugScript::next(uint32_t now) { step.action = Action::WaitRoute; return step; } - if (strcmp(command, "replay") == 0 && sscanf(argument, "%31s", step.name) == 1) { + if (strcmp(command, "replay") == 0 && sscanf(argument, "%31s %f %f %f", step.name, &step.values[0], &step.values[1], &step.values[2]) >= 1) { + // replay [accuracy m] [noise m] [seed] step.action = Action::Replay; return step; } + if (strcmp(command, "searchindex") == 0) { + step.action = Action::SearchIndex; + step.zoom = strncmp(argument, "force", 5) == 0 ? 1 : 0; + // "searchindex force ": that region only + if (step.zoom != 0 && sscanf(argument, "force %31s", step.name) != 1) { + step.name[0] = '\0'; + } + return step; + } + if (strcmp(command, "waitsearchindex") == 0) { + step.action = Action::WaitSearchIndex; + return step; + } + if (strcmp(command, "filecheck") == 0 && sscanf(argument, "%31s", step.name) == 1) { + step.action = Action::FileCheck; + return step; + } + if (strcmp(command, "offroute") == 0 && sscanf(argument, "%31s %f", step.name, &step.values[0]) == 2) { + step.action = Action::OffRoute; + return step; + } float v[6] = {}; const int found = sscanf(argument, "%f %f %f %f %f %f", &v[0], &v[1], &v[2], &v[3], &v[4], &v[5]); struct TouchCommand { diff --git a/Apps/Maps/main/Source/app/Debug.h b/Apps/Maps/main/Source/app/Debug.h index fde62d7..159edfb 100644 --- a/Apps/Maps/main/Source/app/Debug.h +++ b/Apps/Maps/main/Source/app/Debug.h @@ -20,7 +20,13 @@ bool saveBmp(const char* path, const uint16_t* pixels, int width, int height); * Navigation (M5c): "routebench " runs the route search over the pairs of a file in the app's * folder (host/routing/export_pairs.py) and logs each; "navigate " asks for a route from the * position (or the screen centre) as "Navigate here" does; "waitroute" waits until no search runs; "replay " - * plays a GPX track from the app's folder as positions, one point per second ("replay stop" ends it). + * plays a GPX track from the app's folder as positions, one point per second ("replay stop" ends it); + * "replay [seed]" reports that accuracy in metres and moves the points by correlated random + * noise of that many metres (0.4.1). "offroute " sets a value of the off-route rule (OffRouteTuning). + * Search (M6): "searchindex [force [region]]" starts building the search indexes now ("force" also rebuilds current + * ones, of that region only when one is named), + * "waitsearchindex" waits until the build is over, "filecheck " logs the size and CRC-32 of a file under the + * app's folder (to compare an index built on the device with the PC's). */ class DebugScript { public: @@ -38,7 +44,11 @@ class DebugScript { RouteBench, Navigate, WaitRoute, - Replay + Replay, + OffRoute, + SearchIndex, + WaitSearchIndex, + FileCheck }; enum class TouchKind : uint8_t { Tap, Hold, Drag, Pinch, Twist }; diff --git a/Apps/Maps/main/Source/app/MapInput.cpp b/Apps/Maps/main/Source/app/MapInput.cpp index 73f6592..3aa65f3 100644 --- a/Apps/Maps/main/Source/app/MapInput.cpp +++ b/Apps/Maps/main/Source/app/MapInput.cpp @@ -60,6 +60,10 @@ void MapView::handleKey(uint32_t key) { handleCardButton(downloadCard.handleKey(key)); return; } + if (searchSheet.isOpen() && !menuOpen && screen == Screen::Map) { + handleSearchEvent(searchSheet.handleKey(key)); + return; + } if (sheet.isOpen() && !menuOpen && screen == Screen::Map) { handleSheetButton(sheet.handleKey(key)); return; @@ -211,6 +215,12 @@ void MapView::handleMapKey(uint32_t key) { exitRequested = true; } break; + case '/': + case 'f': + case 'F': + // Search (M6): "/" as in many apps, F for find on keyboards where "/" needs a modifier + openSearch(); + break; case 'p': case 'P': openNewWaypointAtPosition(); @@ -299,7 +309,7 @@ void MapView::pointerClick() { return; } cursorMovedAt = nowMs(); - if (prompt.isOpen() || dialog.isOpen() || menuOpen || sheet.isOpen() || downloadCard.isOpen() || screen != Screen::Map || + if (prompt.isOpen() || dialog.isOpen() || menuOpen || searchSheet.isOpen() || sheet.isOpen() || downloadCard.isOpen() || screen != Screen::Map || (pillRect.width > 0 && pillRect.contains(cursorX, cursorY))) { handleTap(cursorX, cursorY); return; @@ -329,7 +339,7 @@ void MapView::click(bool fromPointer) { placeOrderCount > 0 && !menuOpen && !downloadCard.isOpen()) { // Enter again: the next nearest place selectNextPlace(); - } else if (menuOpen || sheet.isOpen() || downloadCard.isOpen() || workCard.isOpen() || screen != Screen::Map || pickingStart) { + } else if (menuOpen || searchSheet.isOpen() || sheet.isOpen() || downloadCard.isOpen() || workCard.isOpen() || screen != Screen::Map || pickingStart) { handleKey(keys::ENTER); } else if (controls.isVisible() && controls.usesFocus()) { pressControl(controls.focused()); @@ -553,6 +563,11 @@ void MapView::handleTap(int x, int y) { gestures.consumeTap(); return; } + if (searchSheet.isOpen() && !menuOpen && screen == Screen::Map) { + handleSearchEvent(searchSheet.handleTap(x, y)); + gestures.consumeTap(); + return; + } if (dialog.isOpen()) { if (dialog.handleTap(x, y) == UiAction::Chosen) { confirmDialog(); @@ -655,6 +670,11 @@ void MapView::handleTap(int x, int y) { void MapView::handleLongPress(int x, int y) { dirty = true; + if (searchSheet.isOpen()) { + // Holding a result row opens it like a tap; nothing behind the sheet reacts + handleTap(x, y); + return; + } // Holding a row opens its actions, like a tap if (screen == Screen::Regions && !dialog.isOpen() && !menuOpen) { const int row = regionList.rowAt(x, y); @@ -691,7 +711,7 @@ void MapView::handleLongPress(int x, int y) { } void MapView::handleGesture(const Gesture& gesture) { - const bool onMap = !menuOpen && !dialog.isOpen() && !isFullScreen(); + const bool onMap = !menuOpen && !dialog.isOpen() && !isFullScreen() && !searchSheet.isOpen(); switch (gesture.type) { case GestureType::Tap: handleTap(gesture.x, gesture.y); @@ -713,7 +733,9 @@ void MapView::handleGesture(const Gesture& gesture) { handleLongPress(gesture.x, gesture.y); break; case GestureType::Drag: - if (onMap) { + if (searchSheet.isOpen() && !menuOpen && !dialog.isOpen()) { + handleSearchEvent(searchSheet.handleDrag(gesture.dy)); + } else if (onMap) { pan(-gesture.dx, -gesture.dy); } else if (!dialog.isOpen() && !prompt.isOpen() && std::abs(gesture.dy) >= std::abs(gesture.dx)) { // Lists and the About page scroll with the finger, a row at a time; the highlight stays where it is diff --git a/Apps/Maps/main/Source/app/MapNavigation.cpp b/Apps/Maps/main/Source/app/MapNavigation.cpp index a8a4e7f..5a9a4d7 100644 --- a/Apps/Maps/main/Source/app/MapNavigation.cpp +++ b/Apps/Maps/main/Source/app/MapNavigation.cpp @@ -19,8 +19,6 @@ constexpr uint16_t ROUTE_CASING = 0xFFFF; constexpr uint16_t ROUTE_PASSED = 0x9CF3; constexpr float ROUTE_WIDTH = 5.0f; constexpr float ROUTE_CASING_WIDTH = 8.0f; -/** At least this long between two recalculations after leaving the route. */ -constexpr uint32_t RECALCULATE_MS = 20000; constexpr uint32_t ARRIVED_BAR_MS = 8000; constexpr char CAR_NOTE[] = "Car routes follow road types only, without turn restrictions or access rules."; @@ -57,9 +55,33 @@ bool ReplaySource::load(const char* text, size_t size) { } cursor = close + 1; } + setNoise(0.0f, 0.0f, 0); return count > 1; } +void ReplaySource::setNoise(float accuracyM, float noiseM, uint32_t seed) { + accuracy = accuracyM > 0.0f ? accuracyM : 5.0f; + // Correlated noise as from a GPS: each second keeps 80 % of the last offset (AR(1)), with a standard deviation of noiseM + uint32_t state = seed != 0 ? seed : 12345u; + auto uniform = [&]() { + state = state * 1664525u + 1013904223u; + return (static_cast(state >> 8) + 0.5f) / 16777216.0f; + }; + auto gaussian = [&]() { return std::sqrt(-2.0f * std::log(uniform())) * std::cos(6.2831853f * uniform()); }; + const float keep = 0.8f; + const float fresh = std::sqrt(1.0f - keep * keep); + float north = noiseM > 0.0f ? noiseM * gaussian() : 0.0f; + float east = noiseM > 0.0f ? noiseM * gaussian() : 0.0f; + for (int i = 0; i < count; i++) { + if (noiseM > 0.0f && i > 0) { + north = keep * north + fresh * noiseM * gaussian(); + east = keep * east + fresh * noiseM * gaussian(); + } + noise[i][0] = north; + noise[i][1] = east; + } +} + PositionRead ReplaySource::read(PositionFix& fix) { if (count == 0) { return PositionRead::NoFix; @@ -72,10 +94,12 @@ PositionRead ReplaySource::read(PositionFix& fix) { } const int index = std::min(count - 1, static_cast((now - lastStep) / STEP_MS)); const int previous = std::max(0, index - 1); + lastIndex = index; fix = {}; - fix.latitude = points[index][0]; - fix.longitude = points[index][1]; - fix.accuracy = 5.0f; + const double cosLat = std::cos(points[index][0] * 3.14159265358979 / 180.0); + fix.latitude = points[index][0] + noise[index][0] / 111320.0; + fix.longitude = points[index][1] + noise[index][1] / (111320.0 * cosLat); + fix.accuracy = accuracy; // Course and speed from the step since the last point const double dy = (points[index][0] - points[previous][0]) * 111320.0; const double dx = (points[index][1] - points[previous][1]) * 111320.0 * std::cos(points[index][0] * 3.14159265358979 / 180.0); @@ -85,7 +109,7 @@ PositionRead ReplaySource::read(PositionFix& fix) { return PositionRead::Fix; } -void MapView::startReplay(const char* name) { +void MapView::startReplay(const char* name, float accuracyM, float noiseM, uint32_t seed) { if (strcmp(name, "stop") == 0) { replay.clear(); replaying = false; @@ -100,8 +124,10 @@ void MapView::startReplay(const char* name) { showToast("Replay: no track points in that file"); return; } + replay.setNoise(accuracyM, noiseM, seed); replaying = true; - logInfo("Replay: %d points from %s", replay.size(), path); + scriptLog("Replay: %d points from %s, accuracy %.0f m, noise %.0f m", replay.size(), path, static_cast(accuracyM), + static_cast(noiseM)); } // --------------------------------------------------------------------------------------------------------------- @@ -418,8 +444,8 @@ void MapView::updateNavigation(uint32_t now) { recalculating = false; dirty = true; } - // Also without a new position: standing still off the route still counts its 10 s - if (navigating && now - lastFollowAt >= 1000) { + // No fix for 2 s (a GPS that stopped sending): the last one counts again, so standing still off the route is noticed + if (navigating && static_cast(now - lastFollowAt) >= 2000) { followRoute(now); } if (navigating && arrivedAt != 0 && now - arrivedAt >= ARRIVED_BAR_MS) { @@ -430,6 +456,12 @@ void MapView::updateNavigation(uint32_t now) { } void MapView::followRoute(uint32_t now) { + // The position is read on a timer and on location events: a second call for the same fix (fixes come once a + // second) must not count as another fix for the off-route rule + // Signed: the loop passes the time it started, which can be before the follow a location read just did + if (lastFollowAt != 0 && static_cast(now - lastFollowAt) < 500) { + return; + } lastFollowAt = now; if (!navigating || !position.hasFix() || arrivedAt != 0) { return; @@ -442,7 +474,16 @@ void MapView::followRoute(uint32_t now) { stopNavigation(); return; } - const RouteFollower::Status status = follower.update(route, fix.latitude, fix.longitude, fix.accuracy, now); + // The 2 s timer shows the last fix again: it updates the bar but never counts as another fix + const bool newFix = fix.timeMs != lastFollowFixMs || fix.timeMs == 0; + lastFollowFixMs = fix.timeMs; + const RouteFollower::Status status = follower.update(route, fix.latitude, fix.longitude, fix.accuracy, now, fix.course, fix.speed, newFix); + // One line per fix while navigating, for tuning the off-route rule (0.4.1) + scriptLog("Follow: route %.1f m, acc %.1f m (fix %.1f), off %.1f m, heading %.0f, side %.1f m, %.1f m/s, away %u%s%s%s, replay %d", + status.distanceFromRouteM, status.accuracyM, fix.accuracy, status.offM, status.headingDifference, status.sideM, fix.speed, + static_cast(status.awayFixes), status.awayRule != OffRouteRule::None ? " by " : "", + status.awayRule != OffRouteRule::None ? offRouteRuleName(status.awayRule) : "", status.weakGps ? ", weak GPS" : "", + replaying ? replay.pointIndex() : -1); char instruction[NavigationBar::INSTRUCTION_LENGTH]; follower.instruction(route, instruction, sizeof(instruction)); navigator.unlock(); @@ -454,26 +495,29 @@ void MapView::followRoute(uint32_t now) { following = false; headingUp = false; arrivedAt = now; - logInfo("Navigation: arrived"); + scriptLog("Navigation: arrived"); break; case RouteFollower::State::OffRoute: - if (!recalculating && (lastRecalculation == 0 || now - lastRecalculation >= RECALCULATE_MS) && + // The first recalculation at once, then at least recalculateMs apart + if (!recalculating && (lastRecalculation == 0 || now - lastRecalculation >= offRouteTuning().recalculateMs) && navigator.state() == Navigator::State::Idle) { lastRecalculation = now; recalculating = navigator.request(fix.latitude, fix.longitude, goalLat, goalLon, routeProfile); routeRequested = recalculating; - logInfo("Navigation: off route by %.0f m, recalculating", status.distanceFromRouteM); + scriptLog("Navigation: off route by %.0f m (acc %.1f m, off %.1f m, rule %s), recalculating", status.distanceFromRouteM, + status.accuracyM, status.offM, offRouteRuleName(status.firedRule)); } navBar.setMessage("Off route \xE2\x80\x94 recalculating"); break; default: if (!recalculating) { - navBar.set(status.nextTurn, instruction, remaining); + // Weak GPS: the route is followed by projection, off route is not checked + navBar.set(status.nextTurn, instruction, status.weakGps ? "Weak GPS" : remaining); } break; } // Redrawn when the bar or the state changed, not every second (e-paper) - const uint32_t signature = (static_cast(status.state) << 24) ^ static_cast(status.toTurnM) ^ + const uint32_t signature = (static_cast(status.state) << 24) ^ (status.weakGps ? 0x800000u : 0u) ^ static_cast(status.toTurnM) ^ (static_cast(status.remainingM) << 8) ^ static_cast(status.progressM * 10.0f); if (signature != lastFollowSignature) { lastFollowSignature = signature; diff --git a/Apps/Maps/main/Source/app/MapScreens.cpp b/Apps/Maps/main/Source/app/MapScreens.cpp index 03a22fa..a70e0a5 100644 --- a/Apps/Maps/main/Source/app/MapScreens.cpp +++ b/Apps/Maps/main/Source/app/MapScreens.cpp @@ -102,6 +102,9 @@ void MapView::closeScreen() { void MapView::goBack() { if (menuOpen) { handlePopupAction(UiAction::Back); + } else if (searchSheet.isOpen() && screen == Screen::Map) { + // One level back in the search sheet (a category list to the main list), else closes it + handleSearchEvent(searchSheet.handleKey(keys::ESCAPE)); } else if (screen == Screen::About) { closeScreen(); } else if (activeList() != nullptr) { @@ -154,6 +157,8 @@ void MapView::openMenu() { menu.add("Clear route"); popupActions[popupActionCount++] = MENU_ROUTE_CLEAR; } + menu.add("Search"); + popupActions[popupActionCount++] = static_cast(MenuItem::Search); int item = 0; for (const char* label : MENU_LABELS) { popupActions[popupActionCount++] = item++; @@ -218,6 +223,10 @@ void MapView::chooseMenu(MenuItem item) { case MenuItem::About: openAbout(); break; + case MenuItem::Search: + returnToMenu = false; + openSearch(); + break; case MenuItem::Exit: // The only way out by trackball or touch, LVGL's swipe back does not run while the map has the screen exitRequested = true; diff --git a/Apps/Maps/main/Source/app/MapSearch.cpp b/Apps/Maps/main/Source/app/MapSearch.cpp new file mode 100644 index 0000000..b22e606 --- /dev/null +++ b/Apps/Maps/main/Source/app/MapSearch.cpp @@ -0,0 +1,755 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Offline search on the map (M6): the search sheet, the region indexes on the card, opening a result. +#include "MapView.h" + +#include "../platform/Platform.h" +#include "../route/RouteSearch.h" + +#include + +#include +#include +#include +#include +#include +#include + +namespace maps { + +namespace { + +/** Results fetched at most (the first page shows at once, the rest while scrolling). */ +constexpr int SEARCH_MAX_RESULTS = SearchSheet::MAX_ROWS; +constexpr char RECENT_FILE[] = "recent.txt"; + +struct RangeReadJob { + FILE** file; + const char* path; + uint32_t offset; + void* out; + uint32_t size; +}; + +bool rangeReadJob(IoTask& io, void* context) { + (void)io; + auto* job = static_cast(context); + if (*job->file == nullptr) { + *job->file = fopen(job->path, "rb"); + if (*job->file == nullptr) { + return false; + } + } + if (fseek(*job->file, static_cast(job->offset), SEEK_SET) != 0) { + return false; + } + return fread(job->out, 1, job->size, *job->file) == job->size; +} + +bool closeFileJob(IoTask& io, void* context) { + (void)io; + auto** file = static_cast(context); + if (*file != nullptr) { + fclose(*file); + *file = nullptr; + } + return true; +} + +void searchCoordinates(double lat, double lon, char* out, size_t size) { + snprintf(out, size, "%.5f %c %.5f %c", std::fabs(lat), lat >= 0.0 ? 'N' : 'S', std::fabs(lon), lon >= 0.0 ? 'E' : 'W'); +} + +/** Runs a callable on the search task (its stack is large enough for the engine), waiting for it. */ +template +void onSearchTask(SearchTask& task, Function& function) { + task.run([](void* context) { (*static_cast(context))(); }, &function); +} + +uint64_t searchClockUs() { + return nowUs(); +} + +const char* plainDistance(uint32_t metres, char* out, size_t size) { + formatDistance(static_cast(metres), out, size); + return out; +} + +} // namespace + +/** + * A region's index file read through the I/O task. The file stays open between reads, but only one index file is + * open at a time: the card's FAT mount allows four open files for the whole system (the index build keeps two). + */ +class MapView::CardSearchFile final : public SearchFile { +public: + CardSearchFile(IoTask& io, const char* path) : io(io) { snprintf(this->path, sizeof(this->path), "%s", path); } + ~CardSearchFile() override { + close(); + if (openOne == this) { + openOne = nullptr; + } + } + bool read(uint32_t offset, void* out, uint32_t size) override { + if (openOne != this) { + if (openOne != nullptr) { + openOne->close(); + } + openOne = this; + } + RangeReadJob job { &file, path, offset, out, size }; + reads++; + return io.run(IoPriority::Visible, rangeReadJob, &job); + } + void close() { + if (file != nullptr) { + io.run(IoPriority::Visible, closeFileJob, &file); + } + } + uint32_t reads = 0; + +private: + static CardSearchFile* openOne; + IoTask& io; + char path[PATH_MAX_LENGTH] {}; + FILE* file = nullptr; +}; + +MapView::CardSearchFile* MapView::CardSearchFile::openOne = nullptr; + +namespace { + +/** Block files of a big region (the .idx files in maps//search). */ +struct BlockFiles { + char names[64][16]; + int count = 0; +}; + +bool collectBlocks(const char* name, bool isDirectory, void* context) { + auto* list = static_cast(context); + const size_t length = strlen(name); + if (!isDirectory && length > 4 && length < sizeof(list->names[0]) && strcmp(name + length - 4, ".idx") == 0 && list->count < 64) { + snprintf(list->names[list->count++], sizeof(list->names[0]), "%s", name); + } + return true; +} + +} // namespace + +bool MapView::loadSearchIndexes() { + if (searchEngine == nullptr) { + void* memory = largeAlloc(sizeof(SearchEngine)); + void* results = largeAlloc(sizeof(SearchResult) * SEARCH_MAX_RESULTS); + void* recent = largeAlloc(sizeof(RecentSearches)); + if (memory == nullptr || results == nullptr || recent == nullptr) { + largeFree(memory); + largeFree(results); + largeFree(recent); + return false; + } + searchEngine = new (memory) SearchEngine(); + searchEngine->setClock(searchClockUs); + searchResults = static_cast(results); + recentSearches = new (recent) RecentSearches(); + loadRecentSearches(); + } + // The indexes of the regions on the card, the one under the map centre first + for (int i = 0; i < searchFileCount; i++) { + searchFiles[i]->~CardSearchFile(); + largeFree(searchFiles[i]); + searchFiles[i] = nullptr; + } + searchFileCount = 0; + auto removeAll = [&]() { searchEngine->removeAll(); }; + onSearchTask(searchTask, removeAll); + const double lat = worldYToLat(centerY, zoom); + const double lon = worldXToLon(centerX, zoom); + const TileKey centre = tileAt(lat, lon, 14); + int order[TileStore::MAX_REGIONS]; + int orderCount = 0; + for (int pass = 0; pass < 2; pass++) { + for (int i = 0; i < store.regionCount(); i++) { + const bool here = store.mayHave(i, centre); + if ((pass == 0) == here) { + order[orderCount++] = i; + } + } + } + auto addFile = [&](const char* path, const char* region) { + void* memory = largeAlloc(sizeof(CardSearchFile)); + if (memory == nullptr) { + return; + } + auto* file = new (memory) CardSearchFile(io, path); + // The profile and the x/y rule come from the index header (the engine reads it) + int slot = -1; + auto add = [&]() { slot = searchEngine->addIndex(file, region, Translit::Russian, true); }; + onSearchTask(searchTask, add); + if (slot < 0) { + logError("Search: %s does not open", path); + file->~CardSearchFile(); + largeFree(memory); + return; + } + searchFiles[searchFileCount++] = file; + }; + for (int k = 0; k < orderCount && searchFileCount < SEARCH_MAX_INDEXES; k++) { + const char* region = store.regionName(order[k]); + char path[PATH_MAX_LENGTH]; + snprintf(path, sizeof(path), "%s/maps/%s/search.idx", rootPath, region); + bool exists = false; + auto blocks = new BlockFiles(); + char folder[PATH_MAX_LENGTH]; + snprintf(folder, sizeof(folder), "%s/maps/%s/search", rootPath, region); + // A big region has blocks instead: their origins (bytes 8..15 of the header) say which are nearest the centre + int64_t distance[64] = {}; + auto check = [&]() { + exists = fileExists(path); + if (!exists && listDirectory(folder, collectBlocks, blocks)) { + for (int i = 0; i < blocks->count; i++) { + char blockPath[PATH_MAX_LENGTH]; + snprintf(blockPath, sizeof(blockPath), "%s/%s", folder, blocks->names[i]); + uint8_t header[16] = {}; + FILE* file = fopen(blockPath, "rb"); + distance[i] = INT64_MAX; + if (file != nullptr && fread(header, 1, sizeof(header), file) == sizeof(header)) { + const auto ox = static_cast(header[8] | header[9] << 8 | header[10] << 16 | static_cast(header[11]) << 24); + const auto oy = static_cast(header[12] | header[13] << 8 | header[14] << 16 | static_cast(header[15]) << 24); + // The middle of a block of 15 x 15 tiles + const int64_t dx = static_cast(ox) + 15 * 2048 - (static_cast(centre.x) * 4096 + 2048); + const int64_t dy = static_cast(oy) + 15 * 2048 - (static_cast(centre.y) * 4096 + 2048); + distance[i] = dx * dx + dy * dy; + } + if (file != nullptr) { + fclose(file); + } + } + } + return true; + }; + runOnCard(io, check); + if (exists) { + addFile(path, region); + } else { + int index[64]; + for (int i = 0; i < blocks->count; i++) { + index[i] = i; + } + std::sort(index, index + blocks->count, [&](int a, int b) { return distance[a] < distance[b]; }); + for (int i = 0; i < blocks->count && searchFileCount < SEARCH_MAX_INDEXES; i++) { + char blockPath[PATH_MAX_LENGTH]; + snprintf(blockPath, sizeof(blockPath), "%s/%s", folder, blocks->names[index[i]]); + addFile(blockPath, region); + } + } + delete blocks; + } + // The name area and the key pages of the nearest index in PSRAM, when there is room (study: 0.76 MB for Bratislava) + if (searchFileCount > 0 && heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM) > 1200 * 1024) { + const uint32_t started = nowMs(); + bool loaded = false; + auto load = [&]() { loaded = searchEngine->loadResident(0, true, true); }; + onSearchTask(searchTask, load); + logInfo("Search: name area and key pages of %s %s in %u ms", searchEngine->regionName(0), loaded ? "loaded" : "not loaded", + static_cast(nowMs() - started)); + } + logInfo("Search: %d region indexes", searchFileCount); + return searchFileCount > 0; +} + +void MapView::releaseSearch() { + if (searchEngine != nullptr) { + auto drop = [&]() { + searchEngine->dropResident(); + searchEngine->removeAll(); + }; + onSearchTask(searchTask, drop); + } + for (int i = 0; i < searchFileCount; i++) { + searchFiles[i]->~CardSearchFile(); + largeFree(searchFiles[i]); + searchFiles[i] = nullptr; + } + searchFileCount = 0; +} + +SearchOrigin MapView::searchOrigin(bool& fromPosition) const { + fromPosition = position.hasFix() && position.state() == PositionState::Precise; + const double lat = fromPosition ? position.fix().latitude : worldYToLat(centerY, zoom); + const double lon = fromPosition ? position.fix().longitude : worldXToLon(centerX, zoom); + const RoutePoint point = routePointFrom(lat, lon); + return { point.x, point.y }; +} + +void MapView::openSearch() { + closeMenu(); + sheet.close(); + controls.hide(); + if (!loadSearchIndexes()) { + showToast("No search index yet: it is made in idle time for downloaded maps"); + return; + } + searchSheet.open(); + searchSheet.setTypingKeyboard(input.hasKeyboard()); + int count = 0; + const SearchCategory* categories = SearchEngine::categories(&count); + const char* names[SearchSheet::MAX_CATEGORIES]; + count = std::min(count, SearchSheet::MAX_CATEGORIES); + for (int i = 0; i < count; i++) { + names[i] = categories[i].name; + } + searchSheet.setCategories(names, count); + showRecentRows(); + layoutSearch(); + dirty = true; +} + +void MapView::closeSearch() { + searchSheet.close(); + releaseSearch(); + syncTouchKeyboard(); + dirty = true; +} + +void MapView::layoutSearch() { + syncTouchKeyboard(); + if (touchKeyboard.isOpen()) { + searchSheet.layout(width, height - TouchKeyboard::heightFor(width), sheetFont(), true); + } else { + searchSheet.layout(width, height, sheetFont()); + } +} + +void MapView::showRecentRows() { + // Static: the map task's stack is small + static SearchSheet::Row rows[SearchSheet::MAX_RECENT]; + const int count = std::min(recentSearches != nullptr ? recentSearches->count() : 0, SearchSheet::MAX_RECENT); + bool fromPosition; + const SearchOrigin origin = searchOrigin(fromPosition); + static char distances[SearchSheet::MAX_RECENT][16]; + for (int i = 0; i < count; i++) { + const RecentSearches::Entry& entry = recentSearches->at(i); + const double metres = std::sqrt(static_cast(entry.x - origin.x) * (entry.x - origin.x) + static_cast(entry.y - origin.y) * (entry.y - origin.y)) * + routeMetresPerUnit(origin.y); + rows[i] = { entry.name, entry.type, nullptr, plainDistance(static_cast(metres), distances[i], sizeof(distances[i])), + static_cast(std::atan2(static_cast(entry.x - origin.x), static_cast(origin.y - entry.y)) * 180.0 / 3.14159265) - + bearing }; + } + searchSheet.setRecent(rows, count); + searchSheet.setHeader(count > 0 ? "Recent" : ""); +} + +void MapView::fillSearchRows(int from, int count, bool more) { + // Static: the map task's stack is small + static SearchSheet::Row rows[SearchSheet::MAX_ROWS]; + static char distances[SearchSheet::MAX_ROWS][16]; + static char notes[SearchSheet::MAX_ROWS][80]; + static char types[SearchSheet::MAX_ROWS][SEARCH_LINE_BYTES]; + for (int i = 0; i < count; i++) { + const SearchResult& r = searchResults[from + i]; + searchTypeLine(r.type, r.place, types[i], sizeof(types[i])); + notes[i][0] = '\0'; + if (r.lowConfidence) { + snprintf(notes[i], sizeof(notes[i]), "Street estimated from the map"); + } else if (r.villageEstimated) { + snprintf(notes[i], sizeof(notes[i]), "Village estimated from the map"); + } + if (r.also[0][0] != '\0') { + const size_t used = strlen(notes[i]); + snprintf(notes[i] + used, sizeof(notes[i]) - used, "%sAlso: %s%s%s", used > 0 ? " \xC2\xB7 " : "", r.also[0], + r.also[1][0] != '\0' ? ", " : "", r.also[1]); + } + rows[i] = { r.name, types[i], notes[i][0] != '\0' ? notes[i] : nullptr, plainDistance(r.distanceM, distances[i], sizeof(distances[i])), + static_cast(r.bearing) - bearing }; + } + if (from == 0) { + searchSheet.setRows(rows, count, more); + } else { + searchSheet.appendRows(rows, count, more); + } +} + +void MapView::runSearch() { + const char* text = searchSheet.text(); + // Results from 3 characters (a typed letter is 1 to 2 bytes) + int characters = 0; + for (const char* p = text; *p != '\0'; p++) { + characters += (static_cast(*p) & 0xC0) != 0x80 ? 1 : 0; + } + searchResultCount = 0; + if (characters < 3) { + searchSheet.setRows(nullptr, 0, false); + showRecentRows(); + return; + } + bool fromPosition; + const SearchOrigin origin = searchOrigin(fromPosition); + static SearchStats stats; + stats = {}; + uint32_t readsBefore = 0; + for (int i = 0; i < searchFileCount; i++) { + readsBefore += searchFiles[i]->reads; + } + const uint32_t started = nowMs(); + int count = 0; + auto run = [&]() { count = searchEngine->search(text, origin, searchResults, SEARCH_FIRST_PAGE, &stats); }; + onSearchTask(searchTask, run); + const uint32_t elapsed = nowMs() - started; + uint32_t reads = 0; + for (int i = 0; i < searchFileCount; i++) { + reads += searchFiles[i]->reads; + } + searchResultCount = std::max(0, count); + const bool more = searchEngine->resultCount() > searchResultCount; + fillSearchRows(0, searchResultCount, more); + if (searchResultCount == 0) { + searchSheet.setHeader("No results"); + } else if (stats.header[0] != '\0') { + searchSheet.setHeader(stats.header); + } else { + searchSheet.setHeader(fromPosition ? "" : "Distances from the map centre"); + } + scriptLog("Search '%s': %d of %d results in %u ms, %u card reads, engine %u us, search stack left %u", text, searchResultCount, + searchEngine->resultCount(), static_cast(elapsed), static_cast(reads - readsBefore), static_cast(stats.elapsedUs), + static_cast(searchTask.stackLeft())); +} + +void MapView::scriptLog(const char* format, ...) { + static char line[256]; + va_list arguments; + va_start(arguments, format); + vsnprintf(line, sizeof(line), format, arguments); + va_end(arguments); + logInfo("%s", line); + if (!script.isActive()) { + return; + } + // Debug scripts also get the lines in scriptlog.txt with the time: measurements without a serial log + char path[PATH_MAX_LENGTH]; + snprintf(path, sizeof(path), "%s/scriptlog.txt", rootPath); + const uint32_t now = nowMs(); + auto append = [&]() { + FILE* file = fopen(path, "a"); + if (file == nullptr) { + return false; + } + fprintf(file, "%u %s\n", static_cast(now), line); + fclose(file); + return true; + }; + runOnCard(io, append); +} + +void MapView::moreSearchResults() { + if (searchEngine == nullptr || searchResultCount >= SEARCH_MAX_RESULTS) { + return; + } + int count = 0; + auto run = [&]() { count = searchEngine->more(searchResults + searchResultCount, std::min(SEARCH_FIRST_PAGE, SEARCH_MAX_RESULTS - searchResultCount)); }; + onSearchTask(searchTask, run); + if (count <= 0) { + searchSheet.appendRows(nullptr, 0, false); + return; + } + const int from = searchResultCount; + searchResultCount += count; + fillSearchRows(from, count, searchEngine->resultCount() > searchResultCount && searchResultCount < SEARCH_MAX_RESULTS); +} + +void MapView::searchNearby(int category) { + bool fromPosition; + const SearchOrigin origin = searchOrigin(fromPosition); + static SearchStats stats; + stats = {}; + const uint32_t started = nowMs(); + int count = 0; + auto run = [&]() { count = searchEngine->nearby(category, origin, searchResults, std::min(SEARCH_NEARBY, SEARCH_MAX_RESULTS), &stats); }; + onSearchTask(searchTask, run); + searchResultCount = std::max(0, count); + fillSearchRows(0, searchResultCount, false); + searchSheet.showResults(); + searchSheet.setHeader(searchResultCount == 0 ? "Nothing of that kind nearby" : (fromPosition ? "" : "Distances from the map centre")); + logInfo("Search nearby %d: %d results in %u ms", category, searchResultCount, static_cast(nowMs() - started)); +} + +void MapView::openSearchResult(const char* name, const char* type, int32_t x, int32_t y, const int32_t* box, const char* note) { + double lat; + double lon; + routePointTo({ x, y }, lat, lon); + int targetZoom = 17; + if (box != nullptr) { + // A street: zoomed so its box fits the screen + const double metres = std::max(box[2] - box[0], box[3] - box[1]) * routeMetresPerUnit(y); + const double metresPerPixel = metres / (std::min(width, height) * 0.8); + targetZoom = static_cast(std::floor(std::log(156543.03 * std::cos(lat * 3.14159265358979 / 180.0) / std::max(0.3, metresPerPixel)) / std::log(2.0))); + targetZoom = std::clamp(targetZoom, 12, 18); + } + closeSearch(); + following = false; + centerOn(lat, lon, targetZoom); + sheetLat = lat; + sheetLon = lon; + selectedWaypoint = -1; + char coordinates[48]; + searchCoordinates(lat, lon, coordinates, sizeof(coordinates)); + char distance[96]; + describeDistance(lat, lon, distance, sizeof(distance)); + if (note != nullptr && note[0] != '\0') { + const size_t used = strlen(distance); + snprintf(distance + used, sizeof(distance) - used, "%s%s", used > 0 ? " \xC2\xB7 " : "", note); + } + char title[WaypointSheet::NAME_LENGTH]; + snprintf(title, sizeof(title), "%s", name); + sheet.openPlace(title, type, coordinates, distance); + sheet.layout(width, height, sheetFont()); + // The place shows above the sheet, not under it + pan(0.0, sheet.bounds().height / 2.0); + searchPinLat = lat; + searchPinLon = lon; + searchPinShown = true; + dirty = true; +} + +void MapView::chooseSearchRow(int index, bool recent) { + if (recent) { + if (recentSearches == nullptr || index < 0 || index >= recentSearches->count()) { + return; + } + const RecentSearches::Entry entry = recentSearches->at(index); + openSearchResult(entry.name, entry.type, entry.x, entry.y, nullptr, nullptr); + return; + } + if (index < 0 || index >= searchResultCount) { + return; + } + const SearchResult& result = searchResults[index]; + recentSearches->add(result); + saveRecentSearches(); + const int32_t box[4] = { result.minX, result.minY, result.maxX, result.maxY }; + const char* note = result.lowConfidence ? "Street estimated from the map" : (result.villageEstimated ? "Village estimated from the map" : nullptr); + char type[SEARCH_LINE_BYTES]; + searchTypeLine(result.type, result.place, type, sizeof(type)); + openSearchResult(result.name, type, result.x, result.y, result.hasBox ? box : nullptr, note); +} + +void MapView::handleSearchEvent(SearchSheet::Event event) { + dirty = true; + switch (event.type) { + case SearchSheet::EventType::TextChanged: + runSearch(); + break; + case SearchSheet::EventType::OpenRow: + chooseSearchRow(event.index, false); + return; + case SearchSheet::EventType::OpenRecent: + chooseSearchRow(event.index, true); + return; + case SearchSheet::EventType::OpenCategory: + searchNearby(event.index); + break; + case SearchSheet::EventType::WantMore: + moreSearchResults(); + break; + case SearchSheet::EventType::Close: + closeSearch(); + return; + default: + break; + } + if (searchSheet.isOpen() && searchSheet.inCategories()) { + searchSheet.setHeader(""); + } + if (searchSheet.isOpen()) { + layoutSearch(); + } +} + +// --------------------------------------------------------------------------------------------------------------- +// Recent searches: a small text file in the app's folder + +namespace { + +struct RecentWriter { + char* buffer; + size_t size; + size_t used; +}; + +bool recentWrite(void* context, const void* data, size_t size) { + auto* writer = static_cast(context); + if (writer->used + size > writer->size) { + return false; + } + memcpy(writer->buffer + writer->used, data, size); + writer->used += size; + return true; +} + +struct RecentReader { + const uint8_t* data; + size_t size; + size_t at; +}; + +size_t recentRead(void* context, void* out, size_t capacity) { + auto* reader = static_cast(context); + const size_t count = std::min(capacity, reader->size - reader->at); + memcpy(out, reader->data + reader->at, count); + reader->at += count; + return count; +} + +} // namespace + +void MapView::loadRecentSearches() { + char path[PATH_MAX_LENGTH]; + snprintf(path, sizeof(path), "%s/%s", rootPath, RECENT_FILE); + FileBuffer file; + auto read = [&]() { return io.readFile(path, file); }; + if (!runOnCard(io, read)) { + return; + } + RecentReader reader { file.bytes(), file.size(), 0 }; + recentSearches->load(recentRead, &reader); +} + +void MapView::saveRecentSearches() { + static char text[4096]; + RecentWriter writer { text, sizeof(text), 0 }; + if (!recentSearches->save(recentWrite, &writer)) { + return; + } + char path[PATH_MAX_LENGTH]; + char temporary[PATH_MAX_LENGTH + 8]; + snprintf(path, sizeof(path), "%s/%s", rootPath, RECENT_FILE); + snprintf(temporary, sizeof(temporary), "%s.tmp", path); + auto write = [&]() { return io.writeFile(path, temporary, reinterpret_cast(text), writer.used, false); }; + if (!runOnCard(io, write)) { + logError("Search: recent searches not saved"); + } +} + +// ------------------------------------------------------------------------------------------------ index build (M6) + +namespace { + +/** The map must be left alone this long before an index build starts. */ +constexpr uint32_t SEARCH_INDEX_IDLE_MS = 20000; +/** After a failed build, the next try waits this long. */ +constexpr uint32_t SEARCH_INDEX_RETRY_MS = 10 * 60 * 1000; +/** PSRAM the build needs free when it starts (its peak is about 0.8 MB for Bratislava on the PC). */ +constexpr size_t SEARCH_INDEX_MIN_FREE = 1100 * 1024; +constexpr size_t SEARCH_INDEX_MIN_BLOCK = 256 * 1024; + +uint32_t crc32Update(uint32_t crc, const uint8_t* data, size_t size) { + crc = ~crc; + for (size_t i = 0; i < size; i++) { + crc ^= data[i]; + for (int k = 0; k < 8; k++) { + crc = (crc >> 1) ^ (0xEDB88320u & (0u - (crc & 1u))); + } + } + return ~crc; +} + +} // namespace + +void MapView::startSearchIndex(bool force, const char* only) { + if (searchIndexJob.state() == SearchIndexJob::State::Busy) { + return; + } + searchIndexJob.acknowledge(); + // The region under the map centre first + const TileKey centre = tileAt(worldYToLat(centerY, zoom), worldXToLon(centerX, zoom), 14); + const char* first = nullptr; + for (int i = 0; i < store.regionCount() && first == nullptr; i++) { + if (store.mayHave(i, centre)) { + first = store.regionName(i); + } + } + if (searchIndexJob.start(io, rootPath, first, force, only)) { + scriptLog("Search index build started%s (PSRAM %u KB free, largest block %u KB)", force ? " (force)" : "", + static_cast(heap_caps_get_free_size(MALLOC_CAP_SPIRAM) / 1024), + static_cast(heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM) / 1024)); + } + searchIndexDue = false; +} + +void MapView::pollSearchIndex(uint32_t now) { + // A download or an update that ended may have brought new tiles + const Downloader::State downloadState = lastStatus.state; + const bool downloading = downloadState == Downloader::State::Downloading || downloadState == Downloader::State::Updating; + const bool wasDownloading = + searchIndexDownloadState == Downloader::State::Downloading || searchIndexDownloadState == Downloader::State::Updating; + if (wasDownloading && !downloading) { + searchIndexDue = true; + } + searchIndexDownloadState = downloadState; + // Reports for the log (measurements) + while (const char* line = searchIndexJob.nextReport()) { + scriptLog("%s", line); + } + switch (searchIndexJob.state()) { + case SearchIndexJob::State::Busy: + // Downloads come first: the build waits with its files closed + searchIndexJob.pause(downloading); + return; + case SearchIndexJob::State::Done: + case SearchIndexJob::State::Failed: { + const bool failed = searchIndexJob.state() == SearchIndexJob::State::Failed; + scriptLog("Search index build %s, lowest free PSRAM %u KB", failed ? "failed" : "done", + static_cast(searchIndexJob.lowestFreePsram() / 1024)); + if (searchIndexJob.builtAny() && !searchSheet.isOpen()) { + // The next search opens the new indexes + releaseSearch(); + } + searchIndexJob.acknowledge(); + if (failed) { + searchIndexRetryAt = now + SEARCH_INDEX_RETRY_MS; + searchIndexDue = true; + } + return; + } + case SearchIndexJob::State::Idle: + break; + } + if (!searchIndexDue || script.isActive() || static_cast(now - searchIndexRetryAt) < 0 || now - lastInputAt < SEARCH_INDEX_IDLE_MS || + downloading || navigating || worker.isPreparing()) { + return; + } + if (heap_caps_get_free_size(MALLOC_CAP_SPIRAM) < SEARCH_INDEX_MIN_FREE || + heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM) < SEARCH_INDEX_MIN_BLOCK) { + return; + } + startSearchIndex(false); +} + +void MapView::checkFile(const char* name) { + char path[PATH_MAX_LENGTH]; + snprintf(path, sizeof(path), "%s/%s", rootPath, name); + uint32_t crc = 0; + uint32_t size = 0; + bool ok = false; + auto job = [&]() { + FILE* file = fopen(path, "rb"); + if (file == nullptr) { + return false; + } + static uint8_t chunk[4096]; + size_t n = 0; + while ((n = fread(chunk, 1, sizeof(chunk), file)) > 0) { + crc = crc32Update(crc, chunk, n); + size += static_cast(n); + } + fclose(file); + ok = true; + return true; + }; + runOnCard(io, job); + if (ok) { + scriptLog("File %s: %u bytes, CRC-32 %08x", name, static_cast(size), static_cast(crc)); + } else { + scriptLog("File %s: cannot read", name); + } +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/app/MapView.cpp b/Apps/Maps/main/Source/app/MapView.cpp index 14914dc..926c8ab 100644 --- a/Apps/Maps/main/Source/app/MapView.cpp +++ b/Apps/Maps/main/Source/app/MapView.cpp @@ -279,6 +279,21 @@ bool MapView::begin(Device* display, const char* root, const char* dataPath) { } void MapView::end() { + // The index build stops first (its scratch files are removed), while the I/O task still runs + searchIndexJob.stop(); + searchIndexJob.acknowledge(); + releaseSearch(); + if (searchEngine != nullptr) { + searchEngine->~SearchEngine(); + largeFree(searchEngine); + searchEngine = nullptr; + largeFree(searchResults); + searchResults = nullptr; + recentSearches->~RecentSearches(); + largeFree(recentSearches); + recentSearches = nullptr; + } + searchTask.stop(); if (viewReady) { saveView(); viewReady = false; @@ -928,10 +943,11 @@ void MapView::pollDownloader(uint32_t now) { } lastStatusPoll = now; const Downloader::Status status = downloader.status(); - // Tile cache slots given to a large download come back once nothing downloads + // Tile cache slots given to a large download come back once nothing downloads, and not while a search index is + // built (its tables grow; it was started with the memory free that it had) if (status.state != Downloader::State::Connecting && status.state != Downloader::State::Downloading && status.state != Downloader::State::Updating && !downloader.isBrowsing() && worker.cacheShrunk() && !worker.isPreparing() && - navigator.state() != Navigator::State::Searching) { + navigator.state() != Navigator::State::Searching && searchIndexJob.state() != SearchIndexJob::State::Busy) { worker.restoreMemory(); } downloader.browseProgress(browseDone, browseTotal); @@ -1239,6 +1255,7 @@ void MapView::updateWork(uint32_t now) { set(WorkKind::IdleUpdate, lastStatus.state == Downloader::State::Updating, static_cast(lastStatus.done), static_cast(lastStatus.total)); set(WorkKind::GpxImport, gpxImport.state() == GpxImport::State::Busy, 0, 0); + set(WorkKind::AddressIndex, searchIndexJob.state() == SearchIndexJob::State::Busy, searchIndexJob.done(), searchIndexJob.total()); const int moved = downloader.movedTiles(); set(WorkKind::MoveBrowsed, lastStatus.state == Downloader::State::Downloading && moved > 0, moved, 0); if (work.generation() != before) { @@ -1260,7 +1277,7 @@ bool MapView::wantsTouchKeyboard() const { if (input.hasKeyboard() || !input.hasTouch()) { return false; } - return prompt.isOpen() || (sheet.isOpen() && sheet.mode() == WaypointSheet::Mode::New && sheet.isEditable()); + return prompt.isOpen() || searchSheet.isOpen() || (sheet.isOpen() && sheet.mode() == WaypointSheet::Mode::New && sheet.isEditable()); } void MapView::syncTouchKeyboard() { @@ -1282,6 +1299,9 @@ void MapView::syncTouchKeyboard() { if (sheet.isOpen()) { sheet.layout(width, above, sheetFont(), true); } + if (searchSheet.isOpen()) { + searchSheet.layout(width, above, sheetFont(), true); + } } bool MapView::isSmoothView() const { @@ -1438,6 +1458,16 @@ void MapView::render() { canvas.line(x - 6.0f, y, x + 6.0f, y, 1.5f, theme.text); canvas.line(x, y - 6.0f, x, y + 6.0f, 1.5f, theme.text); } + if (sheet.isOpen() && searchPinShown) { + // The place of the chosen search result + double sx; + double sy; + worldToScreenOffset(lonToWorldX(searchPinLon, zoom) - centerX, latToWorldY(searchPinLat, zoom) - centerY, sx, sy); + const auto px = static_cast(sx + width / 2.0); + const auto py = static_cast(sy + height / 2.0); + canvas.fillCircle(px, py, 7.0f, theme.panel); + canvas.fillCircle(px, py, 5.0f, theme.danger); + } if (sheet.isOpen()) { sheet.draw(canvas, sheetFont(), theme); } @@ -1459,6 +1489,10 @@ void MapView::render() { if (workDotShown) { drawWorkDot(); } + if (screen == Screen::Map && searchSheet.isOpen()) { + // Over the navigation bar and the status: the search's field and rows are where those are + searchSheet.draw(canvas, sheetFont(), theme); + } } if (menuOpen) { menu.draw(canvas, overlay.font(), theme); @@ -1705,13 +1739,15 @@ void MapView::run() { downloader.setUpdatesAllowed(true); } } - if (now - lastLocationCheck >= LOCATION_CHECK_MS) { + // A replay has no fix event to wake the loop: it is read every second, as a GPS sends fixes + if (now - lastLocationCheck >= (replaying ? ReplaySource::STEP_MS : LOCATION_CHECK_MS)) { readLocation(); } const uint32_t afterLocation = nowMs(); probe.phase("downloader"); pollDownloader(now); pollGpxImport(); + pollSearchIndex(now); recorder.flushIfDue(io, now); if (recorder.state() == TrackRecorder::State::Recording && now - lastRecordingDraw >= 1000 && screen == Screen::Map) { // The elapsed time on the indicator @@ -1736,7 +1772,10 @@ void MapView::run() { if (waitingRoute && navigator.state() != Navigator::State::Searching) { waitingRoute = false; } - if (!waitingIdle && !waitingRoute) { + if (waitingSearchIndex && searchIndexJob.state() != SearchIndexJob::State::Busy) { + waitingSearchIndex = false; + } + if (!waitingIdle && !waitingRoute && !waitingSearchIndex) { const DebugScript::Step step = script.next(now); switch (step.action) { case DebugScript::Action::Key: @@ -1786,7 +1825,23 @@ void MapView::run() { waitingRoute = true; break; case DebugScript::Action::Replay: - startReplay(step.name); + startReplay(step.name, step.values[0], step.values[1], static_cast(step.values[2])); + break; + case DebugScript::Action::SearchIndex: + startSearchIndex(step.zoom != 0, step.name[0] != '\0' ? step.name : nullptr); + break; + case DebugScript::Action::WaitSearchIndex: + waitingSearchIndex = true; + break; + case DebugScript::Action::FileCheck: + checkFile(step.name); + break; + case DebugScript::Action::OffRoute: + if (setOffRouteTuning(step.name, step.values[0])) { + logInfo("Off route: %s = %g", step.name, static_cast(step.values[0])); + } else { + logError("Off route: no value named %s", step.name); + } break; case DebugScript::Action::TouchOnly: input.setTouchOnly(true); diff --git a/Apps/Maps/main/Source/app/MapView.h b/Apps/Maps/main/Source/app/MapView.h index fd89c12..8fc9de9 100644 --- a/Apps/Maps/main/Source/app/MapView.h +++ b/Apps/Maps/main/Source/app/MapView.h @@ -36,6 +36,10 @@ #include "../ui/MapOverlay.h" #include "../ui/PointerAcceleration.h" #include "../ui/WaypointSheet.h" +#include "../ui/SearchSheet.h" +#include "SearchIndexJob.h" +#include "SearchTask.h" +#include "../search/SearchEngine.h" #include "../ui/PositionTracker.h" #include "../ui/Settings.h" #include "../ui/GpxLayer.h" @@ -51,7 +55,7 @@ namespace maps { struct AffineScratch; -#define APP_VERSION "0.4.0" +#define APP_VERSION "0.5.0" constexpr const char* APP_ID = "tactility.maps"; /** @@ -64,14 +68,22 @@ class ReplaySource final : public PositionSource { static constexpr int MAX_POINTS = 2000; static constexpr uint32_t STEP_MS = 1000; bool load(const char* text, size_t size); + /** The accuracy reported (0: 5 m) and correlated noise of noiseM metres added to the points (0.4.1). */ + void setNoise(float accuracyM, float noiseM, uint32_t seed); void clear() { count = 0; } bool isActive() const { return count > 0; } int size() const { return count; } + /** The track point played last, for the log. */ + int pointIndex() const { return lastIndex; } PositionRead read(PositionFix& fix) override; private: double points[MAX_POINTS][2] {}; + /** Per point: noise north and east in metres. */ + float noise[MAX_POINTS][2] {}; + float accuracy = 5.0f; int count = 0; + int lastIndex = -1; int next = 0; uint32_t lastStep = 0; }; @@ -106,6 +118,7 @@ class MapView { Settings, About, Exit, + Search, Count }; @@ -166,7 +179,7 @@ class MapView { void followRoute(uint32_t now); /** The route line, above the roads and below labels and pins. */ void drawRoute(); - void startReplay(const char* name); + void startReplay(const char* name, float accuracyM = 0.0f, float noiseM = 0.0f, uint32_t seed = 0); /** Remembers the centre and zoom, so the map opens there next time. */ void saveView(); void setZoom(int zoom, int anchorX, int anchorY); @@ -338,6 +351,26 @@ class MapView { void openNewWaypointAtPosition(); /** Opens the sheet of an existing waypoint: Rename, Delete, Close. */ void openWaypointSheet(int index); + // Search (M6, MapSearch.cpp) + class CardSearchFile; + bool loadSearchIndexes(); + void releaseSearch(); + SearchOrigin searchOrigin(bool& fromPosition) const; + void openSearch(); + void closeSearch(); + void layoutSearch(); + void showRecentRows(); + void fillSearchRows(int from, int count, bool more); + void runSearch(); + void moreSearchResults(); + void searchNearby(int category); + void openSearchResult(const char* name, const char* type, int32_t x, int32_t y, const int32_t* box, const char* note); + void chooseSearchRow(int index, bool recent); + void handleSearchEvent(SearchSheet::Event event); + void loadRecentSearches(); + void saveRecentSearches(); + /** logInfo, and while a debug script runs also a line in scriptlog.txt (measurements without a serial log). */ + void scriptLog(const char* format, ...); void handleSheetButton(WaypointSheet::Button button); /** Saves the new waypoint of the sheet and says so. */ void saveNewWaypoint(); @@ -439,6 +472,7 @@ class MapView { /** A mode was chosen while a search ran: a new search starts once it has stopped. */ bool restartRoute = false; uint32_t lastRecalculation = 0; + int64_t lastFollowFixMs = -1; uint32_t lastFollowAt = 0; uint32_t lastFollowSignature = 0; uint32_t arrivedAt = 0; @@ -492,6 +526,18 @@ class MapView { /** The waypoint a rename or delete is asked for, from the sheet or the list. */ int editingWaypoint = -1; WaypointSheet sheet; + SearchSheet searchSheet; + SearchEngine* searchEngine = nullptr; + SearchTask searchTask; + SearchResult* searchResults = nullptr; + int searchResultCount = 0; + CardSearchFile* searchFiles[SEARCH_MAX_INDEXES] {}; + int searchFileCount = 0; + RecentSearches* recentSearches = nullptr; + /** The place of the last search result, marked on the map until another sheet opens. */ + double searchPinLat = 0.0; + double searchPinLon = 0.0; + bool searchPinShown = false; /** The places Enter walks through, nearest the centre first, and the one shown. */ static constexpr int MAX_PLACE_ORDER = 16; int placeOrder[MAX_PLACE_ORDER] {}; @@ -514,6 +560,17 @@ class MapView { int gpxFileCount = 0; ListView gpxList; GpxImport gpxImport; + /** The offline search indexes, built in idle time (M6). */ + SearchIndexJob searchIndexJob; + /** A build is due: at start, and after a download or an update ended. */ + bool searchIndexDue = true; + /** No new build before this time after a failed one. */ + uint32_t searchIndexRetryAt = 0; + bool waitingSearchIndex = false; + Downloader::State searchIndexDownloadState = Downloader::State::Idle; + void pollSearchIndex(uint32_t now); + void startSearchIndex(bool force, const char* only = nullptr); + void checkFile(const char* name); int gpxImporting = -1; char gpxFolder[PATH_MAX_LENGTH] = ""; Dialog dialog; diff --git a/Apps/Maps/main/Source/app/Navigator.cpp b/Apps/Maps/main/Source/app/Navigator.cpp index ab0c839..193569e 100644 --- a/Apps/Maps/main/Source/app/Navigator.cpp +++ b/Apps/Maps/main/Source/app/Navigator.cpp @@ -151,13 +151,19 @@ bool Navigator::start(IoTask& ioTask, TileStore& tileStore, TileWorker& tileWork task = nullptr; return false; } + if (routeReady.load()) { + // A route request coming first makes its own side roads + job = Job::Sides; + xSemaphoreGive(wake); + } return true; } void Navigator::stop() { if (task != nullptr) { - search.cancel(); + // Quit first, then cancel: runSides() resets the cancel and then looks for Quit job = Job::Quit; + search.cancel(); xSemaphoreGive(wake); xSemaphoreTake(stopped, portMAX_DELAY); vTaskDeleteWithCaps(task); @@ -244,6 +250,8 @@ void Navigator::loop() { } if (next == Job::Route) { runRoute(); + } else if (next == Job::Sides) { + runSides(); } else if (next == Job::Bench) { runBench(); } @@ -306,12 +314,48 @@ void Navigator::runRoute() { saveRoute(routes[current]); routes[1 - current].clear(); currentState = State::Done; + // The roads beside the route, for the follower's edge rule: after the route is shown, not before + runSides(); } else { next.clear(); currentState = State::Failed; } } +void Navigator::runSides() { + // The route's side roads: worked out on a copy of its points, then given to the route if it is still the same + search.resetCancel(); + if (job.load() == Job::Quit) { + return; + } + GrowArray points; + lock(); + const uint32_t generation = changes.load(); + const Route& route = routes[current]; + if (points.reserve(route.points.size())) { + for (const RoutePoint& point : route.points) { + points.push(point); + } + } + unlock(); + if (points.size() < 2) { + return; + } + CardSource source(*io, *store); + GrowArray sides; + const uint32_t started = nowMs(); + const bool complete = search.addSides(source, points, sides); + lock(); + if (complete && changes.load() == generation) { + routes[current].sides.swap(sides); + } + // Counted under the lock: clearRoute() may change the route right after + const auto sideCount = static_cast(routes[current].sides.size()); + unlock(); + logInfo("Route: %u side segments in %u ms", sideCount, + static_cast(nowMs() - started)); +} + void Navigator::runBench() { FileBuffer file; auto read = [&]() { return io->readFile(benchPath, file); }; diff --git a/Apps/Maps/main/Source/app/Navigator.h b/Apps/Maps/main/Source/app/Navigator.h index 3debbf8..86638f9 100644 --- a/Apps/Maps/main/Source/app/Navigator.h +++ b/Apps/Maps/main/Source/app/Navigator.h @@ -55,12 +55,13 @@ class Navigator { void clearRoute(); private: - enum class Job : uint8_t { None, Route, Bench, Quit }; + enum class Job : uint8_t { None, Route, Bench, Sides, Quit }; static void taskMain(void* context); void loop(); void runRoute(); void runBench(); + void runSides(); bool takeMemory(); void giveMemory(); bool saveRoute(const Route& route); diff --git a/Apps/Maps/main/Source/app/SearchIndexJob.cpp b/Apps/Maps/main/Source/app/SearchIndexJob.cpp new file mode 100644 index 0000000..2c82181 --- /dev/null +++ b/Apps/Maps/main/Source/app/SearchIndexJob.cpp @@ -0,0 +1,804 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "SearchIndexJob.h" + +#include "../mvt/MvtDecoder.h" +#include "../platform/Platform.h" +#include "../route/GrowArray.h" +#include "../search/SearchStream.h" +#include "../tiles/RegionMeta.h" +#include "../tiles/TileStore.h" + +#include +#include + +#include +#include +#include +#include +#include +#include + +namespace maps { + +namespace { + +constexpr uint32_t TASK_STACK = 24 * 1024; +/** Below the render lanes and the I/O task: the build gets the CPU when the map has nothing to do. */ +constexpr UBaseType_t TASK_PRIORITY = 1; +/** Tiles per side of an index block. */ +constexpr int32_t BLOCK_TILES = 15; +/** A tile is read in pieces of this size, each piece one I/O job that opens and closes the file. */ +constexpr uint32_t TILE_CHUNK = 16 * 1024; +/** The sort buffers on the device (two at most at once). */ +constexpr uint32_t SORT_BUFFER = 64 * 1024; +/** Files the build keeps open at once: the card's FAT mount allows four for the whole system. */ +constexpr int REAL_FILES = 2; +/** The stamp's format: change it to rebuild every index (e.g. after a change of the builder's output). */ +constexpr char STAMP_VERSION[] = "sidx1"; + +template +bool onCard(IoTask& io, Function& function) { + return runOnCard(io, function, IoPriority::Background); +} + +/** + * ScratchFiles on the card through the I/O task. Handles are virtual: at most REAL_FILES are open as FILE*, the others + * are closed and opened again at their position when used (written files are only appended to). + */ +class CardScratchFiles final : public ScratchFiles { +public: + explicit CardScratchFiles(IoTask& io) : io(io) {} + ~CardScratchFiles() override { + for (int i = 0; i < MAX_OPEN; i++) { + if (handles[i].used) { + close(i); + } + } + } + + int create(const char* path) override { return add(path, true, true); } + int openRead(const char* path) override { return add(path, false, false); } + int openAppend(const char* path) override { return add(path, true, false); } + + bool write(int h, const void* data, uint32_t size) override { + Handle* handle = at(h); + if (handle == nullptr || !handle->writing) { + return false; + } + bool ok = false; + auto job = [&]() { + ok = ensure(*handle) && (size == 0 || fwrite(data, 1, size, handle->file) == size); + return ok; + }; + onCard(io, job); + if (ok) { + handle->position += size; + } + return ok; + } + + int read(int h, void* data, uint32_t size) override { + Handle* handle = at(h); + if (handle == nullptr || handle->writing) { + return -1; + } + int result = -1; + auto job = [&]() { + if (!ensure(*handle)) { + return false; + } + const size_t n = fread(data, 1, size, handle->file); + result = n == 0 && ferror(handle->file) ? -1 : static_cast(n); + return result >= 0; + }; + onCard(io, job); + if (result > 0) { + handle->position += static_cast(result); + } + return result; + } + + bool seek(int h, uint32_t offset) override { + Handle* handle = at(h); + if (handle == nullptr || handle->writing) { + return false; + } + handle->position = offset; + if (handle->file == nullptr) { + return true; // positioned when it is opened again + } + bool ok = false; + auto job = [&]() { + ok = fseek(handle->file, static_cast(offset), SEEK_SET) == 0; + return ok; + }; + onCard(io, job); + return ok; + } + + bool close(int h) override { + Handle* handle = at(h); + if (handle == nullptr) { + return false; + } + bool ok = true; + if (handle->file != nullptr) { + auto job = [&]() { + ok = fclose(handle->file) == 0; + return ok; + }; + onCard(io, job); + handle->file = nullptr; + realOpen--; + } + handle->used = false; + return ok; + } + + bool remove(const char* path) override { + bool gone = false; + auto job = [&]() { + ::remove(path); + gone = !fileExists(path); + return gone; + }; + onCard(io, job); + return gone; + } + + bool rename(const char* from, const char* to) override { + bool ok = false; + auto job = [&]() { + ::remove(to); + ok = ::rename(from, to) == 0; + return ok; + }; + onCard(io, job); + return ok; + } + + /** Closes every real file (the handles stay valid): while the build waits, the card's few files are free. */ + void suspend() { + for (Handle& handle : handles) { + if (handle.used && handle.file != nullptr) { + auto job = [&]() { + fclose(handle.file); + return true; + }; + onCard(io, job); + handle.file = nullptr; + realOpen--; + } + } + } + +private: + struct Handle { + bool used; + bool writing; + bool truncate; + char path[PATH_MAX_LENGTH]; + uint32_t position; + uint32_t lastUse; + FILE* file; + }; + + int add(const char* path, bool writing, bool truncate) { + for (int i = 0; i < MAX_OPEN; i++) { + if (!handles[i].used) { + Handle& handle = handles[i]; + handle = {}; + handle.used = true; + handle.writing = writing; + handle.truncate = truncate; + snprintf(handle.path, sizeof(handle.path), "%s", path); + // Opened at once, so a missing file fails here + bool ok = false; + auto job = [&]() { + ok = ensure(handle); + return ok; + }; + onCard(io, job); + if (!ok) { + handle.used = false; + return -1; + } + return i; + } + } + return -1; + } + + Handle* at(int h) { return h >= 0 && h < MAX_OPEN && handles[h].used ? &handles[h] : nullptr; } + + /** On the I/O task: the handle's FILE*, opening it (and closing the least recently used one) when needed. */ + bool ensure(Handle& handle) { + handle.lastUse = ++useCounter; + if (handle.file != nullptr) { + return true; + } + if (realOpen >= REAL_FILES) { + Handle* oldest = nullptr; + for (Handle& other : handles) { + if (other.used && other.file != nullptr && (oldest == nullptr || other.lastUse < oldest->lastUse)) { + oldest = &other; + } + } + if (oldest != nullptr) { + fclose(oldest->file); + oldest->file = nullptr; + realOpen--; + } + } + if (handle.writing) { + handle.file = fopen(handle.path, handle.truncate ? "wb" : "ab"); + handle.truncate = false; + } else { + handle.file = fopen(handle.path, "rb"); + if (handle.file != nullptr && handle.position > 0 && fseek(handle.file, static_cast(handle.position), SEEK_SET) != 0) { + fclose(handle.file); + handle.file = nullptr; + } + } + if (handle.file == nullptr) { + return false; + } + realOpen++; + return true; + } + + IoTask& io; + Handle handles[MAX_OPEN] = {}; + int realOpen = 0; + uint32_t useCounter = 0; +}; + +/** A tile's .pbf read in pieces through the I/O task (each piece opens and closes the file). */ +class CardTileInput final : public ByteInput { +public: + CardTileInput(IoTask& io, uint8_t* buffer) : io(io), buffer(buffer) {} + void reset(const char* tilePath) { + snprintf(path, sizeof(path), "%s", tilePath); + offset = 0; + filled = 0; + at = 0; + failed = false; + } + int read(uint8_t* out, size_t size) override { + size_t copied = 0; + while (copied < size) { + if (at == filled) { + if (!refill()) { + return failed ? -1 : static_cast(copied); + } + } + const size_t n = std::min(size - copied, static_cast(filled - at)); + memcpy(out + copied, buffer + at, n); + at += static_cast(n); + copied += n; + } + return static_cast(copied); + } + bool rewind() override { + offset = 0; + filled = 0; + at = 0; + return true; + } + +private: + bool refill() { + uint32_t got = 0; + bool ok = false; + auto job = [&]() { + FILE* file = fopen(path, "rb"); + if (file == nullptr) { + return false; + } + ok = fseek(file, static_cast(offset), SEEK_SET) == 0; + if (ok) { + got = static_cast(fread(buffer, 1, TILE_CHUNK, file)); + } + fclose(file); + return ok; + }; + onCard(io, job); + if (!ok) { + failed = true; + return false; + } + offset += got; + filled = got; + at = 0; + return got > 0; + } + + IoTask& io; + uint8_t* buffer; + char path[PATH_MAX_LENGTH] = ""; + uint32_t offset = 0; + uint32_t filled = 0; + uint32_t at = 0; + bool failed = false; +}; + +class CardTileSource final : public StreamTileSource { +public: + CardTileSource(IoTask& io, const char* regionPath, uint8_t* buffer, MvtDecoder*& decoder) : input(io, buffer), decoder(decoder) { + snprintf(region, sizeof(region), "%s", regionPath); + } + ByteInput* open(int32_t x, int32_t y) override { + char path[PATH_MAX_LENGTH]; + snprintf(path, sizeof(path), "%s/14/%d/%d.pbf", region, static_cast(x), static_cast(y)); + input.reset(path); + return &input; + } + /** The decoder's pools are freed here (its memory block is the job's). */ + void tilesDone() override { + if (decoder != nullptr) { + decoder->~MvtDecoder(); + decoder = nullptr; + } + } + +private: + char region[PATH_MAX_LENGTH] = ""; + CardTileInput input; + MvtDecoder*& decoder; +}; + +struct TileList { + GrowArray tiles; + char path[PATH_MAX_LENGTH]; + int32_t x = 0; +}; + +bool collectTiles(const char* name, bool isDirectory, void* context) { + auto* list = static_cast(context); + if (isDirectory) { + return true; + } + const size_t length = strlen(name); + if (length < 5 || strcmp(name + length - 4, ".pbf") != 0) { + return true; + } + char* end = nullptr; + const long y = strtol(name, &end, 10); + if (end == name || strcmp(end, ".pbf") != 0) { + return true; + } + list->tiles.push({ list->x, static_cast(y) }); + return true; +} + +struct ColumnList { + GrowArray columns; +}; + +bool collectColumns(const char* name, bool isDirectory, void* context) { + if (!isDirectory) { + return true; + } + char* end = nullptr; + const long x = strtol(name, &end, 10); + if (end != name && *end == '\0') { + static_cast(context)->columns.push(static_cast(x)); + } + return true; +} + +struct RegionList { + char names[TileStore::MAX_REGIONS][16]; + int count = 0; +}; + +bool collectRegions(const char* name, bool isDirectory, void* context) { + auto* list = static_cast(context); + if (isDirectory && strlen(name) < sizeof(list->names[0]) && strcmp(name, BROWSE_REGION) != 0 && list->count < TileStore::MAX_REGIONS) { + snprintf(list->names[list->count++], sizeof(list->names[0]), "%s", name); + } + return true; +} + +struct FileName { + char text[16]; +}; + +/** Names of index files (in PSRAM: a country has many blocks). */ +struct IndexFiles { + GrowArray names; + void add(const char* name) { + FileName entry = {}; + snprintf(entry.text, sizeof(entry.text), "%s", name); + names.push(entry); + } + bool has(const char* name) const { + for (const FileName& entry : names) { + if (strcmp(entry.text, name) == 0) { + return true; + } + } + return false; + } +}; + +bool collectIndexFiles(const char* name, bool isDirectory, void* context) { + auto* list = static_cast(context); + const size_t length = strlen(name); + if (!isDirectory && length > 4 && strcmp(name + length - 4, ".idx") == 0 && length < sizeof(FileName::text)) { + list->add(name); + } + return true; +} + +/** What the builder's hook needs: the job, and the build's files (closed while the build waits for a download). */ +struct HookContext { + SearchIndexJob* job; + CardScratchFiles* files; +}; + +bool keepGoingHook(const StreamProgress& progress, void* context) { + auto* hook = static_cast(context); + if (hook->job->isPaused()) { + // The FAT mount allows 4 open files for the whole system: the download needs them + hook->files->suspend(); + } + return hook->job->keepGoing(static_cast(progress.done), static_cast(progress.total)); +} + +/** The task suspends itself when its work is over: it must not run any more when it is deleted. */ +void waitUntilSuspended(TaskHandle_t task) { + TaskStatus_t status {}; + for (;;) { + vTaskGetInfo(task, &status, pdFALSE, eInvalid); + if (status.eCurrentState == eSuspended) { + return; + } + vTaskDelay(pdMS_TO_TICKS(5)); + } +} + +} // namespace + +SearchIndexJob::~SearchIndexJob() { + stop(); +} + +bool SearchIndexJob::start(IoTask& ioTask, const char* rootPath, const char* first, bool force, const char* only) { + if (task != nullptr || currentState.load() == State::Busy) { + return false; + } + io = &ioTask; + snprintf(root, sizeof(root), "%s", rootPath); + snprintf(firstRegion, sizeof(firstRegion), "%s", first != nullptr ? first : ""); + snprintf(onlyRegion, sizeof(onlyRegion), "%s", only != nullptr ? only : ""); + forceBuild = force; + written = 0; + stopping = false; + paused = false; + progressDone = 0; + progressTotal = 0; + lowestFree = static_cast(heap_caps_get_free_size(MALLOC_CAP_SPIRAM)); + currentState = State::Busy; + if (xTaskCreatePinnedToCoreWithCaps(taskMain, "maps-sidx", TASK_STACK, this, TASK_PRIORITY, &task, tskNO_AFFINITY, MALLOC_CAP_SPIRAM) != + pdPASS) { + task = nullptr; + currentState = State::Failed; + return false; + } + return true; +} + +void SearchIndexJob::stop() { + if (task == nullptr) { + return; + } + stopping = true; + paused = false; + while (currentState.load() == State::Busy) { + vTaskDelay(pdMS_TO_TICKS(20)); + } + // The state is set just before the task suspends itself; delete it from here once it has + waitUntilSuspended(task); + vTaskDeleteWithCaps(task); + task = nullptr; +} + +void SearchIndexJob::acknowledge() { + if (currentState.load() != State::Busy) { + if (task != nullptr) { + waitUntilSuspended(task); + vTaskDeleteWithCaps(task); + task = nullptr; + } + currentState = State::Idle; + } +} + +void SearchIndexJob::taskMain(void* context) { + auto* self = static_cast(context); + self->run(); + vTaskSuspend(nullptr); +} + +bool SearchIndexJob::keepGoing(int32_t done, int32_t total) { + progressDone = done; + progressTotal = total; + const auto free = static_cast(heap_caps_get_free_size(MALLOC_CAP_SPIRAM)); + lowestFree = std::min(lowestFree, free); + while (paused.load() && !stopping.load()) { + vTaskDelay(pdMS_TO_TICKS(250)); + } + return !stopping.load(); +} + +void SearchIndexJob::report(const char* format, ...) { + const uint32_t index = reportWritten.load(); + if (index - reportRead.load() >= REPORT_LINES) { + return; // the map loop did not read the older lines yet: drop this one + } + va_list arguments; + va_start(arguments, format); + vsnprintf(reports[index % REPORT_LINES], sizeof(reports[0]), format, arguments); + va_end(arguments); + logInfo("%s", reports[index % REPORT_LINES]); + reportWritten = index + 1; +} + +const char* SearchIndexJob::nextReport() { + const uint32_t index = reportRead.load(); + if (index == reportWritten.load()) { + return nullptr; + } + snprintf(reportOut, sizeof(reportOut), "%s", reports[index % REPORT_LINES]); + reportRead = index + 1; + return reportOut; +} + +void SearchIndexJob::run() { + RegionList regions; + char path[PATH_MAX_LENGTH]; + snprintf(path, sizeof(path), "%s/maps", root); + auto list = [&]() { return listDirectory(path, collectRegions, ®ions); }; + onCard(*io, list); + // The map's region first + for (int i = 0; i < regions.count; i++) { + if (strcmp(regions.names[i], firstRegion) == 0 && i > 0) { + char swap[16]; + memcpy(swap, regions.names[0], sizeof(swap)); + memcpy(regions.names[0], regions.names[i], sizeof(swap)); + memcpy(regions.names[i], swap, sizeof(swap)); + } + } + bool ok = true; + for (int i = 0; i < regions.count && !stopping.load(); i++) { + if (onlyRegion[0] != '\0' && strcmp(regions.names[i], onlyRegion) != 0) { + continue; + } + ok = buildRegion(regions.names[i]) && ok; + } + currentState = ok ? State::Done : State::Failed; +} + +bool SearchIndexJob::buildRegion(const char* region) { + char regionPath[PATH_MAX_LENGTH]; + snprintf(regionPath, sizeof(regionPath), "%s/maps/%s", root, region); + // The stamp: what the index was built from + char stamp[200] = ""; + { + char metaPath[PATH_MAX_LENGTH]; + snprintf(metaPath, sizeof(metaPath), "%s/meta.json", regionPath); + RegionMeta meta = {}; + bool haveMeta = false; + auto readMeta = [&]() { + haveMeta = readRegionMeta(metaPath, meta); + return true; + }; + onCard(*io, readMeta); + if (haveMeta && meta.total > 0 && !meta.isComplete()) { + return true; // still downloading: built when it is complete + } + snprintf(stamp, sizeof(stamp), "%s %s %s %lld %llu", STAMP_VERSION, haveMeta ? meta.version : "-", haveMeta ? meta.date : "-", + haveMeta ? static_cast(meta.done) : -1LL, haveMeta ? static_cast(meta.bytes) : 0ULL); + } + char stampPath[PATH_MAX_LENGTH]; + snprintf(stampPath, sizeof(stampPath), "%s/search.inf", regionPath); + if (!forceBuild) { + char old[200] = ""; + auto readStamp = [&]() { + FILE* file = fopen(stampPath, "rb"); + if (file == nullptr) { + return false; + } + const size_t n = fread(old, 1, sizeof(old) - 1, file); + old[n] = '\0'; + fclose(file); + return true; + }; + onCard(*io, readStamp); + char* newline = strchr(old, '\n'); + if (newline != nullptr) { + *newline = '\0'; + } + if (strcmp(old, stamp) == 0) { + return true; // current + } + } + // The z14 tiles, sorted by x, then y + TileList list; + { + ColumnList columns; + snprintf(list.path, sizeof(list.path), "%s/14", regionPath); + auto walkColumns = [&]() { return listDirectory(list.path, collectColumns, &columns); }; + onCard(*io, walkColumns); + for (int32_t x : columns.columns) { + char columnPath[PATH_MAX_LENGTH]; + snprintf(columnPath, sizeof(columnPath), "%s/14/%d", regionPath, static_cast(x)); + list.x = x; + auto walk = [&]() { return listDirectory(columnPath, collectTiles, &list); }; + onCard(*io, walk); + } + } + if (list.tiles.size() == 0 || list.tiles.failed()) { + return true; // nothing to index + } + std::sort(list.tiles.begin(), list.tiles.end(), [](const StreamTile& a, const StreamTile& b) { return a.x != b.x ? a.x < b.x : a.y < b.y; }); + int32_t minX = list.tiles[0].x, maxX = minX, minY = list.tiles[0].y, maxY = minY; + for (const StreamTile& t : list.tiles) { + minX = std::min(minX, t.x); + maxX = std::max(maxX, t.x); + minY = std::min(minY, t.y); + maxY = std::max(maxY, t.y); + } + const bool blocks = maxX - minX >= BLOCK_TILES || maxY - minY >= BLOCK_TILES; + char scratch[PATH_MAX_LENGTH]; + snprintf(scratch, sizeof(scratch), "%s/srch", regionPath); + char blockFolder[PATH_MAX_LENGTH]; + snprintf(blockFolder, sizeof(blockFolder), "%s/search", regionPath); + auto makeFolders = [&]() { + mkdir(scratch, 0777); + if (blocks) { + mkdir(blockFolder, 0777); + } + return true; + }; + onCard(*io, makeFolders); + // Memory for the build: decoder, a tile buffer, the builder (freed at the end) + void* decoderMemory = largeAlloc(sizeof(MvtDecoder)); + auto* buffer = static_cast(largeAlloc(TILE_CHUNK)); + void* builderMemory = largeAlloc(sizeof(SearchStreamBuilder)); + if (decoderMemory == nullptr || buffer == nullptr || builderMemory == nullptr) { + largeFree(decoderMemory); + largeFree(buffer); + largeFree(builderMemory); + report("Search index %s: no memory to start", region); + return false; + } + auto* builder = new (builderMemory) SearchStreamBuilder(); + CardScratchFiles files(*io); + HookContext hook { this, &files }; + StreamBuildOptions options; + options.sortBufferBytes = SORT_BUFFER; + options.keepGoing = keepGoingHook; + options.context = &hook; + const char* profile = strcmp(region, "kyiv") == 0 ? "uk" : (strcmp(region, "belgrade") == 0 ? "sr" : "ru"); + options.config.profile = profile[0] == 'u' ? Translit::Ukrainian : (profile[0] == 's' ? Translit::Serbian : Translit::Russian); + // One block at a time (all tiles when the region is small) + GrowArray blockTiles; + IndexFiles made; + bool ok = true; + const uint32_t regionStarted = nowMs(); + uint64_t indexBytes = 0; + for (int32_t by = 0; ok && by <= (blocks ? (maxY - minY) / BLOCK_TILES : 0); by++) { + for (int32_t bx = 0; ok && bx <= (blocks ? (maxX - minX) / BLOCK_TILES : 0); bx++) { + blockTiles.clear(); + for (const StreamTile& t : list.tiles) { + if (!blocks || ((t.x - minX) / BLOCK_TILES == bx && (t.y - minY) / BLOCK_TILES == by)) { + blockTiles.push(t); + } + } + if (blockTiles.size() == 0) { + continue; + } + char outPath[PATH_MAX_LENGTH]; + char temporary[PATH_MAX_LENGTH]; + char blockName[16] = ""; + if (blocks) { + snprintf(blockName, sizeof(blockName), "%d_%d.idx", static_cast(bx), static_cast(by)); + made.add(blockName); + snprintf(outPath, sizeof(outPath), "%s/%s", blockFolder, blockName); + snprintf(temporary, sizeof(temporary), "%s/%d_%d.tmp", blockFolder, static_cast(bx), static_cast(by)); + } else { + snprintf(outPath, sizeof(outPath), "%s/search.idx", regionPath); + snprintf(temporary, sizeof(temporary), "%s/search.tmp", regionPath); + } + auto* decoder = new (decoderMemory) MvtDecoder(); + if (!decoder->init()) { + decoder->~MvtDecoder(); + ok = false; + report("Search index %s: no memory for the decoder", region); + break; + } + MvtDecoder* owned = decoder; + CardTileSource source(*io, regionPath, buffer, owned); + const uint32_t freeBefore = static_cast(heap_caps_get_free_size(MALLOC_CAP_SPIRAM)); + lowestFree = freeBefore; + const uint32_t started = nowMs(); + ok = builder->build(files, scratch, blockTiles.begin(), static_cast(blockTiles.size()), source, *decoder, outPath, temporary, + options); + files.suspend(); + if (owned != nullptr) { + owned->~MvtDecoder(); + } + const StreamBuildStats& s = builder->stats(); + const RegionStats& r = s.region; + if (!ok) { + report("Search index %s: %s", region, builder->cancelled() ? "stopped" : builder->error()); + break; + } + indexBytes += s.index.totalBytes; + written++; + report("Search index %s%s%s: %u tiles in %u ms, %u bytes; PSRAM %u KB free before, %u KB lowest", region, blocks ? " " : "", + blockName, static_cast(blockTiles.size()), static_cast(nowMs() - started), + static_cast(s.index.totalBytes), static_cast(freeBefore / 1024), static_cast(lowestFree / 1024)); + report("Search index phases ms: tiles %u points %u clusters %u matching %u addresses %u groups %u keys %u file %u", s.phaseMs[0], + s.phaseMs[1], s.phaseMs[2], s.phaseMs[3], s.phaseMs[4], s.phaseMs[5], s.phaseMs[6], s.phaseMs[7]); + report("Search index slowest tile %d/%d %u ms, matching %d/%d %u ms; scratch %u KB, sort %u KB, tables %u KB, matcher %u KB", + static_cast(s.slowestExtractTile.x), static_cast(s.slowestExtractTile.y), s.slowestExtractMs, + static_cast(s.slowestMatchTile.x), static_cast(s.slowestMatchTile.y), s.slowestMatchMs, + static_cast(s.scratchBytes / 1024), static_cast(s.sortPeakBytes / 1024), static_cast(s.tableBytes / 1024), + static_cast(s.matcherPeakBytes / 1024)); + report("Search index counts: places %u named POIs %u streets %u clusters %u houses %u (plain %.1f %%, village mode %d); street50 %u " + "street100 %u village %u none %u, %u village streets", + r.places, r.namedPois, r.streetNames, r.clusters, r.houses, 100.0 * r.plainShare, r.villageMode ? 1 : 0, r.pathStreet50, + r.pathStreet100, r.pathVillage, r.pathNone, r.pseudoStreets); + } + } + builder->~SearchStreamBuilder(); + largeFree(builderMemory); + largeFree(buffer); + largeFree(decoderMemory); + // The scratch folder, files of an older layout (one file vs blocks), then the stamp + auto cleanup = [&]() { + removeFolder(scratch); + if (!ok) { + return true; + } + char other[PATH_MAX_LENGTH]; + if (blocks) { + snprintf(other, sizeof(other), "%s/search.idx", regionPath); + ::remove(other); + IndexFiles existing; + listDirectory(blockFolder, collectIndexFiles, &existing); + for (const FileName& entry : existing.names) { + if (!made.has(entry.text)) { + snprintf(other, sizeof(other), "%s/%s", blockFolder, entry.text); + ::remove(other); + } + } + } else { + IndexFiles existing; + if (listDirectory(blockFolder, collectIndexFiles, &existing)) { + for (const FileName& entry : existing.names) { + snprintf(other, sizeof(other), "%s/%s", blockFolder, entry.text); + ::remove(other); + } + removeFolder(blockFolder); + } + } + FILE* file = fopen(stampPath, "wb"); + if (file != nullptr) { + fprintf(file, "%s\n", stamp); + fclose(file); + } + return true; + }; + onCard(*io, cleanup); + if (ok) { + report("Search index %s ready: %u blocks, %u bytes in %u ms", region, static_cast(blocks ? made.names.size() : 1), + static_cast(indexBytes), static_cast(nowMs() - regionStarted)); + } + return ok; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/app/SearchIndexJob.h b/Apps/Maps/main/Source/app/SearchIndexJob.h new file mode 100644 index 0000000..8e9ff71 --- /dev/null +++ b/Apps/Maps/main/Source/app/SearchIndexJob.h @@ -0,0 +1,88 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "IoTask.h" + +#include "../tiles/TilePaths.h" + +#include +#include + +#include +#include + +namespace maps { + +/** + * Builds the offline search indexes in idle time (M6): for every region on the card whose index is missing or older + * than its tiles (maps//search.inf holds the meta.json version, date and tile count it was built from), the + * region the map shows first. Uses SearchStreamBuilder on its own low-priority task with the scratch files in + * maps//srch; all card access goes through the I/O task at background priority, so tiles on screen and + * downloads come first. A region of at most 15 x 15 z14 tiles gets maps//search.idx; a larger one is split + * into blocks of 15 x 15 tiles from its first tile, maps//search/_.idx (15, not 16: a block then stays + * below 65536 units with the tiles' buffers, so its points keep 4 bytes). + */ +class SearchIndexJob { +public: + enum class State : uint8_t { + Idle, + Busy, + Done, + Failed + }; + + ~SearchIndexJob(); + /** + * Starts the background build; first is the region to build first (the map centre's), may be null. force builds + * even indexes that are current, only that one region when only is set (debug scripts). False while a build runs. + */ + bool start(IoTask& io, const char* rootPath, const char* first, bool force = false, const char* only = nullptr); + /** While paused the build waits between steps with its files closed (a download runs). */ + void pause(bool value) { paused = value; } + bool isPaused() const { return paused.load(); } + /** Stops a running build (its scratch files are removed) and waits for the task. */ + void stop(); + State state() const { return currentState.load(); } + /** Progress of the region being built (steps of the build), for "Building search index N/M". */ + int32_t done() const { return progressDone.load(); } + int32_t total() const { return progressTotal.load(); } + /** After Done or Failed: back to Idle. */ + void acknowledge(); + /** Whether an index was written by the last build (the map then reloads the search). */ + bool builtAny() const { return written > 0; } + /** One report line at a time for the log (measurements), nullptr when there is none; the line stays until the next call. */ + const char* nextReport(); + /** The lowest free PSRAM seen during the build, in bytes. */ + uint32_t lowestFreePsram() const { return lowestFree; } + + /** Called by the builder between steps: waits while paused, false when stopping. */ + bool keepGoing(int32_t done, int32_t total); + void report(const char* format, ...); + +private: + static void taskMain(void* context); + void run(); + bool buildRegion(const char* region); + + IoTask* io = nullptr; + TaskHandle_t task = nullptr; + std::atomic currentState = State::Idle; + std::atomic paused = false; + std::atomic stopping = false; + std::atomic progressDone = 0; + std::atomic progressTotal = 0; + char root[PATH_MAX_LENGTH] = ""; + char firstRegion[16] = ""; + char onlyRegion[16] = ""; + bool forceBuild = false; + int written = 0; + uint32_t lowestFree = 0; + /** Report lines: written by the task, read by the map loop (a small ring, guarded by the counters). */ + static constexpr int REPORT_LINES = 24; + char reports[REPORT_LINES][200] = {}; + std::atomic reportWritten = 0; + std::atomic reportRead = 0; + char reportOut[200] = ""; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/app/SearchTask.cpp b/Apps/Maps/main/Source/app/SearchTask.cpp new file mode 100644 index 0000000..b97cabb --- /dev/null +++ b/Apps/Maps/main/Source/app/SearchTask.cpp @@ -0,0 +1,81 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "SearchTask.h" + +#include "../platform/Platform.h" + +#include + +namespace maps { + +namespace { + +constexpr uint32_t TASK_STACK = 16 * 1024; +constexpr UBaseType_t TASK_PRIORITY = 4; + +} // namespace + +bool SearchTask::start() { + if (task != nullptr) { + return true; + } + wake = xSemaphoreCreateBinary(); + done = xSemaphoreCreateBinary(); + if (wake == nullptr || done == nullptr) { + stop(); + return false; + } + quit = false; + if (xTaskCreatePinnedToCoreWithCaps(taskMain, "maps-search", TASK_STACK, this, TASK_PRIORITY, &task, tskNO_AFFINITY, MALLOC_CAP_SPIRAM) != + pdPASS) { + task = nullptr; + stop(); + logError("Search: no task"); + return false; + } + return true; +} + +void SearchTask::stop() { + if (task != nullptr) { + quit = true; + xSemaphoreGive(wake); + xSemaphoreTake(done, portMAX_DELAY); + vTaskDeleteWithCaps(task); + task = nullptr; + } + if (wake != nullptr) { + vSemaphoreDelete(wake); + wake = nullptr; + } + if (done != nullptr) { + vSemaphoreDelete(done); + done = nullptr; + } +} + +bool SearchTask::run(Job job, void* context) { + if (!start()) { + return false; + } + pending = job; + pendingContext = context; + xSemaphoreGive(wake); + xSemaphoreTake(done, portMAX_DELAY); + return true; +} + +void SearchTask::taskMain(void* context) { + auto* self = static_cast(context); + while (true) { + xSemaphoreTake(self->wake, portMAX_DELAY); + if (self->quit) { + break; + } + self->pending(self->pendingContext); + xSemaphoreGive(self->done); + } + xSemaphoreGive(self->done); + vTaskSuspend(nullptr); +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/app/SearchTask.h b/Apps/Maps/main/Source/app/SearchTask.h new file mode 100644 index 0000000..733a3f7 --- /dev/null +++ b/Apps/Maps/main/Source/app/SearchTask.h @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include +#include +#include + +#include + +namespace maps { + +/** + * Runs the search engine's work on its own task with a large stack in PSRAM (M6). The engine reads 4 KB key pages and + * name groups into stack buffers; the map task has only a few KB of stack left (it overflowed on the T-Deck). The map + * waits for each job, so it behaves as a direct call. Started on the first search, stopped when the app ends. + */ +class SearchTask { +public: + using Job = void (*)(void* context); + + ~SearchTask() { stop(); } + bool start(); + void stop(); + /** Runs job(context) on the task and waits for it. False when the task could not start. */ + bool run(Job job, void* context); + /** The least stack the task has had left, in bytes (for the log). */ + uint32_t stackLeft() const { return task != nullptr ? static_cast(uxTaskGetStackHighWaterMark(task)) : 0; } + +private: + static void taskMain(void* context); + + TaskHandle_t task = nullptr; + SemaphoreHandle_t wake = nullptr; + SemaphoreHandle_t done = nullptr; + Job pending = nullptr; + void* pendingContext = nullptr; + bool quit = false; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/app/TileWorker.cpp b/Apps/Maps/main/Source/app/TileWorker.cpp index 15b7a03..c2881a3 100644 --- a/Apps/Maps/main/Source/app/TileWorker.cpp +++ b/Apps/Maps/main/Source/app/TileWorker.cpp @@ -29,6 +29,8 @@ constexpr uint32_t IDLE_WAIT_MS = 30; * much PSRAM free in all for the original, the features and the labels. */ constexpr size_t CONVERT_MIN_BLOCK = 336 * 1024; constexpr size_t CONVERT_MIN_FREE = 900 * 1024; +/** While a download runs, a tile on screen is prepared only with this much more PSRAM free, for the download's bodies. */ +constexpr size_t CONVERT_DOWNLOAD_SPARE = 400 * 1024; bool memoryForConversion() { return heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM) >= CONVERT_MIN_BLOCK && @@ -342,6 +344,20 @@ bool labelReadJob(IoTask& io, void* context) { return ok; } +struct FindJob { + const TileStore* store; + TileKey tile; + TileFile file; + bool found; +}; + +bool findJob(IoTask& io, void* context) { + (void)io; + auto* job = static_cast(context); + job->found = job->store->find(job->tile, job->file); + return true; +} + struct ScanJob { PreparedScanner* scanner; const TileStore* store; @@ -561,6 +577,15 @@ bool TileWorker::queueConversion(const TileKey& tile, int region, bool urgent, b for (int i = 0; i < conversionCount; i++) { if (conversions[i].tile == tile) { conversions[i].split = conversions[i].split || split; + if (urgent && !conversions[i].urgent) { + // Now on screen: first in line (during a download only the first, urgent one is prepared) + Conversion entry = conversions[i]; + entry.urgent = true; + for (int j = i; j > 0; j--) { + conversions[j] = conversions[j - 1]; + } + conversions[0] = entry; + } return true; } } @@ -674,10 +699,32 @@ bool TileWorker::loadLabels(Lane& lane) { } bool TileWorker::convertOne(Lane& lane) { - // The compressor needs one 320 KB block of PSRAM, which a download leaves too fragmented to find - if (conversionsPaused || routing) { + if (routing) { return false; } + // While a download runs (it leaves PSRAM fragmented), only the tiles on screen are prepared, so their POIs and house + // numbers show at once instead of after the download; the others wait until it is over. The download keeps memory + // to spare: the tile cache gives up slots that are not on screen for it (they come back after the download). + if (conversionsPaused) { + lock(); + const bool onScreen = conversionCount > 0 && conversions[0].urgent; + unlock(); + if (!onScreen) { + return false; + } + const size_t wanted = CONVERT_MIN_FREE + CONVERT_DOWNLOAD_SPARE; + if (heap_caps_get_free_size(MALLOC_CAP_SPIRAM) < wanted || heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM) < CONVERT_MIN_BLOCK) { + const TickType_t now = xTaskGetTickCount(); + if (static_cast(now - conversionsWaitUntil.load()) < 0) { + return false; + } + relieveMemory(CONVERT_MIN_BLOCK, wanted); + if (heap_caps_get_free_size(MALLOC_CAP_SPIRAM) < wanted || heap_caps_get_largest_free_block(MALLOC_CAP_SPIRAM) < CONVERT_MIN_BLOCK) { + conversionsWaitUntil = now + pdMS_TO_TICKS(DOWNLOAD_BACKOFF_MS); + return false; + } + } + } if (lane.trimBuffers.exchange(false)) { lane.heldKind = static_cast(Found::Nothing); lane.fileBuffer.trim(0); @@ -775,12 +822,20 @@ bool TileWorker::convertOne(Lane& lane) { ConvertReadJob read { root, region, job.tile, &lane.fileBuffer, 0, false, 0, false, region[0] == '\0' }; bool ok = region[0] != '\0' && io->run(IoPriority::Background, convertReadJob, &read); if (read.missing) { - // The scan finds it again where it is now - logInfo("[%d] %d/%d/%d is no longer in %s, not prepared there", lane.index, job.tile.z, job.tile.x, job.tile.y, - region[0] != '\0' ? region : "its region"); + // A download moved it (browsed tiles go into the new region). A tile on screen is queued again where it is + // now, still first; the others the scan finds again + FindJob findMoved { store, job.tile, {}, false }; + const bool found = job.urgent && io->run(IoPriority::Background, findJob, &findMoved) && findMoved.found && + findMoved.file.tile == job.tile && findMoved.file.region != job.region; + const TileFile& moved = findMoved.file; + logInfo("[%d] %d/%d/%d is no longer in %s, %s", lane.index, job.tile.z, job.tile.x, job.tile.y, region[0] != '\0' ? region : "its region", + found ? "queued again in its new region" : "not prepared there"); lane.fileBuffer.trim(KEEP_FILE_BUFFER); lock(); lane.converting = false; + if (found) { + queueConversion(job.tile, moved.region, true, job.split); + } unlock(); return true; } diff --git a/Apps/Maps/main/Source/app/TileWorker.h b/Apps/Maps/main/Source/app/TileWorker.h index 5f98eb6..0d62490 100644 --- a/Apps/Maps/main/Source/app/TileWorker.h +++ b/Apps/Maps/main/Source/app/TileWorker.h @@ -124,6 +124,8 @@ class TileWorker { static constexpr uint32_t BACKGROUND_PREPARE_MS = 10000; /** How long conversions wait after PSRAM had no block large enough for one. */ static constexpr uint32_t MEMORY_BACKOFF_MS = 30000; + /** During a download: the next try to make room for a tile on screen. */ + static constexpr uint32_t DOWNLOAD_BACKOFF_MS = 3000; /** Input arrived: prefetch reads and renders stop at once, and idle work waits. */ void cancelPrefetch(); /** diff --git a/Apps/Maps/main/Source/mvt/MvtDecoder.cpp b/Apps/Maps/main/Source/mvt/MvtDecoder.cpp index 1e659f3..f1a2c55 100644 --- a/Apps/Maps/main/Source/mvt/MvtDecoder.cpp +++ b/Apps/Maps/main/Source/mvt/MvtDecoder.cpp @@ -297,6 +297,10 @@ bool MvtDecoder::scanLayer(size_t layerEnd, LayerTable& table, Layer layer) { table.nameKey = index; } else if (strcmp(name, "name:latin") == 0) { table.nameLatinKey = index; + } else if (strcmp(name, "name_en") == 0) { + table.nameEnKey = index; + } else if (strcmp(name, "name:en") == 0) { + table.nameEnColonKey = index; } else if (strcmp(name, "rank") == 0) { table.rankKey = index; } else if (strcmp(name, "housenumber") == 0) { @@ -423,6 +427,13 @@ void MvtDecoder::applyTag(const LayerTable& table, uint32_t key, uint32_t value, } else if (keyIndex == table.nameLatinKey && entry->hasText != 0) { info.nameLatin = namePool + entry->textOffset; info.nameLatinLength = entry->textLength; + } else if (keyIndex == table.nameEnKey && entry->hasText != 0) { + info.nameEn = namePool + entry->textOffset; + info.nameEnLength = entry->textLength; + info.nameEnPreferred = true; + } else if (keyIndex == table.nameEnColonKey && entry->hasText != 0 && !info.nameEnPreferred) { + info.nameEn = namePool + entry->textOffset; + info.nameEnLength = entry->textLength; } else if (keyIndex == table.rankKey && isInteger) { info.rank = entry->rank; } else if (keyIndex == table.houseNumberKey && entry->hasText != 0) { diff --git a/Apps/Maps/main/Source/mvt/MvtDecoder.h b/Apps/Maps/main/Source/mvt/MvtDecoder.h index f0f6c24..6d12ed2 100644 --- a/Apps/Maps/main/Source/mvt/MvtDecoder.h +++ b/Apps/Maps/main/Source/mvt/MvtDecoder.h @@ -98,6 +98,9 @@ class MvtDecoder { int16_t layerKey = -1; int16_t nameKey = -1; int16_t nameLatinKey = -1; + /** "name_en" and "name:en": a layer can have both, on different features; name_en wins on a feature with both. */ + int16_t nameEnKey = -1; + int16_t nameEnColonKey = -1; int16_t rankKey = -1; int16_t houseNumberKey = -1; uint32_t valueOffset = 0; diff --git a/Apps/Maps/main/Source/mvt/Schema.h b/Apps/Maps/main/Source/mvt/Schema.h index 87c27ed..c9b19ea 100644 --- a/Apps/Maps/main/Source/mvt/Schema.h +++ b/Apps/Maps/main/Source/mvt/Schema.h @@ -70,6 +70,11 @@ struct FeatureInfo { /** The same from "name:latin" (OpenMapTiles' Latin-script name), nullptr when absent. */ const char* nameLatin = nullptr; uint16_t nameLatinLength = 0; + /** The same from "name_en" (or "name:en"), nullptr when absent; the search uses it for places, parks, water, peaks. */ + const char* nameEn = nullptr; + uint16_t nameEnLength = 0; + /** nameEn came from "name_en" (which wins over "name:en" on the same feature). */ + bool nameEnPreferred = false; /** "rank" as an integer, -1 when absent (place, mountain_peak and poi layers). */ int32_t rank = -1; /** The raw "class" and "subclass" strings of any layer, nullptr when absent. Valid during beginFeature(). */ diff --git a/Apps/Maps/main/Source/route/GrowArray.h b/Apps/Maps/main/Source/route/GrowArray.h index 2cf5e16..ba6af61 100644 --- a/Apps/Maps/main/Source/route/GrowArray.h +++ b/Apps/Maps/main/Source/route/GrowArray.h @@ -5,6 +5,7 @@ #include #include +#include namespace maps { @@ -73,6 +74,13 @@ class GrowArray { length--; } } + /** Exchanges the contents with another array, without copying. */ + void swap(GrowArray& other) { + std::swap(data, other.data); + std::swap(capacity, other.capacity); + std::swap(length, other.length); + std::swap(error, other.error); + } T& operator[](size_t index) { return data[index]; } const T& operator[](size_t index) const { return data[index]; } T* begin() { return data; } diff --git a/Apps/Maps/main/Source/route/RouteFollower.cpp b/Apps/Maps/main/Source/route/RouteFollower.cpp index 7cc91c9..a688290 100644 --- a/Apps/Maps/main/Source/route/RouteFollower.cpp +++ b/Apps/Maps/main/Source/route/RouteFollower.cpp @@ -4,6 +4,7 @@ #include #include #include +#include namespace maps { @@ -12,6 +13,8 @@ namespace { constexpr float PI_F = 3.14159265f; /** Directions at a turn are measured over this many metres before and after it, so a kink in the drawing is no turn. */ constexpr float BEARING_METRES = 15.0f; +/** The heading rule compares the course with the route's direction this far before and after the matched point. */ +constexpr float HEADING_METRES = 20.0f; /** Compass bearing in degrees of the step from a to b (graph y grows southwards). */ float bearingOf(const RoutePoint& a, const RoutePoint& b) { @@ -106,6 +109,121 @@ void routeManeuvers(const Route& route, GrowArray& out) { out.push({ route.lengthM, Turn::Arrive, -1 }); } +OffRouteTuning& offRouteTuning() { + static OffRouteTuning tuning; + return tuning; +} + +bool setOffRouteTuning(const char* name, float value) { + if (!std::isfinite(value)) { + return false; + } + // Set on a copy, kept only when the bounds still make sense (clamp needs min <= max) + OffRouteTuning t = offRouteTuning(); + struct Field { + const char* name; + float* number; + int* whole; + uint32_t* ms; + }; + const Field fields[] = { + { "accuracyFactor", &t.accuracyFactor, nullptr, nullptr }, + { "accuracyAddM", &t.accuracyAddM, nullptr, nullptr }, + { "minOffM", &t.minOffM, nullptr, nullptr }, + { "maxOffM", &t.maxOffM, nullptr, nullptr }, + { "noAccuracyM", &t.noAccuracyM, nullptr, nullptr }, + { "weakAccuracyM", &t.weakAccuracyM, nullptr, nullptr }, + { "windowMs", nullptr, nullptr, &t.windowMs }, + { "triggerFixes", nullptr, &t.triggerFixes, nullptr }, + { "poorFromM", &t.poorFromM, nullptr, nullptr }, + { "poorFixesPerM", &t.poorFixesPerM, nullptr, nullptr }, + { "headingSpeed", &t.headingSpeed, nullptr, nullptr }, + { "headingDegrees", &t.headingDegrees, nullptr, nullptr }, + { "headingShare", &t.headingShare, nullptr, nullptr }, + { "edgeRule", nullptr, &t.edgeRule, nullptr }, + { "edgeShare", &t.edgeShare, nullptr, nullptr }, + { "edgeMarginM", &t.edgeMarginM, nullptr, nullptr }, + { "edgeMaxAccuracyM", &t.edgeMaxAccuracyM, nullptr, nullptr }, + { "recalculateMs", nullptr, nullptr, &t.recalculateMs }, + { "settleFixes", nullptr, &t.settleFixes, nullptr }, + { "startGuardM", &t.startGuardM, nullptr, nullptr }, + }; + for (const Field& field : fields) { + if (strcmp(field.name, name) != 0) { + continue; + } + if (field.number != nullptr) { + *field.number = value; + } else if (field.whole != nullptr) { + *field.whole = static_cast(value); + } else { + *field.ms = static_cast(std::max(0.0f, value)); + } + if (t.minOffM < 0.0f || t.minOffM > t.maxOffM || t.triggerFixes < 1 || t.settleFixes < 0) { + return false; + } + offRouteTuning() = t; + return true; + } + return false; +} + +float offRouteThreshold(const OffRouteTuning& tuning, float accuracyM) { + return std::clamp(tuning.accuracyFactor * accuracyM + tuning.accuracyAddM, tuning.minOffM, tuning.maxOffM); +} + +int offRouteFixes(const OffRouteTuning& tuning, float accuracyM) { + const float extra = accuracyM > tuning.poorFromM ? tuning.poorFixesPerM * (accuracyM - tuning.poorFromM) : 0.0f; + return tuning.triggerFixes + static_cast(extra + 0.5f); +} + +const char* offRouteRuleName(OffRouteRule rule) { + switch (rule) { + case OffRouteRule::Distance: + return "distance"; + case OffRouteRule::Heading: + return "heading"; + case OffRouteRule::Edge: + return "edge"; + default: + return "none"; + } +} + +void AccuracyWindow::add(float accuracyM, uint32_t now, const OffRouteTuning& tuning) { + const float value = std::isfinite(accuracyM) && accuracyM > 0.0f ? accuracyM : tuning.noAccuracyM; + // Fixes older than the window go; the newest is always kept + int kept = 0; + for (int i = 0; i < count; i++) { + if (now - times[i] < tuning.windowMs) { + values[kept] = values[i]; + times[kept] = times[i]; + kept++; + } + } + count = kept; + if (count == MAX) { + for (int i = 1; i < count; i++) { + values[i - 1] = values[i]; + times[i - 1] = times[i]; + } + count--; + } + values[count] = value; + times[count] = now; + count++; +} + +float AccuracyWindow::median() const { + if (count == 0) { + return NAN; + } + float sorted[MAX]; + std::copy(values, values + count, sorted); + std::sort(sorted, sorted + count); + return count % 2 == 1 ? sorted[count / 2] : 0.5f * (sorted[count / 2 - 1] + sorted[count / 2]); +} + void RouteFollower::begin(const Route& route) { routeManeuvers(route, maneuvers); current = {}; @@ -115,16 +233,31 @@ void RouteFollower::begin(const Route& route) { current.nextTurn = maneuvers.size() > 0 ? maneuvers[0].turn : Turn::Arrive; current.toTurnM = maneuvers.size() > 0 ? maneuvers[0].atM : route.lengthM; current.nextStep = maneuvers.size() > 0 ? maneuvers[0].step : -1; + current.headingDifference = NAN; + current.sideM = INFINITY; lastSegment = 0; - away = false; - awaySince = 0; + awayFixes = 0; + // A new route (also a recalculated one) settles for a few fixes; the accuracy history stays + settleLeft = offRouteTuning().settleFixes; active = route.points.size() >= 2; } -const RouteFollower::Status& RouteFollower::update(const Route& route, double lat, double lon, float accuracyM, uint32_t now) { +const RouteFollower::Status& RouteFollower::update(const Route& route, double lat, double lon, float accuracyM, uint32_t now, float courseDeg, + float speedMps, bool newFix) { if (!active || route.points.size() < 2 || current.state == State::Arrived) { return current; } + const OffRouteTuning& tuning = offRouteTuning(); + if (newFix) { + accuracy.add(accuracyM, now, tuning); + } + const float acc = accuracy.median(); + const float off = offRouteThreshold(tuning, acc); + const bool weak = acc > tuning.weakAccuracyM; + current.accuracyM = acc; + current.offM = off; + current.weakGps = weak; + const RoutePoint here = routePointFrom(lat, lon); const auto metresPerUnit = static_cast(routeMetresPerUnit(here.y)); const size_t segments = route.points.size() - 1; @@ -159,28 +292,113 @@ const RouteFollower::Status& RouteFollower::update(const Route& route, double la const size_t windowFirst = lastSegment > 20 ? lastSegment - 20 : 0; const size_t windowLast = std::min(segments, lastSegment + 400); nearest(windowFirst, windowLast, segment, t, distance); - const float matchLimit = accuracyM + MATCH_METRES; + const float matchLimit = std::max(off, acc + MATCH_METRES); if (distance > matchLimit && (windowFirst > 0 || windowLast < segments)) { nearest(0, segments, segment, t, distance); } const float segmentMetres = route.distances[segment + 1] - route.distances[segment]; const float progress = route.distances[segment] + t * segmentMetres; current.distanceFromRouteM = distance; - const float offLimit = std::max(OFF_ROUTE_METRES, accuracyM + 25.0f); - if (distance <= matchLimit) { - away = false; + + // The direction of travel against the route's at the matched segment + const RoutePoint& a = route.points[segment]; + const RoutePoint& b = route.points[segment + 1]; + current.headingDifference = NAN; + if (std::isfinite(courseDeg)) { + // The smallest difference to the route's direction within HEADING_METRES along it: near a bend of the route, a + // position matched to the segment before it while walking the one after it is no wrong direction + const float at = route.distances[segment] + t * segmentMetres; + size_t first = segment; + while (first > 0 && at - route.distances[first] < HEADING_METRES) { + first--; + } + size_t last = segment; + while (last + 1 < segments && route.distances[last + 1] - at < HEADING_METRES) { + last++; + } + float smallest = INFINITY; + for (size_t i = first; i <= last; i++) { + const RoutePoint& p = route.points[i]; + const RoutePoint& q = route.points[i + 1]; + if (p.x == q.x && p.y == q.y) { + continue; + } + // fmod, not a loop: a broken course of 1e10 degrees must not hang the update (remainderf is not exported) + float difference = std::fmod(courseDeg - bearingOf(p, q), 360.0f); + if (difference > 180.0f) { + difference -= 360.0f; + } else if (difference <= -180.0f) { + difference += 360.0f; + } + smallest = std::min(smallest, std::fabs(difference)); + } + current.headingDifference = std::isfinite(smallest) ? smallest : NAN; + } + // The nearest other road, for the edge rule + current.sideM = INFINITY; + if (tuning.edgeRule != 0 && route.sides.size() > 0) { + float best = INFINITY; + for (const RouteSide& side : route.sides) { + const float dx = static_cast(side.b.x - side.a.x); + const float dy = static_cast(side.b.y - side.a.y); + const float lengthSquared = dx * dx + dy * dy; + float u = lengthSquared > 0.0f ? ((here.x - side.a.x) * dx + (here.y - side.a.y) * dy) / lengthSquared : 0.0f; + u = std::clamp(u, 0.0f, 1.0f); + const float px = side.a.x + u * dx - here.x; + const float py = side.a.y + u * dy - here.y; + best = std::min(best, px * px + py * py); + } + current.sideM = std::sqrt(best) * metresPerUnit; + } + + // Is this fix away from the route, and by which rule? Weak GPS never triggers. + OffRouteRule rule = OffRouteRule::None; + if (!weak) { + const float edgeMargin = std::max(tuning.edgeMarginM, 0.3f * acc); + // Near the start the route was snapped from one position: only the distance counts there + const bool pastStart = progress > tuning.startGuardM; + if (distance > off) { + rule = OffRouteRule::Distance; + } else if (pastStart && std::isfinite(current.headingDifference) && std::isfinite(speedMps) && speedMps > tuning.headingSpeed && + current.headingDifference > tuning.headingDegrees && distance > tuning.headingShare * off) { + rule = OffRouteRule::Heading; + } else if (pastStart && tuning.edgeRule != 0 && acc <= tuning.edgeMaxAccuracyM && distance > tuning.edgeShare * off && + current.sideM + edgeMargin < distance) { + rule = OffRouteRule::Edge; + } + } + if (settleLeft > 0) { + if (newFix) { + settleLeft--; + } + rule = OffRouteRule::None; + } + current.awayRule = rule; + if (rule == OffRouteRule::None) { + awayFixes = 0; + current.firedRule = OffRouteRule::None; + } else if (newFix) { + awayFixes++; + } + current.awayFixes = static_cast(std::min(awayFixes, 255)); + + // Weak GPS only stops the off-route rule: a fix further than the match limit (accuracy + MATCH_METRES), beyond the + // goal say, moves neither the progress nor makes an arrival + if (rule == OffRouteRule::None && distance <= matchLimit) { + // On the route (in weak GPS too, followed by projection) lastSegment = segment; current.progressM = progress; - const RoutePoint& a = route.points[segment]; - const RoutePoint& b = route.points[segment + 1]; current.matched = { a.x + static_cast((b.x - a.x) * t), a.y + static_cast((b.y - a.y) * t) }; current.state = State::OnRoute; - } else if (distance > offLimit) { - if (!away) { - away = true; - awaySince = now; + } else if (rule != OffRouteRule::None) { + if (awayFixes >= offRouteFixes(tuning, acc)) { + if (current.state != State::OffRoute) { + current.firedRule = rule; + } + current.state = State::OffRoute; + } else if (current.state != State::OffRoute) { + current.state = State::Away; } - current.state = now - awaySince >= OFF_ROUTE_MS ? State::OffRoute : State::Away; } // Arrival: close to the goal, or the end of the route reached on it const RoutePoint& goal = route.points[route.points.size() - 1]; diff --git a/Apps/Maps/main/Source/route/RouteFollower.h b/Apps/Maps/main/Source/route/RouteFollower.h index 6886a06..9f19731 100644 --- a/Apps/Maps/main/Source/route/RouteFollower.h +++ b/Apps/Maps/main/Source/route/RouteFollower.h @@ -5,6 +5,7 @@ #include "RouteSearch.h" #include "RouteTurn.h" +#include #include #include @@ -26,14 +27,81 @@ Turn turnFor(float degrees); const char* turnPhrase(Turn turn); /** - * Follows a route with positions (M5c-4): snaps a position to the route within accuracy + MATCH_METRES, notices when - * it has been more than OFF_ROUTE_METRES (or accuracy + 25 m) away for OFF_ROUTE_MS, finds the next turn and arrival. + * When a position counts as off the route (0.4.1). The threshold follows the GPS precision: the median accuracy of the + * fixes of the last windowMs gives acc, and off = clamp(accuracyFactor * acc + accuracyAddM, minOffM, maxOffM). A fix is + * away when it is farther than off from the route, or farther than headingShare * off while moving faster than + * headingSpeed in a direction more than headingDegrees from the route's, or (edge rule) nearer to another road than to + * the route by edgeMarginM while farther than edgeShare * off. triggerFixes away fixes in a row are off route. Above + * weakAccuracyM nothing triggers ("Weak GPS"). The values can be changed by the "offroute" debug command. + */ +struct OffRouteTuning { + float accuracyFactor = 1.5f; + float accuracyAddM = 5.0f; + float minOffM = 15.0f; + float maxOffM = 60.0f; + /** The accuracy of a fix that reports none (Tactility gives HDOP x 5 m when the GPS sends HDOP). */ + float noAccuracyM = 30.0f; + float weakAccuracyM = 50.0f; + uint32_t windowMs = 10000; + int triggerFixes = 3; + /** + * Above poorFromM of accuracy, poorFixesPerM more fixes per metre are needed (25 m: 3 + 5 = 8): GPS noise of that size + * alone crosses the threshold for 3 fixes now and then on a long walk (PC replays, MEASUREMENTS.md "0.4.1"). + */ + float poorFromM = 10.0f; + float poorFixesPerM = 0.33f; + float headingSpeed = 1.0f; + float headingDegrees = 60.0f; + float headingShare = 0.667f; + /** The edge rule: 0 turns it off. */ + int edgeRule = 1; + float edgeShare = 0.667f; + float edgeMarginM = 4.0f; + /** The edge rule only with GPS this good: with poor GPS the noise alone puts fixes nearer to side streets. */ + float edgeMaxAccuracyM = 10.0f; + /** At least this long between two recalculations after the first one. */ + uint32_t recalculateMs = 10000; + /** Fixes not checked after a new route, so it settles. */ + int settleFixes = 2; + /** The heading and edge rules count only past this many metres along a route (device replays: false alarms at the start). */ + float startGuardM = 30.0f; +}; + +/** The tuning in use, shared by the follower and the map (the debug script changes it). */ +OffRouteTuning& offRouteTuning(); +/** Sets one tuning value by its name ("minOffM", "triggerFixes", ...). False for an unknown name, a value that is not finite, or one that reverses a bound (minOffM above maxOffM); nothing changes then. */ +bool setOffRouteTuning(const char* name, float value); +/** The threshold for a median accuracy: clamp(factor * acc + add, min, max). */ +float offRouteThreshold(const OffRouteTuning& tuning, float accuracyM); +/** The away fixes in a row that make off route at a median accuracy. */ +int offRouteFixes(const OffRouteTuning& tuning, float accuracyM); + +/** The median of the accuracies of the last window of fixes; a single bad fix doesn't move it. */ +class AccuracyWindow { +public: + static constexpr int MAX = 32; + void clear() { count = 0; } + /** Adds a fix's accuracy (NAN or <= 0: noAccuracyM) at time now. */ + void add(float accuracyM, uint32_t now, const OffRouteTuning& tuning); + float median() const; + int size() const { return count; } + +private: + float values[MAX] {}; + uint32_t times[MAX] {}; + int count = 0; +}; + +enum class OffRouteRule : uint8_t { None, Distance, Heading, Edge }; +const char* offRouteRuleName(OffRouteRule rule); + +/** + * Follows a route with positions (M5c-4, 0.4.1): projects a position onto the route, notices when it left the route + * (OffRouteTuning), finds the next turn and arrival. */ class RouteFollower { public: static constexpr float MATCH_METRES = 20.0f; - static constexpr float OFF_ROUTE_METRES = 40.0f; - static constexpr uint32_t OFF_ROUTE_MS = 10000; static constexpr float ARRIVE_METRES = 25.0f; enum class State : uint8_t { Waiting, OnRoute, Away, OffRoute, Arrived }; @@ -50,14 +118,31 @@ class RouteFollower { float toTurnM; /** The step the next turn goes onto, -1 for the arrival. */ int nextStep; + /** The median accuracy and the off-route threshold used for the last fix. */ + float accuracyM; + float offM; + /** Course against the route's direction in degrees, NAN without a course. */ + float headingDifference; + /** Distance to the nearest other road (Route::sides), INFINITY without one. */ + float sideM; + /** The rule that counted the last fix as away, and the one that made it off route. */ + OffRouteRule awayRule; + OffRouteRule firedRule; + uint8_t awayFixes; + bool weakGps; }; /** Starts following a route; the maneuvers are worked out once here. */ void begin(const Route& route); void end() { active = false; } bool isActive() const { return active; } - /** A new position (accuracy in metres) at time now. */ - const Status& update(const Route& route, double lat, double lon, float accuracyM, uint32_t now); + /** + * A position at time now: accuracy in metres (NAN: unknown), course in degrees and speed in m/s (NAN: unknown). + * Each call with newFix counts as one fix for the off-route rule; newFix false (the last fix shown again) updates + * the status without counting, so one wrong fix never counts twice. + */ + const Status& update(const Route& route, double lat, double lon, float accuracyM, uint32_t now, float courseDeg = NAN, + float speedMps = NAN, bool newFix = true); const Status& status() const { return current; } /** The text of the next instruction, e.g. "In 120 m turn left onto Štefánikova". */ void instruction(const Route& route, char* out, size_t size) const; @@ -70,8 +155,9 @@ class RouteFollower { Status current {}; /** The segment matched last, where the next match looks first. */ size_t lastSegment = 0; - uint32_t awaySince = 0; - bool away = false; + int awayFixes = 0; + int settleLeft = 0; + AccuracyWindow accuracy; }; /** "120 m", "850 m", "2.4 km". */ diff --git a/Apps/Maps/main/Source/route/RouteSearch.cpp b/Apps/Maps/main/Source/route/RouteSearch.cpp index 01026a8..e1552b4 100644 --- a/Apps/Maps/main/Source/route/RouteSearch.cpp +++ b/Apps/Maps/main/Source/route/RouteSearch.cpp @@ -173,6 +173,7 @@ void Route::clear() { distances.clear(); steps.clear(); names.clear(); + sides.clear(); lengthM = 0.0f; seconds = 0.0f; } diff --git a/Apps/Maps/main/Source/route/RouteSearch.h b/Apps/Maps/main/Source/route/RouteSearch.h index 79c1acc..0187444 100644 --- a/Apps/Maps/main/Source/route/RouteSearch.h +++ b/Apps/Maps/main/Source/route/RouteSearch.h @@ -33,6 +33,12 @@ struct RouteStep { uint8_t flags; }; +/** A road segment near a route that is not part of it, for the follower's edge rule (0.4.1). */ +struct RouteSide { + RoutePoint a; + RoutePoint b; +}; + /** A found route: its line, the distance from the start at each point, its roads, length and time. */ struct Route { RouteProfile profile = RouteProfile::Walk; @@ -43,6 +49,8 @@ struct Route { GrowArray distances; GrowArray steps; GrowArray names; + /** Road segments within SIDE_METRES of the route that leave it at an angle (RouteSearch::addSides), not saved. */ + GrowArray sides; float lengthM = 0.0f; float seconds = 0.0f; @@ -107,8 +115,17 @@ class RouteSearch { RouteResult find(GraphSource& source, double fromLat, double fromLon, double toLat, double toLon, RouteProfile profile, Route& route, RouteStats* stats); + /** + * Fills sides from the graphs of the tiles along a route's points: road segments within SIDE_METRES of it that are + * not part of it and not parallel to it (a separately drawn pavement is no other road). Needs no search memory. + */ + bool addSides(GraphSource& source, const GrowArray& points, GrowArray& sides); + static constexpr float SIDE_METRES = 60.0f; + static constexpr size_t MAX_SIDES = 8000; /** Makes a running find() return Cancelled soon. */ void cancel() { cancelled = true; } + /** For work after a search (addSides): a cancel from before no longer counts. */ + void resetCancel() { cancelled = false; } /** The node states the memory allows, for the log. */ uint32_t nodeCapacity() const { return maxNodes; } diff --git a/Apps/Maps/main/Source/route/RouteSides.cpp b/Apps/Maps/main/Source/route/RouteSides.cpp new file mode 100644 index 0000000..60bde85 --- /dev/null +++ b/Apps/Maps/main/Source/route/RouteSides.cpp @@ -0,0 +1,178 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// The roads next to a route (0.4.1): segments of other roads near the route, for the follower's edge rule. +#include "RouteSearch.h" + +#include +#include + +namespace maps { + +namespace { + +/** Route points per chunk; each chunk has a box, so a test looks only at the few chunks nearby. */ +constexpr size_t CHUNK_POINTS = 16; +/** A road segment is cut into pieces of at most this many metres before it is tested. */ +constexpr float PIECE_METRES = 15.0f; +/** A piece this near the route is part of it. */ +constexpr float ON_ROUTE_METRES = 3.0f; +/** A piece at less than this angle to the route runs beside it (a separate pavement, a parallel lane): not kept. */ +constexpr float PARALLEL_DEGREES = 30.0f; + +struct Chunk { + size_t first; + size_t last; + int32_t minX; + int32_t minY; + int32_t maxX; + int32_t maxY; +}; + +float pointSegment(float px, float py, const RoutePoint& a, const RoutePoint& b, float& t) { + const float dx = static_cast(b.x - a.x); + const float dy = static_cast(b.y - a.y); + const float lengthSquared = dx * dx + dy * dy; + t = lengthSquared > 0.0f ? ((px - a.x) * dx + (py - a.y) * dy) / lengthSquared : 0.0f; + t = std::clamp(t, 0.0f, 1.0f); + const float ex = a.x + t * dx - px; + const float ey = a.y + t * dy - py; + return std::sqrt(ex * ex + ey * ey); +} + +} // namespace + +bool RouteSearch::addSides(GraphSource& source, const GrowArray& points, GrowArray& sides) { + sides.clear(); + const size_t count = points.size(); + if (count < 2) { + return false; + } + if (!inflaterReady) { + inflaterReady = inflater.init(); + if (!inflaterReady) { + return false; + } + } + const float metresPerUnit = static_cast(routeMetresPerUnit(points[0].y)); + const auto radius = static_cast(SIDE_METRES / metresPerUnit) + 1; + const float onRoute = ON_ROUTE_METRES / metresPerUnit; + const float pieceUnits = PIECE_METRES / metresPerUnit; + const float parallelCos = std::cos(PARALLEL_DEGREES * 3.14159265f / 180.0f); + + GrowArray chunks; + for (size_t first = 0; first + 1 < count; first += CHUNK_POINTS) { + const size_t last = std::min(count - 1, first + CHUNK_POINTS); + Chunk chunk { first, last, INT32_MAX, INT32_MAX, INT32_MIN, INT32_MIN }; + for (size_t i = first; i <= last; i++) { + chunk.minX = std::min(chunk.minX, points[i].x); + chunk.minY = std::min(chunk.minY, points[i].y); + chunk.maxX = std::max(chunk.maxX, points[i].x); + chunk.maxY = std::max(chunk.maxY, points[i].y); + } + chunk.minX -= radius; + chunk.minY -= radius; + chunk.maxX += radius; + chunk.maxY += radius; + chunks.push(chunk); + if (chunks.failed()) { + return false; + } + } + // The nearest route segment to a point: its distance in units and the segment + auto nearestOnRoute = [&](float px, float py, size_t& segment) { + float best = INFINITY; + for (const Chunk& chunk : chunks) { + if (px < chunk.minX || px > chunk.maxX || py < chunk.minY || py > chunk.maxY) { + continue; + } + for (size_t i = chunk.first; i < chunk.last; i++) { + float t; + const float d = pointSegment(px, py, points[i], points[i + 1], t); + if (d < best) { + best = d; + segment = i; + } + } + } + return best; + }; + + // The tiles the route passes, and their neighbours where the route comes near a side + GrowArray sideTiles; + auto addTile = [&](int32_t tx, int32_t ty) { + const TileKey key { 14, tx, ty }; + for (const TileKey& known : sideTiles) { + if (known == key) { + return; + } + } + sideTiles.push(key); + }; + for (size_t i = 0; i < count; i++) { + const RoutePoint& p = points[i]; + for (int oy = -1; oy <= 1; oy++) { + for (int ox = -1; ox <= 1; ox++) { + addTile((p.x + ox * radius) / GRAPH_EXTENT, (p.y + oy * radius) / GRAPH_EXTENT); + } + } + } + + TileGraph graph; + for (const TileKey& tile : sideTiles) { + if (sides.size() >= MAX_SIDES) { + break; + } + if (cancelled) { + // The navigator stops: the side roads are optional + sides.clear(); + return false; + } + if (!source.readGraph(tile, fileBuffer) || !loadTileGraph(fileBuffer.begin(), fileBuffer.size(), inflater, graph)) { + continue; + } + const int32_t baseX = tile.x * GRAPH_EXTENT; + const int32_t baseY = tile.y * GRAPH_EXTENT; + for (uint32_t e = 0; e < graph.edges.size() && sides.size() < MAX_SIDES; e++) { + bool havePrevious = false; + RoutePoint previous {}; + graph.shape(e, false, [&](int32_t x, int32_t y) { + const RoutePoint point { baseX + x, baseY + y }; + if (havePrevious && sides.size() < MAX_SIDES) { + const float dx = static_cast(point.x - previous.x); + const float dy = static_cast(point.y - previous.y); + const float length = std::sqrt(dx * dx + dy * dy); + const int pieces = std::max(1, static_cast(std::ceil(length / pieceUnits))); + for (int k = 0; k < pieces; k++) { + const RoutePoint a { previous.x + static_cast(dx * k / pieces), previous.y + static_cast(dy * k / pieces) }; + const RoutePoint b { previous.x + static_cast(dx * (k + 1) / pieces), + previous.y + static_cast(dy * (k + 1) / pieces) }; + const float mx = 0.5f * static_cast(a.x + b.x); + const float my = 0.5f * static_cast(a.y + b.y); + size_t segment = 0; + const float d = nearestOnRoute(mx, my, segment); + if (!(d <= radius) || d < onRoute) { + continue; + } + // Leaves out pieces running beside the route + const RoutePoint& ra = points[segment]; + const RoutePoint& rb = points[segment + 1]; + const float rx = static_cast(rb.x - ra.x); + const float ry = static_cast(rb.y - ra.y); + const float px = static_cast(b.x - a.x); + const float py = static_cast(b.y - a.y); + const float norms = std::sqrt((rx * rx + ry * ry) * (px * px + py * py)); + if (norms > 0.0f && std::fabs(rx * px + ry * py) / norms > parallelCos) { + continue; + } + sides.push({ a, b }); + } + } + previous = point; + havePrevious = true; + }); + } + } + graph.clear(); + return true; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/AddressMatch.cpp b/Apps/Maps/main/Source/search/AddressMatch.cpp new file mode 100644 index 0000000..fd277cc --- /dev/null +++ b/Apps/Maps/main/Source/search/AddressMatch.cpp @@ -0,0 +1,318 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "AddressMatch.h" + +#include +#include + +namespace maps { + +namespace { + +constexpr uint32_t NONE = UINT32_MAX; + +float segmentDistance(float px, float py, float ax, float ay, float bx, float by) { + const float dx = bx - ax; + const float dy = by - ay; + const float lengthSquared = dx * dx + dy * dy; + float t = lengthSquared > 0.0f ? ((px - ax) * dx + (py - ay) * dy) / lengthSquared : 0.0f; + t = std::clamp(t, 0.0f, 1.0f); + const float ex = ax + t * dx - px; + const float ey = ay + t * dy - py; + return std::sqrt(ex * ex + ey * ey); +} + +/** Insertion sort for the few candidates of a lookup (std::sort on a 3-entry array upsets GCC's bounds check). */ +template +void smallSort(T* items, int count, Less less) { + for (int i = 1; i < count; i++) { + T item = items[i]; + int j = i - 1; + while (j >= 0 && less(item, items[j])) { + items[j + 1] = items[j]; + j--; + } + items[j + 1] = item; + } +} + +int32_t floorDiv(int32_t value, int32_t by) { + return value >= 0 ? value / by : -((-value + by - 1) / by); +} + +} // namespace + +float placeClassScale(PlaceClass placeClass) { + switch (placeClass) { + case PlaceClass::City: + return 3000.0f; + case PlaceClass::Town: + return 1500.0f; + case PlaceClass::Village: + return 800.0f; + case PlaceClass::Hamlet: + return 300.0f; + case PlaceClass::IsolatedDwelling: + return 150.0f; + case PlaceClass::Suburb: + return 700.0f; + case PlaceClass::Quarter: + return 500.0f; + case PlaceClass::Neighbourhood: + return 400.0f; + default: + return 1.0f; + } +} + +void AddressMatcher::begin(double unitMetres) { + metresPerUnit = unitMetres; + // Cells of about the search radius: a lookup visits 3 x 3 cells + segmentCell = std::max(16, static_cast(LOW_STREET_M / metresPerUnit)); + houseCell = std::max(16, static_cast(VOTE_M / metresPerUnit)); + placeCell = std::max(16, static_cast(PLACE_M / metresPerUnit)); + clipping = false; + segments.clear(); + houses.clear(); + places.clear(); + segmentGrid.clear(); + houseGrid.clear(); + placeGrid.clear(); + results.clear(); +} + +void AddressMatcher::clip(int32_t x0, int32_t y0, int32_t x1, int32_t y1) { + clipping = true; + clipX0 = x0; + clipY0 = y0; + clipX1 = x1; + clipY1 = y1; +} + +void AddressMatcher::clipToTile(int32_t tileX, int32_t tileY) { + // The vote's reach plus the search radius, 20 m and two units to spare + const auto margin = static_cast((VOTE_M + LOW_STREET_M + 20.0) / metresPerUnit) + 2; + clip(tileX * 4096 - margin, tileY * 4096 - margin, tileX * 4096 + 4095 + margin, tileY * 4096 + 4095 + margin); +} + +void AddressMatcher::release() { + segments.release(); + houses.release(); + places.release(); + segmentGrid.release(); + houseGrid.release(); + placeGrid.release(); + results.release(); + candidates.release(); +} + +uint32_t AddressMatcher::cellOf(int32_t x, int32_t y, int32_t cellUnits) const { + // Cells far apart may share a key: lookups measure every item they find, so that only costs time + const auto cx = static_cast(floorDiv(x, cellUnits)) & 0xFFFF; + const auto cy = static_cast(floorDiv(y, cellUnits)) & 0xFFFF; + return cx << 16 | cy; +} + +void AddressMatcher::addLine(uint32_t name, const int32_t* xy, size_t pointCount) { + for (size_t i = 1; i < pointCount; i++) { + const Segment segment { xy[2 * i - 2], xy[2 * i - 1], xy[2 * i], xy[2 * i + 1], name }; + if (clipping && (std::max(segment.ax, segment.bx) < clipX0 || std::min(segment.ax, segment.bx) > clipX1 || + std::max(segment.ay, segment.by) < clipY0 || std::min(segment.ay, segment.by) > clipY1)) { + continue; + } + const auto index = static_cast(segments.size()); + segments.push(segment); + // In every cell its box touches (segments are short in vector tiles; long ones cover a few cells) + const int32_t x0 = floorDiv(std::min(segment.ax, segment.bx), segmentCell); + const int32_t x1 = floorDiv(std::max(segment.ax, segment.bx), segmentCell); + const int32_t y0 = floorDiv(std::min(segment.ay, segment.by), segmentCell); + const int32_t y1 = floorDiv(std::max(segment.ay, segment.by), segmentCell); + for (int32_t cx = x0; cx <= x1 && cx - x0 < 64; cx++) { + for (int32_t cy = y0; cy <= y1 && cy - y0 < 64; cy++) { + segmentGrid.push({ cellOf(cx * segmentCell, cy * segmentCell, segmentCell), index }); + } + } + } +} + +void AddressMatcher::addPlace(uint32_t name, PlaceClass placeClass, int32_t x, int32_t y) { + placeGrid.push({ cellOf(x, y, placeCell), static_cast(places.size()) }); + places.push({ x, y, name, placeClass }); +} + +uint32_t AddressMatcher::addHouse(int32_t x, int32_t y, int32_t number, bool villageStyle) { + const auto index = static_cast(houses.size()); + houseGrid.push({ cellOf(x, y, houseCell), index }); + houses.push({ x, y, number, villageStyle }); + return index; +} + +template +void AddressMatcher::visitCells(const GrowArray& grid, int32_t cellUnits, int32_t x, int32_t y, int32_t radiusUnits, Visit visit) const { + const int32_t x0 = floorDiv(x - radiusUnits, cellUnits); + const int32_t x1 = floorDiv(x + radiusUnits, cellUnits); + const int32_t y0 = floorDiv(y - radiusUnits, cellUnits); + const int32_t y1 = floorDiv(y + radiusUnits, cellUnits); + for (int32_t cx = x0; cx <= x1; cx++) { + for (int32_t cy = y0; cy <= y1; cy++) { + const uint32_t key = cellOf(cx * cellUnits, cy * cellUnits, cellUnits); + const CellItem probe { key, 0 }; + const CellItem* first = + std::lower_bound(grid.begin(), grid.end(), probe, [](const CellItem& a, const CellItem& b) { return a.cell < b.cell; }); + for (const CellItem* item = first; item != grid.end() && item->cell == key; item++) { + visit(item->item); + } + } + } +} + +void AddressMatcher::nearestStreets(const House& house, Candidate* out, int& count) const { + count = 0; + const auto radius = static_cast(LOW_STREET_M / metresPerUnit) + 1; + const auto px = static_cast(house.x); + const auto py = static_cast(house.y); + visitCells(segmentGrid, segmentCell, house.x, house.y, radius, [&](uint32_t index) { + const Segment& s = segments[index]; + const float metres = segmentDistance(px, py, static_cast(s.ax), static_cast(s.ay), static_cast(s.bx), + static_cast(s.by)) * + static_cast(metresPerUnit); + if (metres > LOW_STREET_M) { + return; + } + // One entry per street name, its nearest distance, sorted + for (int i = 0; i < count; i++) { + if (out[i].name == s.name) { + if (metres < out[i].metres) { + out[i].metres = metres; + smallSort(out, count, [](const Candidate& a, const Candidate& b) { return a.metres < b.metres; }); + } + return; + } + } + if (count < CANDIDATES) { + out[count++] = { s.name, metres }; + } else if (metres < out[count - 1].metres) { + out[count - 1] = { s.name, metres }; + } else { + return; + } + smallSort(out, count, [](const Candidate& a, const Candidate& b) { return a.metres < b.metres; }); + }); +} + +void AddressMatcher::villagesFor(const House& house, uint32_t* out) const { + struct Scored { + float score; + uint32_t name; + }; + Scored best[3]; + int count = 0; + const auto radius = static_cast(PLACE_M / metresPerUnit); + visitCells(placeGrid, placeCell, house.x, house.y, radius, [&](uint32_t index) { + const Place& place = places[index]; + const auto dx = static_cast(place.x - house.x); + const auto dy = static_cast(place.y - house.y); + const float metres = std::sqrt(dx * dx + dy * dy) * static_cast(metresPerUnit); + if (metres > PLACE_M) { + return; + } + const float score = metres / placeClassScale(place.placeClass); + // Distinct names, the best score of each + for (int i = 0; i < count; i++) { + if (best[i].name == place.name) { + best[i].score = std::min(best[i].score, score); + smallSort(best, count, [](const Scored& a, const Scored& b) { return a.score < b.score; }); + return; + } + } + if (count < 3) { + best[count++] = { score, place.name }; + } else if (score < best[2].score) { + best[2] = { score, place.name }; + } else { + return; + } + smallSort(best, count, [](const Scored& a, const Scored& b) { return a.score < b.score; }); + }); + for (int i = 0; i < 3; i++) { + out[i] = i < count ? best[i].name : NONE; + } +} + +bool AddressMatcher::run() { + auto byCell = [](const CellItem& a, const CellItem& b) { return a.cell < b.cell || (a.cell == b.cell && a.item < b.item); }; + std::sort(segmentGrid.begin(), segmentGrid.end(), byCell); + std::sort(houseGrid.begin(), houseGrid.end(), byCell); + std::sort(placeGrid.begin(), placeGrid.end(), byCell); + const size_t count = houses.size(); + if (!results.resize(count) || !candidates.resize(count * CANDIDATES) || segments.failed() || segmentGrid.failed() || houses.failed() || + houseGrid.failed() || places.failed() || placeGrid.failed()) { + return false; + } + // 1. The nearest street of each house (the vote needs every neighbour's first) + for (size_t i = 0; i < count; i++) { + int found = 0; + Candidate* own = &candidates[i * CANDIDATES]; + nearestStreets(houses[i], own, found); + for (int k = found; k < CANDIDATES; k++) { + own[k] = { NONE, INFINITY }; + } + AddressMatch& match = results[i]; + match.street = found > 0 ? own[0].name : NONE; + match.streetM = found > 0 ? own[0].metres : INFINITY; + match.tier = found == 0 ? AddressTier::None : (own[0].metres <= STREET_M ? AddressTier::Street : AddressTier::LowStreet); + match.villages[0] = match.villages[1] = match.villages[2] = NONE; + } + // 2. The street-run vote, where a second street is nearly as near + const auto voteUnits = static_cast(VOTE_M / metresPerUnit) + 1; + const float voteSquared = static_cast(VOTE_M / metresPerUnit) * static_cast(VOTE_M / metresPerUnit); + for (size_t i = 0; i < count; i++) { + const House& house = houses[i]; + const Candidate* own = &candidates[i * CANDIDATES]; + if (house.number < 0 || own[0].name == NONE || own[1].name == NONE || own[1].metres > own[0].metres + VOTE_SLACK_M) { + continue; + } + int votes[CANDIDATES] = {}; + visitCells(houseGrid, houseCell, house.x, house.y, voteUnits, [&](uint32_t j) { + if (j == i) { + return; + } + const House& other = houses[j]; + if (other.number < 0 || (other.number & 1) != (house.number & 1) || std::abs(other.number - house.number) > 4) { + return; + } + const auto dx = static_cast(other.x - house.x); + const auto dy = static_cast(other.y - house.y); + if (dx * dx + dy * dy > voteSquared) { + return; + } + // A neighbour votes for its own nearest street + const uint32_t street = candidates[j * CANDIDATES].name; + for (int k = 0; k < CANDIDATES; k++) { + if (own[k].name == street) { + votes[k]++; + } + } + }); + int best = 0; + for (int k = 1; k < CANDIDATES; k++) { + if (own[k].name != NONE && own[k].metres <= own[0].metres + VOTE_SLACK_M && own[k].metres <= LOW_STREET_M && votes[k] > votes[best]) { + best = k; + } + } + if (best != 0) { + AddressMatch& match = results[i]; + match.street = own[best].name; + match.streetM = own[best].metres; + match.tier = own[best].metres <= STREET_M ? AddressTier::Street : AddressTier::LowStreet; + } + } + // 3. Settlements for village-style numbers + for (size_t i = 0; i < count; i++) { + if (houses[i].villageStyle) { + villagesFor(houses[i], results[i].villages); + } + } + return true; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/AddressMatch.h b/Apps/Maps/main/Source/search/AddressMatch.h new file mode 100644 index 0000000..70a3bc5 --- /dev/null +++ b/Apps/Maps/main/Source/search/AddressMatch.h @@ -0,0 +1,123 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "../route/GrowArray.h" + +#include +#include + +namespace maps { + +/** Settlement classes for the village rule; locality is never one (M6 study, gt_village.py). */ +enum class PlaceClass : uint8_t { City, Town, Village, Hamlet, IsolatedDwelling, Suburb, Quarter, Neighbourhood, Count }; + +/** The scale of a place class in metres: a village 800 m away counts as much as a hamlet 300 m away. */ +float placeClassScale(PlaceClass placeClass); + +/** How a house number found its place in the index. */ +enum class AddressTier : uint8_t { None, Street, LowStreet }; + +/** The result for one house number. */ +struct AddressMatch { + /** The street's name id (the caller's), UINT32_MAX for none. */ + uint32_t street; + float streetM; + AddressTier tier; + /** Up to three settlements, best first (name ids, UINT32_MAX for none), for village-style numbers. */ + uint32_t villages[3]; +}; + +/** + * Matches house numbers to streets and settlements (M6, the study's measured rules, docs/M6-search-study.md): + * - the nearest named street line (point to segment, in metres) within STREET_M is the street, within LOW_STREET_M a + * low-confidence one, never farther; + * - street-run vote: when a second street is within the nearest + VOTE_SLACK_M, neighbours within VOTE_M with the same + * parity and numbers at most 4 apart vote for their street; the street changes only with strictly more votes; + * - village style numbers get the best three settlements within 3 km by distance divided by the class scale. + * Coordinates are global z14 units (4096 per tile). One matcher works on a tile and its buffer, then is cleared; its + * arrays keep their memory for the next tile. + */ +class AddressMatcher { +public: + static constexpr float STREET_M = 50.0f; + static constexpr float LOW_STREET_M = 100.0f; + static constexpr float VOTE_M = 60.0f; + static constexpr float VOTE_SLACK_M = 12.0f; + static constexpr float PLACE_M = 3000.0f; + + /** Starts a tile; metres per unit at its latitude. */ + void begin(double metresPerUnit); + /** + * Leaves out the segments that lie wholly outside this box (global units) until the next begin(). For one tile: + * its houses and the neighbours' that vote lie within VOTE_M of it and look LOW_STREET_M around them, so segments + * farther out can never be a candidate; leaving them out saves memory and changes no result. + */ + void clip(int32_t x0, int32_t y0, int32_t x1, int32_t y1); + /** Clips to a tile (x, y at z14) and the reach of its houses at this latitude, with a margin. */ + void clipToTile(int32_t tileX, int32_t tileY); + /** A named street line (its points, global units). */ + void addLine(uint32_t name, const int32_t* xy, size_t pointCount); + void addPlace(uint32_t name, PlaceClass placeClass, int32_t x, int32_t y); + /** A house number: its street number for the vote (-1: none), and whether it looks village style (plain). */ + uint32_t addHouse(int32_t x, int32_t y, int32_t number, bool villageStyle); + /** Works out every house's match. False when memory ran out. */ + bool run(); + const AddressMatch& result(uint32_t house) const { return results[house]; } + size_t houseCount() const { return houses.size(); } + /** Frees the memory. */ + void release(); + /** The memory its arrays hold. */ + size_t allocated() const { + return segments.allocated() + houses.allocated() + places.allocated() + segmentGrid.allocated() + houseGrid.allocated() + + placeGrid.allocated() + results.allocated() + candidates.allocated(); + } + +private: + struct Segment { + int32_t ax, ay, bx, by; + uint32_t name; + }; + struct CellItem { + uint32_t cell; + uint32_t item; + }; + struct House { + int32_t x, y; + int32_t number; + bool villageStyle; + }; + struct Place { + int32_t x, y; + uint32_t name; + PlaceClass placeClass; + }; + /** The nearest distinct streets of a house, nearest first. */ + struct Candidate { + uint32_t name; + float metres; + }; + static constexpr int CANDIDATES = 4; + + uint32_t cellOf(int32_t x, int32_t y, int32_t cellUnits) const; + void nearestStreets(const House& house, Candidate* out, int& count) const; + void villagesFor(const House& house, uint32_t* out) const; + template + void visitCells(const GrowArray& grid, int32_t cellUnits, int32_t x, int32_t y, int32_t radiusUnits, Visit visit) const; + + double metresPerUnit = 1.0; + bool clipping = false; + int32_t clipX0 = 0, clipY0 = 0, clipX1 = 0, clipY1 = 0; + int32_t segmentCell = 256; + int32_t houseCell = 128; + int32_t placeCell = 8192; + GrowArray segments; + GrowArray houses; + GrowArray places; + GrowArray segmentGrid; + GrowArray houseGrid; + GrowArray placeGrid; + GrowArray results; + GrowArray candidates; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/ExternalSort.cpp b/Apps/Maps/main/Source/search/ExternalSort.cpp new file mode 100644 index 0000000..2290126 --- /dev/null +++ b/Apps/Maps/main/Source/search/ExternalSort.cpp @@ -0,0 +1,537 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "ExternalSort.h" + +#include "../platform/Platform.h" + +#include +#include +#include +#include + +namespace maps { + +namespace { + +constexpr uint32_t OUT_BUFFER = 4096; +constexpr uint32_t MIN_BUFFER = 16 * 1024; + +uint32_t get16(const uint8_t* p) { + return static_cast(p[0]) | static_cast(p[1]) << 8; +} + +void put16(uint8_t* p, uint32_t v) { + p[0] = static_cast(v); + p[1] = static_cast(v >> 8); +} + +int compareDefault(const uint8_t* a, uint32_t aSize, const uint8_t* b, uint32_t bSize, void*) { + const uint32_t common = aSize < bSize ? aSize : bSize; + const int c = common > 0 ? memcmp(a, b, common) : 0; + if (c != 0) { + return c; + } + return aSize < bSize ? -1 : (aSize > bSize ? 1 : 0); +} + +} // namespace + +/** A run in a run file: where it starts and how many bytes it has. */ +struct RunInfo { + uint32_t offset; + uint32_t size; +}; + +/** One run being read in a merge: a window of the file in its own buffer, at least one whole record in it. */ +struct RunReader { + uint8_t* buffer; + uint32_t filled; + uint32_t at; + /** File position of the next byte not yet in the buffer, and the run's end. */ + uint32_t next; + uint32_t end; + bool done; +}; + +struct ExternalSorter::Impl { + ScratchFiles* files = nullptr; + char runPath[136] = {}; + char mergePath[136] = {}; + Compare compare = nullptr; + void* context = nullptr; + uint32_t bufferBytes = 0; + // Filling + uint8_t* buffer = nullptr; + uint32_t used = 0; + uint32_t count = 0; + // Runs in the run file, and the merge + RunInfo* runs = nullptr; + uint32_t runCapacity = 0; + uint32_t runTotal = 0; + uint32_t runFileSize = 0; + bool runFileExists = false; + bool mergeFileExists = false; + RunReader readers[MAX_FAN_IN] = {}; + uint8_t* readMemory = nullptr; + uint32_t readerCount = 0; + int readHandle = -1; + int lastReader = -1; + // Memory mode: served from the sorted buffer + bool memoryMode = false; + uint32_t memoryAt = 0; + bool finished = false; + uint8_t out[OUT_BUFFER] = {}; + uint32_t outUsed = 0; + size_t held = 0; +}; + +ExternalSorter::~ExternalSorter() { + end(); +} + +namespace { + +void setError(char* error, size_t size, const char* format, ...) { + if (error[0] != '\0') { + return; + } + va_list args; + va_start(args, format); + vsnprintf(error, size, format, args); + va_end(args); +} + +} // namespace + +bool ExternalSorter::begin(ScratchFiles* files, const char* prefix, uint32_t bufferBytes, Compare compare, void* context) { + end(); + error[0] = '\0'; + records = 0; + runs = 0; + written = 0; + peak = 0; + if (files == nullptr || prefix == nullptr || bufferBytes < MIN_BUFFER) { + setError(error, sizeof(error), "bad sort setup"); + return false; + } + // The record offsets sit at the buffer's end as uint32_t: keep them aligned + bufferBytes &= ~3u; + void* memory = largeAlloc(sizeof(Impl)); + if (memory == nullptr) { + setError(error, sizeof(error), "no memory for a sort"); + return false; + } + impl = new (memory) Impl(); + impl->files = files; + snprintf(impl->runPath, sizeof(impl->runPath), "%s.run", prefix); + snprintf(impl->mergePath, sizeof(impl->mergePath), "%s.mrg", prefix); + impl->compare = compare != nullptr ? compare : compareDefault; + impl->context = context; + impl->bufferBytes = bufferBytes; + impl->buffer = static_cast(largeAlloc(bufferBytes)); + if (impl->buffer == nullptr) { + setError(error, sizeof(error), "no memory for the sort buffer (%u bytes)", static_cast(bufferBytes)); + end(); + return false; + } + impl->held = sizeof(Impl) + bufferBytes; + peak = impl->held; + return true; +} + +namespace { + +/** The offsets of the records sit at the end of the buffer, the merge sort's second array before them. */ +uint32_t* offsetsOf(uint8_t* buffer, uint32_t bufferBytes, uint32_t count) { + return reinterpret_cast(buffer + bufferBytes) - count; +} + +/** A stable bottom-up merge sort of record offsets. */ +void sortOffsets(const uint8_t* records, uint32_t* a, uint32_t* b, uint32_t count, ExternalSorter::Compare compare, void* context) { + uint32_t* from = a; + uint32_t* to = b; + for (uint32_t width = 1; width < count; width *= 2) { + for (uint32_t left = 0; left < count; left += 2 * width) { + const uint32_t middle = left + width < count ? left + width : count; + const uint32_t right = left + 2 * width < count ? left + 2 * width : count; + uint32_t i = left; + uint32_t j = middle; + uint32_t k = left; + while (i < middle && j < right) { + const uint8_t* x = records + from[i]; + const uint8_t* y = records + from[j]; + // Take the right one only when strictly smaller: equal records keep their order + if (compare(y + 2, get16(y), x + 2, get16(x), context) < 0) { + to[k++] = from[j++]; + } else { + to[k++] = from[i++]; + } + } + while (i < middle) { + to[k++] = from[i++]; + } + while (j < right) { + to[k++] = from[j++]; + } + } + uint32_t* swap = from; + from = to; + to = swap; + } + if (from != a) { + memcpy(a, from, count * sizeof(uint32_t)); + } +} + +} // namespace + +/** + * Sorts the buffer's records (their offsets). The offsets were stored from the end of the buffer downwards, newest + * first: they are put in add() order first, so the stable sort keeps equal records in that order. The merge sort's + * second array lies just before the offsets. + */ +static void sortBuffer(uint8_t* buffer, uint32_t bufferBytes, uint32_t count, ExternalSorter::Compare compare, void* context) { + uint32_t* offsets = offsetsOf(buffer, bufferBytes, count); + for (uint32_t i = 0, j = count; i + 1 < j; i++, j--) { + const uint32_t swap = offsets[i]; + offsets[i] = offsets[j - 1]; + offsets[j - 1] = swap; + } + uint32_t* temp = offsets - count; + sortOffsets(buffer, offsets, temp, count, compare, context); +} + +bool ExternalSorter::writeRun() { + sortBuffer(impl->buffer, impl->bufferBytes, impl->count, impl->compare, impl->context); + const uint32_t* offsets = offsetsOf(impl->buffer, impl->bufferBytes, impl->count); + if (impl->runTotal == impl->runCapacity) { + const uint32_t capacity = impl->runCapacity < 16 ? 16 : impl->runCapacity * 2; + auto* grown = static_cast(largeAlloc(capacity * sizeof(RunInfo))); + if (grown == nullptr) { + setError(error, sizeof(error), "no memory for the run list"); + return false; + } + if (impl->runs != nullptr) { + memcpy(grown, impl->runs, impl->runTotal * sizeof(RunInfo)); + largeFree(impl->runs); + } + impl->held += (capacity - impl->runCapacity) * sizeof(RunInfo); + peak = impl->held > peak ? impl->held : peak; + impl->runs = grown; + impl->runCapacity = capacity; + } + const int handle = impl->runFileExists ? impl->files->openAppend(impl->runPath) : impl->files->create(impl->runPath); + if (handle < 0) { + setError(error, sizeof(error), "cannot open the run file"); + return false; + } + impl->runFileExists = true; + bool ok = true; + uint32_t runBytes = 0; + impl->outUsed = 0; + for (uint32_t i = 0; i < impl->count && ok; i++) { + const uint8_t* r = impl->buffer + offsets[i]; + const uint32_t total = 2 + get16(r); + if (impl->outUsed + total > OUT_BUFFER) { + ok = impl->files->write(handle, impl->out, impl->outUsed); + impl->outUsed = 0; + } + memcpy(impl->out + impl->outUsed, r, total); + impl->outUsed += total; + runBytes += total; + } + if (ok && impl->outUsed > 0) { + ok = impl->files->write(handle, impl->out, impl->outUsed); + } + ok = impl->files->close(handle) && ok; + if (!ok) { + setError(error, sizeof(error), "cannot write run %u", static_cast(impl->runTotal)); + return false; + } + impl->runs[impl->runTotal++] = { impl->runFileSize, runBytes }; + impl->runFileSize += runBytes; + written += runBytes; + runs++; + impl->used = 0; + impl->count = 0; + return true; +} + +bool ExternalSorter::add(const void* record, uint32_t size) { + if (impl == nullptr || impl->finished || failed()) { + return false; + } + if (size == 0 || size > MAX_RECORD) { + setError(error, sizeof(error), "a sort record of %u bytes", static_cast(size)); + return false; + } + // Per record: its length, its bytes, its offset and the merge sort's second offset + if (impl->used + 2 + size + (impl->count + 1) * 8 > impl->bufferBytes) { + if (impl->count == 0) { + setError(error, sizeof(error), "the sort buffer is too small"); + return false; + } + if (!writeRun()) { + return false; + } + } + uint8_t* at = impl->buffer + impl->used; + put16(at, size); + memcpy(at + 2, record, size); + impl->count++; + offsetsOf(impl->buffer, impl->bufferBytes, impl->count)[0] = impl->used; + impl->used += 2 + size; + records++; + return true; +} + +namespace { + +/** Reads more of a run into its buffer, keeping the unread bytes. False on a read error. */ +bool refill(ScratchFiles* files, int handle, RunReader& r) { + if (r.at > 0) { + memmove(r.buffer, r.buffer + r.at, r.filled - r.at); + r.filled -= r.at; + r.at = 0; + } + uint32_t want = ExternalSorter::READ_BUFFER - r.filled; + if (want > r.end - r.next) { + want = r.end - r.next; + } + if (want == 0) { + return true; + } + if (!files->seek(handle, r.next)) { + return false; + } + uint32_t got = 0; + while (got < want) { + const int n = files->read(handle, r.buffer + r.filled + got, want - got); + if (n <= 0) { + return false; + } + got += static_cast(n); + } + r.filled += got; + r.next += got; + return true; +} + +/** Makes sure the reader holds a whole record at r.at, or marks it done. False on a read error or a broken run. */ +bool ready(ScratchFiles* files, int handle, RunReader& r) { + if (r.done) { + return true; + } + if (r.filled - r.at < 2 || r.filled - r.at < 2 + get16(r.buffer + r.at)) { + if (!refill(files, handle, r)) { + return false; + } + } + if (r.filled - r.at == 0) { + r.done = true; + return true; + } + return r.filled - r.at >= 2 && r.filled - r.at >= 2 + get16(r.buffer + r.at); +} + +/** The reader with the smallest record (the lowest run on a tie, which keeps the sort stable); -1 when all are done. */ +int smallest(RunReader* readers, uint32_t count, ExternalSorter::Compare compare, void* context) { + int best = -1; + for (uint32_t i = 0; i < count; i++) { + const RunReader& r = readers[i]; + if (r.done) { + continue; + } + if (best < 0) { + best = static_cast(i); + continue; + } + const RunReader& b = readers[best]; + const uint8_t* x = r.buffer + r.at; + const uint8_t* y = b.buffer + b.at; + if (compare(x + 2, get16(x), y + 2, get16(y), context) < 0) { + best = static_cast(i); + } + } + return best; +} + +} // namespace + +bool ExternalSorter::finish() { + if (impl == nullptr || impl->finished || failed()) { + return false; + } + impl->finished = true; + if (impl->runTotal == 0) { + sortBuffer(impl->buffer, impl->bufferBytes, impl->count, impl->compare, impl->context); + impl->memoryMode = true; + impl->memoryAt = 0; + return true; + } + if (impl->count > 0 && !writeRun()) { + return false; + } + // The record buffer is not needed any more: the merge has its own buffers + largeFree(impl->buffer); + impl->buffer = nullptr; + impl->held -= impl->bufferBytes; + impl->readMemory = static_cast(largeAlloc(MAX_FAN_IN * READ_BUFFER)); + if (impl->readMemory == nullptr) { + setError(error, sizeof(error), "no memory for the merge buffers"); + return false; + } + impl->held += MAX_FAN_IN * READ_BUFFER; + peak = impl->held > peak ? impl->held : peak; + for (uint32_t i = 0; i < MAX_FAN_IN; i++) { + impl->readers[i].buffer = impl->readMemory + i * READ_BUFFER; + } + // Merge passes until at most MAX_FAN_IN runs are left: groups of consecutive runs into the other file + while (impl->runTotal > MAX_FAN_IN) { + const int input = impl->files->openRead(impl->runPath); + const int output = input >= 0 ? impl->files->create(impl->mergePath) : -1; + if (input < 0 || output < 0) { + if (input >= 0) { + impl->files->close(input); + } + setError(error, sizeof(error), "cannot open the files of a merge pass"); + return false; + } + impl->mergeFileExists = true; + uint32_t newTotal = 0; + uint32_t outputSize = 0; + bool ok = true; + for (uint32_t first = 0; first < impl->runTotal && ok; first += MAX_FAN_IN) { + const uint32_t group = impl->runTotal - first < MAX_FAN_IN ? impl->runTotal - first : MAX_FAN_IN; + for (uint32_t i = 0; i < group; i++) { + RunReader& r = impl->readers[i]; + r.filled = 0; + r.at = 0; + r.next = impl->runs[first + i].offset; + r.end = impl->runs[first + i].offset + impl->runs[first + i].size; + r.done = false; + ok = ok && ready(impl->files, input, r); + } + const uint32_t start = outputSize; + impl->outUsed = 0; + while (ok) { + const int best = smallest(impl->readers, group, impl->compare, impl->context); + if (best < 0) { + break; + } + RunReader& r = impl->readers[best]; + const uint32_t total = 2 + get16(r.buffer + r.at); + if (impl->outUsed + total > OUT_BUFFER) { + ok = impl->files->write(output, impl->out, impl->outUsed); + impl->outUsed = 0; + } + memcpy(impl->out + impl->outUsed, r.buffer + r.at, total); + impl->outUsed += total; + outputSize += total; + r.at += total; + ok = ok && ready(impl->files, input, r); + } + if (ok && impl->outUsed > 0) { + ok = impl->files->write(output, impl->out, impl->outUsed); + impl->outUsed = 0; + } + // Earlier entries are already read: the merged runs can take their places + impl->runs[newTotal++] = { start, outputSize - start }; + runs++; + } + impl->files->close(input); + ok = impl->files->close(output) && ok; + if (!ok) { + setError(error, sizeof(error), "a merge pass failed"); + return false; + } + written += outputSize; + impl->runTotal = newTotal; + impl->runFileSize = outputSize; + // The merged file is the run file now + if (!impl->files->rename(impl->mergePath, impl->runPath)) { + setError(error, sizeof(error), "cannot rename the merged runs"); + return false; + } + impl->mergeFileExists = false; + } + // The final merge is done by next() + impl->readHandle = impl->files->openRead(impl->runPath); + if (impl->readHandle < 0) { + setError(error, sizeof(error), "cannot open the run file"); + return false; + } + impl->readerCount = impl->runTotal; + for (uint32_t i = 0; i < impl->readerCount; i++) { + RunReader& r = impl->readers[i]; + r.filled = 0; + r.at = 0; + r.next = impl->runs[i].offset; + r.end = impl->runs[i].offset + impl->runs[i].size; + r.done = false; + if (!ready(impl->files, impl->readHandle, r)) { + setError(error, sizeof(error), "cannot read run %u", static_cast(i)); + return false; + } + } + impl->lastReader = -1; + return true; +} + +bool ExternalSorter::next(const uint8_t*& data, uint32_t& size) { + if (impl == nullptr || !impl->finished || failed()) { + return false; + } + if (impl->memoryMode) { + if (impl->memoryAt >= impl->count) { + return false; + } + const uint32_t* offsets = offsetsOf(impl->buffer, impl->bufferBytes, impl->count); + const uint8_t* r = impl->buffer + offsets[impl->memoryAt++]; + size = get16(r); + data = r + 2; + return true; + } + // Step past the record returned last time (its bytes stayed valid until now) + if (impl->lastReader >= 0) { + RunReader& r = impl->readers[impl->lastReader]; + r.at += 2 + get16(r.buffer + r.at); + if (!ready(impl->files, impl->readHandle, r)) { + setError(error, sizeof(error), "cannot read run %d", impl->lastReader); + return false; + } + impl->lastReader = -1; + } + const int best = smallest(impl->readers, impl->readerCount, impl->compare, impl->context); + if (best < 0) { + return false; + } + const RunReader& r = impl->readers[best]; + size = get16(r.buffer + r.at); + data = r.buffer + r.at + 2; + impl->lastReader = best; + return true; +} + +void ExternalSorter::end() { + if (impl == nullptr) { + return; + } + if (impl->readHandle >= 0) { + impl->files->close(impl->readHandle); + impl->readHandle = -1; + } + if (impl->runFileExists) { + impl->files->remove(impl->runPath); + } + if (impl->mergeFileExists) { + impl->files->remove(impl->mergePath); + } + largeFree(impl->readMemory); + largeFree(impl->buffer); + largeFree(impl->runs); + impl->~Impl(); + largeFree(impl); + impl = nullptr; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/ExternalSort.h b/Apps/Maps/main/Source/search/ExternalSort.h new file mode 100644 index 0000000..c406441 --- /dev/null +++ b/Apps/Maps/main/Source/search/ExternalSort.h @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "ScratchFiles.h" + +#include +#include + +namespace maps { + +/** + * Sorts a stream of variable-size records with a fixed amount of memory (M6 streaming index build): add() fills a + * buffer; a full buffer is sorted and appended to the run file as the next run; finish() sorts the rest and merges the + * runs; next() then returns the records in order. All runs of a sort are in one file (.run, and .mrg + * for merge passes), so a merge reads through a single handle: the card's FAT mount allows only four open files for + * the whole system. Records that compare equal keep the order of add() (the sort is stable). When all records fit the + * buffer no file is written. Platform free: memory through largeAlloc (PSRAM on the device), files through + * ScratchFiles. + */ +class ExternalSorter { +public: + /** The largest record add() takes. */ + static constexpr uint32_t MAX_RECORD = 2048; + /** At most this many runs are merged at once; more runs are first merged into longer runs. */ + static constexpr uint32_t MAX_FAN_IN = 16; + /** The read buffer of one run while merging (holds at least one whole record). */ + static constexpr uint32_t READ_BUFFER = 4096; + + /** <0, 0, >0 like memcmp; context is the one given to begin(). */ + using Compare = int (*)(const uint8_t* a, uint32_t aSize, const uint8_t* b, uint32_t bSize, void* context); + + ExternalSorter() = default; + ~ExternalSorter(); + ExternalSorter(const ExternalSorter&) = delete; + ExternalSorter& operator=(const ExternalSorter&) = delete; + + /** + * Starts a sort. The run files are .run and .mrg (prefix is a full path such as + * "/sdcard/maps/x/srch/k"). bufferBytes (at least 16 KB) holds the records of one run and their index (2 + size + * + 8 bytes per record); the merge needs MAX_FAN_IN * READ_BUFFER bytes instead (the buffer is freed first). + * compare == nullptr compares the bytes (memcmp, then the shorter first). + */ + bool begin(ScratchFiles* files, const char* prefix, uint32_t bufferBytes, Compare compare, void* context); + /** Adds a record (1 to MAX_RECORD bytes). Only before finish(). */ + bool add(const void* record, uint32_t size); + /** Sorts what is left and prepares the merge. Only once, after the last add(). */ + bool finish(); + /** The next record in order: data stays valid until the next call. False at the end or on an error (failed()). */ + bool next(const uint8_t*& data, uint32_t& size); + /** Closes and removes the run files and frees the memory. Safe to call at any time, also twice. */ + void end(); + + bool failed() const { return error[0] != '\0'; } + const char* errorText() const { return error; } + + /** For logs and tests. */ + uint64_t recordCount() const { return records; } + /** Runs made, merged runs included. */ + uint32_t runCount() const { return runs; } + /** Bytes written to run files, intermediate merges included. */ + uint64_t bytesWritten() const { return written; } + /** The most memory this sorter held at once. */ + size_t peakBytes() const { return peak; } + +private: + bool writeRun(); + struct Impl; + Impl* impl = nullptr; + uint64_t records = 0; + uint32_t runs = 0; + uint64_t written = 0; + size_t peak = 0; + char error[96] = {}; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/FoldTable.h b/Apps/Maps/main/Source/search/FoldTable.h new file mode 100644 index 0000000..f826ce2 --- /dev/null +++ b/Apps/Maps/main/Source/search/FoldTable.h @@ -0,0 +1,4251 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// Made by tools/make_fold_table.py from fold.py and Python's unicodedata 13.0.0; don't edit. +#pragma once + +#include + +namespace maps { + +struct FoldEntry { + uint32_t codePoint; + const char* text; +}; + +constexpr FoldEntry FOLD_LATIN[] = { + { 0x0041, "\x61" }, + { 0x0042, "\x62" }, + { 0x0043, "\x63" }, + { 0x0044, "\x64" }, + { 0x0045, "\x65" }, + { 0x0046, "\x66" }, + { 0x0047, "\x67" }, + { 0x0048, "\x68" }, + { 0x0049, "\x69" }, + { 0x004A, "\x6A" }, + { 0x004B, "\x6B" }, + { 0x004C, "\x6C" }, + { 0x004D, "\x6D" }, + { 0x004E, "\x6E" }, + { 0x004F, "\x6F" }, + { 0x0050, "\x70" }, + { 0x0051, "\x71" }, + { 0x0052, "\x72" }, + { 0x0053, "\x73" }, + { 0x0054, "\x74" }, + { 0x0055, "\x75" }, + { 0x0056, "\x76" }, + { 0x0057, "\x77" }, + { 0x0058, "\x78" }, + { 0x0059, "\x79" }, + { 0x005A, "\x7A" }, + { 0x00C0, "\x61" }, + { 0x00C1, "\x61" }, + { 0x00C2, "\x61" }, + { 0x00C3, "\x61" }, + { 0x00C4, "\x61" }, + { 0x00C5, "\x61" }, + { 0x00C6, "\x61\x65" }, + { 0x00C7, "\x63" }, + { 0x00C8, "\x65" }, + { 0x00C9, "\x65" }, + { 0x00CA, "\x65" }, + { 0x00CB, "\x65" }, + { 0x00CC, "\x69" }, + { 0x00CD, "\x69" }, + { 0x00CE, "\x69" }, + { 0x00CF, "\x69" }, + { 0x00D0, "\x64" }, + { 0x00D1, "\x6E" }, + { 0x00D2, "\x6F" }, + { 0x00D3, "\x6F" }, + { 0x00D4, "\x6F" }, + { 0x00D5, "\x6F" }, + { 0x00D6, "\x6F" }, + { 0x00D8, "\x6F" }, + { 0x00D9, "\x75" }, + { 0x00DA, "\x75" }, + { 0x00DB, "\x75" }, + { 0x00DC, "\x75" }, + { 0x00DD, "\x79" }, + { 0x00DE, "\x74\x68" }, + { 0x00DF, "\x73\x73" }, + { 0x00E0, "\x61" }, + { 0x00E1, "\x61" }, + { 0x00E2, "\x61" }, + { 0x00E3, "\x61" }, + { 0x00E4, "\x61" }, + { 0x00E5, "\x61" }, + { 0x00E6, "\x61\x65" }, + { 0x00E7, "\x63" }, + { 0x00E8, "\x65" }, + { 0x00E9, "\x65" }, + { 0x00EA, "\x65" }, + { 0x00EB, "\x65" }, + { 0x00EC, "\x69" }, + { 0x00ED, "\x69" }, + { 0x00EE, "\x69" }, + { 0x00EF, "\x69" }, + { 0x00F0, "\x64" }, + { 0x00F1, "\x6E" }, + { 0x00F2, "\x6F" }, + { 0x00F3, "\x6F" }, + { 0x00F4, "\x6F" }, + { 0x00F5, "\x6F" }, + { 0x00F6, "\x6F" }, + { 0x00F8, "\x6F" }, + { 0x00F9, "\x75" }, + { 0x00FA, "\x75" }, + { 0x00FB, "\x75" }, + { 0x00FC, "\x75" }, + { 0x00FD, "\x79" }, + { 0x00FE, "\x74\x68" }, + { 0x00FF, "\x79" }, + { 0x0100, "\x61" }, + { 0x0101, "\x61" }, + { 0x0102, "\x61" }, + { 0x0103, "\x61" }, + { 0x0104, "\x61" }, + { 0x0105, "\x61" }, + { 0x0106, "\x63" }, + { 0x0107, "\x63" }, + { 0x0108, "\x63" }, + { 0x0109, "\x63" }, + { 0x010A, "\x63" }, + { 0x010B, "\x63" }, + { 0x010C, "\x63" }, + { 0x010D, "\x63" }, + { 0x010E, "\x64" }, + { 0x010F, "\x64" }, + { 0x0110, "\x64" }, + { 0x0111, "\x64" }, + { 0x0112, "\x65" }, + { 0x0113, "\x65" }, + { 0x0114, "\x65" }, + { 0x0115, "\x65" }, + { 0x0116, "\x65" }, + { 0x0117, "\x65" }, + { 0x0118, "\x65" }, + { 0x0119, "\x65" }, + { 0x011A, "\x65" }, + { 0x011B, "\x65" }, + { 0x011C, "\x67" }, + { 0x011D, "\x67" }, + { 0x011E, "\x67" }, + { 0x011F, "\x67" }, + { 0x0120, "\x67" }, + { 0x0121, "\x67" }, + { 0x0122, "\x67" }, + { 0x0123, "\x67" }, + { 0x0124, "\x68" }, + { 0x0125, "\x68" }, + { 0x0126, "\x68" }, + { 0x0127, "\x68" }, + { 0x0128, "\x69" }, + { 0x0129, "\x69" }, + { 0x012A, "\x69" }, + { 0x012B, "\x69" }, + { 0x012C, "\x69" }, + { 0x012D, "\x69" }, + { 0x012E, "\x69" }, + { 0x012F, "\x69" }, + { 0x0130, "\x69" }, + { 0x0131, "\x69" }, + { 0x0132, "\x69\x6A" }, + { 0x0133, "\x69\x6A" }, + { 0x0134, "\x6A" }, + { 0x0135, "\x6A" }, + { 0x0136, "\x6B" }, + { 0x0137, "\x6B" }, + { 0x0138, "\x6B" }, + { 0x0139, "\x6C" }, + { 0x013A, "\x6C" }, + { 0x013B, "\x6C" }, + { 0x013C, "\x6C" }, + { 0x013D, "\x6C" }, + { 0x013E, "\x6C" }, + { 0x013F, "\xC5\x80" }, + { 0x0141, "\x6C" }, + { 0x0142, "\x6C" }, + { 0x0143, "\x6E" }, + { 0x0144, "\x6E" }, + { 0x0145, "\x6E" }, + { 0x0146, "\x6E" }, + { 0x0147, "\x6E" }, + { 0x0148, "\x6E" }, + { 0x014A, "\x6E" }, + { 0x014B, "\x6E" }, + { 0x014C, "\x6F" }, + { 0x014D, "\x6F" }, + { 0x014E, "\x6F" }, + { 0x014F, "\x6F" }, + { 0x0150, "\x6F" }, + { 0x0151, "\x6F" }, + { 0x0152, "\x6F\x65" }, + { 0x0153, "\x6F\x65" }, + { 0x0154, "\x72" }, + { 0x0155, "\x72" }, + { 0x0156, "\x72" }, + { 0x0157, "\x72" }, + { 0x0158, "\x72" }, + { 0x0159, "\x72" }, + { 0x015A, "\x73" }, + { 0x015B, "\x73" }, + { 0x015C, "\x73" }, + { 0x015D, "\x73" }, + { 0x015E, "\x73" }, + { 0x015F, "\x73" }, + { 0x0160, "\x73" }, + { 0x0161, "\x73" }, + { 0x0162, "\x74" }, + { 0x0163, "\x74" }, + { 0x0164, "\x74" }, + { 0x0165, "\x74" }, + { 0x0166, "\x74" }, + { 0x0167, "\x74" }, + { 0x0168, "\x75" }, + { 0x0169, "\x75" }, + { 0x016A, "\x75" }, + { 0x016B, "\x75" }, + { 0x016C, "\x75" }, + { 0x016D, "\x75" }, + { 0x016E, "\x75" }, + { 0x016F, "\x75" }, + { 0x0170, "\x75" }, + { 0x0171, "\x75" }, + { 0x0172, "\x75" }, + { 0x0173, "\x75" }, + { 0x0174, "\x77" }, + { 0x0175, "\x77" }, + { 0x0176, "\x79" }, + { 0x0177, "\x79" }, + { 0x0178, "\x79" }, + { 0x0179, "\x7A" }, + { 0x017A, "\x7A" }, + { 0x017B, "\x7A" }, + { 0x017C, "\x7A" }, + { 0x017D, "\x7A" }, + { 0x017E, "\x7A" }, + { 0x017F, "\x73" }, + { 0x0181, "\xC9\x93" }, + { 0x0182, "\xC6\x83" }, + { 0x0184, "\xC6\x85" }, + { 0x0186, "\xC9\x94" }, + { 0x0187, "\xC6\x88" }, + { 0x0189, "\xC9\x96" }, + { 0x018A, "\xC9\x97" }, + { 0x018B, "\xC6\x8C" }, + { 0x018E, "\xC7\x9D" }, + { 0x018F, "\x65" }, + { 0x0190, "\xC9\x9B" }, + { 0x0191, "\xC6\x92" }, + { 0x0193, "\xC9\xA0" }, + { 0x0194, "\xC9\xA3" }, + { 0x0196, "\xC9\xA9" }, + { 0x0197, "\xC9\xA8" }, + { 0x0198, "\xC6\x99" }, + { 0x019C, "\xC9\xAF" }, + { 0x019D, "\xC9\xB2" }, + { 0x019F, "\xC9\xB5" }, + { 0x01A0, "\x6F" }, + { 0x01A1, "\x6F" }, + { 0x01A2, "\xC6\xA3" }, + { 0x01A4, "\xC6\xA5" }, + { 0x01A6, "\xCA\x80" }, + { 0x01A7, "\xC6\xA8" }, + { 0x01A9, "\xCA\x83" }, + { 0x01AC, "\xC6\xAD" }, + { 0x01AE, "\xCA\x88" }, + { 0x01AF, "\x75" }, + { 0x01B0, "\x75" }, + { 0x01B1, "\xCA\x8A" }, + { 0x01B2, "\xCA\x8B" }, + { 0x01B3, "\xC6\xB4" }, + { 0x01B5, "\xC6\xB6" }, + { 0x01B7, "\xCA\x92" }, + { 0x01B8, "\xC6\xB9" }, + { 0x01BC, "\xC6\xBD" }, + { 0x01C4, "\xC7\x86" }, + { 0x01C5, "\xC7\x86" }, + { 0x01C7, "\xC7\x89" }, + { 0x01C8, "\xC7\x89" }, + { 0x01CA, "\xC7\x8C" }, + { 0x01CB, "\xC7\x8C" }, + { 0x01CD, "\x61" }, + { 0x01CE, "\x61" }, + { 0x01CF, "\x69" }, + { 0x01D0, "\x69" }, + { 0x01D1, "\x6F" }, + { 0x01D2, "\x6F" }, + { 0x01D3, "\x75" }, + { 0x01D4, "\x75" }, + { 0x01D5, "\x75" }, + { 0x01D6, "\x75" }, + { 0x01D7, "\x75" }, + { 0x01D8, "\x75" }, + { 0x01D9, "\x75" }, + { 0x01DA, "\x75" }, + { 0x01DB, "\x75" }, + { 0x01DC, "\x75" }, + { 0x01DE, "\x61" }, + { 0x01DF, "\x61" }, + { 0x01E0, "\x61" }, + { 0x01E1, "\x61" }, + { 0x01E2, "\xC3\xA6" }, + { 0x01E3, "\xC3\xA6" }, + { 0x01E4, "\xC7\xA5" }, + { 0x01E6, "\x67" }, + { 0x01E7, "\x67" }, + { 0x01E8, "\x6B" }, + { 0x01E9, "\x6B" }, + { 0x01EA, "\x6F" }, + { 0x01EB, "\x6F" }, + { 0x01EC, "\x6F" }, + { 0x01ED, "\x6F" }, + { 0x01EE, "\xCA\x92" }, + { 0x01EF, "\xCA\x92" }, + { 0x01F0, "\x6A" }, + { 0x01F1, "\xC7\xB3" }, + { 0x01F2, "\xC7\xB3" }, + { 0x01F4, "\x67" }, + { 0x01F5, "\x67" }, + { 0x01F6, "\xC6\x95" }, + { 0x01F7, "\xC6\xBF" }, + { 0x01F8, "\x6E" }, + { 0x01F9, "\x6E" }, + { 0x01FA, "\x61" }, + { 0x01FB, "\x61" }, + { 0x01FC, "\xC3\xA6" }, + { 0x01FD, "\xC3\xA6" }, + { 0x01FE, "\xC3\xB8" }, + { 0x01FF, "\xC3\xB8" }, + { 0x0200, "\x61" }, + { 0x0201, "\x61" }, + { 0x0202, "\x61" }, + { 0x0203, "\x61" }, + { 0x0204, "\x65" }, + { 0x0205, "\x65" }, + { 0x0206, "\x65" }, + { 0x0207, "\x65" }, + { 0x0208, "\x69" }, + { 0x0209, "\x69" }, + { 0x020A, "\x69" }, + { 0x020B, "\x69" }, + { 0x020C, "\x6F" }, + { 0x020D, "\x6F" }, + { 0x020E, "\x6F" }, + { 0x020F, "\x6F" }, + { 0x0210, "\x72" }, + { 0x0211, "\x72" }, + { 0x0212, "\x72" }, + { 0x0213, "\x72" }, + { 0x0214, "\x75" }, + { 0x0215, "\x75" }, + { 0x0216, "\x75" }, + { 0x0217, "\x75" }, + { 0x0218, "\x73" }, + { 0x0219, "\x73" }, + { 0x021A, "\x74" }, + { 0x021B, "\x74" }, + { 0x021C, "\xC8\x9D" }, + { 0x021E, "\x68" }, + { 0x021F, "\x68" }, + { 0x0220, "\xC6\x9E" }, + { 0x0222, "\xC8\xA3" }, + { 0x0224, "\xC8\xA5" }, + { 0x0226, "\x61" }, + { 0x0227, "\x61" }, + { 0x0228, "\x65" }, + { 0x0229, "\x65" }, + { 0x022A, "\x6F" }, + { 0x022B, "\x6F" }, + { 0x022C, "\x6F" }, + { 0x022D, "\x6F" }, + { 0x022E, "\x6F" }, + { 0x022F, "\x6F" }, + { 0x0230, "\x6F" }, + { 0x0231, "\x6F" }, + { 0x0232, "\x79" }, + { 0x0233, "\x79" }, + { 0x023A, "\xE2\xB1\xA5" }, + { 0x023B, "\xC8\xBC" }, + { 0x023D, "\xC6\x9A" }, + { 0x023E, "\xE2\xB1\xA6" }, + { 0x0241, "\xC9\x82" }, + { 0x0243, "\xC6\x80" }, + { 0x0244, "\xCA\x89" }, + { 0x0245, "\xCA\x8C" }, + { 0x0246, "\xC9\x87" }, + { 0x0248, "\xC9\x89" }, + { 0x024A, "\xC9\x8B" }, + { 0x024C, "\xC9\x8D" }, + { 0x024E, "\xC9\x8F" }, + { 0x0259, "\x65" }, + { 0x02BB, "" }, + { 0x02BC, "" }, + { 0x0370, "\xCD\xB1" }, + { 0x0372, "\xCD\xB3" }, + { 0x0376, "\xCD\xB7" }, + { 0x037F, "\xCF\xB3" }, + { 0x0386, "\xCE\xAC" }, + { 0x0388, "\xCE\xAD" }, + { 0x0389, "\xCE\xAE" }, + { 0x038A, "\xCE\xAF" }, + { 0x038C, "\xCF\x8C" }, + { 0x038E, "\xCF\x8D" }, + { 0x038F, "\xCF\x8E" }, + { 0x0391, "\xCE\xB1" }, + { 0x0392, "\xCE\xB2" }, + { 0x0393, "\xCE\xB3" }, + { 0x0394, "\xCE\xB4" }, + { 0x0395, "\xCE\xB5" }, + { 0x0396, "\xCE\xB6" }, + { 0x0397, "\xCE\xB7" }, + { 0x0398, "\xCE\xB8" }, + { 0x0399, "\xCE\xB9" }, + { 0x039A, "\xCE\xBA" }, + { 0x039B, "\xCE\xBB" }, + { 0x039C, "\xCE\xBC" }, + { 0x039D, "\xCE\xBD" }, + { 0x039E, "\xCE\xBE" }, + { 0x039F, "\xCE\xBF" }, + { 0x03A0, "\xCF\x80" }, + { 0x03A1, "\xCF\x81" }, + { 0x03A3, "\xCF\x83" }, + { 0x03A4, "\xCF\x84" }, + { 0x03A5, "\xCF\x85" }, + { 0x03A6, "\xCF\x86" }, + { 0x03A7, "\xCF\x87" }, + { 0x03A8, "\xCF\x88" }, + { 0x03A9, "\xCF\x89" }, + { 0x03AA, "\xCF\x8A" }, + { 0x03AB, "\xCF\x8B" }, + { 0x03CF, "\xCF\x97" }, + { 0x03D8, "\xCF\x99" }, + { 0x03DA, "\xCF\x9B" }, + { 0x03DC, "\xCF\x9D" }, + { 0x03DE, "\xCF\x9F" }, + { 0x03E0, "\xCF\xA1" }, + { 0x03E2, "\xCF\xA3" }, + { 0x03E4, "\xCF\xA5" }, + { 0x03E6, "\xCF\xA7" }, + { 0x03E8, "\xCF\xA9" }, + { 0x03EA, "\xCF\xAB" }, + { 0x03EC, "\xCF\xAD" }, + { 0x03EE, "\xCF\xAF" }, + { 0x03F4, "\xCE\xB8" }, + { 0x03F7, "\xCF\xB8" }, + { 0x03F9, "\xCF\xB2" }, + { 0x03FA, "\xCF\xBB" }, + { 0x03FD, "\xCD\xBB" }, + { 0x03FE, "\xCD\xBC" }, + { 0x03FF, "\xCD\xBD" }, + { 0x0400, "\xD1\x90" }, + { 0x0401, "\xD1\x91" }, + { 0x0402, "\xD1\x92" }, + { 0x0403, "\xD1\x93" }, + { 0x0404, "\xD1\x94" }, + { 0x0405, "\xD1\x95" }, + { 0x0406, "\xD1\x96" }, + { 0x0407, "\xD1\x97" }, + { 0x0408, "\xD1\x98" }, + { 0x0409, "\xD1\x99" }, + { 0x040A, "\xD1\x9A" }, + { 0x040B, "\xD1\x9B" }, + { 0x040C, "\xD1\x9C" }, + { 0x040D, "\xD1\x9D" }, + { 0x040E, "\xD1\x9E" }, + { 0x040F, "\xD1\x9F" }, + { 0x0410, "\xD0\xB0" }, + { 0x0411, "\xD0\xB1" }, + { 0x0412, "\xD0\xB2" }, + { 0x0413, "\xD0\xB3" }, + { 0x0414, "\xD0\xB4" }, + { 0x0415, "\xD0\xB5" }, + { 0x0416, "\xD0\xB6" }, + { 0x0417, "\xD0\xB7" }, + { 0x0418, "\xD0\xB8" }, + { 0x0419, "\xD0\xB9" }, + { 0x041A, "\xD0\xBA" }, + { 0x041B, "\xD0\xBB" }, + { 0x041C, "\xD0\xBC" }, + { 0x041D, "\xD0\xBD" }, + { 0x041E, "\xD0\xBE" }, + { 0x041F, "\xD0\xBF" }, + { 0x0420, "\xD1\x80" }, + { 0x0421, "\xD1\x81" }, + { 0x0422, "\xD1\x82" }, + { 0x0423, "\xD1\x83" }, + { 0x0424, "\xD1\x84" }, + { 0x0425, "\xD1\x85" }, + { 0x0426, "\xD1\x86" }, + { 0x0427, "\xD1\x87" }, + { 0x0428, "\xD1\x88" }, + { 0x0429, "\xD1\x89" }, + { 0x042A, "\xD1\x8A" }, + { 0x042B, "\xD1\x8B" }, + { 0x042C, "\xD1\x8C" }, + { 0x042D, "\xD1\x8D" }, + { 0x042E, "\xD1\x8E" }, + { 0x042F, "\xD1\x8F" }, + { 0x0460, "\xD1\xA1" }, + { 0x0462, "\xD1\xA3" }, + { 0x0464, "\xD1\xA5" }, + { 0x0466, "\xD1\xA7" }, + { 0x0468, "\xD1\xA9" }, + { 0x046A, "\xD1\xAB" }, + { 0x046C, "\xD1\xAD" }, + { 0x046E, "\xD1\xAF" }, + { 0x0470, "\xD1\xB1" }, + { 0x0472, "\xD1\xB3" }, + { 0x0474, "\xD1\xB5" }, + { 0x0476, "\xD1\xB7" }, + { 0x0478, "\xD1\xB9" }, + { 0x047A, "\xD1\xBB" }, + { 0x047C, "\xD1\xBD" }, + { 0x047E, "\xD1\xBF" }, + { 0x0480, "\xD2\x81" }, + { 0x048A, "\xD2\x8B" }, + { 0x048C, "\xD2\x8D" }, + { 0x048E, "\xD2\x8F" }, + { 0x0490, "\xD2\x91" }, + { 0x0492, "\xD2\x93" }, + { 0x0494, "\xD2\x95" }, + { 0x0496, "\xD2\x97" }, + { 0x0498, "\xD2\x99" }, + { 0x049A, "\xD2\x9B" }, + { 0x049C, "\xD2\x9D" }, + { 0x049E, "\xD2\x9F" }, + { 0x04A0, "\xD2\xA1" }, + { 0x04A2, "\xD2\xA3" }, + { 0x04A4, "\xD2\xA5" }, + { 0x04A6, "\xD2\xA7" }, + { 0x04A8, "\xD2\xA9" }, + { 0x04AA, "\xD2\xAB" }, + { 0x04AC, "\xD2\xAD" }, + { 0x04AE, "\xD2\xAF" }, + { 0x04B0, "\xD2\xB1" }, + { 0x04B2, "\xD2\xB3" }, + { 0x04B4, "\xD2\xB5" }, + { 0x04B6, "\xD2\xB7" }, + { 0x04B8, "\xD2\xB9" }, + { 0x04BA, "\xD2\xBB" }, + { 0x04BC, "\xD2\xBD" }, + { 0x04BE, "\xD2\xBF" }, + { 0x04C0, "\xD3\x8F" }, + { 0x04C1, "\xD3\x82" }, + { 0x04C3, "\xD3\x84" }, + { 0x04C5, "\xD3\x86" }, + { 0x04C7, "\xD3\x88" }, + { 0x04C9, "\xD3\x8A" }, + { 0x04CB, "\xD3\x8C" }, + { 0x04CD, "\xD3\x8E" }, + { 0x04D0, "\xD3\x91" }, + { 0x04D2, "\xD3\x93" }, + { 0x04D4, "\xD3\x95" }, + { 0x04D6, "\xD3\x97" }, + { 0x04D8, "\xD3\x99" }, + { 0x04DA, "\xD3\x9B" }, + { 0x04DC, "\xD3\x9D" }, + { 0x04DE, "\xD3\x9F" }, + { 0x04E0, "\xD3\xA1" }, + { 0x04E2, "\xD3\xA3" }, + { 0x04E4, "\xD3\xA5" }, + { 0x04E6, "\xD3\xA7" }, + { 0x04E8, "\xD3\xA9" }, + { 0x04EA, "\xD3\xAB" }, + { 0x04EC, "\xD3\xAD" }, + { 0x04EE, "\xD3\xAF" }, + { 0x04F0, "\xD3\xB1" }, + { 0x04F2, "\xD3\xB3" }, + { 0x04F4, "\xD3\xB5" }, + { 0x04F6, "\xD3\xB7" }, + { 0x04F8, "\xD3\xB9" }, + { 0x04FA, "\xD3\xBB" }, + { 0x04FC, "\xD3\xBD" }, + { 0x04FE, "\xD3\xBF" }, + { 0x0500, "\xD4\x81" }, + { 0x0502, "\xD4\x83" }, + { 0x0504, "\xD4\x85" }, + { 0x0506, "\xD4\x87" }, + { 0x0508, "\xD4\x89" }, + { 0x050A, "\xD4\x8B" }, + { 0x050C, "\xD4\x8D" }, + { 0x050E, "\xD4\x8F" }, + { 0x0510, "\xD4\x91" }, + { 0x0512, "\xD4\x93" }, + { 0x0514, "\xD4\x95" }, + { 0x0516, "\xD4\x97" }, + { 0x0518, "\xD4\x99" }, + { 0x051A, "\xD4\x9B" }, + { 0x051C, "\xD4\x9D" }, + { 0x051E, "\xD4\x9F" }, + { 0x0520, "\xD4\xA1" }, + { 0x0522, "\xD4\xA3" }, + { 0x0524, "\xD4\xA5" }, + { 0x0526, "\xD4\xA7" }, + { 0x0528, "\xD4\xA9" }, + { 0x052A, "\xD4\xAB" }, + { 0x052C, "\xD4\xAD" }, + { 0x052E, "\xD4\xAF" }, + { 0x0531, "\xD5\xA1" }, + { 0x0532, "\xD5\xA2" }, + { 0x0533, "\xD5\xA3" }, + { 0x0534, "\xD5\xA4" }, + { 0x0535, "\xD5\xA5" }, + { 0x0536, "\xD5\xA6" }, + { 0x0537, "\xD5\xA7" }, + { 0x0538, "\xD5\xA8" }, + { 0x0539, "\xD5\xA9" }, + { 0x053A, "\xD5\xAA" }, + { 0x053B, "\xD5\xAB" }, + { 0x053C, "\xD5\xAC" }, + { 0x053D, "\xD5\xAD" }, + { 0x053E, "\xD5\xAE" }, + { 0x053F, "\xD5\xAF" }, + { 0x0540, "\xD5\xB0" }, + { 0x0541, "\xD5\xB1" }, + { 0x0542, "\xD5\xB2" }, + { 0x0543, "\xD5\xB3" }, + { 0x0544, "\xD5\xB4" }, + { 0x0545, "\xD5\xB5" }, + { 0x0546, "\xD5\xB6" }, + { 0x0547, "\xD5\xB7" }, + { 0x0548, "\xD5\xB8" }, + { 0x0549, "\xD5\xB9" }, + { 0x054A, "\xD5\xBA" }, + { 0x054B, "\xD5\xBB" }, + { 0x054C, "\xD5\xBC" }, + { 0x054D, "\xD5\xBD" }, + { 0x054E, "\xD5\xBE" }, + { 0x054F, "\xD5\xBF" }, + { 0x0550, "\xD6\x80" }, + { 0x0551, "\xD6\x81" }, + { 0x0552, "\xD6\x82" }, + { 0x0553, "\xD6\x83" }, + { 0x0554, "\xD6\x84" }, + { 0x0555, "\xD6\x85" }, + { 0x0556, "\xD6\x86" }, + { 0x0622, "\xD8\xA7" }, + { 0x0623, "\xD8\xA7" }, + { 0x0624, "\xD9\x88" }, + { 0x0625, "\xD8\xA7" }, + { 0x0626, "\xD9\x8A" }, + { 0x06C0, "\xDB\x95" }, + { 0x06C2, "\xDB\x81" }, + { 0x06D3, "\xDB\x92" }, + { 0x0929, "\xE0\xA4\xA8" }, + { 0x0931, "\xE0\xA4\xB0" }, + { 0x0934, "\xE0\xA4\xB3" }, + { 0x0958, "\xE0\xA4\x95" }, + { 0x0959, "\xE0\xA4\x96" }, + { 0x095A, "\xE0\xA4\x97" }, + { 0x095B, "\xE0\xA4\x9C" }, + { 0x095C, "\xE0\xA4\xA1" }, + { 0x095D, "\xE0\xA4\xA2" }, + { 0x095E, "\xE0\xA4\xAB" }, + { 0x095F, "\xE0\xA4\xAF" }, + { 0x09DC, "\xE0\xA6\xA1" }, + { 0x09DD, "\xE0\xA6\xA2" }, + { 0x09DF, "\xE0\xA6\xAF" }, + { 0x0A33, "\xE0\xA8\xB2" }, + { 0x0A36, "\xE0\xA8\xB8" }, + { 0x0A59, "\xE0\xA8\x96" }, + { 0x0A5A, "\xE0\xA8\x97" }, + { 0x0A5B, "\xE0\xA8\x9C" }, + { 0x0A5E, "\xE0\xA8\xAB" }, + { 0x0B5C, "\xE0\xAC\xA1" }, + { 0x0B5D, "\xE0\xAC\xA2" }, + { 0x0B94, "\xE0\xAE\x92\xE0\xAF\x97" }, + { 0x0F43, "\xE0\xBD\x82" }, + { 0x0F4D, "\xE0\xBD\x8C" }, + { 0x0F52, "\xE0\xBD\x91" }, + { 0x0F57, "\xE0\xBD\x96" }, + { 0x0F5C, "\xE0\xBD\x9B" }, + { 0x0F69, "\xE0\xBD\x80" }, + { 0x1026, "\xE1\x80\xA5" }, + { 0x10A0, "\xE2\xB4\x80" }, + { 0x10A1, "\xE2\xB4\x81" }, + { 0x10A2, "\xE2\xB4\x82" }, + { 0x10A3, "\xE2\xB4\x83" }, + { 0x10A4, "\xE2\xB4\x84" }, + { 0x10A5, "\xE2\xB4\x85" }, + { 0x10A6, "\xE2\xB4\x86" }, + { 0x10A7, "\xE2\xB4\x87" }, + { 0x10A8, "\xE2\xB4\x88" }, + { 0x10A9, "\xE2\xB4\x89" }, + { 0x10AA, "\xE2\xB4\x8A" }, + { 0x10AB, "\xE2\xB4\x8B" }, + { 0x10AC, "\xE2\xB4\x8C" }, + { 0x10AD, "\xE2\xB4\x8D" }, + { 0x10AE, "\xE2\xB4\x8E" }, + { 0x10AF, "\xE2\xB4\x8F" }, + { 0x10B0, "\xE2\xB4\x90" }, + { 0x10B1, "\xE2\xB4\x91" }, + { 0x10B2, "\xE2\xB4\x92" }, + { 0x10B3, "\xE2\xB4\x93" }, + { 0x10B4, "\xE2\xB4\x94" }, + { 0x10B5, "\xE2\xB4\x95" }, + { 0x10B6, "\xE2\xB4\x96" }, + { 0x10B7, "\xE2\xB4\x97" }, + { 0x10B8, "\xE2\xB4\x98" }, + { 0x10B9, "\xE2\xB4\x99" }, + { 0x10BA, "\xE2\xB4\x9A" }, + { 0x10BB, "\xE2\xB4\x9B" }, + { 0x10BC, "\xE2\xB4\x9C" }, + { 0x10BD, "\xE2\xB4\x9D" }, + { 0x10BE, "\xE2\xB4\x9E" }, + { 0x10BF, "\xE2\xB4\x9F" }, + { 0x10C0, "\xE2\xB4\xA0" }, + { 0x10C1, "\xE2\xB4\xA1" }, + { 0x10C2, "\xE2\xB4\xA2" }, + { 0x10C3, "\xE2\xB4\xA3" }, + { 0x10C4, "\xE2\xB4\xA4" }, + { 0x10C5, "\xE2\xB4\xA5" }, + { 0x10C7, "\xE2\xB4\xA7" }, + { 0x10CD, "\xE2\xB4\xAD" }, + { 0x13A0, "\xEA\xAD\xB0" }, + { 0x13A1, "\xEA\xAD\xB1" }, + { 0x13A2, "\xEA\xAD\xB2" }, + { 0x13A3, "\xEA\xAD\xB3" }, + { 0x13A4, "\xEA\xAD\xB4" }, + { 0x13A5, "\xEA\xAD\xB5" }, + { 0x13A6, "\xEA\xAD\xB6" }, + { 0x13A7, "\xEA\xAD\xB7" }, + { 0x13A8, "\xEA\xAD\xB8" }, + { 0x13A9, "\xEA\xAD\xB9" }, + { 0x13AA, "\xEA\xAD\xBA" }, + { 0x13AB, "\xEA\xAD\xBB" }, + { 0x13AC, "\xEA\xAD\xBC" }, + { 0x13AD, "\xEA\xAD\xBD" }, + { 0x13AE, "\xEA\xAD\xBE" }, + { 0x13AF, "\xEA\xAD\xBF" }, + { 0x13B0, "\xEA\xAE\x80" }, + { 0x13B1, "\xEA\xAE\x81" }, + { 0x13B2, "\xEA\xAE\x82" }, + { 0x13B3, "\xEA\xAE\x83" }, + { 0x13B4, "\xEA\xAE\x84" }, + { 0x13B5, "\xEA\xAE\x85" }, + { 0x13B6, "\xEA\xAE\x86" }, + { 0x13B7, "\xEA\xAE\x87" }, + { 0x13B8, "\xEA\xAE\x88" }, + { 0x13B9, "\xEA\xAE\x89" }, + { 0x13BA, "\xEA\xAE\x8A" }, + { 0x13BB, "\xEA\xAE\x8B" }, + { 0x13BC, "\xEA\xAE\x8C" }, + { 0x13BD, "\xEA\xAE\x8D" }, + { 0x13BE, "\xEA\xAE\x8E" }, + { 0x13BF, "\xEA\xAE\x8F" }, + { 0x13C0, "\xEA\xAE\x90" }, + { 0x13C1, "\xEA\xAE\x91" }, + { 0x13C2, "\xEA\xAE\x92" }, + { 0x13C3, "\xEA\xAE\x93" }, + { 0x13C4, "\xEA\xAE\x94" }, + { 0x13C5, "\xEA\xAE\x95" }, + { 0x13C6, "\xEA\xAE\x96" }, + { 0x13C7, "\xEA\xAE\x97" }, + { 0x13C8, "\xEA\xAE\x98" }, + { 0x13C9, "\xEA\xAE\x99" }, + { 0x13CA, "\xEA\xAE\x9A" }, + { 0x13CB, "\xEA\xAE\x9B" }, + { 0x13CC, "\xEA\xAE\x9C" }, + { 0x13CD, "\xEA\xAE\x9D" }, + { 0x13CE, "\xEA\xAE\x9E" }, + { 0x13CF, "\xEA\xAE\x9F" }, + { 0x13D0, "\xEA\xAE\xA0" }, + { 0x13D1, "\xEA\xAE\xA1" }, + { 0x13D2, "\xEA\xAE\xA2" }, + { 0x13D3, "\xEA\xAE\xA3" }, + { 0x13D4, "\xEA\xAE\xA4" }, + { 0x13D5, "\xEA\xAE\xA5" }, + { 0x13D6, "\xEA\xAE\xA6" }, + { 0x13D7, "\xEA\xAE\xA7" }, + { 0x13D8, "\xEA\xAE\xA8" }, + { 0x13D9, "\xEA\xAE\xA9" }, + { 0x13DA, "\xEA\xAE\xAA" }, + { 0x13DB, "\xEA\xAE\xAB" }, + { 0x13DC, "\xEA\xAE\xAC" }, + { 0x13DD, "\xEA\xAE\xAD" }, + { 0x13DE, "\xEA\xAE\xAE" }, + { 0x13DF, "\xEA\xAE\xAF" }, + { 0x13E0, "\xEA\xAE\xB0" }, + { 0x13E1, "\xEA\xAE\xB1" }, + { 0x13E2, "\xEA\xAE\xB2" }, + { 0x13E3, "\xEA\xAE\xB3" }, + { 0x13E4, "\xEA\xAE\xB4" }, + { 0x13E5, "\xEA\xAE\xB5" }, + { 0x13E6, "\xEA\xAE\xB6" }, + { 0x13E7, "\xEA\xAE\xB7" }, + { 0x13E8, "\xEA\xAE\xB8" }, + { 0x13E9, "\xEA\xAE\xB9" }, + { 0x13EA, "\xEA\xAE\xBA" }, + { 0x13EB, "\xEA\xAE\xBB" }, + { 0x13EC, "\xEA\xAE\xBC" }, + { 0x13ED, "\xEA\xAE\xBD" }, + { 0x13EE, "\xEA\xAE\xBE" }, + { 0x13EF, "\xEA\xAE\xBF" }, + { 0x13F0, "\xE1\x8F\xB8" }, + { 0x13F1, "\xE1\x8F\xB9" }, + { 0x13F2, "\xE1\x8F\xBA" }, + { 0x13F3, "\xE1\x8F\xBB" }, + { 0x13F4, "\xE1\x8F\xBC" }, + { 0x13F5, "\xE1\x8F\xBD" }, + { 0x1B06, "\xE1\xAC\x85\xE1\xAC\xB5" }, + { 0x1B08, "\xE1\xAC\x87\xE1\xAC\xB5" }, + { 0x1B0A, "\xE1\xAC\x89\xE1\xAC\xB5" }, + { 0x1B0C, "\xE1\xAC\x8B\xE1\xAC\xB5" }, + { 0x1B0E, "\xE1\xAC\x8D\xE1\xAC\xB5" }, + { 0x1B12, "\xE1\xAC\x91\xE1\xAC\xB5" }, + { 0x1C90, "\xE1\x83\x90" }, + { 0x1C91, "\xE1\x83\x91" }, + { 0x1C92, "\xE1\x83\x92" }, + { 0x1C93, "\xE1\x83\x93" }, + { 0x1C94, "\xE1\x83\x94" }, + { 0x1C95, "\xE1\x83\x95" }, + { 0x1C96, "\xE1\x83\x96" }, + { 0x1C97, "\xE1\x83\x97" }, + { 0x1C98, "\xE1\x83\x98" }, + { 0x1C99, "\xE1\x83\x99" }, + { 0x1C9A, "\xE1\x83\x9A" }, + { 0x1C9B, "\xE1\x83\x9B" }, + { 0x1C9C, "\xE1\x83\x9C" }, + { 0x1C9D, "\xE1\x83\x9D" }, + { 0x1C9E, "\xE1\x83\x9E" }, + { 0x1C9F, "\xE1\x83\x9F" }, + { 0x1CA0, "\xE1\x83\xA0" }, + { 0x1CA1, "\xE1\x83\xA1" }, + { 0x1CA2, "\xE1\x83\xA2" }, + { 0x1CA3, "\xE1\x83\xA3" }, + { 0x1CA4, "\xE1\x83\xA4" }, + { 0x1CA5, "\xE1\x83\xA5" }, + { 0x1CA6, "\xE1\x83\xA6" }, + { 0x1CA7, "\xE1\x83\xA7" }, + { 0x1CA8, "\xE1\x83\xA8" }, + { 0x1CA9, "\xE1\x83\xA9" }, + { 0x1CAA, "\xE1\x83\xAA" }, + { 0x1CAB, "\xE1\x83\xAB" }, + { 0x1CAC, "\xE1\x83\xAC" }, + { 0x1CAD, "\xE1\x83\xAD" }, + { 0x1CAE, "\xE1\x83\xAE" }, + { 0x1CAF, "\xE1\x83\xAF" }, + { 0x1CB0, "\xE1\x83\xB0" }, + { 0x1CB1, "\xE1\x83\xB1" }, + { 0x1CB2, "\xE1\x83\xB2" }, + { 0x1CB3, "\xE1\x83\xB3" }, + { 0x1CB4, "\xE1\x83\xB4" }, + { 0x1CB5, "\xE1\x83\xB5" }, + { 0x1CB6, "\xE1\x83\xB6" }, + { 0x1CB7, "\xE1\x83\xB7" }, + { 0x1CB8, "\xE1\x83\xB8" }, + { 0x1CB9, "\xE1\x83\xB9" }, + { 0x1CBA, "\xE1\x83\xBA" }, + { 0x1CBD, "\xE1\x83\xBD" }, + { 0x1CBE, "\xE1\x83\xBE" }, + { 0x1CBF, "\xE1\x83\xBF" }, + { 0x1E00, "\x61" }, + { 0x1E01, "\x61" }, + { 0x1E02, "\x62" }, + { 0x1E03, "\x62" }, + { 0x1E04, "\x62" }, + { 0x1E05, "\x62" }, + { 0x1E06, "\x62" }, + { 0x1E07, "\x62" }, + { 0x1E08, "\x63" }, + { 0x1E09, "\x63" }, + { 0x1E0A, "\x64" }, + { 0x1E0B, "\x64" }, + { 0x1E0C, "\x64" }, + { 0x1E0D, "\x64" }, + { 0x1E0E, "\x64" }, + { 0x1E0F, "\x64" }, + { 0x1E10, "\x64" }, + { 0x1E11, "\x64" }, + { 0x1E12, "\x64" }, + { 0x1E13, "\x64" }, + { 0x1E14, "\x65" }, + { 0x1E15, "\x65" }, + { 0x1E16, "\x65" }, + { 0x1E17, "\x65" }, + { 0x1E18, "\x65" }, + { 0x1E19, "\x65" }, + { 0x1E1A, "\x65" }, + { 0x1E1B, "\x65" }, + { 0x1E1C, "\x65" }, + { 0x1E1D, "\x65" }, + { 0x1E1E, "\x66" }, + { 0x1E1F, "\x66" }, + { 0x1E20, "\x67" }, + { 0x1E21, "\x67" }, + { 0x1E22, "\x68" }, + { 0x1E23, "\x68" }, + { 0x1E24, "\x68" }, + { 0x1E25, "\x68" }, + { 0x1E26, "\x68" }, + { 0x1E27, "\x68" }, + { 0x1E28, "\x68" }, + { 0x1E29, "\x68" }, + { 0x1E2A, "\x68" }, + { 0x1E2B, "\x68" }, + { 0x1E2C, "\x69" }, + { 0x1E2D, "\x69" }, + { 0x1E2E, "\x69" }, + { 0x1E2F, "\x69" }, + { 0x1E30, "\x6B" }, + { 0x1E31, "\x6B" }, + { 0x1E32, "\x6B" }, + { 0x1E33, "\x6B" }, + { 0x1E34, "\x6B" }, + { 0x1E35, "\x6B" }, + { 0x1E36, "\x6C" }, + { 0x1E37, "\x6C" }, + { 0x1E38, "\x6C" }, + { 0x1E39, "\x6C" }, + { 0x1E3A, "\x6C" }, + { 0x1E3B, "\x6C" }, + { 0x1E3C, "\x6C" }, + { 0x1E3D, "\x6C" }, + { 0x1E3E, "\x6D" }, + { 0x1E3F, "\x6D" }, + { 0x1E40, "\x6D" }, + { 0x1E41, "\x6D" }, + { 0x1E42, "\x6D" }, + { 0x1E43, "\x6D" }, + { 0x1E44, "\x6E" }, + { 0x1E45, "\x6E" }, + { 0x1E46, "\x6E" }, + { 0x1E47, "\x6E" }, + { 0x1E48, "\x6E" }, + { 0x1E49, "\x6E" }, + { 0x1E4A, "\x6E" }, + { 0x1E4B, "\x6E" }, + { 0x1E4C, "\x6F" }, + { 0x1E4D, "\x6F" }, + { 0x1E4E, "\x6F" }, + { 0x1E4F, "\x6F" }, + { 0x1E50, "\x6F" }, + { 0x1E51, "\x6F" }, + { 0x1E52, "\x6F" }, + { 0x1E53, "\x6F" }, + { 0x1E54, "\x70" }, + { 0x1E55, "\x70" }, + { 0x1E56, "\x70" }, + { 0x1E57, "\x70" }, + { 0x1E58, "\x72" }, + { 0x1E59, "\x72" }, + { 0x1E5A, "\x72" }, + { 0x1E5B, "\x72" }, + { 0x1E5C, "\x72" }, + { 0x1E5D, "\x72" }, + { 0x1E5E, "\x72" }, + { 0x1E5F, "\x72" }, + { 0x1E60, "\x73" }, + { 0x1E61, "\x73" }, + { 0x1E62, "\x73" }, + { 0x1E63, "\x73" }, + { 0x1E64, "\x73" }, + { 0x1E65, "\x73" }, + { 0x1E66, "\x73" }, + { 0x1E67, "\x73" }, + { 0x1E68, "\x73" }, + { 0x1E69, "\x73" }, + { 0x1E6A, "\x74" }, + { 0x1E6B, "\x74" }, + { 0x1E6C, "\x74" }, + { 0x1E6D, "\x74" }, + { 0x1E6E, "\x74" }, + { 0x1E6F, "\x74" }, + { 0x1E70, "\x74" }, + { 0x1E71, "\x74" }, + { 0x1E72, "\x75" }, + { 0x1E73, "\x75" }, + { 0x1E74, "\x75" }, + { 0x1E75, "\x75" }, + { 0x1E76, "\x75" }, + { 0x1E77, "\x75" }, + { 0x1E78, "\x75" }, + { 0x1E79, "\x75" }, + { 0x1E7A, "\x75" }, + { 0x1E7B, "\x75" }, + { 0x1E7C, "\x76" }, + { 0x1E7D, "\x76" }, + { 0x1E7E, "\x76" }, + { 0x1E7F, "\x76" }, + { 0x1E80, "\x77" }, + { 0x1E81, "\x77" }, + { 0x1E82, "\x77" }, + { 0x1E83, "\x77" }, + { 0x1E84, "\x77" }, + { 0x1E85, "\x77" }, + { 0x1E86, "\x77" }, + { 0x1E87, "\x77" }, + { 0x1E88, "\x77" }, + { 0x1E89, "\x77" }, + { 0x1E8A, "\x78" }, + { 0x1E8B, "\x78" }, + { 0x1E8C, "\x78" }, + { 0x1E8D, "\x78" }, + { 0x1E8E, "\x79" }, + { 0x1E8F, "\x79" }, + { 0x1E90, "\x7A" }, + { 0x1E91, "\x7A" }, + { 0x1E92, "\x7A" }, + { 0x1E93, "\x7A" }, + { 0x1E94, "\x7A" }, + { 0x1E95, "\x7A" }, + { 0x1E96, "\x68" }, + { 0x1E97, "\x74" }, + { 0x1E98, "\x77" }, + { 0x1E99, "\x79" }, + { 0x1E9B, "\xC5\xBF" }, + { 0x1E9E, "\x73\x73" }, + { 0x1EA0, "\x61" }, + { 0x1EA1, "\x61" }, + { 0x1EA2, "\x61" }, + { 0x1EA3, "\x61" }, + { 0x1EA4, "\x61" }, + { 0x1EA5, "\x61" }, + { 0x1EA6, "\x61" }, + { 0x1EA7, "\x61" }, + { 0x1EA8, "\x61" }, + { 0x1EA9, "\x61" }, + { 0x1EAA, "\x61" }, + { 0x1EAB, "\x61" }, + { 0x1EAC, "\x61" }, + { 0x1EAD, "\x61" }, + { 0x1EAE, "\x61" }, + { 0x1EAF, "\x61" }, + { 0x1EB0, "\x61" }, + { 0x1EB1, "\x61" }, + { 0x1EB2, "\x61" }, + { 0x1EB3, "\x61" }, + { 0x1EB4, "\x61" }, + { 0x1EB5, "\x61" }, + { 0x1EB6, "\x61" }, + { 0x1EB7, "\x61" }, + { 0x1EB8, "\x65" }, + { 0x1EB9, "\x65" }, + { 0x1EBA, "\x65" }, + { 0x1EBB, "\x65" }, + { 0x1EBC, "\x65" }, + { 0x1EBD, "\x65" }, + { 0x1EBE, "\x65" }, + { 0x1EBF, "\x65" }, + { 0x1EC0, "\x65" }, + { 0x1EC1, "\x65" }, + { 0x1EC2, "\x65" }, + { 0x1EC3, "\x65" }, + { 0x1EC4, "\x65" }, + { 0x1EC5, "\x65" }, + { 0x1EC6, "\x65" }, + { 0x1EC7, "\x65" }, + { 0x1EC8, "\x69" }, + { 0x1EC9, "\x69" }, + { 0x1ECA, "\x69" }, + { 0x1ECB, "\x69" }, + { 0x1ECC, "\x6F" }, + { 0x1ECD, "\x6F" }, + { 0x1ECE, "\x6F" }, + { 0x1ECF, "\x6F" }, + { 0x1ED0, "\x6F" }, + { 0x1ED1, "\x6F" }, + { 0x1ED2, "\x6F" }, + { 0x1ED3, "\x6F" }, + { 0x1ED4, "\x6F" }, + { 0x1ED5, "\x6F" }, + { 0x1ED6, "\x6F" }, + { 0x1ED7, "\x6F" }, + { 0x1ED8, "\x6F" }, + { 0x1ED9, "\x6F" }, + { 0x1EDA, "\x6F" }, + { 0x1EDB, "\x6F" }, + { 0x1EDC, "\x6F" }, + { 0x1EDD, "\x6F" }, + { 0x1EDE, "\x6F" }, + { 0x1EDF, "\x6F" }, + { 0x1EE0, "\x6F" }, + { 0x1EE1, "\x6F" }, + { 0x1EE2, "\x6F" }, + { 0x1EE3, "\x6F" }, + { 0x1EE4, "\x75" }, + { 0x1EE5, "\x75" }, + { 0x1EE6, "\x75" }, + { 0x1EE7, "\x75" }, + { 0x1EE8, "\x75" }, + { 0x1EE9, "\x75" }, + { 0x1EEA, "\x75" }, + { 0x1EEB, "\x75" }, + { 0x1EEC, "\x75" }, + { 0x1EED, "\x75" }, + { 0x1EEE, "\x75" }, + { 0x1EEF, "\x75" }, + { 0x1EF0, "\x75" }, + { 0x1EF1, "\x75" }, + { 0x1EF2, "\x79" }, + { 0x1EF3, "\x79" }, + { 0x1EF4, "\x79" }, + { 0x1EF5, "\x79" }, + { 0x1EF6, "\x79" }, + { 0x1EF7, "\x79" }, + { 0x1EF8, "\x79" }, + { 0x1EF9, "\x79" }, + { 0x1EFA, "\xE1\xBB\xBB" }, + { 0x1EFC, "\xE1\xBB\xBD" }, + { 0x1EFE, "\xE1\xBB\xBF" }, + { 0x1F00, "\xCE\xB1" }, + { 0x1F01, "\xCE\xB1" }, + { 0x1F02, "\xCE\xB1" }, + { 0x1F03, "\xCE\xB1" }, + { 0x1F04, "\xCE\xB1" }, + { 0x1F05, "\xCE\xB1" }, + { 0x1F06, "\xCE\xB1" }, + { 0x1F07, "\xCE\xB1" }, + { 0x1F08, "\xCE\xB1" }, + { 0x1F09, "\xCE\xB1" }, + { 0x1F0A, "\xCE\xB1" }, + { 0x1F0B, "\xCE\xB1" }, + { 0x1F0C, "\xCE\xB1" }, + { 0x1F0D, "\xCE\xB1" }, + { 0x1F0E, "\xCE\xB1" }, + { 0x1F0F, "\xCE\xB1" }, + { 0x1F10, "\xCE\xB5" }, + { 0x1F11, "\xCE\xB5" }, + { 0x1F12, "\xCE\xB5" }, + { 0x1F13, "\xCE\xB5" }, + { 0x1F14, "\xCE\xB5" }, + { 0x1F15, "\xCE\xB5" }, + { 0x1F18, "\xCE\xB5" }, + { 0x1F19, "\xCE\xB5" }, + { 0x1F1A, "\xCE\xB5" }, + { 0x1F1B, "\xCE\xB5" }, + { 0x1F1C, "\xCE\xB5" }, + { 0x1F1D, "\xCE\xB5" }, + { 0x1F20, "\xCE\xB7" }, + { 0x1F21, "\xCE\xB7" }, + { 0x1F22, "\xCE\xB7" }, + { 0x1F23, "\xCE\xB7" }, + { 0x1F24, "\xCE\xB7" }, + { 0x1F25, "\xCE\xB7" }, + { 0x1F26, "\xCE\xB7" }, + { 0x1F27, "\xCE\xB7" }, + { 0x1F28, "\xCE\xB7" }, + { 0x1F29, "\xCE\xB7" }, + { 0x1F2A, "\xCE\xB7" }, + { 0x1F2B, "\xCE\xB7" }, + { 0x1F2C, "\xCE\xB7" }, + { 0x1F2D, "\xCE\xB7" }, + { 0x1F2E, "\xCE\xB7" }, + { 0x1F2F, "\xCE\xB7" }, + { 0x1F30, "\xCE\xB9" }, + { 0x1F31, "\xCE\xB9" }, + { 0x1F32, "\xCE\xB9" }, + { 0x1F33, "\xCE\xB9" }, + { 0x1F34, "\xCE\xB9" }, + { 0x1F35, "\xCE\xB9" }, + { 0x1F36, "\xCE\xB9" }, + { 0x1F37, "\xCE\xB9" }, + { 0x1F38, "\xCE\xB9" }, + { 0x1F39, "\xCE\xB9" }, + { 0x1F3A, "\xCE\xB9" }, + { 0x1F3B, "\xCE\xB9" }, + { 0x1F3C, "\xCE\xB9" }, + { 0x1F3D, "\xCE\xB9" }, + { 0x1F3E, "\xCE\xB9" }, + { 0x1F3F, "\xCE\xB9" }, + { 0x1F40, "\xCE\xBF" }, + { 0x1F41, "\xCE\xBF" }, + { 0x1F42, "\xCE\xBF" }, + { 0x1F43, "\xCE\xBF" }, + { 0x1F44, "\xCE\xBF" }, + { 0x1F45, "\xCE\xBF" }, + { 0x1F48, "\xCE\xBF" }, + { 0x1F49, "\xCE\xBF" }, + { 0x1F4A, "\xCE\xBF" }, + { 0x1F4B, "\xCE\xBF" }, + { 0x1F4C, "\xCE\xBF" }, + { 0x1F4D, "\xCE\xBF" }, + { 0x1F50, "\xCF\x85" }, + { 0x1F51, "\xCF\x85" }, + { 0x1F52, "\xCF\x85" }, + { 0x1F53, "\xCF\x85" }, + { 0x1F54, "\xCF\x85" }, + { 0x1F55, "\xCF\x85" }, + { 0x1F56, "\xCF\x85" }, + { 0x1F57, "\xCF\x85" }, + { 0x1F59, "\xCF\x85" }, + { 0x1F5B, "\xCF\x85" }, + { 0x1F5D, "\xCF\x85" }, + { 0x1F5F, "\xCF\x85" }, + { 0x1F60, "\xCF\x89" }, + { 0x1F61, "\xCF\x89" }, + { 0x1F62, "\xCF\x89" }, + { 0x1F63, "\xCF\x89" }, + { 0x1F64, "\xCF\x89" }, + { 0x1F65, "\xCF\x89" }, + { 0x1F66, "\xCF\x89" }, + { 0x1F67, "\xCF\x89" }, + { 0x1F68, "\xCF\x89" }, + { 0x1F69, "\xCF\x89" }, + { 0x1F6A, "\xCF\x89" }, + { 0x1F6B, "\xCF\x89" }, + { 0x1F6C, "\xCF\x89" }, + { 0x1F6D, "\xCF\x89" }, + { 0x1F6E, "\xCF\x89" }, + { 0x1F6F, "\xCF\x89" }, + { 0x1F70, "\xCE\xB1" }, + { 0x1F71, "\xCE\xB1" }, + { 0x1F72, "\xCE\xB5" }, + { 0x1F73, "\xCE\xB5" }, + { 0x1F74, "\xCE\xB7" }, + { 0x1F75, "\xCE\xB7" }, + { 0x1F76, "\xCE\xB9" }, + { 0x1F77, "\xCE\xB9" }, + { 0x1F78, "\xCE\xBF" }, + { 0x1F79, "\xCE\xBF" }, + { 0x1F7A, "\xCF\x85" }, + { 0x1F7B, "\xCF\x85" }, + { 0x1F7C, "\xCF\x89" }, + { 0x1F7D, "\xCF\x89" }, + { 0x1F80, "\xCE\xB1" }, + { 0x1F81, "\xCE\xB1" }, + { 0x1F82, "\xCE\xB1" }, + { 0x1F83, "\xCE\xB1" }, + { 0x1F84, "\xCE\xB1" }, + { 0x1F85, "\xCE\xB1" }, + { 0x1F86, "\xCE\xB1" }, + { 0x1F87, "\xCE\xB1" }, + { 0x1F88, "\xCE\xB1" }, + { 0x1F89, "\xCE\xB1" }, + { 0x1F8A, "\xCE\xB1" }, + { 0x1F8B, "\xCE\xB1" }, + { 0x1F8C, "\xCE\xB1" }, + { 0x1F8D, "\xCE\xB1" }, + { 0x1F8E, "\xCE\xB1" }, + { 0x1F8F, "\xCE\xB1" }, + { 0x1F90, "\xCE\xB7" }, + { 0x1F91, "\xCE\xB7" }, + { 0x1F92, "\xCE\xB7" }, + { 0x1F93, "\xCE\xB7" }, + { 0x1F94, "\xCE\xB7" }, + { 0x1F95, "\xCE\xB7" }, + { 0x1F96, "\xCE\xB7" }, + { 0x1F97, "\xCE\xB7" }, + { 0x1F98, "\xCE\xB7" }, + { 0x1F99, "\xCE\xB7" }, + { 0x1F9A, "\xCE\xB7" }, + { 0x1F9B, "\xCE\xB7" }, + { 0x1F9C, "\xCE\xB7" }, + { 0x1F9D, "\xCE\xB7" }, + { 0x1F9E, "\xCE\xB7" }, + { 0x1F9F, "\xCE\xB7" }, + { 0x1FA0, "\xCF\x89" }, + { 0x1FA1, "\xCF\x89" }, + { 0x1FA2, "\xCF\x89" }, + { 0x1FA3, "\xCF\x89" }, + { 0x1FA4, "\xCF\x89" }, + { 0x1FA5, "\xCF\x89" }, + { 0x1FA6, "\xCF\x89" }, + { 0x1FA7, "\xCF\x89" }, + { 0x1FA8, "\xCF\x89" }, + { 0x1FA9, "\xCF\x89" }, + { 0x1FAA, "\xCF\x89" }, + { 0x1FAB, "\xCF\x89" }, + { 0x1FAC, "\xCF\x89" }, + { 0x1FAD, "\xCF\x89" }, + { 0x1FAE, "\xCF\x89" }, + { 0x1FAF, "\xCF\x89" }, + { 0x1FB0, "\xCE\xB1" }, + { 0x1FB1, "\xCE\xB1" }, + { 0x1FB2, "\xCE\xB1" }, + { 0x1FB3, "\xCE\xB1" }, + { 0x1FB4, "\xCE\xB1" }, + { 0x1FB6, "\xCE\xB1" }, + { 0x1FB7, "\xCE\xB1" }, + { 0x1FB8, "\xCE\xB1" }, + { 0x1FB9, "\xCE\xB1" }, + { 0x1FBA, "\xCE\xB1" }, + { 0x1FBB, "\xCE\xB1" }, + { 0x1FBC, "\xCE\xB1" }, + { 0x1FBE, "\xCE\xB9" }, + { 0x1FC2, "\xCE\xB7" }, + { 0x1FC3, "\xCE\xB7" }, + { 0x1FC4, "\xCE\xB7" }, + { 0x1FC6, "\xCE\xB7" }, + { 0x1FC7, "\xCE\xB7" }, + { 0x1FC8, "\xCE\xB5" }, + { 0x1FC9, "\xCE\xB5" }, + { 0x1FCA, "\xCE\xB7" }, + { 0x1FCB, "\xCE\xB7" }, + { 0x1FCC, "\xCE\xB7" }, + { 0x1FD0, "\xCE\xB9" }, + { 0x1FD1, "\xCE\xB9" }, + { 0x1FD2, "\xCE\xB9" }, + { 0x1FD3, "\xCE\xB9" }, + { 0x1FD6, "\xCE\xB9" }, + { 0x1FD7, "\xCE\xB9" }, + { 0x1FD8, "\xCE\xB9" }, + { 0x1FD9, "\xCE\xB9" }, + { 0x1FDA, "\xCE\xB9" }, + { 0x1FDB, "\xCE\xB9" }, + { 0x1FE0, "\xCF\x85" }, + { 0x1FE1, "\xCF\x85" }, + { 0x1FE2, "\xCF\x85" }, + { 0x1FE3, "\xCF\x85" }, + { 0x1FE4, "\xCF\x81" }, + { 0x1FE5, "\xCF\x81" }, + { 0x1FE6, "\xCF\x85" }, + { 0x1FE7, "\xCF\x85" }, + { 0x1FE8, "\xCF\x85" }, + { 0x1FE9, "\xCF\x85" }, + { 0x1FEA, "\xCF\x85" }, + { 0x1FEB, "\xCF\x85" }, + { 0x1FEC, "\xCF\x81" }, + { 0x1FF2, "\xCF\x89" }, + { 0x1FF3, "\xCF\x89" }, + { 0x1FF4, "\xCF\x89" }, + { 0x1FF6, "\xCF\x89" }, + { 0x1FF7, "\xCF\x89" }, + { 0x1FF8, "\xCE\xBF" }, + { 0x1FF9, "\xCE\xBF" }, + { 0x1FFA, "\xCF\x89" }, + { 0x1FFB, "\xCF\x89" }, + { 0x1FFC, "\xCF\x89" }, + { 0x2126, "\xCF\x89" }, + { 0x212A, "\x6B" }, + { 0x212B, "\x61" }, + { 0x2132, "\xE2\x85\x8E" }, + { 0x2160, "\xE2\x85\xB0" }, + { 0x2161, "\xE2\x85\xB1" }, + { 0x2162, "\xE2\x85\xB2" }, + { 0x2163, "\xE2\x85\xB3" }, + { 0x2164, "\xE2\x85\xB4" }, + { 0x2165, "\xE2\x85\xB5" }, + { 0x2166, "\xE2\x85\xB6" }, + { 0x2167, "\xE2\x85\xB7" }, + { 0x2168, "\xE2\x85\xB8" }, + { 0x2169, "\xE2\x85\xB9" }, + { 0x216A, "\xE2\x85\xBA" }, + { 0x216B, "\xE2\x85\xBB" }, + { 0x216C, "\xE2\x85\xBC" }, + { 0x216D, "\xE2\x85\xBD" }, + { 0x216E, "\xE2\x85\xBE" }, + { 0x216F, "\xE2\x85\xBF" }, + { 0x2183, "\xE2\x86\x84" }, + { 0x2C00, "\xE2\xB0\xB0" }, + { 0x2C01, "\xE2\xB0\xB1" }, + { 0x2C02, "\xE2\xB0\xB2" }, + { 0x2C03, "\xE2\xB0\xB3" }, + { 0x2C04, "\xE2\xB0\xB4" }, + { 0x2C05, "\xE2\xB0\xB5" }, + { 0x2C06, "\xE2\xB0\xB6" }, + { 0x2C07, "\xE2\xB0\xB7" }, + { 0x2C08, "\xE2\xB0\xB8" }, + { 0x2C09, "\xE2\xB0\xB9" }, + { 0x2C0A, "\xE2\xB0\xBA" }, + { 0x2C0B, "\xE2\xB0\xBB" }, + { 0x2C0C, "\xE2\xB0\xBC" }, + { 0x2C0D, "\xE2\xB0\xBD" }, + { 0x2C0E, "\xE2\xB0\xBE" }, + { 0x2C0F, "\xE2\xB0\xBF" }, + { 0x2C10, "\xE2\xB1\x80" }, + { 0x2C11, "\xE2\xB1\x81" }, + { 0x2C12, "\xE2\xB1\x82" }, + { 0x2C13, "\xE2\xB1\x83" }, + { 0x2C14, "\xE2\xB1\x84" }, + { 0x2C15, "\xE2\xB1\x85" }, + { 0x2C16, "\xE2\xB1\x86" }, + { 0x2C17, "\xE2\xB1\x87" }, + { 0x2C18, "\xE2\xB1\x88" }, + { 0x2C19, "\xE2\xB1\x89" }, + { 0x2C1A, "\xE2\xB1\x8A" }, + { 0x2C1B, "\xE2\xB1\x8B" }, + { 0x2C1C, "\xE2\xB1\x8C" }, + { 0x2C1D, "\xE2\xB1\x8D" }, + { 0x2C1E, "\xE2\xB1\x8E" }, + { 0x2C1F, "\xE2\xB1\x8F" }, + { 0x2C20, "\xE2\xB1\x90" }, + { 0x2C21, "\xE2\xB1\x91" }, + { 0x2C22, "\xE2\xB1\x92" }, + { 0x2C23, "\xE2\xB1\x93" }, + { 0x2C24, "\xE2\xB1\x94" }, + { 0x2C25, "\xE2\xB1\x95" }, + { 0x2C26, "\xE2\xB1\x96" }, + { 0x2C27, "\xE2\xB1\x97" }, + { 0x2C28, "\xE2\xB1\x98" }, + { 0x2C29, "\xE2\xB1\x99" }, + { 0x2C2A, "\xE2\xB1\x9A" }, + { 0x2C2B, "\xE2\xB1\x9B" }, + { 0x2C2C, "\xE2\xB1\x9C" }, + { 0x2C2D, "\xE2\xB1\x9D" }, + { 0x2C2E, "\xE2\xB1\x9E" }, + { 0x2C60, "\xE2\xB1\xA1" }, + { 0x2C62, "\xC9\xAB" }, + { 0x2C63, "\xE1\xB5\xBD" }, + { 0x2C64, "\xC9\xBD" }, + { 0x2C67, "\xE2\xB1\xA8" }, + { 0x2C69, "\xE2\xB1\xAA" }, + { 0x2C6B, "\xE2\xB1\xAC" }, + { 0x2C6D, "\xC9\x91" }, + { 0x2C6E, "\xC9\xB1" }, + { 0x2C6F, "\xC9\x90" }, + { 0x2C70, "\xC9\x92" }, + { 0x2C72, "\xE2\xB1\xB3" }, + { 0x2C75, "\xE2\xB1\xB6" }, + { 0x2C7E, "\xC8\xBF" }, + { 0x2C7F, "\xC9\x80" }, + { 0x2C80, "\xE2\xB2\x81" }, + { 0x2C82, "\xE2\xB2\x83" }, + { 0x2C84, "\xE2\xB2\x85" }, + { 0x2C86, "\xE2\xB2\x87" }, + { 0x2C88, "\xE2\xB2\x89" }, + { 0x2C8A, "\xE2\xB2\x8B" }, + { 0x2C8C, "\xE2\xB2\x8D" }, + { 0x2C8E, "\xE2\xB2\x8F" }, + { 0x2C90, "\xE2\xB2\x91" }, + { 0x2C92, "\xE2\xB2\x93" }, + { 0x2C94, "\xE2\xB2\x95" }, + { 0x2C96, "\xE2\xB2\x97" }, + { 0x2C98, "\xE2\xB2\x99" }, + { 0x2C9A, "\xE2\xB2\x9B" }, + { 0x2C9C, "\xE2\xB2\x9D" }, + { 0x2C9E, "\xE2\xB2\x9F" }, + { 0x2CA0, "\xE2\xB2\xA1" }, + { 0x2CA2, "\xE2\xB2\xA3" }, + { 0x2CA4, "\xE2\xB2\xA5" }, + { 0x2CA6, "\xE2\xB2\xA7" }, + { 0x2CA8, "\xE2\xB2\xA9" }, + { 0x2CAA, "\xE2\xB2\xAB" }, + { 0x2CAC, "\xE2\xB2\xAD" }, + { 0x2CAE, "\xE2\xB2\xAF" }, + { 0x2CB0, "\xE2\xB2\xB1" }, + { 0x2CB2, "\xE2\xB2\xB3" }, + { 0x2CB4, "\xE2\xB2\xB5" }, + { 0x2CB6, "\xE2\xB2\xB7" }, + { 0x2CB8, "\xE2\xB2\xB9" }, + { 0x2CBA, "\xE2\xB2\xBB" }, + { 0x2CBC, "\xE2\xB2\xBD" }, + { 0x2CBE, "\xE2\xB2\xBF" }, + { 0x2CC0, "\xE2\xB3\x81" }, + { 0x2CC2, "\xE2\xB3\x83" }, + { 0x2CC4, "\xE2\xB3\x85" }, + { 0x2CC6, "\xE2\xB3\x87" }, + { 0x2CC8, "\xE2\xB3\x89" }, + { 0x2CCA, "\xE2\xB3\x8B" }, + { 0x2CCC, "\xE2\xB3\x8D" }, + { 0x2CCE, "\xE2\xB3\x8F" }, + { 0x2CD0, "\xE2\xB3\x91" }, + { 0x2CD2, "\xE2\xB3\x93" }, + { 0x2CD4, "\xE2\xB3\x95" }, + { 0x2CD6, "\xE2\xB3\x97" }, + { 0x2CD8, "\xE2\xB3\x99" }, + { 0x2CDA, "\xE2\xB3\x9B" }, + { 0x2CDC, "\xE2\xB3\x9D" }, + { 0x2CDE, "\xE2\xB3\x9F" }, + { 0x2CE0, "\xE2\xB3\xA1" }, + { 0x2CE2, "\xE2\xB3\xA3" }, + { 0x2CEB, "\xE2\xB3\xAC" }, + { 0x2CED, "\xE2\xB3\xAE" }, + { 0x2CF2, "\xE2\xB3\xB3" }, + { 0x304C, "\xE3\x81\x8B" }, + { 0x304E, "\xE3\x81\x8D" }, + { 0x3050, "\xE3\x81\x8F" }, + { 0x3052, "\xE3\x81\x91" }, + { 0x3054, "\xE3\x81\x93" }, + { 0x3056, "\xE3\x81\x95" }, + { 0x3058, "\xE3\x81\x97" }, + { 0x305A, "\xE3\x81\x99" }, + { 0x305C, "\xE3\x81\x9B" }, + { 0x305E, "\xE3\x81\x9D" }, + { 0x3060, "\xE3\x81\x9F" }, + { 0x3062, "\xE3\x81\xA1" }, + { 0x3065, "\xE3\x81\xA4" }, + { 0x3067, "\xE3\x81\xA6" }, + { 0x3069, "\xE3\x81\xA8" }, + { 0x3070, "\xE3\x81\xAF" }, + { 0x3071, "\xE3\x81\xAF" }, + { 0x3073, "\xE3\x81\xB2" }, + { 0x3074, "\xE3\x81\xB2" }, + { 0x3076, "\xE3\x81\xB5" }, + { 0x3077, "\xE3\x81\xB5" }, + { 0x3079, "\xE3\x81\xB8" }, + { 0x307A, "\xE3\x81\xB8" }, + { 0x307C, "\xE3\x81\xBB" }, + { 0x307D, "\xE3\x81\xBB" }, + { 0x3094, "\xE3\x81\x86" }, + { 0x309E, "\xE3\x82\x9D" }, + { 0x30AC, "\xE3\x82\xAB" }, + { 0x30AE, "\xE3\x82\xAD" }, + { 0x30B0, "\xE3\x82\xAF" }, + { 0x30B2, "\xE3\x82\xB1" }, + { 0x30B4, "\xE3\x82\xB3" }, + { 0x30B6, "\xE3\x82\xB5" }, + { 0x30B8, "\xE3\x82\xB7" }, + { 0x30BA, "\xE3\x82\xB9" }, + { 0x30BC, "\xE3\x82\xBB" }, + { 0x30BE, "\xE3\x82\xBD" }, + { 0x30C0, "\xE3\x82\xBF" }, + { 0x30C2, "\xE3\x83\x81" }, + { 0x30C5, "\xE3\x83\x84" }, + { 0x30C7, "\xE3\x83\x86" }, + { 0x30C9, "\xE3\x83\x88" }, + { 0x30D0, "\xE3\x83\x8F" }, + { 0x30D1, "\xE3\x83\x8F" }, + { 0x30D3, "\xE3\x83\x92" }, + { 0x30D4, "\xE3\x83\x92" }, + { 0x30D6, "\xE3\x83\x95" }, + { 0x30D7, "\xE3\x83\x95" }, + { 0x30D9, "\xE3\x83\x98" }, + { 0x30DA, "\xE3\x83\x98" }, + { 0x30DC, "\xE3\x83\x9B" }, + { 0x30DD, "\xE3\x83\x9B" }, + { 0x30F4, "\xE3\x82\xA6" }, + { 0x30F7, "\xE3\x83\xAF" }, + { 0x30F8, "\xE3\x83\xB0" }, + { 0x30F9, "\xE3\x83\xB1" }, + { 0x30FA, "\xE3\x83\xB2" }, + { 0x30FE, "\xE3\x83\xBD" }, + { 0xA640, "\xEA\x99\x81" }, + { 0xA642, "\xEA\x99\x83" }, + { 0xA644, "\xEA\x99\x85" }, + { 0xA646, "\xEA\x99\x87" }, + { 0xA648, "\xEA\x99\x89" }, + { 0xA64A, "\xEA\x99\x8B" }, + { 0xA64C, "\xEA\x99\x8D" }, + { 0xA64E, "\xEA\x99\x8F" }, + { 0xA650, "\xEA\x99\x91" }, + { 0xA652, "\xEA\x99\x93" }, + { 0xA654, "\xEA\x99\x95" }, + { 0xA656, "\xEA\x99\x97" }, + { 0xA658, "\xEA\x99\x99" }, + { 0xA65A, "\xEA\x99\x9B" }, + { 0xA65C, "\xEA\x99\x9D" }, + { 0xA65E, "\xEA\x99\x9F" }, + { 0xA660, "\xEA\x99\xA1" }, + { 0xA662, "\xEA\x99\xA3" }, + { 0xA664, "\xEA\x99\xA5" }, + { 0xA666, "\xEA\x99\xA7" }, + { 0xA668, "\xEA\x99\xA9" }, + { 0xA66A, "\xEA\x99\xAB" }, + { 0xA66C, "\xEA\x99\xAD" }, + { 0xA680, "\xEA\x9A\x81" }, + { 0xA682, "\xEA\x9A\x83" }, + { 0xA684, "\xEA\x9A\x85" }, + { 0xA686, "\xEA\x9A\x87" }, + { 0xA688, "\xEA\x9A\x89" }, + { 0xA68A, "\xEA\x9A\x8B" }, + { 0xA68C, "\xEA\x9A\x8D" }, + { 0xA68E, "\xEA\x9A\x8F" }, + { 0xA690, "\xEA\x9A\x91" }, + { 0xA692, "\xEA\x9A\x93" }, + { 0xA694, "\xEA\x9A\x95" }, + { 0xA696, "\xEA\x9A\x97" }, + { 0xA698, "\xEA\x9A\x99" }, + { 0xA69A, "\xEA\x9A\x9B" }, + { 0xA722, "\xEA\x9C\xA3" }, + { 0xA724, "\xEA\x9C\xA5" }, + { 0xA726, "\xEA\x9C\xA7" }, + { 0xA728, "\xEA\x9C\xA9" }, + { 0xA72A, "\xEA\x9C\xAB" }, + { 0xA72C, "\xEA\x9C\xAD" }, + { 0xA72E, "\xEA\x9C\xAF" }, + { 0xA732, "\xEA\x9C\xB3" }, + { 0xA734, "\xEA\x9C\xB5" }, + { 0xA736, "\xEA\x9C\xB7" }, + { 0xA738, "\xEA\x9C\xB9" }, + { 0xA73A, "\xEA\x9C\xBB" }, + { 0xA73C, "\xEA\x9C\xBD" }, + { 0xA73E, "\xEA\x9C\xBF" }, + { 0xA740, "\xEA\x9D\x81" }, + { 0xA742, "\xEA\x9D\x83" }, + { 0xA744, "\xEA\x9D\x85" }, + { 0xA746, "\xEA\x9D\x87" }, + { 0xA748, "\xEA\x9D\x89" }, + { 0xA74A, "\xEA\x9D\x8B" }, + { 0xA74C, "\xEA\x9D\x8D" }, + { 0xA74E, "\xEA\x9D\x8F" }, + { 0xA750, "\xEA\x9D\x91" }, + { 0xA752, "\xEA\x9D\x93" }, + { 0xA754, "\xEA\x9D\x95" }, + { 0xA756, "\xEA\x9D\x97" }, + { 0xA758, "\xEA\x9D\x99" }, + { 0xA75A, "\xEA\x9D\x9B" }, + { 0xA75C, "\xEA\x9D\x9D" }, + { 0xA75E, "\xEA\x9D\x9F" }, + { 0xA760, "\xEA\x9D\xA1" }, + { 0xA762, "\xEA\x9D\xA3" }, + { 0xA764, "\xEA\x9D\xA5" }, + { 0xA766, "\xEA\x9D\xA7" }, + { 0xA768, "\xEA\x9D\xA9" }, + { 0xA76A, "\xEA\x9D\xAB" }, + { 0xA76C, "\xEA\x9D\xAD" }, + { 0xA76E, "\xEA\x9D\xAF" }, + { 0xA779, "\xEA\x9D\xBA" }, + { 0xA77B, "\xEA\x9D\xBC" }, + { 0xA77D, "\xE1\xB5\xB9" }, + { 0xA77E, "\xEA\x9D\xBF" }, + { 0xA780, "\xEA\x9E\x81" }, + { 0xA782, "\xEA\x9E\x83" }, + { 0xA784, "\xEA\x9E\x85" }, + { 0xA786, "\xEA\x9E\x87" }, + { 0xA78B, "\xEA\x9E\x8C" }, + { 0xA78D, "\xC9\xA5" }, + { 0xA790, "\xEA\x9E\x91" }, + { 0xA792, "\xEA\x9E\x93" }, + { 0xA796, "\xEA\x9E\x97" }, + { 0xA798, "\xEA\x9E\x99" }, + { 0xA79A, "\xEA\x9E\x9B" }, + { 0xA79C, "\xEA\x9E\x9D" }, + { 0xA79E, "\xEA\x9E\x9F" }, + { 0xA7A0, "\xEA\x9E\xA1" }, + { 0xA7A2, "\xEA\x9E\xA3" }, + { 0xA7A4, "\xEA\x9E\xA5" }, + { 0xA7A6, "\xEA\x9E\xA7" }, + { 0xA7A8, "\xEA\x9E\xA9" }, + { 0xA7AA, "\xC9\xA6" }, + { 0xA7AB, "\xC9\x9C" }, + { 0xA7AC, "\xC9\xA1" }, + { 0xA7AD, "\xC9\xAC" }, + { 0xA7AE, "\xC9\xAA" }, + { 0xA7B0, "\xCA\x9E" }, + { 0xA7B1, "\xCA\x87" }, + { 0xA7B2, "\xCA\x9D" }, + { 0xA7B3, "\xEA\xAD\x93" }, + { 0xA7B4, "\xEA\x9E\xB5" }, + { 0xA7B6, "\xEA\x9E\xB7" }, + { 0xA7B8, "\xEA\x9E\xB9" }, + { 0xA7BA, "\xEA\x9E\xBB" }, + { 0xA7BC, "\xEA\x9E\xBD" }, + { 0xA7BE, "\xEA\x9E\xBF" }, + { 0xA7C2, "\xEA\x9F\x83" }, + { 0xA7C4, "\xEA\x9E\x94" }, + { 0xA7C5, "\xCA\x82" }, + { 0xA7C6, "\xE1\xB6\x8E" }, + { 0xA7C7, "\xEA\x9F\x88" }, + { 0xA7C9, "\xEA\x9F\x8A" }, + { 0xA7F5, "\xEA\x9F\xB6" }, + { 0xF900, "\xE8\xB1\x88" }, + { 0xF901, "\xE6\x9B\xB4" }, + { 0xF902, "\xE8\xBB\x8A" }, + { 0xF903, "\xE8\xB3\x88" }, + { 0xF904, "\xE6\xBB\x91" }, + { 0xF905, "\xE4\xB8\xB2" }, + { 0xF906, "\xE5\x8F\xA5" }, + { 0xF907, "\xE9\xBE\x9C" }, + { 0xF908, "\xE9\xBE\x9C" }, + { 0xF909, "\xE5\xA5\x91" }, + { 0xF90A, "\xE9\x87\x91" }, + { 0xF90B, "\xE5\x96\x87" }, + { 0xF90C, "\xE5\xA5\x88" }, + { 0xF90D, "\xE6\x87\xB6" }, + { 0xF90E, "\xE7\x99\xA9" }, + { 0xF90F, "\xE7\xBE\x85" }, + { 0xF910, "\xE8\x98\xBF" }, + { 0xF911, "\xE8\x9E\xBA" }, + { 0xF912, "\xE8\xA3\xB8" }, + { 0xF913, "\xE9\x82\x8F" }, + { 0xF914, "\xE6\xA8\x82" }, + { 0xF915, "\xE6\xB4\x9B" }, + { 0xF916, "\xE7\x83\x99" }, + { 0xF917, "\xE7\x8F\x9E" }, + { 0xF918, "\xE8\x90\xBD" }, + { 0xF919, "\xE9\x85\xAA" }, + { 0xF91A, "\xE9\xA7\xB1" }, + { 0xF91B, "\xE4\xBA\x82" }, + { 0xF91C, "\xE5\x8D\xB5" }, + { 0xF91D, "\xE6\xAC\x84" }, + { 0xF91E, "\xE7\x88\x9B" }, + { 0xF91F, "\xE8\x98\xAD" }, + { 0xF920, "\xE9\xB8\x9E" }, + { 0xF921, "\xE5\xB5\x90" }, + { 0xF922, "\xE6\xBF\xAB" }, + { 0xF923, "\xE8\x97\x8D" }, + { 0xF924, "\xE8\xA5\xA4" }, + { 0xF925, "\xE6\x8B\x89" }, + { 0xF926, "\xE8\x87\x98" }, + { 0xF927, "\xE8\xA0\x9F" }, + { 0xF928, "\xE5\xBB\x8A" }, + { 0xF929, "\xE6\x9C\x97" }, + { 0xF92A, "\xE6\xB5\xAA" }, + { 0xF92B, "\xE7\x8B\xBC" }, + { 0xF92C, "\xE9\x83\x8E" }, + { 0xF92D, "\xE4\xBE\x86" }, + { 0xF92E, "\xE5\x86\xB7" }, + { 0xF92F, "\xE5\x8B\x9E" }, + { 0xF930, "\xE6\x93\x84" }, + { 0xF931, "\xE6\xAB\x93" }, + { 0xF932, "\xE7\x88\x90" }, + { 0xF933, "\xE7\x9B\xA7" }, + { 0xF934, "\xE8\x80\x81" }, + { 0xF935, "\xE8\x98\x86" }, + { 0xF936, "\xE8\x99\x9C" }, + { 0xF937, "\xE8\xB7\xAF" }, + { 0xF938, "\xE9\x9C\xB2" }, + { 0xF939, "\xE9\xAD\xAF" }, + { 0xF93A, "\xE9\xB7\xBA" }, + { 0xF93B, "\xE7\xA2\x8C" }, + { 0xF93C, "\xE7\xA5\xBF" }, + { 0xF93D, "\xE7\xB6\xA0" }, + { 0xF93E, "\xE8\x8F\x89" }, + { 0xF93F, "\xE9\x8C\x84" }, + { 0xF940, "\xE9\xB9\xBF" }, + { 0xF941, "\xE8\xAB\x96" }, + { 0xF942, "\xE5\xA3\x9F" }, + { 0xF943, "\xE5\xBC\x84" }, + { 0xF944, "\xE7\xB1\xA0" }, + { 0xF945, "\xE8\x81\xBE" }, + { 0xF946, "\xE7\x89\xA2" }, + { 0xF947, "\xE7\xA3\x8A" }, + { 0xF948, "\xE8\xB3\x82" }, + { 0xF949, "\xE9\x9B\xB7" }, + { 0xF94A, "\xE5\xA3\x98" }, + { 0xF94B, "\xE5\xB1\xA2" }, + { 0xF94C, "\xE6\xA8\x93" }, + { 0xF94D, "\xE6\xB7\x9A" }, + { 0xF94E, "\xE6\xBC\x8F" }, + { 0xF94F, "\xE7\xB4\xAF" }, + { 0xF950, "\xE7\xB8\xB7" }, + { 0xF951, "\xE9\x99\x8B" }, + { 0xF952, "\xE5\x8B\x92" }, + { 0xF953, "\xE8\x82\x8B" }, + { 0xF954, "\xE5\x87\x9C" }, + { 0xF955, "\xE5\x87\x8C" }, + { 0xF956, "\xE7\xA8\x9C" }, + { 0xF957, "\xE7\xB6\xBE" }, + { 0xF958, "\xE8\x8F\xB1" }, + { 0xF959, "\xE9\x99\xB5" }, + { 0xF95A, "\xE8\xAE\x80" }, + { 0xF95B, "\xE6\x8B\x8F" }, + { 0xF95C, "\xE6\xA8\x82" }, + { 0xF95D, "\xE8\xAB\xBE" }, + { 0xF95E, "\xE4\xB8\xB9" }, + { 0xF95F, "\xE5\xAF\xA7" }, + { 0xF960, "\xE6\x80\x92" }, + { 0xF961, "\xE7\x8E\x87" }, + { 0xF962, "\xE7\x95\xB0" }, + { 0xF963, "\xE5\x8C\x97" }, + { 0xF964, "\xE7\xA3\xBB" }, + { 0xF965, "\xE4\xBE\xBF" }, + { 0xF966, "\xE5\xBE\xA9" }, + { 0xF967, "\xE4\xB8\x8D" }, + { 0xF968, "\xE6\xB3\x8C" }, + { 0xF969, "\xE6\x95\xB8" }, + { 0xF96A, "\xE7\xB4\xA2" }, + { 0xF96B, "\xE5\x8F\x83" }, + { 0xF96C, "\xE5\xA1\x9E" }, + { 0xF96D, "\xE7\x9C\x81" }, + { 0xF96E, "\xE8\x91\x89" }, + { 0xF96F, "\xE8\xAA\xAA" }, + { 0xF970, "\xE6\xAE\xBA" }, + { 0xF971, "\xE8\xBE\xB0" }, + { 0xF972, "\xE6\xB2\x88" }, + { 0xF973, "\xE6\x8B\xBE" }, + { 0xF974, "\xE8\x8B\xA5" }, + { 0xF975, "\xE6\x8E\xA0" }, + { 0xF976, "\xE7\x95\xA5" }, + { 0xF977, "\xE4\xBA\xAE" }, + { 0xF978, "\xE5\x85\xA9" }, + { 0xF979, "\xE5\x87\x89" }, + { 0xF97A, "\xE6\xA2\x81" }, + { 0xF97B, "\xE7\xB3\xA7" }, + { 0xF97C, "\xE8\x89\xAF" }, + { 0xF97D, "\xE8\xAB\x92" }, + { 0xF97E, "\xE9\x87\x8F" }, + { 0xF97F, "\xE5\x8B\xB5" }, + { 0xF980, "\xE5\x91\x82" }, + { 0xF981, "\xE5\xA5\xB3" }, + { 0xF982, "\xE5\xBB\xAC" }, + { 0xF983, "\xE6\x97\x85" }, + { 0xF984, "\xE6\xBF\xBE" }, + { 0xF985, "\xE7\xA4\xAA" }, + { 0xF986, "\xE9\x96\xAD" }, + { 0xF987, "\xE9\xA9\xAA" }, + { 0xF988, "\xE9\xBA\x97" }, + { 0xF989, "\xE9\xBB\x8E" }, + { 0xF98A, "\xE5\x8A\x9B" }, + { 0xF98B, "\xE6\x9B\x86" }, + { 0xF98C, "\xE6\xAD\xB7" }, + { 0xF98D, "\xE8\xBD\xA2" }, + { 0xF98E, "\xE5\xB9\xB4" }, + { 0xF98F, "\xE6\x86\x90" }, + { 0xF990, "\xE6\x88\x80" }, + { 0xF991, "\xE6\x92\x9A" }, + { 0xF992, "\xE6\xBC\xA3" }, + { 0xF993, "\xE7\x85\x89" }, + { 0xF994, "\xE7\x92\x89" }, + { 0xF995, "\xE7\xA7\x8A" }, + { 0xF996, "\xE7\xB7\xB4" }, + { 0xF997, "\xE8\x81\xAF" }, + { 0xF998, "\xE8\xBC\xA6" }, + { 0xF999, "\xE8\x93\xAE" }, + { 0xF99A, "\xE9\x80\xA3" }, + { 0xF99B, "\xE9\x8D\x8A" }, + { 0xF99C, "\xE5\x88\x97" }, + { 0xF99D, "\xE5\x8A\xA3" }, + { 0xF99E, "\xE5\x92\xBD" }, + { 0xF99F, "\xE7\x83\x88" }, + { 0xF9A0, "\xE8\xA3\x82" }, + { 0xF9A1, "\xE8\xAA\xAA" }, + { 0xF9A2, "\xE5\xBB\x89" }, + { 0xF9A3, "\xE5\xBF\xB5" }, + { 0xF9A4, "\xE6\x8D\xBB" }, + { 0xF9A5, "\xE6\xAE\xAE" }, + { 0xF9A6, "\xE7\xB0\xBE" }, + { 0xF9A7, "\xE7\x8D\xB5" }, + { 0xF9A8, "\xE4\xBB\xA4" }, + { 0xF9A9, "\xE5\x9B\xB9" }, + { 0xF9AA, "\xE5\xAF\xA7" }, + { 0xF9AB, "\xE5\xB6\xBA" }, + { 0xF9AC, "\xE6\x80\x9C" }, + { 0xF9AD, "\xE7\x8E\xB2" }, + { 0xF9AE, "\xE7\x91\xA9" }, + { 0xF9AF, "\xE7\xBE\x9A" }, + { 0xF9B0, "\xE8\x81\x86" }, + { 0xF9B1, "\xE9\x88\xB4" }, + { 0xF9B2, "\xE9\x9B\xB6" }, + { 0xF9B3, "\xE9\x9D\x88" }, + { 0xF9B4, "\xE9\xA0\x98" }, + { 0xF9B5, "\xE4\xBE\x8B" }, + { 0xF9B6, "\xE7\xA6\xAE" }, + { 0xF9B7, "\xE9\x86\xB4" }, + { 0xF9B8, "\xE9\x9A\xB8" }, + { 0xF9B9, "\xE6\x83\xA1" }, + { 0xF9BA, "\xE4\xBA\x86" }, + { 0xF9BB, "\xE5\x83\x9A" }, + { 0xF9BC, "\xE5\xAF\xAE" }, + { 0xF9BD, "\xE5\xB0\xBF" }, + { 0xF9BE, "\xE6\x96\x99" }, + { 0xF9BF, "\xE6\xA8\x82" }, + { 0xF9C0, "\xE7\x87\x8E" }, + { 0xF9C1, "\xE7\x99\x82" }, + { 0xF9C2, "\xE8\x93\xBC" }, + { 0xF9C3, "\xE9\x81\xBC" }, + { 0xF9C4, "\xE9\xBE\x8D" }, + { 0xF9C5, "\xE6\x9A\x88" }, + { 0xF9C6, "\xE9\x98\xAE" }, + { 0xF9C7, "\xE5\x8A\x89" }, + { 0xF9C8, "\xE6\x9D\xBB" }, + { 0xF9C9, "\xE6\x9F\xB3" }, + { 0xF9CA, "\xE6\xB5\x81" }, + { 0xF9CB, "\xE6\xBA\x9C" }, + { 0xF9CC, "\xE7\x90\x89" }, + { 0xF9CD, "\xE7\x95\x99" }, + { 0xF9CE, "\xE7\xA1\xAB" }, + { 0xF9CF, "\xE7\xB4\x90" }, + { 0xF9D0, "\xE9\xA1\x9E" }, + { 0xF9D1, "\xE5\x85\xAD" }, + { 0xF9D2, "\xE6\x88\xAE" }, + { 0xF9D3, "\xE9\x99\xB8" }, + { 0xF9D4, "\xE5\x80\xAB" }, + { 0xF9D5, "\xE5\xB4\x99" }, + { 0xF9D6, "\xE6\xB7\xAA" }, + { 0xF9D7, "\xE8\xBC\xAA" }, + { 0xF9D8, "\xE5\xBE\x8B" }, + { 0xF9D9, "\xE6\x85\x84" }, + { 0xF9DA, "\xE6\xA0\x97" }, + { 0xF9DB, "\xE7\x8E\x87" }, + { 0xF9DC, "\xE9\x9A\x86" }, + { 0xF9DD, "\xE5\x88\xA9" }, + { 0xF9DE, "\xE5\x90\x8F" }, + { 0xF9DF, "\xE5\xB1\xA5" }, + { 0xF9E0, "\xE6\x98\x93" }, + { 0xF9E1, "\xE6\x9D\x8E" }, + { 0xF9E2, "\xE6\xA2\xA8" }, + { 0xF9E3, "\xE6\xB3\xA5" }, + { 0xF9E4, "\xE7\x90\x86" }, + { 0xF9E5, "\xE7\x97\xA2" }, + { 0xF9E6, "\xE7\xBD\xB9" }, + { 0xF9E7, "\xE8\xA3\x8F" }, + { 0xF9E8, "\xE8\xA3\xA1" }, + { 0xF9E9, "\xE9\x87\x8C" }, + { 0xF9EA, "\xE9\x9B\xA2" }, + { 0xF9EB, "\xE5\x8C\xBF" }, + { 0xF9EC, "\xE6\xBA\xBA" }, + { 0xF9ED, "\xE5\x90\x9D" }, + { 0xF9EE, "\xE7\x87\x90" }, + { 0xF9EF, "\xE7\x92\x98" }, + { 0xF9F0, "\xE8\x97\xBA" }, + { 0xF9F1, "\xE9\x9A\xA3" }, + { 0xF9F2, "\xE9\xB1\x97" }, + { 0xF9F3, "\xE9\xBA\x9F" }, + { 0xF9F4, "\xE6\x9E\x97" }, + { 0xF9F5, "\xE6\xB7\x8B" }, + { 0xF9F6, "\xE8\x87\xA8" }, + { 0xF9F7, "\xE7\xAB\x8B" }, + { 0xF9F8, "\xE7\xAC\xA0" }, + { 0xF9F9, "\xE7\xB2\x92" }, + { 0xF9FA, "\xE7\x8B\x80" }, + { 0xF9FB, "\xE7\x82\x99" }, + { 0xF9FC, "\xE8\xAD\x98" }, + { 0xF9FD, "\xE4\xBB\x80" }, + { 0xF9FE, "\xE8\x8C\xB6" }, + { 0xF9FF, "\xE5\x88\xBA" }, + { 0xFA00, "\xE5\x88\x87" }, + { 0xFA01, "\xE5\xBA\xA6" }, + { 0xFA02, "\xE6\x8B\x93" }, + { 0xFA03, "\xE7\xB3\x96" }, + { 0xFA04, "\xE5\xAE\x85" }, + { 0xFA05, "\xE6\xB4\x9E" }, + { 0xFA06, "\xE6\x9A\xB4" }, + { 0xFA07, "\xE8\xBC\xBB" }, + { 0xFA08, "\xE8\xA1\x8C" }, + { 0xFA09, "\xE9\x99\x8D" }, + { 0xFA0A, "\xE8\xA6\x8B" }, + { 0xFA0B, "\xE5\xBB\x93" }, + { 0xFA0C, "\xE5\x85\x80" }, + { 0xFA0D, "\xE5\x97\x80" }, + { 0xFA10, "\xE5\xA1\x9A" }, + { 0xFA12, "\xE6\x99\xB4" }, + { 0xFA15, "\xE5\x87\x9E" }, + { 0xFA16, "\xE7\x8C\xAA" }, + { 0xFA17, "\xE7\x9B\x8A" }, + { 0xFA18, "\xE7\xA4\xBC" }, + { 0xFA19, "\xE7\xA5\x9E" }, + { 0xFA1A, "\xE7\xA5\xA5" }, + { 0xFA1B, "\xE7\xA6\x8F" }, + { 0xFA1C, "\xE9\x9D\x96" }, + { 0xFA1D, "\xE7\xB2\xBE" }, + { 0xFA1E, "\xE7\xBE\xBD" }, + { 0xFA20, "\xE8\x98\x92" }, + { 0xFA22, "\xE8\xAB\xB8" }, + { 0xFA25, "\xE9\x80\xB8" }, + { 0xFA26, "\xE9\x83\xBD" }, + { 0xFA2A, "\xE9\xA3\xAF" }, + { 0xFA2B, "\xE9\xA3\xBC" }, + { 0xFA2C, "\xE9\xA4\xA8" }, + { 0xFA2D, "\xE9\xB6\xB4" }, + { 0xFA2E, "\xE9\x83\x9E" }, + { 0xFA2F, "\xE9\x9A\xB7" }, + { 0xFA30, "\xE4\xBE\xAE" }, + { 0xFA31, "\xE5\x83\xA7" }, + { 0xFA32, "\xE5\x85\x8D" }, + { 0xFA33, "\xE5\x8B\x89" }, + { 0xFA34, "\xE5\x8B\xA4" }, + { 0xFA35, "\xE5\x8D\x91" }, + { 0xFA36, "\xE5\x96\x9D" }, + { 0xFA37, "\xE5\x98\x86" }, + { 0xFA38, "\xE5\x99\xA8" }, + { 0xFA39, "\xE5\xA1\x80" }, + { 0xFA3A, "\xE5\xA2\xA8" }, + { 0xFA3B, "\xE5\xB1\xA4" }, + { 0xFA3C, "\xE5\xB1\xAE" }, + { 0xFA3D, "\xE6\x82\x94" }, + { 0xFA3E, "\xE6\x85\xA8" }, + { 0xFA3F, "\xE6\x86\x8E" }, + { 0xFA40, "\xE6\x87\xB2" }, + { 0xFA41, "\xE6\x95\x8F" }, + { 0xFA42, "\xE6\x97\xA2" }, + { 0xFA43, "\xE6\x9A\x91" }, + { 0xFA44, "\xE6\xA2\x85" }, + { 0xFA45, "\xE6\xB5\xB7" }, + { 0xFA46, "\xE6\xB8\x9A" }, + { 0xFA47, "\xE6\xBC\xA2" }, + { 0xFA48, "\xE7\x85\xAE" }, + { 0xFA49, "\xE7\x88\xAB" }, + { 0xFA4A, "\xE7\x90\xA2" }, + { 0xFA4B, "\xE7\xA2\x91" }, + { 0xFA4C, "\xE7\xA4\xBE" }, + { 0xFA4D, "\xE7\xA5\x89" }, + { 0xFA4E, "\xE7\xA5\x88" }, + { 0xFA4F, "\xE7\xA5\x90" }, + { 0xFA50, "\xE7\xA5\x96" }, + { 0xFA51, "\xE7\xA5\x9D" }, + { 0xFA52, "\xE7\xA6\x8D" }, + { 0xFA53, "\xE7\xA6\x8E" }, + { 0xFA54, "\xE7\xA9\x80" }, + { 0xFA55, "\xE7\xAA\x81" }, + { 0xFA56, "\xE7\xAF\x80" }, + { 0xFA57, "\xE7\xB7\xB4" }, + { 0xFA58, "\xE7\xB8\x89" }, + { 0xFA59, "\xE7\xB9\x81" }, + { 0xFA5A, "\xE7\xBD\xB2" }, + { 0xFA5B, "\xE8\x80\x85" }, + { 0xFA5C, "\xE8\x87\xAD" }, + { 0xFA5D, "\xE8\x89\xB9" }, + { 0xFA5E, "\xE8\x89\xB9" }, + { 0xFA5F, "\xE8\x91\x97" }, + { 0xFA60, "\xE8\xA4\x90" }, + { 0xFA61, "\xE8\xA6\x96" }, + { 0xFA62, "\xE8\xAC\x81" }, + { 0xFA63, "\xE8\xAC\xB9" }, + { 0xFA64, "\xE8\xB3\x93" }, + { 0xFA65, "\xE8\xB4\x88" }, + { 0xFA66, "\xE8\xBE\xB6" }, + { 0xFA67, "\xE9\x80\xB8" }, + { 0xFA68, "\xE9\x9B\xA3" }, + { 0xFA69, "\xE9\x9F\xBF" }, + { 0xFA6A, "\xE9\xA0\xBB" }, + { 0xFA6B, "\xE6\x81\xB5" }, + { 0xFA6C, "\xF0\xA4\x8B\xAE" }, + { 0xFA6D, "\xE8\x88\x98" }, + { 0xFA70, "\xE4\xB8\xA6" }, + { 0xFA71, "\xE5\x86\xB5" }, + { 0xFA72, "\xE5\x85\xA8" }, + { 0xFA73, "\xE4\xBE\x80" }, + { 0xFA74, "\xE5\x85\x85" }, + { 0xFA75, "\xE5\x86\x80" }, + { 0xFA76, "\xE5\x8B\x87" }, + { 0xFA77, "\xE5\x8B\xBA" }, + { 0xFA78, "\xE5\x96\x9D" }, + { 0xFA79, "\xE5\x95\x95" }, + { 0xFA7A, "\xE5\x96\x99" }, + { 0xFA7B, "\xE5\x97\xA2" }, + { 0xFA7C, "\xE5\xA1\x9A" }, + { 0xFA7D, "\xE5\xA2\xB3" }, + { 0xFA7E, "\xE5\xA5\x84" }, + { 0xFA7F, "\xE5\xA5\x94" }, + { 0xFA80, "\xE5\xA9\xA2" }, + { 0xFA81, "\xE5\xAC\xA8" }, + { 0xFA82, "\xE5\xBB\x92" }, + { 0xFA83, "\xE5\xBB\x99" }, + { 0xFA84, "\xE5\xBD\xA9" }, + { 0xFA85, "\xE5\xBE\xAD" }, + { 0xFA86, "\xE6\x83\x98" }, + { 0xFA87, "\xE6\x85\x8E" }, + { 0xFA88, "\xE6\x84\x88" }, + { 0xFA89, "\xE6\x86\x8E" }, + { 0xFA8A, "\xE6\x85\xA0" }, + { 0xFA8B, "\xE6\x87\xB2" }, + { 0xFA8C, "\xE6\x88\xB4" }, + { 0xFA8D, "\xE6\x8F\x84" }, + { 0xFA8E, "\xE6\x90\x9C" }, + { 0xFA8F, "\xE6\x91\x92" }, + { 0xFA90, "\xE6\x95\x96" }, + { 0xFA91, "\xE6\x99\xB4" }, + { 0xFA92, "\xE6\x9C\x97" }, + { 0xFA93, "\xE6\x9C\x9B" }, + { 0xFA94, "\xE6\x9D\x96" }, + { 0xFA95, "\xE6\xAD\xB9" }, + { 0xFA96, "\xE6\xAE\xBA" }, + { 0xFA97, "\xE6\xB5\x81" }, + { 0xFA98, "\xE6\xBB\x9B" }, + { 0xFA99, "\xE6\xBB\x8B" }, + { 0xFA9A, "\xE6\xBC\xA2" }, + { 0xFA9B, "\xE7\x80\x9E" }, + { 0xFA9C, "\xE7\x85\xAE" }, + { 0xFA9D, "\xE7\x9E\xA7" }, + { 0xFA9E, "\xE7\x88\xB5" }, + { 0xFA9F, "\xE7\x8A\xAF" }, + { 0xFAA0, "\xE7\x8C\xAA" }, + { 0xFAA1, "\xE7\x91\xB1" }, + { 0xFAA2, "\xE7\x94\x86" }, + { 0xFAA3, "\xE7\x94\xBB" }, + { 0xFAA4, "\xE7\x98\x9D" }, + { 0xFAA5, "\xE7\x98\x9F" }, + { 0xFAA6, "\xE7\x9B\x8A" }, + { 0xFAA7, "\xE7\x9B\x9B" }, + { 0xFAA8, "\xE7\x9B\xB4" }, + { 0xFAA9, "\xE7\x9D\x8A" }, + { 0xFAAA, "\xE7\x9D\x80" }, + { 0xFAAB, "\xE7\xA3\x8C" }, + { 0xFAAC, "\xE7\xAA\xB1" }, + { 0xFAAD, "\xE7\xAF\x80" }, + { 0xFAAE, "\xE7\xB1\xBB" }, + { 0xFAAF, "\xE7\xB5\x9B" }, + { 0xFAB0, "\xE7\xB7\xB4" }, + { 0xFAB1, "\xE7\xBC\xBE" }, + { 0xFAB2, "\xE8\x80\x85" }, + { 0xFAB3, "\xE8\x8D\x92" }, + { 0xFAB4, "\xE8\x8F\xAF" }, + { 0xFAB5, "\xE8\x9D\xB9" }, + { 0xFAB6, "\xE8\xA5\x81" }, + { 0xFAB7, "\xE8\xA6\x86" }, + { 0xFAB8, "\xE8\xA6\x96" }, + { 0xFAB9, "\xE8\xAA\xBF" }, + { 0xFABA, "\xE8\xAB\xB8" }, + { 0xFABB, "\xE8\xAB\x8B" }, + { 0xFABC, "\xE8\xAC\x81" }, + { 0xFABD, "\xE8\xAB\xBE" }, + { 0xFABE, "\xE8\xAB\xAD" }, + { 0xFABF, "\xE8\xAC\xB9" }, + { 0xFAC0, "\xE8\xAE\x8A" }, + { 0xFAC1, "\xE8\xB4\x88" }, + { 0xFAC2, "\xE8\xBC\xB8" }, + { 0xFAC3, "\xE9\x81\xB2" }, + { 0xFAC4, "\xE9\x86\x99" }, + { 0xFAC5, "\xE9\x89\xB6" }, + { 0xFAC6, "\xE9\x99\xBC" }, + { 0xFAC7, "\xE9\x9B\xA3" }, + { 0xFAC8, "\xE9\x9D\x96" }, + { 0xFAC9, "\xE9\x9F\x9B" }, + { 0xFACA, "\xE9\x9F\xBF" }, + { 0xFACB, "\xE9\xA0\x8B" }, + { 0xFACC, "\xE9\xA0\xBB" }, + { 0xFACD, "\xE9\xAC\x92" }, + { 0xFACE, "\xE9\xBE\x9C" }, + { 0xFACF, "\xF0\xA2\xA1\x8A" }, + { 0xFAD0, "\xF0\xA2\xA1\x84" }, + { 0xFAD1, "\xF0\xA3\x8F\x95" }, + { 0xFAD2, "\xE3\xAE\x9D" }, + { 0xFAD3, "\xE4\x80\x98" }, + { 0xFAD4, "\xE4\x80\xB9" }, + { 0xFAD5, "\xF0\xA5\x89\x89" }, + { 0xFAD6, "\xF0\xA5\xB3\x90" }, + { 0xFAD7, "\xF0\xA7\xBB\x93" }, + { 0xFAD8, "\xE9\xBD\x83" }, + { 0xFAD9, "\xE9\xBE\x8E" }, + { 0xFB1D, "\xD7\x99" }, + { 0xFB1F, "\xD7\xB2" }, + { 0xFB2A, "\xD7\xA9" }, + { 0xFB2B, "\xD7\xA9" }, + { 0xFB2C, "\xD7\xA9" }, + { 0xFB2D, "\xD7\xA9" }, + { 0xFB2E, "\xD7\x90" }, + { 0xFB2F, "\xD7\x90" }, + { 0xFB30, "\xD7\x90" }, + { 0xFB31, "\xD7\x91" }, + { 0xFB32, "\xD7\x92" }, + { 0xFB33, "\xD7\x93" }, + { 0xFB34, "\xD7\x94" }, + { 0xFB35, "\xD7\x95" }, + { 0xFB36, "\xD7\x96" }, + { 0xFB38, "\xD7\x98" }, + { 0xFB39, "\xD7\x99" }, + { 0xFB3A, "\xD7\x9A" }, + { 0xFB3B, "\xD7\x9B" }, + { 0xFB3C, "\xD7\x9C" }, + { 0xFB3E, "\xD7\x9E" }, + { 0xFB40, "\xD7\xA0" }, + { 0xFB41, "\xD7\xA1" }, + { 0xFB43, "\xD7\xA3" }, + { 0xFB44, "\xD7\xA4" }, + { 0xFB46, "\xD7\xA6" }, + { 0xFB47, "\xD7\xA7" }, + { 0xFB48, "\xD7\xA8" }, + { 0xFB49, "\xD7\xA9" }, + { 0xFB4A, "\xD7\xAA" }, + { 0xFB4B, "\xD7\x95" }, + { 0xFB4C, "\xD7\x91" }, + { 0xFB4D, "\xD7\x9B" }, + { 0xFB4E, "\xD7\xA4" }, + { 0xFF21, "\xEF\xBD\x81" }, + { 0xFF22, "\xEF\xBD\x82" }, + { 0xFF23, "\xEF\xBD\x83" }, + { 0xFF24, "\xEF\xBD\x84" }, + { 0xFF25, "\xEF\xBD\x85" }, + { 0xFF26, "\xEF\xBD\x86" }, + { 0xFF27, "\xEF\xBD\x87" }, + { 0xFF28, "\xEF\xBD\x88" }, + { 0xFF29, "\xEF\xBD\x89" }, + { 0xFF2A, "\xEF\xBD\x8A" }, + { 0xFF2B, "\xEF\xBD\x8B" }, + { 0xFF2C, "\xEF\xBD\x8C" }, + { 0xFF2D, "\xEF\xBD\x8D" }, + { 0xFF2E, "\xEF\xBD\x8E" }, + { 0xFF2F, "\xEF\xBD\x8F" }, + { 0xFF30, "\xEF\xBD\x90" }, + { 0xFF31, "\xEF\xBD\x91" }, + { 0xFF32, "\xEF\xBD\x92" }, + { 0xFF33, "\xEF\xBD\x93" }, + { 0xFF34, "\xEF\xBD\x94" }, + { 0xFF35, "\xEF\xBD\x95" }, + { 0xFF36, "\xEF\xBD\x96" }, + { 0xFF37, "\xEF\xBD\x97" }, + { 0xFF38, "\xEF\xBD\x98" }, + { 0xFF39, "\xEF\xBD\x99" }, + { 0xFF3A, "\xEF\xBD\x9A" }, +}; + +constexpr FoldEntry FOLD_ORIGINAL[] = { + { 0x0370, "\xCD\xB1" }, + { 0x0372, "\xCD\xB3" }, + { 0x0374, "\xCA\xB9" }, + { 0x0376, "\xCD\xB7" }, + { 0x037F, "\xCF\xB3" }, + { 0x0386, "\xCE\xB1" }, + { 0x0388, "\xCE\xB5" }, + { 0x0389, "\xCE\xB7" }, + { 0x038A, "\xCE\xB9" }, + { 0x038C, "\xCE\xBF" }, + { 0x038E, "\xCF\x85" }, + { 0x038F, "\xCF\x89" }, + { 0x0390, "\xCE\xB9" }, + { 0x0391, "\xCE\xB1" }, + { 0x0392, "\xCE\xB2" }, + { 0x0393, "\xCE\xB3" }, + { 0x0394, "\xCE\xB4" }, + { 0x0395, "\xCE\xB5" }, + { 0x0396, "\xCE\xB6" }, + { 0x0397, "\xCE\xB7" }, + { 0x0398, "\xCE\xB8" }, + { 0x0399, "\xCE\xB9" }, + { 0x039A, "\xCE\xBA" }, + { 0x039B, "\xCE\xBB" }, + { 0x039C, "\xCE\xBC" }, + { 0x039D, "\xCE\xBD" }, + { 0x039E, "\xCE\xBE" }, + { 0x039F, "\xCE\xBF" }, + { 0x03A0, "\xCF\x80" }, + { 0x03A1, "\xCF\x81" }, + { 0x03A3, "\xCF\x83" }, + { 0x03A4, "\xCF\x84" }, + { 0x03A5, "\xCF\x85" }, + { 0x03A6, "\xCF\x86" }, + { 0x03A7, "\xCF\x87" }, + { 0x03A8, "\xCF\x88" }, + { 0x03A9, "\xCF\x89" }, + { 0x03AA, "\xCE\xB9" }, + { 0x03AB, "\xCF\x85" }, + { 0x03AC, "\xCE\xB1" }, + { 0x03AD, "\xCE\xB5" }, + { 0x03AE, "\xCE\xB7" }, + { 0x03AF, "\xCE\xB9" }, + { 0x03B0, "\xCF\x85" }, + { 0x03C2, "\xCF\x83" }, + { 0x03CA, "\xCE\xB9" }, + { 0x03CB, "\xCF\x85" }, + { 0x03CC, "\xCE\xBF" }, + { 0x03CD, "\xCF\x85" }, + { 0x03CE, "\xCF\x89" }, + { 0x03CF, "\xCF\x97" }, + { 0x03D3, "\xCF\x92" }, + { 0x03D4, "\xCF\x92" }, + { 0x03D8, "\xCF\x99" }, + { 0x03DA, "\xCF\x9B" }, + { 0x03DC, "\xCF\x9D" }, + { 0x03DE, "\xCF\x9F" }, + { 0x03E0, "\xCF\xA1" }, + { 0x03E2, "\xCF\xA3" }, + { 0x03E4, "\xCF\xA5" }, + { 0x03E6, "\xCF\xA7" }, + { 0x03E8, "\xCF\xA9" }, + { 0x03EA, "\xCF\xAB" }, + { 0x03EC, "\xCF\xAD" }, + { 0x03EE, "\xCF\xAF" }, + { 0x03F4, "\xCE\xB8" }, + { 0x03F7, "\xCF\xB8" }, + { 0x03F9, "\xCF\xB2" }, + { 0x03FA, "\xCF\xBB" }, + { 0x03FD, "\xCD\xBB" }, + { 0x03FE, "\xCD\xBC" }, + { 0x03FF, "\xCD\xBD" }, + { 0x0400, "\xD1\x90" }, + { 0x0401, "\xD0\xB5" }, + { 0x0402, "\xD1\x92" }, + { 0x0403, "\xD1\x93" }, + { 0x0404, "\xD1\x94" }, + { 0x0405, "\xD1\x95" }, + { 0x0406, "\xD1\x96" }, + { 0x0407, "\xD1\x97" }, + { 0x0408, "\xD1\x98" }, + { 0x0409, "\xD1\x99" }, + { 0x040A, "\xD1\x9A" }, + { 0x040B, "\xD1\x9B" }, + { 0x040C, "\xD1\x9C" }, + { 0x040D, "\xD1\x9D" }, + { 0x040E, "\xD1\x9E" }, + { 0x040F, "\xD1\x9F" }, + { 0x0410, "\xD0\xB0" }, + { 0x0411, "\xD0\xB1" }, + { 0x0412, "\xD0\xB2" }, + { 0x0413, "\xD0\xB3" }, + { 0x0414, "\xD0\xB4" }, + { 0x0415, "\xD0\xB5" }, + { 0x0416, "\xD0\xB6" }, + { 0x0417, "\xD0\xB7" }, + { 0x0418, "\xD0\xB8" }, + { 0x0419, "\xD0\xB9" }, + { 0x041A, "\xD0\xBA" }, + { 0x041B, "\xD0\xBB" }, + { 0x041C, "\xD0\xBC" }, + { 0x041D, "\xD0\xBD" }, + { 0x041E, "\xD0\xBE" }, + { 0x041F, "\xD0\xBF" }, + { 0x0420, "\xD1\x80" }, + { 0x0421, "\xD1\x81" }, + { 0x0422, "\xD1\x82" }, + { 0x0423, "\xD1\x83" }, + { 0x0424, "\xD1\x84" }, + { 0x0425, "\xD1\x85" }, + { 0x0426, "\xD1\x86" }, + { 0x0427, "\xD1\x87" }, + { 0x0428, "\xD1\x88" }, + { 0x0429, "\xD1\x89" }, + { 0x042A, "\xD1\x8A" }, + { 0x042B, "\xD1\x8B" }, + { 0x042C, "\xD1\x8C" }, + { 0x042D, "\xD1\x8D" }, + { 0x042E, "\xD1\x8E" }, + { 0x042F, "\xD1\x8F" }, + { 0x0451, "\xD0\xB5" }, + { 0x0460, "\xD1\xA1" }, + { 0x0462, "\xD1\xA3" }, + { 0x0464, "\xD1\xA5" }, + { 0x0466, "\xD1\xA7" }, + { 0x0468, "\xD1\xA9" }, + { 0x046A, "\xD1\xAB" }, + { 0x046C, "\xD1\xAD" }, + { 0x046E, "\xD1\xAF" }, + { 0x0470, "\xD1\xB1" }, + { 0x0472, "\xD1\xB3" }, + { 0x0474, "\xD1\xB5" }, + { 0x0476, "\xD1\xB7" }, + { 0x0478, "\xD1\xB9" }, + { 0x047A, "\xD1\xBB" }, + { 0x047C, "\xD1\xBD" }, + { 0x047E, "\xD1\xBF" }, + { 0x0480, "\xD2\x81" }, + { 0x048A, "\xD2\x8B" }, + { 0x048C, "\xD2\x8D" }, + { 0x048E, "\xD2\x8F" }, + { 0x0490, "\xD0\xB3" }, + { 0x0491, "\xD0\xB3" }, + { 0x0492, "\xD2\x93" }, + { 0x0494, "\xD2\x95" }, + { 0x0496, "\xD2\x97" }, + { 0x0498, "\xD2\x99" }, + { 0x049A, "\xD2\x9B" }, + { 0x049C, "\xD2\x9D" }, + { 0x049E, "\xD2\x9F" }, + { 0x04A0, "\xD2\xA1" }, + { 0x04A2, "\xD2\xA3" }, + { 0x04A4, "\xD2\xA5" }, + { 0x04A6, "\xD2\xA7" }, + { 0x04A8, "\xD2\xA9" }, + { 0x04AA, "\xD2\xAB" }, + { 0x04AC, "\xD2\xAD" }, + { 0x04AE, "\xD2\xAF" }, + { 0x04B0, "\xD2\xB1" }, + { 0x04B2, "\xD2\xB3" }, + { 0x04B4, "\xD2\xB5" }, + { 0x04B6, "\xD2\xB7" }, + { 0x04B8, "\xD2\xB9" }, + { 0x04BA, "\xD2\xBB" }, + { 0x04BC, "\xD2\xBD" }, + { 0x04BE, "\xD2\xBF" }, + { 0x04C0, "\xD3\x8F" }, + { 0x04C1, "\xD3\x82" }, + { 0x04C3, "\xD3\x84" }, + { 0x04C5, "\xD3\x86" }, + { 0x04C7, "\xD3\x88" }, + { 0x04C9, "\xD3\x8A" }, + { 0x04CB, "\xD3\x8C" }, + { 0x04CD, "\xD3\x8E" }, + { 0x04D0, "\xD3\x91" }, + { 0x04D2, "\xD3\x93" }, + { 0x04D4, "\xD3\x95" }, + { 0x04D6, "\xD3\x97" }, + { 0x04D8, "\xD3\x99" }, + { 0x04DA, "\xD3\x9B" }, + { 0x04DC, "\xD3\x9D" }, + { 0x04DE, "\xD3\x9F" }, + { 0x04E0, "\xD3\xA1" }, + { 0x04E2, "\xD3\xA3" }, + { 0x04E4, "\xD3\xA5" }, + { 0x04E6, "\xD3\xA7" }, + { 0x04E8, "\xD3\xA9" }, + { 0x04EA, "\xD3\xAB" }, + { 0x04EC, "\xD3\xAD" }, + { 0x04EE, "\xD3\xAF" }, + { 0x04F0, "\xD3\xB1" }, + { 0x04F2, "\xD3\xB3" }, + { 0x04F4, "\xD3\xB5" }, + { 0x04F6, "\xD3\xB7" }, + { 0x04F8, "\xD3\xB9" }, + { 0x04FA, "\xD3\xBB" }, + { 0x04FC, "\xD3\xBD" }, + { 0x04FE, "\xD3\xBF" }, + { 0x0500, "\xD4\x81" }, + { 0x0502, "\xD4\x83" }, + { 0x0504, "\xD4\x85" }, + { 0x0506, "\xD4\x87" }, + { 0x0508, "\xD4\x89" }, + { 0x050A, "\xD4\x8B" }, + { 0x050C, "\xD4\x8D" }, + { 0x050E, "\xD4\x8F" }, + { 0x0510, "\xD4\x91" }, + { 0x0512, "\xD4\x93" }, + { 0x0514, "\xD4\x95" }, + { 0x0516, "\xD4\x97" }, + { 0x0518, "\xD4\x99" }, + { 0x051A, "\xD4\x9B" }, + { 0x051C, "\xD4\x9D" }, + { 0x051E, "\xD4\x9F" }, + { 0x0520, "\xD4\xA1" }, + { 0x0522, "\xD4\xA3" }, + { 0x0524, "\xD4\xA5" }, + { 0x0526, "\xD4\xA7" }, + { 0x0528, "\xD4\xA9" }, + { 0x052A, "\xD4\xAB" }, + { 0x052C, "\xD4\xAD" }, + { 0x052E, "\xD4\xAF" }, +}; + +constexpr FoldEntry FOLD_LOWER[] = { + { 0x0041, "\x61" }, + { 0x0042, "\x62" }, + { 0x0043, "\x63" }, + { 0x0044, "\x64" }, + { 0x0045, "\x65" }, + { 0x0046, "\x66" }, + { 0x0047, "\x67" }, + { 0x0048, "\x68" }, + { 0x0049, "\x69" }, + { 0x004A, "\x6A" }, + { 0x004B, "\x6B" }, + { 0x004C, "\x6C" }, + { 0x004D, "\x6D" }, + { 0x004E, "\x6E" }, + { 0x004F, "\x6F" }, + { 0x0050, "\x70" }, + { 0x0051, "\x71" }, + { 0x0052, "\x72" }, + { 0x0053, "\x73" }, + { 0x0054, "\x74" }, + { 0x0055, "\x75" }, + { 0x0056, "\x76" }, + { 0x0057, "\x77" }, + { 0x0058, "\x78" }, + { 0x0059, "\x79" }, + { 0x005A, "\x7A" }, + { 0x00C0, "\xC3\xA0" }, + { 0x00C1, "\xC3\xA1" }, + { 0x00C2, "\xC3\xA2" }, + { 0x00C3, "\xC3\xA3" }, + { 0x00C4, "\xC3\xA4" }, + { 0x00C5, "\xC3\xA5" }, + { 0x00C6, "\xC3\xA6" }, + { 0x00C7, "\xC3\xA7" }, + { 0x00C8, "\xC3\xA8" }, + { 0x00C9, "\xC3\xA9" }, + { 0x00CA, "\xC3\xAA" }, + { 0x00CB, "\xC3\xAB" }, + { 0x00CC, "\xC3\xAC" }, + { 0x00CD, "\xC3\xAD" }, + { 0x00CE, "\xC3\xAE" }, + { 0x00CF, "\xC3\xAF" }, + { 0x00D0, "\xC3\xB0" }, + { 0x00D1, "\xC3\xB1" }, + { 0x00D2, "\xC3\xB2" }, + { 0x00D3, "\xC3\xB3" }, + { 0x00D4, "\xC3\xB4" }, + { 0x00D5, "\xC3\xB5" }, + { 0x00D6, "\xC3\xB6" }, + { 0x00D8, "\xC3\xB8" }, + { 0x00D9, "\xC3\xB9" }, + { 0x00DA, "\xC3\xBA" }, + { 0x00DB, "\xC3\xBB" }, + { 0x00DC, "\xC3\xBC" }, + { 0x00DD, "\xC3\xBD" }, + { 0x00DE, "\xC3\xBE" }, + { 0x0100, "\xC4\x81" }, + { 0x0102, "\xC4\x83" }, + { 0x0104, "\xC4\x85" }, + { 0x0106, "\xC4\x87" }, + { 0x0108, "\xC4\x89" }, + { 0x010A, "\xC4\x8B" }, + { 0x010C, "\xC4\x8D" }, + { 0x010E, "\xC4\x8F" }, + { 0x0110, "\xC4\x91" }, + { 0x0112, "\xC4\x93" }, + { 0x0114, "\xC4\x95" }, + { 0x0116, "\xC4\x97" }, + { 0x0118, "\xC4\x99" }, + { 0x011A, "\xC4\x9B" }, + { 0x011C, "\xC4\x9D" }, + { 0x011E, "\xC4\x9F" }, + { 0x0120, "\xC4\xA1" }, + { 0x0122, "\xC4\xA3" }, + { 0x0124, "\xC4\xA5" }, + { 0x0126, "\xC4\xA7" }, + { 0x0128, "\xC4\xA9" }, + { 0x012A, "\xC4\xAB" }, + { 0x012C, "\xC4\xAD" }, + { 0x012E, "\xC4\xAF" }, + { 0x0130, "\x69" }, + { 0x0132, "\xC4\xB3" }, + { 0x0134, "\xC4\xB5" }, + { 0x0136, "\xC4\xB7" }, + { 0x0139, "\xC4\xBA" }, + { 0x013B, "\xC4\xBC" }, + { 0x013D, "\xC4\xBE" }, + { 0x013F, "\xC5\x80" }, + { 0x0141, "\xC5\x82" }, + { 0x0143, "\xC5\x84" }, + { 0x0145, "\xC5\x86" }, + { 0x0147, "\xC5\x88" }, + { 0x014A, "\xC5\x8B" }, + { 0x014C, "\xC5\x8D" }, + { 0x014E, "\xC5\x8F" }, + { 0x0150, "\xC5\x91" }, + { 0x0152, "\xC5\x93" }, + { 0x0154, "\xC5\x95" }, + { 0x0156, "\xC5\x97" }, + { 0x0158, "\xC5\x99" }, + { 0x015A, "\xC5\x9B" }, + { 0x015C, "\xC5\x9D" }, + { 0x015E, "\xC5\x9F" }, + { 0x0160, "\xC5\xA1" }, + { 0x0162, "\xC5\xA3" }, + { 0x0164, "\xC5\xA5" }, + { 0x0166, "\xC5\xA7" }, + { 0x0168, "\xC5\xA9" }, + { 0x016A, "\xC5\xAB" }, + { 0x016C, "\xC5\xAD" }, + { 0x016E, "\xC5\xAF" }, + { 0x0170, "\xC5\xB1" }, + { 0x0172, "\xC5\xB3" }, + { 0x0174, "\xC5\xB5" }, + { 0x0176, "\xC5\xB7" }, + { 0x0178, "\xC3\xBF" }, + { 0x0179, "\xC5\xBA" }, + { 0x017B, "\xC5\xBC" }, + { 0x017D, "\xC5\xBE" }, + { 0x0181, "\xC9\x93" }, + { 0x0182, "\xC6\x83" }, + { 0x0184, "\xC6\x85" }, + { 0x0186, "\xC9\x94" }, + { 0x0187, "\xC6\x88" }, + { 0x0189, "\xC9\x96" }, + { 0x018A, "\xC9\x97" }, + { 0x018B, "\xC6\x8C" }, + { 0x018E, "\xC7\x9D" }, + { 0x018F, "\xC9\x99" }, + { 0x0190, "\xC9\x9B" }, + { 0x0191, "\xC6\x92" }, + { 0x0193, "\xC9\xA0" }, + { 0x0194, "\xC9\xA3" }, + { 0x0196, "\xC9\xA9" }, + { 0x0197, "\xC9\xA8" }, + { 0x0198, "\xC6\x99" }, + { 0x019C, "\xC9\xAF" }, + { 0x019D, "\xC9\xB2" }, + { 0x019F, "\xC9\xB5" }, + { 0x01A0, "\xC6\xA1" }, + { 0x01A2, "\xC6\xA3" }, + { 0x01A4, "\xC6\xA5" }, + { 0x01A6, "\xCA\x80" }, + { 0x01A7, "\xC6\xA8" }, + { 0x01A9, "\xCA\x83" }, + { 0x01AC, "\xC6\xAD" }, + { 0x01AE, "\xCA\x88" }, + { 0x01AF, "\xC6\xB0" }, + { 0x01B1, "\xCA\x8A" }, + { 0x01B2, "\xCA\x8B" }, + { 0x01B3, "\xC6\xB4" }, + { 0x01B5, "\xC6\xB6" }, + { 0x01B7, "\xCA\x92" }, + { 0x01B8, "\xC6\xB9" }, + { 0x01BC, "\xC6\xBD" }, + { 0x01C4, "\xC7\x86" }, + { 0x01C5, "\xC7\x86" }, + { 0x01C7, "\xC7\x89" }, + { 0x01C8, "\xC7\x89" }, + { 0x01CA, "\xC7\x8C" }, + { 0x01CB, "\xC7\x8C" }, + { 0x01CD, "\xC7\x8E" }, + { 0x01CF, "\xC7\x90" }, + { 0x01D1, "\xC7\x92" }, + { 0x01D3, "\xC7\x94" }, + { 0x01D5, "\xC7\x96" }, + { 0x01D7, "\xC7\x98" }, + { 0x01D9, "\xC7\x9A" }, + { 0x01DB, "\xC7\x9C" }, + { 0x01DE, "\xC7\x9F" }, + { 0x01E0, "\xC7\xA1" }, + { 0x01E2, "\xC7\xA3" }, + { 0x01E4, "\xC7\xA5" }, + { 0x01E6, "\xC7\xA7" }, + { 0x01E8, "\xC7\xA9" }, + { 0x01EA, "\xC7\xAB" }, + { 0x01EC, "\xC7\xAD" }, + { 0x01EE, "\xC7\xAF" }, + { 0x01F1, "\xC7\xB3" }, + { 0x01F2, "\xC7\xB3" }, + { 0x01F4, "\xC7\xB5" }, + { 0x01F6, "\xC6\x95" }, + { 0x01F7, "\xC6\xBF" }, + { 0x01F8, "\xC7\xB9" }, + { 0x01FA, "\xC7\xBB" }, + { 0x01FC, "\xC7\xBD" }, + { 0x01FE, "\xC7\xBF" }, + { 0x0200, "\xC8\x81" }, + { 0x0202, "\xC8\x83" }, + { 0x0204, "\xC8\x85" }, + { 0x0206, "\xC8\x87" }, + { 0x0208, "\xC8\x89" }, + { 0x020A, "\xC8\x8B" }, + { 0x020C, "\xC8\x8D" }, + { 0x020E, "\xC8\x8F" }, + { 0x0210, "\xC8\x91" }, + { 0x0212, "\xC8\x93" }, + { 0x0214, "\xC8\x95" }, + { 0x0216, "\xC8\x97" }, + { 0x0218, "\xC8\x99" }, + { 0x021A, "\xC8\x9B" }, + { 0x021C, "\xC8\x9D" }, + { 0x021E, "\xC8\x9F" }, + { 0x0220, "\xC6\x9E" }, + { 0x0222, "\xC8\xA3" }, + { 0x0224, "\xC8\xA5" }, + { 0x0226, "\xC8\xA7" }, + { 0x0228, "\xC8\xA9" }, + { 0x022A, "\xC8\xAB" }, + { 0x022C, "\xC8\xAD" }, + { 0x022E, "\xC8\xAF" }, + { 0x0230, "\xC8\xB1" }, + { 0x0232, "\xC8\xB3" }, + { 0x023A, "\xE2\xB1\xA5" }, + { 0x023B, "\xC8\xBC" }, + { 0x023D, "\xC6\x9A" }, + { 0x023E, "\xE2\xB1\xA6" }, + { 0x0241, "\xC9\x82" }, + { 0x0243, "\xC6\x80" }, + { 0x0244, "\xCA\x89" }, + { 0x0245, "\xCA\x8C" }, + { 0x0246, "\xC9\x87" }, + { 0x0248, "\xC9\x89" }, + { 0x024A, "\xC9\x8B" }, + { 0x024C, "\xC9\x8D" }, + { 0x024E, "\xC9\x8F" }, + { 0x0370, "\xCD\xB1" }, + { 0x0372, "\xCD\xB3" }, + { 0x0376, "\xCD\xB7" }, + { 0x037F, "\xCF\xB3" }, + { 0x0386, "\xCE\xAC" }, + { 0x0388, "\xCE\xAD" }, + { 0x0389, "\xCE\xAE" }, + { 0x038A, "\xCE\xAF" }, + { 0x038C, "\xCF\x8C" }, + { 0x038E, "\xCF\x8D" }, + { 0x038F, "\xCF\x8E" }, + { 0x0391, "\xCE\xB1" }, + { 0x0392, "\xCE\xB2" }, + { 0x0393, "\xCE\xB3" }, + { 0x0394, "\xCE\xB4" }, + { 0x0395, "\xCE\xB5" }, + { 0x0396, "\xCE\xB6" }, + { 0x0397, "\xCE\xB7" }, + { 0x0398, "\xCE\xB8" }, + { 0x0399, "\xCE\xB9" }, + { 0x039A, "\xCE\xBA" }, + { 0x039B, "\xCE\xBB" }, + { 0x039C, "\xCE\xBC" }, + { 0x039D, "\xCE\xBD" }, + { 0x039E, "\xCE\xBE" }, + { 0x039F, "\xCE\xBF" }, + { 0x03A0, "\xCF\x80" }, + { 0x03A1, "\xCF\x81" }, + { 0x03A3, "\xCF\x83" }, + { 0x03A4, "\xCF\x84" }, + { 0x03A5, "\xCF\x85" }, + { 0x03A6, "\xCF\x86" }, + { 0x03A7, "\xCF\x87" }, + { 0x03A8, "\xCF\x88" }, + { 0x03A9, "\xCF\x89" }, + { 0x03AA, "\xCF\x8A" }, + { 0x03AB, "\xCF\x8B" }, + { 0x03CF, "\xCF\x97" }, + { 0x03D8, "\xCF\x99" }, + { 0x03DA, "\xCF\x9B" }, + { 0x03DC, "\xCF\x9D" }, + { 0x03DE, "\xCF\x9F" }, + { 0x03E0, "\xCF\xA1" }, + { 0x03E2, "\xCF\xA3" }, + { 0x03E4, "\xCF\xA5" }, + { 0x03E6, "\xCF\xA7" }, + { 0x03E8, "\xCF\xA9" }, + { 0x03EA, "\xCF\xAB" }, + { 0x03EC, "\xCF\xAD" }, + { 0x03EE, "\xCF\xAF" }, + { 0x03F4, "\xCE\xB8" }, + { 0x03F7, "\xCF\xB8" }, + { 0x03F9, "\xCF\xB2" }, + { 0x03FA, "\xCF\xBB" }, + { 0x03FD, "\xCD\xBB" }, + { 0x03FE, "\xCD\xBC" }, + { 0x03FF, "\xCD\xBD" }, + { 0x0400, "\xD1\x90" }, + { 0x0401, "\xD1\x91" }, + { 0x0402, "\xD1\x92" }, + { 0x0403, "\xD1\x93" }, + { 0x0404, "\xD1\x94" }, + { 0x0405, "\xD1\x95" }, + { 0x0406, "\xD1\x96" }, + { 0x0407, "\xD1\x97" }, + { 0x0408, "\xD1\x98" }, + { 0x0409, "\xD1\x99" }, + { 0x040A, "\xD1\x9A" }, + { 0x040B, "\xD1\x9B" }, + { 0x040C, "\xD1\x9C" }, + { 0x040D, "\xD1\x9D" }, + { 0x040E, "\xD1\x9E" }, + { 0x040F, "\xD1\x9F" }, + { 0x0410, "\xD0\xB0" }, + { 0x0411, "\xD0\xB1" }, + { 0x0412, "\xD0\xB2" }, + { 0x0413, "\xD0\xB3" }, + { 0x0414, "\xD0\xB4" }, + { 0x0415, "\xD0\xB5" }, + { 0x0416, "\xD0\xB6" }, + { 0x0417, "\xD0\xB7" }, + { 0x0418, "\xD0\xB8" }, + { 0x0419, "\xD0\xB9" }, + { 0x041A, "\xD0\xBA" }, + { 0x041B, "\xD0\xBB" }, + { 0x041C, "\xD0\xBC" }, + { 0x041D, "\xD0\xBD" }, + { 0x041E, "\xD0\xBE" }, + { 0x041F, "\xD0\xBF" }, + { 0x0420, "\xD1\x80" }, + { 0x0421, "\xD1\x81" }, + { 0x0422, "\xD1\x82" }, + { 0x0423, "\xD1\x83" }, + { 0x0424, "\xD1\x84" }, + { 0x0425, "\xD1\x85" }, + { 0x0426, "\xD1\x86" }, + { 0x0427, "\xD1\x87" }, + { 0x0428, "\xD1\x88" }, + { 0x0429, "\xD1\x89" }, + { 0x042A, "\xD1\x8A" }, + { 0x042B, "\xD1\x8B" }, + { 0x042C, "\xD1\x8C" }, + { 0x042D, "\xD1\x8D" }, + { 0x042E, "\xD1\x8E" }, + { 0x042F, "\xD1\x8F" }, + { 0x0460, "\xD1\xA1" }, + { 0x0462, "\xD1\xA3" }, + { 0x0464, "\xD1\xA5" }, + { 0x0466, "\xD1\xA7" }, + { 0x0468, "\xD1\xA9" }, + { 0x046A, "\xD1\xAB" }, + { 0x046C, "\xD1\xAD" }, + { 0x046E, "\xD1\xAF" }, + { 0x0470, "\xD1\xB1" }, + { 0x0472, "\xD1\xB3" }, + { 0x0474, "\xD1\xB5" }, + { 0x0476, "\xD1\xB7" }, + { 0x0478, "\xD1\xB9" }, + { 0x047A, "\xD1\xBB" }, + { 0x047C, "\xD1\xBD" }, + { 0x047E, "\xD1\xBF" }, + { 0x0480, "\xD2\x81" }, + { 0x048A, "\xD2\x8B" }, + { 0x048C, "\xD2\x8D" }, + { 0x048E, "\xD2\x8F" }, + { 0x0490, "\xD2\x91" }, + { 0x0492, "\xD2\x93" }, + { 0x0494, "\xD2\x95" }, + { 0x0496, "\xD2\x97" }, + { 0x0498, "\xD2\x99" }, + { 0x049A, "\xD2\x9B" }, + { 0x049C, "\xD2\x9D" }, + { 0x049E, "\xD2\x9F" }, + { 0x04A0, "\xD2\xA1" }, + { 0x04A2, "\xD2\xA3" }, + { 0x04A4, "\xD2\xA5" }, + { 0x04A6, "\xD2\xA7" }, + { 0x04A8, "\xD2\xA9" }, + { 0x04AA, "\xD2\xAB" }, + { 0x04AC, "\xD2\xAD" }, + { 0x04AE, "\xD2\xAF" }, + { 0x04B0, "\xD2\xB1" }, + { 0x04B2, "\xD2\xB3" }, + { 0x04B4, "\xD2\xB5" }, + { 0x04B6, "\xD2\xB7" }, + { 0x04B8, "\xD2\xB9" }, + { 0x04BA, "\xD2\xBB" }, + { 0x04BC, "\xD2\xBD" }, + { 0x04BE, "\xD2\xBF" }, + { 0x04C0, "\xD3\x8F" }, + { 0x04C1, "\xD3\x82" }, + { 0x04C3, "\xD3\x84" }, + { 0x04C5, "\xD3\x86" }, + { 0x04C7, "\xD3\x88" }, + { 0x04C9, "\xD3\x8A" }, + { 0x04CB, "\xD3\x8C" }, + { 0x04CD, "\xD3\x8E" }, + { 0x04D0, "\xD3\x91" }, + { 0x04D2, "\xD3\x93" }, + { 0x04D4, "\xD3\x95" }, + { 0x04D6, "\xD3\x97" }, + { 0x04D8, "\xD3\x99" }, + { 0x04DA, "\xD3\x9B" }, + { 0x04DC, "\xD3\x9D" }, + { 0x04DE, "\xD3\x9F" }, + { 0x04E0, "\xD3\xA1" }, + { 0x04E2, "\xD3\xA3" }, + { 0x04E4, "\xD3\xA5" }, + { 0x04E6, "\xD3\xA7" }, + { 0x04E8, "\xD3\xA9" }, + { 0x04EA, "\xD3\xAB" }, + { 0x04EC, "\xD3\xAD" }, + { 0x04EE, "\xD3\xAF" }, + { 0x04F0, "\xD3\xB1" }, + { 0x04F2, "\xD3\xB3" }, + { 0x04F4, "\xD3\xB5" }, + { 0x04F6, "\xD3\xB7" }, + { 0x04F8, "\xD3\xB9" }, + { 0x04FA, "\xD3\xBB" }, + { 0x04FC, "\xD3\xBD" }, + { 0x04FE, "\xD3\xBF" }, + { 0x0500, "\xD4\x81" }, + { 0x0502, "\xD4\x83" }, + { 0x0504, "\xD4\x85" }, + { 0x0506, "\xD4\x87" }, + { 0x0508, "\xD4\x89" }, + { 0x050A, "\xD4\x8B" }, + { 0x050C, "\xD4\x8D" }, + { 0x050E, "\xD4\x8F" }, + { 0x0510, "\xD4\x91" }, + { 0x0512, "\xD4\x93" }, + { 0x0514, "\xD4\x95" }, + { 0x0516, "\xD4\x97" }, + { 0x0518, "\xD4\x99" }, + { 0x051A, "\xD4\x9B" }, + { 0x051C, "\xD4\x9D" }, + { 0x051E, "\xD4\x9F" }, + { 0x0520, "\xD4\xA1" }, + { 0x0522, "\xD4\xA3" }, + { 0x0524, "\xD4\xA5" }, + { 0x0526, "\xD4\xA7" }, + { 0x0528, "\xD4\xA9" }, + { 0x052A, "\xD4\xAB" }, + { 0x052C, "\xD4\xAD" }, + { 0x052E, "\xD4\xAF" }, + { 0x0531, "\xD5\xA1" }, + { 0x0532, "\xD5\xA2" }, + { 0x0533, "\xD5\xA3" }, + { 0x0534, "\xD5\xA4" }, + { 0x0535, "\xD5\xA5" }, + { 0x0536, "\xD5\xA6" }, + { 0x0537, "\xD5\xA7" }, + { 0x0538, "\xD5\xA8" }, + { 0x0539, "\xD5\xA9" }, + { 0x053A, "\xD5\xAA" }, + { 0x053B, "\xD5\xAB" }, + { 0x053C, "\xD5\xAC" }, + { 0x053D, "\xD5\xAD" }, + { 0x053E, "\xD5\xAE" }, + { 0x053F, "\xD5\xAF" }, + { 0x0540, "\xD5\xB0" }, + { 0x0541, "\xD5\xB1" }, + { 0x0542, "\xD5\xB2" }, + { 0x0543, "\xD5\xB3" }, + { 0x0544, "\xD5\xB4" }, + { 0x0545, "\xD5\xB5" }, + { 0x0546, "\xD5\xB6" }, + { 0x0547, "\xD5\xB7" }, + { 0x0548, "\xD5\xB8" }, + { 0x0549, "\xD5\xB9" }, + { 0x054A, "\xD5\xBA" }, + { 0x054B, "\xD5\xBB" }, + { 0x054C, "\xD5\xBC" }, + { 0x054D, "\xD5\xBD" }, + { 0x054E, "\xD5\xBE" }, + { 0x054F, "\xD5\xBF" }, + { 0x0550, "\xD6\x80" }, + { 0x0551, "\xD6\x81" }, + { 0x0552, "\xD6\x82" }, + { 0x0553, "\xD6\x83" }, + { 0x0554, "\xD6\x84" }, + { 0x0555, "\xD6\x85" }, + { 0x0556, "\xD6\x86" }, + { 0x10A0, "\xE2\xB4\x80" }, + { 0x10A1, "\xE2\xB4\x81" }, + { 0x10A2, "\xE2\xB4\x82" }, + { 0x10A3, "\xE2\xB4\x83" }, + { 0x10A4, "\xE2\xB4\x84" }, + { 0x10A5, "\xE2\xB4\x85" }, + { 0x10A6, "\xE2\xB4\x86" }, + { 0x10A7, "\xE2\xB4\x87" }, + { 0x10A8, "\xE2\xB4\x88" }, + { 0x10A9, "\xE2\xB4\x89" }, + { 0x10AA, "\xE2\xB4\x8A" }, + { 0x10AB, "\xE2\xB4\x8B" }, + { 0x10AC, "\xE2\xB4\x8C" }, + { 0x10AD, "\xE2\xB4\x8D" }, + { 0x10AE, "\xE2\xB4\x8E" }, + { 0x10AF, "\xE2\xB4\x8F" }, + { 0x10B0, "\xE2\xB4\x90" }, + { 0x10B1, "\xE2\xB4\x91" }, + { 0x10B2, "\xE2\xB4\x92" }, + { 0x10B3, "\xE2\xB4\x93" }, + { 0x10B4, "\xE2\xB4\x94" }, + { 0x10B5, "\xE2\xB4\x95" }, + { 0x10B6, "\xE2\xB4\x96" }, + { 0x10B7, "\xE2\xB4\x97" }, + { 0x10B8, "\xE2\xB4\x98" }, + { 0x10B9, "\xE2\xB4\x99" }, + { 0x10BA, "\xE2\xB4\x9A" }, + { 0x10BB, "\xE2\xB4\x9B" }, + { 0x10BC, "\xE2\xB4\x9C" }, + { 0x10BD, "\xE2\xB4\x9D" }, + { 0x10BE, "\xE2\xB4\x9E" }, + { 0x10BF, "\xE2\xB4\x9F" }, + { 0x10C0, "\xE2\xB4\xA0" }, + { 0x10C1, "\xE2\xB4\xA1" }, + { 0x10C2, "\xE2\xB4\xA2" }, + { 0x10C3, "\xE2\xB4\xA3" }, + { 0x10C4, "\xE2\xB4\xA4" }, + { 0x10C5, "\xE2\xB4\xA5" }, + { 0x10C7, "\xE2\xB4\xA7" }, + { 0x10CD, "\xE2\xB4\xAD" }, + { 0x13A0, "\xEA\xAD\xB0" }, + { 0x13A1, "\xEA\xAD\xB1" }, + { 0x13A2, "\xEA\xAD\xB2" }, + { 0x13A3, "\xEA\xAD\xB3" }, + { 0x13A4, "\xEA\xAD\xB4" }, + { 0x13A5, "\xEA\xAD\xB5" }, + { 0x13A6, "\xEA\xAD\xB6" }, + { 0x13A7, "\xEA\xAD\xB7" }, + { 0x13A8, "\xEA\xAD\xB8" }, + { 0x13A9, "\xEA\xAD\xB9" }, + { 0x13AA, "\xEA\xAD\xBA" }, + { 0x13AB, "\xEA\xAD\xBB" }, + { 0x13AC, "\xEA\xAD\xBC" }, + { 0x13AD, "\xEA\xAD\xBD" }, + { 0x13AE, "\xEA\xAD\xBE" }, + { 0x13AF, "\xEA\xAD\xBF" }, + { 0x13B0, "\xEA\xAE\x80" }, + { 0x13B1, "\xEA\xAE\x81" }, + { 0x13B2, "\xEA\xAE\x82" }, + { 0x13B3, "\xEA\xAE\x83" }, + { 0x13B4, "\xEA\xAE\x84" }, + { 0x13B5, "\xEA\xAE\x85" }, + { 0x13B6, "\xEA\xAE\x86" }, + { 0x13B7, "\xEA\xAE\x87" }, + { 0x13B8, "\xEA\xAE\x88" }, + { 0x13B9, "\xEA\xAE\x89" }, + { 0x13BA, "\xEA\xAE\x8A" }, + { 0x13BB, "\xEA\xAE\x8B" }, + { 0x13BC, "\xEA\xAE\x8C" }, + { 0x13BD, "\xEA\xAE\x8D" }, + { 0x13BE, "\xEA\xAE\x8E" }, + { 0x13BF, "\xEA\xAE\x8F" }, + { 0x13C0, "\xEA\xAE\x90" }, + { 0x13C1, "\xEA\xAE\x91" }, + { 0x13C2, "\xEA\xAE\x92" }, + { 0x13C3, "\xEA\xAE\x93" }, + { 0x13C4, "\xEA\xAE\x94" }, + { 0x13C5, "\xEA\xAE\x95" }, + { 0x13C6, "\xEA\xAE\x96" }, + { 0x13C7, "\xEA\xAE\x97" }, + { 0x13C8, "\xEA\xAE\x98" }, + { 0x13C9, "\xEA\xAE\x99" }, + { 0x13CA, "\xEA\xAE\x9A" }, + { 0x13CB, "\xEA\xAE\x9B" }, + { 0x13CC, "\xEA\xAE\x9C" }, + { 0x13CD, "\xEA\xAE\x9D" }, + { 0x13CE, "\xEA\xAE\x9E" }, + { 0x13CF, "\xEA\xAE\x9F" }, + { 0x13D0, "\xEA\xAE\xA0" }, + { 0x13D1, "\xEA\xAE\xA1" }, + { 0x13D2, "\xEA\xAE\xA2" }, + { 0x13D3, "\xEA\xAE\xA3" }, + { 0x13D4, "\xEA\xAE\xA4" }, + { 0x13D5, "\xEA\xAE\xA5" }, + { 0x13D6, "\xEA\xAE\xA6" }, + { 0x13D7, "\xEA\xAE\xA7" }, + { 0x13D8, "\xEA\xAE\xA8" }, + { 0x13D9, "\xEA\xAE\xA9" }, + { 0x13DA, "\xEA\xAE\xAA" }, + { 0x13DB, "\xEA\xAE\xAB" }, + { 0x13DC, "\xEA\xAE\xAC" }, + { 0x13DD, "\xEA\xAE\xAD" }, + { 0x13DE, "\xEA\xAE\xAE" }, + { 0x13DF, "\xEA\xAE\xAF" }, + { 0x13E0, "\xEA\xAE\xB0" }, + { 0x13E1, "\xEA\xAE\xB1" }, + { 0x13E2, "\xEA\xAE\xB2" }, + { 0x13E3, "\xEA\xAE\xB3" }, + { 0x13E4, "\xEA\xAE\xB4" }, + { 0x13E5, "\xEA\xAE\xB5" }, + { 0x13E6, "\xEA\xAE\xB6" }, + { 0x13E7, "\xEA\xAE\xB7" }, + { 0x13E8, "\xEA\xAE\xB8" }, + { 0x13E9, "\xEA\xAE\xB9" }, + { 0x13EA, "\xEA\xAE\xBA" }, + { 0x13EB, "\xEA\xAE\xBB" }, + { 0x13EC, "\xEA\xAE\xBC" }, + { 0x13ED, "\xEA\xAE\xBD" }, + { 0x13EE, "\xEA\xAE\xBE" }, + { 0x13EF, "\xEA\xAE\xBF" }, + { 0x13F0, "\xE1\x8F\xB8" }, + { 0x13F1, "\xE1\x8F\xB9" }, + { 0x13F2, "\xE1\x8F\xBA" }, + { 0x13F3, "\xE1\x8F\xBB" }, + { 0x13F4, "\xE1\x8F\xBC" }, + { 0x13F5, "\xE1\x8F\xBD" }, + { 0x1C90, "\xE1\x83\x90" }, + { 0x1C91, "\xE1\x83\x91" }, + { 0x1C92, "\xE1\x83\x92" }, + { 0x1C93, "\xE1\x83\x93" }, + { 0x1C94, "\xE1\x83\x94" }, + { 0x1C95, "\xE1\x83\x95" }, + { 0x1C96, "\xE1\x83\x96" }, + { 0x1C97, "\xE1\x83\x97" }, + { 0x1C98, "\xE1\x83\x98" }, + { 0x1C99, "\xE1\x83\x99" }, + { 0x1C9A, "\xE1\x83\x9A" }, + { 0x1C9B, "\xE1\x83\x9B" }, + { 0x1C9C, "\xE1\x83\x9C" }, + { 0x1C9D, "\xE1\x83\x9D" }, + { 0x1C9E, "\xE1\x83\x9E" }, + { 0x1C9F, "\xE1\x83\x9F" }, + { 0x1CA0, "\xE1\x83\xA0" }, + { 0x1CA1, "\xE1\x83\xA1" }, + { 0x1CA2, "\xE1\x83\xA2" }, + { 0x1CA3, "\xE1\x83\xA3" }, + { 0x1CA4, "\xE1\x83\xA4" }, + { 0x1CA5, "\xE1\x83\xA5" }, + { 0x1CA6, "\xE1\x83\xA6" }, + { 0x1CA7, "\xE1\x83\xA7" }, + { 0x1CA8, "\xE1\x83\xA8" }, + { 0x1CA9, "\xE1\x83\xA9" }, + { 0x1CAA, "\xE1\x83\xAA" }, + { 0x1CAB, "\xE1\x83\xAB" }, + { 0x1CAC, "\xE1\x83\xAC" }, + { 0x1CAD, "\xE1\x83\xAD" }, + { 0x1CAE, "\xE1\x83\xAE" }, + { 0x1CAF, "\xE1\x83\xAF" }, + { 0x1CB0, "\xE1\x83\xB0" }, + { 0x1CB1, "\xE1\x83\xB1" }, + { 0x1CB2, "\xE1\x83\xB2" }, + { 0x1CB3, "\xE1\x83\xB3" }, + { 0x1CB4, "\xE1\x83\xB4" }, + { 0x1CB5, "\xE1\x83\xB5" }, + { 0x1CB6, "\xE1\x83\xB6" }, + { 0x1CB7, "\xE1\x83\xB7" }, + { 0x1CB8, "\xE1\x83\xB8" }, + { 0x1CB9, "\xE1\x83\xB9" }, + { 0x1CBA, "\xE1\x83\xBA" }, + { 0x1CBD, "\xE1\x83\xBD" }, + { 0x1CBE, "\xE1\x83\xBE" }, + { 0x1CBF, "\xE1\x83\xBF" }, + { 0x1E00, "\xE1\xB8\x81" }, + { 0x1E02, "\xE1\xB8\x83" }, + { 0x1E04, "\xE1\xB8\x85" }, + { 0x1E06, "\xE1\xB8\x87" }, + { 0x1E08, "\xE1\xB8\x89" }, + { 0x1E0A, "\xE1\xB8\x8B" }, + { 0x1E0C, "\xE1\xB8\x8D" }, + { 0x1E0E, "\xE1\xB8\x8F" }, + { 0x1E10, "\xE1\xB8\x91" }, + { 0x1E12, "\xE1\xB8\x93" }, + { 0x1E14, "\xE1\xB8\x95" }, + { 0x1E16, "\xE1\xB8\x97" }, + { 0x1E18, "\xE1\xB8\x99" }, + { 0x1E1A, "\xE1\xB8\x9B" }, + { 0x1E1C, "\xE1\xB8\x9D" }, + { 0x1E1E, "\xE1\xB8\x9F" }, + { 0x1E20, "\xE1\xB8\xA1" }, + { 0x1E22, "\xE1\xB8\xA3" }, + { 0x1E24, "\xE1\xB8\xA5" }, + { 0x1E26, "\xE1\xB8\xA7" }, + { 0x1E28, "\xE1\xB8\xA9" }, + { 0x1E2A, "\xE1\xB8\xAB" }, + { 0x1E2C, "\xE1\xB8\xAD" }, + { 0x1E2E, "\xE1\xB8\xAF" }, + { 0x1E30, "\xE1\xB8\xB1" }, + { 0x1E32, "\xE1\xB8\xB3" }, + { 0x1E34, "\xE1\xB8\xB5" }, + { 0x1E36, "\xE1\xB8\xB7" }, + { 0x1E38, "\xE1\xB8\xB9" }, + { 0x1E3A, "\xE1\xB8\xBB" }, + { 0x1E3C, "\xE1\xB8\xBD" }, + { 0x1E3E, "\xE1\xB8\xBF" }, + { 0x1E40, "\xE1\xB9\x81" }, + { 0x1E42, "\xE1\xB9\x83" }, + { 0x1E44, "\xE1\xB9\x85" }, + { 0x1E46, "\xE1\xB9\x87" }, + { 0x1E48, "\xE1\xB9\x89" }, + { 0x1E4A, "\xE1\xB9\x8B" }, + { 0x1E4C, "\xE1\xB9\x8D" }, + { 0x1E4E, "\xE1\xB9\x8F" }, + { 0x1E50, "\xE1\xB9\x91" }, + { 0x1E52, "\xE1\xB9\x93" }, + { 0x1E54, "\xE1\xB9\x95" }, + { 0x1E56, "\xE1\xB9\x97" }, + { 0x1E58, "\xE1\xB9\x99" }, + { 0x1E5A, "\xE1\xB9\x9B" }, + { 0x1E5C, "\xE1\xB9\x9D" }, + { 0x1E5E, "\xE1\xB9\x9F" }, + { 0x1E60, "\xE1\xB9\xA1" }, + { 0x1E62, "\xE1\xB9\xA3" }, + { 0x1E64, "\xE1\xB9\xA5" }, + { 0x1E66, "\xE1\xB9\xA7" }, + { 0x1E68, "\xE1\xB9\xA9" }, + { 0x1E6A, "\xE1\xB9\xAB" }, + { 0x1E6C, "\xE1\xB9\xAD" }, + { 0x1E6E, "\xE1\xB9\xAF" }, + { 0x1E70, "\xE1\xB9\xB1" }, + { 0x1E72, "\xE1\xB9\xB3" }, + { 0x1E74, "\xE1\xB9\xB5" }, + { 0x1E76, "\xE1\xB9\xB7" }, + { 0x1E78, "\xE1\xB9\xB9" }, + { 0x1E7A, "\xE1\xB9\xBB" }, + { 0x1E7C, "\xE1\xB9\xBD" }, + { 0x1E7E, "\xE1\xB9\xBF" }, + { 0x1E80, "\xE1\xBA\x81" }, + { 0x1E82, "\xE1\xBA\x83" }, + { 0x1E84, "\xE1\xBA\x85" }, + { 0x1E86, "\xE1\xBA\x87" }, + { 0x1E88, "\xE1\xBA\x89" }, + { 0x1E8A, "\xE1\xBA\x8B" }, + { 0x1E8C, "\xE1\xBA\x8D" }, + { 0x1E8E, "\xE1\xBA\x8F" }, + { 0x1E90, "\xE1\xBA\x91" }, + { 0x1E92, "\xE1\xBA\x93" }, + { 0x1E94, "\xE1\xBA\x95" }, + { 0x1E9E, "\xC3\x9F" }, + { 0x1EA0, "\xE1\xBA\xA1" }, + { 0x1EA2, "\xE1\xBA\xA3" }, + { 0x1EA4, "\xE1\xBA\xA5" }, + { 0x1EA6, "\xE1\xBA\xA7" }, + { 0x1EA8, "\xE1\xBA\xA9" }, + { 0x1EAA, "\xE1\xBA\xAB" }, + { 0x1EAC, "\xE1\xBA\xAD" }, + { 0x1EAE, "\xE1\xBA\xAF" }, + { 0x1EB0, "\xE1\xBA\xB1" }, + { 0x1EB2, "\xE1\xBA\xB3" }, + { 0x1EB4, "\xE1\xBA\xB5" }, + { 0x1EB6, "\xE1\xBA\xB7" }, + { 0x1EB8, "\xE1\xBA\xB9" }, + { 0x1EBA, "\xE1\xBA\xBB" }, + { 0x1EBC, "\xE1\xBA\xBD" }, + { 0x1EBE, "\xE1\xBA\xBF" }, + { 0x1EC0, "\xE1\xBB\x81" }, + { 0x1EC2, "\xE1\xBB\x83" }, + { 0x1EC4, "\xE1\xBB\x85" }, + { 0x1EC6, "\xE1\xBB\x87" }, + { 0x1EC8, "\xE1\xBB\x89" }, + { 0x1ECA, "\xE1\xBB\x8B" }, + { 0x1ECC, "\xE1\xBB\x8D" }, + { 0x1ECE, "\xE1\xBB\x8F" }, + { 0x1ED0, "\xE1\xBB\x91" }, + { 0x1ED2, "\xE1\xBB\x93" }, + { 0x1ED4, "\xE1\xBB\x95" }, + { 0x1ED6, "\xE1\xBB\x97" }, + { 0x1ED8, "\xE1\xBB\x99" }, + { 0x1EDA, "\xE1\xBB\x9B" }, + { 0x1EDC, "\xE1\xBB\x9D" }, + { 0x1EDE, "\xE1\xBB\x9F" }, + { 0x1EE0, "\xE1\xBB\xA1" }, + { 0x1EE2, "\xE1\xBB\xA3" }, + { 0x1EE4, "\xE1\xBB\xA5" }, + { 0x1EE6, "\xE1\xBB\xA7" }, + { 0x1EE8, "\xE1\xBB\xA9" }, + { 0x1EEA, "\xE1\xBB\xAB" }, + { 0x1EEC, "\xE1\xBB\xAD" }, + { 0x1EEE, "\xE1\xBB\xAF" }, + { 0x1EF0, "\xE1\xBB\xB1" }, + { 0x1EF2, "\xE1\xBB\xB3" }, + { 0x1EF4, "\xE1\xBB\xB5" }, + { 0x1EF6, "\xE1\xBB\xB7" }, + { 0x1EF8, "\xE1\xBB\xB9" }, + { 0x1EFA, "\xE1\xBB\xBB" }, + { 0x1EFC, "\xE1\xBB\xBD" }, + { 0x1EFE, "\xE1\xBB\xBF" }, + { 0x1F08, "\xE1\xBC\x80" }, + { 0x1F09, "\xE1\xBC\x81" }, + { 0x1F0A, "\xE1\xBC\x82" }, + { 0x1F0B, "\xE1\xBC\x83" }, + { 0x1F0C, "\xE1\xBC\x84" }, + { 0x1F0D, "\xE1\xBC\x85" }, + { 0x1F0E, "\xE1\xBC\x86" }, + { 0x1F0F, "\xE1\xBC\x87" }, + { 0x1F18, "\xE1\xBC\x90" }, + { 0x1F19, "\xE1\xBC\x91" }, + { 0x1F1A, "\xE1\xBC\x92" }, + { 0x1F1B, "\xE1\xBC\x93" }, + { 0x1F1C, "\xE1\xBC\x94" }, + { 0x1F1D, "\xE1\xBC\x95" }, + { 0x1F28, "\xE1\xBC\xA0" }, + { 0x1F29, "\xE1\xBC\xA1" }, + { 0x1F2A, "\xE1\xBC\xA2" }, + { 0x1F2B, "\xE1\xBC\xA3" }, + { 0x1F2C, "\xE1\xBC\xA4" }, + { 0x1F2D, "\xE1\xBC\xA5" }, + { 0x1F2E, "\xE1\xBC\xA6" }, + { 0x1F2F, "\xE1\xBC\xA7" }, + { 0x1F38, "\xE1\xBC\xB0" }, + { 0x1F39, "\xE1\xBC\xB1" }, + { 0x1F3A, "\xE1\xBC\xB2" }, + { 0x1F3B, "\xE1\xBC\xB3" }, + { 0x1F3C, "\xE1\xBC\xB4" }, + { 0x1F3D, "\xE1\xBC\xB5" }, + { 0x1F3E, "\xE1\xBC\xB6" }, + { 0x1F3F, "\xE1\xBC\xB7" }, + { 0x1F48, "\xE1\xBD\x80" }, + { 0x1F49, "\xE1\xBD\x81" }, + { 0x1F4A, "\xE1\xBD\x82" }, + { 0x1F4B, "\xE1\xBD\x83" }, + { 0x1F4C, "\xE1\xBD\x84" }, + { 0x1F4D, "\xE1\xBD\x85" }, + { 0x1F59, "\xE1\xBD\x91" }, + { 0x1F5B, "\xE1\xBD\x93" }, + { 0x1F5D, "\xE1\xBD\x95" }, + { 0x1F5F, "\xE1\xBD\x97" }, + { 0x1F68, "\xE1\xBD\xA0" }, + { 0x1F69, "\xE1\xBD\xA1" }, + { 0x1F6A, "\xE1\xBD\xA2" }, + { 0x1F6B, "\xE1\xBD\xA3" }, + { 0x1F6C, "\xE1\xBD\xA4" }, + { 0x1F6D, "\xE1\xBD\xA5" }, + { 0x1F6E, "\xE1\xBD\xA6" }, + { 0x1F6F, "\xE1\xBD\xA7" }, + { 0x1F88, "\xE1\xBE\x80" }, + { 0x1F89, "\xE1\xBE\x81" }, + { 0x1F8A, "\xE1\xBE\x82" }, + { 0x1F8B, "\xE1\xBE\x83" }, + { 0x1F8C, "\xE1\xBE\x84" }, + { 0x1F8D, "\xE1\xBE\x85" }, + { 0x1F8E, "\xE1\xBE\x86" }, + { 0x1F8F, "\xE1\xBE\x87" }, + { 0x1F98, "\xE1\xBE\x90" }, + { 0x1F99, "\xE1\xBE\x91" }, + { 0x1F9A, "\xE1\xBE\x92" }, + { 0x1F9B, "\xE1\xBE\x93" }, + { 0x1F9C, "\xE1\xBE\x94" }, + { 0x1F9D, "\xE1\xBE\x95" }, + { 0x1F9E, "\xE1\xBE\x96" }, + { 0x1F9F, "\xE1\xBE\x97" }, + { 0x1FA8, "\xE1\xBE\xA0" }, + { 0x1FA9, "\xE1\xBE\xA1" }, + { 0x1FAA, "\xE1\xBE\xA2" }, + { 0x1FAB, "\xE1\xBE\xA3" }, + { 0x1FAC, "\xE1\xBE\xA4" }, + { 0x1FAD, "\xE1\xBE\xA5" }, + { 0x1FAE, "\xE1\xBE\xA6" }, + { 0x1FAF, "\xE1\xBE\xA7" }, + { 0x1FB8, "\xE1\xBE\xB0" }, + { 0x1FB9, "\xE1\xBE\xB1" }, + { 0x1FBA, "\xE1\xBD\xB0" }, + { 0x1FBB, "\xE1\xBD\xB1" }, + { 0x1FBC, "\xE1\xBE\xB3" }, + { 0x1FC8, "\xE1\xBD\xB2" }, + { 0x1FC9, "\xE1\xBD\xB3" }, + { 0x1FCA, "\xE1\xBD\xB4" }, + { 0x1FCB, "\xE1\xBD\xB5" }, + { 0x1FCC, "\xE1\xBF\x83" }, + { 0x1FD8, "\xE1\xBF\x90" }, + { 0x1FD9, "\xE1\xBF\x91" }, + { 0x1FDA, "\xE1\xBD\xB6" }, + { 0x1FDB, "\xE1\xBD\xB7" }, + { 0x1FE8, "\xE1\xBF\xA0" }, + { 0x1FE9, "\xE1\xBF\xA1" }, + { 0x1FEA, "\xE1\xBD\xBA" }, + { 0x1FEB, "\xE1\xBD\xBB" }, + { 0x1FEC, "\xE1\xBF\xA5" }, + { 0x1FF8, "\xE1\xBD\xB8" }, + { 0x1FF9, "\xE1\xBD\xB9" }, + { 0x1FFA, "\xE1\xBD\xBC" }, + { 0x1FFB, "\xE1\xBD\xBD" }, + { 0x1FFC, "\xE1\xBF\xB3" }, + { 0x2126, "\xCF\x89" }, + { 0x212A, "\x6B" }, + { 0x212B, "\xC3\xA5" }, + { 0x2132, "\xE2\x85\x8E" }, + { 0x2160, "\xE2\x85\xB0" }, + { 0x2161, "\xE2\x85\xB1" }, + { 0x2162, "\xE2\x85\xB2" }, + { 0x2163, "\xE2\x85\xB3" }, + { 0x2164, "\xE2\x85\xB4" }, + { 0x2165, "\xE2\x85\xB5" }, + { 0x2166, "\xE2\x85\xB6" }, + { 0x2167, "\xE2\x85\xB7" }, + { 0x2168, "\xE2\x85\xB8" }, + { 0x2169, "\xE2\x85\xB9" }, + { 0x216A, "\xE2\x85\xBA" }, + { 0x216B, "\xE2\x85\xBB" }, + { 0x216C, "\xE2\x85\xBC" }, + { 0x216D, "\xE2\x85\xBD" }, + { 0x216E, "\xE2\x85\xBE" }, + { 0x216F, "\xE2\x85\xBF" }, + { 0x2183, "\xE2\x86\x84" }, + { 0x24B6, "\xE2\x93\x90" }, + { 0x24B7, "\xE2\x93\x91" }, + { 0x24B8, "\xE2\x93\x92" }, + { 0x24B9, "\xE2\x93\x93" }, + { 0x24BA, "\xE2\x93\x94" }, + { 0x24BB, "\xE2\x93\x95" }, + { 0x24BC, "\xE2\x93\x96" }, + { 0x24BD, "\xE2\x93\x97" }, + { 0x24BE, "\xE2\x93\x98" }, + { 0x24BF, "\xE2\x93\x99" }, + { 0x24C0, "\xE2\x93\x9A" }, + { 0x24C1, "\xE2\x93\x9B" }, + { 0x24C2, "\xE2\x93\x9C" }, + { 0x24C3, "\xE2\x93\x9D" }, + { 0x24C4, "\xE2\x93\x9E" }, + { 0x24C5, "\xE2\x93\x9F" }, + { 0x24C6, "\xE2\x93\xA0" }, + { 0x24C7, "\xE2\x93\xA1" }, + { 0x24C8, "\xE2\x93\xA2" }, + { 0x24C9, "\xE2\x93\xA3" }, + { 0x24CA, "\xE2\x93\xA4" }, + { 0x24CB, "\xE2\x93\xA5" }, + { 0x24CC, "\xE2\x93\xA6" }, + { 0x24CD, "\xE2\x93\xA7" }, + { 0x24CE, "\xE2\x93\xA8" }, + { 0x24CF, "\xE2\x93\xA9" }, + { 0x2C00, "\xE2\xB0\xB0" }, + { 0x2C01, "\xE2\xB0\xB1" }, + { 0x2C02, "\xE2\xB0\xB2" }, + { 0x2C03, "\xE2\xB0\xB3" }, + { 0x2C04, "\xE2\xB0\xB4" }, + { 0x2C05, "\xE2\xB0\xB5" }, + { 0x2C06, "\xE2\xB0\xB6" }, + { 0x2C07, "\xE2\xB0\xB7" }, + { 0x2C08, "\xE2\xB0\xB8" }, + { 0x2C09, "\xE2\xB0\xB9" }, + { 0x2C0A, "\xE2\xB0\xBA" }, + { 0x2C0B, "\xE2\xB0\xBB" }, + { 0x2C0C, "\xE2\xB0\xBC" }, + { 0x2C0D, "\xE2\xB0\xBD" }, + { 0x2C0E, "\xE2\xB0\xBE" }, + { 0x2C0F, "\xE2\xB0\xBF" }, + { 0x2C10, "\xE2\xB1\x80" }, + { 0x2C11, "\xE2\xB1\x81" }, + { 0x2C12, "\xE2\xB1\x82" }, + { 0x2C13, "\xE2\xB1\x83" }, + { 0x2C14, "\xE2\xB1\x84" }, + { 0x2C15, "\xE2\xB1\x85" }, + { 0x2C16, "\xE2\xB1\x86" }, + { 0x2C17, "\xE2\xB1\x87" }, + { 0x2C18, "\xE2\xB1\x88" }, + { 0x2C19, "\xE2\xB1\x89" }, + { 0x2C1A, "\xE2\xB1\x8A" }, + { 0x2C1B, "\xE2\xB1\x8B" }, + { 0x2C1C, "\xE2\xB1\x8C" }, + { 0x2C1D, "\xE2\xB1\x8D" }, + { 0x2C1E, "\xE2\xB1\x8E" }, + { 0x2C1F, "\xE2\xB1\x8F" }, + { 0x2C20, "\xE2\xB1\x90" }, + { 0x2C21, "\xE2\xB1\x91" }, + { 0x2C22, "\xE2\xB1\x92" }, + { 0x2C23, "\xE2\xB1\x93" }, + { 0x2C24, "\xE2\xB1\x94" }, + { 0x2C25, "\xE2\xB1\x95" }, + { 0x2C26, "\xE2\xB1\x96" }, + { 0x2C27, "\xE2\xB1\x97" }, + { 0x2C28, "\xE2\xB1\x98" }, + { 0x2C29, "\xE2\xB1\x99" }, + { 0x2C2A, "\xE2\xB1\x9A" }, + { 0x2C2B, "\xE2\xB1\x9B" }, + { 0x2C2C, "\xE2\xB1\x9C" }, + { 0x2C2D, "\xE2\xB1\x9D" }, + { 0x2C2E, "\xE2\xB1\x9E" }, + { 0x2C60, "\xE2\xB1\xA1" }, + { 0x2C62, "\xC9\xAB" }, + { 0x2C63, "\xE1\xB5\xBD" }, + { 0x2C64, "\xC9\xBD" }, + { 0x2C67, "\xE2\xB1\xA8" }, + { 0x2C69, "\xE2\xB1\xAA" }, + { 0x2C6B, "\xE2\xB1\xAC" }, + { 0x2C6D, "\xC9\x91" }, + { 0x2C6E, "\xC9\xB1" }, + { 0x2C6F, "\xC9\x90" }, + { 0x2C70, "\xC9\x92" }, + { 0x2C72, "\xE2\xB1\xB3" }, + { 0x2C75, "\xE2\xB1\xB6" }, + { 0x2C7E, "\xC8\xBF" }, + { 0x2C7F, "\xC9\x80" }, + { 0x2C80, "\xE2\xB2\x81" }, + { 0x2C82, "\xE2\xB2\x83" }, + { 0x2C84, "\xE2\xB2\x85" }, + { 0x2C86, "\xE2\xB2\x87" }, + { 0x2C88, "\xE2\xB2\x89" }, + { 0x2C8A, "\xE2\xB2\x8B" }, + { 0x2C8C, "\xE2\xB2\x8D" }, + { 0x2C8E, "\xE2\xB2\x8F" }, + { 0x2C90, "\xE2\xB2\x91" }, + { 0x2C92, "\xE2\xB2\x93" }, + { 0x2C94, "\xE2\xB2\x95" }, + { 0x2C96, "\xE2\xB2\x97" }, + { 0x2C98, "\xE2\xB2\x99" }, + { 0x2C9A, "\xE2\xB2\x9B" }, + { 0x2C9C, "\xE2\xB2\x9D" }, + { 0x2C9E, "\xE2\xB2\x9F" }, + { 0x2CA0, "\xE2\xB2\xA1" }, + { 0x2CA2, "\xE2\xB2\xA3" }, + { 0x2CA4, "\xE2\xB2\xA5" }, + { 0x2CA6, "\xE2\xB2\xA7" }, + { 0x2CA8, "\xE2\xB2\xA9" }, + { 0x2CAA, "\xE2\xB2\xAB" }, + { 0x2CAC, "\xE2\xB2\xAD" }, + { 0x2CAE, "\xE2\xB2\xAF" }, + { 0x2CB0, "\xE2\xB2\xB1" }, + { 0x2CB2, "\xE2\xB2\xB3" }, + { 0x2CB4, "\xE2\xB2\xB5" }, + { 0x2CB6, "\xE2\xB2\xB7" }, + { 0x2CB8, "\xE2\xB2\xB9" }, + { 0x2CBA, "\xE2\xB2\xBB" }, + { 0x2CBC, "\xE2\xB2\xBD" }, + { 0x2CBE, "\xE2\xB2\xBF" }, + { 0x2CC0, "\xE2\xB3\x81" }, + { 0x2CC2, "\xE2\xB3\x83" }, + { 0x2CC4, "\xE2\xB3\x85" }, + { 0x2CC6, "\xE2\xB3\x87" }, + { 0x2CC8, "\xE2\xB3\x89" }, + { 0x2CCA, "\xE2\xB3\x8B" }, + { 0x2CCC, "\xE2\xB3\x8D" }, + { 0x2CCE, "\xE2\xB3\x8F" }, + { 0x2CD0, "\xE2\xB3\x91" }, + { 0x2CD2, "\xE2\xB3\x93" }, + { 0x2CD4, "\xE2\xB3\x95" }, + { 0x2CD6, "\xE2\xB3\x97" }, + { 0x2CD8, "\xE2\xB3\x99" }, + { 0x2CDA, "\xE2\xB3\x9B" }, + { 0x2CDC, "\xE2\xB3\x9D" }, + { 0x2CDE, "\xE2\xB3\x9F" }, + { 0x2CE0, "\xE2\xB3\xA1" }, + { 0x2CE2, "\xE2\xB3\xA3" }, + { 0x2CEB, "\xE2\xB3\xAC" }, + { 0x2CED, "\xE2\xB3\xAE" }, + { 0x2CF2, "\xE2\xB3\xB3" }, + { 0xA640, "\xEA\x99\x81" }, + { 0xA642, "\xEA\x99\x83" }, + { 0xA644, "\xEA\x99\x85" }, + { 0xA646, "\xEA\x99\x87" }, + { 0xA648, "\xEA\x99\x89" }, + { 0xA64A, "\xEA\x99\x8B" }, + { 0xA64C, "\xEA\x99\x8D" }, + { 0xA64E, "\xEA\x99\x8F" }, + { 0xA650, "\xEA\x99\x91" }, + { 0xA652, "\xEA\x99\x93" }, + { 0xA654, "\xEA\x99\x95" }, + { 0xA656, "\xEA\x99\x97" }, + { 0xA658, "\xEA\x99\x99" }, + { 0xA65A, "\xEA\x99\x9B" }, + { 0xA65C, "\xEA\x99\x9D" }, + { 0xA65E, "\xEA\x99\x9F" }, + { 0xA660, "\xEA\x99\xA1" }, + { 0xA662, "\xEA\x99\xA3" }, + { 0xA664, "\xEA\x99\xA5" }, + { 0xA666, "\xEA\x99\xA7" }, + { 0xA668, "\xEA\x99\xA9" }, + { 0xA66A, "\xEA\x99\xAB" }, + { 0xA66C, "\xEA\x99\xAD" }, + { 0xA680, "\xEA\x9A\x81" }, + { 0xA682, "\xEA\x9A\x83" }, + { 0xA684, "\xEA\x9A\x85" }, + { 0xA686, "\xEA\x9A\x87" }, + { 0xA688, "\xEA\x9A\x89" }, + { 0xA68A, "\xEA\x9A\x8B" }, + { 0xA68C, "\xEA\x9A\x8D" }, + { 0xA68E, "\xEA\x9A\x8F" }, + { 0xA690, "\xEA\x9A\x91" }, + { 0xA692, "\xEA\x9A\x93" }, + { 0xA694, "\xEA\x9A\x95" }, + { 0xA696, "\xEA\x9A\x97" }, + { 0xA698, "\xEA\x9A\x99" }, + { 0xA69A, "\xEA\x9A\x9B" }, + { 0xA722, "\xEA\x9C\xA3" }, + { 0xA724, "\xEA\x9C\xA5" }, + { 0xA726, "\xEA\x9C\xA7" }, + { 0xA728, "\xEA\x9C\xA9" }, + { 0xA72A, "\xEA\x9C\xAB" }, + { 0xA72C, "\xEA\x9C\xAD" }, + { 0xA72E, "\xEA\x9C\xAF" }, + { 0xA732, "\xEA\x9C\xB3" }, + { 0xA734, "\xEA\x9C\xB5" }, + { 0xA736, "\xEA\x9C\xB7" }, + { 0xA738, "\xEA\x9C\xB9" }, + { 0xA73A, "\xEA\x9C\xBB" }, + { 0xA73C, "\xEA\x9C\xBD" }, + { 0xA73E, "\xEA\x9C\xBF" }, + { 0xA740, "\xEA\x9D\x81" }, + { 0xA742, "\xEA\x9D\x83" }, + { 0xA744, "\xEA\x9D\x85" }, + { 0xA746, "\xEA\x9D\x87" }, + { 0xA748, "\xEA\x9D\x89" }, + { 0xA74A, "\xEA\x9D\x8B" }, + { 0xA74C, "\xEA\x9D\x8D" }, + { 0xA74E, "\xEA\x9D\x8F" }, + { 0xA750, "\xEA\x9D\x91" }, + { 0xA752, "\xEA\x9D\x93" }, + { 0xA754, "\xEA\x9D\x95" }, + { 0xA756, "\xEA\x9D\x97" }, + { 0xA758, "\xEA\x9D\x99" }, + { 0xA75A, "\xEA\x9D\x9B" }, + { 0xA75C, "\xEA\x9D\x9D" }, + { 0xA75E, "\xEA\x9D\x9F" }, + { 0xA760, "\xEA\x9D\xA1" }, + { 0xA762, "\xEA\x9D\xA3" }, + { 0xA764, "\xEA\x9D\xA5" }, + { 0xA766, "\xEA\x9D\xA7" }, + { 0xA768, "\xEA\x9D\xA9" }, + { 0xA76A, "\xEA\x9D\xAB" }, + { 0xA76C, "\xEA\x9D\xAD" }, + { 0xA76E, "\xEA\x9D\xAF" }, + { 0xA779, "\xEA\x9D\xBA" }, + { 0xA77B, "\xEA\x9D\xBC" }, + { 0xA77D, "\xE1\xB5\xB9" }, + { 0xA77E, "\xEA\x9D\xBF" }, + { 0xA780, "\xEA\x9E\x81" }, + { 0xA782, "\xEA\x9E\x83" }, + { 0xA784, "\xEA\x9E\x85" }, + { 0xA786, "\xEA\x9E\x87" }, + { 0xA78B, "\xEA\x9E\x8C" }, + { 0xA78D, "\xC9\xA5" }, + { 0xA790, "\xEA\x9E\x91" }, + { 0xA792, "\xEA\x9E\x93" }, + { 0xA796, "\xEA\x9E\x97" }, + { 0xA798, "\xEA\x9E\x99" }, + { 0xA79A, "\xEA\x9E\x9B" }, + { 0xA79C, "\xEA\x9E\x9D" }, + { 0xA79E, "\xEA\x9E\x9F" }, + { 0xA7A0, "\xEA\x9E\xA1" }, + { 0xA7A2, "\xEA\x9E\xA3" }, + { 0xA7A4, "\xEA\x9E\xA5" }, + { 0xA7A6, "\xEA\x9E\xA7" }, + { 0xA7A8, "\xEA\x9E\xA9" }, + { 0xA7AA, "\xC9\xA6" }, + { 0xA7AB, "\xC9\x9C" }, + { 0xA7AC, "\xC9\xA1" }, + { 0xA7AD, "\xC9\xAC" }, + { 0xA7AE, "\xC9\xAA" }, + { 0xA7B0, "\xCA\x9E" }, + { 0xA7B1, "\xCA\x87" }, + { 0xA7B2, "\xCA\x9D" }, + { 0xA7B3, "\xEA\xAD\x93" }, + { 0xA7B4, "\xEA\x9E\xB5" }, + { 0xA7B6, "\xEA\x9E\xB7" }, + { 0xA7B8, "\xEA\x9E\xB9" }, + { 0xA7BA, "\xEA\x9E\xBB" }, + { 0xA7BC, "\xEA\x9E\xBD" }, + { 0xA7BE, "\xEA\x9E\xBF" }, + { 0xA7C2, "\xEA\x9F\x83" }, + { 0xA7C4, "\xEA\x9E\x94" }, + { 0xA7C5, "\xCA\x82" }, + { 0xA7C6, "\xE1\xB6\x8E" }, + { 0xA7C7, "\xEA\x9F\x88" }, + { 0xA7C9, "\xEA\x9F\x8A" }, + { 0xA7F5, "\xEA\x9F\xB6" }, + { 0xFF21, "\xEF\xBD\x81" }, + { 0xFF22, "\xEF\xBD\x82" }, + { 0xFF23, "\xEF\xBD\x83" }, + { 0xFF24, "\xEF\xBD\x84" }, + { 0xFF25, "\xEF\xBD\x85" }, + { 0xFF26, "\xEF\xBD\x86" }, + { 0xFF27, "\xEF\xBD\x87" }, + { 0xFF28, "\xEF\xBD\x88" }, + { 0xFF29, "\xEF\xBD\x89" }, + { 0xFF2A, "\xEF\xBD\x8A" }, + { 0xFF2B, "\xEF\xBD\x8B" }, + { 0xFF2C, "\xEF\xBD\x8C" }, + { 0xFF2D, "\xEF\xBD\x8D" }, + { 0xFF2E, "\xEF\xBD\x8E" }, + { 0xFF2F, "\xEF\xBD\x8F" }, + { 0xFF30, "\xEF\xBD\x90" }, + { 0xFF31, "\xEF\xBD\x91" }, + { 0xFF32, "\xEF\xBD\x92" }, + { 0xFF33, "\xEF\xBD\x93" }, + { 0xFF34, "\xEF\xBD\x94" }, + { 0xFF35, "\xEF\xBD\x95" }, + { 0xFF36, "\xEF\xBD\x96" }, + { 0xFF37, "\xEF\xBD\x97" }, + { 0xFF38, "\xEF\xBD\x98" }, + { 0xFF39, "\xEF\xBD\x99" }, + { 0xFF3A, "\xEF\xBD\x9A" }, +}; + +constexpr FoldEntry TRANSLIT_RU[] = { + { 0x03B1, "\x61" }, + { 0x03B2, "\x76" }, + { 0x03B3, "\x67" }, + { 0x03B4, "\x64" }, + { 0x03B5, "\x65" }, + { 0x03B6, "\x7A" }, + { 0x03B7, "\x69" }, + { 0x03B8, "\x74\x68" }, + { 0x03B9, "\x69" }, + { 0x03BA, "\x6B" }, + { 0x03BB, "\x6C" }, + { 0x03BC, "\x6D" }, + { 0x03BD, "\x6E" }, + { 0x03BE, "\x78" }, + { 0x03BF, "\x6F" }, + { 0x03C0, "\x70" }, + { 0x03C1, "\x72" }, + { 0x03C2, "\x73" }, + { 0x03C3, "\x73" }, + { 0x03C4, "\x74" }, + { 0x03C5, "\x79" }, + { 0x03C6, "\x66" }, + { 0x03C7, "\x63\x68" }, + { 0x03C8, "\x70\x73" }, + { 0x03C9, "\x6F" }, + { 0x0430, "\x61" }, + { 0x0431, "\x62" }, + { 0x0432, "\x76" }, + { 0x0433, "\x67" }, + { 0x0434, "\x64" }, + { 0x0435, "\x65" }, + { 0x0436, "\x7A\x68" }, + { 0x0437, "\x7A" }, + { 0x0438, "\x69" }, + { 0x0439, "\x79" }, + { 0x043A, "\x6B" }, + { 0x043B, "\x6C" }, + { 0x043C, "\x6D" }, + { 0x043D, "\x6E" }, + { 0x043E, "\x6F" }, + { 0x043F, "\x70" }, + { 0x0440, "\x72" }, + { 0x0441, "\x73" }, + { 0x0442, "\x74" }, + { 0x0443, "\x75" }, + { 0x0444, "\x66" }, + { 0x0445, "\x6B\x68" }, + { 0x0446, "\x74\x73" }, + { 0x0447, "\x63\x68" }, + { 0x0448, "\x73\x68" }, + { 0x0449, "\x73\x68\x63\x68" }, + { 0x044A, "" }, + { 0x044B, "\x79" }, + { 0x044C, "" }, + { 0x044D, "\x65" }, + { 0x044E, "\x79\x75" }, + { 0x044F, "\x79\x61" }, + { 0x0451, "\x79\x6F" }, + { 0x0454, "\x79\x65" }, + { 0x0456, "\x69" }, + { 0x0457, "\x69" }, + { 0x045E, "\x75" }, + { 0x0491, "\x67" }, +}; + +constexpr FoldEntry TRANSLIT_UK[] = { + { 0x03B1, "\x61" }, + { 0x03B2, "\x76" }, + { 0x03B3, "\x67" }, + { 0x03B4, "\x64" }, + { 0x03B5, "\x65" }, + { 0x03B6, "\x7A" }, + { 0x03B7, "\x69" }, + { 0x03B8, "\x74\x68" }, + { 0x03B9, "\x69" }, + { 0x03BA, "\x6B" }, + { 0x03BB, "\x6C" }, + { 0x03BC, "\x6D" }, + { 0x03BD, "\x6E" }, + { 0x03BE, "\x78" }, + { 0x03BF, "\x6F" }, + { 0x03C0, "\x70" }, + { 0x03C1, "\x72" }, + { 0x03C2, "\x73" }, + { 0x03C3, "\x73" }, + { 0x03C4, "\x74" }, + { 0x03C5, "\x79" }, + { 0x03C6, "\x66" }, + { 0x03C7, "\x63\x68" }, + { 0x03C8, "\x70\x73" }, + { 0x03C9, "\x6F" }, + { 0x0430, "\x61" }, + { 0x0431, "\x62" }, + { 0x0432, "\x76" }, + { 0x0433, "\x68" }, + { 0x0434, "\x64" }, + { 0x0435, "\x65" }, + { 0x0436, "\x7A\x68" }, + { 0x0437, "\x7A" }, + { 0x0438, "\x79" }, + { 0x0439, "\x79" }, + { 0x043A, "\x6B" }, + { 0x043B, "\x6C" }, + { 0x043C, "\x6D" }, + { 0x043D, "\x6E" }, + { 0x043E, "\x6F" }, + { 0x043F, "\x70" }, + { 0x0440, "\x72" }, + { 0x0441, "\x73" }, + { 0x0442, "\x74" }, + { 0x0443, "\x75" }, + { 0x0444, "\x66" }, + { 0x0445, "\x6B\x68" }, + { 0x0446, "\x74\x73" }, + { 0x0447, "\x63\x68" }, + { 0x0448, "\x73\x68" }, + { 0x0449, "\x73\x68\x63\x68" }, + { 0x044A, "" }, + { 0x044B, "\x79" }, + { 0x044C, "" }, + { 0x044D, "\x65" }, + { 0x044E, "\x79\x75" }, + { 0x044F, "\x79\x61" }, + { 0x0451, "\x79\x6F" }, + { 0x0454, "\x79\x65" }, + { 0x0456, "\x69" }, + { 0x0457, "\x69" }, + { 0x045E, "\x75" }, + { 0x0491, "\x67" }, +}; + +constexpr FoldEntry TRANSLIT_SR[] = { + { 0x03B1, "\x61" }, + { 0x03B2, "\x76" }, + { 0x03B3, "\x67" }, + { 0x03B4, "\x64" }, + { 0x03B5, "\x65" }, + { 0x03B6, "\x7A" }, + { 0x03B7, "\x69" }, + { 0x03B8, "\x74\x68" }, + { 0x03B9, "\x69" }, + { 0x03BA, "\x6B" }, + { 0x03BB, "\x6C" }, + { 0x03BC, "\x6D" }, + { 0x03BD, "\x6E" }, + { 0x03BE, "\x78" }, + { 0x03BF, "\x6F" }, + { 0x03C0, "\x70" }, + { 0x03C1, "\x72" }, + { 0x03C2, "\x73" }, + { 0x03C3, "\x73" }, + { 0x03C4, "\x74" }, + { 0x03C5, "\x79" }, + { 0x03C6, "\x66" }, + { 0x03C7, "\x63\x68" }, + { 0x03C8, "\x70\x73" }, + { 0x03C9, "\x6F" }, + { 0x0430, "\x61" }, + { 0x0431, "\x62" }, + { 0x0432, "\x76" }, + { 0x0433, "\x67" }, + { 0x0434, "\x64" }, + { 0x0435, "\x65" }, + { 0x0436, "\x7A" }, + { 0x0437, "\x7A" }, + { 0x0438, "\x69" }, + { 0x0439, "\x79" }, + { 0x043A, "\x6B" }, + { 0x043B, "\x6C" }, + { 0x043C, "\x6D" }, + { 0x043D, "\x6E" }, + { 0x043E, "\x6F" }, + { 0x043F, "\x70" }, + { 0x0440, "\x72" }, + { 0x0441, "\x73" }, + { 0x0442, "\x74" }, + { 0x0443, "\x75" }, + { 0x0444, "\x66" }, + { 0x0445, "\x68" }, + { 0x0446, "\x63" }, + { 0x0447, "\x63" }, + { 0x0448, "\x73" }, + { 0x0449, "\x73\x68\x63\x68" }, + { 0x044A, "" }, + { 0x044B, "\x79" }, + { 0x044C, "" }, + { 0x044D, "\x65" }, + { 0x044E, "\x79\x75" }, + { 0x044F, "\x79\x61" }, + { 0x0451, "\x79\x6F" }, + { 0x0452, "\x64\x6A" }, + { 0x0454, "\x79\x65" }, + { 0x0456, "\x69" }, + { 0x0457, "\x69" }, + { 0x0458, "\x6A" }, + { 0x0459, "\x6C\x6A" }, + { 0x045A, "\x6E\x6A" }, + { 0x045B, "\x63" }, + { 0x045E, "\x75" }, + { 0x045F, "\x64\x7A" }, + { 0x0491, "\x67" }, +}; + +/** Ranges of U+0080..U+FFFF that fold to a space (not letters, digits or combining marks), sorted. */ +constexpr uint32_t FOLD_SEPARATORS[][2] = { + { 0x0080, 0x00A9 }, + { 0x00AB, 0x00B1 }, + { 0x00B4, 0x00B4 }, + { 0x00B6, 0x00B8 }, + { 0x00BB, 0x00BB }, + { 0x00BF, 0x00BF }, + { 0x00D7, 0x00D7 }, + { 0x00F7, 0x00F7 }, + { 0x02C2, 0x02C5 }, + { 0x02D2, 0x02DF }, + { 0x02E5, 0x02EB }, + { 0x02ED, 0x02ED }, + { 0x02EF, 0x02FF }, + { 0x0375, 0x0375 }, + { 0x0378, 0x0379 }, + { 0x037E, 0x037E }, + { 0x0380, 0x0385 }, + { 0x0387, 0x0387 }, + { 0x038B, 0x038B }, + { 0x038D, 0x038D }, + { 0x03A2, 0x03A2 }, + { 0x03F6, 0x03F6 }, + { 0x0482, 0x0482 }, + { 0x0488, 0x0489 }, + { 0x0530, 0x0530 }, + { 0x0557, 0x0558 }, + { 0x055A, 0x055F }, + { 0x0589, 0x0590 }, + { 0x05BE, 0x05BE }, + { 0x05C0, 0x05C0 }, + { 0x05C3, 0x05C3 }, + { 0x05C6, 0x05C6 }, + { 0x05C8, 0x05CF }, + { 0x05EB, 0x05EE }, + { 0x05F3, 0x060F }, + { 0x061B, 0x061F }, + { 0x066A, 0x066D }, + { 0x06D4, 0x06D4 }, + { 0x06DD, 0x06DE }, + { 0x06E9, 0x06E9 }, + { 0x06FD, 0x06FE }, + { 0x0700, 0x070F }, + { 0x074B, 0x074C }, + { 0x07B2, 0x07BF }, + { 0x07F6, 0x07F9 }, + { 0x07FB, 0x07FC }, + { 0x07FE, 0x07FF }, + { 0x082E, 0x083F }, + { 0x085C, 0x085F }, + { 0x086B, 0x089F }, + { 0x08B5, 0x08B5 }, + { 0x08C8, 0x08D2 }, + { 0x08E2, 0x08E2 }, + { 0x0903, 0x0903 }, + { 0x093B, 0x093B }, + { 0x093E, 0x0940 }, + { 0x0949, 0x094C }, + { 0x094E, 0x094F }, + { 0x0964, 0x0965 }, + { 0x0970, 0x0970 }, + { 0x0982, 0x0984 }, + { 0x098D, 0x098E }, + { 0x0991, 0x0992 }, + { 0x09A9, 0x09A9 }, + { 0x09B1, 0x09B1 }, + { 0x09B3, 0x09B5 }, + { 0x09BA, 0x09BB }, + { 0x09BE, 0x09C0 }, + { 0x09C5, 0x09CC }, + { 0x09CF, 0x09DB }, + { 0x09DE, 0x09DE }, + { 0x09E4, 0x09E5 }, + { 0x09F2, 0x09F3 }, + { 0x09FA, 0x09FB }, + { 0x09FD, 0x09FD }, + { 0x09FF, 0x0A00 }, + { 0x0A03, 0x0A04 }, + { 0x0A0B, 0x0A0E }, + { 0x0A11, 0x0A12 }, + { 0x0A29, 0x0A29 }, + { 0x0A31, 0x0A31 }, + { 0x0A34, 0x0A34 }, + { 0x0A37, 0x0A37 }, + { 0x0A3A, 0x0A3B }, + { 0x0A3D, 0x0A40 }, + { 0x0A43, 0x0A46 }, + { 0x0A49, 0x0A4A }, + { 0x0A4E, 0x0A50 }, + { 0x0A52, 0x0A58 }, + { 0x0A5D, 0x0A5D }, + { 0x0A5F, 0x0A65 }, + { 0x0A76, 0x0A80 }, + { 0x0A83, 0x0A84 }, + { 0x0A8E, 0x0A8E }, + { 0x0A92, 0x0A92 }, + { 0x0AA9, 0x0AA9 }, + { 0x0AB1, 0x0AB1 }, + { 0x0AB4, 0x0AB4 }, + { 0x0ABA, 0x0ABB }, + { 0x0ABE, 0x0AC0 }, + { 0x0AC6, 0x0AC6 }, + { 0x0AC9, 0x0ACC }, + { 0x0ACE, 0x0ACF }, + { 0x0AD1, 0x0ADF }, + { 0x0AE4, 0x0AE5 }, + { 0x0AF0, 0x0AF8 }, + { 0x0B00, 0x0B00 }, + { 0x0B02, 0x0B04 }, + { 0x0B0D, 0x0B0E }, + { 0x0B11, 0x0B12 }, + { 0x0B29, 0x0B29 }, + { 0x0B31, 0x0B31 }, + { 0x0B34, 0x0B34 }, + { 0x0B3A, 0x0B3B }, + { 0x0B3E, 0x0B3E }, + { 0x0B40, 0x0B40 }, + { 0x0B45, 0x0B4C }, + { 0x0B4E, 0x0B54 }, + { 0x0B57, 0x0B5B }, + { 0x0B5E, 0x0B5E }, + { 0x0B64, 0x0B65 }, + { 0x0B70, 0x0B70 }, + { 0x0B78, 0x0B81 }, + { 0x0B84, 0x0B84 }, + { 0x0B8B, 0x0B8D }, + { 0x0B91, 0x0B91 }, + { 0x0B96, 0x0B98 }, + { 0x0B9B, 0x0B9B }, + { 0x0B9D, 0x0B9D }, + { 0x0BA0, 0x0BA2 }, + { 0x0BA5, 0x0BA7 }, + { 0x0BAB, 0x0BAD }, + { 0x0BBA, 0x0BBF }, + { 0x0BC1, 0x0BCC }, + { 0x0BCE, 0x0BCF }, + { 0x0BD1, 0x0BE5 }, + { 0x0BF3, 0x0BFF }, + { 0x0C01, 0x0C03 }, + { 0x0C0D, 0x0C0D }, + { 0x0C11, 0x0C11 }, + { 0x0C29, 0x0C29 }, + { 0x0C3A, 0x0C3C }, + { 0x0C41, 0x0C45 }, + { 0x0C49, 0x0C49 }, + { 0x0C4E, 0x0C54 }, + { 0x0C57, 0x0C57 }, + { 0x0C5B, 0x0C5F }, + { 0x0C64, 0x0C65 }, + { 0x0C70, 0x0C77 }, + { 0x0C7F, 0x0C7F }, + { 0x0C82, 0x0C84 }, + { 0x0C8D, 0x0C8D }, + { 0x0C91, 0x0C91 }, + { 0x0CA9, 0x0CA9 }, + { 0x0CB4, 0x0CB4 }, + { 0x0CBA, 0x0CBB }, + { 0x0CBE, 0x0CBE }, + { 0x0CC0, 0x0CC5 }, + { 0x0CC7, 0x0CCB }, + { 0x0CCE, 0x0CDD }, + { 0x0CDF, 0x0CDF }, + { 0x0CE4, 0x0CE5 }, + { 0x0CF0, 0x0CF0 }, + { 0x0CF3, 0x0CFF }, + { 0x0D02, 0x0D03 }, + { 0x0D0D, 0x0D0D }, + { 0x0D11, 0x0D11 }, + { 0x0D3E, 0x0D40 }, + { 0x0D45, 0x0D4C }, + { 0x0D4F, 0x0D53 }, + { 0x0D57, 0x0D57 }, + { 0x0D64, 0x0D65 }, + { 0x0D79, 0x0D79 }, + { 0x0D80, 0x0D80 }, + { 0x0D82, 0x0D84 }, + { 0x0D97, 0x0D99 }, + { 0x0DB2, 0x0DB2 }, + { 0x0DBC, 0x0DBC }, + { 0x0DBE, 0x0DBF }, + { 0x0DC7, 0x0DC9 }, + { 0x0DCB, 0x0DD1 }, + { 0x0DD5, 0x0DD5 }, + { 0x0DD7, 0x0DE5 }, + { 0x0DF0, 0x0E00 }, + { 0x0E3B, 0x0E3F }, + { 0x0E4F, 0x0E4F }, + { 0x0E5A, 0x0E80 }, + { 0x0E83, 0x0E83 }, + { 0x0E85, 0x0E85 }, + { 0x0E8B, 0x0E8B }, + { 0x0EA4, 0x0EA4 }, + { 0x0EA6, 0x0EA6 }, + { 0x0EBE, 0x0EBF }, + { 0x0EC5, 0x0EC5 }, + { 0x0EC7, 0x0EC7 }, + { 0x0ECE, 0x0ECF }, + { 0x0EDA, 0x0EDB }, + { 0x0EE0, 0x0EFF }, + { 0x0F01, 0x0F17 }, + { 0x0F1A, 0x0F1F }, + { 0x0F34, 0x0F34 }, + { 0x0F36, 0x0F36 }, + { 0x0F38, 0x0F38 }, + { 0x0F3A, 0x0F3F }, + { 0x0F48, 0x0F48 }, + { 0x0F6D, 0x0F70 }, + { 0x0F7F, 0x0F7F }, + { 0x0F85, 0x0F85 }, + { 0x0F98, 0x0F98 }, + { 0x0FBD, 0x0FC5 }, + { 0x0FC7, 0x0FFF }, + { 0x102B, 0x102C }, + { 0x1031, 0x1031 }, + { 0x1038, 0x1038 }, + { 0x103B, 0x103C }, + { 0x104A, 0x104F }, + { 0x1056, 0x1057 }, + { 0x1062, 0x1064 }, + { 0x1067, 0x106D }, + { 0x1083, 0x1084 }, + { 0x1087, 0x108C }, + { 0x108F, 0x108F }, + { 0x109A, 0x109C }, + { 0x109E, 0x109F }, + { 0x10C6, 0x10C6 }, + { 0x10C8, 0x10CC }, + { 0x10CE, 0x10CF }, + { 0x10FB, 0x10FB }, + { 0x1249, 0x1249 }, + { 0x124E, 0x124F }, + { 0x1257, 0x1257 }, + { 0x1259, 0x1259 }, + { 0x125E, 0x125F }, + { 0x1289, 0x1289 }, + { 0x128E, 0x128F }, + { 0x12B1, 0x12B1 }, + { 0x12B6, 0x12B7 }, + { 0x12BF, 0x12BF }, + { 0x12C1, 0x12C1 }, + { 0x12C6, 0x12C7 }, + { 0x12D7, 0x12D7 }, + { 0x1311, 0x1311 }, + { 0x1316, 0x1317 }, + { 0x135B, 0x135C }, + { 0x1360, 0x1368 }, + { 0x137D, 0x137F }, + { 0x1390, 0x139F }, + { 0x13F6, 0x13F7 }, + { 0x13FE, 0x1400 }, + { 0x166D, 0x166E }, + { 0x1680, 0x1680 }, + { 0x169B, 0x169F }, + { 0x16EB, 0x16ED }, + { 0x16F9, 0x16FF }, + { 0x170D, 0x170D }, + { 0x1715, 0x171F }, + { 0x1735, 0x173F }, + { 0x1754, 0x175F }, + { 0x176D, 0x176D }, + { 0x1771, 0x1771 }, + { 0x1774, 0x177F }, + { 0x17B6, 0x17B6 }, + { 0x17BE, 0x17C5 }, + { 0x17C7, 0x17C8 }, + { 0x17D4, 0x17D6 }, + { 0x17D8, 0x17DB }, + { 0x17DE, 0x17DF }, + { 0x17EA, 0x17EF }, + { 0x17FA, 0x180A }, + { 0x180E, 0x180F }, + { 0x181A, 0x181F }, + { 0x1879, 0x187F }, + { 0x18AB, 0x18AF }, + { 0x18F6, 0x18FF }, + { 0x191F, 0x191F }, + { 0x1923, 0x1926 }, + { 0x1929, 0x1931 }, + { 0x1933, 0x1938 }, + { 0x193C, 0x1945 }, + { 0x196E, 0x196F }, + { 0x1975, 0x197F }, + { 0x19AC, 0x19AF }, + { 0x19CA, 0x19CF }, + { 0x19DB, 0x19FF }, + { 0x1A19, 0x1A1A }, + { 0x1A1C, 0x1A1F }, + { 0x1A55, 0x1A55 }, + { 0x1A57, 0x1A57 }, + { 0x1A5F, 0x1A5F }, + { 0x1A61, 0x1A61 }, + { 0x1A63, 0x1A64 }, + { 0x1A6D, 0x1A72 }, + { 0x1A7D, 0x1A7E }, + { 0x1A8A, 0x1A8F }, + { 0x1A9A, 0x1AA6 }, + { 0x1AA8, 0x1AAF }, + { 0x1ABE, 0x1ABE }, + { 0x1AC1, 0x1AFF }, + { 0x1B04, 0x1B04 }, + { 0x1B35, 0x1B35 }, + { 0x1B3B, 0x1B3B }, + { 0x1B3D, 0x1B41 }, + { 0x1B43, 0x1B44 }, + { 0x1B4C, 0x1B4F }, + { 0x1B5A, 0x1B6A }, + { 0x1B74, 0x1B7F }, + { 0x1B82, 0x1B82 }, + { 0x1BA1, 0x1BA1 }, + { 0x1BA6, 0x1BA7 }, + { 0x1BAA, 0x1BAA }, + { 0x1BE7, 0x1BE7 }, + { 0x1BEA, 0x1BEC }, + { 0x1BEE, 0x1BEE }, + { 0x1BF2, 0x1BFF }, + { 0x1C24, 0x1C2B }, + { 0x1C34, 0x1C35 }, + { 0x1C38, 0x1C3F }, + { 0x1C4A, 0x1C4C }, + { 0x1C7E, 0x1C7F }, + { 0x1C89, 0x1C8F }, + { 0x1CBB, 0x1CBC }, + { 0x1CC0, 0x1CCF }, + { 0x1CD3, 0x1CD3 }, + { 0x1CE1, 0x1CE1 }, + { 0x1CF7, 0x1CF7 }, + { 0x1CFB, 0x1CFF }, + { 0x1DFA, 0x1DFA }, + { 0x1F16, 0x1F17 }, + { 0x1F1E, 0x1F1F }, + { 0x1F46, 0x1F47 }, + { 0x1F4E, 0x1F4F }, + { 0x1F58, 0x1F58 }, + { 0x1F5A, 0x1F5A }, + { 0x1F5C, 0x1F5C }, + { 0x1F5E, 0x1F5E }, + { 0x1F7E, 0x1F7F }, + { 0x1FB5, 0x1FB5 }, + { 0x1FBD, 0x1FBD }, + { 0x1FBF, 0x1FC1 }, + { 0x1FC5, 0x1FC5 }, + { 0x1FCD, 0x1FCF }, + { 0x1FD4, 0x1FD5 }, + { 0x1FDC, 0x1FDF }, + { 0x1FED, 0x1FF1 }, + { 0x1FF5, 0x1FF5 }, + { 0x1FFD, 0x206F }, + { 0x2072, 0x2073 }, + { 0x207A, 0x207E }, + { 0x208A, 0x208F }, + { 0x209D, 0x20CF }, + { 0x20DD, 0x20E0 }, + { 0x20E2, 0x20E4 }, + { 0x20F1, 0x2101 }, + { 0x2103, 0x2106 }, + { 0x2108, 0x2109 }, + { 0x2114, 0x2114 }, + { 0x2116, 0x2118 }, + { 0x211E, 0x2123 }, + { 0x2125, 0x2125 }, + { 0x2127, 0x2127 }, + { 0x2129, 0x2129 }, + { 0x212E, 0x212E }, + { 0x213A, 0x213B }, + { 0x2140, 0x2144 }, + { 0x214A, 0x214D }, + { 0x214F, 0x214F }, + { 0x218A, 0x245F }, + { 0x249C, 0x24E9 }, + { 0x2500, 0x2775 }, + { 0x2794, 0x2BFF }, + { 0x2C2F, 0x2C2F }, + { 0x2C5F, 0x2C5F }, + { 0x2CE5, 0x2CEA }, + { 0x2CF4, 0x2CFC }, + { 0x2CFE, 0x2CFF }, + { 0x2D26, 0x2D26 }, + { 0x2D28, 0x2D2C }, + { 0x2D2E, 0x2D2F }, + { 0x2D68, 0x2D6E }, + { 0x2D70, 0x2D7E }, + { 0x2D97, 0x2D9F }, + { 0x2DA7, 0x2DA7 }, + { 0x2DAF, 0x2DAF }, + { 0x2DB7, 0x2DB7 }, + { 0x2DBF, 0x2DBF }, + { 0x2DC7, 0x2DC7 }, + { 0x2DCF, 0x2DCF }, + { 0x2DD7, 0x2DD7 }, + { 0x2DDF, 0x2DDF }, + { 0x2E00, 0x2E2E }, + { 0x2E30, 0x3004 }, + { 0x3008, 0x3020 }, + { 0x302E, 0x3030 }, + { 0x3036, 0x3037 }, + { 0x303D, 0x3040 }, + { 0x3097, 0x3098 }, + { 0x309B, 0x309C }, + { 0x30A0, 0x30A0 }, + { 0x30FB, 0x30FB }, + { 0x3100, 0x3104 }, + { 0x3130, 0x3130 }, + { 0x318F, 0x3191 }, + { 0x3196, 0x319F }, + { 0x31C0, 0x31EF }, + { 0x3200, 0x321F }, + { 0x322A, 0x3247 }, + { 0x3250, 0x3250 }, + { 0x3260, 0x327F }, + { 0x328A, 0x32B0 }, + { 0x32C0, 0x33FF }, + { 0x4DC0, 0x4DFF }, + { 0x9FFD, 0x9FFF }, + { 0xA48D, 0xA4CF }, + { 0xA4FE, 0xA4FF }, + { 0xA60D, 0xA60F }, + { 0xA62C, 0xA63F }, + { 0xA670, 0xA673 }, + { 0xA67E, 0xA67E }, + { 0xA6F2, 0xA716 }, + { 0xA720, 0xA721 }, + { 0xA789, 0xA78A }, + { 0xA7C0, 0xA7C1 }, + { 0xA7CB, 0xA7F4 }, + { 0xA823, 0xA824 }, + { 0xA827, 0xA82B }, + { 0xA82D, 0xA82F }, + { 0xA836, 0xA83F }, + { 0xA874, 0xA881 }, + { 0xA8B4, 0xA8C3 }, + { 0xA8C6, 0xA8CF }, + { 0xA8DA, 0xA8DF }, + { 0xA8F8, 0xA8FA }, + { 0xA8FC, 0xA8FC }, + { 0xA92E, 0xA92F }, + { 0xA952, 0xA95F }, + { 0xA97D, 0xA97F }, + { 0xA983, 0xA983 }, + { 0xA9B4, 0xA9B5 }, + { 0xA9BA, 0xA9BB }, + { 0xA9BE, 0xA9CE }, + { 0xA9DA, 0xA9DF }, + { 0xA9FF, 0xA9FF }, + { 0xAA2F, 0xAA30 }, + { 0xAA33, 0xAA34 }, + { 0xAA37, 0xAA3F }, + { 0xAA4D, 0xAA4F }, + { 0xAA5A, 0xAA5F }, + { 0xAA77, 0xAA79 }, + { 0xAA7B, 0xAA7B }, + { 0xAA7D, 0xAA7D }, + { 0xAAC3, 0xAADA }, + { 0xAADE, 0xAADF }, + { 0xAAEB, 0xAAEB }, + { 0xAAEE, 0xAAF1 }, + { 0xAAF5, 0xAAF5 }, + { 0xAAF7, 0xAB00 }, + { 0xAB07, 0xAB08 }, + { 0xAB0F, 0xAB10 }, + { 0xAB17, 0xAB1F }, + { 0xAB27, 0xAB27 }, + { 0xAB2F, 0xAB2F }, + { 0xAB5B, 0xAB5B }, + { 0xAB6A, 0xAB6F }, + { 0xABE3, 0xABE4 }, + { 0xABE6, 0xABE7 }, + { 0xABE9, 0xABEC }, + { 0xABEE, 0xABEF }, + { 0xABFA, 0xABFF }, + { 0xD7A4, 0xD7AF }, + { 0xD7C7, 0xD7CA }, + { 0xD7FC, 0xF8FF }, + { 0xFA6E, 0xFA6F }, + { 0xFADA, 0xFAFF }, + { 0xFB07, 0xFB12 }, + { 0xFB18, 0xFB1C }, + { 0xFB29, 0xFB29 }, + { 0xFB37, 0xFB37 }, + { 0xFB3D, 0xFB3D }, + { 0xFB3F, 0xFB3F }, + { 0xFB42, 0xFB42 }, + { 0xFB45, 0xFB45 }, + { 0xFBB2, 0xFBD2 }, + { 0xFD3E, 0xFD4F }, + { 0xFD90, 0xFD91 }, + { 0xFDC8, 0xFDEF }, + { 0xFDFC, 0xFDFF }, + { 0xFE10, 0xFE1F }, + { 0xFE30, 0xFE6F }, + { 0xFE75, 0xFE75 }, + { 0xFEFD, 0xFF0F }, + { 0xFF1A, 0xFF20 }, + { 0xFF3B, 0xFF40 }, + { 0xFF5B, 0xFF65 }, + { 0xFFBF, 0xFFC1 }, + { 0xFFC8, 0xFFC9 }, + { 0xFFD0, 0xFFD1 }, + { 0xFFD8, 0xFFD9 }, + { 0xFFDD, 0xFFFF }, +}; + +/** Apostrophes, dropped by both folds. */ +constexpr uint32_t FOLD_APOSTROPHES[] = { 0x0027, 0x0060, 0x00B4, 0x02BB, 0x02BC, 0x2019 }; + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/QueryParse.cpp b/Apps/Maps/main/Source/search/QueryParse.cpp new file mode 100644 index 0000000..ca5e294 --- /dev/null +++ b/Apps/Maps/main/Source/search/QueryParse.cpp @@ -0,0 +1,237 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "QueryParse.h" + +#include "TextFold.h" + +#include +#include +#include +#include + +namespace maps { + +static uint32_t decode_utf8(const char* s, size_t len, size_t* out_len) { + const unsigned char* u = (const unsigned char*)s; + if (len == 0) { *out_len = 0; return 0; } + if (u[0] < 0x80) { *out_len = 1; return u[0]; } + if ((u[0] & 0xE0) == 0xC0 && len >= 2) { *out_len = 2; return ((u[0] & 0x1F) << 6) | (u[1] & 0x3F); } + if ((u[0] & 0xF0) == 0xE0 && len >= 3) { *out_len = 3; return ((u[0] & 0x0F) << 12) | ((u[1] & 0x3F) << 6) | (u[2] & 0x3F); } + if ((u[0] & 0xF8) == 0xF0 && len >= 4) { *out_len = 4; return ((u[0] & 0x07) << 18) | ((u[1] & 0x3F) << 12) | ((u[2] & 0x3F) << 6) | (u[3] & 0x3F); } + *out_len = 1; return u[0]; +} + +static bool is_letter(uint32_t cp) { + if (cp < 128) return (cp >= 'a' && cp <= 'z') || (cp >= 'A' && cp <= 'Z'); + return true; +} + +/** The suffix letter as the study stores it (data.py _sfx): the first letter of its Russian-profile transliteration. */ +static char translit(uint32_t cp) { + char letter[5]; + const size_t used = utf8Append(cp, letter, sizeof(letter), 0); + if (used == 0) return 0; + char folded[16]; + transliterate(letter, Translit::Russian, folded, sizeof(folded)); + const char first = folded[0]; + return (first >= 'a' && first <= 'z') ? first : 0; +} + +static void trim(char* s) { + size_t len = strlen(s); + while (len > 0 && std::isspace((unsigned char)s[len - 1])) s[--len] = 0; + size_t start = 0; + while (s[start] && std::isspace((unsigned char)s[start])) start++; + if (start > 0) memmove(s, s + start, len - start + 1); +} + +bool parseHouseNumber(const char* text, size_t length, HouseNumber& out) { + char buf[32]; + if (length >= sizeof(buf)) return false; + memcpy(buf, text, length); + buf[length] = 0; + trim(buf); + + const char* s = buf; + uint32_t num = 0; + int digits = 0; + while (*s && std::isdigit((unsigned char)*s) && digits < 5) { + num = num * 10 + (*s - '0'); + s++; digits++; + } + if (digits == 0 || num > 65535) return false; + + while (*s && std::isspace((unsigned char)*s)) s++; + + if (*s == 0) { + out.number = static_cast(num); out.aux = 0; out.flags = 0; + out.suffix = 0; + return true; + } + + if (*s == '/') { + s++; + while (*s && std::isspace((unsigned char)*s)) s++; + if (std::isdigit((unsigned char)*s)) { + uint32_t aux = 0; int aux_digits = 0; + while (*s && std::isdigit((unsigned char)*s) && aux_digits < 5) { + aux = aux * 10 + (*s - '0'); + s++; aux_digits++; + } + if (aux > 65535) return false; + while (*s && std::isspace((unsigned char)*s)) s++; + size_t l; uint32_t cp = decode_utf8(s, strlen(s), &l); + char sfx = (l > 0 && is_letter(cp)) ? translit(cp) : 0; + if (s[l] == 0) { + out.number = static_cast(num); out.aux = static_cast(aux); out.suffix = sfx; out.flags = HOUSE_SLASH; + return true; + } + } else { + size_t l; uint32_t cp = decode_utf8(s, strlen(s), &l); + if (l > 0 && is_letter(cp) && s[l] == 0) { + out.number = static_cast(num); out.aux = 0; out.suffix = translit(cp); out.flags = HOUSE_SLASH; + return true; + } + } + } else if (*s == '-') { + s++; + while (*s && std::isspace((unsigned char)*s)) s++; + uint32_t aux = 0; int aux_digits = 0; + while (*s && std::isdigit((unsigned char)*s) && aux_digits < 5) { + aux = aux * 10 + (*s - '0'); + s++; aux_digits++; + } + if (aux_digits > 0 && *s == 0 && aux <= 65535) { + out.number = static_cast(num); out.aux = static_cast(aux); out.suffix = 0; out.flags = HOUSE_RANGE; + return true; + } + } else { + size_t l; uint32_t cp = decode_utf8(s, strlen(s), &l); + if (l > 0 && is_letter(cp) && s[l] == 0) { + out.number = static_cast(num); out.aux = 0; out.suffix = translit(cp); out.flags = 0; + return true; + } + } + return false; +} + +static bool is_numtok(const char* s) { + if (!*s) return false; + int digits = 0; + while (*s && std::isdigit((unsigned char)*s) && digits < 5) { + s++; digits++; + } + if (digits == 0) return false; + if (*s == 0) return true; + if (*s == '/') { + s++; + if (std::isdigit((unsigned char)*s)) { + int aux_digits = 0; + while (*s && std::isdigit((unsigned char)*s) && aux_digits < 5) { + s++; aux_digits++; + } + if (aux_digits == 0) return false; + size_t l; uint32_t cp = decode_utf8(s, strlen(s), &l); + if (l == 0 || (l > 0 && is_letter(cp) && s[l] == 0)) return true; + } else { + size_t l; uint32_t cp = decode_utf8(s, strlen(s), &l); + if (l > 0 && is_letter(cp) && s[l] == 0) return true; + } + } else { + size_t l; uint32_t cp = decode_utf8(s, strlen(s), &l); + if (l > 0 && is_letter(cp) && s[l] == 0) return true; + } + return false; +} + +int parseQuery(const char* query, QueryReading out[3]) { + char q[256]; + strncpy(q, query, sizeof(q) - 1); + q[sizeof(q) - 1] = 0; + trim(q); + + // re.sub(r'\s*/\s*', '/', q) + char q2[256]; int j = 0; + for (int i = 0; q[i]; i++) { + if (q[i] == '/') { + while (j > 0 && std::isspace((unsigned char)q2[j - 1])) q2[--j] = 0; + q2[j++] = '/'; + while (q[i + 1] && std::isspace((unsigned char)q[i + 1])) i++; + } else { + q2[j++] = q[i]; + } + } + q2[j] = 0; + + // re.sub(r'(\d)\s+([A-Za-z])$', r'\1\2', q) + int len = strlen(q2); + if (len >= 2 && std::isalpha((unsigned char)q2[len - 1])) { + int k = len - 2; + while (k >= 0 && std::isspace((unsigned char)q2[k])) k--; + if (k >= 0 && std::isdigit((unsigned char)q2[k])) { + char q3[256]; int m = 0; + for (int n = 0; n <= k; n++) q3[m++] = q2[n]; + q3[m++] = q2[len - 1]; + q3[m] = 0; + strcpy(q2, q3); + } + } + + char toks[16][32]; int tok_count = 0; + char* cur = q2; + while (*cur && tok_count < 16) { + while (*cur && std::isspace((unsigned char)*cur)) cur++; + if (!*cur) break; + int start = 0; + while (*cur && !std::isspace((unsigned char)*cur) && start < 31) { + toks[tok_count][start++] = *cur++; + } + toks[tok_count][start] = 0; + tok_count++; + } + + int readings = 0; + if (tok_count >= 2) { + char last[32]; strcpy(last, toks[tok_count - 1]); + size_t last_len = strlen(last); + if (last_len > 0 && last[last_len - 1] == '.') last[last_len - 1] = 0; + if (is_numtok(last)) { + int s_len = 0; + for (int i = 0; i < tok_count - 1; i++) { + int t_len = strlen(toks[i]); + if (s_len + t_len + 1 >= 96) break; + if (s_len > 0) out[readings].street[s_len++] = ' '; + memcpy(out[readings].street + s_len, toks[i], t_len); + s_len += t_len; + } + out[readings].street[s_len] = 0; + strncpy(out[readings].number, last, 23); + out[readings].number[23] = 0; + readings++; + } + if (readings < 3 && is_numtok(toks[0])) { + int s_len = 0; + for (int i = 1; i < tok_count; i++) { + int t_len = strlen(toks[i]); + if (s_len + t_len + 1 >= 96) break; + if (s_len > 0) out[readings].street[s_len++] = ' '; + memcpy(out[readings].street + s_len, toks[i], t_len); + s_len += t_len; + } + out[readings].street[s_len] = 0; + strncpy(out[readings].number, toks[0], 23); + out[readings].number[23] = 0; + readings++; + } + } + + if (readings < 3) { + strncpy(out[readings].street, q2, 95); + out[readings].street[95] = 0; + out[readings].number[0] = 0; + readings++; + } + + return readings; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/QueryParse.h b/Apps/Maps/main/Source/search/QueryParse.h new file mode 100644 index 0000000..b0ccb0a --- /dev/null +++ b/Apps/Maps/main/Source/search/QueryParse.h @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include +#include + +namespace maps { + +/** A house number as stored in the index: number, second part, a one-letter suffix and flags. */ +struct HouseNumber { + uint16_t number; + /** The part after "/" ("1394/12": 12), or the end of a range ("3-5": 5); 0 for none. */ + uint16_t aux; + /** Lowercase ASCII letter, 0 for none. */ + char suffix; + /** HOUSE_SLASH, HOUSE_RANGE. */ + uint8_t flags; +}; + +constexpr uint8_t HOUSE_SLASH = 1; +constexpr uint8_t HOUSE_RANGE = 2; + +/** parseHouseNumber(): parses a house number string; false when it doesn't fit any form (stored as text then). */ +bool parseHouseNumber(const char* text, size_t length, HouseNumber& out); + +/** One reading of a query: the street (or name) text and the house number text ("" for none). */ +struct QueryReading { + char street[96]; + char number[24]; +}; + +/** parseQuery(): the readings of a query, most likely first (at most 3). Returns how many. */ +int parseQuery(const char* query, QueryReading out[3]); + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/ScratchFiles.h b/Apps/Maps/main/Source/search/ScratchFiles.h new file mode 100644 index 0000000..28f4bde --- /dev/null +++ b/Apps/Maps/main/Source/search/ScratchFiles.h @@ -0,0 +1,117 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include +#include + +namespace maps { + +/** + * Files for the streaming index build (M6): sorted runs, per-tile extracts and the index being written. The host + * uses stdio (StdioScratchFiles); the device does every call on its I/O task and keeps at most two files really open + * (the card's FAT mount allows four for the whole system), reopening the others at their position when used. Callers + * write and read in pieces of a few KB, never byte by byte. Paths are full paths. A handle is a small number >= 0. + */ +class ScratchFiles { +public: + static constexpr int MAX_OPEN = 24; + + virtual ~ScratchFiles() = default; + /** Creates (or empties) a file for writing; -1 on error. */ + virtual int create(const char* path) = 0; + /** Opens a file for reading; -1 on error. */ + virtual int openRead(const char* path) = 0; + /** Opens a file for writing at its end (creates it when missing); -1 on error. */ + virtual int openAppend(const char* path) = 0; + virtual bool write(int handle, const void* data, uint32_t size) = 0; + /** Reads up to size bytes at the current position: the count, 0 at the end, -1 on error. */ + virtual int read(int handle, void* data, uint32_t size) = 0; + /** Moves the position of a file opened for reading. */ + virtual bool seek(int handle, uint32_t offset) = 0; + /** Closes the handle; for a written file false when its data could not be written. */ + virtual bool close(int handle) = 0; + /** Removes a file; true when it is gone afterwards (also when it was not there). */ + virtual bool remove(const char* path) = 0; + /** Renames a file, replacing the target. */ + virtual bool rename(const char* from, const char* to) = 0; +}; + +/** ScratchFiles over stdio, for the host tools and tests. */ +class StdioScratchFiles final : public ScratchFiles { +public: + ~StdioScratchFiles() override { + for (FILE*& f : files) { + if (f != nullptr) { + fclose(f); + f = nullptr; + } + } + } + int create(const char* path) override { return put(fopen(path, "wb")); } + int openRead(const char* path) override { return put(fopen(path, "rb")); } + int openAppend(const char* path) override { return put(fopen(path, "ab")); } + bool write(int handle, const void* data, uint32_t size) override { + FILE* f = at(handle); + return f != nullptr && (size == 0 || fwrite(data, 1, size, f) == size); + } + int read(int handle, void* data, uint32_t size) override { + FILE* f = at(handle); + if (f == nullptr) { + return -1; + } + const size_t n = fread(data, 1, size, f); + return n == 0 && ferror(f) ? -1 : static_cast(n); + } + bool seek(int handle, uint32_t offset) override { + FILE* f = at(handle); + return f != nullptr && fseek(f, static_cast(offset), SEEK_SET) == 0; + } + bool close(int handle) override { + FILE* f = at(handle); + if (f == nullptr) { + return false; + } + files[handle] = nullptr; + return fclose(f) == 0; + } + bool remove(const char* path) override { + ::remove(path); + FILE* f = fopen(path, "rb"); + if (f != nullptr) { + fclose(f); + return false; + } + return true; + } + bool rename(const char* from, const char* to) override { + ::remove(to); + return ::rename(from, to) == 0; + } + /** How many files are open now (for tests that check every handle is closed). */ + int openCount() const { + int n = 0; + for (FILE* f : files) { + n += f != nullptr ? 1 : 0; + } + return n; + } + +private: + int put(FILE* f) { + if (f == nullptr) { + return -1; + } + for (int i = 0; i < MAX_OPEN; i++) { + if (files[i] == nullptr) { + files[i] = f; + return i; + } + } + fclose(f); + return -1; + } + FILE* at(int handle) const { return handle >= 0 && handle < MAX_OPEN ? files[handle] : nullptr; } + FILE* files[MAX_OPEN] = {}; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchEngine.cpp b/Apps/Maps/main/Source/search/SearchEngine.cpp new file mode 100644 index 0000000..1ef560e --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchEngine.cpp @@ -0,0 +1,1110 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "SearchEngine.h" + +#include "AddressMatch.h" +#include "QueryParse.h" + +#include "../render/PoiTypes.h" + +#include +#include +#include +#include +#include +#include +#include + +namespace maps { + +namespace { + +constexpr double PI_VALUE = 3.14159265358979323846; +/** The side of the world in z14 units: 2^14 tiles of 4096 units. */ +constexpr double WORLD_UNITS = 16384.0 * 4096.0; +constexpr size_t KEY_BYTES = 256; +/** Candidates kept after a ranking, and the size at which the list is ranked and cut down in between. */ +constexpr size_t MAX_RANKED = 300; +constexpr size_t PRUNE_AT = 1200; +/** Points of one name group (40 x Billa) and of a category run that are kept: the nearest. */ +constexpr size_t NAME_POINTS = 64; +constexpr size_t CATEGORY_POINTS = 100; +constexpr size_t STREET_CLUSTERS = 16; +constexpr uint32_t RECORD_CHUNK = 512; +constexpr int MIN_CHARACTERS = 3; +constexpr int DUPLICATE_UNITS = 4; +/** A settlement counts while distance / class size is at most this (a village 3.2 km, a city 12 km). */ +constexpr float PLACE_SCORE_LIMIT = 4.0f; +/** A suburb of the city or town counts while distance / 700 m is at most this (2.1 km). */ +constexpr float SUBURB_SCORE_LIMIT = 3.0f; + +// ---------------------------------------------------------------------------------------------------- text helpers + +/** Copies text, cut on a UTF-8 character boundary to fit size (0-terminated). */ +void copyText(char* out, size_t size, const char* text) { + if (size == 0) { + return; + } + size_t length = strlen(text); + if (length >= size) { + length = size - 1; + while (length > 0 && (static_cast(text[length]) & 0xC0) == 0x80) { + length--; + } + } + memcpy(out, text, length); + out[length] = 0; +} + +int characterCount(const char* text) { + int count = 0; + for (; *text != 0; text++) { + if ((static_cast(*text) & 0xC0) != 0x80) { + count++; + } + } + return count; +} + +/** Byte length of the first n characters. */ +size_t bytesOfCharacters(const char* text, int n) { + size_t at = 0; + int seen = 0; + while (text[at] != 0) { + if ((static_cast(text[at]) & 0xC0) != 0x80) { + if (seen == n) { + break; + } + seen++; + } + at++; + } + return at; +} + +void trimBlanks(char* text) { + size_t length = strlen(text); + while (length > 0 && text[length - 1] == ' ') { + text[--length] = 0; + } + size_t start = 0; + while (text[start] == ' ') { + start++; + } + if (start > 0) { + memmove(text, text + start, length - start + 1); + } +} + +/** The lookup keys of a text: folded Latin, and for Greek and Cyrillic also the folded original script (first). */ +int makeKeys(const char* text, char keys[2][KEY_BYTES]) { + int count = 0; + if (hasNonLatin(text)) { + foldOriginal(text, keys[count], KEY_BYTES); + trimBlanks(keys[count]); + count++; + } + foldLatin(text, keys[count], KEY_BYTES); + trimBlanks(keys[count]); + if (count == 1 && strcmp(keys[0], keys[1]) == 0) { + return 1; + } + return count + 1; +} + +/** "cafe" -> "Cafe", "isolated_dwelling" -> "Isolated dwelling". */ +void humanise(const char* text, char* out, size_t size) { + size_t i = 0; + for (; text[i] != 0 && text[i] != '/' && i + 1 < size; i++) { + out[i] = text[i] == '_' ? ' ' : text[i]; + } + out[i] = 0; + if (out[0] >= 'a' && out[0] <= 'z') { + out[0] = static_cast(out[0] - 'a' + 'A'); + } +} + +/** The group and the plain-word type of a class table label ("poi/cafe/cafe", "place/town", "peak/peak"). */ +ResultGroup describeLabel(const char* label, char* type, size_t typeSize) { + if (strncmp(label, "poi/", 4) == 0) { + const char* classText = label + 4; + const char* slash = strchr(classText, '/'); + size_t classLength = slash != nullptr ? static_cast(slash - classText) : strlen(classText); + const char* sub = slash != nullptr ? slash + 1 : nullptr; + size_t subLength = sub != nullptr ? strlen(sub) : 0; + if (subLength == 1 && sub[0] == '*') { + sub = nullptr; + subLength = 0; + } + uint8_t id = poiTypeFor(classText, classLength, sub, subLength); + if (id != 0) { + copyText(type, typeSize, poiType(id).label); + } else { + const char* source = (sub != nullptr && subLength > 0 && strcmp(sub, "None") != 0) ? sub : classText; + char text[64]; + humanise(source, text, sizeof(text)); + copyText(type, typeSize, text); + } + return ResultGroup::Poi; + } + if (strncmp(label, "place/", 6) == 0) { + static const char* const NAMES[][2] = { + { "city", "City" }, { "town", "Town" }, { "village", "Village" }, { "hamlet", "Hamlet" }, { "suburb", "Suburb" }, + { "quarter", "Quarter" }, { "neighbourhood", "Neighbourhood" }, { "island", "Island" }, + { "isolated_dwelling", "Dwelling" }, { "locality", "Locality" }, + }; + const char* classText = label + 6; + for (const auto& entry : NAMES) { + if (strcmp(classText, entry[0]) == 0) { + copyText(type, typeSize, entry[1]); + return ResultGroup::Place; + } + } + char text[64]; + humanise(classText, text, sizeof(text)); + copyText(type, typeSize, text); + return ResultGroup::Place; + } + if (strncmp(label, "peak/", 5) == 0) { + copyText(type, typeSize, strcmp(label + 5, "cliff") == 0 ? "Cliff" : "Peak"); + } else if (strncmp(label, "water/", 6) == 0) { + copyText(type, typeSize, strcmp(label + 6, "lake") == 0 ? "Lake" : "Water"); + } else if (strncmp(label, "park/", 5) == 0) { + copyText(type, typeSize, "Park"); + } else if (strncmp(label, "aerodrome/", 10) == 0) { + copyText(type, typeSize, "Airport"); + } else { + char text[64]; + humanise(label, text, sizeof(text)); + copyText(type, typeSize, text); + } + return ResultGroup::Natural; +} + +/** Whether the folded text of a name equals one of the keys (also its transliteration in the region's profile). */ +bool isExactName(const char* name, const char* const* keys, int keyCount, Translit profile) { + char latin[KEY_BYTES * 2]; + char original[KEY_BYTES * 2]; + char translit[KEY_BYTES * 2]; + foldLatin(name, latin, sizeof(latin)); + trimBlanks(latin); + bool nonLatin = hasNonLatin(name); + if (nonLatin) { + foldOriginal(name, original, sizeof(original)); + transliterate(name, profile, translit, sizeof(translit)); + trimBlanks(original); + trimBlanks(translit); + } + for (int i = 0; i < keyCount; i++) { + if (strcmp(latin, keys[i]) == 0 || (nonLatin && (strcmp(original, keys[i]) == 0 || strcmp(translit, keys[i]) == 0))) { + return true; + } + } + return false; +} + +/** "Štefánikova 4": the number text of an address record as the street shows it. */ +void numberText(const SearchAddress& a, bool slashSecond, SearchIndexReader& reader, char* out, size_t size) { + if (a.isText) { + if (!reader.overflowText(a.textOffset, out, size)) { + out[0] = 0; + } + return; + } + char letter[3] = { 0, 0, 0 }; + if (a.suffix != 0) { + letter[0] = static_cast(a.suffix >= 'a' && a.suffix <= 'z' ? a.suffix - 'a' + 'A' : a.suffix); + } + if ((a.flags & HOUSE_RANGE) != 0 && a.aux != 0) { + snprintf(out, size, "%u-%u", a.number, a.aux); + } else if ((a.flags & HOUSE_SLASH) != 0 && a.aux != 0 && !slashSecond) { + snprintf(out, size, "%u/%u%s", a.number, a.aux, letter); + } else if ((a.flags & HOUSE_SLASH) != 0 && a.aux == 0 && letter[0] != 0 && !slashSecond) { + snprintf(out, size, "%u/%s", a.number, letter); + } else { + snprintf(out, size, "%u%s", a.number, letter); + } +} + +// ---------------------------------------------------------------------------------------------------- nearest records + +struct NearPoint { + uint64_t distanceSq; + SearchPointRecord record; +}; + +uint64_t distanceSquared(int32_t x, int32_t y, SearchOrigin origin) { + int64_t dx = static_cast(x) - origin.x; + int64_t dy = static_cast(y) - origin.y; + return static_cast(dx * dx + dy * dy); +} + +/** Reads a run of point records in chunks and keeps the k nearest to the origin, nearest first, appended to out. */ +bool nearestPoints(SearchIndexReader& reader, uint32_t first, uint32_t count, SearchOrigin origin, size_t k, GrowArray& out) { + GrowArray chunk; + GrowArray best; + for (uint32_t done = 0; done < count; done += RECORD_CHUNK) { + uint32_t n = std::min(RECORD_CHUNK, count - done); + if (!reader.pointRecords(first + done, n, chunk)) { + return false; + } + for (const SearchPointRecord& record : chunk) { + uint64_t d = distanceSquared(record.x, record.y, origin); + if (best.size() >= k && d >= best[best.size() - 1].distanceSq) { + continue; + } + NearPoint item = { d, record }; + size_t at; + if (best.size() < k) { + best.push(item); + at = best.size() - 1; + } else { + at = k - 1; + best[at] = item; + } + while (at > 0 && best[at - 1].distanceSq > best[at].distanceSq) { + std::swap(best[at - 1], best[at]); + at--; + } + } + } + for (const NearPoint& item : best) { + out.push(item); + } + return !out.failed(); +} + +// ---------------------------------------------------------------------------------------------------- categories + +const char* const LABELS_FOOD[] = { "poi/restaurant/*", "poi/fast_food/*", "poi/bar/*", "poi/beer/*", "poi/ice_cream/*", "poi/bakery/*" }; +const char* const LABELS_CAFE[] = { "poi/cafe/*", "poi/shop/coffee" }; +const char* const LABELS_SHOP[] = { "poi/shop/*", "poi/clothing_store/*", "poi/alcohol_shop/*", "poi/butcher/*", "poi/grocery/greengrocer", + "poi/grocery/deli", "poi/grocery/marketplace", "poi/grocery/department_store", "poi/bicycle/*", + "poi/music/*" }; +const char* const LABELS_SUPERMARKET[] = { "poi/grocery/supermarket", "poi/shop/supermarket", "poi/shop/convenience" }; +const char* const LABELS_PHARMACY[] = { "poi/pharmacy/*" }; +const char* const LABELS_TOILETS[] = { "poi/toilets/*" }; +const char* const LABELS_STOP[] = { "poi/bus/*", "poi/railway/tram_stop" }; +const char* const LABELS_PARKING[] = { "poi/parking/*" }; +const char* const LABELS_FUEL[] = { "poi/fuel/*" }; +const char* const LABELS_BANK[] = { "poi/atm/*", "poi/bank/*" }; +const char* const LABELS_HOSPITAL[] = { "poi/hospital/*" }; +const char* const LABELS_VIEWPOINT[] = { "poi/attraction/viewpoint" }; + +#define CATEGORY(name, labels) { name, labels, static_cast(sizeof(labels) / sizeof(labels[0])) } +const SearchCategory CATEGORIES[] = { + CATEGORY("Food and drink", LABELS_FOOD), CATEGORY("Cafe", LABELS_CAFE), CATEGORY("Shops", LABELS_SHOP), + CATEGORY("Supermarket", LABELS_SUPERMARKET), CATEGORY("Pharmacy", LABELS_PHARMACY), CATEGORY("Toilets", LABELS_TOILETS), + CATEGORY("Bus or tram stop", LABELS_STOP), CATEGORY("Parking", LABELS_PARKING), CATEGORY("Fuel and charging", LABELS_FUEL), + CATEGORY("Bank or ATM", LABELS_BANK), CATEGORY("Hospital", LABELS_HOSPITAL), CATEGORY("Viewpoint", LABELS_VIEWPOINT), +}; +#undef CATEGORY + +/** A pattern that ends in "/" and an asterisk covers every subclass of the class; any other pattern must be equal. */ +bool labelMatches(const char* pattern, const char* label) { + size_t length = strlen(pattern); + if (length >= 2 && pattern[length - 1] == '*' && pattern[length - 2] == '/') { + return strncmp(label, pattern, length - 1) == 0; + } + return strcmp(label, pattern) == 0; +} + +} // namespace + +/** The typed text prepared for lookups. */ +struct SearchEngine::Query { + char keys[2][KEY_BYTES]; + int keyCount; + QueryReading readings[3]; + int readingCount; +}; + +// ---------------------------------------------------------------------------------------------------- registry + +SearchEngine::SearchEngine() = default; + +SearchEngine::~SearchEngine() { + removeAll(); +} + +int SearchEngine::addIndex(SearchFile* file, const char* regionName, Translit profile, bool slashSecond) { + if (slotCount >= SEARCH_MAX_INDEXES || file == nullptr) { + return -1; + } + Slot* slot = new (std::nothrow) Slot(); + if (slot == nullptr) { + return -1; + } + if (!slot->reader.open(file)) { + delete slot; + return -1; + } + slot->file = file; + copyText(slot->region, sizeof(slot->region), regionName != nullptr ? regionName : ""); + slot->profile = profile; + slot->slashSecond = slashSecond; + int index = slotCount++; + slots[index] = slot; + order[index] = static_cast(index); + return index; +} + +void SearchEngine::removeAll() { + for (int i = 0; i < slotCount; i++) { + delete slots[i]; + slots[i] = nullptr; + } + slotCount = 0; + ranked.release(); + cursor = 0; +} + +const char* SearchEngine::regionName(int slot) const { + return slot >= 0 && slot < slotCount ? slots[slot]->region : ""; +} + +const SearchIndexReader* SearchEngine::reader(int slot) const { + return slot >= 0 && slot < slotCount ? &slots[slot]->reader : nullptr; +} + +void SearchEngine::reorder(const int* wanted, int count) { + uint8_t next[SEARCH_MAX_INDEXES]; + bool used[SEARCH_MAX_INDEXES] = {}; + int n = 0; + for (int i = 0; i < count && n < slotCount; i++) { + if (wanted[i] >= 0 && wanted[i] < slotCount && !used[wanted[i]]) { + used[wanted[i]] = true; + next[n++] = static_cast(wanted[i]); + } + } + for (int i = 0; i < slotCount; i++) { + int slot = order[i]; + if (!used[slot]) { + used[slot] = true; + next[n++] = static_cast(slot); + } + } + for (int i = 0; i < slotCount; i++) { + order[i] = next[i]; + } +} + +bool SearchEngine::loadResident(int slot, bool nameGroups, bool keyPages) { + return slot >= 0 && slot < slotCount && slots[slot]->reader.loadResident(nameGroups, keyPages); +} + +void SearchEngine::dropResident() { + for (int i = 0; i < slotCount; i++) { + slots[i]->reader.loadResident(false, false); + } +} + +void SearchEngine::setCandidateLimit(uint32_t limit) { + candidateLimit = std::max(8, std::min(limit, 512)); +} + +int SearchEngine::priorityOf(int slotIndex) const { + for (int i = 0; i < slotCount; i++) { + if (order[i] == slotIndex) { + return i; + } + } + return slotCount; +} + +const SearchCategory* SearchEngine::categories(int* count) { + if (count != nullptr) { + *count = static_cast(sizeof(CATEGORIES) / sizeof(CATEGORIES[0])); + } + return CATEGORIES; +} + +// ---------------------------------------------------------------------------------------------------- candidates + +void SearchEngine::startSearch(SearchOrigin origin) { + currentOrigin = origin; + ranked.clear(); + cursor = 0; + double n = PI_VALUE * (1.0 - 2.0 * static_cast(origin.y) / WORLD_UNITS); + double latitude = std::atan(std::sinh(n)); + unitMetres = 40075016.686 * std::cos(latitude) / WORLD_UNITS; + for (int i = 0; i < slotCount; i++) { + slots[i]->reader.resetIo(); + } +} + +void SearchEngine::setGeo(SearchResult& result, uint64_t& distanceSq) const { + distanceSq = distanceSquared(result.x, result.y, currentOrigin); + result.distanceM = static_cast(std::sqrt(static_cast(distanceSq)) * unitMetres + 0.5); + double dx = static_cast(result.x) - currentOrigin.x; + double dy = static_cast(result.y) - currentOrigin.y; + double degrees = std::atan2(dx, -dy) * 180.0 / PI_VALUE; + if (degrees < 0) { + degrees += 360.0; + } + result.bearing = static_cast(static_cast(degrees + 0.5) % 360); +} + +namespace { + +/** A new candidate for a point: filled by the caller further. Declared here because Candidate is private. */ +template +void zero(T& value) { + memset(static_cast(&value), 0, sizeof(T)); +} + +} // namespace + +void SearchEngine::rankCandidates(size_t cap) { + const size_t total = ranked.size(); + GrowArray sorted; + if (!sorted.resize(total)) { + return; + } + for (size_t i = 0; i < total; i++) { + sorted[i] = static_cast(i); + } + const Candidate* items = ranked.begin(); + std::sort(sorted.begin(), sorted.end(), [items](uint32_t a, uint32_t b) { + const Candidate& x = items[a]; + const Candidate& y = items[b]; + if (x.result.group != y.result.group) { + return x.result.group < y.result.group; + } + if (x.rank != y.rank) { + return x.rank < y.rank; + } + if (x.distanceSq != y.distanceSq) { + return x.distanceSq < y.distanceSq; + } + if (x.priority != y.priority) { + return x.priority < y.priority; + } + return a < b; + }); + GrowArray kept; + kept.reserve(std::min(total, cap)); + for (size_t s = 0; s < total && kept.size() < cap; s++) { + const Candidate& c = items[sorted[s]]; + bool duplicate = false; + for (size_t k = 0; k < kept.size() && !duplicate; k++) { + Candidate& other = kept[k]; + if (other.result.group != c.result.group || strcmp(other.result.type, c.result.type) != 0) { + continue; + } + bool names = other.nameLazy || c.nameLazy || strcmp(other.result.name, c.result.name) == 0; + if (!names) { + continue; + } + if (std::abs(other.result.x - c.result.x) <= DUPLICATE_UNITS && std::abs(other.result.y - c.result.y) <= DUPLICATE_UNITS) { + duplicate = true; + } else if (c.result.hasBox && other.result.hasBox && other.result.slot != c.result.slot && + c.result.minX <= other.result.maxX && c.result.maxX >= other.result.minX && + c.result.minY <= other.result.maxY && c.result.maxY >= other.result.minY) { + // the same street cut by a block border: one result with the union of the boxes + other.result.minX = std::min(other.result.minX, c.result.minX); + other.result.minY = std::min(other.result.minY, c.result.minY); + other.result.maxX = std::max(other.result.maxX, c.result.maxX); + other.result.maxY = std::max(other.result.maxY, c.result.maxY); + duplicate = true; + } + } + if (!duplicate) { + kept.push(c); + } + } + ranked.swap(kept); +} + +bool SearchEngine::materialise(Candidate& candidate, SearchResult& out) { + if (candidate.nameLazy) { + SearchNg group; + if (candidate.result.slot < slotCount && slots[candidate.result.slot]->reader.ng(candidate.ng, group)) { + copyText(candidate.result.name, sizeof(candidate.result.name), group.name); + } else { + copyText(candidate.result.name, sizeof(candidate.result.name), candidate.result.type); + } + candidate.nameLazy = false; + } + if (!candidate.isPlace) { + findPlace(candidate.result); + candidate.isPlace = true; + } + out = candidate.result; + return true; +} + +void SearchEngine::loadPlaces(Slot& slot) { + slot.placesLoaded = true; + for (uint32_t pair = 0; pair < slot.reader.info().classCount; pair++) { + char label[96]; + uint32_t first, count; + if (!slot.reader.classPair(pair, label, sizeof(label), first, count) || count == 0 || strncmp(label, "place/", 6) != 0) { + continue; + } + static const struct { + const char* name; + PlaceClass placeClass; + PlacePoint::Level level; + } SETTLEMENTS[] = { + { "city", PlaceClass::City, PlacePoint::Big }, { "town", PlaceClass::Town, PlacePoint::Big }, + { "village", PlaceClass::Village, PlacePoint::Small }, { "hamlet", PlaceClass::Hamlet, PlacePoint::Small }, + { "suburb", PlaceClass::Suburb, PlacePoint::Suburb }, + }; + // "place/city" or, when the pairs were folded, "place/city/*" + const char* classText = label + 6; + const char* slash = strchr(classText, '/'); + const size_t classLength = slash != nullptr ? static_cast(slash - classText) : strlen(classText); + float scale = 0.0f; + PlacePoint::Level level = PlacePoint::Small; + for (const auto& settlement : SETTLEMENTS) { + if (strlen(settlement.name) == classLength && strncmp(classText, settlement.name, classLength) == 0) { + scale = placeClassScale(settlement.placeClass); + level = settlement.level; + } + } + if (scale == 0.0f) { + continue; + } + for (uint32_t done = 0; done < count; done += RECORD_CHUNK) { + GrowArray records; + if (!slot.reader.pointRecords(first + done, std::min(RECORD_CHUNK, count - done), records)) { + break; + } + for (const SearchPointRecord& record : records) { + if (record.ng != SIDX_UNNAMED) { + slot.places.push({ record.x, record.y, record.ng, scale, level }); + } + } + } + } +} + +SearchEngine::BestPlace SearchEngine::bestPlace(int32_t x, int32_t y, PlacePoint::Level from, PlacePoint::Level to, float limit) { + BestPlace best; + float bestScore = limit; + for (int s = 0; s < slotCount; s++) { + Slot& slot = *slots[s]; + if (!slot.placesLoaded) { + loadPlaces(slot); + } + for (const PlacePoint& place : slot.places) { + if (place.level < from || place.level > to) { + continue; + } + const double dx = static_cast(place.x) - x; + const double dy = static_cast(place.y) - y; + const auto score = static_cast(std::sqrt(dx * dx + dy * dy) * unitMetres / place.scale); + if (score < bestScore) { + bestScore = score; + best.slot = s; + best.ng = place.ng; + best.level = place.level; + } + } + } + return best; +} + +void SearchEngine::findPlace(SearchResult& result) { + result.place[0] = 0; + const BestPlace settlement = bestPlace(result.x, result.y, PlacePoint::Big, PlacePoint::Small, PLACE_SCORE_LIMIT); + SearchNg group; + if (settlement.slot < 0 || !slots[settlement.slot]->reader.ng(settlement.ng, group)) { + return; + } + char text[2 * SIDX_MAX_NAME + 4]; + copyText(text, sizeof(text), group.name); + if (settlement.level == PlacePoint::Big) { + // Inside a city or town the suburb tells more: "Ružinov, Bratislava" + const BestPlace suburb = bestPlace(result.x, result.y, PlacePoint::Suburb, PlacePoint::Suburb, SUBURB_SCORE_LIMIT); + SearchNg suburbGroup; + if (suburb.slot >= 0 && slots[suburb.slot]->reader.ng(suburb.ng, suburbGroup) && strcmp(suburbGroup.name, group.name) != 0) { + snprintf(text, sizeof(text), "%s, %s", suburbGroup.name, group.name); + } + } + // "Holice 238" (a village address) and the village's own street already name the place + const size_t length = strlen(group.name); + if (length == 0 || strncmp(result.name, group.name, length) == 0 || strncmp(result.name, text, strlen(text)) == 0) { + return; + } + copyText(result.place, sizeof(result.place), text); +} + +int SearchEngine::takeResults(SearchResult* out, int max) { + int n = 0; + while (n < max && cursor < ranked.size()) { + materialise(ranked[cursor], out[n]); + n++; + cursor++; + } + return n; +} + +int SearchEngine::more(SearchResult* out, int max) { + return takeResults(out, max); +} + +// ---------------------------------------------------------------------------------------------------- address search + +bool SearchEngine::collectAddresses(const Query& query, SearchOrigin) { + for (int r = 0; r < query.readingCount; r++) { + const QueryReading& reading = query.readings[r]; + if (reading.number[0] == 0) { + continue; + } + HouseNumber number; + if (!parseHouseNumber(reading.number, strlen(reading.number), number)) { + continue; + } + char keys[2][KEY_BYTES]; + int keyCount = makeKeys(reading.street, keys); + if (characterCount(keys[0]) < MIN_CHARACTERS) { + continue; + } + const size_t before = ranked.size(); + for (int p = 0; p < slotCount; p++) { + const int slotIndex = order[p]; + Slot& slot = *slots[slotIndex]; + uint16_t wanted = number.number; + if ((number.flags & HOUSE_SLASH) != 0 && number.aux != 0 && slot.slashSecond) { + wanted = number.aux; + } + int suffix = number.suffix != 0 ? number.suffix : -1; + for (int k = 0; k < keyCount; k++) { + GrowArray hits; + if (!slot.reader.searchAddress(keys[k], wanted, suffix, 20, hits)) { + continue; + } + for (const SearchAddressHit& hit : hits) { + Candidate c; + zero(c); + char text[40]; + numberText(hit.address, slot.slashSecond, slot.reader, text, sizeof(text)); + snprintf(c.result.name, sizeof(c.result.name), "%s %s", hit.street, text); + copyText(c.result.type, sizeof(c.result.type), "Address"); + c.result.x = hit.address.x; + c.result.y = hit.address.y; + c.result.minX = c.result.maxX = c.result.x; + c.result.minY = c.result.maxY = c.result.y; + c.result.group = ResultGroup::Address; + c.result.slot = static_cast(slotIndex); + c.result.lowConfidence = (hit.address.flags & ADDRESS_LOW) != 0; + c.result.villageEstimated = (hit.address.flags & ADDRESS_VILLAGE) != 0; + c.rank = (suffix < 0 && hit.address.suffix != 0) ? 1 : 0; + c.priority = static_cast(p); + setGeo(c.result, c.distanceSq); + ranked.push(c); + } + } + } + if (ranked.size() > before) { + return true; + } + } + return false; +} + +// ---------------------------------------------------------------------------------------------------- name search + +void SearchEngine::collectNames(const char* const* keys, int keyCount, SearchOrigin origin) { + GrowArray found; + GrowArray buffer; + buffer.resize(candidateLimit); + for (int p = 0; p < slotCount; p++) { + const int slotIndex = order[p]; + Slot& slot = *slots[slotIndex]; + found.clear(); + for (int k = 0; k < keyCount; k++) { + uint32_t n = slot.reader.prefix(keys[k], candidateLimit, buffer.begin()); + for (uint32_t i = 0; i < n; i++) { + bool seen = false; + for (uint32_t f : found) { + seen = seen || f == buffer[i]; + } + if (!seen) { + found.push(buffer[i]); + } + } + } + for (uint32_t ngOffset : found) { + SearchNg group; + if (!slot.reader.ng(ngOffset, group)) { + continue; + } + const bool exact = isExactName(group.name, keys, keyCount, slot.profile); + if (group.kind == SearchKind::Category) { + if (!exact) { + continue; + } + GrowArray nearest; + if (!nearestPoints(slot.reader, group.first, group.count, origin, CATEGORY_POINTS, nearest)) { + continue; + } + for (const NearPoint& item : nearest) { + Candidate c; + zero(c); + char label[96]; + uint32_t first, count; + slot.reader.classPair(item.record.pair, label, sizeof(label), first, count); + c.isPlace = describeLabel(label, c.result.type, sizeof(c.result.type)) == ResultGroup::Place; + c.result.x = item.record.x; + c.result.y = item.record.y; + c.result.minX = c.result.maxX = c.result.x; + c.result.minY = c.result.maxY = c.result.y; + c.result.group = ResultGroup::Category; + c.result.slot = static_cast(slotIndex); + c.priority = static_cast(p); + if (item.record.ng == SIDX_UNNAMED) { + copyText(c.result.name, sizeof(c.result.name), c.result.type); + } else { + c.nameLazy = true; + c.ng = item.record.ng; + } + setGeo(c.result, c.distanceSq); + ranked.push(c); + } + } else if (group.kind == SearchKind::Point) { + GrowArray nearest; + if (!nearestPoints(slot.reader, group.first, group.count, origin, NAME_POINTS, nearest)) { + continue; + } + for (const NearPoint& item : nearest) { + Candidate c; + zero(c); + char label[96]; + uint32_t first, count; + slot.reader.classPair(item.record.pair, label, sizeof(label), first, count); + ResultGroup kind = describeLabel(label, c.result.type, sizeof(c.result.type)); + c.isPlace = kind == ResultGroup::Place; + copyText(c.result.name, sizeof(c.result.name), group.name); + c.result.x = item.record.x; + c.result.y = item.record.y; + c.result.minX = c.result.maxX = c.result.x; + c.result.minY = c.result.maxY = c.result.y; + c.result.group = exact ? ResultGroup::ExactName : kind; + c.result.slot = static_cast(slotIndex); + c.priority = static_cast(p); + setGeo(c.result, c.distanceSq); + ranked.push(c); + } + } else { + GrowArray records; + if (!slot.reader.streetRecords(group.first, std::min(group.count, 512), records)) { + continue; + } + GrowArray streets; + for (const SearchStreetRecord& record : records) { + Candidate c; + zero(c); + copyText(c.result.name, sizeof(c.result.name), group.name); + copyText(c.result.type, sizeof(c.result.type), "Street"); + c.result.x = record.x; + c.result.y = record.y; + c.result.minX = record.minX; + c.result.minY = record.minY; + c.result.maxX = record.maxX; + c.result.maxY = record.maxY; + c.result.hasBox = true; + c.result.group = exact ? ResultGroup::ExactName : ResultGroup::Street; + c.result.slot = static_cast(slotIndex); + c.priority = static_cast(p); + setGeo(c.result, c.distanceSq); + streets.push(c); + } + std::sort(streets.begin(), streets.end(), [](const Candidate& a, const Candidate& b) { return a.distanceSq < b.distanceSq; }); + for (size_t i = 0; i < streets.size() && i < STREET_CLUSTERS; i++) { + ranked.push(streets[i]); + } + } + } + if (ranked.size() > PRUNE_AT) { + rankCandidates(MAX_RANKED); + } + } +} + +bool SearchEngine::prefixExists(const char* key) { + uint32_t ref; + for (int p = 0; p < slotCount; p++) { + if (slots[order[p]]->reader.prefix(key, 1, &ref) > 0) { + return true; + } + } + return false; +} + +// ---------------------------------------------------------------------------------------------------- search + +void SearchEngine::fillStats(SearchStats* stats, uint64_t startUs, bool short_) { + if (stats == nullptr) { + return; + } + memset(stats, 0, sizeof(*stats)); + for (int i = 0; i < slotCount; i++) { + SearchIoStats io = slots[i]->reader.io(); + stats->readCalls += io.calls; + stats->readBlocks += io.blocks; + stats->readRuns += io.runs; + } + stats->total = static_cast(ranked.size()); + for (const Candidate& c : ranked) { + stats->groupCount[static_cast(c.result.group)]++; + } + stats->indexesSearched = static_cast(slotCount); + stats->tooShort = short_; + if (clock != nullptr) { + stats->elapsedUs = static_cast(clock() - startUs); + } +} + +int SearchEngine::search(const char* typed, SearchOrigin origin, SearchResult* out, int max, SearchStats* stats) { + const uint64_t startUs = clock != nullptr ? clock() : 0; + startSearch(origin); + Query query; + memset(&query, 0, sizeof(query)); + query.keyCount = makeKeys(typed != nullptr ? typed : "", query.keys); + query.readingCount = parseQuery(typed != nullptr ? typed : "", query.readings); + const bool shortText = characterCount(query.keys[0]) < MIN_CHARACTERS; + bool backedOff = false; + char usedKey[KEY_BYTES]; + copyText(usedKey, sizeof(usedKey), query.keys[0]); + + if (!shortText && slotCount > 0) { + if (!collectAddresses(query, origin)) { + const char* keys[2] = { query.keys[0], query.keys[1] }; + collectNames(keys, query.keyCount, origin); + } + if (ranked.size() == 0) { + // the longest prefix that still has results: the existence of a prefix only goes one way, so bisect + const char* primary = query.keys[0]; + int lo = MIN_CHARACTERS, hi = characterCount(primary) - 1, best = 0; + char prefix[KEY_BYTES]; + while (lo <= hi) { + int mid = (lo + hi) / 2; + size_t length = bytesOfCharacters(primary, mid); + memcpy(prefix, primary, length); + prefix[length] = 0; + trimBlanks(prefix); + if (characterCount(prefix) >= MIN_CHARACTERS && prefixExists(prefix)) { + best = mid; + lo = mid + 1; + } else { + hi = mid - 1; + } + } + if (best > 0) { + size_t length = bytesOfCharacters(primary, best); + memcpy(prefix, primary, length); + prefix[length] = 0; + trimBlanks(prefix); + const char* keys[1] = { prefix }; + collectNames(keys, 1, origin); + backedOff = ranked.size() > 0; + copyText(usedKey, sizeof(usedKey), prefix); + } + } + rankCandidates(MAX_RANKED); + } + fillStats(stats, startUs, shortText); + if (stats != nullptr) { + stats->backedOff = backedOff; + copyText(stats->key, sizeof(stats->key), usedKey); + if (backedOff) { + snprintf(stats->header, sizeof(stats->header), "Showing results for '%s'", usedKey); + } + } + return takeResults(out, max); +} + +// ---------------------------------------------------------------------------------------------------- nearby + +int SearchEngine::nearby(int category, SearchOrigin origin, SearchResult* out, int max, SearchStats* stats) { + const uint64_t startUs = clock != nullptr ? clock() : 0; + startSearch(origin); + const int categoryTotal = static_cast(sizeof(CATEGORIES) / sizeof(CATEGORIES[0])); + if (category >= 0 && category < categoryTotal && max > 0) { + const SearchCategory& wanted = CATEGORIES[category]; + const size_t k = static_cast(std::min(max, 64)); + for (int p = 0; p < slotCount; p++) { + const int slotIndex = order[p]; + Slot& slot = *slots[slotIndex]; + for (uint32_t pair = 0; pair < slot.reader.info().classCount; pair++) { + char label[96]; + uint32_t first, count; + if (!slot.reader.classPair(pair, label, sizeof(label), first, count) || count == 0) { + continue; + } + bool match = false; + for (int l = 0; l < wanted.labelCount && !match; l++) { + match = labelMatches(wanted.labels[l], label); + } + if (!match) { + continue; + } + GrowArray nearest; + if (!nearestPoints(slot.reader, first, count, origin, k, nearest)) { + continue; + } + for (const NearPoint& item : nearest) { + Candidate c; + zero(c); + c.isPlace = describeLabel(label, c.result.type, sizeof(c.result.type)) == ResultGroup::Place; + c.result.x = item.record.x; + c.result.y = item.record.y; + c.result.minX = c.result.maxX = c.result.x; + c.result.minY = c.result.maxY = c.result.y; + c.result.group = ResultGroup::Nearby; + c.result.slot = static_cast(slotIndex); + c.priority = static_cast(p); + if (item.record.ng == SIDX_UNNAMED) { + copyText(c.result.name, sizeof(c.result.name), c.result.type); + } else { + c.nameLazy = true; + c.ng = item.record.ng; + } + setGeo(c.result, c.distanceSq); + ranked.push(c); + } + } + if (ranked.size() > PRUNE_AT) { + rankCandidates(MAX_RANKED); + } + } + rankCandidates(std::max(k, 64)); + } + // the reads of the names are part of the measurement: the stats come after taking the results + int n = takeResults(out, max); + if (stats != nullptr) { + fillStats(stats, startUs, false); + } + return n; +} + +// ---------------------------------------------------------------------------------------------------- recent searches + +void searchTypeLine(const char* type, const char* place, char* out, size_t size) { + if (place != nullptr && place[0] != 0) { + char text[SEARCH_LINE_BYTES + SEARCH_PLACE_BYTES]; + snprintf(text, sizeof(text), "%s \xC2\xB7 %s", type, place); + copyText(out, size, text); + } else { + copyText(out, size, type); + } +} + +void RecentSearches::add(const SearchResult& result) { + Entry entry = {}; + copyText(entry.name, sizeof(entry.name), result.name); + searchTypeLine(result.type, result.place, entry.type, sizeof(entry.type)); + entry.x = result.x; + entry.y = result.y; + int at = used; + for (int i = 0; i < used; i++) { + if (strcmp(entries[i].name, entry.name) == 0 && entries[i].x == entry.x && entries[i].y == entry.y) { + at = i; + break; + } + } + if (at == used) { + if (used < CAPACITY) { + used++; + } + at = used - 1; + } + for (int i = at; i > 0; i--) { + entries[i] = entries[i - 1]; + } + entries[0] = entry; +} + +namespace { + +/** One line of the recent searches file: x TAB y TAB type TAB name. */ +constexpr size_t RECENT_LINE_BYTES = SEARCH_NAME_BYTES + SEARCH_LINE_BYTES + 32; + +} // namespace + +bool RecentSearches::save(WriteBytes write, void* context) const { + // Line by line: the map task that calls this has little stack + const char header[] = "recent 1\n"; + if (!write(context, header, sizeof(header) - 1)) { + return false; + } + for (int i = 0; i < used; i++) { + char name[SEARCH_NAME_BYTES]; + char type[SEARCH_LINE_BYTES]; + copyText(name, sizeof(name), entries[i].name); + copyText(type, sizeof(type), entries[i].type); + for (char* s : { name, type }) { + for (; *s != 0; s++) { + if (*s == '\t' || *s == '\n' || *s == '\r') { + *s = ' '; + } + } + } + char line[RECENT_LINE_BYTES]; + int n = snprintf(line, sizeof(line), "%ld\t%ld\t%s\t%s\n", static_cast(entries[i].x), static_cast(entries[i].y), type, name); + if (n < 0 || static_cast(n) >= sizeof(line) || !write(context, line, static_cast(n))) { + return false; + } + } + return true; +} + +bool RecentSearches::load(ReadBytes read, void* context) { + // Read in small pieces and parse line by line: the map task that calls this has little stack + used = 0; + bool header = false; + bool bad = false; + char line[RECENT_LINE_BYTES]; + size_t length = 0; + auto takeLine = [&]() { + line[length] = 0; + length = 0; + if (!header) { + header = strncmp(line, "recent 1", 8) == 0; + bad = !header; + return; + } + if (used >= CAPACITY) { + return; + } + char* fields[4]; + int n = 0; + char* at = line; + fields[n++] = at; + while (n < 4 && (at = strchr(at, '\t')) != nullptr) { + *at++ = 0; + fields[n++] = at; + } + if (n == 4) { + Entry entry = {}; + entry.x = atoi(fields[0]); + entry.y = atoi(fields[1]); + copyText(entry.type, sizeof(entry.type), fields[2]); + copyText(entry.name, sizeof(entry.name), fields[3]); + entries[used++] = entry; + } + }; + char chunk[64]; + size_t got; + while (!bad && (got = read(context, chunk, sizeof(chunk))) > 0) { + for (size_t i = 0; i < got && !bad; i++) { + if (chunk[i] == '\n') { + takeLine(); + } else if (length + 1 < sizeof(line)) { + line[length++] = chunk[i]; + } + } + } + if (!bad && length > 0) { + takeLine(); + } + if (bad || !header) { + used = 0; + return false; + } + return true; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchEngine.h b/Apps/Maps/main/Source/search/SearchEngine.h new file mode 100644 index 0000000..8d5772b --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchEngine.h @@ -0,0 +1,282 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "SearchIndex.h" +#include "TextFold.h" + +#include "../route/GrowArray.h" + +#include +#include + +namespace maps { + +/** + * The query side of offline search (M6 study, sections 5 and 6): a set of open region indexes, the ranking of a typed + * text over all of them, "nearby" lists by category and the recent searches. No Tactility, no file API: the indexes + * are read through SearchFile objects that the caller provides and keeps alive. + */ + +constexpr int SEARCH_MAX_INDEXES = 8; +/** Results whose names the first page needs (search() with max = this); more() fetches the next ones. */ +constexpr int SEARCH_FIRST_PAGE = 8; +/** The size of a "nearby" list. */ +constexpr int SEARCH_NEARBY = 20; +constexpr int SEARCH_NAME_BYTES = 96; +constexpr int SEARCH_TYPE_BYTES = 28; +constexpr int SEARCH_ALSO_BYTES = 32; +constexpr int SEARCH_ALSO_COUNT = 2; +constexpr int SEARCH_HEADER_BYTES = 128; +/** The settlement a result lies in ("Holice"). */ +constexpr int SEARCH_PLACE_BYTES = 40; +/** A type line with its settlement: "Street · Holice". */ +constexpr int SEARCH_LINE_BYTES = SEARCH_TYPE_BYTES + SEARCH_PLACE_BYTES + 4; + +/** A position in global z14 tile units (the unit of the index points). */ +struct SearchOrigin { + int32_t x; + int32_t y; +}; + +/** The ranking groups in the order the results come: the study's recommendation 8. */ +enum class ResultGroup : uint8_t { + /** Street and number found. */ + Address, + /** The query is a category word: the nearest of that category. */ + Category, + /** The name equals the query. */ + ExactName, + /** Cities, towns, villages, suburbs. */ + Place, + /** Parks, water, peaks, airports. */ + Natural, + Street, + /** Points of interest whose name starts with the query. */ + Poi, + /** A nearby() list. */ + Nearby, +}; + +struct SearchResult { + /** The display name (UTF-8); for an address the address text, "Štefánikova 4". An unnamed point shows its type. */ + char name[SEARCH_NAME_BYTES]; + /** The type in plain words: "Cafe", "Street", "Village", "Address" ... */ + char type[SEARCH_TYPE_BYTES]; + /** + * The city, town, village or hamlet the result lies in, estimated from the place points of the open indexes (the + * nearest by distance over the class's size, as the village rule scores them); empty for places themselves, when + * no settlement is near, or when the name already starts with it ("Holice 238"). + */ + char place[SEARCH_PLACE_BYTES]; + /** Up to two alternative village names ("Also: ..."), empty when the index does not carry them (it does not yet). */ + char also[SEARCH_ALSO_COUNT][SEARCH_ALSO_BYTES]; + /** The point in global z14 units. */ + int32_t x; + int32_t y; + /** The bounding box of a street (hasBox), else the point. */ + int32_t minX, minY, maxX, maxY; + bool hasBox; + /** From the search origin: metres on the ground, and degrees clockwise from north (0..359). */ + uint32_t distanceM; + uint16_t bearing; + /** An address 50..100 m from its street (the index does not carry the flag yet: always false). */ + bool lowConfidence; + /** The address uses an estimated village name (the index does not carry it yet: always false). */ + bool villageEstimated; + ResultGroup group; + /** The slot of the index (addIndex() result) that holds it. */ + uint8_t slot; +}; + +/** What a search did, for the UI header, the tests and the measurements. */ +struct SearchStats { + /** Reads made through the SearchFile objects: calls, distinct 4 KB blocks, runs of consecutive blocks. */ + uint32_t readCalls; + uint32_t readBlocks; + uint32_t readRuns; + /** Microseconds by the caller's clock (0 without one). */ + uint32_t elapsedUs; + /** Ranked results found (more() can fetch them), and per group. */ + uint32_t total; + uint32_t groupCount[8]; + uint32_t indexesSearched; + /** Fewer than 3 characters: nothing was searched. */ + bool tooShort; + /** No result for the text: the longest prefix that has some was used. */ + bool backedOff; + /** The folded text that was looked up (after a back-off: the prefix that has results). */ + char key[96]; + /** "Showing results for 'stefan'" after a back-off, else empty. */ + char header[SEARCH_HEADER_BYTES]; +}; + +/** A clock in microseconds, supplied by the caller (monotonic; CPU time when the platform has it). */ +using SearchClock = uint64_t (*)(); + +/** A "nearby" category: a plain-word name and the class-table labels it covers. */ +struct SearchCategory { + const char* name; + /** Labels of the index class table: "poi/cafe/cafe" matches itself; "poi/restaurant/" followed by a star matches all subclasses. */ + const char* const* labels; + uint8_t labelCount; +}; + +class SearchEngine { +public: + using Result = SearchResult; + + SearchEngine(); + ~SearchEngine(); + SearchEngine(const SearchEngine&) = delete; + SearchEngine& operator=(const SearchEngine&) = delete; + + // ---- the open indexes. A region that is cut into several block files adds each file as its own index. + + /** + * Opens an index over the file (kept, not owned: it must outlive the engine or removeAll()). profile is the + * region's transliteration profile; slashSecond tells that "conscription/street" numbers have the street number + * second (the writer's slashSecond flag, not stored in the file). The indexes should be added nearest to the map + * centre first: that order breaks ties between equally distant results. Returns the slot, or -1 (all slots used, or + * the file is not an index). + */ + int addIndex(SearchFile* file, const char* regionName, Translit profile, bool slashSecond); + void removeAll(); + int indexCount() const { return slotCount; } + const char* regionName(int slot) const; + const SearchIndexReader* reader(int slot) const; + /** New priority order: the slots, nearest first; slots that are not listed go last in their old order. */ + void reorder(const int* slots, int count); + + /** Keeps the name group area and/or the key pages of an index in memory (the reader's loadResident). */ + bool loadResident(int slot, bool nameGroups, bool keyPages); + /** Gives the memory of every index's resident pages back (routing needs it); queries read the files again. */ + void dropResident(); + + /** The clock for SearchStats::elapsedUs. */ + void setClock(SearchClock clockFunction) { clock = clockFunction; } + /** How many name groups one index contributes at most per query (default 64; a bigger number ranks better, reads more). */ + void setCandidateLimit(uint32_t limit); + + // ---- search + + /** + * Searches all indexes for the typed text and ranks the results: exact address (street and number) -> category + * word -> exact names -> places -> streets -> points of interest by prefix; inside a group the nearest to origin + * first, and many results of one name nearest first. Fewer than 3 characters search nothing. When nothing is + * found the longest prefix that has results is used and stats->header says so. Returns the first results, at most + * max (use SEARCH_FIRST_PAGE); the rest is kept for more(). + */ + int search(const char* typed, SearchOrigin origin, SearchResult* out, int max, SearchStats* stats = nullptr); + /** The next results of the last search(), at most max; 0 at the end. Names of unnamed-lookup results are read here. */ + int more(SearchResult* out, int max); + /** How many ranked results the last search or nearby() kept. */ + int resultCount() const { return static_cast(ranked.size()); } + + // ---- nearby + + /** The categories of the nearby list: about 12, built from the study's most common POI classes. */ + static const SearchCategory* categories(int* count); + /** The nearest max (at most 64) points of a category from origin; replaces the results that more() continues. */ + int nearby(int category, SearchOrigin origin, SearchResult* out, int max, SearchStats* stats = nullptr); + +private: + /** A settlement or suburb point of an index: position, name group, class size in metres. */ + struct PlacePoint { + int32_t x, y; + uint32_t ng; + float scale; + /** A city or town (gets its suburb in front: "Ružinov, Bratislava"), a village or hamlet, a suburb. */ + enum Level : uint8_t { Big, Small, Suburb } level; + }; + struct BestPlace { + int slot = -1; + uint32_t ng = 0; + PlacePoint::Level level = PlacePoint::Small; + }; + /** The best scored place of the levels from..to within the score limit. */ + BestPlace bestPlace(int32_t x, int32_t y, PlacePoint::Level from, PlacePoint::Level to, float limit); + struct Slot { + GrowArray places; + bool placesLoaded = false; + SearchFile* file = nullptr; + SearchIndexReader reader; + char region[32] = {}; + Translit profile = Translit::Russian; + bool slashSecond = false; + }; + /** A ranked result before its name is final: the result and what ordering and the lazy name need. */ + struct Candidate { + SearchResult result; + uint64_t distanceSq; + /** Name group offset when the name is read on demand (nameLazy), else unused. */ + uint32_t ng; + uint8_t rank; + uint8_t priority; + bool nameLazy; + /** A settlement or other place: gets no place of its own. */ + bool isPlace; + }; + struct Query; + + int priorityOf(int slotIndex) const; + void startSearch(SearchOrigin origin); + void setGeo(SearchResult& result, uint64_t& distanceSq) const; + void fillStats(SearchStats* stats, uint64_t startUs, bool tooShort); + bool collectAddresses(const Query& query, SearchOrigin origin); + void collectNames(const char* const* keys, int keyCount, SearchOrigin origin); + bool prefixExists(const char* key); + void rankCandidates(size_t cap); + bool materialise(Candidate& candidate, SearchResult& out); + void loadPlaces(Slot& slot); + void findPlace(SearchResult& result); + int takeResults(SearchResult* out, int max); + + Slot* slots[SEARCH_MAX_INDEXES] = {}; + uint8_t order[SEARCH_MAX_INDEXES] = {}; + int slotCount = 0; + SearchClock clock = nullptr; + uint32_t candidateLimit = 64; + + SearchOrigin currentOrigin = {}; + /** Metres of one z14 unit at the latitude of the origin. */ + double unitMetres = 0.4; + GrowArray ranked; + size_t cursor = 0; +}; + +/** The type line a list shows: "Street · Holice", or the type alone without a place. */ +void searchTypeLine(const char* type, const char* place, char* out, size_t size); + +/** The last chosen results, newest first: what the search sheet offers before the user types. */ +class RecentSearches { +public: + static constexpr int CAPACITY = 10; + struct Entry { + char name[SEARCH_NAME_BYTES]; + /** The type line with the settlement (searchTypeLine). */ + char type[SEARCH_LINE_BYTES]; + int32_t x; + int32_t y; + }; + /** Writes bytes somewhere (a file): false on failure. */ + using WriteBytes = bool (*)(void* context, const void* data, size_t size); + /** Reads up to capacity bytes: the count, 0 at the end. */ + using ReadBytes = size_t (*)(void* context, void* out, size_t capacity); + + void clear() { used = 0; } + int count() const { return used; } + /** 0 is the newest. */ + const Entry& at(int index) const { return entries[index]; } + /** A chosen result goes to the front; the same name at the same point moves instead of repeating; the oldest drops. */ + void add(const SearchResult& result); + /** Text format, a header line then one line per entry: x TAB y TAB type TAB name. */ + bool save(WriteBytes write, void* context) const; + /** Replaces the list; false (and an empty list) when the data is not this format. */ + bool load(ReadBytes read, void* context); + +private: + Entry entries[CAPACITY] = {}; + int used = 0; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchExtract.cpp b/Apps/Maps/main/Source/search/SearchExtract.cpp new file mode 100644 index 0000000..c42c312 --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchExtract.cpp @@ -0,0 +1,329 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "SearchExtract.h" + +#include +#include +#include + +namespace maps { + +namespace { + +constexpr int TILE_UNITS = 4096; + +bool textIs(const char* text, size_t length, const char* value) { + return text != nullptr && strlen(value) == length && memcmp(text, value, length) == 0; +} + +/** Unnamed POI classes that are not worth an index entry (study.py JUNK). */ +bool isJunkClass(const char* text, size_t length) { + static const char* const JUNK[] = { "gate", "bollard", "lift_gate", "waste_basket", "bicycle_parking", "information", "shelter", "recycling" }; + for (const char* junk : JUNK) { + if (textIs(text, length, junk)) { + return true; + } + } + return false; +} + +/** Classes the study's street matching left out (data.py PATHCLS). */ +bool isPathClass(const char* text, size_t length) { + static const char* const PATH[] = { "path", "track", "aerialway", "ferry", "rail" }; + for (const char* path : PATH) { + if (textIs(text, length, path)) { + return true; + } + } + return false; +} + +uint8_t settlementClass(const char* text, size_t length) { + static const char* const NAMES[] = { "city", "town", "village", "hamlet", "isolated_dwelling", "suburb", "quarter", "neighbourhood" }; + for (size_t i = 0; i < sizeof(NAMES) / sizeof(NAMES[0]); i++) { + if (textIs(text, length, NAMES[i])) { + return static_cast(i); + } + } + return EXTRACT_NOT_SETTLEMENT; +} + +} // namespace + +void ExtractedTile::clear() { + text.clear(); + points.clear(); + lines.clear(); + linePoints.clear(); + houses.clear(); + refOnlyLines = 0; + junkPois = 0; +} + +void ExtractedTile::release() { + text.release(); + points.release(); + lines.release(); + linePoints.release(); + houses.release(); + refOnlyLines = 0; + junkPois = 0; +} + +uint32_t TileExtractor::addText(const char* text, size_t length) { + const auto offset = static_cast(out->text.size()); + for (size_t i = 0; i < length; i++) { + out->text.push(text[i]); + } + out->text.push('\0'); + return offset; +} + +uint32_t TileExtractor::addLabel(const char* kind, const char* classText, size_t classLength, const char* subclassText, size_t subclassLength) { + char buffer[200]; + const bool isPoi = strcmp(kind, "poi") == 0; + char cls[80]; + size_t n = classLength < sizeof(cls) - 1 ? classLength : sizeof(cls) - 1; + if (classText != nullptr && n > 0) { + memcpy(cls, classText, n); + } else { + n = 0; + } + cls[n] = '\0'; + if (n == 0) { + strcpy(cls, "other"); + } + if (isPoi) { + char sub[80]; + size_t m = subclassLength < sizeof(sub) - 1 ? subclassLength : sizeof(sub) - 1; + if (subclassText != nullptr && m > 0) { + memcpy(sub, subclassText, m); + } else { + m = 0; + } + sub[m] = '\0'; + snprintf(buffer, sizeof(buffer), "poi/%s/%s", cls, m > 0 ? sub : cls); + } else { + snprintf(buffer, sizeof(buffer), "%s/%s", kind, cls); + } + return addText(buffer, strlen(buffer)); +} + +int32_t TileExtractor::scaled(int32_t value, int32_t origin) const { + if (extent == TILE_UNITS) { + return origin + value; + } + const double units = static_cast(value) * TILE_UNITS / extent; + return origin + static_cast(std::floor(units + 0.5)); +} + +bool TileExtractor::extract(MvtDecoder& decoder, ByteInput* input, int tileX, int tileY, ExtractedTile& tile, ExtractConsumer* featureConsumer) { + out = &tile; + consumer = featureConsumer; + consumerOk = true; + arrayFailed = false; + tile.clear(); + tile.tileX = tileX; + tile.tileY = tileY; + originX = tileX * TILE_UNITS; + originY = tileY * TILE_UNITS; + shape = Shape::None; + if (!decoder.scan(input, true)) { + decoder.shrinkPools(); + return false; + } + for (int i = 0; i < static_cast(Layer::Count); i++) { + extents[i] = decoder.extent(static_cast(i)); + } + static const Layer LAYERS[] = { Layer::Place, Layer::Poi, Layer::TransportationName, Layer::Housenumber, Layer::WaterName, Layer::MountainPeak, + Layer::Park }; + const bool ok = decoder.sweep(LAYERS, 7, *this); + decoder.shrinkPools(); + consumer = nullptr; + return ok && consumerOk && !arrayFailed && !tile.failed(); +} + +bool TileExtractor::beginFeature(const FeatureInfo& info) { + if (consumer != nullptr) { + // Texts of a skipped feature before this one are not needed + out->text.clear(); + } + shape = Shape::None; + rings = 0; + havePoint = false; + label = name = latin = en = EXTRACT_NONE; + settlement = EXTRACT_NOT_SETTLEMENT; + useEn = 0; + pathClass = 0; + extent = extents[static_cast(info.layer)]; + if (extent <= 0) { + extent = TILE_UNITS; + } + if (info.layer == Layer::Housenumber) { + if (info.houseNumber == nullptr || info.houseNumberLength == 0) { + return false; + } + name = addText(info.houseNumber, info.houseNumberLength); + shape = Shape::House; + return true; + } + // The display name is name, else name:latin, else name_en (data.py nm_of, without name_int) + const char* shown = nullptr; + size_t shownLength = 0; + if (info.name != nullptr && info.nameLength > 0) { + shown = info.name; + shownLength = info.nameLength; + } else if (info.nameLatin != nullptr && info.nameLatinLength > 0) { + shown = info.nameLatin; + shownLength = info.nameLatinLength; + } else if (info.nameEn != nullptr && info.nameEnLength > 0) { + shown = info.nameEn; + shownLength = info.nameEnLength; + } + if (info.layer == Layer::TransportationName) { + if (shown == nullptr) { + out->refOnlyLines++; + return false; + } + name = addText(shown, shownLength); + if (info.nameLatin != nullptr && info.nameLatinLength > 0) { + latin = addText(info.nameLatin, info.nameLatinLength); + } + pathClass = isPathClass(info.classText, info.classTextLength) ? 1 : 0; + shape = Shape::Line; + return true; + } + if (shown == nullptr) { + if (info.layer != Layer::Poi) { + return false; + } + if (isJunkClass(info.classText, info.classTextLength)) { + out->junkPois++; + return false; + } + } + switch (info.layer) { + case Layer::Place: + label = addLabel("place", info.classText, info.classTextLength, nullptr, 0); + settlement = settlementClass(info.classText, info.classTextLength); + useEn = 1; + break; + case Layer::Poi: + label = addLabel("poi", info.classText, info.classTextLength, info.subclassText, info.subclassTextLength); + break; + case Layer::WaterName: + label = addLabel("water", info.classText, info.classTextLength, nullptr, 0); + useEn = 1; + break; + case Layer::MountainPeak: + label = addLabel("peak", info.classText, info.classTextLength, nullptr, 0); + useEn = 1; + break; + case Layer::Park: + label = addLabel("park", info.classText, info.classTextLength, nullptr, 0); + useEn = 1; + break; + default: + return false; + } + if (shown != nullptr) { + name = addText(shown, shownLength); + if (info.nameLatin != nullptr && info.nameLatinLength > 0) { + latin = addText(info.nameLatin, info.nameLatinLength); + } + if (info.nameEn != nullptr && info.nameEnLength > 0) { + en = addText(info.nameEn, info.nameEnLength); + } + } + shape = Shape::Point; + return true; +} + +void TileExtractor::beginRing() { + rings++; + chunks = 0; + if (shape == Shape::Line) { + lineStart = static_cast(out->linePoints.size() / 2); + } else if (rings == 1) { + ring.clear(); + } +} + +void TileExtractor::points(const TilePoint* chunk, int count) { + // The decoder starts each chunk after the first with the last point of the one before: skip that repeat + const int skip = chunks > 0 ? 1 : 0; + chunks++; + if (shape == Shape::Line) { + for (int i = skip; i < count; i++) { + out->linePoints.push(scaled(chunk[i].x, originX)); + out->linePoints.push(scaled(chunk[i].y, originY)); + } + } else if ((shape == Shape::Point || shape == Shape::House) && rings == 1) { + for (int i = skip; i < count; i++) { + ring.push(chunk[i]); + } + } +} + +void TileExtractor::endRing(bool closed) { + if (shape == Shape::Line) { + const auto count = static_cast(out->linePoints.size() / 2 - lineStart); + if (count >= 2) { + out->lines.push({ name, latin, lineStart, count, pathClass }); + } else { + // Not a line: take its points back + while (out->linePoints.size() / 2 > lineStart) { + out->linePoints.pop(); + out->linePoints.pop(); + } + } + return; + } + if (ring.failed()) { + arrayFailed = true; + return; + } + if (rings != 1 || ring.size() == 0) { + return; + } + // data.py first_pt: the middle vertex of a ring of more than 2 points (a closed ring repeats its first point there) + const size_t length = ring.size() + (closed ? 1 : 0); + size_t index = length > 2 ? length / 2 : 0; + if (index >= ring.size()) { + index = ring.size() - 1; + } + havePoint = true; + pointX = ring[index].x; + pointY = ring[index].y; +} + +void TileExtractor::endFeature() { + if (havePoint) { + const int32_t gx = scaled(pointX, originX); + const int32_t gy = scaled(pointY, originY); + if (shape == Shape::Point) { + out->points.push({ label, name, latin, en, gx, gy, settlement, useEn }); + } else if (shape == Shape::House) { + const int32_t lx = gx - originX; + const int32_t ly = gy - originY; + out->houses.push({ name, gx, gy, static_cast(lx >= 0 && lx < TILE_UNITS && ly >= 0 && ly < TILE_UNITS ? 1 : 0) }); + } + } + shape = Shape::None; + if (out->failed()) { + // Never hand partial arrays on; clear() below resets the flags, so remember it + arrayFailed = true; + consumerOk = false; + } + if (consumer != nullptr) { + if (consumerOk && (out->points.size() > 0 || out->lines.size() > 0 || out->houses.size() > 0)) { + consumerOk = consumer->take(*out); + } + out->text.clear(); + out->points.clear(); + out->lines.clear(); + out->linePoints.clear(); + out->houses.clear(); + } +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchExtract.h b/Apps/Maps/main/Source/search/SearchExtract.h new file mode 100644 index 0000000..e29edc6 --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchExtract.h @@ -0,0 +1,146 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "../mvt/MvtDecoder.h" +#include "../route/GrowArray.h" +#include "AddressMatch.h" + +#include +#include + +namespace maps { + +/** "No string" in the offsets below. */ +constexpr uint32_t EXTRACT_NONE = 0xFFFFFFFFu; +/** ExtractPoint::settlement of a point that is not a settlement. */ +constexpr uint8_t EXTRACT_NOT_SETTLEMENT = 255; + +/** + * A point of the search index as the M6 study makes it (host/search/data.py load_region): a place, a POI (named, or + * unnamed and not in the junk list), a water label, a peak or a park. Positions are global z14 units (tile x * 4096 + + * local x after scaling by the layer's extent). Strings are offsets into ExtractedTile::text. + */ +struct ExtractPoint { + /** "place/town", "poi/cafe/cafe", "water/lake", "peak/peak", "park/park" ... */ + uint32_t label; + /** The display name (name, else name:latin, else name_en), EXTRACT_NONE for an unnamed POI. */ + uint32_t name; + /** "name:latin" or EXTRACT_NONE. */ + uint32_t latin; + /** "name_en" or EXTRACT_NONE. */ + uint32_t en; + int32_t x; + int32_t y; + /** A PlaceClass for the village rule (city town village hamlet isolated_dwelling suburb quarter neighbourhood), else EXTRACT_NOT_SETTLEMENT. */ + uint8_t settlement; + /** Whether name_en makes search keys: places, water, peaks and parks, not POIs (the study's rule). */ + uint8_t useEn; +}; + +/** A named line of the transportation_name layer (every class, paths and pedestrian streets too). */ +struct ExtractLine { + uint32_t name; + uint32_t latin; + /** Index of the first point in ExtractedTile::linePoints (x, y pairs), and the point count (at least 2). */ + uint32_t first; + uint32_t count; + /** Class path, track, aerialway, ferry or rail (the study's PATHCLS, which its street matching left out). */ + uint8_t pathClass; +}; + +struct ExtractHouse { + uint32_t text; + int32_t x; + int32_t y; + /** The position is inside the tile (not in its buffer). */ + uint8_t inside; +}; + +/** What one tile gives. The arrays keep their memory between tiles (clear() only empties them). */ +struct ExtractedTile { + int tileX = 0; + int tileY = 0; + /** Zero-terminated strings. */ + GrowArray text; + GrowArray points; + GrowArray lines; + GrowArray linePoints; + GrowArray houses; + /** Lines of the transportation_name layer that have no name at all (only a ref). */ + uint32_t refOnlyLines = 0; + /** Unnamed POIs of the junk list that were not kept. */ + uint32_t junkPois = 0; + + void clear(); + void release(); + const char* str(uint32_t offset) const { return offset == EXTRACT_NONE ? nullptr : &text[offset]; } + bool failed() const { return text.failed() || points.failed() || lines.failed() || linePoints.failed() || houses.failed(); } +}; + +/** + * Takes what the extractor finds feature by feature, for the streaming index build: the tile's arrays then hold only + * one feature's point, house number or lines (a line feature may have several parts) and are emptied after take(). + */ +class ExtractConsumer { +public: + virtual ~ExtractConsumer() = default; + /** False stops the extraction (extract() then returns false). */ + virtual bool take(const ExtractedTile& feature) = 0; +}; + +/** + * Decodes one z14 tile and collects what the search index needs from the layers place, poi, transportation_name, + * housenumber, water_name, mountain_peak and park. Platform free; the decoder and the arrays are the caller's, so a + * whole region reuses them. + */ +class TileExtractor final : public GeometrySink { +public: + /** + * Decodes the tile (the ByteInput of its .pbf) into out. Returns false when the tile cannot be read. With a + * consumer, each feature is handed over as soon as it is decoded and out stays small (the counters keep counting). + */ + bool extract(MvtDecoder& decoder, ByteInput* input, int tileX, int tileY, ExtractedTile& out, ExtractConsumer* consumer = nullptr); + + bool beginFeature(const FeatureInfo& info) override; + void beginRing() override; + void points(const TilePoint* chunk, int count) override; + void endRing(bool closed) override; + void endFeature() override; + +private: + enum class Shape : uint8_t { None, Point, Line, House }; + + uint32_t addText(const char* text, size_t length); + uint32_t addLabel(const char* kind, const char* classText, size_t classLength, const char* subclassText, size_t subclassLength); + int32_t scaled(int32_t value, int32_t origin) const; + + ExtractedTile* out = nullptr; + ExtractConsumer* consumer = nullptr; + bool consumerOk = true; + int originX = 0; + int originY = 0; + int32_t extents[static_cast(Layer::Count)] = {}; + // The feature being decoded + Shape shape = Shape::None; + int32_t extent = 4096; + uint32_t label = EXTRACT_NONE; + uint32_t name = EXTRACT_NONE; + uint32_t latin = EXTRACT_NONE; + uint32_t en = EXTRACT_NONE; + uint8_t settlement = EXTRACT_NOT_SETTLEMENT; + uint8_t useEn = 0; + uint8_t pathClass = 0; + int rings = 0; + int chunks = 0; + uint32_t lineStart = 0; + bool havePoint = false; + int32_t pointX = 0; + int32_t pointY = 0; + bool inside = false; + /** The first ring of a point-like feature (a park's outline is only needed for its middle vertex). */ + GrowArray ring; + /** An array could not grow: the tile's extraction fails. */ + bool arrayFailed = false; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchIndex.cpp b/Apps/Maps/main/Source/search/SearchIndex.cpp new file mode 100644 index 0000000..e3099e3 --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchIndex.cpp @@ -0,0 +1,1477 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "SearchIndex.h" + +#include "QueryParse.h" + +#include +#include +#include + +namespace maps { + +namespace { + +void put16(uint8_t* out, uint32_t value) { + out[0] = static_cast(value); + out[1] = static_cast(value >> 8); +} + +void put24(uint8_t* out, uint32_t value) { + out[0] = static_cast(value); + out[1] = static_cast(value >> 8); + out[2] = static_cast(value >> 16); +} + +void put32(uint8_t* out, uint32_t value) { + out[0] = static_cast(value); + out[1] = static_cast(value >> 8); + out[2] = static_cast(value >> 16); + out[3] = static_cast(value >> 24); +} + +uint32_t get16(const uint8_t* in) { + return static_cast(in[0] | (in[1] << 8)); +} + +uint32_t get24(const uint8_t* in) { + return static_cast(in[0] | (in[1] << 8) | (in[2] << 16)); +} + +uint32_t get32(const uint8_t* in) { + return static_cast(in[0]) | (static_cast(in[1]) << 8) | (static_cast(in[2]) << 16) | + (static_cast(in[3]) << 24); +} + +uint32_t cutUtf8(const char* text, size_t length, uint32_t maximum) { + if (length <= maximum) { + return static_cast(length); + } + uint32_t cut = maximum; + // A continuation byte (10xxxxxx) at the cut means the character began earlier: leave it out whole + while (cut > 0 && (static_cast(text[cut]) & 0xC0) == 0x80) { + cut--; + } + return cut; +} + +/** Compares two byte strings the way Python compares bytes: by content, then the shorter first. */ +int compareBytes(const uint8_t* a, uint32_t aLength, const uint8_t* b, uint32_t bLength) { + uint32_t common = aLength < bLength ? aLength : bLength; + int result = common > 0 ? memcmp(a, b, common) : 0; + if (result != 0) { + return result; + } + return aLength < bLength ? -1 : (aLength > bLength ? 1 : 0); +} + +void appendBytes(GrowArray& array, const void* data, size_t size) { + size_t at = array.size(); + if (size > 0 && array.resize(at + size)) { + memcpy(array.begin() + at, data, size); + } +} + +} // namespace + +uint32_t truncateKey(const char* key, size_t length) { + return cutUtf8(key, length, SIDX_MAX_KEY); +} + +uint32_t truncateName(const char* name, size_t length) { + return cutUtf8(name, length, SIDX_MAX_NAME); +} + +// ------------------------------------------------------------------------------------------------ encoders + +void encodeHeader(const SearchHeader& header, uint8_t out[SIDX_HEADER_SIZE]) { + memset(out, 0, SIDX_HEADER_SIZE); + memcpy(out, "SIDX", 4); + put16(out + 4, SIDX_VERSION); + put16(out + 6, header.pointBytes); + put32(out + 8, static_cast(header.originX)); + put32(out + 12, static_cast(header.originY)); + put32(out + 16, header.classCount); + put32(out + 20, header.fenceCount); + put32(out + 24, header.pageCount); + put32(out + 28, header.pointCount); + put32(out + 32, header.streetCount); + put32(out + 36, header.addressCount); + put32(out + 40, 0); + for (uint32_t i = 0; i < SIDX_SECTION_COUNT; i++) { + put32(out + 44 + 8 * i, header.sectionOffset[i]); + put32(out + 48 + 8 * i, header.sectionSize[i]); + } +} + +uint32_t encodePt(uint8_t* out, uint32_t pointBytes, int32_t relX, int32_t relY) { + if (pointBytes == 8) { + put32(out, static_cast(relX)); + put32(out + 4, static_cast(relY)); + } else { + put16(out, static_cast(relX)); + put16(out + 2, static_cast(relY)); + } + return pointBytes; +} + +bool encodeKeyPage(uint8_t* page, uint32_t& used, const uint8_t* key, uint32_t keyLength, uint32_t ngOffset) { + uint32_t size = 1 + keyLength + 3; + if (keyLength == 0 || used + size + 1 > SIDX_PAGE_SIZE) { + return false; + } + page[used] = static_cast(keyLength); + memcpy(page + used + 1, key, keyLength); + put24(page + used + 1 + keyLength, ngOffset); + used += size; + return true; +} + +uint32_t encodeNameGroup(uint8_t* out, const char* name, uint32_t nameLength, SearchKind kind, uint32_t first, uint16_t count) { + out[0] = static_cast(nameLength); + memcpy(out + 1, name, nameLength); + out[1 + nameLength] = static_cast(kind); + put32(out + 2 + nameLength, first); + put16(out + 6 + nameLength, count); + return 1 + nameLength + 1 + 4 + 2; +} + +uint32_t encodePoint(uint8_t* out, uint32_t pointBytes, int32_t relX, int32_t relY, uint8_t pair, uint32_t ngOffset) { + encodePt(out, pointBytes, relX, relY); + out[pointBytes] = pair; + put24(out + pointBytes + 1, ngOffset); + return pointBytes + 4; +} + +uint32_t encodeStreet(uint8_t* out, uint32_t pointBytes, const int32_t rep[2], const int32_t boxMin[2], const int32_t boxMax[2], + uint32_t firstAddress, uint16_t addressCount) { + encodePt(out, pointBytes, rep[0], rep[1]); + encodePt(out + pointBytes, pointBytes, boxMin[0], boxMin[1]); + encodePt(out + 2 * pointBytes, pointBytes, boxMax[0], boxMax[1]); + put32(out + 3 * pointBytes, firstAddress); + put16(out + 3 * pointBytes + 4, addressCount); + return 3 * pointBytes + 6; +} + +uint32_t encodeAddress(uint8_t* out, uint32_t pointBytes, uint16_t number, uint16_t aux, uint8_t suffix, uint8_t flags, int32_t relX, + int32_t relY) { + put16(out, number); + put16(out + 2, aux); + out[4] = suffix; + out[5] = flags; + encodePt(out + 6, pointBytes, relX, relY); + return 6 + pointBytes; +} + +uint32_t encodeTextAddress(uint8_t* out, uint32_t pointBytes, uint32_t textOffset, int32_t relX, int32_t relY) { + return encodeAddress(out, pointBytes, 0xFFFF, static_cast(textOffset & 0xFFFF), static_cast(textOffset >> 16), + HOUSE_TEXT, relX, relY); +} + +// ------------------------------------------------------------------------------------------------ writer + +namespace { + +enum Category : uint8_t { CAT_CATEGORY = 0, CAT_PLACE = 1, CAT_STREET = 2, CAT_OTHER = 3, CAT_POI = 4 }; + +struct NameGroup { + const char* name; + uint32_t nameLength; + const char* sortKey; + uint32_t sortLength; + uint8_t kind; + uint8_t category; + uint32_t pair; + /** Order of creation, the tie-breaker of the sort (sidx.py sorts stably). */ + uint32_t creation; + /** Point group: its first point (index into the input). Street: the street. Category: unused. */ + uint32_t source; + uint32_t keysSource; + uint32_t first; + uint32_t count; + uint32_t offset; + /** Category: the run of pairs. */ + uint32_t pairFirst; + uint32_t pairCount; +}; + +struct CategoryRun { + uint32_t rawOffset; + uint32_t rawLength; + uint32_t firstPair; + uint32_t lastPair; + uint32_t count; + bool contiguous; +}; + +struct PointRecord { + uint32_t pair; + uint32_t unnamed; + uint32_t ng; + int32_t tileY; + int32_t tileX; + int32_t x; + int32_t y; +}; + +struct KeyEntry { + uint32_t offset; + uint32_t length; + uint32_t ng; + uint8_t priority; +}; + +struct AddressTemp { + uint16_t number; + uint16_t aux; + uint8_t suffix; + uint8_t flags; + int32_t x, y; + uint32_t index; +}; + +struct StreetTemp { + const SearchClusterInput* cluster; + uint32_t firstAddress; + uint32_t count; +}; + +struct Failure { + char* text; + size_t size; + bool fail(const char* what, uint32_t a = 0, uint32_t b = 0) { + snprintf(text, size, what, a, b); + return false; + } +}; + +/** A very small hash set of overflow strings: offsets into the overflow bytes. */ +class OverflowTable { +public: + /** Returns the offset of the text, adding it when new. UINT32_MAX when out of memory. */ + uint32_t find(const char* text, size_t length) { + if ((used + 1) * 2 > slots.size()) { + if (!grow()) { + return UINT32_MAX; + } + } + uint32_t hash = hashOf(text, length); + size_t mask = slots.size() - 1; + for (size_t i = hash & mask;; i = (i + 1) & mask) { + uint32_t slot = slots[i]; + if (slot == 0) { + uint32_t offset = static_cast(bytes.size()); + size_t at = bytes.size(); + if (!bytes.resize(at + length + 1)) { + return UINT32_MAX; + } + memcpy(bytes.begin() + at, text, length); + slots[i] = offset + 1; + used++; + count++; + return offset; + } + const char* stored = bytes.begin() + (slot - 1); + if (strlen(stored) == length && memcmp(stored, text, length) == 0) { + return slot - 1; + } + } + } + GrowArray bytes; + uint32_t count = 0; + +private: + static uint32_t hashOf(const char* text, size_t length) { + uint32_t hash = 2166136261u; + for (size_t i = 0; i < length; i++) { + hash = (hash ^ static_cast(text[i])) * 16777619u; + } + return hash; + } + bool grow() { + GrowArray next; + size_t size = slots.size() < 64 ? 64 : slots.size() * 2; + if (!next.resize(size)) { + return false; + } + size_t mask = size - 1; + for (uint32_t slot : slots) { + if (slot == 0) { + continue; + } + const char* stored = bytes.begin() + (slot - 1); + size_t i = hashOf(stored, strlen(stored)) & mask; + while (next[i] != 0) { + i = (i + 1) & mask; + } + next[i] = slot; + } + slots.swap(next); + return true; + } + GrowArray slots; + size_t used = 0; +}; + +/** Folds a category name (an OSM class made of ASCII): lowercase, anything but letters and digits is a space, runs of spaces are one. */ +uint32_t foldCategory(const char* name, char* out, size_t capacity) { + uint32_t length = 0; + bool space = false; + for (const char* p = name; *p != 0; p++) { + unsigned char c = static_cast(*p); + char value; + if (c >= 'A' && c <= 'Z') { + value = static_cast(c + 32); + } else if ((c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || c >= 0x80) { + value = static_cast(c); + } else { + space = length > 0; + continue; + } + if (space && length + 2 < capacity) { + out[length++] = ' '; + } + space = false; + if (length + 1 < capacity) { + out[length++] = value; + } + } + out[length] = 0; + return length; +} + +/** The number text as the index stores it. Text that the grammar does not cover, or would store wrongly, becomes overflow. */ +bool parseNumber(const char* text, bool slashSecond, uint16_t& number, uint16_t& aux, uint8_t& suffix, uint8_t& flags) { + size_t length = strlen(text); + bool letter = false; + uint32_t run = 0; + for (size_t i = 0; i <= length; i++) { + unsigned char c = static_cast(text[i]); + if (c >= '0' && c <= '9') { + run = run * 10 + (c - '0'); + if (run >= 0xFFFF) { + return false; + } + continue; + } + run = 0; + if (c >= 0x80 || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')) { + letter = true; + } + } + HouseNumber parsed; + if (!parseHouseNumber(text, length, parsed)) { + return false; + } + // A letter that parseHouseNumber could not turn into an ASCII letter would be lost: keep the text instead + if (letter && parsed.suffix == 0) { + return false; + } + number = parsed.number; + aux = parsed.aux; + suffix = static_cast(parsed.suffix); + flags = parsed.flags; + if (slashSecond && flags == HOUSE_SLASH && aux != 0) { + uint16_t first = number; + number = aux; + aux = first; + } + return true; +} + +} // namespace + +int compareSearchBytes(const uint8_t* a, uint32_t aLength, const uint8_t* b, uint32_t bLength) { + return compareBytes(a, aLength, b, bLength); +} + +uint32_t foldCategoryName(const char* name, char* out, size_t capacity) { + return foldCategory(name, out, capacity); +} + +bool indexHouseNumber(const char* text, bool slashSecond, uint16_t& number, uint16_t& aux, uint8_t& suffix, uint8_t& flags) { + return parseNumber(text, slashSecond, number, aux, suffix, flags); +} + +bool SearchIndexWriter::build(const SearchInput& in, ByteSink& sink, SearchBuildStats* stats) { + errorText[0] = 0; + Failure failure = { errorText, sizeof(errorText) }; + const SearchPointInput* points = in.points; + const uint32_t pointCount = in.pointCount; + const SearchStreetInput* streets = in.streets; + const uint32_t streetCount = in.streetCount; + + // ---- origin and point size + bool anyPoint = false; + int32_t minX = 0, minY = 0, maxX = 0, maxY = 0; + auto include = [&](int32_t x, int32_t y) { + if (!anyPoint) { + minX = maxX = x; + minY = maxY = y; + anyPoint = true; + return; + } + minX = std::min(minX, x); + maxX = std::max(maxX, x); + minY = std::min(minY, y); + maxY = std::max(maxY, y); + }; + for (uint32_t i = 0; i < pointCount; i++) { + include(points[i].x, points[i].y); + } + for (uint32_t s = 0; s < streetCount; s++) { + for (uint32_t c = 0; c < streets[s].clusterCount; c++) { + const SearchClusterInput& cl = streets[s].clusters[c]; + include(cl.repX, cl.repY); + include(cl.minX, cl.minY); + include(cl.maxX, cl.maxY); + for (uint32_t a = 0; a < cl.addressCount; a++) { + include(cl.addresses[a].x, cl.addresses[a].y); + } + } + } + const int32_t originX = minX, originY = minY; + const int64_t span = std::max(static_cast(maxX) - minX, static_cast(maxY) - minY); + const bool wide = span >= 65536; + const uint32_t pts = wide ? 8 : 4; + + // ---- class pairs: unique labels (sorted by text), folded into "/(all)" when there are more than 255 + GrowArray order; + order.resize(pointCount); + for (uint32_t i = 0; i < pointCount; i++) { + if (points[i].label == nullptr) { + return failure.fail("point %u has no label", i); + } + order[i] = i; + } + std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) { + int c = strcmp(points[a].label, points[b].label); + return c != 0 ? c < 0 : a < b; + }); + GrowArray uniqueOf, uniqueFirst, uniqueCount; + uniqueOf.resize(pointCount); + for (uint32_t k = 0; k < pointCount; k++) { + uint32_t i = order[k]; + if (k == 0 || strcmp(points[order[k - 1]].label, points[i].label) != 0) { + uniqueFirst.push(i); + uniqueCount.push(0); + } + uniqueOf[i] = static_cast(uniqueFirst.size() - 1); + uniqueCount[uniqueCount.size() - 1]++; + if (i < uniqueFirst[uniqueFirst.size() - 1]) { + uniqueFirst[uniqueFirst.size() - 1] = i; + } + } + const uint32_t uniqueTotal = static_cast(uniqueFirst.size()); + GrowArray keep; + keep.resize(uniqueTotal); + uint32_t collapsed = 0; + if (uniqueTotal > 255) { + GrowArray byCount; + byCount.resize(uniqueTotal); + for (uint32_t u = 0; u < uniqueTotal; u++) { + byCount[u] = u; + } + std::sort(byCount.begin(), byCount.end(), [&](uint32_t a, uint32_t b) { + return uniqueCount[a] != uniqueCount[b] ? uniqueCount[a] > uniqueCount[b] : uniqueFirst[a] < uniqueFirst[b]; + }); + for (uint32_t k = 0; k < 110; k++) { + keep[byCount[k]] = 1; + } + collapsed = uniqueTotal - 110; + } else { + for (uint32_t u = 0; u < uniqueTotal; u++) { + keep[u] = 1; + } + } + GrowArray labelBytes; + GrowArray labelOffset; + labelOffset.resize(uniqueTotal); + for (uint32_t u = 0; u < uniqueTotal; u++) { + const char* label = points[uniqueFirst[u]].label; + size_t length = strlen(label); + size_t keepLength = length; + const char* suffix = ""; + if (!keep[u]) { + const char* slash = strchr(label, '/'); + const char* second = slash != nullptr ? strchr(slash + 1, '/') : nullptr; + keepLength = second != nullptr ? static_cast(second - label) : length; + suffix = "/*"; + } + size_t at = labelBytes.size(); + labelOffset[u] = static_cast(at); + labelBytes.resize(at + keepLength + strlen(suffix) + 1); + memcpy(labelBytes.begin() + at, label, keepLength); + memcpy(labelBytes.begin() + at + keepLength, suffix, strlen(suffix)); + } + if (labelBytes.failed()) { + return failure.fail("out of memory"); + } + GrowArray labelOrder; + labelOrder.resize(uniqueTotal); + for (uint32_t u = 0; u < uniqueTotal; u++) { + labelOrder[u] = u; + } + const char* labelBase = labelBytes.begin(); + std::sort(labelOrder.begin(), labelOrder.end(), [&](uint32_t a, uint32_t b) { + int c = strcmp(labelBase + labelOffset[a], labelBase + labelOffset[b]); + return c != 0 ? c < 0 : a < b; + }); + GrowArray pairOfUnique, pairLabel; // pairLabel: a unique label with that pair's text + pairOfUnique.resize(uniqueTotal); + for (uint32_t k = 0; k < uniqueTotal; k++) { + uint32_t u = labelOrder[k]; + if (k == 0 || strcmp(labelBase + labelOffset[labelOrder[k - 1]], labelBase + labelOffset[u]) != 0) { + pairLabel.push(u); + } + pairOfUnique[u] = static_cast(pairLabel.size() - 1); + } + const uint32_t pairTotal = static_cast(pairLabel.size()); + if (pairTotal > 255) { + return failure.fail("%u class pairs do not fit a byte", pairTotal); + } + auto pairText = [&](uint32_t pair) { return labelBase + labelOffset[pairLabel[pair]]; }; + for (uint32_t pair = 0; pair < pairTotal; pair++) { + if (strlen(pairText(pair)) > 255) { + return failure.fail("class label %u is longer than 255 bytes", pair); + } + } + GrowArray pairOf; + pairOf.resize(pointCount); + for (uint32_t i = 0; i < pointCount; i++) { + pairOf[i] = static_cast(pairOfUnique[uniqueOf[i]]); + } + + // ---- category runs: class and subclass names of "poi//" pairs + GrowArray catChars; + GrowArray classRuns, subRuns; + uint32_t notContiguous = 0; + auto addRun = [&](GrowArray& runs, const char* raw, size_t rawLength, uint32_t pair) { + for (CategoryRun& run : runs) { + if (run.rawLength == rawLength && memcmp(catChars.begin() + run.rawOffset, raw, rawLength) == 0) { + if (pair != run.lastPair + 1) { + run.contiguous = false; + } + run.lastPair = pair; + run.count++; + return; + } + } + CategoryRun run = {}; + run.rawOffset = static_cast(catChars.size()); + run.rawLength = static_cast(rawLength); + run.firstPair = run.lastPair = pair; + run.count = 1; + run.contiguous = true; + size_t at = catChars.size(); + catChars.resize(at + rawLength + 1); + memcpy(catChars.begin() + at, raw, rawLength); + runs.push(run); + }; + for (uint32_t pair = 0; pair < pairTotal; pair++) { + const char* label = pairText(pair); + if (strncmp(label, "poi/", 4) != 0) { + continue; + } + const char* classStart = label + 4; + const char* classEnd = strchr(classStart, '/'); + size_t classLength = classEnd != nullptr ? static_cast(classEnd - classStart) : strlen(classStart); + addRun(classRuns, classStart, classLength, pair); + if (classEnd != nullptr) { + const char* subStart = classEnd + 1; + const char* subEnd = strchr(subStart, '/'); + size_t subLength = subEnd != nullptr ? static_cast(subEnd - subStart) : strlen(subStart); + if (!(subLength == classLength && memcmp(subStart, classStart, subLength) == 0)) { + addRun(subRuns, subStart, subLength, pair); + } + } + } + // display names ("pet_shop" -> "pet shop") and their sort keys, kept after the raw names + struct CategoryText { + uint32_t nameOffset, nameLength, sortOffset, sortLength; + CategoryRun run; + }; + GrowArray categories; + for (int which = 0; which < 2; which++) { + for (const CategoryRun& run : which == 0 ? classRuns : subRuns) { + if (!run.contiguous) { + notContiguous++; + continue; + } + CategoryText text = {}; + text.run = run; + char name[128]; + uint32_t length = std::min(run.rawLength, sizeof(name) - 1); + memcpy(name, catChars.begin() + run.rawOffset, length); + for (uint32_t i = 0; i < length; i++) { + if (name[i] == '_') { + name[i] = ' '; + } + } + name[length] = 0; + char folded[128]; + uint32_t foldedLength = foldCategory(name, folded, sizeof(folded)); + for (uint32_t o = 0; o < in.categoryOverrides; o++) { + if (strcmp(in.categoryNames[o], name) == 0) { + foldedLength = std::min(static_cast(strlen(in.categoryKeys[o])), sizeof(folded) - 1); + memcpy(folded, in.categoryKeys[o], foldedLength); + folded[foldedLength] = 0; + break; + } + } + text.nameOffset = static_cast(catChars.size()); + text.nameLength = length; + catChars.resize(catChars.size() + length + 1); + memcpy(catChars.begin() + text.nameOffset, name, length); + text.sortOffset = static_cast(catChars.size()); + text.sortLength = foldedLength; + catChars.resize(catChars.size() + foldedLength + 1); + memcpy(catChars.begin() + text.sortOffset, folded, foldedLength); + categories.push(text); + } + } + if (catChars.failed() || categories.failed()) { + return failure.fail("out of memory"); + } + + // ---- name groups + GrowArray created; + GrowArray groupOfPoint; // created index of a named point's group + groupOfPoint.resize(pointCount); + { + GrowArray named; + for (uint32_t i = 0; i < pointCount; i++) { + if (points[i].name != nullptr) { + named.push(i); + } + } + std::sort(named.begin(), named.end(), [&](uint32_t a, uint32_t b) { + int c = strcmp(points[a].name, points[b].name); + if (c != 0) { + return c < 0; + } + return pairOf[a] != pairOf[b] ? pairOf[a] < pairOf[b] : a < b; + }); + for (size_t k = 0; k < named.size(); k++) { + uint32_t i = named[k]; + bool head = k == 0 || strcmp(points[named[k - 1]].name, points[i].name) != 0 || pairOf[named[k - 1]] != pairOf[i]; + if (head) { + NameGroup group = {}; + group.name = points[i].name; + group.nameLength = truncateName(group.name, strlen(group.name)); + group.sortKey = points[i].sortKey != nullptr ? points[i].sortKey : ""; + group.sortLength = static_cast(strlen(group.sortKey)); + group.kind = static_cast(SearchKind::Point); + const char* label = points[i].label; + group.category = strncmp(label, "poi/", 4) == 0 ? CAT_POI : (strncmp(label, "place/", 6) == 0 ? CAT_PLACE : CAT_OTHER); + group.pair = pairOf[i]; + group.creation = i; + group.source = i; + created.push(group); + } + groupOfPoint[i] = static_cast(created.size() - 1); + } + } + for (uint32_t s = 0; s < streetCount; s++) { + NameGroup group = {}; + group.name = streets[s].name; + group.nameLength = truncateName(group.name, strlen(group.name)); + group.sortKey = streets[s].sortKey != nullptr ? streets[s].sortKey : ""; + group.sortLength = static_cast(strlen(group.sortKey)); + group.kind = static_cast(SearchKind::Street); + group.category = CAT_STREET; + group.creation = pointCount + s; + group.source = s; + created.push(group); + } + for (uint32_t c = 0; c < categories.size(); c++) { + NameGroup group = {}; + group.name = catChars.begin() + categories[c].nameOffset; + group.nameLength = truncateName(group.name, categories[c].nameLength); + group.sortKey = catChars.begin() + categories[c].sortOffset; + group.sortLength = categories[c].sortLength; + group.kind = static_cast(SearchKind::Category); + group.category = CAT_CATEGORY; + group.creation = pointCount + streetCount + c; + group.source = c; + group.pairFirst = categories[c].run.firstPair; + group.pairCount = categories[c].run.count; + created.push(group); + } + if (created.failed()) { + return failure.fail("out of memory"); + } + const uint32_t groupTotal = static_cast(created.size()); + GrowArray groupOrder; + groupOrder.resize(groupTotal); + for (uint32_t g = 0; g < groupTotal; g++) { + groupOrder[g] = g; + } + std::sort(groupOrder.begin(), groupOrder.end(), [&](uint32_t a, uint32_t b) { + const NameGroup& x = created[a]; + const NameGroup& y = created[b]; + if (x.kind != y.kind) { + return x.kind < y.kind; + } + int c = compareBytes(reinterpret_cast(x.sortKey), x.sortLength, reinterpret_cast(y.sortKey), + y.sortLength); + if (c != 0) { + return c < 0; + } + if (x.pair != y.pair) { + return x.pair < y.pair; + } + return x.creation < y.creation; + }); + GrowArray groups; + GrowArray newIndex; + groups.resize(groupTotal); + newIndex.resize(groupTotal); + uint32_t groupBytes = 0; + for (uint32_t k = 0; k < groupTotal; k++) { + groups[k] = created[groupOrder[k]]; + newIndex[groupOrder[k]] = k; + groups[k].offset = groupBytes; + groupBytes += 1 + groups[k].nameLength + 1 + 4 + 2; + } + if (groupBytes >= SIDX_MAX_OFFSET24) { + return failure.fail("the name groups take %u bytes, more than 16 MB", groupBytes); + } + + // ---- point records, sorted by (pair, named first, name group, tile) + GrowArray records; + records.resize(pointCount); + for (uint32_t i = 0; i < pointCount; i++) { + PointRecord& r = records[i]; + r.pair = pairOf[i]; + r.unnamed = points[i].name == nullptr ? 1 : 0; + r.ng = r.unnamed ? 0 : newIndex[groupOfPoint[i]]; + r.tileY = points[i].y >> 12; + r.tileX = points[i].x >> 12; + r.x = points[i].x; + r.y = points[i].y; + } + std::sort(records.begin(), records.end(), [](const PointRecord& a, const PointRecord& b) { + if (a.pair != b.pair) return a.pair < b.pair; + if (a.unnamed != b.unnamed) return a.unnamed < b.unnamed; + if (a.ng != b.ng) return a.ng < b.ng; + if (a.tileY != b.tileY) return a.tileY < b.tileY; + if (a.tileX != b.tileX) return a.tileX < b.tileX; + if (a.x != b.x) return a.x < b.x; + return a.y < b.y; + }); + GrowArray firstOf, countOf, pairStart, pairCount; + firstOf.resize(groupTotal); + countOf.resize(groupTotal); + pairStart.resize(pairTotal); + pairCount.resize(pairTotal); + for (uint32_t i = 0; i < pointCount; i++) { + const PointRecord& r = records[i]; + if (!r.unnamed) { + if (countOf[r.ng] == 0) { + firstOf[r.ng] = i; + } + countOf[r.ng]++; + } + if (pairCount[r.pair] == 0) { + pairStart[r.pair] = i; + } + pairCount[r.pair]++; + } + + // ---- street records and addresses, in name group order + GrowArray streetRecords; + GrowArray addressBlob; + OverflowTable overflow; + uint32_t addressTotal = 0; + for (uint32_t k = 0; k < groupTotal; k++) { + NameGroup& g = groups[k]; + if (g.kind != static_cast(SearchKind::Street)) { + continue; + } + const SearchStreetInput& street = streets[g.source]; + if (street.clusterCount > 0xFFFF) { + return failure.fail("a street has %u clusters, the limit is 65535", street.clusterCount); + } + g.first = static_cast(streetRecords.size()); + g.count = street.clusterCount; + for (uint32_t c = 0; c < street.clusterCount; c++) { + const SearchClusterInput& cluster = street.clusters[c]; + if (cluster.addressCount > 0xFFFF) { + return failure.fail("a street cluster has %u addresses, the limit is 65535", cluster.addressCount); + } + GrowArray temp; + for (uint32_t a = 0; a < cluster.addressCount; a++) { + const SearchAddressInput& in2 = cluster.addresses[a]; + AddressTemp t = {}; + t.x = in2.x - originX; + t.y = in2.y - originY; + t.index = a; + if (!parseNumber(in2.number, in.slashSecond, t.number, t.aux, t.suffix, t.flags)) { + uint32_t at = overflow.find(in2.number, strlen(in2.number)); + if (at == UINT32_MAX || at >= SIDX_MAX_OFFSET24) { + return failure.fail("the overflow strings pass 16 MB"); + } + t.number = 0xFFFF; + t.aux = static_cast(at & 0xFFFF); + t.suffix = static_cast(at >> 16); + t.flags = HOUSE_TEXT; + } + t.flags = static_cast(t.flags | in2.flags); + temp.push(t); + } + std::sort(temp.begin(), temp.end(), [](const AddressTemp& a, const AddressTemp& b) { + if (a.number != b.number) return a.number < b.number; + if (a.aux != b.aux) return a.aux < b.aux; + if (a.suffix != b.suffix) return a.suffix < b.suffix; + return a.index < b.index; + }); + StreetTemp rec = { &cluster, addressTotal, cluster.addressCount }; + streetRecords.push(rec); + for (const AddressTemp& t : temp) { + uint8_t bytes[16]; + uint32_t size = encodeAddress(bytes, pts, t.number, t.aux, t.suffix, t.flags, t.x, t.y); + appendBytes(addressBlob, bytes, size); + } + addressTotal += cluster.addressCount; + } + } + + // ---- name group counts, then the name group area and the point and street records + for (uint32_t k = 0; k < groupTotal; k++) { + NameGroup& g = groups[k]; + if (g.kind == static_cast(SearchKind::Point)) { + g.first = firstOf[k]; + g.count = countOf[k]; + } else if (g.kind == static_cast(SearchKind::Category)) { + g.first = pairStart[g.pairFirst]; + g.count = 0; + for (uint32_t p = 0; p < g.pairCount; p++) { + g.count += pairCount[g.pairFirst + p]; + } + } + if (g.count > 0xFFFF) { + return failure.fail("name group %u has more than 65535 records", k); + } + } + GrowArray groupBlob; + for (uint32_t k = 0; k < groupTotal; k++) { + uint8_t bytes[SIDX_MAX_NAME + 8]; + uint32_t size = encodeNameGroup(bytes, groups[k].name, groups[k].nameLength, static_cast(groups[k].kind), groups[k].first, + static_cast(groups[k].count)); + appendBytes(groupBlob, bytes, size); + } + GrowArray pointBlob; + for (uint32_t i = 0; i < pointCount; i++) { + const PointRecord& r = records[i]; + uint8_t bytes[16]; + uint32_t size = encodePoint(bytes, pts, r.x - originX, r.y - originY, static_cast(r.pair), + r.unnamed ? SIDX_UNNAMED : groups[r.ng].offset); + appendBytes(pointBlob, bytes, size); + } + GrowArray streetBlob; + for (const StreetTemp& s : streetRecords) { + const SearchClusterInput& c = *s.cluster; + int32_t rep[2] = { c.repX - originX, c.repY - originY }; + int32_t lo[2] = { c.minX - originX, c.minY - originY }; + int32_t hi[2] = { c.maxX - originX, c.maxY - originY }; + uint8_t bytes[32]; + uint32_t size = encodeStreet(bytes, pts, rep, lo, hi, s.firstAddress, static_cast(s.count)); + appendBytes(streetBlob, bytes, size); + } + + // ---- keys, key pages and fence + GrowArray keyBytes; + GrowArray entries; + for (uint32_t k = 0; k < groupTotal; k++) { + const NameGroup& g = groups[k]; + const char* const* keys = nullptr; + uint32_t keyCount = 0; + const char* categoryKey[1]; + if (g.kind == static_cast(SearchKind::Point)) { + keys = points[g.source].keys; + keyCount = points[g.source].keyCount; + } else if (g.kind == static_cast(SearchKind::Street)) { + keys = streets[g.source].keys; + keyCount = streets[g.source].keyCount; + } else { + categoryKey[0] = g.sortKey; + keys = categoryKey; + keyCount = 1; + } + for (uint32_t i = 0; i < keyCount; i++) { + size_t length = strlen(keys[i]); + if (length == 0) { + continue; + } + KeyEntry e = {}; + e.length = truncateKey(keys[i], length); + e.offset = static_cast(keyBytes.size()); + e.ng = g.offset; + e.priority = g.category; + size_t at = keyBytes.size(); + keyBytes.resize(at + e.length); + memcpy(keyBytes.begin() + at, keys[i], e.length); + entries.push(e); + } + } + if (keyBytes.failed() || entries.failed()) { + return failure.fail("out of memory"); + } + const uint8_t* keyBase = reinterpret_cast(keyBytes.begin()); + std::sort(entries.begin(), entries.end(), [&](const KeyEntry& a, const KeyEntry& b) { + int c = compareBytes(keyBase + a.offset, a.length, keyBase + b.offset, b.length); + if (c != 0) return c < 0; + if (a.priority != b.priority) return a.priority < b.priority; + return a.ng < b.ng; + }); + GrowArray keyBlob, fenceBlob; + uint32_t pageCount = 0, fenceCount = 0; + { + uint8_t page[SIDX_PAGE_SIZE]; + uint32_t used = 0; + memset(page, 0, sizeof(page)); + for (const KeyEntry& e : entries) { + if (!encodeKeyPage(page, used, keyBase + e.offset, e.length, e.ng)) { + appendBytes(keyBlob, page, SIDX_PAGE_SIZE); + pageCount++; + memset(page, 0, sizeof(page)); + used = 0; + if (!encodeKeyPage(page, used, keyBase + e.offset, e.length, e.ng)) { + return failure.fail("a key does not fit a page"); + } + } + if (used == 1 + e.length + 3) { + uint8_t length = static_cast(e.length); + appendBytes(fenceBlob, &length, 1); + appendBytes(fenceBlob, keyBase + e.offset, e.length); + fenceCount++; + } + } + if (used > 0) { + appendBytes(keyBlob, page, SIDX_PAGE_SIZE); + pageCount++; + } + } + GrowArray classBlob; + for (uint32_t pair = 0; pair < pairTotal; pair++) { + const char* label = pairText(pair); + uint8_t length = static_cast(strlen(label)); + uint8_t tail[8]; + put32(tail, pairStart[pair]); + put32(tail + 4, pairCount[pair]); + appendBytes(classBlob, &length, 1); + appendBytes(classBlob, label, length); + appendBytes(classBlob, tail, 8); + } + if (pointBlob.failed() || streetBlob.failed() || addressBlob.failed() || groupBlob.failed() || keyBlob.failed() || fenceBlob.failed() || + classBlob.failed() || overflow.bytes.failed() || groups.failed()) { + return failure.fail("out of memory"); + } + + // ---- assemble + const GrowArray* parts[SIDX_SECTION_COUNT] = { &classBlob, &fenceBlob, &keyBlob, &groupBlob, &pointBlob, &streetBlob, &addressBlob, nullptr }; + SearchHeader header = {}; + header.pointBytes = static_cast(pts); + header.originX = originX; + header.originY = originY; + header.classCount = pairTotal; + header.fenceCount = fenceCount; + header.pageCount = pageCount; + header.pointCount = pointCount; + header.streetCount = static_cast(streetRecords.size()); + header.addressCount = addressTotal; + uint64_t offset = SIDX_HEADER_SIZE; + for (uint32_t i = 0; i < SIDX_SECTION_COUNT; i++) { + uint64_t size = parts[i] != nullptr ? parts[i]->size() : overflow.bytes.size(); + header.sectionOffset[i] = static_cast(offset); + header.sectionSize[i] = static_cast(size); + offset += size; + } + if (offset > 0xFFFFFFFFull) { + return failure.fail("the index is larger than 4 GB"); + } + uint8_t headerBytes[SIDX_HEADER_SIZE]; + encodeHeader(header, headerBytes); + bool ok = sink.append(headerBytes, SIDX_HEADER_SIZE); + for (uint32_t i = 0; i < SIDX_SECTION_COUNT && ok; i++) { + const void* data = parts[i] != nullptr ? static_cast(parts[i]->begin()) : static_cast(overflow.bytes.begin()); + if (header.sectionSize[i] > 0) { + ok = sink.append(data, header.sectionSize[i]); + } + } + if (!ok) { + return failure.fail("the sink failed"); + } + if (stats != nullptr) { + *stats = {}; + stats->totalBytes = static_cast(offset); + stats->header = header; + stats->keyCount = static_cast(entries.size()); + stats->ngCount = groupTotal; + for (uint32_t i = 0; i < pointCount; i++) { + if (points[i].name != nullptr) { + stats->namedPoints++; + } else { + stats->unnamedPoints++; + } + } + stats->overflowStrings = overflow.count; + stats->collapsedPairs = collapsed; + stats->categoryNotContiguous = notContiguous; + } + return true; +} + +// ------------------------------------------------------------------------------------------------ reader + +uint32_t SearchIndexReader::readSection(SidxSection section, uint32_t offset, void* out, uint32_t size) { + uint32_t index = static_cast(section); + uint32_t length = header.sectionSize[index]; + if (offset >= length || size == 0) { + return 0; + } + if (size > length - offset) { + size = length - offset; + } + callCount++; + if (section == SidxSection::Ng && ngResident) { + memcpy(out, residentNg.begin() + offset, size); + return size; + } + if (section == SidxSection::Keys && keysResident) { + memcpy(out, residentKeys.begin() + offset, size); + return size; + } + uint32_t absolute = header.sectionOffset[index] + offset; + touch(absolute, size); + return file->read(absolute, out, size) ? size : 0; +} + +void SearchIndexReader::touch(uint32_t offset, uint32_t size) { + uint32_t last = (offset + (size > 0 ? size : 1) - 1) / SIDX_PAGE_SIZE; + for (uint32_t b = offset / SIDX_PAGE_SIZE; b <= last && b < blockSeen.size(); b++) { + blockSeen[b] = 1; + } +} + +void SearchIndexReader::resetIo() { + callCount = 0; + for (uint8_t& seen : blockSeen) { + seen = 0; + } +} + +SearchIoStats SearchIndexReader::io() const { + SearchIoStats stats = {}; + stats.calls = callCount; + bool previous = false; + for (uint8_t seen : blockSeen) { + if (seen) { + stats.blocks++; + if (!previous) { + stats.runs++; + } + } + previous = seen != 0; + } + return stats; +} + +bool SearchIndexReader::open(SearchFile* source) { + file = source; + uint8_t raw[SIDX_HEADER_SIZE]; + if (file == nullptr || !file->read(0, raw, SIDX_HEADER_SIZE)) { + return false; + } + if (memcmp(raw, "SIDX", 4) != 0 || get16(raw + 4) != SIDX_VERSION) { + return false; + } + header = {}; + header.pointBytes = static_cast(get16(raw + 6)); + if (header.pointBytes != 4 && header.pointBytes != 8) { + return false; + } + pointBytes = header.pointBytes; + header.originX = static_cast(get32(raw + 8)); + header.originY = static_cast(get32(raw + 12)); + header.classCount = get32(raw + 16); + header.fenceCount = get32(raw + 20); + header.pageCount = get32(raw + 24); + header.pointCount = get32(raw + 28); + header.streetCount = get32(raw + 32); + header.addressCount = get32(raw + 36); + uint64_t end = SIDX_HEADER_SIZE; + for (uint32_t i = 0; i < SIDX_SECTION_COUNT; i++) { + header.sectionOffset[i] = get32(raw + 44 + 8 * i); + header.sectionSize[i] = get32(raw + 48 + 8 * i); + end = std::max(end, static_cast(header.sectionOffset[i]) + header.sectionSize[i]); + } + ngResident = keysResident = false; + residentNg.release(); + residentKeys.release(); + blockSeen.release(); + blockSeen.resize(static_cast(end / SIDX_PAGE_SIZE + 1)); + classBlob.release(); + classOffsets.release(); + fenceBlob.release(); + fenceOffsets.release(); + if (!classBlob.resize(header.sectionSize[0]) || !fenceBlob.resize(header.sectionSize[1])) { + return false; + } + if (header.sectionSize[0] > 0 && !file->read(header.sectionOffset[0], classBlob.begin(), header.sectionSize[0])) { + return false; + } + if (header.sectionSize[1] > 0 && !file->read(header.sectionOffset[1], fenceBlob.begin(), header.sectionSize[1])) { + return false; + } + for (uint32_t at = 0, i = 0; i < header.classCount; i++) { + if (at >= classBlob.size() || at + 1 + classBlob[at] + 8 > classBlob.size()) { + return false; + } + classOffsets.push(at); + at += 1 + classBlob[at] + 8; + } + for (uint32_t at = 0; at < fenceBlob.size();) { + if (at + 1 + fenceBlob[at] > fenceBlob.size()) { + return false; + } + fenceOffsets.push(at); + at += 1 + fenceBlob[at]; + } + if (classOffsets.failed() || fenceOffsets.failed() || fenceOffsets.size() != header.fenceCount) { + return false; + } + resetIo(); + return true; +} + +bool SearchIndexReader::loadResident(bool nameGroups, bool keyPages) { + if (file == nullptr) { + return false; + } + auto load = [&](SidxSection section, GrowArray& into, bool& flag) { + flag = false; + into.release(); + uint32_t index = static_cast(section); + if (header.sectionSize[index] == 0) { + return true; + } + if (!into.resize(header.sectionSize[index]) || !file->read(header.sectionOffset[index], into.begin(), header.sectionSize[index])) { + into.release(); + return false; + } + flag = true; + return true; + }; + bool ok = true; + if (nameGroups) { + ok = load(SidxSection::Ng, residentNg, ngResident) && ok; + } else { + ngResident = false; + residentNg.release(); + } + if (keyPages) { + ok = load(SidxSection::Keys, residentKeys, keysResident) && ok; + } else { + keysResident = false; + residentKeys.release(); + } + return ok; +} + +bool SearchIndexReader::classPair(uint32_t index, char* label, size_t labelSize, uint32_t& first, uint32_t& count) const { + if (index >= classOffsets.size()) { + return false; + } + const uint8_t* entry = classBlob.begin() + classOffsets[index]; + uint32_t length = entry[0]; + if (labelSize > 0) { + size_t copy = std::min(length, labelSize - 1); + memcpy(label, entry + 1, copy); + label[copy] = 0; + } + first = get32(entry + 1 + length); + count = get32(entry + 5 + length); + return true; +} + +bool SearchIndexReader::readPage(uint32_t page, uint8_t* out) { + return readSection(SidxSection::Keys, page * SIDX_PAGE_SIZE, out, SIDX_PAGE_SIZE) == SIDX_PAGE_SIZE; +} + +uint32_t SearchIndexReader::prefix(const char* key, uint32_t limit, uint32_t* ngOffsets) { + const uint8_t* query = reinterpret_cast(key); + uint32_t queryLength = truncateKey(key, strlen(key)); + // bisect_left over the fence + size_t lo = 0, hi = fenceOffsets.size(); + while (lo < hi) { + size_t mid = (lo + hi) / 2; + const uint8_t* fence = fenceBlob.begin() + fenceOffsets[mid]; + if (compareBytes(fence + 1, fence[0], query, queryLength) < 0) { + lo = mid + 1; + } else { + hi = mid; + } + } + uint32_t page = lo > 0 ? static_cast(lo - 1) : 0; + uint32_t found = 0; + bool done = limit == 0; + for (; page < header.pageCount && !done; page++) { + if (!readPage(page, pageBuffer)) { + break; + } + uint32_t i = 0; + while (i < SIDX_PAGE_SIZE && pageBuffer[i] != 0) { + uint32_t length = pageBuffer[i]; + if (i + 4 + length > SIDX_PAGE_SIZE) { + break; + } + const uint8_t* entry = pageBuffer + i + 1; + uint32_t reference = get24(entry + length); + i += 4 + length; + if (length >= queryLength && memcmp(entry, query, queryLength) == 0) { + bool seen = false; + for (uint32_t r = 0; r < found; r++) { + seen = seen || ngOffsets[r] == reference; + } + if (!seen) { + ngOffsets[found++] = reference; + } + if (found >= limit) { + done = true; + break; + } + } else if (compareBytes(entry, length, query, queryLength) > 0) { + done = true; + break; + } + } + } + return found; +} + +bool SearchIndexReader::ng(uint32_t offset, SearchNg& out) { + uint8_t bytes[SIDX_MAX_NAME + 8]; + uint32_t got = readSection(SidxSection::Ng, offset, bytes, 80); + if (got < 1) { + return false; + } + uint32_t length = bytes[0]; + if (got < length + 8) { + got = readSection(SidxSection::Ng, offset, bytes, length + 8); + if (got < length + 8) { + return false; + } + } + memcpy(out.name, bytes + 1, length); + out.name[length] = 0; + out.kind = static_cast(bytes[length + 1]); + out.first = get32(bytes + length + 2); + out.count = static_cast(get16(bytes + length + 6)); + out.offset = offset; + return true; +} + +void SearchIndexReader::decodePt(const uint8_t* bytes, int32_t& x, int32_t& y) const { + if (pointBytes == 8) { + x = static_cast(get32(bytes)) + header.originX; + y = static_cast(get32(bytes + 4)) + header.originY; + } else { + x = static_cast(get16(bytes)) + header.originX; + y = static_cast(get16(bytes + 2)) + header.originY; + } +} + +bool SearchIndexReader::pointRecords(uint32_t first, uint32_t count, GrowArray& out) { + out.clear(); + if (count == 0) { + return true; + } + uint32_t size = pointRecordBytes(pointBytes); + GrowArray raw; + if (!raw.resize(static_cast(count) * size) || + readSection(SidxSection::Points, first * size, raw.begin(), count * size) != count * size) { + return false; + } + for (uint32_t i = 0; i < count; i++) { + const uint8_t* b = raw.begin() + static_cast(i) * size; + SearchPointRecord r; + decodePt(b, r.x, r.y); + r.pair = b[pointBytes]; + r.ng = get24(b + pointBytes + 1); + out.push(r); + } + return !out.failed(); +} + +bool SearchIndexReader::streetRecords(uint32_t first, uint32_t count, GrowArray& out) { + out.clear(); + if (count == 0) { + return true; + } + uint32_t size = streetRecordBytes(pointBytes); + GrowArray raw; + if (!raw.resize(static_cast(count) * size) || + readSection(SidxSection::Streets, first * size, raw.begin(), count * size) != count * size) { + return false; + } + for (uint32_t i = 0; i < count; i++) { + const uint8_t* b = raw.begin() + static_cast(i) * size; + SearchStreetRecord r; + decodePt(b, r.x, r.y); + decodePt(b + pointBytes, r.minX, r.minY); + decodePt(b + 2 * pointBytes, r.maxX, r.maxY); + r.firstAddress = get32(b + 3 * pointBytes); + r.addressCount = static_cast(get16(b + 3 * pointBytes + 4)); + out.push(r); + } + return !out.failed(); +} + +void SearchIndexReader::decodeAddress(const uint8_t* b, SearchAddress& out) const { + out.number = static_cast(get16(b)); + out.aux = static_cast(get16(b + 2)); + out.flags = b[5]; + out.isText = (out.flags & HOUSE_TEXT) != 0; + out.suffix = out.isText ? 0 : static_cast(b[4]); + out.textOffset = out.isText ? (static_cast(out.aux) | (static_cast(b[4]) << 16)) : 0; + decodePt(b + 6, out.x, out.y); +} + +bool SearchIndexReader::addressAt(uint32_t index, SearchAddress& out) { + uint32_t size = addressRecordBytes(pointBytes); + uint8_t bytes[16]; + if (readSection(SidxSection::Addresses, index * size, bytes, size) != size) { + return false; + } + decodeAddress(bytes, out); + return true; +} + +bool SearchIndexReader::addressFind(uint32_t first, uint32_t count, uint16_t number, int suffix, GrowArray& out) { + out.clear(); + uint32_t lo = 0, hi = count; + SearchAddress a; + while (lo < hi) { + uint32_t mid = (lo + hi) / 2; + if (!addressAt(first + mid, a)) { + return false; + } + if (a.number < number) { + lo = mid + 1; + } else { + hi = mid; + } + } + for (uint32_t i = lo; i < count; i++) { + if (!addressAt(first + i, a)) { + return false; + } + if (a.number != number) { + break; + } + if (suffix < 0 || a.suffix == suffix) { + out.push(a); + } + } + return !out.failed(); +} + +bool SearchIndexReader::overflowText(uint32_t offset, char* out, size_t outSize) { + if (outSize == 0) { + return false; + } + size_t length = 0; + while (true) { + char chunk[32]; + uint32_t got = readSection(SidxSection::Overflow, offset + static_cast(length), chunk, sizeof(chunk)); + if (got == 0) { + out[length < outSize ? length : outSize - 1] = 0; + return length > 0; + } + for (uint32_t i = 0; i < got; i++) { + if (chunk[i] == 0) { + out[std::min(length, outSize - 1)] = 0; + return true; + } + if (length + 1 < outSize) { + out[length] = chunk[i]; + } + length++; + } + } +} + +bool SearchIndexReader::searchName(const char* key, uint32_t limit, uint32_t show, GrowArray& out) { + out.clear(); + GrowArray refs; + if (!refs.resize(limit)) { + return false; + } + uint32_t found = prefix(key, limit, refs.begin()); + for (uint32_t i = 0; i < found; i++) { + SearchNg group; + if (!ng(refs[i], group)) { + return false; + } + out.push(group); + } + for (uint32_t i = 0; i < out.size() && i < show; i++) { + if (out[i].kind == SearchKind::Point) { + GrowArray records; + pointRecords(out[i].first, std::min(out[i].count, 64), records); + } else if (out[i].kind == SearchKind::Street) { + GrowArray records; + streetRecords(out[i].first, std::min(out[i].count, 16), records); + } + } + return !out.failed(); +} + +bool SearchIndexReader::searchAddress(const char* streetKey, uint16_t number, int suffix, uint32_t limit, + GrowArray& out) { + out.clear(); + GrowArray refs; + if (!refs.resize(limit)) { + return false; + } + uint32_t found = prefix(streetKey, limit, refs.begin()); + for (uint32_t i = 0; i < found; i++) { + SearchNg group; + if (!ng(refs[i], group)) { + return false; + } + if (group.kind != SearchKind::Street) { + continue; + } + GrowArray records; + if (!streetRecords(group.first, group.count, records)) { + return false; + } + for (const SearchStreetRecord& record : records) { + if (record.addressCount == 0) { + continue; + } + GrowArray matches; + if (!addressFind(record.firstAddress, record.addressCount, number, suffix, matches)) { + return false; + } + for (const SearchAddress& a : matches) { + SearchAddressHit hit = {}; + snprintf(hit.street, sizeof(hit.street), "%s", group.name); + hit.address = a; + out.push(hit); + } + } + if (out.size() >= 8) { + break; + } + } + return !out.failed(); +} + +bool SearchIndexReader::category(const SearchNg& group, GrowArray& out) { + return pointRecords(group.first, group.count, out); +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchIndex.h b/Apps/Maps/main/Source/search/SearchIndex.h new file mode 100644 index 0000000..fdbb9df --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchIndex.h @@ -0,0 +1,344 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "../route/GrowArray.h" + +#include +#include + +namespace maps { + +/** + * The search index file (.sidx), format version 1. It is the C++ port of host/search/sidx.py (M6 study, section 3). + * All numbers are little endian. + * + * header (128 bytes) | class table | fence table | key pages | name groups | point records | street records | + * address records | overflow strings + * + * - header: "SIDX", u16 version (1), u16 point size (4, or 8 for a wide region), i32 origin x, i32 origin y, + * u32 class count, fence count, key page count, point count, street record count, u32 address count, u32 0, then + * eight (u32 offset, u32 size) pairs for the sections above, in that order. The rest is zero. + * - point "PT": (u16 x, u16 y) relative to the origin, one unit = one z14 tile unit (~0.4 m at 48 N). When the + * region is 65536 units or wider, a PT is (i32 x, i32 y) and the point size is 8 ("wide mode"). + * - class table, per class pair: u8 length, label ("poi/cafe/cafe", "place/town"), u32 first point record, u32 count. + * - fence table: u8 length, key bytes, for the first key of every key page. + * - key page: 4096 bytes of entries { u8 key length (1..40), key bytes, u24 name group offset }, sorted by key bytes, + * a zero length ends the page, no entry straddles a page. A key is cut at 40 bytes on a UTF-8 character boundary. + * - name group ("NG"): u8 name length, name (UTF-8, at most 255 bytes, cut on a character boundary), u8 kind + * (0 points, 1 streets, 2 category), u32 first record, u16 count (records in the point or street table). + * - point record: PT, u8 class pair, u24 NG offset (0xFFFFFF for an unnamed point). Sorted by (pair, named first, + * NG, tile), so a whole class pair is one run. + * - street record: PT representative point, PT box minimum, PT box maximum, u32 first address, u16 address count. + * - address record: u16 number, u16 aux, u8 suffix, u8 flags, PT. Sorted per street record by (number, aux, suffix). + * A house number that does not fit the number grammar (parseHouseNumber) is stored as text in the overflow strings: + * number = 0xFFFF, flags = HOUSE_TEXT, and the text's offset in the overflow section is 24 bits: the low 16 bits + * in aux, the high 8 bits in the suffix byte (version 1 of sidx.py wrapped at 64 KB here). Offsets are limited to + * 16 MB, the builder fails when exceeded. + * - overflow strings: UTF-8 text, each ended by a zero byte. + * + * Counts keep the field widths above (u16 for an NG's records, a street's clusters and a cluster's addresses); the + * writer fails with an error text instead of capping a count. + */ +constexpr uint32_t SIDX_VERSION = 1; +constexpr uint32_t SIDX_HEADER_SIZE = 128; +constexpr uint32_t SIDX_PAGE_SIZE = 4096; +constexpr uint32_t SIDX_MAX_KEY = 40; +constexpr uint32_t SIDX_MAX_NAME = 255; +constexpr uint32_t SIDX_UNNAMED = 0xFFFFFF; +constexpr uint32_t SIDX_MAX_OFFSET24 = 0xFFFFFF; +constexpr uint32_t SIDX_SECTION_COUNT = 8; + +/** The flag of an address record whose number is text in the overflow strings (HOUSE_SLASH = 1, HOUSE_RANGE = 2). */ +constexpr uint8_t HOUSE_TEXT = 4; +/** + * Flags the caller sets through SearchAddressInput::flags (added to version 1, which no device has released yet): + * ADDRESS_LOW marks an address whose street is 50 to 100 m from the house (low confidence). ADDRESS_VILLAGE marks an + * address stored under a settlement: the street record of such a place is an ordinary one named like the place, with + * one cluster whose representative point and box are the place's position (a place may share the name group of a real + * street of the same name), and every address under it carries ADDRESS_VILLAGE. + */ +constexpr uint8_t ADDRESS_LOW = 8; +constexpr uint8_t ADDRESS_VILLAGE = 16; + +enum class SidxSection : uint8_t { Class, Fence, Keys, Ng, Points, Streets, Addresses, Overflow }; +enum class SearchKind : uint8_t { Point = 0, Street = 1, Category = 2 }; + +/** Cuts a key to at most SIDX_MAX_KEY bytes without splitting a UTF-8 character. Returns the new length. */ +uint32_t truncateKey(const char* key, size_t length); +/** The same for a name group's name (SIDX_MAX_NAME). */ +uint32_t truncateName(const char* name, size_t length); +/** Compares byte strings like Python bytes: content, then the shorter first (the key order). */ +int compareSearchBytes(const uint8_t* a, uint32_t aLength, const uint8_t* b, uint32_t bLength); +/** The key of a category name made of ASCII: lowercase, other characters a single blank (the writer's rule). */ +uint32_t foldCategoryName(const char* name, char* out, size_t capacity); +/** + * A house number as an address record stores it (number, aux, suffix, flags; "x/y" swapped when slashSecond). + * False when it must be stored as text in the overflow strings. + */ +bool indexHouseNumber(const char* text, bool slashSecond, uint16_t& number, uint16_t& aux, uint8_t& suffix, uint8_t& flags); + +// ------------------------------------------------------------------------------------------------ encoders + +/** The parts of the header. */ +struct SearchHeader { + uint16_t pointBytes; + int32_t originX; + int32_t originY; + uint32_t classCount; + uint32_t fenceCount; + uint32_t pageCount; + uint32_t pointCount; + uint32_t streetCount; + uint32_t addressCount; + uint32_t sectionOffset[SIDX_SECTION_COUNT]; + uint32_t sectionSize[SIDX_SECTION_COUNT]; +}; + +/** The sizes of the records for a point size. */ +inline uint32_t pointRecordBytes(uint32_t pointBytes) { return pointBytes + 4; } +inline uint32_t streetRecordBytes(uint32_t pointBytes) { return 3 * pointBytes + 6; } +inline uint32_t addressRecordBytes(uint32_t pointBytes) { return 6 + pointBytes; } + +/** Writes the 128-byte header. */ +void encodeHeader(const SearchHeader& header, uint8_t out[SIDX_HEADER_SIZE]); +/** Writes a PT (x, y relative to the origin); returns the point size. */ +uint32_t encodePt(uint8_t* out, uint32_t pointBytes, int32_t relX, int32_t relY); +/** + * Appends one key entry to a key page being filled (page is SIDX_PAGE_SIZE bytes, used = bytes so far, starting at + * 0). Returns false and changes nothing when the entry and the ending zero byte do not fit. The key must be cut + * (truncateKey) and not empty. + */ +bool encodeKeyPage(uint8_t* page, uint32_t& used, const uint8_t* key, uint32_t keyLength, uint32_t ngOffset); +/** One name group; returns its size (1 + name length + 1 + 4 + 2). The name must be cut (truncateName). */ +uint32_t encodeNameGroup(uint8_t* out, const char* name, uint32_t nameLength, SearchKind kind, uint32_t first, uint16_t count); +/** One point record; returns its size. */ +uint32_t encodePoint(uint8_t* out, uint32_t pointBytes, int32_t relX, int32_t relY, uint8_t pair, uint32_t ngOffset); +/** One street record; returns its size. */ +uint32_t encodeStreet(uint8_t* out, uint32_t pointBytes, const int32_t rep[2], const int32_t boxMin[2], const int32_t boxMax[2], + uint32_t firstAddress, uint16_t addressCount); +/** One address record; returns its size. */ +uint32_t encodeAddress(uint8_t* out, uint32_t pointBytes, uint16_t number, uint16_t aux, uint8_t suffix, uint8_t flags, int32_t relX, + int32_t relY); +/** An address record for text: the offset (below 2^24) goes into aux and the suffix byte. */ +uint32_t encodeTextAddress(uint8_t* out, uint32_t pointBytes, uint32_t textOffset, int32_t relX, int32_t relY); + +// ------------------------------------------------------------------------------------------------ writer + +/** Where the writer puts the file's bytes, in order. The host tool writes a file, the device writes through its I/O task. */ +class ByteSink { +public: + virtual ~ByteSink() = default; + virtual bool append(const void* data, uint32_t size) = 0; +}; + +/** One address of a street cluster. */ +struct SearchAddressInput { + /** The house number as in the data ("12/A", "9 к3"). */ + const char* number; + int32_t x; + int32_t y; + /** ADDRESS_LOW, ADDRESS_VILLAGE or 0; stored in the record's flags next to the number's own flags. */ + uint8_t flags; +}; + +/** One place of a street name: the representative vertex, the box and the house numbers assigned to it. */ +struct SearchClusterInput { + int32_t repX, repY; + int32_t minX, minY; + int32_t maxX, maxY; + const SearchAddressInput* addresses; + uint32_t addressCount; +}; + +/** A street name with its clusters. Keys are made by the caller. */ +struct SearchStreetInput { + const char* name; + /** The name's sort key (folded, transliterated for Greek and Cyrillic). Name groups are ordered by it. */ + const char* sortKey; + /** All search keys of the name including word-suffix keys (not cut, empty ones are skipped). */ + const char* const* keys; + uint32_t keyCount; + const SearchClusterInput* clusters; + uint32_t clusterCount; +}; + +/** A point: a named or unnamed place, POI or other feature. */ +struct SearchPointInput { + /** The class pair label as sidx.py makes it: "poi//", "place/", "/". */ + const char* label; + /** The display name, nullptr for an unnamed point. */ + const char* name; + /** For a named point: sort key and keys as for streets. Points of the same name and label share the first one's. */ + const char* sortKey; + const char* const* keys; + uint32_t keyCount; + /** Position in global z14 units. */ + int32_t x; + int32_t y; +}; + +struct SearchInput { + const SearchPointInput* points; + uint32_t pointCount; + const SearchStreetInput* streets; + uint32_t streetCount; + /** "conscription/street" numbers: the second part is the street number and sorts first (sidx.py cfg slash_second). */ + bool slashSecond; + /** + * Folded keys for category names that are not plain ASCII (class names come from OSM tags, almost always ASCII): + * categoryNames[i] is the display name ("pet shop") and categoryKeys[i] its key. Names not in the table are folded + * as ASCII (lowercase, other characters become blanks). May be null. + */ + const char* const* categoryNames; + const char* const* categoryKeys; + uint32_t categoryOverrides; +}; + +/** What the build made, for reports and tests. */ +struct SearchBuildStats { + uint32_t totalBytes; + SearchHeader header; + uint32_t keyCount; + uint32_t ngCount; + uint32_t namedPoints; + uint32_t unnamedPoints; + uint32_t overflowStrings; + /** Class pairs folded into "/(all)" because there were more than 255. */ + uint32_t collapsedPairs; + /** Category runs without a key because the pairs are not contiguous. */ + uint32_t categoryNotContiguous; +}; + +/** + * Builds the whole index in memory and gives it to a sink. For the host tool and tests: the device later streams + * the pieces with the encode functions above. Fails (false, error()) when a count or offset does not fit its field. + */ +class SearchIndexWriter { +public: + bool build(const SearchInput& input, ByteSink& sink, SearchBuildStats* stats = nullptr); + const char* error() const { return errorText; } + +private: + char errorText[200] = {}; +}; + +// ------------------------------------------------------------------------------------------------ reader + +/** Where the reader gets bytes: a file, or memory. Returns false when the range cannot be read. */ +class SearchFile { +public: + virtual ~SearchFile() = default; + virtual bool read(uint32_t offset, void* out, uint32_t size) = 0; +}; + +/** A name group. */ +struct SearchNg { + char name[SIDX_MAX_NAME + 1]; + SearchKind kind; + uint32_t first; + uint16_t count; + uint32_t offset; +}; + +struct SearchPointRecord { + int32_t x, y; + uint8_t pair; + uint32_t ng; +}; + +struct SearchStreetRecord { + int32_t x, y; + int32_t minX, minY; + int32_t maxX, maxY; + uint32_t firstAddress; + uint16_t addressCount; +}; + +struct SearchAddress { + uint16_t number; + uint16_t aux; + /** Lowercase letter or 0. */ + char suffix; + uint8_t flags; + int32_t x, y; + /** Text number: true when flags has HOUSE_TEXT; textOffset is the offset in the overflow section. */ + bool isText; + uint32_t textOffset; +}; + +struct SearchAddressHit { + char street[96]; + SearchAddress address; +}; + +/** Reads made by the queries: calls, distinct 4 KB blocks of the file and runs of consecutive blocks. */ +struct SearchIoStats { + uint32_t calls; + uint32_t blocks; + uint32_t runs; +}; + +/** + * Reads an index through a SearchFile. The header, class table and fence table are kept in RAM; the name group area + * and the key pages can be made resident too (loadResident). Keys passed to the queries are already folded by the + * caller (TextFold), at most 40 bytes are used. + */ +class SearchIndexReader { +public: + /** Reads the header, class table and fence table. */ + bool open(SearchFile* file); + /** Loads the name group area and/or the key pages into memory. Reads inside them cost no file I/O afterwards. */ + bool loadResident(bool nameGroups, bool keyPages); + + void resetIo(); + SearchIoStats io() const; + /** Bytes read by open(): header + class table + fence table. */ + uint32_t openBytes() const { return SIDX_HEADER_SIZE + header.sectionSize[0] + header.sectionSize[1]; } + + const SearchHeader& info() const { return header; } + /** A class pair: label (cut to size), first point record and count. */ + bool classPair(uint32_t index, char* label, size_t labelSize, uint32_t& first, uint32_t& count) const; + + /** Distinct NG offsets of the keys that start with the folded key, in key order, at most limit. Returns the count. */ + uint32_t prefix(const char* key, uint32_t limit, uint32_t* ngOffsets); + bool ng(uint32_t offset, SearchNg& out); + bool pointRecords(uint32_t first, uint32_t count, GrowArray& out); + bool streetRecords(uint32_t first, uint32_t count, GrowArray& out); + bool addressAt(uint32_t index, SearchAddress& out); + /** Binary search in a street's addresses for a number; suffix -1 = any, 0 = none, else a letter. */ + bool addressFind(uint32_t first, uint32_t count, uint16_t number, int suffix, GrowArray& out); + /** The text of an overflow number (cut to size). */ + bool overflowText(uint32_t offset, char* out, size_t outSize); + + /** Name search: the NGs for a prefix; reads the first records of the first `show` results like sidx.Index. */ + bool searchName(const char* key, uint32_t limit, uint32_t show, GrowArray& out); + /** Address search: the street prefix key and a number; stops after a street when 8 addresses are found. */ + bool searchAddress(const char* streetKey, uint16_t number, int suffix, uint32_t limit, GrowArray& out); + /** The whole run of a category NG. */ + bool category(const SearchNg& group, GrowArray& out); + +private: + /** Reads from a section (clamped to its end); returns the bytes read, 0 on failure. */ + uint32_t readSection(SidxSection section, uint32_t offset, void* out, uint32_t size); + bool readPage(uint32_t page, uint8_t* out); + void decodePt(const uint8_t* bytes, int32_t& x, int32_t& y) const; + void decodeAddress(const uint8_t* bytes, SearchAddress& out) const; + void touch(uint32_t offset, uint32_t size); + + SearchFile* file = nullptr; + SearchHeader header = {}; + uint32_t pointBytes = 4; + GrowArray classBlob; + GrowArray classOffsets; + GrowArray fenceBlob; + GrowArray fenceOffsets; + GrowArray residentNg; + GrowArray residentKeys; + bool ngResident = false; + bool keysResident = false; + GrowArray blockSeen; + uint32_t callCount = 0; + uint8_t pageBuffer[SIDX_PAGE_SIZE] = {}; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchRegion.cpp b/Apps/Maps/main/Source/search/SearchRegion.cpp new file mode 100644 index 0000000..4110f74 --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchRegion.cpp @@ -0,0 +1,1154 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "SearchRegion.h" + +#include "QueryParse.h" + +#include +#include +#include +#include + +namespace maps { + +namespace { + +constexpr uint32_t NONE = UINT32_MAX; +constexpr int TILE_UNITS = 4096; +constexpr double PI_D = 3.14159265358979323846; +constexpr size_t KEY_BUFFER = 2048; +/** Fields of the index: a cluster's address count. */ +constexpr uint32_t MAX_CLUSTER_ADDRESSES = 0xFFFF; +/** Lines are added up to this far outside a tile (the 100 m of matching plus the 60 m of the neighbours' vote). */ +constexpr double LINE_BUFFER_M = 160.0; + +uint64_t mix(uint64_t h) { + h ^= h >> 33; + h *= 0xff51afd7ed558ccdULL; + h ^= h >> 33; + h *= 0xc4ceb9fe1a85ec53ULL; + h ^= h >> 33; + return h; +} + +uint64_t hashBytes(uint64_t h, const void* data, size_t size) { + const auto* p = static_cast(data); + for (size_t i = 0; i < size; i++) { + h = (h ^ p[i]) * 0x100000001b3ULL; + } + return h; +} + +uint64_t hashText(uint64_t h, const char* s) { + return s == nullptr ? hashBytes(h, "\1", 1) : hashBytes(h, s, strlen(s) + 1); +} + +uint64_t hashInt(uint64_t h, int64_t v) { + return hashBytes(h, &v, sizeof(v)); +} + +bool sameText(const char* a, const char* b) { + if (a == nullptr || b == nullptr) { + return a == b; + } + return strcmp(a, b) == 0; +} + +int32_t floorDiv(int32_t value, int32_t by) { + return value >= 0 ? value / by : -((-value + by - 1) / by); +} + +bool startsWith(const char* s, const char* prefix) { + return strncmp(s, prefix, strlen(prefix)) == 0; +} + +/** The characters of a UTF-8 word (data.py len() of a Python string). */ +size_t characterCount(const char* s, size_t bytes) { + size_t n = 0; + for (size_t i = 0; i < bytes; i++) { + if ((static_cast(s[i]) & 0xC0) != 0x80) { + n++; + } + } + return n; +} + +uint32_t tileKey(int x, int y) { + return static_cast(x) << 16 | static_cast(y); +} + +} // namespace + +double metresPerUnitAtTileY(double tileY) { + const double n = PI_D * (1.0 - 2.0 * tileY / 16384.0); + const double lat = std::atan(std::sinh(n)); + return 40075016.686 * std::cos(lat) / (16384.0 * 4096.0); +} + +// ------------------------------------------------------------------------------------------------ HashIndex + +void HashIndex::clear() { + full = false; + values.clear(); + hashes.clear(); + used = 0; +} + +void HashIndex::release() { + full = false; + values.release(); + hashes.release(); + used = 0; +} + +void HashIndex::put(uint64_t hash, uint32_t value) { + const size_t mask = values.size() - 1; + size_t i = static_cast(hash) & mask; + while (values[i] != NONE) { + i = (i + 1) & mask; + } + values[i] = value; + hashes[i] = hash; +} + +void HashIndex::insert(uint64_t hash, uint32_t value) { + if (full) { + return; + } + if ((used + 1) * 2 > values.size()) { + // Both new arrays first: without memory for both the old table stays as it was (find() keeps working on it) + // and the index is full, which failed() reports + const size_t capacity = values.size() < 1024 ? 1024 : values.size() * 2; + GrowArray grownValues; + GrowArray grownHashes; + if (!grownValues.resize(capacity) || !grownHashes.resize(capacity)) { + full = true; + return; + } + for (size_t i = 0; i < capacity; i++) { + grownValues[i] = NONE; + } + grownValues.swap(values); + grownHashes.swap(hashes); + // The old table is in the grown* arrays now + for (size_t i = 0; i < grownValues.size(); i++) { + if (grownValues[i] != NONE) { + put(grownHashes[i], grownValues[i]); + } + } + } + put(hash, value); + used++; +} + +// ------------------------------------------------------------------------------------------------ builder + +SearchRegionBuilder::SearchRegionBuilder() = default; +SearchRegionBuilder::~SearchRegionBuilder() = default; + +bool SearchRegionBuilder::fail(const char* message) { + snprintf(errorText, sizeof(errorText), "%s", message); + return false; +} + +bool SearchRegionBuilder::anyFailed() const { + return text.failed() || points.failed() || lines.failed() || linePoints.failed() || houses.failed() || labelIndex.failed() || + nameIndex.failed() || pointIndex.failed() || lineIndex.failed() || houseIndex.failed() || tileIndex.failed() || labelOffsets.failed() || + nameOffsets.failed() || clusters.failed() || streets.failed() || addressRecs.failed() || keyPool.failed() || keyOffsets.failed() || + pointKeys.failed() || streetKeys.failed() || pointInputs.failed() || streetInputs.failed() || clusterInputs.failed() || + addressInputs.failed() || keyPointers.failed() || pseudos.failed() || settlements.failed(); +} + +uint32_t SearchRegionBuilder::copyText(const char* s) { + if (s == nullptr) { + return EXTRACT_NONE; + } + const auto offset = static_cast(text.size()); + const size_t length = strlen(s); + // s may point into text (a settlement's name): grow first, so the pushes below never move it + const bool inside = s >= text.begin() && s < text.begin() + text.size(); + const size_t from = inside ? static_cast(s - text.begin()) : 0; + if (!text.reserve(text.size() + length + 1)) { + text.push(0); // marks the failure (failed()) + return offset; + } + const char* source = inside ? text.begin() + from : s; + for (size_t i = 0; i <= length; i++) { + text.push(source[i]); + } + return offset; +} + +uint32_t SearchRegionBuilder::internLabel(const char* label) { + const uint64_t hash = mix(hashText(1469598103934665603ULL, label)); + const uint32_t found = labelIndex.find(hash, [&](uint32_t id) { return strcmp(&text[labelOffsets[id]], label) == 0; }); + if (found != NONE) { + return found; + } + const auto id = static_cast(labelOffsets.size()); + labelOffsets.push(copyText(label)); + labelIndex.insert(hash, id); + return id; +} + +uint32_t SearchRegionBuilder::internName(const char* name) { + const uint64_t hash = mix(hashText(1469598103934665603ULL, name)); + const uint32_t found = nameIndex.find(hash, [&](uint32_t id) { return strcmp(&text[nameOffsets[id]], name) == 0; }); + if (found != NONE) { + return found; + } + const auto id = static_cast(nameOffsets.size()); + nameOffsets.push(copyText(name)); + nameIndex.insert(hash, id); + return id; +} + +bool SearchRegionBuilder::mergeTile(const ExtractedTile& tile, uint32_t tileNumber) { + for (size_t i = 0; i < tile.points.size(); i++) { + const ExtractPoint& p = tile.points[i]; + const char* name = tile.str(p.name); + const uint32_t labelId = internLabel(tile.str(p.label)); + const uint64_t hash = mix(hashInt(hashInt(hashText(hashInt(1469598103934665603ULL, labelId), name), p.x), p.y)); + const uint32_t found = pointIndex.find(hash, [&](uint32_t c) { + const Pt& q = points[c]; + return q.label == labelId && q.x == p.x && q.y == p.y && sameText(textAt(q.name), name); + }); + if (found != NONE) { + continue; + } + Pt q = {}; + q.label = labelId; + q.name = copyText(name); + q.latin = copyText(tile.str(p.latin)); + q.en = copyText(tile.str(p.en)); + q.x = p.x; + q.y = p.y; + q.settlement = p.settlement; + q.useEn = p.useEn; + pointIndex.insert(hash, static_cast(points.size())); + points.push(q); + } + for (size_t i = 0; i < tile.lines.size(); i++) { + const ExtractLine& l = tile.lines[i]; + const uint32_t nameId = internName(tile.str(l.name)); + const int32_t* xy = &tile.linePoints[2 * static_cast(l.first)]; + const uint64_t hash = mix(hashBytes(hashInt(1469598103934665603ULL, nameId), xy, 2 * sizeof(int32_t) * l.count)); + const uint32_t found = lineIndex.find(hash, [&](uint32_t c) { + const Ln& q = lines[c]; + return q.nameId == nameId && q.count == l.count && memcmp(&linePoints[2 * static_cast(q.first)], xy, 2 * sizeof(int32_t) * l.count) == 0; + }); + if (found != NONE) { + continue; + } + Ln q = {}; + q.nameId = nameId; + q.latin = copyText(tile.str(l.latin)); + q.first = static_cast(linePoints.size() / 2); + q.count = l.count; + q.cluster = NONE; + q.pathClass = l.pathClass; + q.minX = q.maxX = xy[0]; + q.minY = q.maxY = xy[1]; + for (uint32_t k = 0; k < l.count; k++) { + q.minX = std::min(q.minX, xy[2 * k]); + q.maxX = std::max(q.maxX, xy[2 * k]); + q.minY = std::min(q.minY, xy[2 * k + 1]); + q.maxY = std::max(q.maxY, xy[2 * k + 1]); + linePoints.push(xy[2 * k]); + linePoints.push(xy[2 * k + 1]); + } + lineIndex.insert(hash, static_cast(lines.size())); + lines.push(q); + } + for (size_t i = 0; i < tile.houses.size(); i++) { + const ExtractHouse& h = tile.houses[i]; + const char* number = tile.str(h.text); + const uint64_t hash = mix(hashInt(hashInt(hashText(1469598103934665603ULL, number), h.x), h.y)); + const uint32_t found = houseIndex.find(hash, [&](uint32_t c) { + const Hs& q = houses[c]; + return q.x == h.x && q.y == h.y && strcmp(&text[q.text], number) == 0; + }); + if (found != NONE) { + // data.py: the copy inside its own tile wins + if (h.inside != 0 && houses[found].inside == 0) { + houses[found].inside = 1; + houses[found].tile = tileNumber; + houses[found].local = static_cast(i); + } + continue; + } + Hs q = {}; + q.text = copyText(number); + q.x = h.x; + q.y = h.y; + q.tile = tileNumber; + q.local = static_cast(i); + q.inside = h.inside; + q.cluster = NONE; + q.village = NONE; + houseIndex.insert(hash, static_cast(houses.size())); + houses.push(q); + } + return !anyFailed(); +} + +bool SearchRegionBuilder::addTile(MvtDecoder& decoder, ByteInput* input, int tileX, int tileY) { + if (!extractor.extract(decoder, input, tileX, tileY, tileBuffer)) { + return fail("cannot read a tile"); + } + const uint64_t hash = mix(tileKey(tileX, tileY)); + const auto tileNumber = static_cast(tiles.size()); + tileIndex.insert(hash, tileNumber); + tiles.push({ tileX, tileY }); + statistics.refOnlyLines += tileBuffer.refOnlyLines; + statistics.junkPoisSkipped += tileBuffer.junkPois; + return mergeTile(tileBuffer, tileNumber) || fail("no memory"); +} + +HouseView SearchRegionBuilder::house(size_t index) const { + const Hs& h = houses[index]; + HouseView v = {}; + v.text = &text[h.text]; + v.x = h.x; + v.y = h.y; + v.tier = static_cast(h.tier); + v.streetM = h.streetM; + v.villageStyle = h.villageStyle != 0; + if (h.cluster != NONE) { + v.street = textAt(nameOffsets[streets[clusters[h.cluster].street].nameId]); + } + if (h.village != NONE) { + v.village = textAt(nameOffsets[settlements[h.village].nameId]); + } + return v; +} + +// ------------------------------------------------------------------------------------------------ clusters + +namespace { + +struct CellEntry { + uint32_t nameId; + int32_t cx, cy; + uint32_t line; +}; + +struct LineRef { + uint32_t nameId; + uint32_t root; + uint32_t line; +}; + +struct Group { + uint32_t nameId; + uint32_t begin; + uint32_t end; + int32_t minX, minY; +}; + +} // namespace + +bool SearchRegionBuilder::makeClusters(double metresPerUnit) { + const size_t count = lines.size(); + const auto cell = std::max(1, static_cast(150.0 / metresPerUnit)); + GrowArray parent; + parent.resize(count); + for (size_t i = 0; i < count; i++) { + parent[i] = static_cast(i); + } + auto find = [&](uint32_t x) { + while (parent[x] != x) { + parent[x] = parent[parent[x]]; + x = parent[x]; + } + return x; + }; + HashIndex cells; + GrowArray entries; + auto cellHash = [](uint32_t nameId, int32_t cx, int32_t cy) { + return mix((static_cast(nameId) * 0x9E3779B97F4A7C15ULL) ^ (static_cast(static_cast(cx)) << 32 | static_cast(cy))); + }; + for (size_t i = 0; i < count; i++) { + const Ln& line = lines[i]; + int32_t previousX = INT32_MIN; + int32_t previousY = INT32_MIN; + for (uint32_t k = 0; k < line.count; k++) { + const int32_t cx = floorDiv(linePoints[2 * (static_cast(line.first) + k)], cell); + const int32_t cy = floorDiv(linePoints[2 * (static_cast(line.first) + k) + 1], cell); + if (cx == previousX && cy == previousY) { + continue; + } + previousX = cx; + previousY = cy; + for (int dx = -1; dx <= 1; dx++) { + for (int dy = -1; dy <= 1; dy++) { + const uint32_t found = cells.find(cellHash(line.nameId, cx + dx, cy + dy), [&](uint32_t e) { + return entries[e].nameId == line.nameId && entries[e].cx == cx + dx && entries[e].cy == cy + dy; + }); + if (found != NONE) { + const uint32_t a = find(static_cast(i)); + const uint32_t b = find(entries[found].line); + if (a != b) { + parent[a] = b; + } + } + } + } + const uint64_t hash = cellHash(line.nameId, cx, cy); + if (cells.find(hash, [&](uint32_t e) { return entries[e].nameId == line.nameId && entries[e].cx == cx && entries[e].cy == cy; }) == NONE) { + cells.insert(hash, static_cast(entries.size())); + entries.push({ line.nameId, cx, cy, static_cast(i) }); + } + } + } + cells.release(); + entries.release(); + GrowArray refs; + refs.reserve(count); + for (size_t i = 0; i < count; i++) { + refs.push({ lines[i].nameId, find(static_cast(i)), static_cast(i) }); + } + std::sort(refs.begin(), refs.end(), [](const LineRef& a, const LineRef& b) { + if (a.nameId != b.nameId) return a.nameId < b.nameId; + if (a.root != b.root) return a.root < b.root; + return a.line < b.line; + }); + GrowArray groups; + for (size_t i = 0; i < refs.size();) { + size_t j = i; + Group g = { refs[i].nameId, static_cast(i), 0, INT32_MAX, INT32_MAX }; + while (j < refs.size() && refs[j].nameId == refs[i].nameId && refs[j].root == refs[i].root) { + const Ln& line = lines[refs[j].line]; + const int32_t fx = linePoints[2 * static_cast(line.first)]; + const int32_t fy = linePoints[2 * static_cast(line.first) + 1]; + if (fx < g.minX || (fx == g.minX && fy < g.minY)) { + g.minX = fx; + g.minY = fy; + } + j++; + } + g.end = static_cast(j); + groups.push(g); + i = j; + } + // Streets in the byte order of their names, clusters of a street by their smallest first vertex + std::sort(groups.begin(), groups.end(), [&](const Group& a, const Group& b) { + if (a.nameId != b.nameId) return strcmp(&text[nameOffsets[a.nameId]], &text[nameOffsets[b.nameId]]) < 0; + if (a.minX != b.minX) return a.minX < b.minX; + if (a.minY != b.minY) return a.minY < b.minY; + return a.begin < b.begin; + }); + clusters.clear(); + streets.clear(); + for (const Group& g : groups) { + if (streets.size() == 0 || streets[streets.size() - 1].nameId != g.nameId) { + Street s = {}; + s.nameId = g.nameId; + s.latin = EXTRACT_NONE; + s.en = EXTRACT_NONE; + s.firstCluster = static_cast(clusters.size()); + streets.push(s); + } + Street& street = streets[streets.size() - 1]; + Cluster c = {}; + c.street = static_cast(streets.size() - 1); + c.minX = c.minY = INT32_MAX; + c.maxX = c.maxY = INT32_MIN; + double sumX = 0; + double sumY = 0; + uint64_t total = 0; + for (uint32_t r = g.begin; r < g.end; r++) { + Ln& line = lines[refs[r].line]; + line.cluster = static_cast(clusters.size()); + if (street.latin == EXTRACT_NONE && line.latin != EXTRACT_NONE) { + street.latin = line.latin; + } + for (uint32_t k = 0; k < line.count; k++) { + const int32_t x = linePoints[2 * (static_cast(line.first) + k)]; + const int32_t y = linePoints[2 * (static_cast(line.first) + k) + 1]; + sumX += x; + sumY += y; + total++; + c.minX = std::min(c.minX, x); + c.maxX = std::max(c.maxX, x); + c.minY = std::min(c.minY, y); + c.maxY = std::max(c.maxY, y); + } + } + const double cx = sumX / static_cast(total); + const double cy = sumY / static_cast(total); + double bestD = 0; + bool have = false; + for (uint32_t r = g.begin; r < g.end; r++) { + const Ln& line = lines[refs[r].line]; + for (uint32_t k = 0; k < line.count; k++) { + const int32_t x = linePoints[2 * (static_cast(line.first) + k)]; + const int32_t y = linePoints[2 * (static_cast(line.first) + k) + 1]; + const double d = (x - cx) * (x - cx) + (y - cy) * (y - cy); + if (!have || d < bestD || (d == bestD && (x < c.repX || (x == c.repX && y < c.repY)))) { + have = true; + bestD = d; + c.repX = x; + c.repY = y; + } + } + } + street.clusterCount++; + clusters.push(c); + } + return !(clusters.failed() || streets.failed() || parent.failed() || refs.failed() || groups.failed()); +} + +// ------------------------------------------------------------------------------------------------ matching + +bool SearchRegionBuilder::matchHouses(const RegionConfig& config, double metresPerUnit, bool villageMode, bool slashSecond) { + const size_t tileCount = tiles.size(); + const size_t houseCount = houses.size(); + // Own houses per tile + GrowArray houseStart; + GrowArray houseOrder; + houseStart.resize(tileCount + 1); + for (size_t i = 0; i < houseCount; i++) { + houseStart[houses[i].tile + 1]++; + } + for (size_t t = 0; t < tileCount; t++) { + houseStart[t + 1] += houseStart[t]; + } + houseOrder.resize(houseCount); + { + GrowArray fill; + fill.resize(tileCount); + for (size_t i = 0; i < houseCount; i++) { + const uint32_t t = houses[i].tile; + houseOrder[houseStart[t] + fill[t]++] = static_cast(i); + } + } + // A tile's houses in the order of its own extract (the order a streaming build sees them in) + for (size_t t = 0; t < tileCount; t++) { + std::sort(&houseOrder[houseStart[t]], &houseOrder[houseStart[t]] + (houseStart[t + 1] - houseStart[t]), + [&](uint32_t a, uint32_t b) { return houses[a].local < houses[b].local; }); + } + // Lines per tile, with the buffer + const auto lineBuffer = static_cast(LINE_BUFFER_M / metresPerUnit) + 1; + GrowArray lineStart; + GrowArray lineOrder; + lineStart.resize(tileCount + 1); + auto forTiles = [&](const Ln& line, auto visit) { + const int32_t x0 = floorDiv(line.minX - lineBuffer, TILE_UNITS); + const int32_t x1 = floorDiv(line.maxX + lineBuffer, TILE_UNITS); + const int32_t y0 = floorDiv(line.minY - lineBuffer, TILE_UNITS); + const int32_t y1 = floorDiv(line.maxY + lineBuffer, TILE_UNITS); + for (int32_t tx = x0; tx <= x1; tx++) { + for (int32_t ty = y0; ty <= y1; ty++) { + const uint32_t t = tileIndex.find(mix(tileKey(tx, ty)), [&](uint32_t c) { return tiles[c].x == tx && tiles[c].y == ty; }); + if (t != NONE) { + visit(t); + } + } + } + }; + for (size_t i = 0; i < lines.size(); i++) { + if (lines[i].pathClass != 0 && !config.matchPathLines) { + continue; + } + forTiles(lines[i], [&](uint32_t t) { lineStart[t + 1]++; }); + } + for (size_t t = 0; t < tileCount; t++) { + lineStart[t + 1] += lineStart[t]; + } + lineOrder.resize(lineStart[tileCount]); + { + GrowArray fill; + fill.resize(tileCount); + for (size_t i = 0; i < lines.size(); i++) { + if (lines[i].pathClass != 0 && !config.matchPathLines) { + continue; + } + forTiles(lines[i], [&](uint32_t t) { lineOrder[lineStart[t] + fill[t]++] = static_cast(i); }); + } + } + settlementPseudo.resize(settlements.size()); + for (size_t i = 0; i < settlementPseudo.size(); i++) { + settlementPseudo[i] = NONE; + } + nameStreet.resize(nameOffsets.size()); + for (size_t i = 0; i < nameStreet.size(); i++) { + nameStreet[i] = NONE; + } + for (size_t i = 0; i < streets.size(); i++) { + nameStreet[streets[i].nameId] = static_cast(i); + } + + GrowArray contextOf; + for (size_t t = 0; t < tileCount; t++) { + if (houseStart[t + 1] == houseStart[t]) { + continue; + } + const double mpu = metresPerUnitAtTileY(tiles[t].y + 0.5); + const int32_t rx0 = tiles[t].x * TILE_UNITS; + const int32_t ry0 = tiles[t].y * TILE_UNITS; + const int32_t rx1 = rx0 + TILE_UNITS - 1; + const int32_t ry1 = ry0 + TILE_UNITS - 1; + matcher.begin(mpu); + matcher.clipToTile(tiles[t].x, tiles[t].y); + for (uint32_t k = lineStart[t]; k < lineStart[t + 1]; k++) { + const Ln& line = lines[lineOrder[k]]; + matcher.addLine(line.cluster, &linePoints[2 * static_cast(line.first)], line.count); + } + if (villageMode) { + const auto placeBuffer = static_cast(AddressMatcher::PLACE_M / mpu) + 1; + for (size_t s = 0; s < settlements.size(); s++) { + const Pt& p = points[settlements[s].point]; + if (p.x >= rx0 - placeBuffer && p.x <= rx1 + placeBuffer && p.y >= ry0 - placeBuffer && p.y <= ry1 + placeBuffer) { + matcher.addPlace(static_cast(s), static_cast(settlements[s].placeClass), p.x, p.y); + } + } + } + auto streetNumberOf = [&](const char* number) { return houseVoteNumber(number, slashSecond && config.voteSlashSecond); }; + const uint32_t own = houseStart[t + 1] - houseStart[t]; + for (uint32_t k = houseStart[t]; k < houseStart[t + 1]; k++) { + Hs& h = houses[houseOrder[k]]; + const char* number = &text[h.text]; + const bool villageStyle = villageMode && houseNumberStyle(number).plain; + h.villageStyle = villageStyle ? 1 : 0; + matcher.addHouse(h.x, h.y, streetNumberOf(number), villageStyle); + } + // The neighbours' houses near the tile edge take part in the vote + const auto voteBuffer = static_cast(AddressMatcher::VOTE_M / mpu) + 1; + for (int dx = -1; dx <= 1; dx++) { + for (int dy = -1; dy <= 1; dy++) { + if (dx == 0 && dy == 0) { + continue; + } + const int nx = tiles[t].x + dx; + const int ny = tiles[t].y + dy; + const uint32_t n = tileIndex.find(mix(tileKey(nx, ny)), [&](uint32_t c) { return tiles[c].x == nx && tiles[c].y == ny; }); + if (n == NONE) { + continue; + } + for (uint32_t k = houseStart[n]; k < houseStart[n + 1]; k++) { + const Hs& h = houses[houseOrder[k]]; + if (h.x >= rx0 - voteBuffer && h.x <= rx1 + voteBuffer && h.y >= ry0 - voteBuffer && h.y <= ry1 + voteBuffer) { + matcher.addHouse(h.x, h.y, streetNumberOf(&text[h.text]), false); + } + } + } + } + if (!matcher.run()) { + return fail("no memory for the house matching"); + } + for (uint32_t k = 0; k < own; k++) { + const uint32_t index = houseOrder[houseStart[t] + k]; + Hs& h = houses[index]; + const AddressMatch& m = matcher.result(k); + h.tier = static_cast(m.tier); + h.cluster = m.tier == AddressTier::None ? NONE : m.street; + h.streetM = m.streetM; + h.village = (h.villageStyle != 0 && m.villages[0] != NONE) ? m.villages[0] : NONE; + if (h.cluster != NONE) { + addressRecs.push({ h.cluster, index, static_cast(m.tier == AddressTier::LowStreet ? ADDRESS_LOW : 0) }); + if (m.tier == AddressTier::Street) { + statistics.pathStreet50++; + } else { + statistics.pathStreet100++; + } + if (h.village != NONE) { + statistics.alsoVillage++; + } + } else if (h.village != NONE) { + statistics.pathVillage++; + } else { + statistics.pathNone++; + } + if (h.village != NONE) { + uint32_t& pseudo = settlementPseudo[h.village]; + if (pseudo == NONE) { + const Settlement& s = settlements[h.village]; + uint32_t street = nameStreet[s.nameId]; + if (street == NONE) { + const Pt& p = points[s.point]; + Street created = {}; + created.nameId = s.nameId; + created.latin = p.latin; + created.en = p.en; + created.useEn = 1; + street = static_cast(streets.size()); + nameStreet[s.nameId] = street; + streets.push(created); + } + pseudo = static_cast(pseudos.size()); + pseudos.push({ street, h.village }); + } + addressRecs.push({ static_cast(clusters.size()) + pseudo, index, ADDRESS_VILLAGE }); + } + } + } + matcher.release(); + statistics.pseudoStreets = static_cast(pseudos.size()); + return !anyFailed(); +} + +// ------------------------------------------------------------------------------------------------ keys + +SearchRegionBuilder::KeyRef SearchRegionBuilder::makeKeys(const char* name, const char* latin, const char* en, bool useEn, Translit profile) { + KeyRef ref = {}; + keyMaker.make(name, latin, en, useEn, profile); + auto copy = [&](const char* key) { + const auto offset = static_cast(keyPool.size()); + for (const char* c = key;; c++) { + keyPool.push(*c); + if (*c == '\0') { + break; + } + } + return offset; + }; + ref.sortKey = copy(keyMaker.sortKey()); + ref.first = static_cast(keyOffsets.size()); + for (uint32_t i = 0; i < keyMaker.count(); i++) { + keyOffsets.push(copy(keyMaker.key(i))); + } + ref.count = keyMaker.count(); + return ref; +} + +// ------------------------------------------------------------------------------------------------ KeyMaker + +void KeyMaker::sortKeyOf(const char* name, Translit profile, char* out, size_t outSize) { + if (!hasNonLatin(name)) { + foldLatin(name, out, outSize); + } else { + transliterate(name, profile, out, outSize); + } +} + +bool KeyMaker::has(const char* key) const { + for (size_t i = 0; i < offsets.size(); i++) { + if (strcmp(&pool[offsets[i]], key) == 0) { + return true; + } + } + return false; +} + +void KeyMaker::add(const char* key) { + offsets.push(static_cast(pool.size())); + for (const char* c = key;; c++) { + pool.push(*c); + if (*c == '\0') { + break; + } + } +} + +bool KeyMaker::make(const char* name, const char* latin, const char* en, bool useEn, Translit profile) { + pool.clear(); + offsets.clear(); + // The sort key first in the pool (offset 0), not one of the keys + sortKeyOf(name, profile, scratch[4], BUFFER); + for (const char* c = scratch[4];; c++) { + pool.push(*c); + if (*c == '\0') { + break; + } + } + // fold.search_keys + int n = 0; + if (!hasNonLatin(name)) { + foldLatin(name, scratch[n++], BUFFER); + if (latin != nullptr && !hasNonLatin(latin)) { + foldLatin(latin, scratch[n++], BUFFER); + } + } else { + transliterate(name, profile, scratch[n++], BUFFER); + foldOriginal(name, scratch[n++], BUFFER); + if (latin != nullptr) { + foldLatin(latin, scratch[n++], BUFFER); + } + } + if (useEn && en != nullptr && !hasNonLatin(en)) { + foldLatin(en, scratch[n++], BUFFER); + } + for (int i = 0; i < n; i++) { + if (scratch[i][0] != '\0' && !has(scratch[i])) { + add(scratch[i]); + } + } + // fold.suffix_keys of each base key: the text from each later word of at least 3 characters + const size_t baseCount = offsets.size(); + for (size_t b = 0; b < baseCount; b++) { + // The pool may move while keys are added: work on a copy + strncpy(scratch[4], &pool[offsets[b]], BUFFER - 1); + scratch[4][BUFFER - 1] = '\0'; + const char* word = scratch[4]; + bool first = true; + while (true) { + const char* space = strchr(word, ' '); + const size_t length = space != nullptr ? static_cast(space - word) : strlen(word); + if (!first && characterCount(word, length) >= 3 && !has(word)) { + add(word); + } + first = false; + if (space == nullptr) { + break; + } + word = space + 1; + } + } + return !pool.failed() && !offsets.failed(); +} + +// ------------------------------------------------------------------------------------------------ assembly + +bool SearchRegionBuilder::assemble(const RegionConfig& config) { + const size_t realClusters = clusters.size(); + const size_t unified = realClusters + pseudos.size(); + // Streets in the byte order of their names (the settlements' own streets were added at the end) + GrowArray streetOrder; + GrowArray streetRank; + for (size_t i = 0; i < streets.size(); i++) { + streetOrder.push(static_cast(i)); + } + std::sort(streetOrder.begin(), streetOrder.end(), [&](uint32_t a, uint32_t b) { + return strcmp(&text[nameOffsets[streets[a].nameId]], &text[nameOffsets[streets[b].nameId]]) < 0; + }); + streetRank.resize(streets.size()); + for (size_t k = 0; k < streetOrder.size(); k++) { + streetRank[streetOrder[k]] = static_cast(k); + } + // Pseudo clusters in street order, then in the order of their settlements (the points' order) + GrowArray pseudoOrder; + for (size_t i = 0; i < pseudos.size(); i++) { + pseudoOrder.push(static_cast(i)); + } + std::sort(pseudoOrder.begin(), pseudoOrder.end(), [&](uint32_t a, uint32_t b) { + if (pseudos[a].street != pseudos[b].street) return streetRank[pseudos[a].street] < streetRank[pseudos[b].street]; + return pseudos[a].settlement < pseudos[b].settlement; + }); + // Output order of the clusters: per street, the real ones, then the places' + GrowArray finalOf; + finalOf.resize(unified); + GrowArray unifiedAt; + size_t cursor = 0; + for (size_t k = 0; k < streetOrder.size(); k++) { + const uint32_t s = streetOrder[k]; + for (uint32_t c = 0; c < streets[s].clusterCount; c++) { + const uint32_t id = streets[s].firstCluster + c; + finalOf[id] = static_cast(unifiedAt.size()); + unifiedAt.push(id); + } + while (cursor < pseudoOrder.size() && pseudos[pseudoOrder[cursor]].street == s) { + const auto id = static_cast(realClusters + pseudoOrder[cursor]); + finalOf[id] = static_cast(unifiedAt.size()); + unifiedAt.push(id); + cursor++; + } + } + // Addresses per output cluster (counting sort), capped at the field's width + GrowArray start; + start.resize(unified + 1); + for (const AddressRec& r : addressRecs) { + start[finalOf[r.cluster] + 1]++; + } + for (size_t i = 0; i < unified; i++) { + start[i + 1] += start[i]; + } + GrowArray order; + order.resize(addressRecs.size()); + { + GrowArray fill; + fill.resize(unified); + for (size_t i = 0; i < addressRecs.size(); i++) { + const uint32_t f = finalOf[addressRecs[i].cluster]; + order[start[f] + fill[f]++] = static_cast(i); + } + } + addressInputs.clear(); + clusterInputs.clear(); + GrowArray addressFirst; + GrowArray numberOffsets; + for (size_t f = 0; f < unified; f++) { + const uint32_t id = unifiedAt[f]; + SearchClusterInput c = {}; + if (id < realClusters) { + const Cluster& k = clusters[id]; + c.repX = k.repX; + c.repY = k.repY; + c.minX = k.minX; + c.minY = k.minY; + c.maxX = k.maxX; + c.maxY = k.maxY; + } else { + const Pt& p = points[settlements[pseudos[id - realClusters].settlement].point]; + c.repX = c.minX = c.maxX = p.x; + c.repY = c.minY = c.maxY = p.y; + } + addressFirst.push(static_cast(addressInputs.size())); + uint32_t taken = 0; + for (uint32_t k = start[f]; k < start[f + 1]; k++) { + if (taken >= MAX_CLUSTER_ADDRESSES) { + statistics.droppedAddresses++; + continue; + } + const AddressRec& r = addressRecs[order[k]]; + const Hs& h = houses[r.house]; + // The text pointer is set when the pool is final (below): keep the offset for now + addressInputs.push({ nullptr, h.x, h.y, r.flags }); + numberOffsets.push(h.text); + taken++; + } + c.addressCount = taken; + clusterInputs.push(c); + } + for (size_t f = 0; f < unified; f++) { + clusterInputs[f].addresses = clusterInputs[f].addressCount > 0 ? &addressInputs[addressFirst[f]] : nullptr; + } + statistics.addresses = static_cast(addressInputs.size()); + // Number texts: the offsets become pointers (the pool does not grow any more) + for (size_t i = 0; i < addressInputs.size(); i++) { + addressInputs[i].number = &text[numberOffsets[i]]; + } + + // Streets and their keys + streetInputs.clear(); + streetKeys.clear(); + keyPool.clear(); + keyOffsets.clear(); + uint32_t clusterAt = 0; + for (size_t k = 0; k < streetOrder.size(); k++) { + const uint32_t s = streetOrder[k]; + const Street& street = streets[s]; + const KeyRef keys = makeKeys(&text[nameOffsets[street.nameId]], textAt(street.latin), textAt(street.en), street.useEn != 0, config.profile); + streetKeys.push(keys); + SearchStreetInput in = {}; + in.name = &text[nameOffsets[street.nameId]]; + in.keyCount = keys.count; + uint32_t total = street.clusterCount; + for (size_t p = 0; p < pseudoOrder.size(); p++) { + if (pseudos[pseudoOrder[p]].street == s) { + total++; + } + } + in.clusterCount = total; + in.clusters = total > 0 ? &clusterInputs[clusterAt] : nullptr; + clusterAt += total; + streetInputs.push(in); + } + // Points and their keys + pointInputs.clear(); + pointKeys.clear(); + keyShare.clear(); + for (size_t i = 0; i < points.size(); i++) { + const Pt& p = points[i]; + KeyRef keys = {}; + keys.sortKey = NONE; + if (p.name != EXTRACT_NONE) { + const char* name = &text[p.name]; + const uint64_t hash = mix(hashInt(hashText(hashText(hashText(1469598103934665603ULL, name), textAt(p.latin)), textAt(p.en)), p.useEn)); + const uint32_t found = keyShare.find(hash, [&](uint32_t c) { + const Pt& q = points[c]; + return q.useEn == p.useEn && sameText(textAt(q.name), name) && sameText(textAt(q.latin), textAt(p.latin)) && sameText(textAt(q.en), textAt(p.en)); + }); + if (found != NONE) { + keys = pointKeys[found]; + } else { + keys = makeKeys(name, textAt(p.latin), textAt(p.en), p.useEn != 0, config.profile); + keyShare.insert(hash, static_cast(i)); + } + } + pointKeys.push(keys); + } + // Category names that are not ASCII get a key of their own + GrowArray categoryNameOffsets; + GrowArray categoryKeyOffsets; + for (size_t i = 0; i < labelOffsets.size(); i++) { + const char* label = &text[labelOffsets[i]]; + if (!startsWith(label, "poi/")) { + continue; + } + const char* part = label + 4; + for (int k = 0; k < 2 && part != nullptr; k++) { + const char* slash = strchr(part, '/'); + const size_t length = slash != nullptr ? static_cast(slash - part) : strlen(part); + char display[200]; + const size_t n = std::min(length, sizeof(display) - 1); + bool ascii = true; + for (size_t c = 0; c < n; c++) { + display[c] = part[c] == '_' ? ' ' : part[c]; + ascii = ascii && static_cast(part[c]) < 0x80; + } + display[n] = '\0'; + part = slash != nullptr ? slash + 1 : nullptr; + if (ascii) { + continue; + } + bool known = false; + for (size_t c = 0; c < categoryNameOffsets.size(); c++) { + known = known || strcmp(&keyPool[categoryNameOffsets[c]], display) == 0; + } + if (known) { + continue; + } + categoryNameOffsets.push(static_cast(keyPool.size())); + for (const char* c = display;; c++) { + keyPool.push(*c); + if (*c == '\0') { + break; + } + } + KeyMaker::sortKeyOf(display, config.profile, categoryScratch, sizeof(categoryScratch)); + categoryKeyOffsets.push(static_cast(keyPool.size())); + for (const char* c = categoryScratch;; c++) { + keyPool.push(*c); + if (*c == '\0') { + break; + } + } + } + } + if (anyFailed()) { + return false; + } + // The pools are final: pointers + keyPointers.clear(); + for (size_t i = 0; i < keyOffsets.size(); i++) { + keyPointers.push(&keyPool[keyOffsets[i]]); + } + for (size_t i = 0; i < points.size(); i++) { + const Pt& p = points[i]; + SearchPointInput in = {}; + in.label = &text[labelOffsets[p.label]]; + in.x = p.x; + in.y = p.y; + if (p.name != EXTRACT_NONE) { + in.name = &text[p.name]; + in.sortKey = &keyPool[pointKeys[i].sortKey]; + in.keys = pointKeys[i].count > 0 ? &keyPointers[pointKeys[i].first] : nullptr; + in.keyCount = pointKeys[i].count; + } + pointInputs.push(in); + } + for (size_t s = 0; s < streets.size(); s++) { + streetInputs[s].sortKey = &keyPool[streetKeys[s].sortKey]; + streetInputs[s].keys = streetKeys[s].count > 0 ? &keyPointers[streetKeys[s].first] : nullptr; + } + categoryNames.clear(); + categoryKeys.clear(); + for (size_t i = 0; i < categoryNameOffsets.size(); i++) { + categoryNames.push(&keyPool[categoryNameOffsets[i]]); + categoryKeys.push(&keyPool[categoryKeyOffsets[i]]); + } + searchInput = {}; + searchInput.points = pointInputs.begin(); + searchInput.pointCount = static_cast(pointInputs.size()); + searchInput.streets = streetInputs.begin(); + searchInput.streetCount = static_cast(streetInputs.size()); + searchInput.slashSecond = statistics.slashSecond; + searchInput.categoryNames = categoryNames.begin(); + searchInput.categoryKeys = categoryKeys.begin(); + searchInput.categoryOverrides = static_cast(categoryNames.size()); + return !anyFailed(); +} + +bool SearchRegionBuilder::buildSearchInput(const RegionConfig& config) { + errorText[0] = '\0'; + if (tiles.size() == 0) { + return fail("no tiles"); + } + int minY = tiles[0].y; + int maxY = tiles[0].y; + for (const TileRef& t : tiles) { + minY = std::min(minY, t.y); + maxY = std::max(maxY, t.y); + } + const double mpu = config.metresPerUnit > 0 ? config.metresPerUnit : metresPerUnitAtTileY((minY + maxY + 1) / 2.0); + RegionStats& st = statistics; + st.tiles = static_cast(tiles.size()); + st.lines = static_cast(lines.size()); + st.houses = static_cast(houses.size()); + for (const Pt& p : points) { + const char* label = &text[labelOffsets[p.label]]; + if (startsWith(label, "place/")) { + st.places++; + } else if (startsWith(label, "poi/")) { + if (p.name != EXTRACT_NONE) { + st.namedPois++; + } else { + st.unnamedPois++; + } + } else { + st.otherNamed++; + } + } + // Settlements for the village rule + settlements.clear(); + for (size_t i = 0; i < points.size(); i++) { + if (points[i].settlement != EXTRACT_NOT_SETTLEMENT && points[i].name != EXTRACT_NONE) { + settlements.push({ static_cast(i), internName(&text[points[i].name]), points[i].settlement }); + } + } + st.settlements = static_cast(settlements.size()); + // The region's number style + for (const Hs& h : houses) { + const HouseNumberStyle kind = houseNumberStyle(&text[h.text]); + if (!kind.parsed) { + st.textNumbers++; + } + if (kind.plain) { + st.plainNumbers++; + } + if (kind.slash) { + st.slashNumbers++; + if (kind.secondSmaller) { + st.slashSecondSmaller++; + } + } + } + st.plainShare = houses.size() > 0 ? static_cast(st.plainNumbers) / static_cast(houses.size()) : 0.0; + st.villageMode = st.plainShare >= config.villageShare && houses.size() > 0; + st.slashShare = st.slashNumbers > 0 ? static_cast(st.slashSecondSmaller) / static_cast(st.slashNumbers) : 0.0; + st.slashSecond = st.slashShare >= config.slashSecondShare && st.slashNumbers > 0; + if (anyFailed() || !makeClusters(mpu)) { + return fail("no memory for the street clusters"); + } + st.streetNames = static_cast(streets.size()); + st.clusters = static_cast(clusters.size()); + if (!matchHouses(config, mpu, st.villageMode, st.slashSecond)) { + return errorText[0] != '\0' ? false : fail("no memory for the house matching"); + } + if (!assemble(config)) { + return fail("no memory for the index input"); + } + return true; +} + +// ------------------------------------------------------------------------------------------------ number helpers + +HouseNumberStyle houseNumberStyle(const char* text) { + HouseNumberStyle k = {}; + k.parsed = parseHouseNumber(text, strlen(text), k.number); + if (k.parsed) { + k.plain = k.number.aux == 0 && k.number.suffix == 0 && k.number.flags == 0; + k.slash = (k.number.flags & HOUSE_SLASH) != 0 && k.number.aux != 0; + k.secondSmaller = k.slash && k.number.aux < k.number.number; + } + return k; +} + +int32_t houseVoteNumber(const char* text, bool slashSecond) { + const HouseNumberStyle kind = houseNumberStyle(text); + if (kind.parsed) { + if (kind.slash && slashSecond) { + return static_cast(kind.number.aux); + } + return static_cast(kind.number.number); + } + int32_t value = 0; + int digits = 0; + while (text[digits] >= '0' && text[digits] <= '9' && digits < 9) { + value = value * 10 + (text[digits] - '0'); + digits++; + } + return digits > 0 ? value : -1; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchRegion.h b/Apps/Maps/main/Source/search/SearchRegion.h new file mode 100644 index 0000000..74672b6 --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchRegion.h @@ -0,0 +1,324 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "../route/GrowArray.h" +#include "AddressMatch.h" +#include "QueryParse.h" +#include "SearchExtract.h" +#include "SearchIndex.h" +#include "TextFold.h" + +#include +#include + +namespace maps { + +/** Metres per z14 unit at a tile row (fractional, e.g. the middle of a region's rows). */ +double metresPerUnitAtTileY(double tileY); + +/** An open-addressing table from a 64-bit hash to a caller's index; the caller's callback decides equality. */ +class HashIndex { +public: + static constexpr uint32_t NONE = UINT32_MAX; + void clear(); + void release(); + /** The value stored under the hash for which eq(value) is true, or NONE. */ + template + uint32_t find(uint64_t hash, Eq eq) const { + if (values.size() == 0) { + return NONE; + } + const size_t mask = values.size() - 1; + for (size_t i = static_cast(hash) & mask; values[i] != NONE; i = (i + 1) & mask) { + if (hashes[i] == hash && eq(values[i])) { + return values[i]; + } + } + return NONE; + } + /** Adds a value (the caller has checked it is not there). */ + void insert(uint64_t hash, uint32_t value); + /** A value could not be added (no memory to grow): the index misses it. */ + bool failed() const { return full || values.failed() || hashes.failed(); } + +private: + void put(uint64_t hash, uint32_t value); + GrowArray values; + GrowArray hashes; + size_t used = 0; + bool full = false; +}; + +/** + * The search keys of a name as the study makes them (fold.search_keys and suffix_keys): the folded name (transliterated + * and original-folded for Greek and Cyrillic), name:latin, name_en when useEn, then the text from each later word of at + * least 3 characters. Duplicates are left out; the order is fixed. Also the name's sort key. Shared by the in-memory + * builder and the streaming one, so both make the same keys. + */ +class KeyMaker { +public: + /** Makes the keys of a name; false when out of memory. */ + bool make(const char* name, const char* latin, const char* en, bool useEn, Translit profile); + /** The sort key (name groups are ordered by it). */ + const char* sortKey() const { return &pool[0]; } + uint32_t count() const { return static_cast(offsets.size()); } + const char* key(uint32_t index) const { return &pool[offsets[index]]; } + /** Only the sort key, without the other keys. */ + static void sortKeyOf(const char* name, Translit profile, char* out, size_t outSize); + void release() { + pool.release(); + offsets.release(); + } + +private: + static constexpr size_t BUFFER = 2048; + bool has(const char* key) const; + void add(const char* key); + char scratch[5][BUFFER] = {}; + GrowArray pool; + GrowArray offsets; +}; + +struct RegionConfig { + /** Transliteration of Greek and Cyrillic names (the study's PROFILE map: Ukrainian for kyiv, Serbian for belgrade, else Russian). */ + Translit profile = Translit::Russian; + /** Metres per z14 unit; 0 takes it from the middle row of the region's tiles. */ + double metresPerUnit = 0.0; + /** + * Named lines of class path, track, aerialway, ferry and rail take part in the house matching. The study's counts + * leave them out; with them 88.3 % of Bratislava's numbers get the street of OSM's addr:street instead of 87.2 % + * (host/search/gt_cpp.py, 12 Nov 2026), and the brief asks for named paths and pedestrian streets. + */ + bool matchPathLines = true; + /** For "x/y" numbers of a conscription/street region the vote compares the street number (second part). */ + bool voteSlashSecond = true; + /** The share of plain numbers from which a region is in village mode, and of x/y numbers with a smaller second part for slash_second. */ + double villageShare = 0.30; + double slashSecondShare = 0.80; +}; + +struct RegionStats { + uint32_t tiles = 0; + // What was extracted, deduplicated across tiles (for comparison with the study's counts) + uint32_t places = 0; + uint32_t namedPois = 0; + uint32_t unnamedPois = 0; + uint32_t otherNamed = 0; + uint32_t junkPoisSkipped = 0; + uint32_t streetNames = 0; + uint32_t lines = 0; + uint32_t clusters = 0; + uint32_t refOnlyLines = 0; + uint32_t settlements = 0; + uint32_t houses = 0; + // House number grammar + uint32_t plainNumbers = 0; + double plainShare = 0; + bool villageMode = false; + uint32_t slashNumbers = 0; + uint32_t slashSecondSmaller = 0; + double slashShare = 0; + bool slashSecond = false; + uint32_t textNumbers = 0; + // The path of each house number: street within 50 m, street 50 to 100 m, village only, none + uint32_t pathStreet50 = 0; + uint32_t pathStreet100 = 0; + uint32_t pathVillage = 0; + uint32_t pathNone = 0; + /** Houses that also went under a settlement while having a street (counted in the street paths above). */ + uint32_t alsoVillage = 0; + uint32_t pseudoStreets = 0; + uint32_t droppedAddresses = 0; + uint32_t addresses = 0; +}; + +/** One house number and where it went, for dumps and checks. */ +struct HouseView { + const char* text; + int32_t x; + int32_t y; + /** The matched street name, nullptr for none. */ + const char* street; + AddressTier tier; + float streetM; + /** The settlement the number also went under, nullptr for none. */ + const char* village; + bool villageStyle; +}; + +/** + * The region-level extraction and matching, host sized (everything in RAM): addTile() for every z14 tile, then + * buildSearchInput(). The SearchInput points into the builder, which must outlive its use. The device will stream + * the same steps; this is the reference for them. + * + * What it does (M6 study rules, docs/M6-search-study.md): + * - points are deduplicated by (label, name, x, y), lines by (name, points), house numbers by (text, x, y); + * - street clusters: same-name lines whose vertices are in the same or adjacent 150 m cell (union-find), representative + * vertex nearest to the centroid, bounding box; + * - house numbers: matched per tile with AddressMatcher, with the lines of the neighbour tiles up to 160 m outside the + * tile and the neighbouring houses up to 60 m outside (for the street-run vote; their own results are not used). The + * matcher's street ids are cluster ids, so the match is already the cluster nearest to the house; + * - slash_second (street number is the second part of "x/y") when at least RegionConfig::slashSecondShare of the + * region's "x/y" numbers have a smaller second part (the study's rule); the vote uses that part as the street number; + * - village mode when at least RegionConfig::villageShare of all numbers are plain ("245"); then plain numbers also go + * under their best settlement (matcher's villages[0]) as an address with ADDRESS_VILLAGE under a street record named + * like the place, with a one-point cluster; with a street within 50 m they are under the street too, within 50 to + * 100 m under it with ADDRESS_LOW; + * - keys exactly as the study (search_keys and suffix_keys with TextFold), name_en only for places, water, peaks, parks. + */ +class SearchRegionBuilder { +public: + SearchRegionBuilder(); + ~SearchRegionBuilder(); + SearchRegionBuilder(const SearchRegionBuilder&) = delete; + SearchRegionBuilder& operator=(const SearchRegionBuilder&) = delete; + + /** Extracts one tile and merges it. */ + bool addTile(MvtDecoder& decoder, ByteInput* input, int tileX, int tileY); + /** Clusters, matching, keys. Call once, after the last tile. */ + bool buildSearchInput(const RegionConfig& config); + + const SearchInput& input() const { return searchInput; } + const RegionStats& stats() const { return statistics; } + const char* error() const { return errorText; } + + size_t houseCount() const { return houses.size(); } + HouseView house(size_t index) const; + +private: + struct Pt { + uint32_t label; + uint32_t name; + uint32_t latin; + uint32_t en; + int32_t x, y; + uint8_t settlement; + uint8_t useEn; + }; + struct Ln { + uint32_t nameId; + uint32_t latin; + uint32_t first; + uint32_t count; + uint32_t cluster; + int32_t minX, minY, maxX, maxY; + uint8_t pathClass; + }; + struct Hs { + uint32_t text; + int32_t x, y; + uint32_t tile; + /** Its index in the extract of the tile that owns it: the order of the tile's houses (the streaming build's too). */ + uint32_t local; + uint8_t inside; + uint8_t tier; + uint32_t cluster; + uint32_t village; + float streetM; + uint8_t villageStyle; + }; + struct TileRef { + int x, y; + }; + struct Cluster { + uint32_t street; + int32_t repX, repY, minX, minY, maxX, maxY; + }; + struct Street { + uint32_t nameId; + uint32_t latin; + uint32_t en; + uint32_t firstCluster; + uint32_t clusterCount; + uint8_t useEn; + }; + struct Settlement { + uint32_t point; + uint32_t nameId; + uint8_t placeClass; + }; + struct Pseudo { + uint32_t street; + uint32_t settlement; + }; + struct AddressRec { + uint32_t cluster; + uint32_t house; + uint8_t flags; + }; + struct KeyRef { + uint32_t sortKey; + uint32_t first; + uint32_t count; + }; + + uint32_t copyText(const char* text); + uint32_t internLabel(const char* label); + uint32_t internName(const char* name); + const char* textAt(uint32_t offset) const { return offset == EXTRACT_NONE ? nullptr : &text[offset]; } + bool mergeTile(const ExtractedTile& tile, uint32_t tileIndex); + bool makeClusters(double metresPerUnit); + bool matchHouses(const RegionConfig& config, double metresPerUnit, bool villageMode, bool slashSecond); + bool assemble(const RegionConfig& config); + KeyRef makeKeys(const char* name, const char* latin, const char* en, bool useEn, Translit profile); + bool anyFailed() const; + bool fail(const char* message); + + GrowArray text; + HashIndex labelIndex; + GrowArray labelOffsets; + HashIndex nameIndex; + GrowArray nameOffsets; + HashIndex pointIndex; + HashIndex lineIndex; + HashIndex houseIndex; + HashIndex tileIndex; + GrowArray points; + GrowArray lines; + GrowArray linePoints; + GrowArray houses; + GrowArray tiles; + GrowArray clusters; + GrowArray streets; + GrowArray settlements; + GrowArray pseudos; + GrowArray addressRecs; + GrowArray settlementPseudo; + GrowArray nameStreet; + // Keys + GrowArray keyPool; + GrowArray keyOffsets; + GrowArray pointKeys; + GrowArray streetKeys; + HashIndex keyShare; + // The writer's input + GrowArray pointInputs; + GrowArray streetInputs; + GrowArray clusterInputs; + GrowArray addressInputs; + GrowArray keyPointers; + GrowArray categoryNames; + GrowArray categoryKeys; + SearchInput searchInput = {}; + RegionStats statistics; + ExtractedTile tileBuffer; + TileExtractor extractor; + AddressMatcher matcher; + char errorText[160] = {}; + KeyMaker keyMaker; + char categoryScratch[2048] = {}; +}; + +/** The house number's grammar as the region statistics count it (both builders use it, so their files match). */ +struct HouseNumberStyle { + bool parsed; + bool plain; + bool slash; + bool secondSmaller; + HouseNumber number; +}; +HouseNumberStyle houseNumberStyle(const char* text); +/** The street number a house votes with: the second part of "x/y" when slashSecond, else the first; -1 for none. */ +int32_t houseVoteNumber(const char* text, bool slashSecond); + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchStream.cpp b/Apps/Maps/main/Source/search/SearchStream.cpp new file mode 100644 index 0000000..f3416a1 --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchStream.cpp @@ -0,0 +1,2897 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "SearchStream.h" + +#include "../platform/Platform.h" +#include "QueryParse.h" +#include "TextFold.h" + +#include +#include +#include +#include +#include + +namespace maps { + +namespace { + +constexpr uint32_t NONE = UINT32_MAX; +constexpr int32_t TILE_UNITS = 4096; +/** Lines are matched up to this far outside a tile (the 100 m of matching plus the 60 m of the neighbours' vote). */ +constexpr double LINE_BUFFER_M = 160.0; +constexpr uint32_t IO_BUFFER = 4096; +/** An address record's cluster reference with this bit is a settlement's (village) cluster. */ +constexpr uint32_t PSEUDO = 0x80000000u; +/** Texts in sort records are cut to this many bytes (names are far shorter; a record has at most four texts). */ +constexpr uint32_t MAX_TEXT = 500; +constexpr uint32_t MAX_CLUSTER_ADDRESSES = 0xFFFF; +/** Records between two keepGoing() calls in the long loops. */ +constexpr uint32_t CHECK_EVERY = 2048; + +enum Category : uint8_t { CAT_CATEGORY = 0, CAT_PLACE = 1, CAT_STREET = 2, CAT_OTHER = 3, CAT_POI = 4 }; + +int32_t floorDiv(int32_t value, int32_t by) { + return value >= 0 ? value / by : -((-value + by - 1) / by); +} + +uint64_t mix(uint64_t h) { + h ^= h >> 33; + h *= 0xff51afd7ed558ccdULL; + h ^= h >> 33; + h *= 0xc4ceb9fe1a85ec53ULL; + h ^= h >> 33; + return h; +} + +uint64_t hashBytes(uint64_t h, const void* data, size_t size) { + const auto* p = static_cast(data); + for (size_t i = 0; i < size; i++) { + h = (h ^ p[i]) * 0x100000001b3ULL; + } + return h; +} + +uint64_t hashText(uint64_t h, const char* s) { + return s == nullptr ? hashBytes(h, "\1", 1) : hashBytes(h, s, strlen(s) + 1); +} + +uint64_t hashInt(uint64_t h, int64_t v) { + return hashBytes(h, &v, sizeof(v)); +} + +constexpr uint64_t SEED = 1469598103934665603ULL; + +void put16(uint8_t* p, uint32_t v) { + p[0] = static_cast(v); + p[1] = static_cast(v >> 8); +} + +void put32(uint8_t* p, uint32_t v) { + p[0] = static_cast(v); + p[1] = static_cast(v >> 8); + p[2] = static_cast(v >> 16); + p[3] = static_cast(v >> 24); +} + +uint32_t get16(const uint8_t* p) { + return static_cast(p[0]) | static_cast(p[1]) << 8; +} + +uint32_t get32(const uint8_t* p) { + return static_cast(p[0]) | static_cast(p[1]) << 8 | static_cast(p[2]) << 16 | + static_cast(p[3]) << 24; +} + +int compareInt(int64_t a, int64_t b) { + return a < b ? -1 : (a > b ? 1 : 0); +} + +int compareText(const char* a, const char* b) { + return compareSearchBytes(reinterpret_cast(a), static_cast(strlen(a)), reinterpret_cast(b), + static_cast(strlen(b))); +} + +/** A record being built for a sort or a scratch file. Texts end with a zero byte and are cut to MAX_TEXT bytes. */ +class Rec { +public: + void clear() { used = 0; } + void u8(uint32_t v) { bytes(&v, 1); } + void u16(uint32_t v) { + uint8_t b[2]; + put16(b, v); + bytes(b, 2); + } + void u32(uint32_t v) { + uint8_t b[4]; + put32(b, v); + bytes(b, 4); + } + void i32(int32_t v) { u32(static_cast(v)); } + void text(const char* s) { + size_t n = s != nullptr ? strlen(s) : 0; + if (n > MAX_TEXT) { + n = MAX_TEXT; + while (n > 0 && (static_cast(s[n]) & 0xC0) == 0x80) { + n--; + } + } + bytes(s, n); + const uint8_t zero = 0; + bytes(&zero, 1); + } + void bytes(const void* data, size_t size) { + if (used + size > sizeof(buffer)) { + full = true; + return; + } + if (size > 0) { + memcpy(buffer + used, data, size); + } + used += static_cast(size); + } + const uint8_t* data() const { return buffer; } + uint32_t size() const { return used; } + bool overflowed() const { return full; } + +private: + uint8_t buffer[ExternalSorter::MAX_RECORD] = {}; + uint32_t used = 0; + bool full = false; +}; + +/** Reads the fields of a record in the order they were written. */ +class Cursor { +public: + Cursor(const uint8_t* data, uint32_t size) : p(data), end(data + size) {} + uint32_t u8() { return p < end ? *p++ : 0; } + uint32_t u16() { + const uint32_t v = get16(p); + p += 2; + return v; + } + uint32_t u32() { + const uint32_t v = get32(p); + p += 4; + return v; + } + int32_t i32() { return static_cast(u32()); } + /** A text inside the record (it ends with its zero byte). */ + const char* text() { + const char* s = reinterpret_cast(p); + p += strlen(s) + 1; + return s; + } + +private: + const uint8_t* p; + const uint8_t* end; +}; + +/** A scratch file written in IO_BUFFER pieces. */ +class OutFile { +public: + bool open(ScratchFiles* scratch, const char* path) { + files = scratch; + used = 0; + total = 0; + good = true; + handle = files->create(path); + good = handle >= 0; + return good; + } + bool write(const void* data, uint32_t size) { + const auto* p = static_cast(data); + while (size > 0 && good) { + const uint32_t n = std::min(size, IO_BUFFER - used); + memcpy(buffer + used, p, n); + used += n; + p += n; + size -= n; + total += n; + if (used == IO_BUFFER) { + good = files->write(handle, buffer, used); + used = 0; + } + } + return good; + } + bool write(const Rec& rec) { return write(rec.data(), rec.size()); } + bool close() { + if (handle < 0) { + return good; + } + if (good && used > 0) { + good = files->write(handle, buffer, used); + } + used = 0; + good = files->close(handle) && good; + handle = -1; + return good; + } + uint32_t size() const { return total; } + +private: + ScratchFiles* files = nullptr; + int handle = -1; + uint8_t buffer[IO_BUFFER] = {}; + uint32_t used = 0; + uint32_t total = 0; + bool good = false; +}; + +/** A scratch file read in IO_BUFFER pieces, with seek. */ +class InFile { +public: + bool open(ScratchFiles* scratch, const char* path) { + files = scratch; + filled = 0; + at = 0; + base = 0; + handle = files->openRead(path); + return handle >= 0; + } + void close() { + if (handle >= 0) { + files->close(handle); + handle = -1; + } + } + bool seek(uint32_t offset) { + if (offset >= base && offset <= base + filled) { + at = offset - base; + return true; + } + filled = 0; + at = 0; + base = offset; + return files->seek(handle, offset); + } + /** Exactly size bytes; false at the end or on an error. */ + bool read(void* out, uint32_t size) { + auto* p = static_cast(out); + while (size > 0) { + if (at == filled && !refill()) { + return false; + } + const uint32_t n = std::min(size, filled - at); + memcpy(p, buffer + at, n); + at += n; + p += n; + size -= n; + } + return true; + } + bool u8(uint32_t& v) { + uint8_t b = 0; + const bool ok = read(&b, 1); + v = b; + return ok; + } + bool u32(uint32_t& v) { + uint8_t b[4]; + const bool ok = read(b, 4); + v = get32(b); + return ok; + } + bool i32(int32_t& v) { + uint32_t u = 0; + const bool ok = u32(u); + v = static_cast(u); + return ok; + } + /** A text up to its zero byte (cut to capacity - 1). */ + bool text(char* out, uint32_t capacity) { + uint32_t n = 0; + while (true) { + if (at == filled && !refill()) { + return false; + } + const char c = static_cast(buffer[at++]); + if (c == '\0') { + break; + } + if (n + 1 < capacity) { + out[n++] = c; + } + } + out[n] = '\0'; + return true; + } + bool atEnd() { + return at == filled && !refill(); + } + uint32_t position() const { return base + at; } + +private: + bool refill() { + base += filled; + filled = 0; + at = 0; + const int n = files->read(handle, buffer, IO_BUFFER); + if (n <= 0) { + return false; + } + filled = static_cast(n); + return true; + } + ScratchFiles* files = nullptr; + int handle = -1; + uint8_t buffer[IO_BUFFER] = {}; + uint32_t filled = 0; + uint32_t at = 0; + uint32_t base = 0; +}; + +/** Interned strings: id by text. */ +class Strings { +public: + uint32_t intern(const char* s) { + const uint64_t hash = mix(hashText(SEED, s)); + const uint32_t found = index.find(hash, [&](uint32_t id) { return strcmp(&pool[offsets[id]], s) == 0; }); + if (found != HashIndex::NONE) { + return found; + } + const auto id = static_cast(offsets.size()); + offsets.push(static_cast(pool.size())); + for (const char* c = s;; c++) { + pool.push(*c); + if (*c == '\0') { + break; + } + } + index.insert(hash, id); + return id; + } + uint32_t find(const char* s) const { + const uint64_t hash = mix(hashText(SEED, s)); + return index.find(hash, [&](uint32_t id) { return strcmp(&pool[offsets[id]], s) == 0; }); + } + const char* at(uint32_t id) const { return &pool[offsets[id]]; } + uint32_t count() const { return static_cast(offsets.size()); } + size_t bytes() const { return pool.allocated() + offsets.allocated() + 12 * offsets.size(); } + bool failed() const { return pool.failed() || offsets.failed() || index.failed(); } + void release() { + pool.release(); + offsets.release(); + index.release(); + } + +private: + GrowArray pool; + GrowArray offsets; + HashIndex index; +}; + +struct TileInfo { + int32_t x, y; + uint32_t lineFirst; + uint32_t lineCount; + uint32_t lineOffset; + uint32_t rawHouseOffset; + uint32_t rawHouseCount; + uint32_t houseOffset; + uint32_t houseCount; +}; + +struct LabelInfo { + uint32_t count; + uint32_t minSeq; + uint32_t pair; + uint8_t category; +}; + +struct SettlementInfo { + uint32_t label; + uint32_t nameId; + int32_t x, y; + uint8_t placeClass; + /** Offsets into the extra texts, NONE for none. */ + uint32_t latin; + uint32_t en; + uint32_t pseudo; +}; + +struct Comp { + uint32_t nameId; + int32_t firstX, firstY; + int32_t minX, minY, maxX, maxY; + int64_t sumX, sumY; + uint32_t count; + uint32_t latin; + int32_t repX, repY; + double best; + bool haveRep; + uint32_t addresses; + uint32_t finalIndex; +}; + +/** A street cluster as the index needs it after clustering. */ +struct ClusterInfo { + int32_t repX, repY; + int32_t minX, minY, maxX, maxY; + uint32_t addresses; + uint32_t finalIndex; +}; + +struct StreetInfo { + uint32_t nameId; + uint32_t latin; + uint32_t en; + uint8_t useEn; + uint32_t firstCluster; + uint32_t clusterCount; + uint32_t pseudoCount; + /** Position in name order (the writer's creation order). */ + uint32_t order; + uint32_t sortKey; + uint32_t firstRecord; +}; + +struct PseudoInfo { + uint32_t settlement; + uint32_t street; + uint32_t addresses; + uint32_t finalIndex; +}; + +struct CategoryRun { + uint32_t raw; + uint32_t firstPair; + uint32_t lastPair; + uint32_t count; + bool contiguous; +}; + +struct CategoryNg { + uint32_t name; + uint32_t sortKey; + uint32_t firstPair; + uint32_t pairCount; + uint32_t creation; +}; + +/** A house's match, kept until the tile's addresses are written. */ +struct OwnResult { + uint32_t cluster; + uint32_t village; + uint8_t tier; +}; + +struct ColumnCell { + int32_t cy; + uint32_t line; +}; + +// ---------------------------------------------------------------- sort orders + +/** S_PT: label u32, named u8, name, x, y, seq | latin, en, settlement u8, useEn u8. */ +int comparePoints(const uint8_t* a, uint32_t aSize, const uint8_t* b, uint32_t bSize, void*) { + Cursor x(a, aSize); + Cursor y(b, bSize); + int c = compareInt(x.u32(), y.u32()); + if (c != 0) return c; + c = compareInt(x.u8(), y.u8()); + if (c != 0) return c; + c = strcmp(x.text(), y.text()); + if (c != 0) return c; + c = compareInt(x.i32(), y.i32()); + if (c != 0) return c; + c = compareInt(x.i32(), y.i32()); + if (c != 0) return c; + return compareInt(x.u32(), y.u32()); +} + +/** S_CELL: name u32, cx, cy, line u32 (fixed). */ +int compareCells(const uint8_t* a, uint32_t aSize, const uint8_t* b, uint32_t bSize, void*) { + Cursor x(a, aSize); + Cursor y(b, bSize); + int c = compareInt(x.u32(), y.u32()); + if (c != 0) return c; + c = compareInt(x.i32(), y.i32()); + if (c != 0) return c; + c = compareInt(x.i32(), y.i32()); + if (c != 0) return c; + return compareInt(x.u32(), y.u32()); +} + +/** Fixed records compared by their first n u32/i32 fields: the field kinds per sort below. */ +int compareU32Fields(const uint8_t* a, const uint8_t* b, int fields, const bool* isSigned) { + for (int i = 0; i < fields; i++) { + const uint32_t ua = get32(a + 4 * i); + const uint32_t ub = get32(b + 4 * i); + if (ua != ub) { + if (isSigned[i]) { + return static_cast(ua) < static_cast(ub) ? -1 : 1; + } + return ua < ub ? -1 : 1; + } + } + return 0; +} + +/** S_A1: final u32, seq u32 | flags, x, y, text. */ +int compareAddressSeq(const uint8_t* a, uint32_t, const uint8_t* b, uint32_t, void*) { + static const bool kinds[2] = { false, false }; + return compareU32Fields(a, b, 2, kinds); +} + +/** S_A2: final u32, number u32, aux u32, suffix u32, seq u32 | flags, x, y. */ +int compareAddressNumber(const uint8_t* a, uint32_t, const uint8_t* b, uint32_t, void*) { + static const bool kinds[5] = { false, false, false, false, false }; + return compareU32Fields(a, b, 5, kinds); +} + +/** S_GRP: name, pair u8, seq u32 | ... */ +int compareGroups(const uint8_t* a, uint32_t aSize, const uint8_t* b, uint32_t bSize, void*) { + Cursor x(a, aSize); + Cursor y(b, bSize); + int c = strcmp(x.text(), y.text()); + if (c != 0) return c; + c = compareInt(x.u8(), y.u8()); + if (c != 0) return c; + return compareInt(x.u32(), y.u32()); +} + +/** S_NG: sortKey, pair u8, headSeq u32 | ... (the writer's name group order for points) */ +int compareNameGroups(const uint8_t* a, uint32_t aSize, const uint8_t* b, uint32_t bSize, void*) { + Cursor x(a, aSize); + Cursor y(b, bSize); + int c = compareText(x.text(), y.text()); + if (c != 0) return c; + c = compareInt(x.u8(), y.u8()); + if (c != 0) return c; + return compareInt(x.u32(), y.u32()); +} + +/** S_HS: hash high, hash low, not inside, tile, local (all u32): copies of one house together, the winner first. */ +int compareHouseCopies(const uint8_t* a, uint32_t, const uint8_t* b, uint32_t, void*) { + static const bool kinds[5] = { false, false, false, false, false }; + return compareU32Fields(a, b, 5, kinds); +} + +/** S_DROP: tile, local. */ +int compareFirstTwo(const uint8_t* a, uint32_t, const uint8_t* b, uint32_t, void*) { + static const bool kinds[2] = { false, false }; + return compareU32Fields(a, b, 2, kinds); +} + +/** S_G2R and S_FC: the first u32. */ +int compareFirst(const uint8_t* a, uint32_t, const uint8_t* b, uint32_t, void*) { + static const bool kinds[1] = { false }; + return compareU32Fields(a, b, 1, kinds); +} + +/** S_REC: pair, unnamed, rank, tileY, tileX, x, y | offset (the writer's point record order). */ +int comparePointRecords(const uint8_t* a, uint32_t, const uint8_t* b, uint32_t, void*) { + static const bool kinds[7] = { false, false, false, true, true, true, true }; + return compareU32Fields(a, b, 7, kinds); +} + +/** S_KEY: length u8, key, priority u8, offset u32 (the writer's key order). */ +int compareKeys(const uint8_t* a, uint32_t, const uint8_t* b, uint32_t, void*) { + const uint32_t la = a[0]; + const uint32_t lb = b[0]; + int c = compareSearchBytes(a + 1, la, b + 1, lb); + if (c != 0) return c; + c = compareInt(a[1 + la], b[1 + lb]); + if (c != 0) return c; + return compareInt(get32(a + 2 + la), get32(b + 2 + lb)); +} + +} // namespace + +// ------------------------------------------------------------------------------------------------ number helpers + + +// ------------------------------------------------------------------------------------------------ the builder + +struct SearchStreamBuilder::Impl { + SearchStreamBuilder* owner = nullptr; + ScratchFiles* files = nullptr; + StreamTileSource* source = nullptr; + MvtDecoder* decoder = nullptr; + const StreamBuildOptions* options = nullptr; + RegionConfig config; + StreamBuildStats* st = nullptr; + char dir[128] = {}; + char path[160] = {}; + StreamProgress progress = {}; + + // Tiles + GrowArray tiles; + HashIndex tileIndex; + double regionMpu = 0; + + // Phase 1: the extractor hands over feature by feature (extract holds one feature) + struct Consumer final : ExtractConsumer { + Impl* impl = nullptr; + bool take(const ExtractedTile& feature) override { return impl->takeFeature(feature); } + }; + Consumer consumer; + uint32_t currentTile = 0; + ExtractedTile extract; + TileExtractor extractor; + Strings names; + Strings labelNames; + GrowArray labels; + GrowArray lineName; + GrowArray lineHash; + GrowArray settlements; + GrowArray extraText; + uint32_t pointSeq = 0; + uint32_t pointCount = 0; + uint32_t namedCount = 0; + bool villageMode = false; + bool slashSecond = false; + + // Bounds of everything the index stores (the writer's origin) + bool anyPoint = false; + int32_t minX = 0, minY = 0, maxX = 0, maxY = 0; + + // Pairs and categories + Strings pairTexts; + uint32_t pairTotal = 0; + GrowArray pairStart; + GrowArray pairCount; + GrowArray categoryChars; + GrowArray categories; + uint32_t collapsed = 0; + uint32_t notContiguous = 0; + + // Clusters, streets, villages + GrowArray lineCluster; + GrowArray clusters; + GrowArray streets; + GrowArray nameStreet; + GrowArray pseudos; + GrowArray streetNgOrder; + GrowArray sortKeys; + uint32_t streetRecordTotal = 0; + uint32_t addressTotal = 0; + + // Name groups + uint32_t pointNgCount = 0; + uint32_t pointNgBytes = 0; + uint32_t streetNgBytes = 0; + uint32_t ngTotal = 0; + uint32_t keyCount = 0; + uint32_t overflowCount = 0; + uint32_t overflowBytes = 0; + GrowArray fence; + uint32_t fenceCount = 0; + uint32_t pageCount = 0; + + // Work objects (big: kept here, not on the stack) + AddressMatcher matcher; + KeyMaker keys; + ExternalSorter sortA; + ExternalSorter sortB; + Rec rec; + OutFile outA; + OutFile outB; + InFile inA; + InFile inB; + GrowArray points; + GrowArray tileText; + GrowArray tileTextAt; + GrowArray houseXY; + char textA[MAX_TEXT + 8] = {}; + char textB[MAX_TEXT + 8] = {}; + char textC[MAX_TEXT + 8] = {}; + char keyScratch[2048] = {}; + + // ------------------------------------------------------------ helpers + + const char* file(const char* name) { + snprintf(path, sizeof(path), "%s/%s", dir, name); + return path; + } + + bool fail(const char* message) { + if (owner->errorText[0] == '\0') { + snprintf(owner->errorText, sizeof(owner->errorText), "%s", message); + } + return false; + } + + bool fail(const char* format, uint32_t a, uint32_t b = 0) { + if (owner->errorText[0] == '\0') { + snprintf(owner->errorText, sizeof(owner->errorText), format, a, b); + } + return false; + } + + bool keepGoing() { + if (options->keepGoing != nullptr && !options->keepGoing(progress, options->context)) { + owner->wasCancelled = true; + return fail("cancelled"); + } + return true; + } + + void step() { + progress.done++; + } + + void include(int32_t x, int32_t y) { + if (!anyPoint) { + minX = maxX = x; + minY = maxY = y; + anyPoint = true; + return; + } + minX = std::min(minX, x); + maxX = std::max(maxX, x); + minY = std::min(minY, y); + maxY = std::max(maxY, y); + } + + uint32_t tileAt(int32_t x, int32_t y) const { + const uint64_t hash = mix(static_cast(static_cast(x)) << 32 | static_cast(y)); + return tileIndex.find(hash, [&](uint32_t t) { return tiles[t].x == x && tiles[t].y == y; }); + } + + uint32_t addExtra(const char* s) { + if (s == nullptr) { + return NONE; + } + const auto offset = static_cast(extraText.size()); + for (const char* c = s;; c++) { + extraText.push(*c); + if (*c == '\0') { + break; + } + } + return offset; + } + const char* extra(uint32_t offset) const { return offset == NONE ? nullptr : &extraText[offset]; } + + bool sortBegin(ExternalSorter& sorter, const char* name, ExternalSorter::Compare compare) { + char prefix[160]; + snprintf(prefix, sizeof(prefix), "%s/%s", dir, name); + if (!sorter.begin(files, prefix, options->sortBufferBytes, compare, nullptr)) { + return fail(sorter.errorText()); + } + return true; + } + + bool sortAdd(ExternalSorter& sorter) { + if (rec.overflowed()) { + return fail("a record is too long"); + } + if (!sorter.add(rec.data(), rec.size())) { + return fail(sorter.errorText()); + } + return true; + } + + void noteSort(const ExternalSorter& sorter) { + st->scratchBytes += sorter.bytesWritten(); + st->sortPeakBytes = std::max(st->sortPeakBytes, sorter.peakBytes()); + } + + void noteTables() { + size_t bytes = names.bytes() + labelNames.bytes() + labels.allocated() + lineName.allocated() + lineHash.allocated() + + settlements.allocated() + extraText.allocated() + tiles.allocated() + lineCluster.allocated() + clusters.allocated() + + streets.allocated() + nameStreet.allocated() + pseudos.allocated() + streetNgOrder.allocated() + sortKeys.allocated() + + categories.allocated() + categoryChars.allocated() + + fence.allocated() + pairStart.allocated() + pairCount.allocated(); + st->tableBytes = std::max(st->tableBytes, bytes); + } + + // ------------------------------------------------------------ 1. tiles + + bool readTiles() { + const uint32_t started = nowMs(); + if (!outA.open(files, file("l.tmp")) || !outB.open(files, file("h.tmp"))) { + return fail("cannot create the scratch files"); + } + if (!sortBegin(sortA, "s_pt", comparePoints) || !sortBegin(sortB, "s_hs", compareHouseCopies)) { + return false; + } + RegionStats& rs = st->region; + for (uint32_t t = 0; t < tiles.size(); t++) { + if (!keepGoing()) { + return false; + } + TileInfo& tile = tiles[t]; + ByteInput* input = source->open(tile.x, tile.y); + if (input == nullptr) { + return fail("cannot read tile %u/%u", static_cast(tile.x), static_cast(tile.y)); + } + const uint32_t tileStart = nowMs(); + currentTile = t; + tile.lineFirst = static_cast(lineName.size()); + tile.lineOffset = outA.size(); + tile.rawHouseOffset = outB.size(); + tile.rawHouseCount = 0; + consumer.impl = this; + if (!extractor.extract(*decoder, input, tile.x, tile.y, extract, &consumer)) { + return owner->errorText[0] != '\0' ? false + : fail("cannot decode tile %u/%u", static_cast(tile.x), static_cast(tile.y)); + } + tile.lineCount = static_cast(lineName.size()) - tile.lineFirst; + rs.refOnlyLines += extract.refOnlyLines; + rs.junkPoisSkipped += extract.junkPois; + const uint32_t ms = nowMs() - tileStart; + if (ms > st->slowestExtractMs) { + st->slowestExtractMs = ms; + st->slowestExtractTile = { tile.x, tile.y }; + } + step(); + } + if (!outA.close() || !outB.close()) { + return fail("cannot write the scratch files"); + } + st->scratchBytes += outA.size() + outB.size(); + extract.release(); + source->tilesDone(); + lineHash.release(); + rs.tiles = static_cast(tiles.size()); + rs.lines = static_cast(lineName.size()); + rs.settlements = static_cast(settlements.size()); + noteTables(); + st->phaseMs[0] = nowMs() - started; + return !names.failed() && !lineName.failed() && !settlements.failed() ? true : fail("no memory for the tile tables"); + } + + /** One decoded feature of the current tile: its lines, house number or point. */ + bool takeFeature(const ExtractedTile& feature) { + // Out of memory in a table: stop at once (a table that could not grow must not be read further) + if (names.failed() || lineName.failed() || lineHash.failed() || settlements.failed()) { + return fail("no memory for the tile tables"); + } + return addLines(currentTile, feature) && addHouses(currentTile, feature) && addPoints(feature); + } + + bool addLines(uint32_t t, const ExtractedTile& extract) { + const TileInfo& tile = tiles[t]; + for (size_t i = 0; i < extract.lines.size(); i++) { + const ExtractLine& l = extract.lines[i]; + const char* name = extract.str(l.name); + const int32_t* xy = &extract.linePoints[2 * static_cast(l.first)]; + const uint64_t hash = mix(hashBytes(hashText(SEED, name), xy, 2 * sizeof(int32_t) * l.count)); + // A copy from a neighbour's buffer: the same name and points in this tile or a neighbour read before + bool duplicate = false; + for (uint32_t k = tile.lineFirst; k < lineName.size() && !duplicate; k++) { + duplicate = lineHash[k] == hash; + } + for (int dx = -1; dx <= 1 && !duplicate; dx++) { + for (int dy = -1; dy <= 1 && !duplicate; dy++) { + const uint32_t n = (dx == 0 && dy == 0) ? NONE : tileAt(tile.x + dx, tile.y + dy); + if (n == NONE || n >= t) { + continue; + } + for (uint32_t k = tiles[n].lineFirst; k < tiles[n].lineFirst + tiles[n].lineCount && !duplicate; k++) { + duplicate = lineHash[k] == hash; + } + } + } + if (duplicate) { + continue; + } + const uint32_t nameId = names.intern(name); + lineName.push(nameId); + lineHash.push(hash); + rec.clear(); + rec.u32(nameId); + rec.u8(l.pathClass); + const char* latin = extract.str(l.latin); + rec.u8(latin != nullptr ? 1 : 0); + rec.u32(l.count); + if (!outA.write(rec) || (latin != nullptr && !outA.write(latin, static_cast(strlen(latin)) + 1)) || + !outA.write(xy, 2 * sizeof(int32_t) * l.count)) { + return fail("cannot write the lines"); + } + } + return true; + } + + bool addHouses(uint32_t t, const ExtractedTile& extract) { + TileInfo& tile = tiles[t]; + for (size_t i = 0; i < extract.houses.size(); i++) { + const ExtractHouse& h = extract.houses[i]; + const char* text = extract.str(h.text); + // Its index among the tile's house numbers (the tile's own order) + const uint32_t local = tile.rawHouseCount++; + rec.clear(); + rec.u32(local); + rec.text(text); + rec.i32(h.x); + rec.i32(h.y); + if (!outB.write(rec)) { + return fail("cannot write the house numbers"); + } + // Every copy goes into the sort that picks one per (text, position): see filterHouses() + const uint64_t hash = mix(hashInt(hashInt(hashText(SEED, text), h.x), h.y)); + rec.clear(); + rec.u32(static_cast(hash >> 32)); + rec.u32(static_cast(hash)); + rec.u32(h.inside != 0 ? 0 : 1); + rec.u32(t); + rec.u32(local); + if (!sortAdd(sortB)) { + return false; + } + } + return true; + } + + /** + * One copy per house number (text and position), like SearchRegionBuilder: the copy inside its own tile, else the + * first copy read (tile order, then the tile's own order). The others are dropped while h.tmp becomes i.tmp; the + * region's number statistics and modes come from what is kept. + */ + bool filterHouses() { + const uint32_t started = nowMs(); + RegionStats& rs = st->region; + if (!sortB.finish() || !sortBegin(sortA, "s_drop", compareFirstTwo)) { + return fail("cannot sort the house numbers"); + } + const uint8_t* data = nullptr; + uint32_t size = 0; + uint32_t lastHi = 0; + uint32_t lastLo = 0; + bool have = false; + while (sortB.next(data, size)) { + const uint32_t hi = get32(data); + const uint32_t lo = get32(data + 4); + if (have && hi == lastHi && lo == lastLo) { + rec.clear(); + rec.u32(get32(data + 12)); + rec.u32(get32(data + 16)); + if (!sortAdd(sortA)) { + return false; + } + continue; + } + have = true; + lastHi = hi; + lastLo = lo; + } + if (sortB.failed() || !sortA.finish()) { + return fail(sortB.failed() ? sortB.errorText() : sortA.errorText()); + } + noteSort(sortB); + sortB.end(); + if (!inB.open(files, file("h.tmp")) || !outB.open(files, file("i.tmp"))) { + return fail("cannot filter the house numbers"); + } + bool dropPending = sortA.next(data, size); + uint32_t dropTile = dropPending ? get32(data) : NONE; + uint32_t dropLocal = dropPending ? get32(data + 4) : NONE; + for (uint32_t t = 0; t < tiles.size(); t++) { + TileInfo& tile = tiles[t]; + tile.houseOffset = outB.size(); + tile.houseCount = 0; + for (uint32_t k = 0; k < tile.rawHouseCount; k++) { + uint32_t local = 0; + int32_t x = 0; + int32_t y = 0; + if (!inB.u32(local) || !inB.text(textA, sizeof(textA)) || !inB.i32(x) || !inB.i32(y)) { + return fail("cannot read the house numbers"); + } + if (dropPending && dropTile == t && dropLocal == local) { + dropPending = sortA.next(data, size); + dropTile = dropPending ? get32(data) : NONE; + dropLocal = dropPending ? get32(data + 4) : NONE; + continue; + } + rec.clear(); + rec.text(textA); + rec.i32(x); + rec.i32(y); + if (!outB.write(rec)) { + return fail("cannot write the house numbers"); + } + tile.houseCount++; + rs.houses++; + const HouseNumberStyle kind = houseNumberStyle(textA); + rs.textNumbers += kind.parsed ? 0 : 1; + rs.plainNumbers += kind.plain ? 1 : 0; + rs.slashNumbers += kind.slash ? 1 : 0; + rs.slashSecondSmaller += kind.secondSmaller ? 1 : 0; + } + if (t % 16 == 0 && !keepGoing()) { + return false; + } + } + inB.close(); + if (sortA.failed()) { + return fail(sortA.errorText()); + } + noteSort(sortA); + sortA.end(); + if (!outB.close()) { + return fail("cannot write the house numbers"); + } + st->scratchBytes += outB.size(); + files->remove(file("h.tmp")); + rs.plainShare = rs.houses > 0 ? static_cast(rs.plainNumbers) / static_cast(rs.houses) : 0.0; + rs.villageMode = rs.plainShare >= config.villageShare && rs.houses > 0; + rs.slashShare = rs.slashNumbers > 0 ? static_cast(rs.slashSecondSmaller) / static_cast(rs.slashNumbers) : 0.0; + rs.slashSecond = rs.slashShare >= config.slashSecondShare && rs.slashNumbers > 0; + villageMode = rs.villageMode; + slashSecond = rs.slashSecond; + st->phaseMs[1] += nowMs() - started; + step(); + return true; + } + + bool addPoints(const ExtractedTile& extract) { + for (size_t i = 0; i < extract.points.size(); i++) { + const ExtractPoint& p = extract.points[i]; + const uint32_t seq = pointSeq++; + const char* labelText = extract.str(p.label); + const uint32_t label = labelNames.intern(labelText); + if (label == labels.size()) { + LabelInfo info = {}; + info.minSeq = NONE; + info.category = strncmp(labelText, "poi/", 4) == 0 ? CAT_POI : (strncmp(labelText, "place/", 6) == 0 ? CAT_PLACE : CAT_OTHER); + labels.push(info); + } + const char* name = extract.str(p.name); + rec.clear(); + rec.u32(label); + rec.u8(name != nullptr ? 1 : 0); + rec.text(name); + rec.i32(p.x); + rec.i32(p.y); + rec.u32(seq); + rec.text(extract.str(p.latin)); + rec.u8(p.latin != EXTRACT_NONE ? 1 : 0); + rec.text(extract.str(p.en)); + rec.u8(p.en != EXTRACT_NONE ? 1 : 0); + rec.u8(p.useEn); + if (!sortAdd(sortA)) { + return false; + } + // Settlements for the village rule, in the order they are first seen (the points' order) + if (p.settlement != EXTRACT_NOT_SETTLEMENT && name != nullptr) { + const uint32_t nameId = names.intern(name); + bool known = false; + for (const SettlementInfo& s : settlements) { + known = known || (s.label == label && s.nameId == nameId && s.x == p.x && s.y == p.y); + } + if (!known) { + SettlementInfo s = {}; + s.label = label; + s.nameId = nameId; + s.x = p.x; + s.y = p.y; + s.placeClass = p.settlement; + s.latin = addExtra(extract.str(p.latin)); + s.en = addExtra(extract.str(p.en)); + s.pseudo = NONE; + settlements.push(s); + } + } + } + return true; + } + + // ------------------------------------------------------------ 2. points + + bool dedupPoints() { + const uint32_t started = nowMs(); + if (!sortA.finish()) { + return fail(sortA.errorText()); + } + if (!outA.open(files, file("p.tmp"))) { + return fail("cannot create the point file"); + } + RegionStats& rs = st->region; + const uint8_t* data = nullptr; + uint32_t size = 0; + uint32_t lastLabel = NONE; + uint32_t lastNamed = 0; + int32_t lastX = 0, lastY = 0; + bool have = false; + uint32_t n = 0; + while (sortA.next(data, size)) { + if (++n % CHECK_EVERY == 0 && !keepGoing()) { + return false; + } + Cursor c(data, size); + const uint32_t label = c.u32(); + const uint32_t named = c.u8(); + const char* name = c.text(); + const int32_t x = c.i32(); + const int32_t y = c.i32(); + const uint32_t seq = c.u32(); + if (have && label == lastLabel && named == lastNamed && x == lastX && y == lastY && strcmp(name, textA) == 0) { + continue; // a later copy of the same point + } + have = true; + lastLabel = label; + lastNamed = named; + lastX = x; + lastY = y; + snprintf(textA, sizeof(textA), "%s", name); + const char* latin = c.text(); + const uint32_t hasLatin = c.u8(); + const char* en = c.text(); + const uint32_t hasEn = c.u8(); + const uint32_t useEn = c.u8(); + LabelInfo& info = labels[label]; + info.count++; + info.minSeq = std::min(info.minSeq, seq); + pointCount++; + include(x, y); + if (info.category == CAT_PLACE) { + rs.places++; + } else if (info.category == CAT_POI) { + if (named != 0) { + rs.namedPois++; + } else { + rs.unnamedPois++; + } + } else { + rs.otherNamed++; + } + namedCount += named != 0 ? 1 : 0; + rec.clear(); + rec.u32(label); + rec.u8(named); + rec.text(name); + rec.text(hasLatin != 0 ? latin : nullptr); + rec.u8(hasLatin); + rec.text(hasEn != 0 ? en : nullptr); + rec.u8(hasEn); + rec.u8(useEn); + rec.i32(x); + rec.i32(y); + rec.u32(seq); + if (!outA.write(rec)) { + return fail("cannot write the points"); + } + } + if (sortA.failed()) { + return fail(sortA.errorText()); + } + noteSort(sortA); + sortA.end(); + if (!outA.close()) { + return fail("cannot write the points"); + } + st->scratchBytes += outA.size(); + if (!makePairs()) { + return false; + } + st->phaseMs[1] = nowMs() - started; + step(); + return true; + } + + /** Class pairs as the writer makes them: labels by text; beyond 255 the rare ones fold to their kind and class plus a star (top 110 kept). */ + bool makePairs() { + const uint32_t total = static_cast(labels.size()); + GrowArray byText; + for (uint32_t u = 0; u < total; u++) { + byText.push(u); + } + std::sort(byText.begin(), byText.end(), [&](uint32_t a, uint32_t b) { return strcmp(labelNames.at(a), labelNames.at(b)) < 0; }); + GrowArray keep; + keep.resize(total); + if (total > 255) { + GrowArray byCount; + for (uint32_t u = 0; u < total; u++) { + byCount.push(u); + } + std::sort(byCount.begin(), byCount.end(), [&](uint32_t a, uint32_t b) { + return labels[a].count != labels[b].count ? labels[a].count > labels[b].count : labels[a].minSeq < labels[b].minSeq; + }); + for (uint32_t k = 0; k < 110; k++) { + keep[byCount[k]] = 1; + } + collapsed = total - 110; + } else { + for (uint32_t u = 0; u < total; u++) { + keep[u] = 1; + } + } + // The folded texts, then the pairs in their text order + Strings folded; + GrowArray foldedOf; + foldedOf.resize(total); + for (uint32_t u = 0; u < total; u++) { + const char* label = labelNames.at(u); + char text[300]; + if (keep[u]) { + snprintf(text, sizeof(text), "%s", label); + } else { + const char* slash = strchr(label, '/'); + const char* second = slash != nullptr ? strchr(slash + 1, '/') : nullptr; + const size_t keepLength = second != nullptr ? static_cast(second - label) : strlen(label); + snprintf(text, sizeof(text), "%.*s/*", static_cast(keepLength), label); + } + foldedOf[u] = folded.intern(text); + } + GrowArray foldedOrder; + for (uint32_t f = 0; f < folded.count(); f++) { + foldedOrder.push(f); + } + std::sort(foldedOrder.begin(), foldedOrder.end(), [&](uint32_t a, uint32_t b) { return strcmp(folded.at(a), folded.at(b)) < 0; }); + pairTotal = folded.count(); + if (pairTotal > 255) { + return fail("%u class pairs do not fit a byte", pairTotal); + } + GrowArray pairOfFolded; + pairOfFolded.resize(pairTotal); + for (uint32_t k = 0; k < pairTotal; k++) { + pairOfFolded[foldedOrder[k]] = k; + const char* text = folded.at(foldedOrder[k]); + if (strlen(text) > 255) { + return fail("class label %u is longer than 255 bytes", k); + } + pairTexts.intern(text); // pair k has id k + } + for (uint32_t u = 0; u < total; u++) { + labels[u].pair = pairOfFolded[foldedOf[u]]; + } + pairStart.resize(pairTotal); + pairCount.resize(pairTotal); + return makeCategories(); + } + + /** Category name groups as the writer makes them: class and subclass names of the poi pairs, when contiguous. */ + bool makeCategories() { + GrowArray classRuns; + GrowArray subRuns; + auto addRun = [&](GrowArray& runs, const char* raw, size_t length, uint32_t pair) { + for (CategoryRun& run : runs) { + const char* stored = &categoryChars[run.raw]; + if (strlen(stored) == length && memcmp(stored, raw, length) == 0) { + if (pair != run.lastPair + 1) { + run.contiguous = false; + } + run.lastPair = pair; + run.count++; + return; + } + } + CategoryRun run = {}; + run.raw = static_cast(categoryChars.size()); + for (size_t i = 0; i < length; i++) { + categoryChars.push(raw[i]); + } + categoryChars.push('\0'); + run.firstPair = run.lastPair = pair; + run.count = 1; + run.contiguous = true; + runs.push(run); + }; + for (uint32_t pair = 0; pair < pairTotal; pair++) { + const char* label = pairTexts.at(pair); + if (strncmp(label, "poi/", 4) != 0) { + continue; + } + const char* classStart = label + 4; + const char* classEnd = strchr(classStart, '/'); + const size_t classLength = classEnd != nullptr ? static_cast(classEnd - classStart) : strlen(classStart); + addRun(classRuns, classStart, classLength, pair); + if (classEnd != nullptr) { + const char* subStart = classEnd + 1; + const char* subEnd = strchr(subStart, '/'); + const size_t subLength = subEnd != nullptr ? static_cast(subEnd - subStart) : strlen(subStart); + if (!(subLength == classLength && memcmp(subStart, classStart, subLength) == 0)) { + addRun(subRuns, subStart, subLength, pair); + } + } + } + // Keys of category names that are not ASCII (SearchRegionBuilder's overrides, from every raw label) + Strings overrideNames; + GrowArray overrideKeys; + for (uint32_t u = 0; u < labels.size(); u++) { + const char* label = labelNames.at(u); + if (strncmp(label, "poi/", 4) != 0) { + continue; + } + const char* part = label + 4; + for (int k = 0; k < 2 && part != nullptr; k++) { + const char* slash = strchr(part, '/'); + const size_t length = slash != nullptr ? static_cast(slash - part) : strlen(part); + char display[200]; + const size_t n = std::min(length, sizeof(display) - 1); + bool ascii = true; + for (size_t c = 0; c < n; c++) { + display[c] = part[c] == '_' ? ' ' : part[c]; + ascii = ascii && static_cast(part[c]) < 0x80; + } + display[n] = '\0'; + part = slash != nullptr ? slash + 1 : nullptr; + if (ascii || overrideNames.find(display) != HashIndex::NONE) { + continue; + } + overrideNames.intern(display); + KeyMaker::sortKeyOf(display, config.profile, keyScratch, sizeof(keyScratch)); + overrideKeys.push(addExtra(keyScratch)); + } + } + uint32_t creation = 0; + for (int which = 0; which < 2; which++) { + for (const CategoryRun& run : which == 0 ? classRuns : subRuns) { + if (!run.contiguous) { + notContiguous++; + continue; + } + char name[128]; + const char* raw = &categoryChars[run.raw]; + const size_t length = std::min(strlen(raw), sizeof(name) - 1); + memcpy(name, raw, length); + for (size_t i = 0; i < length; i++) { + if (name[i] == '_') { + name[i] = ' '; + } + } + name[length] = '\0'; + char key[128]; + foldCategoryName(name, key, sizeof(key)); + const uint32_t o = overrideNames.find(name); + if (o != HashIndex::NONE) { + snprintf(key, sizeof(key), "%s", extra(overrideKeys[o])); + } + CategoryNg ng = {}; + ng.name = addExtra(name); + ng.sortKey = addExtra(key); + ng.firstPair = run.firstPair; + ng.pairCount = run.count; + ng.creation = creation++; + categories.push(ng); + } + } + std::sort(categories.begin(), categories.end(), [&](const CategoryNg& a, const CategoryNg& b) { + const int c = compareText(extra(a.sortKey), extra(b.sortKey)); + return c != 0 ? c < 0 : a.creation < b.creation; + }); + return !categories.failed() && !categoryChars.failed() && !extraText.failed() ? true : fail("no memory for the categories"); + } + + // ------------------------------------------------------------ 3. street clusters + + /** Reads one line record of l.tmp: name, path class, latin (into textC, "" for none), points. */ + bool readLine(InFile& in, uint32_t& nameId, uint32_t& pathClass, bool& hasLatin, GrowArray& xy) { + uint32_t latinFlag = 0; + uint32_t count = 0; + if (!in.u32(nameId) || !in.u8(pathClass) || !in.u8(latinFlag) || !in.u32(count)) { + return false; + } + hasLatin = latinFlag != 0; + textC[0] = '\0'; + if (hasLatin && !in.text(textC, sizeof(textC))) { + return false; + } + xy.resize(2 * static_cast(count)); + return !xy.failed() && in.read(xy.begin(), 2 * sizeof(int32_t) * count); + } + + bool makeClusters() { + const uint32_t started = nowMs(); + GrowArray comps; + const uint32_t lineTotal = static_cast(lineName.size()); + const auto cell = std::max(1, static_cast(150.0 / regionMpu)); + if (!sortBegin(sortA, "s_cell", compareCells) || !inA.open(files, file("l.tmp"))) { + return fail("cannot read the lines"); + } + for (uint32_t i = 0; i < lineTotal; i++) { + uint32_t nameId = 0; + uint32_t pathClass = 0; + bool hasLatin = false; + if (!readLine(inA, nameId, pathClass, hasLatin, points)) { + return fail("cannot read line %u", i); + } + int32_t previousX = INT32_MIN; + int32_t previousY = INT32_MIN; + for (size_t k = 0; k < points.size() / 2; k++) { + const int32_t cx = floorDiv(points[2 * k], cell); + const int32_t cy = floorDiv(points[2 * k + 1], cell); + if (cx == previousX && cy == previousY) { + continue; + } + previousX = cx; + previousY = cy; + rec.clear(); + rec.u32(nameId); + rec.i32(cx); + rec.i32(cy); + rec.u32(i); + if (!sortAdd(sortA)) { + return false; + } + } + if (i % CHECK_EVERY == 0 && !keepGoing()) { + return false; + } + } + inA.close(); + if (!sortA.finish()) { + return fail(sortA.errorText()); + } + // Union-find over the lines: the same cell, or touching cells (8 neighbours), of the same name join + GrowArray& parent = lineCluster; + parent.resize(lineTotal); + for (uint32_t i = 0; i < lineTotal; i++) { + parent[i] = i; + } + auto find = [&](uint32_t x) { + while (parent[x] != x) { + parent[x] = parent[parent[x]]; + x = parent[x]; + } + return x; + }; + auto join = [&](uint32_t a, uint32_t b) { + a = find(a); + b = find(b); + if (a != b) { + parent[a] = b; + } + }; + GrowArray previous; + GrowArray current; + uint32_t name = NONE; + int32_t column = 0; + const uint8_t* data = nullptr; + uint32_t size = 0; + while (sortA.next(data, size)) { + Cursor c(data, size); + const uint32_t n = c.u32(); + const int32_t cx = c.i32(); + const int32_t cy = c.i32(); + const uint32_t line = c.u32(); + if (n != name) { + name = n; + previous.clear(); + current.clear(); + column = cx; + } else if (cx != column) { + if (cx == column + 1) { + previous.swap(current); + } else { + previous.clear(); + } + current.clear(); + column = cx; + } + if (current.size() > 0 && current[current.size() - 1].cy == cy) { + join(line, current[current.size() - 1].line); + continue; + } + if (current.size() > 0 && current[current.size() - 1].cy == cy - 1) { + join(line, current[current.size() - 1].line); + } + // The column to the left: cells cy - 1 .. cy + 1 (sorted by cy) + const ColumnCell* lo = std::lower_bound(previous.begin(), previous.end(), cy - 1, + [](const ColumnCell& cell, int32_t value) { return cell.cy < value; }); + for (const ColumnCell* p = lo; p != previous.end() && p->cy <= cy + 1; p++) { + join(line, p->line); + } + current.push({ cy, line }); + } + if (sortA.failed()) { + return fail(sortA.errorText()); + } + noteSort(sortA); + sortA.end(); + previous.release(); + current.release(); + // Components: one per root, numbered in line order + GrowArray compOfRoot; + compOfRoot.resize(lineTotal); + for (uint32_t i = 0; i < lineTotal; i++) { + compOfRoot[i] = NONE; + } + GrowArray compOfLine; + compOfLine.resize(lineTotal); + for (uint32_t i = 0; i < lineTotal; i++) { + const uint32_t root = find(i); + if (compOfRoot[root] == NONE) { + compOfRoot[root] = static_cast(comps.size()); + Comp comp = {}; + comp.nameId = lineName[i]; + comp.minX = comp.minY = INT32_MAX; + comp.maxX = comp.maxY = INT32_MIN; + comp.firstX = comp.firstY = INT32_MAX; + comp.latin = NONE; + comps.push(comp); + } + compOfLine[i] = compOfRoot[root]; + } + compOfRoot.release(); // reused below as the map from a component to its sorted cluster + if (comps.failed() || compOfLine.failed()) { + return fail("no memory for the street clusters"); + } + // Their sums, boxes and smallest first vertex + if (!inA.open(files, file("l.tmp"))) { + return fail("cannot read the lines"); + } + for (uint32_t i = 0; i < lineTotal; i++) { + uint32_t nameId = 0; + uint32_t pathClass = 0; + bool hasLatin = false; + if (!readLine(inA, nameId, pathClass, hasLatin, points)) { + return fail("cannot read line %u", i); + } + Comp& comp = comps[compOfLine[i]]; + const int32_t fx = points[0]; + const int32_t fy = points[1]; + if (fx < comp.firstX || (fx == comp.firstX && fy < comp.firstY)) { + comp.firstX = fx; + comp.firstY = fy; + } + if (comp.latin == NONE && hasLatin) { + comp.latin = addExtra(textC); + } + for (size_t k = 0; k < points.size() / 2; k++) { + const int32_t x = points[2 * k]; + const int32_t y = points[2 * k + 1]; + comp.sumX += x; + comp.sumY += y; + comp.count++; + comp.minX = std::min(comp.minX, x); + comp.maxX = std::max(comp.maxX, x); + comp.minY = std::min(comp.minY, y); + comp.maxY = std::max(comp.maxY, y); + } + } + inA.close(); + // The representative vertex: the one nearest the centroid (ties: the smallest x, then y) + if (!inA.open(files, file("l.tmp"))) { + return fail("cannot read the lines"); + } + for (uint32_t i = 0; i < lineTotal; i++) { + uint32_t nameId = 0; + uint32_t pathClass = 0; + bool hasLatin = false; + if (!readLine(inA, nameId, pathClass, hasLatin, points)) { + return fail("cannot read line %u", i); + } + Comp& comp = comps[compOfLine[i]]; + const double cx = static_cast(comp.sumX) / static_cast(comp.count); + const double cy = static_cast(comp.sumY) / static_cast(comp.count); + for (size_t k = 0; k < points.size() / 2; k++) { + const int32_t x = points[2 * k]; + const int32_t y = points[2 * k + 1]; + const double d = (x - cx) * (x - cx) + (y - cy) * (y - cy); + if (!comp.haveRep || d < comp.best || (d == comp.best && (x < comp.repX || (x == comp.repX && y < comp.repY)))) { + comp.haveRep = true; + comp.best = d; + comp.repX = x; + comp.repY = y; + } + } + } + inA.close(); + lineName.release(); + // Cluster order: name (bytes), then the smallest first vertex; the compact clusters in that order + GrowArray order; + if (!order.reserve(comps.size()) || !clusters.reserve(comps.size())) { + return fail("no memory for the street clusters"); + } + for (uint32_t c = 0; c < comps.size(); c++) { + order.push(c); + } + std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) { + const Comp& x = comps[a]; + const Comp& y = comps[b]; + if (x.nameId != y.nameId) { + return strcmp(names.at(x.nameId), names.at(y.nameId)) < 0; + } + if (x.firstX != y.firstX) return x.firstX < y.firstX; + return x.firstY < y.firstY; + }); + nameStreet.resize(names.count()); + for (uint32_t n = 0; n < names.count(); n++) { + nameStreet[n] = NONE; + } + for (uint32_t k = 0; k < order.size(); k++) { + const Comp& comp = comps[order[k]]; + clusters.push({ comp.repX, comp.repY, comp.minX, comp.minY, comp.maxX, comp.maxY, 0, 0 }); + include(comp.repX, comp.repY); + include(comp.minX, comp.minY); + include(comp.maxX, comp.maxY); + if (streets.size() == 0 || streets[streets.size() - 1].nameId != comp.nameId) { + StreetInfo street = {}; + street.nameId = comp.nameId; + street.latin = NONE; + street.en = NONE; + street.firstCluster = k; + nameStreet[comp.nameId] = static_cast(streets.size()); + streets.push(street); + } + StreetInfo& street = streets[streets.size() - 1]; + street.clusterCount++; + if (street.latin == NONE && comp.latin != NONE) { + street.latin = comp.latin; + } + } + // Lines point at the sorted clusters (the order array becomes the inverse map) + compOfRoot.resize(comps.size()); + for (uint32_t k = 0; k < order.size(); k++) { + compOfRoot[order[k]] = k; + } + for (uint32_t i = 0; i < lineTotal; i++) { + lineCluster[i] = compOfRoot[compOfLine[i]]; + } + compOfRoot.release(); + compOfLine.release(); + order.release(); + st->region.streetNames = static_cast(streets.size()); + st->region.clusters = static_cast(comps.size()); + comps.release(); + noteTables(); + st->phaseMs[2] = nowMs() - started; + step(); + return !streets.failed() && !nameStreet.failed() && !extraText.failed() ? true : fail("no memory for the streets"); + } + + // ------------------------------------------------------------ 4. matching + + /** Reads the house numbers of a tile: texts into tileText, positions into houseXY. */ + bool readHouses(uint32_t t, bool keepText) { + houseXY.clear(); + tileText.clear(); + tileTextAt.clear(); + const TileInfo& tile = tiles[t]; + if (tile.houseCount == 0) { + return true; + } + if (!inB.seek(tile.houseOffset)) { + return false; + } + for (uint32_t k = 0; k < tile.houseCount; k++) { + int32_t x = 0; + int32_t y = 0; + if (!inB.text(textA, sizeof(textA)) || !inB.i32(x) || !inB.i32(y)) { + return false; + } + houseXY.push(x); + houseXY.push(y); + if (keepText) { + tileTextAt.push(static_cast(tileText.size())); + for (const char* c = textA;; c++) { + tileText.push(*c); + if (*c == '\0') { + break; + } + } + } else { + // Only the vote number is needed: keep it in place of the text + tileTextAt.push(static_cast(houseVoteNumber(textA, slashSecond && config.voteSlashSecond))); + } + } + return !houseXY.failed() && !tileText.failed() && !tileTextAt.failed(); + } + + /** Adds the lines of the tile and its neighbours that SearchRegionBuilder would add, in line order (clipped by the matcher). */ + bool addTileLines(const TileInfo& tile, int32_t lineBuffer) { + // Each tile's lines are one range of l.tmp, in line order + uint32_t ranges[9]; + int rangeCount = 0; + for (int dx = -1; dx <= 1; dx++) { + for (int dy = -1; dy <= 1; dy++) { + const uint32_t n = tileAt(tile.x + dx, tile.y + dy); + if (n != NONE && tiles[n].lineCount > 0) { + ranges[rangeCount++] = n; + } + } + } + // Insertion sort: GCC's array-bounds check misreads std::sort on this small array + for (int i = 1; i < rangeCount; i++) { + const uint32_t value = ranges[i]; + int j = i; + while (j > 0 && tiles[ranges[j - 1]].lineFirst > tiles[value].lineFirst) { + ranges[j] = ranges[j - 1]; + j--; + } + ranges[j] = value; + } + for (int r = 0; r < rangeCount; r++) { + const TileInfo& from = tiles[ranges[r]]; + if (!inA.seek(from.lineOffset)) { + return fail("cannot read the lines"); + } + for (uint32_t k = 0; k < from.lineCount; k++) { + uint32_t nameId = 0; + uint32_t pathClass = 0; + bool hasLatin = false; + if (!readLine(inA, nameId, pathClass, hasLatin, points)) { + return fail("cannot read the lines"); + } + if (pathClass != 0 && !config.matchPathLines) { + continue; + } + int32_t lx0 = INT32_MAX, ly0 = INT32_MAX, lx1 = INT32_MIN, ly1 = INT32_MIN; + for (size_t p = 0; p < points.size() / 2; p++) { + lx0 = std::min(lx0, points[2 * p]); + lx1 = std::max(lx1, points[2 * p]); + ly0 = std::min(ly0, points[2 * p + 1]); + ly1 = std::max(ly1, points[2 * p + 1]); + } + // SearchRegionBuilder's test: the tiles the line's box (plus the buffer) touches include this one + if (floorDiv(lx0 - lineBuffer, TILE_UNITS) > tile.x || floorDiv(lx1 + lineBuffer, TILE_UNITS) < tile.x || + floorDiv(ly0 - lineBuffer, TILE_UNITS) > tile.y || floorDiv(ly1 + lineBuffer, TILE_UNITS) < tile.y) { + continue; + } + matcher.addLine(lineCluster[from.lineFirst + k], points.begin(), points.size() / 2); + } + } + return true; + } + + /** + * House numbers to streets and settlements, tile by tile (SearchRegionBuilder's rules and results). A dense tile is + * matched in blocks to keep the matcher small: a block's houses with the lines within reach and, as voters, every + * other house within VOTE_M of the block. A result depends only on segments within LOW_STREET_M of the house and on + * voters within VOTE_M (whose own nearest streets depend on segments within LOW_STREET_M of them), so blocks give + * the same results as the whole tile at once; the addresses are written in the tile's own order afterwards. + */ + bool matchHouses() { + const uint32_t started = nowMs(); + RegionStats& rs = st->region; + const auto lineBuffer = static_cast(LINE_BUFFER_M / regionMpu) + 1; + if (!inA.open(files, file("l.tmp")) || !inB.open(files, file("i.tmp")) || !outA.open(files, file("a.tmp"))) { + return fail("cannot open the matching files"); + } + GrowArray ownXY; + GrowArray ownVote; + GrowArray ownPlain; + GrowArray ownResult; + GrowArray voterXY; + GrowArray voterVote; + GrowArray members; + for (uint32_t t = 0; t < tiles.size(); t++) { + if (!keepGoing()) { + return false; + } + const TileInfo& tile = tiles[t]; + step(); + if (tile.houseCount == 0) { + continue; + } + const uint32_t tileStart = nowMs(); + const double mpu = metresPerUnitAtTileY(tile.y + 0.5); + const int32_t rx0 = tile.x * TILE_UNITS; + const int32_t ry0 = tile.y * TILE_UNITS; + const int32_t rx1 = rx0 + TILE_UNITS - 1; + const int32_t ry1 = ry0 + TILE_UNITS - 1; + const auto voteBuffer = static_cast(AddressMatcher::VOTE_M / mpu) + 1; + const auto reach = static_cast((AddressMatcher::VOTE_M + AddressMatcher::LOW_STREET_M + 20.0) / mpu) + 2; + // The tile's own houses (position, vote number, village style), and the neighbours' near the tile that vote + const uint32_t own = tile.houseCount; + ownXY.clear(); + ownVote.clear(); + ownPlain.clear(); + if (!ownXY.reserve(2 * static_cast(own)) || !ownVote.reserve(own) || !ownPlain.reserve(own) || !inB.seek(tile.houseOffset)) { + return fail("no memory for the house matching"); + } + for (uint32_t k = 0; k < own; k++) { + int32_t x = 0; + int32_t y = 0; + if (!inB.text(textA, sizeof(textA)) || !inB.i32(x) || !inB.i32(y)) { + return fail("cannot read the house numbers"); + } + ownXY.push(x); + ownXY.push(y); + ownVote.push(houseVoteNumber(textA, slashSecond && config.voteSlashSecond)); + ownPlain.push(houseNumberStyle(textA).plain ? 1 : 0); + } + voterXY.clear(); + voterVote.clear(); + for (int dx = -1; dx <= 1; dx++) { + for (int dy = -1; dy <= 1; dy++) { + if (dx == 0 && dy == 0) { + continue; + } + const uint32_t n = tileAt(tile.x + dx, tile.y + dy); + if (n == NONE || tiles[n].houseCount == 0) { + continue; + } + if (!inB.seek(tiles[n].houseOffset)) { + return fail("cannot read the house numbers"); + } + for (uint32_t k = 0; k < tiles[n].houseCount; k++) { + int32_t x = 0; + int32_t y = 0; + if (!inB.text(textA, sizeof(textA)) || !inB.i32(x) || !inB.i32(y)) { + return fail("cannot read the house numbers"); + } + if (x >= rx0 - voteBuffer && x <= rx1 + voteBuffer && y >= ry0 - voteBuffer && y <= ry1 + voteBuffer) { + voterXY.push(x); + voterXY.push(y); + voterVote.push(houseVoteNumber(textA, slashSecond && config.voteSlashSecond)); + } + } + } + } + if (!ownResult.resize(own) || ownXY.failed() || ownVote.failed() || ownPlain.failed() || voterXY.failed() || voterVote.failed()) { + return fail("no memory for the house matching"); + } + // Blocks per side: one for most tiles, more for the dense ones (each block reads the lines again) + const int32_t blocks = own <= 200 ? 1 : (own <= 800 ? 2 : (own <= 1800 ? 3 : 4)); + const int32_t blockUnits = (TILE_UNITS + blocks - 1) / blocks; + for (int32_t by = 0; by < blocks; by++) { + for (int32_t bx = 0; bx < blocks; bx++) { + // Members: own houses whose position (held to the tile) falls in the block, in the tile's order + members.clear(); + int32_t mx0 = INT32_MAX, my0 = INT32_MAX, mx1 = INT32_MIN, my1 = INT32_MIN; + for (uint32_t k = 0; k < own; k++) { + const int32_t x = ownXY[2 * k]; + const int32_t y = ownXY[2 * k + 1]; + const int32_t cx = std::min(std::max(x, rx0), rx1) - rx0; + const int32_t cy = std::min(std::max(y, ry0), ry1) - ry0; + if (cx / blockUnits != bx || cy / blockUnits != by) { + continue; + } + members.push(k); + mx0 = std::min(mx0, x); + mx1 = std::max(mx1, x); + my0 = std::min(my0, y); + my1 = std::max(my1, y); + } + if (members.size() == 0) { + continue; + } + matcher.begin(mpu); + if (blocks == 1) { + matcher.clipToTile(tile.x, tile.y); + } else { + matcher.clip(mx0 - reach, my0 - reach, mx1 + reach, my1 + reach); + } + if (!addTileLines(tile, lineBuffer)) { + return false; + } + if (villageMode) { + const auto placeBuffer = static_cast(AddressMatcher::PLACE_M / mpu) + 1; + for (uint32_t s = 0; s < settlements.size(); s++) { + const SettlementInfo& p = settlements[s]; + if (p.x >= rx0 - placeBuffer && p.x <= rx1 + placeBuffer && p.y >= ry0 - placeBuffer && p.y <= ry1 + placeBuffer) { + matcher.addPlace(s, static_cast(p.placeClass), p.x, p.y); + } + } + } + for (uint32_t m : members) { + matcher.addHouse(ownXY[2 * m], ownXY[2 * m + 1], ownVote[m], villageMode && ownPlain[m] != 0); + } + // Voters: the tile's other houses and the neighbours' near this block (all of them for one block) + auto near = [&](int32_t x, int32_t y) { + return blocks == 1 || (x >= mx0 - voteBuffer && x <= mx1 + voteBuffer && y >= my0 - voteBuffer && y <= my1 + voteBuffer); + }; + if (blocks > 1) { + size_t next = 0; + for (uint32_t k = 0; k < own; k++) { + if (next < members.size() && members[next] == k) { + next++; + continue; + } + if (near(ownXY[2 * k], ownXY[2 * k + 1])) { + matcher.addHouse(ownXY[2 * k], ownXY[2 * k + 1], ownVote[k], false); + } + } + } + for (size_t k = 0; k < voterVote.size(); k++) { + if (near(voterXY[2 * k], voterXY[2 * k + 1])) { + matcher.addHouse(voterXY[2 * k], voterXY[2 * k + 1], voterVote[k], false); + } + } + if (!matcher.run()) { + return fail("no memory for the house matching"); + } + for (size_t q = 0; q < members.size(); q++) { + const AddressMatch& m = matcher.result(static_cast(q)); + OwnResult& r = ownResult[members[q]]; + r.cluster = m.tier == AddressTier::None ? NONE : m.street; + r.village = (villageMode && ownPlain[members[q]] != 0 && m.villages[0] != NONE) ? m.villages[0] : NONE; + r.tier = static_cast(m.tier); + } + } + } + // The addresses in the tile's own order, with the texts read again + if (!inB.seek(tile.houseOffset)) { + return fail("cannot read the house numbers"); + } + for (uint32_t k = 0; k < own; k++) { + int32_t x = 0; + int32_t y = 0; + if (!inB.text(textA, sizeof(textA)) || !inB.i32(x) || !inB.i32(y)) { + return fail("cannot read the house numbers"); + } + const char* text = textA; + const OwnResult& m = ownResult[k]; + const uint32_t cluster = m.cluster; + const uint32_t village = m.village; + if (cluster != NONE) { + if (!writeAddress(cluster, m.tier == static_cast(AddressTier::LowStreet) ? ADDRESS_LOW : 0, x, y, text)) { + return false; + } + if (m.tier == static_cast(AddressTier::Street)) { + rs.pathStreet50++; + } else { + rs.pathStreet100++; + } + if (village != NONE) { + rs.alsoVillage++; + } + } else if (village != NONE) { + rs.pathVillage++; + } else { + rs.pathNone++; + } + if (village != NONE) { + SettlementInfo& s = settlements[village]; + if (s.pseudo == NONE) { + uint32_t street = nameStreet[s.nameId]; + if (street == NONE) { + StreetInfo created = {}; + created.nameId = s.nameId; + created.latin = s.latin; + created.en = s.en; + created.useEn = 1; + created.firstCluster = NONE; + street = static_cast(streets.size()); + nameStreet[s.nameId] = street; + streets.push(created); + } + s.pseudo = static_cast(pseudos.size()); + pseudos.push({ village, street, 0, 0 }); + streets[street].pseudoCount++; + } + if (!writeAddress(PSEUDO | s.pseudo, ADDRESS_VILLAGE, x, y, text)) { + return false; + } + } + } + const uint32_t ms = nowMs() - tileStart; + if (ms > st->slowestMatchMs) { + st->slowestMatchMs = ms; + st->slowestMatchTile = { tile.x, tile.y }; + } + st->matcherPeakBytes = std::max(st->matcherPeakBytes, matcher.allocated()); + } + matcher.release(); + inA.close(); + inB.close(); + if (!outA.close()) { + return fail("cannot write the addresses"); + } + st->scratchBytes += outA.size(); + rs.pseudoStreets = static_cast(pseudos.size()); + noteTables(); + st->phaseMs[3] = nowMs() - started; + return !streets.failed() && !pseudos.failed() ? true : fail("no memory for the village streets"); + } + + bool writeAddress(uint32_t cluster, uint32_t flags, int32_t x, int32_t y, const char* text) { + rec.clear(); + rec.u32(cluster); + rec.u8(flags); + rec.i32(x); + rec.i32(y); + rec.text(text); + return outA.write(rec) ? true : fail("cannot write the addresses"); + } + + // ------------------------------------------------------------ 5. streets and addresses + + /** Street records in name group order: per street (sort key, then name order) its clusters, then its villages. */ + bool orderStreets() { + GrowArray byName; + for (uint32_t s = 0; s < streets.size(); s++) { + byName.push(s); + } + std::sort(byName.begin(), byName.end(), [&](uint32_t a, uint32_t b) { return strcmp(names.at(streets[a].nameId), names.at(streets[b].nameId)) < 0; }); + for (uint32_t k = 0; k < byName.size(); k++) { + StreetInfo& s = streets[byName[k]]; + s.order = k; + KeyMaker::sortKeyOf(names.at(s.nameId), config.profile, keyScratch, sizeof(keyScratch)); + s.sortKey = static_cast(sortKeys.size()); + for (const char* c = keyScratch;; c++) { + sortKeys.push(*c); + if (*c == '\0') { + break; + } + } + } + streetNgOrder.clear(); + for (uint32_t k = 0; k < byName.size(); k++) { + streetNgOrder.push(byName[k]); + } + std::sort(streetNgOrder.begin(), streetNgOrder.end(), [&](uint32_t a, uint32_t b) { + const int c = compareText(&sortKeys[streets[a].sortKey], &sortKeys[streets[b].sortKey]); + return c != 0 ? c < 0 : streets[a].order < streets[b].order; + }); + // The villages of a street in the order of their settlements + GrowArray pseudoOrder; + for (uint32_t p = 0; p < pseudos.size(); p++) { + pseudoOrder.push(p); + } + std::sort(pseudoOrder.begin(), pseudoOrder.end(), [&](uint32_t a, uint32_t b) { + if (pseudos[a].street != pseudos[b].street) return pseudos[a].street < pseudos[b].street; + return pseudos[a].settlement < pseudos[b].settlement; + }); + GrowArray pseudoFirst; + pseudoFirst.resize(streets.size()); + for (auto k = static_cast(pseudoOrder.size()); k > 0; k--) { + pseudoFirst[pseudos[pseudoOrder[k - 1]].street] = k - 1; + } + uint32_t record = 0; + for (uint32_t street : streetNgOrder) { + StreetInfo& s = streets[street]; + if (s.clusterCount + s.pseudoCount > 0xFFFF) { + return fail("a street has %u clusters, the limit is 65535", s.clusterCount + s.pseudoCount); + } + s.firstRecord = record; + for (uint32_t c = 0; c < s.clusterCount; c++) { + clusters[s.firstCluster + c].finalIndex = record++; + } + for (uint32_t p = 0; p < s.pseudoCount; p++) { + pseudos[pseudoOrder[pseudoFirst[street] + p]].finalIndex = record++; + } + } + streetRecordTotal = record; + noteTables(); + return !streetNgOrder.failed() && !sortKeys.failed() ? true : fail("no memory for the street order"); + } + + /** The addresses by street record (capped at 65535 each), then sorted by number: a.tmp -> d.tmp, o.tmp. */ + bool makeAddresses() { + const uint32_t started = nowMs(); + RegionStats& rs = st->region; + if (!orderStreets() || !sortBegin(sortA, "s_a1", compareAddressSeq) || !inA.open(files, file("a.tmp"))) { + return fail("cannot read the addresses"); + } + uint32_t seq = 0; + while (!inA.atEnd()) { + uint32_t cluster = 0; + uint32_t flags = 0; + int32_t x = 0; + int32_t y = 0; + if (!inA.u32(cluster) || !inA.u8(flags) || !inA.i32(x) || !inA.i32(y) || !inA.text(textA, sizeof(textA))) { + return fail("cannot read the addresses"); + } + uint32_t& count = (cluster & PSEUDO) != 0 ? pseudos[cluster & ~PSEUDO].addresses : clusters[cluster].addresses; + const uint32_t final = (cluster & PSEUDO) != 0 ? pseudos[cluster & ~PSEUDO].finalIndex : clusters[cluster].finalIndex; + if (count >= MAX_CLUSTER_ADDRESSES) { + rs.droppedAddresses++; + continue; + } + count++; + include(x, y); + rec.clear(); + rec.u32(final); + rec.u32(seq++); + rec.u8(flags); + rec.i32(x); + rec.i32(y); + rec.text(textA); + if (!sortAdd(sortA) || (seq % CHECK_EVERY == 0 && !keepGoing())) { + return false; + } + } + inA.close(); + addressTotal = seq; + rs.addresses = seq; + if (!sortA.finish()) { + return fail(sortA.errorText()); + } + // Numbers as the records store them; text numbers into the overflow strings in this order (the writer's) + if (!sortBegin(sortB, "s_a2", compareAddressNumber) || !outA.open(files, file("o.tmp"))) { + return fail("cannot start the address sort"); + } + HashIndex overflow; + GrowArray overflowHashes; + GrowArray overflowOffsets; + const uint8_t* data = nullptr; + uint32_t size = 0; + while (sortA.next(data, size)) { + Cursor c(data, size); + const uint32_t final = c.u32(); + const uint32_t order = c.u32(); + const uint32_t flags = c.u8(); + const int32_t x = c.i32(); + const int32_t y = c.i32(); + const char* text = c.text(); + uint16_t number = 0; + uint16_t aux = 0; + uint8_t suffix = 0; + uint8_t numberFlags = 0; + if (!indexHouseNumber(text, slashSecond, number, aux, suffix, numberFlags)) { + const uint64_t hash = mix(hashText(SEED, text)); + uint32_t at = overflow.find(hash, [&](uint32_t k) { return overflowHashes[k] == hash; }); + uint32_t offset = 0; + if (at != HashIndex::NONE) { + offset = overflowOffsets[at]; + } else { + offset = overflowBytes; + overflow.insert(hash, static_cast(overflowHashes.size())); + overflowHashes.push(hash); + overflowOffsets.push(offset); + const auto length = static_cast(strlen(text)); + if (!outA.write(text, length + 1)) { + return fail("cannot write the overflow strings"); + } + overflowBytes += length + 1; + overflowCount++; + } + if (offset >= SIDX_MAX_OFFSET24) { + return fail("the overflow strings pass 16 MB"); + } + number = 0xFFFF; + aux = static_cast(offset & 0xFFFF); + suffix = static_cast(offset >> 16); + numberFlags = HOUSE_TEXT; + } + rec.clear(); + rec.u32(final); + rec.u32(number); + rec.u32(aux); + rec.u32(suffix); + rec.u32(order); + rec.u8(numberFlags | flags); + rec.i32(x); + rec.i32(y); + if (!sortAdd(sortB)) { + return false; + } + } + if (sortA.failed()) { + return fail(sortA.errorText()); + } + noteSort(sortA); + sortA.end(); + overflow.release(); + if (!outA.close() || !sortB.finish()) { + return fail("cannot write the overflow strings"); + } + st->scratchBytes += outA.size(); + st->phaseMs[4] = nowMs() - started; + step(); + return true; + } + + /** The address and street records, once the origin is known: d.tmp and t.tmp. */ + bool writeAddressRecords(uint32_t pts) { + if (!outA.open(files, file("d.tmp"))) { + return fail("cannot create the address records"); + } + const uint8_t* data = nullptr; + uint32_t size = 0; + uint8_t bytes[32]; + uint32_t n = 0; + while (sortB.next(data, size)) { + Cursor c(data, size); + c.u32(); + const uint32_t number = c.u32(); + const uint32_t aux = c.u32(); + const uint32_t suffix = c.u32(); + c.u32(); + const uint32_t flags = c.u8(); + const int32_t x = c.i32(); + const int32_t y = c.i32(); + const uint32_t length = encodeAddress(bytes, pts, static_cast(number), static_cast(aux), static_cast(suffix), + static_cast(flags), x - minX, y - minY); + if (!outA.write(bytes, length)) { + return fail("cannot write the address records"); + } + if (++n % CHECK_EVERY == 0 && !keepGoing()) { + return false; + } + } + if (sortB.failed()) { + return fail(sortB.errorText()); + } + noteSort(sortB); + sortB.end(); + if (!outA.close() || !outB.open(files, file("t.tmp"))) { + return fail("cannot write the address records"); + } + st->scratchBytes += outA.size(); + uint32_t firstAddress = 0; + for (uint32_t street : streetNgOrder) { + const StreetInfo& s = streets[street]; + auto put = [&](int32_t rx, int32_t ry, int32_t lx, int32_t ly, int32_t hx, int32_t hy, uint32_t count) { + const int32_t rep[2] = { rx - minX, ry - minY }; + const int32_t lo[2] = { lx - minX, ly - minY }; + const int32_t hi[2] = { hx - minX, hy - minY }; + const uint32_t length = encodeStreet(bytes, pts, rep, lo, hi, firstAddress, static_cast(count)); + firstAddress += count; + return outB.write(bytes, length); + }; + for (uint32_t c = 0; c < s.clusterCount; c++) { + const ClusterInfo& comp = clusters[s.firstCluster + c]; + if (!put(comp.repX, comp.repY, comp.minX, comp.minY, comp.maxX, comp.maxY, comp.addresses)) { + return fail("cannot write the street records"); + } + } + // The villages, in their final order (finalIndex follows the street's real clusters) + for (uint32_t k = 0; k < s.pseudoCount; k++) { + const uint32_t wanted = s.firstRecord + s.clusterCount + k; + for (const PseudoInfo& p : pseudos) { + if (p.street == street && p.finalIndex == wanted) { + const SettlementInfo& place = settlements[p.settlement]; + if (!put(place.x, place.y, place.x, place.y, place.x, place.y, p.addresses)) { + return fail("cannot write the street records"); + } + break; + } + } + } + } + if (!outB.close()) { + return fail("cannot write the street records"); + } + st->scratchBytes += outB.size(); + return true; + } + + // ------------------------------------------------------------ 6. name groups, point records, keys + + bool pointGroups() { + const uint32_t started = nowMs(); + // Named points by (name, pair, seq); unnamed ones straight to their record list + if (!sortBegin(sortA, "s_grp", compareGroups) || !inA.open(files, file("p.tmp")) || !outA.open(files, file("u.tmp"))) { + return fail("cannot read the points"); + } + for (uint32_t i = 0; i < pointCount; i++) { + uint32_t label = 0, named = 0, hasLatin = 0, hasEn = 0, useEn = 0, seq = 0; + int32_t x = 0, y = 0; + if (!inA.u32(label) || !inA.u8(named) || !inA.text(textA, sizeof(textA)) || !inA.text(textB, sizeof(textB)) || !inA.u8(hasLatin) || + !inA.text(textC, sizeof(textC)) || !inA.u8(hasEn) || !inA.u8(useEn) || !inA.i32(x) || !inA.i32(y) || !inA.u32(seq)) { + return fail("cannot read the points"); + } + const uint32_t pair = labels[label].pair; + rec.clear(); + if (named == 0) { + rec.u8(pair); + rec.i32(x); + rec.i32(y); + if (!outA.write(rec)) { + return fail("cannot write the points"); + } + continue; + } + rec.text(textA); + rec.u8(pair); + rec.u32(seq); + rec.i32(x); + rec.i32(y); + rec.text(textB); + rec.u8(hasLatin); + rec.text(textC); + rec.u8(hasEn); + rec.u8(useEn); + rec.u8(labels[label].category); + if (!sortAdd(sortA) || (i % CHECK_EVERY == 0 && !keepGoing())) { + return false; + } + } + inA.close(); + if (!outA.close() || !sortA.finish()) { + return fail("cannot sort the points"); + } + st->scratchBytes += outA.size(); + // Groups (name, pair): the head (first seen) gives the group its keys; members listed by group number + if (!sortBegin(sortB, "s_ng", compareNameGroups) || !outA.open(files, file("m.tmp"))) { + return fail("cannot start the name group sort"); + } + const uint8_t* data = nullptr; + uint32_t size = 0; + uint32_t group = NONE; + uint32_t lastPair = NONE; + while (sortA.next(data, size)) { + Cursor c(data, size); + const char* name = c.text(); + const uint32_t pair = c.u8(); + const uint32_t seq = c.u32(); + const int32_t x = c.i32(); + const int32_t y = c.i32(); + if (group == NONE || pair != lastPair || strcmp(name, textA) != 0) { + group = group == NONE ? 0 : group + 1; + lastPair = pair; + snprintf(textA, sizeof(textA), "%s", name); + const char* latin = c.text(); + const uint32_t hasLatin = c.u8(); + const char* en = c.text(); + const uint32_t hasEn = c.u8(); + const uint32_t useEn = c.u8(); + const uint32_t category = c.u8(); + KeyMaker::sortKeyOf(name, config.profile, keyScratch, sizeof(keyScratch)); + rec.clear(); + rec.text(keyScratch); + rec.u8(pair); + rec.u32(seq); + rec.u32(group); + rec.text(name); + rec.text(latin); + rec.u8(hasLatin); + rec.text(en); + rec.u8(hasEn); + rec.u8(useEn); + rec.u8(category); + if (!sortAdd(sortB)) { + return false; + } + } + rec.clear(); + rec.u32(group); + rec.u8(pair); + rec.i32(x); + rec.i32(y); + if (!outA.write(rec)) { + return fail("cannot write the group members"); + } + } + if (sortA.failed()) { + return fail(sortA.errorText()); + } + noteSort(sortA); + sortA.end(); + if (!outA.close() || !sortB.finish()) { + return fail("cannot sort the name groups"); + } + st->scratchBytes += outA.size(); + // Ranks and offsets of the point groups in the writer's order; n.tmp keeps them in that order + if (!sortBegin(sortA, "s_g2r", compareFirst) || !outA.open(files, file("n.tmp"))) { + return fail("cannot start the rank sort"); + } + uint32_t rank = 0; + uint32_t offset = 0; + while (sortB.next(data, size)) { + Cursor c(data, size); + c.text(); + c.u8(); + c.u32(); + const uint32_t g = c.u32(); + const char* name = c.text(); + const char* latin = c.text(); + const uint32_t hasLatin = c.u8(); + const char* en = c.text(); + const uint32_t hasEn = c.u8(); + const uint32_t useEn = c.u8(); + const uint32_t category = c.u8(); + rec.clear(); + rec.u32(g); + rec.u32(rank); + rec.u32(offset); + if (!sortAdd(sortA)) { + return false; + } + rec.clear(); + rec.text(name); + rec.text(latin); + rec.u8(hasLatin); + rec.text(en); + rec.u8(hasEn); + rec.u8(useEn); + rec.u8(category); + if (!outA.write(rec)) { + return fail("cannot write the name groups"); + } + offset += 1 + truncateName(name, strlen(name)) + 1 + 4 + 2; + rank++; + } + if (sortB.failed()) { + return fail(sortB.errorText()); + } + noteSort(sortB); + sortB.end(); + pointNgCount = rank; + pointNgBytes = offset; + if (!outA.close() || !sortA.finish()) { + return fail("cannot sort the ranks"); + } + st->scratchBytes += outA.size(); + // Point records: members get their group's rank and offset (both lists in group order) + if (!sortBegin(sortB, "s_rec", comparePointRecords) || !inA.open(files, file("m.tmp"))) { + return fail("cannot start the point record sort"); + } + uint32_t currentGroup = NONE; + uint32_t currentRank = 0; + uint32_t currentOffset = 0; + for (uint32_t i = 0; i < namedCount; i++) { + uint32_t g = 0, pair = 0; + int32_t x = 0, y = 0; + if (!inA.u32(g) || !inA.u8(pair) || !inA.i32(x) || !inA.i32(y)) { + return fail("cannot read the group members"); + } + while (currentGroup != g) { + if (!sortA.next(data, size)) { + return fail("the group ranks are short"); + } + currentGroup = get32(data); + currentRank = get32(data + 4); + currentOffset = get32(data + 8); + } + if (!addPointRecord(pair, 0, currentRank, x, y, currentOffset)) { + return false; + } + } + inA.close(); + noteSort(sortA); + sortA.end(); + if (!inA.open(files, file("u.tmp"))) { + return fail("cannot read the unnamed points"); + } + for (uint32_t i = namedCount; i < pointCount; i++) { + uint32_t pair = 0; + int32_t x = 0, y = 0; + if (!inA.u8(pair) || !inA.i32(x) || !inA.i32(y)) { + return fail("cannot read the unnamed points"); + } + if (!addPointRecord(pair, 1, 0, x, y, SIDX_UNNAMED)) { + return false; + } + } + inA.close(); + if (!sortB.finish()) { + return fail(sortB.errorText()); + } + st->phaseMs[5] = nowMs() - started; + step(); + return true; + } + + bool addPointRecord(uint32_t pair, uint32_t unnamed, uint32_t rank, int32_t x, int32_t y, uint32_t offset) { + rec.clear(); + rec.u32(pair); + rec.u32(unnamed); + rec.u32(rank); + rec.i32(y >> 12); + rec.i32(x >> 12); + rec.i32(x); + rec.i32(y); + rec.u32(offset); + return sortAdd(sortB); + } + + /** Point records (r.tmp) and the name group area (g.tmp): point groups, streets, categories. */ + bool writeGroups(uint32_t pts) { + const uint32_t started = nowMs(); + if (!outA.open(files, file("r.tmp")) || !sortBegin(sortA, "s_fc", compareFirst)) { + return fail("cannot create the point records"); + } + const uint8_t* data = nullptr; + uint32_t size = 0; + uint8_t bytes[SIDX_MAX_NAME + 16]; + uint32_t index = 0; + uint32_t runRank = NONE; + uint32_t runFirst = 0; + uint32_t runCount = 0; + auto flushRun = [&]() { + if (runRank == NONE) { + return true; + } + rec.clear(); + rec.u32(runRank); + rec.u32(runFirst); + rec.u32(runCount); + return sortAdd(sortA); + }; + while (sortB.next(data, size)) { + const uint32_t pair = get32(data); + const uint32_t unnamed = get32(data + 4); + const uint32_t rank = get32(data + 8); + const auto x = static_cast(get32(data + 20)); + const auto y = static_cast(get32(data + 24)); + const uint32_t offset = get32(data + 28); + const uint32_t length = encodePoint(bytes, pts, x - minX, y - minY, static_cast(pair), offset); + if (!outA.write(bytes, length)) { + return fail("cannot write the point records"); + } + if (pairCount[pair] == 0) { + pairStart[pair] = index; + } + pairCount[pair]++; + if (unnamed == 0) { + if (rank != runRank) { + if (!flushRun()) { + return false; + } + runRank = rank; + runFirst = index; + runCount = 0; + } + runCount++; + } + index++; + if (index % CHECK_EVERY == 0 && !keepGoing()) { + return false; + } + } + if (sortB.failed() || !flushRun()) { + return fail(sortB.failed() ? sortB.errorText() : "cannot sort the group runs"); + } + noteSort(sortB); + sortB.end(); + if (!outA.close() || !sortA.finish()) { + return fail("cannot write the point records"); + } + st->scratchBytes += outA.size(); + // The name group area + if (!outB.open(files, file("g.tmp")) || !inA.open(files, file("n.tmp"))) { + return fail("cannot create the name groups"); + } + uint32_t groupBytes = 0; + for (uint32_t r = 0; r < pointNgCount; r++) { + uint32_t hasLatin = 0, hasEn = 0, useEn = 0, category = 0; + if (!inA.text(textA, sizeof(textA)) || !inA.text(textB, sizeof(textB)) || !inA.u8(hasLatin) || !inA.text(textC, sizeof(textC)) || + !inA.u8(hasEn) || !inA.u8(useEn) || !inA.u8(category)) { + return fail("cannot read the name groups"); + } + if (!sortA.next(data, size) || get32(data) != r) { + return fail("the group runs do not match the groups"); + } + const uint32_t first = get32(data + 4); + const uint32_t count = get32(data + 8); + if (count > 0xFFFF) { + return fail("name group %u has more than 65535 records", r); + } + const uint32_t nameLength = truncateName(textA, strlen(textA)); + const uint32_t length = encodeNameGroup(bytes, textA, nameLength, SearchKind::Point, first, static_cast(count)); + if (!outB.write(bytes, length)) { + return fail("cannot write the name groups"); + } + groupBytes += length; + } + inA.close(); + noteSort(sortA); + sortA.end(); + for (uint32_t street : streetNgOrder) { + const StreetInfo& s = streets[street]; + const char* name = names.at(s.nameId); + const uint32_t nameLength = truncateName(name, strlen(name)); + const uint32_t length = encodeNameGroup(bytes, name, nameLength, SearchKind::Street, s.firstRecord, + static_cast(s.clusterCount + s.pseudoCount)); + if (!outB.write(bytes, length)) { + return fail("cannot write the name groups"); + } + groupBytes += length; + } + streetNgBytes = groupBytes - pointNgBytes; + uint32_t k = pointNgCount + static_cast(streets.size()); + for (const CategoryNg& c : categories) { + uint32_t count = 0; + for (uint32_t p = 0; p < c.pairCount; p++) { + count += pairCount[c.firstPair + p]; + } + if (count > 0xFFFF) { + return fail("name group %u has more than 65535 records", k); + } + const char* name = extra(c.name); + const uint32_t nameLength = truncateName(name, strlen(name)); + const uint32_t length = encodeNameGroup(bytes, name, nameLength, SearchKind::Category, pairStart[c.firstPair], static_cast(count)); + if (!outB.write(bytes, length)) { + return fail("cannot write the name groups"); + } + groupBytes += length; + k++; + } + ngTotal = k; + if (groupBytes >= SIDX_MAX_OFFSET24) { + return fail("the name groups take %u bytes, more than 16 MB", groupBytes); + } + if (!outB.close()) { + return fail("cannot write the name groups"); + } + st->scratchBytes += outB.size(); + st->phaseMs[5] += nowMs() - started; + return true; + } + + bool addKey(const char* key, uint32_t priority, uint32_t offset) { + const size_t length = strlen(key); + if (length == 0) { + return true; + } + const uint32_t cut = truncateKey(key, length); + rec.clear(); + rec.u8(cut); + rec.bytes(key, cut); + rec.u8(priority); + rec.u32(offset); + keyCount++; + return sortAdd(sortA); + } + + /** Keys of every name group, sorted, into key pages (k.tmp) and the fence table. */ + bool writeKeys() { + const uint32_t started = nowMs(); + if (!sortBegin(sortA, "s_key", compareKeys) || !inA.open(files, file("n.tmp"))) { + return fail("cannot start the key sort"); + } + uint32_t offset = 0; + for (uint32_t r = 0; r < pointNgCount; r++) { + uint32_t hasLatin = 0, hasEn = 0, useEn = 0, category = 0; + if (!inA.text(textA, sizeof(textA)) || !inA.text(textB, sizeof(textB)) || !inA.u8(hasLatin) || !inA.text(textC, sizeof(textC)) || + !inA.u8(hasEn) || !inA.u8(useEn) || !inA.u8(category)) { + return fail("cannot read the name groups"); + } + if (!keys.make(textA, hasLatin != 0 ? textB : nullptr, hasEn != 0 ? textC : nullptr, useEn != 0, config.profile)) { + return fail("no memory for the keys"); + } + for (uint32_t k = 0; k < keys.count(); k++) { + if (!addKey(keys.key(k), category, offset)) { + return false; + } + } + offset += 1 + truncateName(textA, strlen(textA)) + 1 + 4 + 2; + if (r % CHECK_EVERY == 0 && !keepGoing()) { + return false; + } + } + inA.close(); + for (uint32_t street : streetNgOrder) { + const StreetInfo& s = streets[street]; + const char* name = names.at(s.nameId); + if (!keys.make(name, extra(s.latin), extra(s.en), s.useEn != 0, config.profile)) { + return fail("no memory for the keys"); + } + for (uint32_t k = 0; k < keys.count(); k++) { + if (!addKey(keys.key(k), CAT_STREET, offset)) { + return false; + } + } + offset += 1 + truncateName(name, strlen(name)) + 1 + 4 + 2; + } + for (const CategoryNg& c : categories) { + if (!addKey(extra(c.sortKey), CAT_CATEGORY, offset)) { + return false; + } + const char* name = extra(c.name); + offset += 1 + truncateName(name, strlen(name)) + 1 + 4 + 2; + } + keys.release(); + if (!sortA.finish() || !outA.open(files, file("k.tmp"))) { + return fail("cannot sort the keys"); + } + uint8_t page[SIDX_PAGE_SIZE]; + uint32_t used = 0; + memset(page, 0, sizeof(page)); + const uint8_t* data = nullptr; + uint32_t size = 0; + while (sortA.next(data, size)) { + const uint32_t length = data[0]; + const uint8_t* key = data + 1; + const uint32_t ng = get32(data + 2 + length); + if (!encodeKeyPage(page, used, key, length, ng)) { + if (!outA.write(page, SIDX_PAGE_SIZE)) { + return fail("cannot write the key pages"); + } + pageCount++; + memset(page, 0, sizeof(page)); + used = 0; + if (!encodeKeyPage(page, used, key, length, ng)) { + return fail("a key does not fit a page"); + } + } + if (used == 1 + length + 3) { + fence.push(static_cast(length)); + for (uint32_t i = 0; i < length; i++) { + fence.push(key[i]); + } + fenceCount++; + } + } + if (sortA.failed()) { + return fail(sortA.errorText()); + } + if (used > 0) { + if (!outA.write(page, SIDX_PAGE_SIZE)) { + return fail("cannot write the key pages"); + } + pageCount++; + } + noteSort(sortA); + sortA.end(); + if (!outA.close() || fence.failed()) { + return fail("cannot write the key pages"); + } + st->scratchBytes += outA.size(); + st->phaseMs[6] = nowMs() - started; + step(); + return true; + } + + // ------------------------------------------------------------ 7. the file + + bool copyInto(OutFile& out, const char* name, uint32_t expected) { + if (expected == 0) { + return true; + } + if (!inA.open(files, file(name))) { + return fail("cannot read a section"); + } + uint8_t chunk[1024]; + uint32_t left = expected; + while (left > 0) { + const uint32_t n = std::min(left, sizeof(chunk)); + if (!inA.read(chunk, n) || !out.write(chunk, n)) { + inA.close(); + return fail("cannot copy a section"); + } + left -= n; + } + inA.close(); + return true; + } + + bool writeFile(const char* outPath, const char* temporaryPath, uint32_t pts) { + const uint32_t started = nowMs(); + GrowArray classBlob; + for (uint32_t pair = 0; pair < pairTotal; pair++) { + const char* label = pairTexts.at(pair); + const auto length = static_cast(strlen(label)); + uint8_t tail[8]; + put32(tail, pairStart[pair]); + put32(tail + 4, pairCount[pair]); + classBlob.push(length); + for (uint32_t i = 0; i < length; i++) { + classBlob.push(static_cast(label[i])); + } + for (uint8_t b : tail) { + classBlob.push(b); + } + } + const uint32_t addressBytes = addressTotal * addressRecordBytes(pts); + const uint32_t sizes[SIDX_SECTION_COUNT] = { + static_cast(classBlob.size()), + static_cast(fence.size()), + pageCount * SIDX_PAGE_SIZE, + pointNgBytes + streetNgBytes + categoryBytes(), + pointCount * pointRecordBytes(pts), + streetRecordTotal * streetRecordBytes(pts), + addressBytes, + overflowBytes, + }; + SearchHeader header = {}; + header.pointBytes = static_cast(pts); + header.originX = minX; + header.originY = minY; + header.classCount = pairTotal; + header.fenceCount = fenceCount; + header.pageCount = pageCount; + header.pointCount = pointCount; + header.streetCount = streetRecordTotal; + header.addressCount = addressTotal; + uint64_t offset = SIDX_HEADER_SIZE; + for (uint32_t i = 0; i < SIDX_SECTION_COUNT; i++) { + header.sectionOffset[i] = static_cast(offset); + header.sectionSize[i] = sizes[i]; + offset += sizes[i]; + } + if (offset > 0xFFFFFFFFull) { + return fail("the index is larger than 4 GB"); + } + uint8_t headerBytes[SIDX_HEADER_SIZE]; + encodeHeader(header, headerBytes); + OutFile& out = outB; + if (!out.open(files, temporaryPath)) { + return fail("cannot create the index file"); + } + bool ok = out.write(headerBytes, SIDX_HEADER_SIZE) && out.write(classBlob.begin(), static_cast(classBlob.size())) && + out.write(fence.begin(), static_cast(fence.size())); + ok = ok && copyInto(out, "k.tmp", sizes[2]) && copyInto(out, "g.tmp", sizes[3]) && copyInto(out, "r.tmp", sizes[4]) && + copyInto(out, "t.tmp", sizes[5]) && copyInto(out, "d.tmp", sizes[6]) && copyInto(out, "o.tmp", sizes[7]); + ok = out.close() && ok; + if (!ok) { + files->remove(temporaryPath); + return fail("cannot write the index file"); + } + if (!files->rename(temporaryPath, outPath)) { + files->remove(temporaryPath); + return fail("cannot rename the index file"); + } + SearchBuildStats& is = st->index; + is.totalBytes = static_cast(offset); + is.header = header; + is.keyCount = keyCount; + is.ngCount = ngTotal; + is.namedPoints = namedCount; + is.unnamedPoints = pointCount - namedCount; + is.overflowStrings = overflowCount; + is.collapsedPairs = collapsed; + is.categoryNotContiguous = notContiguous; + st->phaseMs[7] = nowMs() - started; + step(); + return true; + } + + uint32_t categoryBytes() const { + uint32_t bytes = 0; + for (const CategoryNg& c : categories) { + const char* name = extra(c.name); + bytes += 1 + truncateName(name, strlen(name)) + 1 + 4 + 2; + } + return bytes; + } + + void removeScratch() { + static const char* const NAMES[] = { "l.tmp", "h.tmp", "i.tmp", "p.tmp", "a.tmp", "o.tmp", "d.tmp", "t.tmp", "u.tmp", + "m.tmp", "n.tmp", "r.tmp", "g.tmp", "k.tmp" }; + inA.close(); + inB.close(); + outA.close(); + outB.close(); + sortA.end(); + sortB.end(); + for (const char* name : NAMES) { + files->remove(file(name)); + } + } +}; + +SearchStreamBuilder::SearchStreamBuilder() = default; + +SearchStreamBuilder::~SearchStreamBuilder() { + if (impl != nullptr) { + impl->~Impl(); + largeFree(impl); + } +} + +bool SearchStreamBuilder::build(ScratchFiles& files, const char* scratchDir, const StreamTile* tiles, uint32_t tileCount, StreamTileSource& source, + MvtDecoder& decoder, const char* outPath, const char* temporaryPath, const StreamBuildOptions& options) { + statistics = {}; + errorText[0] = '\0'; + wasCancelled = false; + if (impl != nullptr) { + impl->~Impl(); + largeFree(impl); + impl = nullptr; + } + if (tileCount == 0) { + snprintf(errorText, sizeof(errorText), "no tiles"); + return false; + } + void* memory = largeAlloc(sizeof(Impl)); + if (memory == nullptr) { + snprintf(errorText, sizeof(errorText), "no memory for the builder"); + return false; + } + impl = new (memory) Impl(); + Impl& b = *impl; + b.owner = this; + b.files = &files; + b.source = &source; + b.decoder = &decoder; + b.options = &options; + b.config = options.config; + b.st = &statistics; + snprintf(b.dir, sizeof(b.dir), "%s", scratchDir); + int32_t minY = tiles[0].y; + int32_t maxY = tiles[0].y; + for (uint32_t i = 0; i < tileCount; i++) { + TileInfo info = {}; + info.x = tiles[i].x; + info.y = tiles[i].y; + b.tileIndex.insert(mix(static_cast(static_cast(info.x)) << 32 | static_cast(info.y)), i); + b.tiles.push(info); + minY = std::min(minY, tiles[i].y); + maxY = std::max(maxY, tiles[i].y); + } + b.regionMpu = b.config.metresPerUnit > 0 ? b.config.metresPerUnit : metresPerUnitAtTileY((minY + maxY + 1) / 2.0); + b.progress.total = 2 * tileCount + 7; + bool ok = !b.tiles.failed() && !b.tileIndex.failed(); + ok = ok && b.readTiles() && b.dedupPoints() && b.filterHouses() && b.makeClusters() && b.matchHouses() && b.makeAddresses(); + uint32_t pts = 4; + if (ok) { + const int64_t span = std::max(static_cast(b.maxX) - b.minX, static_cast(b.maxY) - b.minY); + pts = span >= 65536 ? 8 : 4; + if (!b.anyPoint) { + b.minX = b.minY = 0; + } + } + ok = ok && b.writeAddressRecords(pts) && b.pointGroups() && b.writeGroups(pts) && b.writeKeys() && + b.writeFile(outPath, temporaryPath, pts); + b.noteTables(); + b.removeScratch(); + impl->~Impl(); + largeFree(impl); + impl = nullptr; + if (!ok && errorText[0] == '\0') { + snprintf(errorText, sizeof(errorText), "the build failed"); + } + return ok; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/SearchStream.h b/Apps/Maps/main/Source/search/SearchStream.h new file mode 100644 index 0000000..0d9f19c --- /dev/null +++ b/Apps/Maps/main/Source/search/SearchStream.h @@ -0,0 +1,118 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "../mvt/ByteInput.h" +#include "../mvt/MvtDecoder.h" +#include "../route/GrowArray.h" +#include "AddressMatch.h" +#include "ExternalSort.h" +#include "QueryParse.h" +#include "ScratchFiles.h" +#include "SearchExtract.h" +#include "SearchIndex.h" +#include "SearchRegion.h" + +#include +#include + +namespace maps { + +/** A z14 tile of the region being indexed. */ +struct StreamTile { + int32_t x; + int32_t y; +}; + +/** Gives the builder a tile's original vector tile (.pbf), one at a time. */ +class StreamTileSource { +public: + virtual ~StreamTileSource() = default; + /** The tile's bytes, or nullptr when it cannot be read. Stays valid until the next call. */ + virtual ByteInput* open(int32_t x, int32_t y) = 0; + /** Every tile was read: the decoder and the tile buffers are not needed any more (the device frees them). */ + virtual void tilesDone() {} +}; + +/** What the builder does now, for the progress line ("Building search index N/M"). */ +struct StreamProgress { + uint32_t done; + uint32_t total; +}; + +struct StreamBuildOptions { + RegionConfig config; + /** The buffer of the sorted runs (each sort; at most two sorts hold memory at once). */ + uint32_t sortBufferBytes = 64 * 1024; + /** Called between steps; returning false cancels the build (scratch files are removed). */ + bool (*keepGoing)(const StreamProgress& progress, void* context) = nullptr; + void* context = nullptr; +}; + +/** Measurements of one build. */ +struct StreamBuildStats { + RegionStats region; + SearchBuildStats index; + /** Time per phase in ms: tiles, points, clusters, matching, addresses, name groups and records, keys, file. */ + uint32_t phaseMs[8]; + /** The slowest tile's extraction and its matching, in ms, and which tile. */ + uint32_t slowestExtractMs; + StreamTile slowestExtractTile; + uint32_t slowestMatchMs; + StreamTile slowestMatchTile; + /** Bytes written to scratch files (runs and extracts), and the most memory a sort held. */ + uint64_t scratchBytes; + size_t sortPeakBytes; + /** The builder's own tables at their largest (lines, clusters, names, labels, settlements, tiles). */ + size_t tableBytes; + /** The house matcher's arrays at their largest (one tile, or one block of a dense tile). */ + size_t matcherPeakBytes; +}; + +/** + * Builds a search index (.sidx) for a set of z14 tiles with little memory (M6, built on the device in idle time). It + * makes the same file as SearchRegionBuilder + SearchIndexWriter (byte for byte; the host check compares them), but + * keeps only small tables in memory and puts everything that grows with the number of points, house numbers and keys + * through sorted runs on the card (ExternalSorter): + * + * 1. tiles: each tile is extracted (TileExtractor); its named lines and owned house numbers go to two scratch files, + * its points into a sort. Duplicate lines are found among the tile and its neighbours already read; a house number + * belongs to the tile it lies in (a copy in another tile's buffer is skipped), or, outside the region, to the first + * tile that has it; + * 2. points: duplicates (label, name, position) dropped, the first one kept; label counts for the class pairs; + * 3. street clusters: union-find over same-name lines whose vertex cells (150 m) touch, through a sort of the cells; + * 4. matching: per tile, its houses with the lines of the tile and its neighbours and the neighbours' houses for the + * vote (AddressMatcher), written as address records; + * 5. addresses: per street record, sorted by number; text numbers into the overflow strings; + * 6. name groups, point records and keys through sorts; 7. the file: header, tables and the sections, then a rename. + * + * Memory: the decoder's pools while extracting (about 260 KB), one tile's extract, two sort buffers, and per region + * about 12 bytes per line, 64 per street cluster and the street names. Platform free; files through ScratchFiles. + */ +class SearchStreamBuilder { +public: + SearchStreamBuilder(); + ~SearchStreamBuilder(); + SearchStreamBuilder(const SearchStreamBuilder&) = delete; + SearchStreamBuilder& operator=(const SearchStreamBuilder&) = delete; + + /** + * Builds the index of the tiles (in this order; the host tool sorts by x, then y) into outPath, written as + * temporaryPath and renamed. scratchDir must exist; the builder's files inside it are removed at the end. The + * decoder must be initialised (init()); it is used with its POI pools and shrunk again afterwards. + */ + bool build(ScratchFiles& files, const char* scratchDir, const StreamTile* tiles, uint32_t tileCount, StreamTileSource& source, + MvtDecoder& decoder, const char* outPath, const char* temporaryPath, const StreamBuildOptions& options); + + const StreamBuildStats& stats() const { return statistics; } + const char* error() const { return errorText; } + bool cancelled() const { return wasCancelled; } + +private: + struct Impl; + Impl* impl = nullptr; + StreamBuildStats statistics = {}; + char errorText[160] = {}; + bool wasCancelled = false; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/TextFold.cpp b/Apps/Maps/main/Source/search/TextFold.cpp new file mode 100644 index 0000000..b1b8370 --- /dev/null +++ b/Apps/Maps/main/Source/search/TextFold.cpp @@ -0,0 +1,506 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +// The study's text folding (host/search/fold.py) for search keys and queries. The character tables are made from +// fold.py by tools/make_fold_table.py (FoldTable.h). +#include "TextFold.h" + +#include "FoldTable.h" + +#include + +namespace maps { + +namespace { + +/** Code points of a text, at most this many are folded (names are short; longer texts are cut). */ +constexpr size_t MAX_POINTS = 256; + +template +const char* lookup(const FoldEntry (&table)[N], uint32_t codePoint) { + size_t low = 0; + size_t high = N; + while (low < high) { + const size_t middle = (low + high) / 2; + if (table[middle].codePoint < codePoint) { + low = middle + 1; + } else { + high = middle; + } + } + return low < N && table[low].codePoint == codePoint ? table[low].text : nullptr; +} + +bool isApostrophe(uint32_t c) { + for (uint32_t a : FOLD_APOSTROPHES) { + if (a == c) { + return true; + } + } + return false; +} + +bool isMark(uint32_t c) { + return (c >= 0xFE00 && c <= 0xFE0F) || c == 0x200D || (c >= 0x300 && c <= 0x36F) || (c >= 0x483 && c <= 0x489) || (c >= 0x1AB0 && c <= 0x1AFF) || (c >= 0x1DC0 && c <= 0x1DFF) || + (c >= 0x20D0 && c <= 0x20FF) || (c >= 0xFE20 && c <= 0xFE2F); +} + +/** Not a letter or digit for Python (isalnum) outside ASCII: FoldTable's ranges, emoji and symbols above U+FFFF. */ +bool isNonAsciiSeparator(uint32_t c) { + if (c >= 0x10000) { + return c >= 0x1F000 && c <= 0x1FAFF; + } + size_t low = 0; + size_t high = sizeof(FOLD_SEPARATORS) / sizeof(FOLD_SEPARATORS[0]); + while (low < high) { + const size_t middle = (low + high) / 2; + if (FOLD_SEPARATORS[middle][1] < c) { + low = middle + 1; + } else { + high = middle; + } + } + return low < sizeof(FOLD_SEPARATORS) / sizeof(FOLD_SEPARATORS[0]) && FOLD_SEPARATORS[low][0] <= c; +} + +bool isAsciiAlnum(uint32_t c) { + return (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); +} + +bool isLetter(uint32_t c) { + if (c < 0x80) { + return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); + } + return !isNonAsciiSeparator(c) && !isMark(c); +} + +/** Lowercase for Greek and Cyrillic (and ASCII); other scripts unchanged. */ +uint32_t lowerGreekCyrillic(uint32_t c) { + if (c >= 'A' && c <= 'Z') { + return c + 32; + } + if ((c >= 0x391 && c <= 0x3A9) && c != 0x3A2) { + return c + 0x20; + } + switch (c) { + case 0x386: + return 0x3AC; + case 0x388: + return 0x3AD; + case 0x389: + return 0x3AE; + case 0x38A: + return 0x3AF; + case 0x38C: + return 0x3CC; + case 0x38E: + return 0x3CD; + case 0x38F: + return 0x3CE; + default: + break; + } + if (c >= 0x410 && c <= 0x42F) { + return c + 0x20; + } + if (c >= 0x400 && c <= 0x40F) { + return c + 0x50; + } + if (((c >= 0x460 && c <= 0x481) || (c >= 0x48A && c <= 0x4BF) || (c >= 0x4D0 && c <= 0x52F)) && (c & 1) == 0) { + return c + 1; + } + if (c >= 0x4C1 && c <= 0x4CE && (c & 1) == 1) { + return c + 1; + } + return c; +} + +/** The text as code points. */ +size_t decodeAll(const char* text, uint32_t* points, size_t capacity) { + const size_t length = strlen(text); + size_t at = 0; + size_t count = 0; + while (at < length && count < capacity) { + uint32_t c; + const size_t used = utf8Decode(text + at, length - at, c); + if (used == 0) { + break; + } + at += used; + points[count++] = c; + } + return count; +} + +/** Writes text pieces with single spaces between words and none at the ends. */ +class SpacedWriter { +public: + SpacedWriter(char* out, size_t size) : out(out), size(size) { + if (size > 0) { + out[0] = '\0'; + } + } + void space() { pendingSpace = used > 0; } + void text(const char* piece) { + for (const char* p = piece; *p != '\0'; p++) { + if (*p == ' ') { + space(); + } else { + byte(*p); + } + } + } + void point(uint32_t c) { + if (c == ' ') { + space(); + return; + } + char buffer[5]; + const size_t n = utf8Append(c, buffer, sizeof(buffer), 0); + // A code point is written whole or not at all + if (used + (pendingSpace ? 1 : 0) + n + 1 > size) { + full = true; + return; + } + for (size_t i = 0; i < n; i++) { + byte(buffer[i]); + } + } + size_t finish() { + if (size > 0) { + out[used] = '\0'; + } + return used; + } + +private: + void byte(char c) { + if (full) { + return; + } + if (pendingSpace) { + if (used + 2 > size) { + full = true; + return; + } + out[used++] = ' '; + pendingSpace = false; + } + if (used + 1 >= size) { + full = true; + return; + } + out[used++] = c; + } + char* out; + size_t size; + size_t used = 0; + bool pendingSpace = false; + bool full = false; +}; + +void foldPointLatin(uint32_t c, SpacedWriter& writer) { + if (isApostrophe(c) || isMark(c)) { + return; + } + if (c >= 0xAC00 && c <= 0xD7A3) { + // Hangul syllables decompose into their jamo (NFD), as in fold.py + const uint32_t index = c - 0xAC00; + writer.point(0x1100 + index / (21 * 28)); + writer.point(0x1161 + (index % (21 * 28)) / 28); + if (index % 28 != 0) { + writer.point(0x11A7 + index % 28); + } + return; + } + const char* folded = lookup(FOLD_LATIN, c); + if (folded != nullptr) { + writer.text(folded); + } else if (c < 0x80) { + if (isAsciiAlnum(c)) { + writer.point(c); + } else { + writer.space(); + } + } else if (isNonAsciiSeparator(c)) { + writer.space(); + } else { + writer.point(lowerGreekCyrillic(c)); + } +} + +} // namespace + +size_t utf8Decode(const char* text, size_t length, uint32_t& codePoint) { + if (length == 0) { + codePoint = 0; + return 0; + } + const auto* u = reinterpret_cast(text); + size_t need; + uint32_t c; + if (u[0] < 0x80) { + codePoint = u[0]; + return 1; + } else if ((u[0] & 0xE0) == 0xC0) { + need = 2; + c = u[0] & 0x1F; + } else if ((u[0] & 0xF0) == 0xE0) { + need = 3; + c = u[0] & 0x0F; + } else if ((u[0] & 0xF8) == 0xF0) { + need = 4; + c = u[0] & 0x07; + } else { + codePoint = 0xFFFD; + return 1; + } + if (need > length) { + codePoint = 0xFFFD; + return 1; + } + for (size_t i = 1; i < need; i++) { + if ((u[i] & 0xC0) != 0x80) { + codePoint = 0xFFFD; + return 1; + } + c = (c << 6) | (u[i] & 0x3F); + } + codePoint = c; + return need; +} + +size_t utf8Append(uint32_t c, char* out, size_t size, size_t used) { + char bytes[4]; + size_t n; + if (c < 0x80) { + bytes[0] = static_cast(c); + n = 1; + } else if (c < 0x800) { + bytes[0] = static_cast(0xC0 | (c >> 6)); + bytes[1] = static_cast(0x80 | (c & 0x3F)); + n = 2; + } else if (c < 0x10000) { + bytes[0] = static_cast(0xE0 | (c >> 12)); + bytes[1] = static_cast(0x80 | ((c >> 6) & 0x3F)); + bytes[2] = static_cast(0x80 | (c & 0x3F)); + n = 3; + } else { + bytes[0] = static_cast(0xF0 | (c >> 18)); + bytes[1] = static_cast(0x80 | ((c >> 12) & 0x3F)); + bytes[2] = static_cast(0x80 | ((c >> 6) & 0x3F)); + bytes[3] = static_cast(0x80 | (c & 0x3F)); + n = 4; + } + if (used + n + 1 > size) { + return used; + } + memcpy(out + used, bytes, n); + used += n; + out[used] = '\0'; + return used; +} + +size_t foldLatin(const char* text, char* out, size_t size) { + uint32_t points[MAX_POINTS]; + const size_t count = decodeAll(text, points, MAX_POINTS); + SpacedWriter writer(out, size); + for (size_t i = 0; i < count; i++) { + // Python's lower(): a capital sigma ending a word is a final sigma + if (points[i] == 0x3A3) { + // Unicode's rule: a cased letter before it and none after it, skipping case-ignorable characters (". : ' ·") + auto ignorable = [](uint32_t c) { return c == '.' || c == ':' || c == '\'' || c == 0x2019 || c == 0xB7 || isMark(c); }; + size_t before = i; + while (before > 0 && ignorable(points[before - 1])) { + before--; + } + size_t after = i + 1; + while (after < count && ignorable(points[after])) { + after++; + } + const bool letterBefore = before > 0 && isLetter(points[before - 1]) && !(points[before - 1] >= '0' && points[before - 1] <= '9'); + const bool letterAfter = after < count && isLetter(points[after]) && !(points[after] >= '0' && points[after] <= '9'); + if (letterBefore && !letterAfter) { + writer.point(0x3C2); + continue; + } + } + foldPointLatin(points[i], writer); + } + return writer.finish(); +} + +size_t foldOriginal(const char* text, char* out, size_t size) { + uint32_t points[MAX_POINTS]; + const size_t count = decodeAll(text, points, MAX_POINTS); + SpacedWriter writer(out, size); + for (size_t i = 0; i < count; i++) { + const uint32_t c = points[i]; + if (isApostrophe(c) || isMark(c)) { + continue; + } + if (c >= 0x370 && c <= 0x52F) { + const char* folded = lookup(FOLD_ORIGINAL, c); + if (folded != nullptr) { + writer.text(folded); + } else if (isNonAsciiSeparator(c)) { + writer.space(); + } else { + writer.point(lowerGreekCyrillic(c)); + } + } else if (c < 0x80) { + if (isAsciiAlnum(c)) { + writer.point(lowerGreekCyrillic(c)); + } else { + writer.space(); + } + } else if (isNonAsciiSeparator(c)) { + writer.space(); + } else { + const char* lower = lookup(FOLD_LOWER, c); + if (lower != nullptr) { + writer.text(lower); + } else { + writer.point(c); + } + } + } + return writer.finish(); +} + +bool hasNonLatin(const char* text) { + const size_t length = strlen(text); + size_t at = 0; + while (at < length) { + uint32_t c; + const size_t used = utf8Decode(text + at, length - at, c); + if (used == 0) { + break; + } + if (c >= 0x370 && c <= 0x52F) { + return true; + } + at += used; + } + return false; +} + +size_t transliterate(const char* text, Translit profile, char* out, size_t size) { + uint32_t points[MAX_POINTS]; + size_t count = decodeAll(text, points, MAX_POINTS); + // Lowercase; Greek without tonos and with final sigma as sigma (FOLD_ORIGINAL does both for Greek) + for (size_t i = 0; i < count; i++) { + uint32_t c = lowerGreekCyrillic(points[i]); + if (c >= 0x370 && c <= 0x3FF) { + const char* plain = lookup(FOLD_ORIGINAL, c); + if (plain != nullptr) { + utf8Decode(plain, strlen(plain), c); + } + } + points[i] = c; + } + // Into Latin, then fold_latin over the result (as fold.py does) + char latin[4 * MAX_POINTS + 1]; + size_t used = 0; + latin[0] = '\0'; + auto appendText = [&](const char* piece) { + for (const char* p = piece; *p != '\0' && used + 1 < sizeof(latin); p++) { + latin[used++] = *p; + } + latin[used] = '\0'; + }; + auto wordStart = [&](size_t i) { return i == 0 || !isLetter(points[i - 1]); }; + auto isOneOf = [](uint32_t c, const char* utf8) { + size_t at = 0; + const size_t length = strlen(utf8); + while (at < length) { + uint32_t d; + at += utf8Decode(utf8 + at, length - at, d); + if (d == c) { + return true; + } + } + return false; + }; + const FoldEntry* table = profile == Translit::Ukrainian ? TRANSLIT_UK : (profile == Translit::Serbian ? TRANSLIT_SR : TRANSLIT_RU); + const size_t tableSize = profile == Translit::Ukrainian ? sizeof(TRANSLIT_UK) / sizeof(FoldEntry) + : (profile == Translit::Serbian ? sizeof(TRANSLIT_SR) / sizeof(FoldEntry) + : sizeof(TRANSLIT_RU) / sizeof(FoldEntry)); + auto tableText = [&](uint32_t c) -> const char* { + size_t low = 0; + size_t high = tableSize; + while (low < high) { + const size_t middle = (low + high) / 2; + if (table[middle].codePoint < c) { + low = middle + 1; + } else { + high = middle; + } + } + return low < tableSize && table[low].codePoint == c ? table[low].text : nullptr; + }; + for (size_t i = 0; i < count;) { + const uint32_t c = points[i]; + const uint32_t next = i + 1 < count ? points[i + 1] : 0; + if ((c >= 0x3B1 && c <= 0x3C9) || c == 0x3C2) { + // Greek digraphs (ELOT 743) + if (c == 0x3BF && next == 0x3C5) { // ου + appendText("ou"); + i += 2; + continue; + } + if ((c == 0x3B1 || c == 0x3B5) && next == 0x3C5) { // αυ, ευ + const uint32_t after = i + 2 < count ? points[i + 2] : 0; + appendText(c == 0x3B1 ? "a" : "e"); + appendText(after == 0 || isOneOf(after, "\xCE\xB8\xCE\xBA\xCE\xBE\xCF\x80\xCF\x83\xCF\x84\xCF\x86\xCF\x87\xCF\x88") ? "f" : "v"); + i += 2; + continue; + } + if (c == 0x3BC && next == 0x3C0) { // μπ + appendText(wordStart(i) ? "b" : "mp"); + i += 2; + continue; + } + if (c == 0x3BD && next == 0x3C4) { // ντ + appendText(wordStart(i) ? "d" : "nt"); + i += 2; + continue; + } + if (c == 0x3B3 && next == 0x3B3) { // γγ + appendText("ng"); + i += 2; + continue; + } + if (c == 0x3B3 && next == 0x3BA) { // γκ + appendText(wordStart(i) ? "g" : "nk"); + i += 2; + continue; + } + const char* latinText = tableText(c); + if (latinText != nullptr) { + appendText(latinText); + } else { + char one[5]; + utf8Append(c, one, sizeof(one), 0); + appendText(one); + } + i++; + continue; + } + if (c >= 0x400 && c <= 0x52F) { + if (c == 0x457 && profile == Translit::Ukrainian && wordStart(i)) { // ї at a word start + appendText("yi"); + } else { + const char* latinText = tableText(c); + appendText(latinText != nullptr ? latinText : ""); + } + i++; + continue; + } + char one[5]; + utf8Append(c, one, sizeof(one), 0); + appendText(one); + i++; + } + return foldLatin(latin, out, size); +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/search/TextFold.h b/Apps/Maps/main/Source/search/TextFold.h new file mode 100644 index 0000000..f4dbc75 --- /dev/null +++ b/Apps/Maps/main/Source/search/TextFold.h @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include +#include + +namespace maps { + +/** Transliteration profiles of the study (fold.py PROFILES): Russian, Ukrainian, Serbian; Greek is the same in all. */ +enum class Translit : uint8_t { Russian, Ukrainian, Serbian }; + +/** Decodes one UTF-8 code point at text[0..length); returns the bytes used (0 at the end, 1 for a bad byte, U+FFFD). */ +size_t utf8Decode(const char* text, size_t length, uint32_t& codePoint); +/** Appends a code point as UTF-8 at out[used] within size (room for the final 0 kept); returns the new used. */ +size_t utf8Append(uint32_t codePoint, char* out, size_t size, size_t used); + +/** + * The search keys' text form (fold.py fold_latin): lowercase, Latin diacritics removed (Štefánikova -> stefanikova, + * Łódź -> lodz, Đường -> duong), apostrophes dropped, punctuation to one space. Greek and Cyrillic are only lowercased. + * Returns the length written to out (0-terminated). + */ +size_t foldLatin(const char* text, char* out, size_t size); +/** fold_orig: Greek and Cyrillic kept in their script (Київ -> київ), tonos removed, final sigma, ё -> е, ґ -> г. */ +size_t foldOriginal(const char* text, char* out, size_t size); +/** Whether the text has a Greek or Cyrillic letter (U+0370..U+052F). */ +bool hasNonLatin(const char* text); +/** Greek and Cyrillic to Latin with the profile's table, then foldLatin (Київ -> kyiv, Αθήνα -> athina). */ +size_t transliterate(const char* text, Translit profile, char* out, size_t size); + +} // namespace maps diff --git a/Apps/Maps/main/Source/ui/BackgroundWork.cpp b/Apps/Maps/main/Source/ui/BackgroundWork.cpp index 8f9ba2c..5a473a4 100644 --- a/Apps/Maps/main/Source/ui/BackgroundWork.cpp +++ b/Apps/Maps/main/Source/ui/BackgroundWork.cpp @@ -22,7 +22,7 @@ const char* workName(WorkKind kind) { case WorkKind::CacheCleanup: return "Cleaning the cache"; case WorkKind::GpxImport: return "Importing GPX"; case WorkKind::MoveBrowsed: return "Moving browsed tiles"; - case WorkKind::AddressIndex: return "Building the address index"; + case WorkKind::AddressIndex: return "Building search index"; case WorkKind::SplitTiles: return "Splitting tiles"; default: return ""; } diff --git a/Apps/Maps/main/Source/ui/BackgroundWork.h b/Apps/Maps/main/Source/ui/BackgroundWork.h index 1864ec3..1f7feee 100644 --- a/Apps/Maps/main/Source/ui/BackgroundWork.h +++ b/Apps/Maps/main/Source/ui/BackgroundWork.h @@ -19,7 +19,7 @@ enum class WorkKind : uint8_t { CacheCleanup, // "Cleaning the cache" GpxImport, // "Importing GPX" MoveBrowsed, // "Moving browsed tiles" - AddressIndex, // "Building the address index" + AddressIndex, // "Building search index" (M6) SplitTiles, // "Splitting tiles" Count }; diff --git a/Apps/Maps/main/Source/ui/SearchSheet.cpp b/Apps/Maps/main/Source/ui/SearchSheet.cpp new file mode 100644 index 0000000..b20f820 --- /dev/null +++ b/Apps/Maps/main/Source/ui/SearchSheet.cpp @@ -0,0 +1,667 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#include "SearchSheet.h" + +#include +#include +#include +#include +#include + +namespace maps { + +namespace { + +constexpr int PADDING = 8; +constexpr int ROW_PADDING = 10; +constexpr const char* PLACEHOLDER = "Search places, streets, addresses"; +constexpr const char* CATEGORY_ENTRY = "Nearby by category"; + +/** Copies text, cut on a UTF-8 character boundary to fit size. */ +void copyText(char* target, size_t size, const char* source) { + if (size == 0) { + return; + } + const char* text = source != nullptr ? source : ""; + size_t length = strlen(text); + if (length >= size) { + length = size - 1; + while (length > 0 && (static_cast(text[length]) & 0xC0) == 0x80) { + length--; + } + } + memcpy(target, text, length); + target[length] = '\0'; +} + +} // namespace + +void SearchSheet::store(Slot& slot, const Row& row) { + copyText(slot.name, sizeof(slot.name), row.name); + copyText(slot.type, sizeof(slot.type), row.type); + copyText(slot.note, sizeof(slot.note), row.note); + copyText(slot.distance, sizeof(slot.distance), row.distance); + slot.bearing = row.bearing; +} + +void SearchSheet::open() { + value[0] = '\0'; + headerText[0] = '\0'; + resultCount = 0; + moreRows = false; + categoryView = false; + resultView = false; + selection = 0; + first = 0; + dragPixels = 0; + askedAt = -1; + focusField = true; + highlighted = typing; + visible = true; + relayout(); +} + +void SearchSheet::setTypingKeyboard(bool enabled) { + typing = enabled; + highlighted = enabled; + relayout(); +} + +void SearchSheet::setText(const char* text) { + copyText(value, sizeof(value), text); + categoryView = false; + resultView = false; + selection = 0; + first = 0; +} + +void SearchSheet::setHeader(const char* text) { + copyText(headerText, sizeof(headerText), text); + relayout(); +} + +void SearchSheet::setRows(const Row* rows, int count, bool more) { + const int kept = std::clamp(count, 0, MAX_ROWS); + for (int i = 0; i < kept; i++) { + store(results[i], rows[i]); + } + resultCount = kept; + moreRows = more && count <= MAX_ROWS; + selection = 0; + first = 0; + dragPixels = 0; + askedAt = -1; + highlighted = typing; + keepVisible(); +} + +void SearchSheet::appendRows(const Row* rows, int count, bool more) { + const int room = MAX_ROWS - resultCount; + const int kept = std::clamp(count, 0, room); + for (int i = 0; i < kept; i++) { + store(results[resultCount + i], rows[i]); + } + resultCount += kept; + moreRows = more && resultCount < MAX_ROWS; + keepVisible(); +} + +void SearchSheet::setRecent(const Row* rows, int count) { + recentCount = std::clamp(count, 0, MAX_RECENT); + for (int i = 0; i < recentCount; i++) { + store(recent[i], rows[i]); + } + keepVisible(); +} + +void SearchSheet::setCategories(const char* const* names, int count) { + categoryCount = std::clamp(count, 0, MAX_CATEGORIES); + for (int i = 0; i < categoryCount; i++) { + copyText(categories[i], sizeof(categories[i]), names[i]); + } + keepVisible(); +} + +void SearchSheet::showResults() { + categoryView = false; + resultView = true; + selection = 0; + first = 0; + highlighted = highlighted || typing; + keepVisible(); +} + +void SearchSheet::showCategories() { + resultView = false; + categoryView = true; + selection = 0; + first = 0; + highlighted = highlighted || typing; + keepVisible(); +} + +// --------------------------------------------------------------------------------------------------------------- + +int SearchSheet::entryCount() const { + if (categoryView) { + return categoryCount; + } + if (value[0] != '\0' || resultView) { + return resultCount; + } + return (categoryCount > 0 ? 1 : 0) + recentCount; +} + +SearchSheet::Source SearchSheet::sourceOf(int entry, int& index) const { + if (categoryView) { + index = entry; + return Source::Category; + } + if (value[0] != '\0' || resultView) { + index = entry; + return Source::Result; + } + const int offset = categoryCount > 0 ? 1 : 0; + if (entry < offset) { + index = -1; + return Source::Browse; + } + index = entry - offset; + return Source::Recent; +} + +const SearchSheet::Slot* SearchSheet::slotOf(int entry) const { + int index; + switch (sourceOf(entry, index)) { + case Source::Result: + return index >= 0 && index < resultCount ? &results[index] : nullptr; + case Source::Recent: + return index >= 0 && index < recentCount ? &recent[index] : nullptr; + default: + return nullptr; + } +} + +int SearchSheet::entryHeight(int entry) const { + const Slot* slot = slotOf(entry); + if (slot == nullptr || compact) { + return lineHeight + ROW_PADDING; + } + return 2 * lineHeight + 6 + (slot->note[0] != '\0' ? lineHeight : 0); +} + +int SearchSheet::rowHeight() const { + return std::max(8, entryHeight(first)); +} + +int SearchSheet::fitting(int firstEntry) const { + const int count = entryCount(); + int used = 0; + int rows = 0; + for (int e = firstEntry; e < count; e++) { + const int h = entryHeight(e); + if (used + h > listBottom - listTop) { + break; + } + used += h; + rows++; + } + return std::max(rows, 1); +} + +int SearchSheet::maxFirst() const { + int used = 0; + int f = entryCount(); + while (f > 0 && used + entryHeight(f - 1) <= listBottom - listTop) { + used += entryHeight(f - 1); + f--; + } + return std::min(f, std::max(0, entryCount() - 1)); +} + +void SearchSheet::keepVisible() { + const int count = entryCount(); + selection = std::clamp(selection, 0, std::max(0, count - 1)); + first = std::clamp(first, 0, maxFirst()); + if (selection < first) { + first = selection; + } + while (first < selection && selection >= first + fitting(first)) { + first++; + } +} + +void SearchSheet::relayout() { + barHeight = lineHeight + 12; + const int headerHeight = headerText[0] != '\0' ? lineHeight + 2 : 2; + listTop = barHeight + headerHeight; + const int hintHeight = typing && !compact ? lineHeight + 4 : 0; + listBottom = std::max(listTop + 1, frame.height - hintHeight); + keepVisible(); +} + +void SearchSheet::layout(int width, int height, Font& font, bool compactLayout) { + compact = compactLayout; + lineHeight = font.lineHeight(); + frame = { 0, 0, width, height }; + barHeight = lineHeight + 12; + backButton = { 0, 0, lineHeight + 16, barHeight }; + field = { backButton.width, 3, width - backButton.width - 6, lineHeight + 8 }; + relayout(); +} + +bool SearchSheet::wantsMore() const { + if (!moreRows || categoryView || value[0] == '\0' || resultCount == 0 || askedAt == resultCount) { + return false; + } + return selection >= resultCount - 3 || first + fitting(first) >= resultCount - 1; +} + +SearchSheet::Event SearchSheet::wantMoreIfNeeded(Event event) { + if (event.type == EventType::None && wantsMore()) { + askedAt = resultCount; + return { EventType::WantMore, -1 }; + } + return event; +} + +// --------------------------------------------------------------------------------------------------------------- + +void SearchSheet::drawArrow(Canvas& canvas, float cx, float cy, float degrees, uint16_t color) const { + const float radians = degrees * 3.14159265f / 180.0f; + const float s = sinf(radians); + const float c = cosf(radians); + // A triangle pointing up, turned clockwise + const float base[3][2] = { { 0.0f, -6.0f }, { 4.5f, 5.0f }, { -4.5f, 5.0f } }; + float px[3]; + float py[3]; + for (int i = 0; i < 3; i++) { + px[i] = cx + base[i][0] * c - base[i][1] * s; + py[i] = cy + base[i][0] * s + base[i][1] * c; + } + const int left = static_cast(floorf(cx)) - 8; + const int top = static_cast(floorf(cy)) - 8; + for (int y = top; y <= top + 16; y++) { + for (int x = left; x <= left + 16; x++) { + const float tx = static_cast(x) + 0.5f; + const float ty = static_cast(y) + 0.5f; + bool negative = false; + bool positive = false; + for (int i = 0; i < 3; i++) { + const int j = (i + 1) % 3; + const float cross = (px[j] - px[i]) * (ty - py[i]) - (py[j] - py[i]) * (tx - px[i]); + negative = negative || cross < 0.0f; + positive = positive || cross > 0.0f; + } + if (!(negative && positive)) { + canvas.fillRect(x, y, 1, 1, color); + } + } + } +} + +void SearchSheet::draw(Canvas& canvas, Font& font, const UiTheme& theme) { + // The map stays visible but dimmed behind the sheet, as behind the menus + canvas.fillRect(0, 0, canvas.width, canvas.height, theme.scrim, 90); + canvas.fillRect(frame.x, frame.y, frame.width, frame.height, theme.panel, theme.panelOpacity); + + // < at the top left, drawn as lines + const auto backX = static_cast(backButton.x + backButton.width / 2); + const auto backY = static_cast(field.y + field.height / 2); + canvas.line(backX + 3.0f, backY - 6.0f, backX - 3.0f, backY, 2.0f, theme.text); + canvas.line(backX - 3.0f, backY, backX + 3.0f, backY + 6.0f, 2.0f, theme.text); + + // The text field: the end of a long text stays in view; with the placeholder when empty + canvas.fillRoundRect(field.x, field.y, field.width, field.height, 4, focusField ? theme.accent : theme.border); + canvas.fillRoundRect(field.x + 1, field.y + 1, field.width - 2, field.height - 2, 3, theme.panel); + if (value[0] == '\0') { + drawClippedText(canvas, font, field.x + 4, field.y + 4, categoryView ? CATEGORY_ENTRY : PLACEHOLDER, field.width - 8, theme.muted); + if (focusField) { + canvas.fillRect(field.x + 4, field.y + 3, 2, field.height - 6, theme.accent, 255); + } + } else { + const char* shown = value; + while (*shown != '\0' && font.textWidth(shown) > field.width - 14) { + shown++; + while ((static_cast(*shown) & 0xC0) == 0x80) { + shown++; + } + } + font.drawText(canvas, field.x + 4, field.y + 4, shown, theme.text); + if (focusField) { + canvas.fillRect(field.x + 4 + font.textWidth(shown) + 1, field.y + 3, 2, field.height - 6, theme.accent, 255); + } + } + + // The note line + if (headerText[0] != '\0') { + drawClippedText(canvas, font, PADDING, barHeight + 1, headerText, frame.width - 2 * PADDING, theme.muted); + } + canvas.fillRect(0, listTop - 1, frame.width, 1, theme.border); + + // The rows that fit completely + const int count = entryCount(); + const int left = 4; + const int width = frame.width - 8; + int y = listTop; + int shownRows = 0; + for (int e = first; e < count; e++) { + const int h = entryHeight(e); + if (y + h > listBottom) { + break; + } + shownRows++; + int index; + const Source source = sourceOf(e, index); + const Slot* slot = slotOf(e); + const bool isHighlighted = highlighted && e == selection; + if (isHighlighted) { + canvas.fillRoundRect(left, y, width, h - 2, 5, theme.accent); + } + const uint16_t color = isHighlighted ? theme.accentText : theme.text; + const uint16_t quiet = isHighlighted ? theme.accentText : theme.muted; + const int textLeft = left + PADDING; + const int textRight = left + width - PADDING; + if (slot == nullptr) { + const char* label = source == Source::Browse ? CATEGORY_ENTRY : categories[index]; + const int textY = y + (h - 2 - lineHeight) / 2; + const int room = source == Source::Browse ? textRight - textLeft - 14 : textRight - textLeft; + drawClippedText(canvas, font, textLeft, textY, label, room, color); + if (source == Source::Browse) { + const auto cx = static_cast(textRight - 4); + const auto cy = static_cast(y + (h - 2) / 2); + canvas.line(cx - 3.0f, cy - 6.0f, cx + 3.0f, cy, 2.0f, color); + canvas.line(cx + 3.0f, cy, cx - 3.0f, cy + 6.0f, 2.0f, color); + } + } else { + const bool hasDistance = slot->distance[0] != '\0'; + const int distanceWidth = hasDistance ? font.textWidth(slot->distance) : 0; + const int distanceX = textRight - distanceWidth; + // The distance, with the arrow to its left + const int arrowCentreX = distanceX - 10; + const int rightLimit = hasDistance ? arrowCentreX - 8 : textRight; + if (compact) { + const int textY = y + (h - 2 - lineHeight) / 2; + if (hasDistance) { + font.drawText(canvas, distanceX, textY, slot->distance, color); + drawArrow(canvas, static_cast(arrowCentreX), static_cast(textY + lineHeight / 2), slot->bearing, color); + } + const int typeWidth = slot->type[0] != '\0' ? font.textWidth(slot->type) : 0; + const int reserved = typeWidth > 0 ? 8 + std::min(typeWidth, 56) : 0; + const int nameRoom = rightLimit - textLeft - reserved; + drawClippedText(canvas, font, textLeft, textY, slot->name, nameRoom, color); + const int nameEnd = textLeft + std::min(font.textWidth(slot->name), nameRoom); + const int typeRoom = rightLimit - nameEnd - 8; + if (typeWidth > 0 && typeRoom >= 24) { + drawClippedText(canvas, font, nameEnd + 8, textY, slot->type, typeRoom, quiet); + } + } else { + const int line1 = y + 3; + const int line2 = line1 + lineHeight; + drawClippedText(canvas, font, textLeft, line1, slot->name, textRight - textLeft, color); + drawClippedText(canvas, font, textLeft, line2, slot->type, rightLimit - textLeft, quiet); + if (hasDistance) { + font.drawText(canvas, distanceX, line2, slot->distance, quiet); + drawArrow(canvas, static_cast(arrowCentreX), static_cast(line2 + lineHeight / 2), slot->bearing, quiet); + } + if (slot->note[0] != '\0') { + drawClippedText(canvas, font, textLeft, line2 + lineHeight, slot->note, textRight - textLeft, quiet); + } + } + } + y += h; + } + + // A thin bar shows where the visible rows are in a longer list + if (shownRows > 0 && shownRows < count) { + const int trackHeight = y - listTop; + const int barLength = std::max(8, trackHeight * shownRows / count); + const int barTop = listTop + (trackHeight - barLength) * first / std::max(1, maxFirst()); + canvas.fillRect(frame.width - 3, barTop, 2, barLength, theme.muted); + } + + if (typing && !compact) { + const int hintY = frame.height - lineHeight - 2; + canvas.fillRect(0, hintY - 2, frame.width, 1, theme.border); + drawClippedText(canvas, font, PADDING, hintY, "Roll: choose Enter: open Esc: close", frame.width - 2 * PADDING, theme.muted); + } +} + +// --------------------------------------------------------------------------------------------------------------- + +void SearchSheet::append(uint32_t codepoint) { + char bytes[4]; + size_t count = 0; + if (codepoint < 0x80) { + bytes[count++] = static_cast(codepoint); + } else if (codepoint < 0x800) { + bytes[count++] = static_cast(0xC0 | (codepoint >> 6)); + bytes[count++] = static_cast(0x80 | (codepoint & 0x3F)); + } else { + bytes[count++] = static_cast(0xE0 | (codepoint >> 12)); + bytes[count++] = static_cast(0x80 | ((codepoint >> 6) & 0x3F)); + bytes[count++] = static_cast(0x80 | (codepoint & 0x3F)); + } + const size_t length = strlen(value); + if (length + count < sizeof(value)) { + memcpy(value + length, bytes, count); + value[length + count] = '\0'; + } +} + +bool SearchSheet::erase() { + size_t length = strlen(value); + if (length == 0) { + return false; + } + while (length > 0 && (static_cast(value[length - 1]) & 0xC0) == 0x80) { + length--; + } + if (length > 0) { + length--; + } + value[length] = '\0'; + return true; +} + +SearchSheet::Event SearchSheet::activate(int entry) { + int index; + switch (sourceOf(entry, index)) { + case Source::Browse: + showCategories(); + return {}; + case Source::Category: + return { EventType::OpenCategory, index }; + case Source::Recent: + return index < recentCount ? Event { EventType::OpenRecent, index } : Event {}; + case Source::Result: + return index < resultCount ? Event { EventType::OpenRow, index } : Event {}; + } + return {}; +} + +SearchSheet::Event SearchSheet::openSelected() { + if (entryCount() == 0) { + return {}; + } + // Without a highlight (touch) the first visible row is the one Enter means + if (!highlighted) { + selection = first; + } + return activate(selection); +} + +SearchSheet::Event SearchSheet::goBack() { + if (resultView) { + // A nearby list goes back to the categories + resultView = false; + categoryView = true; + selection = 0; + first = 0; + return {}; + } + if (categoryView) { + categoryView = false; + selection = 0; + first = 0; + return {}; + } + return { EventType::Close, -1 }; +} + +SearchSheet::Event SearchSheet::move(int rows) { + if (entryCount() == 0 || rows == 0) { + if (rows < 0) { + focusField = true; + } + return {}; + } + if (!highlighted) { + // The first step shows the highlight where the list is, it does not move yet + highlighted = true; + if (selection < first || selection >= first + fitting(first)) { + selection = first; + } + return wantMoreIfNeeded({}); + } + if (rows < 0 && selection == 0) { + focusField = true; + return {}; + } + focusField = false; + selection = std::clamp(selection + rows, 0, entryCount() - 1); + keepVisible(); + return wantMoreIfNeeded({}); +} + +SearchSheet::Event SearchSheet::scrollBy(int rows) { + first = std::clamp(first + rows, 0, maxFirst()); + return wantMoreIfNeeded({}); +} + +SearchSheet::Event SearchSheet::handleDrag(int dy) { + dragPixels -= dy; + Event result; + const int height = rowHeight(); + while (std::abs(dragPixels) >= height) { + const int step = dragPixels > 0 ? 1 : -1; + dragPixels -= step * height; + const Event event = scrollBy(step); + if (event.type != EventType::None) { + result = event; + } + } + return result; +} + +SearchSheet::Event SearchSheet::handleKey(uint32_t key) { + switch (key) { + case keys::ENTER: + case keys::YES: + return openSelected(); + case keys::ESCAPE: + case keys::NO: + return goBack(); + case keys::UP: + case keys::FOCUS_PREVIOUS: + return move(-1); + case keys::DOWN: + case keys::FOCUS_NEXT: + return move(1); + case keys::TAB: + focusField = !focusField; + if (!focusField) { + highlighted = true; + } + return {}; + case keys::BACKSPACE: + case keys::DELETE: + if (categoryView || resultView || value[0] == '\0') { + return goBack(); + } + erase(); + selection = 0; + first = 0; + highlighted = typing; + focusField = true; + return { EventType::TextChanged, -1 }; + default: + break; + } + if ((key >= 0x20 && key < 0x7F) || (key >= 0xA0 && key < 0x2000)) { + categoryView = false; + resultView = false; + const size_t before = strlen(value); + append(key); + focusField = true; + if (strlen(value) == before) { + return {}; + } + selection = 0; + first = 0; + highlighted = typing; + return { EventType::TextChanged, -1 }; + } + return {}; +} + +int SearchSheet::rowAt(int x, int y) const { + if (x < 0 || x >= frame.width || y < listTop || y >= listBottom) { + return -1; + } + int top = listTop; + const int count = entryCount(); + for (int e = first; e < count; e++) { + const int h = entryHeight(e); + if (top + h > listBottom) { + break; + } + if (y < top + h) { + return e; + } + top += h; + } + return -1; +} + +bool SearchSheet::entryRect(int entry, Rect& rect) const { + int top = listTop; + const int count = entryCount(); + for (int e = first; e < count; e++) { + const int h = entryHeight(e); + if (top + h > listBottom) { + break; + } + if (e == entry) { + rect = { 0, top, frame.width, h }; + return true; + } + top += h; + } + return false; +} + +void SearchSheet::hover(int entry) { + if (entry >= 0 && entry < entryCount()) { + selection = entry; + highlighted = true; + focusField = false; + } +} + +SearchSheet::Event SearchSheet::handleTap(int x, int y) { + if (backButton.contains(x, y)) { + return goBack(); + } + if (field.contains(x, y)) { + focusField = true; + return { EventType::FocusField, -1 }; + } + const int entry = rowAt(x, y); + if (entry >= 0) { + selection = entry; + return activate(entry); + } + return {}; +} + +} // namespace maps diff --git a/Apps/Maps/main/Source/ui/SearchSheet.h b/Apps/Maps/main/Source/ui/SearchSheet.h new file mode 100644 index 0000000..ba61d72 --- /dev/null +++ b/Apps/Maps/main/Source/ui/SearchSheet.h @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: GPL-3.0-or-later +#pragma once + +#include "Widgets.h" + +#include + +namespace maps { + +/** + * The search sheet: a full screen overlay over the dimmed map with a < at the top left, a text field with a cursor + * (placeholder "Search places, streets, addresses"), a note line under it, and a list of rows. It knows nothing about + * how the results are found: the caller passes plain row data and reacts to the events. + * + * Rows: the name, under it the type in plain words with the distance and a small arrow (rotated by the bearing) on the + * right, and an optional third line with a note ("Street estimated from the map", "Also: X, Y"). With the on-screen + * keyboard (compact layout, see layout) a row is one line: name, type, distance. + * + * Empty field: the list shows the entry "Nearby by category" (opens the category list inside the sheet) and the recent + * rows. A non-empty field shows the result rows. The caller answers every TextChanged event with setRows (an empty + * list when the text is too short to search): the sheet keeps showing the old rows until it does. + * + * Rows come in pages: setRows gives the first page, the sheet raises WantMore when the selection or the scroll position + * nears the end of a list that has more, and the caller answers with appendRows. At most MAX_ROWS rows are kept. + * + * Keys: typing edits the field (UTF-8, code points as Tactility sends them), Backspace deletes, Backspace on an empty + * field closes (in the category list: back), Esc closes (in the category list: back), Enter opens the selected row, + * Up/Down (the trackball) and focus keys move the selection, Tab moves the focus between the field and the list. + * Touch: a tap on a row opens it, dragging scrolls, a tap on the field asks for the keyboard (FocusField), < closes. + * A trackball in mouse mode uses rowAt/hover/move, as MapView does for its lists. + */ +class SearchSheet { +public: + static constexpr int TEXT_LENGTH = 64; + static constexpr int MAX_ROWS = 64; + static constexpr int MAX_RECENT = 8; + static constexpr int MAX_CATEGORIES = 16; + /** How many rows the caller gives at once. */ + static constexpr int PAGE_ROWS = 8; + + /** One row, plain texts (copied, the pointers need not outlive the call). Empty strings and nullptr are the same. */ + struct Row { + const char* name; + /** The type in plain words: Cafe, Street, Village, Address */ + const char* type; + /** Optional third line */ + const char* note; + /** "850 m", "2.4 km"; empty: no distance and no arrow */ + const char* distance; + /** Direction of the place in degrees, clockwise, 0 = up on the screen (the caller makes it relative to north-up or heading-up) */ + float bearing; + }; + + enum class EventType : uint8_t { + None, + /** The text changed: query again (from 3 characters) and call setRows */ + TextChanged, + /** A result row was chosen; index counts the rows given to setRows/appendRows */ + OpenRow, + /** A recent row was chosen; index counts the rows given to setRecent */ + OpenRecent, + /** A category was chosen; index counts the names given to setCategories */ + OpenCategory, + Close, + /** The list nears its end and has more: call appendRows */ + WantMore, + /** The field was tapped: show the on-screen keyboard (touch boards) */ + FocusField + }; + + struct Event { + EventType type = EventType::None; + int index = -1; + }; + + void open(); + void close() { visible = false; } + bool isOpen() const { return visible; } + /** Boards with a typing keyboard show key hints and highlight the selection from the start. */ + void setTypingKeyboard(bool value); + bool hasTypingKeyboard() const { return typing; } + + const char* text() const { return value; } + /** Replaces the text without an event (for example a recent search). */ + void setText(const char* text); + /** The line under the field; empty hides it. */ + void setHeader(const char* text); + const char* header() const { return headerText; } + + /** Replaces the result rows (the first page); selects the first. more: the caller has further rows. */ + void setRows(const Row* rows, int count, bool more); + void appendRows(const Row* rows, int count, bool more); + int rowCount() const { return resultCount; } + /** Rows shown with an empty field, after "Nearby by category". */ + void setRecent(const Row* rows, int count); + /** The names of the category list (12 at the moment). With none the entry "Nearby by category" is not shown. */ + void setCategories(const char* const* names, int count); + /** Shows the category list (the entry "Nearby by category" does this by itself). */ + void showCategories(); + /** Shows the result rows although no text is typed (a nearby list); Back returns to the categories. */ + void showResults(); + bool inCategories() const { return categoryView; } + /** True when the list nears its end, more rows exist and the caller was not asked yet for this many rows. */ + bool wantsMore() const; + + /** + * Fills width x height from the top left. height is the room above the on-screen keyboard when compact (then rows + * are one line, no hints), the whole screen height otherwise. + */ + void layout(int width, int height, Font& font, bool compact = false); + void draw(Canvas& canvas, Font& font, const UiTheme& theme); + const Rect& bounds() const { return frame; } + const Rect& fieldBounds() const { return field; } + bool fieldFocused() const { return focusField; } + + Event handleKey(uint32_t key); + /** A tap on the sheet (the caller sends taps on the on-screen keyboard to TouchKeyboard::tap and the key here). */ + Event handleTap(int x, int y); + /** A finger moved dy pixels (positive: down) while dragging on the list: scrolls whole rows, the selection stays. */ + Event handleDrag(int dy); + /** The end of a drag: forget the part of a row. */ + void endDrag() { dragPixels = 0; } + /** The selection moves by rows (trackball steps); the first step only shows the highlight. */ + Event move(int rows); + /** Scrolls the list by rows, the selection stays. */ + Event scrollBy(int rows); + /** The entry (a row of the list as shown) under a screen position, -1 for none. */ + int rowAt(int x, int y) const; + /** The pointer is over an entry: it becomes the highlighted one. */ + void hover(int entry); + int selected() const { return selection; } + int firstVisible() const { return first; } + int entryCount() const; + /** Pixels a scrolling step of one entry takes, for mapping pointer or drag movement to rows. */ + int rowHeight() const; + /** For tests: the rectangle of an entry when visible. */ + bool entryRect(int entry, Rect& rect) const; + +private: + struct Slot { + char name[64]; + /** The type line, with the settlement: "Street · Holice" */ + char type[72]; + char note[64]; + char distance[12]; + float bearing; + }; + enum class Source : uint8_t { Browse, Recent, Category, Result }; + + static void store(Slot& slot, const Row& row); + Source sourceOf(int entry, int& index) const; + const Slot* slotOf(int entry) const; + int entryHeight(int entry) const; + int fitting(int firstEntry) const; + int maxFirst() const; + void keepVisible(); + void relayout(); + Event activate(int entry); + Event openSelected(); + Event goBack(); + Event wantMoreIfNeeded(Event event); + void append(uint32_t codepoint); + bool erase(); + void drawArrow(Canvas& canvas, float cx, float cy, float degrees, uint16_t color) const; + + bool visible = false; + bool typing = false; + bool compact = false; + bool focusField = true; + bool highlighted = false; + bool categoryView = false; + /** Result rows without typed text (a nearby list). */ + bool resultView = false; + bool moreRows = false; + int askedAt = -1; + int selection = 0; + int first = 0; + int dragPixels = 0; + char value[TEXT_LENGTH] = ""; + char headerText[64] = ""; + Slot results[MAX_ROWS] {}; + int resultCount = 0; + Slot recent[MAX_RECENT] {}; + int recentCount = 0; + char categories[MAX_CATEGORIES][32] {}; + int categoryCount = 0; + + int lineHeight = 16; + int barHeight = 28; + int listTop = 0; + int listBottom = 0; + Rect frame {}; + Rect backButton {}; + Rect field {}; +}; + +} // namespace maps diff --git a/Apps/Maps/main/Source/ui/WaypointSheet.h b/Apps/Maps/main/Source/ui/WaypointSheet.h index 232b995..28e88b8 100644 --- a/Apps/Maps/main/Source/ui/WaypointSheet.h +++ b/Apps/Maps/main/Source/ui/WaypointSheet.h @@ -80,7 +80,7 @@ class WaypointSheet { int focus = 0; char value[NAME_LENGTH] = ""; char coordinateText[48] = ""; - char typeText[48] = ""; + char typeText[72] = ""; bool compactLayout = false; char distanceText[64] = ""; Rect frame {}; diff --git a/Apps/Maps/manifest.properties b/Apps/Maps/manifest.properties index cb3e5ce..b324e71 100644 --- a/Apps/Maps/manifest.properties +++ b/Apps/Maps/manifest.properties @@ -2,8 +2,8 @@ manifest.version=0.3 target.sdk=0.8.0-dev target.platforms=esp32s3,esp32p4 id=tactility.maps -version.name=0.4.0 -version.code=5 +version.name=0.5.0 +version.code=7 app.0.id=tactility.maps app.0.name=Maps app.0.binary=maps diff --git a/Apps/Maps/tools/fold.py b/Apps/Maps/tools/fold.py new file mode 100644 index 0000000..9ebe3a8 --- /dev/null +++ b/Apps/Maps/tools/fold.py @@ -0,0 +1,118 @@ +# SPDX-License-Identifier: GPL-3.0-or-later +"""Diacritic folding + Greek/Cyrillic transliteration for search keys (no deps).""" +import unicodedata, re + +SPECIAL = {'ł':'l','đ':'d','ø':'o','ß':'ss','æ':'ae','œ':'oe','ð':'d','þ':'th','ı':'i','ħ':'h','ŧ':'t','ŋ':'n','ə':'e','ij':'ij','ĸ':'k','ſ':'s'} +APOS = "'’ʼ`´ʻ" + +def _strip_marks(ch): + d = unicodedata.normalize('NFD', ch) + return ''.join(c for c in d if unicodedata.category(c) != 'Mn') + +def fold_latin(s): + """lowercase, strip diacritics from Latin(-like) letters, punctuation -> single space. + Greek/Cyrillic letters pass through (lowercased) - see fold_orig for those.""" + out = [] + for ch in s.lower(): + if ch in APOS or unicodedata.category(ch) == 'Mn': continue + if ch in SPECIAL: out.append(SPECIAL[ch]); continue + o = ord(ch) + if o < 128: + out.append(ch if ch.isalnum() else ' ') + elif ch.isalnum(): + if 0x370 <= o <= 0x52F: # Greek/Cyrillic: handled elsewhere + out.append(ch) + else: + out.append(_strip_marks(ch)) # latin w/ marks, vietnamese tones etc. + else: out.append(' ') + return re.sub(r'\s+', ' ', ''.join(out)).strip() + +def fold_orig(s): + """folded form kept in the ORIGINAL script (Greek/Cyrillic): lowercase, strip tonos/stress, final sigma, ё->е, ґ->г""" + out = [] + for ch in s.lower(): + if ch in APOS: continue + o = ord(ch) + if 0x370 <= o <= 0x3FF: + ch = 'σ' if ch == 'ς' else _strip_marks(ch) + elif 0x400 <= o <= 0x52F: + ch = {'ё':'е','ґ':'г'}.get(ch, ch) + ch = ''.join(c for c in ch if c != '́') + if ch.isalnum() or ch.isspace(): out.append(ch) + elif unicodedata.category(ch) == 'Mn': continue + else: out.append(' ') + return re.sub(r'\s+', ' ', ''.join(out)).strip() + +def has_nonlatin(s): + return any(0x370 <= ord(c) <= 0x52F for c in s) + +GREEK = dict(zip('αβγδεζηθικλμνξοπρστυφχψω', ['a','v','g','d','e','z','i','th','i','k','l','m','n','x','o','p','r','s','t','y','f','ch','ps','o'])) +GREEK['ς'] = 's' +CYR_BASE = {'а':'a','б':'b','в':'v','г':'g','д':'d','е':'e','ё':'yo','ж':'zh','з':'z','и':'i','й':'y','к':'k','л':'l','м':'m','н':'n','о':'o','п':'p','р':'r','с':'s','т':'t','у':'u','ф':'f','х':'kh','ц':'ts','ч':'ch','ш':'sh','щ':'shch','ъ':'','ы':'y','ь':'','э':'e','ю':'yu','я':'ya','і':'i','ї':'i','є':'ye','ґ':'g','ў':'u'} +CYR_UK = dict(CYR_BASE, **{'г':'h','и':'y','й':'y','ґ':'g','ї':'i','є':'ye','ю':'yu','я':'ya','ь':'','ъ':''}) +CYR_SR = dict(CYR_BASE, **{'ђ':'dj','ж':'z','ј':'j','љ':'lj','њ':'nj','ћ':'c','ч':'c','џ':'dz','ш':'s','х':'h','и':'i','ц':'c','ъ':'','ь':''}) +PROFILES = {'ru': CYR_BASE, 'uk': CYR_UK, 'sr': CYR_SR} + +def translit(s, profile='ru'): + """Greek (ELOT-743-ish) + Cyrillic -> Latin, then fold_latin.""" + tab = PROFILES[profile] + s = s.lower() + # greek: strip accents first, digraphs + g = [] + for ch in s: + if 0x370 <= ord(ch) <= 0x3FF: ch = 'σ' if ch == 'ς' else _strip_marks(ch) + g.append(ch) + s = ''.join(g) + out = []; i = 0; n = len(s) + def wordstart(i): return i == 0 or not s[i-1].isalpha() + while i < n: + ch = s[i]; nx = s[i+1] if i+1 < n else '' + if 0x3B1 <= ord(ch) <= 0x3C9 or ch == 'ς': + two = ch + nx + if two == 'ου': out.append('ou'); i += 2; continue + if two in ('αυ','ευ'): + nn = s[i+2] if i+2 < n else '' + out.append(('a' if ch == 'α' else 'e') + ('f' if (nn == '' or nn in 'θκξπστφχψ') else 'v')); i += 2; continue + if two == 'μπ': out.append('b' if wordstart(i) else 'mp'); i += 2; continue + if two == 'ντ': out.append('d' if wordstart(i) else 'nt'); i += 2; continue + if two == 'γγ': out.append('ng'); i += 2; continue + if two == 'γκ': out.append('g' if wordstart(i) else 'nk'); i += 2; continue + out.append(GREEK.get(ch, ch)); i += 1; continue + if 0x400 <= ord(ch) <= 0x52F: + if ch == 'ї' and profile == 'uk' and wordstart(i): out.append('yi') + else: out.append(tab.get(ch, '')) + i += 1; continue + out.append(ch); i += 1 + return fold_latin(''.join(out)) + +def search_keys(name, latin=None, en=None, profile='ru', use_en=False): + """set of folded keys for one feature name.""" + keys = [] + k = fold_latin(name) if not has_nonlatin(name) else None + if k is None: + keys.append(translit(name, profile)) + keys.append(fold_orig(name)) + if latin: keys.append(fold_latin(latin)) + else: + keys.append(k) + if latin and not has_nonlatin(latin): keys.append(fold_latin(latin)) + if use_en and en and not has_nonlatin(en): keys.append(fold_latin(en)) + seen = []; + for x in keys: + if x and x not in seen: seen.append(x) + return seen + +def suffix_keys(key, minlen=3): + """word-start suffix keys (so 'stefanika' finds 'milana rastislava stefanika')""" + w = key.split(' ') + out = [] + for i in range(1, len(w)): + if len(w[i]) >= minlen: out.append(' '.join(w[i:])) + return out + +if __name__ == '__main__': + tests = [('Štefánikova','stefanikova'),('Łódź','lodz'),('Đường','duong'),('Mäsiarska','masiarska'),('İstanbul','istanbul'),('Çankaya','cankaya'), + ('Székesfehérvár','szekesfehervar'),('Hlavná stanica','hlavna stanica'),('Straße','strasse'),('Ørsted','orsted'),('Nguyễn Thiếp','nguyen thiep'), + ('Wrocław','wroclaw'),('Győr','gyor'),('nám. Slobody','nam slobody'),('Ulica 29. augusta','ulica 29 augusta'),('Phố Hàng Bạc','pho hang bac')] + for a,b in tests: assert fold_latin(a)==b,(a,fold_latin(a),b) + print(translit('Київ','uk'), fold_orig('Київ'), translit('Αθήνα'), fold_orig('Αθήνα'), translit('Београд','sr'), translit('Ποσειδώνος'), translit('Μπακάλικο'), translit('Рейтарська вулиця','uk'), translit('Москва','ru'),translit('Ђорђа Станојевића','sr')) diff --git a/Apps/Maps/tools/make_fold_table.py b/Apps/Maps/tools/make_fold_table.py new file mode 100644 index 0000000..b39faaf --- /dev/null +++ b/Apps/Maps/tools/make_fold_table.py @@ -0,0 +1,139 @@ +# SPDX-License-Identifier: GPL-3.0-or-later +"""Writes main/Source/search/FoldTable.h: the character tables of fold.py as C++ arrays, made with Python's own +unicodedata so the device folds exactly as the study did. + +Usage: python tools/make_fold_table.py +Tables (each sorted by code point): + FOLD_LATIN: code point -> folded UTF-8 text, for every letter or digit up to U+24F and U+1E00..U+1EFF whose + fold_latin() result differs from itself (lowercasing, marks, SPECIAL letters), and for Greek and + Cyrillic capitals (lowercased; fold_latin keeps those scripts) + FOLD_ORIGINAL: code point -> text for Greek and Cyrillic in fold_orig() (lowercase, tonos removed, final sigma, + ё -> е, ґ -> г) + TRANSLIT_RU / _UK / _SR: lowercase Greek and Cyrillic letter -> Latin text, as translit() +""" +import os +import sys +import unicodedata + +HERE = os.path.dirname(os.path.abspath(__file__)) +sys.path.insert(0, HERE) +import fold # noqa: E402 + +OUT = os.path.join(HERE, "..", "main", "Source", "search", "FoldTable.h") +RANGES = [(0x41, 0x5A), (0xC0, 0xABFF), (0xD7A4, 0xFFFF)] + + +def c_string(text): + return '"' + "".join("\\x%02X" % b for b in text.encode("utf-8")) + '"' + + +def table(name, pairs): + lines = [f"constexpr FoldEntry {name}[] = {{"] + for cp, text in pairs: + lines.append(f" {{ 0x{cp:04X}, {c_string(text)} }},") + lines.append("};") + return "\n".join(lines) + + +def main(): + latin = [] + original = [] + for start, end in RANGES: + for cp in range(start, end + 1): + ch = chr(cp) + if not ch.isalnum(): + continue + # fold_latin works on lowercased text; a single letter's lowercase may be 1 char + folded = fold.fold_latin(ch) + if folded != ch and len(folded) <= 3: + latin.append((cp, folded)) + if 0x370 <= cp <= 0x52F: + o = fold.fold_orig(ch) + if o != ch and len(o) <= 2: + original.append((cp, o)) + # SPECIAL letters that are not alphanumeric by Python's rules are still letters there + for ch, text in fold.SPECIAL.items(): + if ord(ch) not in {cp for cp, _ in latin}: + latin.append((ord(ch), text)) + latin.sort() + original.sort() + # Plain lowercasing for fold_orig of Latin letters (marks kept there) + lower = [] + for start, end in RANGES: + for cp in range(start, end + 1): + ch = chr(cp) + low = ch.lower() + if low != ch and len(low) == 1: + lower.append((cp, low)) + elif len(low) == 2 and unicodedata.category(low[1]) == "Mn": + # İ -> i + combining dot; the folds drop the dot + lower.append((cp, low[0])) + lower.sort() + # Code points U+0080..U+FFFF that are neither letters nor digits for Python (isalnum) and no combining marks: + # the folds turn them into spaces. Stored as ranges. + separators = [] + start = None + for cp in range(0x80, 0x10000): + ch = chr(cp) + is_sep = not ch.isalnum() and unicodedata.category(ch) != "Mn" + if is_sep and start is None: + start = cp + elif not is_sep and start is not None: + separators.append((start, cp - 1)) + start = None + if start is not None: + separators.append((start, 0xFFFF)) + translit = {} + for profile in ("ru", "uk", "sr"): + rows = [] + letters = set(fold.GREEK) | set(fold.PROFILES[profile]) + table_ = fold.PROFILES[profile] + for ch in sorted(letters): + # The letter's own value, not its word-start form (ї at a word start is handled in code) + raw = fold.GREEK.get(ch) if ch in fold.GREEK else table_.get(ch, "") + rows.append((ord(ch), fold.fold_latin(raw) if raw else "")) + translit[profile] = sorted(rows) + parts = [ + "// SPDX-License-Identifier: GPL-3.0-or-later", + "// Made by tools/make_fold_table.py from fold.py and Python's unicodedata " + unicodedata.unidata_version + "; don't edit.", + "#pragma once", + "", + "#include ", + "", + "namespace maps {", + "", + "struct FoldEntry {", + " uint32_t codePoint;", + " const char* text;", + "};", + "", + table("FOLD_LATIN", latin), + "", + table("FOLD_ORIGINAL", original), + "", + table("FOLD_LOWER", lower), + "", + table("TRANSLIT_RU", translit["ru"]), + "", + table("TRANSLIT_UK", translit["uk"]), + "", + table("TRANSLIT_SR", translit["sr"]), + "", + "/** Ranges of U+0080..U+FFFF that fold to a space (not letters, digits or combining marks), sorted. */", + "constexpr uint32_t FOLD_SEPARATORS[][2] = {", + *[f" {{ 0x{a:04X}, 0x{b:04X} }}," for a, b in separators], + "};", + "", + "/** Apostrophes, dropped by both folds. */", + "constexpr uint32_t FOLD_APOSTROPHES[] = { " + ", ".join("0x%04X" % ord(c) for c in sorted(fold.APOS)) + " };", + "", + "} // namespace maps", + "", + ] + with open(OUT, "w", encoding="utf-8", newline="\n") as out: + out.write("\n".join(parts)) + print(f"{OUT}: separators {len(separators)} ranges, latin {len(latin)}, original {len(original)}, translit {[len(v) for v in translit.values()]}") + + +if __name__ == "__main__": + main()