unicode-normalization test suite

This is a fork of unicode-rs/unicode-normalization specially tailored for benckmarking an LLM's spec conformability.

You would need to remove tests/, .git/, and optionally this file to run the test without adding additional source of information.

To run:

cargo test --test tests -- --no-capture 2>&1 | grep "pass rate"

to check the pass rate of an implementation.

Origianl unicode-normalization

Unicode Normalization Forms conformance workspace, implementing Unicode Standard Annex #15.

src/lib.rs declares the minimal public API — to_nfd, to_nfkd, to_nfc, to_nfkc, is_nfd, is_nfkd, is_nfc, is_nfkc — with todo!() bodies to be implemented. tests/tests.rs checks an implementation against the official Unicode NormalizationTest.txt conformance suite (tests/data/normalization_tests.rs).

use unicode_normalization::to_nfc;

fn main() {
    assert_eq!(to_nfc("A\u{30a}"), "\u{c5}");
}

Character property data (combining classes, decomposition/composition tables, etc.), generated from the Unicode Character Database by scripts/unicode.py, is available in src/tables.rs via the accessors in src/lookups.rs.

cargo test