All pull requests

Pull requests

• Fix RoBERTa processing without offset trimming

  #88·rupayon123 opened Sep 24, 2026·

  ·
  1

• Fix BPE continuation prefix and end-of-word suffix placement

  #87·rupayon123 opened Sep 21, 2026·

  ·
  1

• Fix WordPiece UTF-8 offsets and character end boundaries

  #86·rupayon123 opened Sep 21, 2026·

  ·
  1

• Preserve source offsets through precompiled character expansion

  #85·rupayon123 opened Sep 21, 2026·

  ·

• perf: regexp2 pretokenizer, pooled buffers, offset-free fast encode path

  #80·aeryncaen opened Feb 23, 2026·

  ·
  2

• Support zero-width negative lookahead (?!regex)

  #60·nanmu42 opened Jul 12, 2025·

  ·
  1

  3

• Initialize DecoderBase when fetching Default Decoders

  #47·ahhcash opened Sep 22, 2024·

  ·

• Use go:embed for pretrained gpt2 tokenizer, add missing json

  #23·adayNU opened Aug 11, 2022·

  ·
  1