Skip to content

Fix GPT-4o token IDs in the tokenizer package README - #7701

Merged
matouskozak merged 1 commit into
mainfrom
matouskozak-nuget-readme-error
Sep 9, 2026
Merged

Fix GPT-4o token IDs in the tokenizer package README#7701
matouskozak merged 1 commit into
mainfrom
matouskozak-nuget-readme-error

Conversation

@matouskozak

Copy link
Copy Markdown
Member

Description

Update the expected token IDs in src/Microsoft.ML.Tokenizers/PACKAGE.md to match TiktokenTokenizer.CreateForModel("gpt-4o").

#7360 changed the example from GPT-4 to GPT-4o but left the GPT-4 token IDs in the output comment. This change corrects only that comment. The token count and both trimming examples remain correct.

Validation

Ran the example with the published Microsoft.ML.Tokenizers version 3.0.0-preview.26457.2 and matching data packages. The new list matches GPT-4o output; the old list matches GPT-4 output.

Checklist

  • The title describes the change.
  • No new issue is needed for this one-line documentation correction. Related PR: Some tweaks to the Microsoft.ML.Tokenizers PACKAGE.md #7360.
  • The description explains the change and its cause.
  • No unit test change is needed; the example output was checked against the published package.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 9, 2026 12:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change is a verified one-line documentation correction with no code or behavioral impact.

Pull request overview

Updates the Microsoft.ML.Tokenizers package documentation to correct the expected token ID sequence for the gpt-4o example, aligning the commented output with what TiktokenTokenizer.CreateForModel("gpt-4o") actually produces.

Changes:

  • Corrected the sample EncodeToIds output comment for the gpt-4o tokenizer example in PACKAGE.md.
File summaries
File Description
src/Microsoft.ML.Tokenizers/PACKAGE.md Updates the documented token ID list for the gpt-4o example to match current tokenizer output.
Review details
  • Files reviewed: 1/1 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@matouskozak
matouskozak merged commit cb1c64e into main Sep 9, 2026
8 checks passed
@matouskozak
matouskozak deleted the matouskozak-nuget-readme-error branch September 9, 2026 14:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants