Skip to content

reject mixed stack metadata within reporting repetitions - #7

Open
anxkhn wants to merge 1 commit into
agentgateway:mainfrom
anxkhn:fix/report-stage-stack-consistency
Open

anxkhn wants to merge 1 commit into
agentgateway:mainfrom
anxkhn:fix/report-stage-stack-consistency

Conversation

@anxkhn

@anxkhn anxkhn commented Oct 12, 2026 •

Copy link
Copy Markdown

Summary

The llm-d report adapter checks stack metadata only in the first stage file of each repetition. A later stage can name a different backend or component version and still contribute metrics to a comparison labeled with the first stage's backend.

Check the existing canonical stack metadata in every stage file before aggregation. The error identifies the mismatched file. Metric definitions, duplicate-rate warm-up handling and output formats remain unchanged.

Regression fixtures cover backend drift in warm-up and measured stages, non-engine component drift, and drift in a later repetition. A consistent two-repetition fixture checks median output rates, warm-up exclusion, the backend label and Prism bundles. Five regression subcases failed with the original adapter and pass with the fix.

Validation

  • Reporting suite: 9 tests passed, including Markdown, PNG, CSV and Prism checks.
  • llm-d Python suite: 15 tests passed.
  • All five mocked shell harness suites passed.
  • ShellCheck and Bash syntax passed for all 15 shell files.
  • Python syntax passed for all 18 Python files.
  • Standalone repository reference check and git diff --check passed.

These checks ran locally on macOS with isolated Python 3.12.13. Shell tests use mocks and fail-closed tool-presence stubs. No cloud resources, GPUs or live clusters were provisioned.

The existing main-branch Inference tooling run fails during the shell-test step before Python tests. That hosted CI failure is separate from the passing local results above.

@danehans

Copy link
Copy Markdown
Contributor

@anxkhn thanks for finding and fixing this bug!

/lgtm (as long as CI passes)

@howardjohn can you kickoff CI?

@danehans

Copy link
Copy Markdown
Contributor

@anxkhn to fix the CI failure, please rebase after #8 merges.

@danehans

danehans commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@anxkhn PR #8 merged. Can you rebase and update your PR?

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@anxkhn
anxkhn force-pushed the fix/report-stage-stack-consistency branch from 14ada54 to 62d0a56 Compare November 7, 2026 00:46
@anxkhn

anxkhn commented Nov 7, 2026

Copy link
Copy Markdown
Author

Rebased onto main after #8 merged and pushed 62d0a56. The original reporting patch is unchanged, verified with git range-diff.

Local validation passed: 9 reporting tests, 15 llm-d Python tests, all five shell harness suites, Bash syntax, ShellCheck, and the standalone-reference check. For the local shell harnesses I used fail-closed tool-presence stubs and mawk to match Ubuntu runner behavior.

Thanks for the CI fix!

@anxkhn

anxkhn commented Nov 7, 2026

Copy link
Copy Markdown
Author

@howardjohn the rebased commit is ready. The new CI run is waiting for maintainer approval: https://github.com/agentgateway/benchmarks/actions/runs/42320318143 . I cannot approve it from the fork account. Could you approve this run? DCO has passed.

@danehans

Copy link
Copy Markdown
Contributor

/lgtm

cc: @howardjohn

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants