Repository navigation
Conversation
|
@anxkhn thanks for finding and fixing this bug! /lgtm (as long as CI passes) @howardjohn can you kickoff CI? |
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
14ada54 to
62d0a56
Compare
|
Rebased onto main after #8 merged and pushed 62d0a56. The original reporting patch is unchanged, verified with git range-diff. Local validation passed: 9 reporting tests, 15 llm-d Python tests, all five shell harness suites, Bash syntax, ShellCheck, and the standalone-reference check. For the local shell harnesses I used fail-closed tool-presence stubs and mawk to match Ubuntu runner behavior. Thanks for the CI fix! |
|
@howardjohn the rebased commit is ready. The new CI run is waiting for maintainer approval: https://github.com/agentgateway/benchmarks/actions/runs/42320318143 . I cannot approve it from the fork account. Could you approve this run? DCO has passed. |
|
/lgtm cc: @howardjohn |
Summary
The llm-d report adapter checks stack metadata only in the first stage file of each repetition. A later stage can name a different backend or component version and still contribute metrics to a comparison labeled with the first stage's backend.
Check the existing canonical stack metadata in every stage file before aggregation. The error identifies the mismatched file. Metric definitions, duplicate-rate warm-up handling and output formats remain unchanged.
Regression fixtures cover backend drift in warm-up and measured stages, non-engine component drift, and drift in a later repetition. A consistent two-repetition fixture checks median output rates, warm-up exclusion, the backend label and Prism bundles. Five regression subcases failed with the original adapter and pass with the fix.
Validation
git diff --checkpassed.These checks ran locally on macOS with isolated Python 3.12.13. Shell tests use mocks and fail-closed tool-presence stubs. No cloud resources, GPUs or live clusters were provisioned.
The existing main-branch Inference tooling run fails during the shell-test step before Python tests. That hosted CI failure is separate from the passing local results above.