Skip to content

fix(delivery): preserve observed benchmark token usage - #547

Merged
Dong1017 merged 1 commit into
InferMatrixLab:mainfrom
Inference1:fix/preserve-observed-benchmark-tokens
Nov 21, 2026
Merged

Dong1017 merged 1 commit into
InferMatrixLab:mainfrom
Inference1:fix/preserve-observed-benchmark-tokens

Conversation

@Inference1

Copy link
Copy Markdown
Contributor

Governing Issue and delivery slice

Refs #536

Standalone delivery follow-up to the versioned metrics implementation in #539. The slice covers a native terminal receipt with an identity-bound coverage declaration and absent token counters.

Problem

When export --agent-job attaches a completed native observation with no usage field, the Benchmark report replaces the campaign's retained caller-exported token subtotal with unknown values. The artifact index still contains both sources, but the reported subtotal and its provenance disappear.

What

Keep the campaign's token projection when the selected native receipt has no usable counters. Continue indexing the attached native job and its receipts for audit.

Why needed

A user can attach a job to audit a benchmark even when its host did not export token counters. The report should continue to show the observed subtotal and mark complete session cost as unknown.

Design and implementation

Reuse terminal_token_usage for identity, interval and coverage checks. Apply its projection only when it contains an observed token total. An observed zero qualifies. The complete total still requires the existing full-scope checks, and the result and its rendered views continue to share one projection.

Knowledge authorship declaration

Not applicable. The patch touches standalone Apps delivery and its tests.

Checks and dependencies

  • Linux, Python 3.12.14. Delivery, standalone and paired Benchmark metrics, comparison reporting and import gates. 224 passed, 2 skipped, 43 subtests passed. The skipped tests require Playwright.
  • Linux, Python 3.11.18. 17 passed for focused native usage and terminal receipt cases.
  • The new regression uses a retained completed native receipt and an observed campaign subtotal. On the base it lost the 1,200-token subtotal. It now preserves the value, source and unknown complete total.
  • ruff check src tests tools and git diff --check passed.

Remaining work and Issue closure

This software regression is ready for independent review. The governing Issue's broader acceptance stays with its owner.

Only select native usage when its receipt supplies observed counters. Preserve the existing campaign subtotal and source when the attached receipt has no counters.

Agent-assisted-by: Codex (author)
Signed-off-by: Inference1 <3617237115@qq.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 9466bc8 and head 0d731b8 (full diff, first review). prior-reviews.md shows no formal reviews and no outstanding CHANGES_REQUESTED, so this Approve is the first independent review on this head.

Verified from the diff:

  • src/inferrsi/apps/delivery/benchmark_usage.py::attach_native_usage: usage now starts as None. The selected_native_usage flag is gone. usage is only set when the policy is supported, the snapshot is workspace-bound, and coverage.campaign_plan_digest equals control["plan_digest"]. In that case it comes from terminal_token_usage(...).
  • The native override of result["usage"]["session_tokens"] and the source rewrite now run only when usage is not None and usage["observed_tokens"] is not None. A terminal receipt with no counters therefore no longer replaces the campaign-level benchmark_metrics.token_usage and usage.session_tokens with an all-null native record. This matches the PR title.
  • The missing-receipt case with observed tokens still takes the native path. The existing test_single_benchmark_missing_terminal_receipt_preserves_observed_subtotal has observed_tokens == 1200, so it is not affected.
  • The new test test_single_benchmark_terminal_receipt_without_counters_preserves_campaign_usage deletes observation["usage"] and projects with the agent job. It asserts these results:
    • token_usage equals the original campaign projection.
    • observed_tokens == 1200 and total_tokens is None.
    • session_tokens is unchanged.
    • A native_agent_usage source is still recorded, because snapshot.include(native, ...) still runs unconditionally.
      This covers the regression path.

Not verified here:

  • I did not run the test suite, and no CI status was seeded.
  • The diff context ends a few lines after the changed block, so I could not see the rest of attach_native_usage. I assume nothing after the if dereferences usage when it is None. The existing unsupported-policy test (test_native_job_preserves_unsupported_policy_reason) exercises that path and should catch a None dereference. Please confirm it is green in CI.

Non-blocking: the repo ruleset still needs two eligible approvals. This is one vote.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

首次独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30…0d731b8efba7fb99859340e7b7484954256cbeb4,范围是 #536 的独立 Benchmark 导出。附加的原生回执仍被索引;缺少可用计数时,不再覆盖已有的 1,200 token 观察小计、来源或未知的完整总量。合法的 0 仍可选用,原有身份、范围和完整性门槛不变。

已阅读两个变更文件并在隔离环境对精确函数做了缺失、0、正数三种投影检查(使用快照替身),同时核对原生来源仍被索引。未跑完整导出套件或设备基准。本批准仅适用于这项导出修复,不代表 #536 的整体验收。

@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 21, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 9466bc8 and head 0d731b8 (full diff).

Verified:

  • attach_native_usage now applies terminal_token_usage only when the receipt has observed_tokens is not None (zero still qualifies). A completed native receipt with no usage field no longer overwrites the campaign's retained subtotal or its source; snapshot.include still indexes the job for audit.
  • The new test_single_benchmark_terminal_receipt_without_counters_preserves_campaign_usage covers that regression (1,200 observed tokens, total_tokens is None, native source still present).
  • I read the rest of attach_native_usage: when usage is None it is not dereferenced after the if. Identity, interval and completeness checks in terminal_token_usage are unchanged.

This Approve is for this delivery fix only, not an acceptance of #536 as a whole. Required checks were still running at review time; please keep them green before merge.

@Dong1017
Dong1017 merged commit 7a7ab84 into InferMatrixLab:main Nov 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants