Skip to content

feat(benchmarks): filter the task list with --iterative - #553

Open
Inference1 wants to merge 1 commit into
InferMatrixLab:mainfrom
Inference1:feat/benchmarks-list-iterative
Open

Inference1 wants to merge 1 commit into
InferMatrixLab:mainfrom
Inference1:feat/benchmarks-list-iterative

Conversation

@Inference1

Copy link
Copy Markdown
Contributor

Governing Issue and delivery slice

Refs #549. Dong1017’s delegate accepted this scope and assigned implementation to Inference1.

inferrsi benchmarks list --iterative lists the installed tasks that ship optimization.py as a file at their package root. Both the CLI and comparison configuration use the same check in the Benchmark registry. Discovery reads the package entries without loading task code or probing devices.

Changed artifacts and evidence

The CLI preserves the ordering from builtin_tasks(). I built the wheel, installed it in a fresh environment with no optional dependencies, and ran both commands outside the checkout. Their JSON output is shown here with whitespace condensed.

$ inferrsi benchmarks list
["moe-forward-npu", "qwen21-denoise-npu", "qwen3-tts-npu"]
$ inferrsi benchmarks list --iterative
["moe-forward-npu", "qwen21-denoise-npu"]

CPU tests cover the existing list, filtered order, help text, and an empty match ([], exit 0). A temporary registry excludes a directory named optimization.py and includes a file that raises on import. The device-library discovery test also exercises the new flag.

Checks and dependencies

  • pytest tests/test_engineering_benchmarks.py tests/test_benchmark_operations.py tests/test_benchmark_compare.py — 73 passed.
  • ruff check src tests tools and git diff --check — passed.
  • Full pytest on macOS arm64, Python 3.12: base 9466bc8d had 5,776 passed, 118 failed and 33 skipped. The feature tree had 5,780 passed, 117 failed and 33 skipped. Every failing test also failed on the base. Most failures occur in test-service process cleanup on this host.

Knowledge authorship declaration

The change covers benchmark discovery, its CLI, tests and user guides. The protected knowledge-content declaration is inapplicable. OpenAI Codex assisted with implementation, tests and documentation.

Remaining work and Issue closure

Acceptance remains with Dong1017’s queue. The commands above and the empty-result regression provide the checks requested in #549.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 9466bc8 and head 9a4b587 (full diff, diffComplete: true). This is Dong-bot (authorized delegate) acting as one reviewer. No prior formal reviews and no outstanding CHANGES_REQUESTED are seeded for this head.

Verified from the diff:

  • registry.has_iterative_protocol(name) is a file-existence check only: task_resources(name).joinpath("optimization.py").is_file(). It never imports or executes task code, which matches the docs claim. Unknown names still go through task_resources validation.
  • benchmark_compare._validate_config now calls this helper instead of an inline Path(str(task_resources(task))) / "optimization.py" check. The condition and error message are unchanged, so there is one source of truth for "iterative" across the CLI and compare.
  • benchmarks list --iterative in apps/cli/__init__.py filters the already-sorted builtin_tasks() result. Plain list is unchanged. The flag is store_true on the list subparser with help text.
  • Tests cover the behaviour:
    • A directory named optimization.py is excluded.
    • Output order is stable (a-task, z-task).
    • An optimization.py that raises on execution does not break listing.
    • The packaged list is [moe-forward-npu, qwen21-denoise-npu].
    • list --help mentions --iterative.
    • The subprocess device-library import guard now also runs main(["benchmarks","list","--iterative"]).
  • Docs (engineering-benchmarks.md, operations.md, optimization-benchmarks.md) match the implementation. The TTS single-candidate note is kept, and the hard-coded task names in the prose are replaced by the command, with the current list still stated.

Not verified here (I can't run commands):

  • I did not execute the test suite or see CI results. Please confirm CI is green, especially test_discovery_does_not_import_device_libraries, since it now imports inferrsi.apps.cli in the subprocess.
  • I could not see other tests that might monkeypatch benchmark_compare.task_resources. Such a patch would no longer affect the validation path now that the helper lives in registry. This is a non-blocking check for the author.
  • No governing issue was linked in the seed, so I did not check accepted scope.

The merge ruleset still requires two eligible approvals. This Approve counts as one vote.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

首轮独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30 → 9a4b5871ac6220f86207e6be8a5b5335e7fb2952。实现符合 #549 的约定范围:CLI 与 comparison 共用 registry 的文件存在判断,默认列表仍按 builtin_tasks() 排序,session start 不变。文件存在并不代表协议或设备可用。

独立执行了当前提交的 registry、参数定义和列表分支,用临时资源验证默认排序、空结果、排除同名目录、包含导入即报错的文件(未执行该文件)和帮助说明。核对了已打包的 MoE、Qwen denoise 和 TTS 入口。没有独立重建 wheel、运行完整测试集或查看 CI。此批准针对代码改动,不替代 #549 的最终验收。

@Inference1

Copy link
Copy Markdown
Contributor Author

I checked the two points from the review on 9a4b5871. The device-library test runs benchmarks list --iterative in a fresh Python process, then checks that the five listed runtime libraries stayed unloaded. The comparison fixture patches both compare.task_resources and registry.task_resources to the same synthetic package. I reran those two tests and the import-raising protocol fixture together on CPU. All three passed. Discovery test · Comparison fixture

Could a maintainer approve the Ready CI run for this head? GitHub currently shows action_required and zero jobs. I’ll check the results as soon as it runs.

@zengchuang-hw, thank you for checking the resource edge cases. If your approval still stands, could you resubmit it with LGTM alone on the first line, as requested by the repository review guide?

Copy link
Copy Markdown
Contributor

The code approval still stands for 9a4b5871. The request to reformat it is separate from code review: this approval starts with PASS, so it does not meet the repository’s exact first-line LGTM requirement for merge. I haven’t resubmitted or approved the Ready workflow.

@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 21, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 9466bc8 and head 9a4b587 (full diff).

Verified against the accepted checklist in #549:

  • has_iterative_protocol in src/inferrsi/benchmark/registry.py is task_resources(name).joinpath("optimization.py").is_file() and does not import task code.
  • benchmarks list --iterative filters the already-sorted builtin_tasks() result; the unfiltered list is unchanged. session start is not touched.
  • benchmark_compare._validate_config uses the same helper instead of an inline Path(str(task_resources(...))) check; the error message is unchanged. task_resources remains used later in the file.
  • Tests cover the packaged inventory ["moe-forward-npu", "qwen21-denoise-npu"], help text, empty match, excluding a directory named optimization.py, including an import-raising file without executing it, and the device-library import guard now also runs list --iterative.
  • Docs in engineering-benchmarks.md, operations.md, and optimization-benchmarks.md match the implementation.

Accuracy and performance comparison evidence is not applicable (CLI discovery filter, docs, and tests; no model output or runtime path change). No redundant architecture, code, or test concerns.

Note: Ready lane CI (run 44891405026) is action_required with no jobs executed yet, so I have not independently confirmed lint/tests in CI. Merge should wait until that lane actually runs and is green.

@Inference1

Copy link
Copy Markdown
Contributor Author

@Dong1017, the two first-line LGTM approvals are now in for 9a4b5871. Could you approve the Ready CI run for this fork PR, or route it to the maintainer who handles Actions approvals? It still shows action_required with zero jobs. I’ll check the Linux results when they’re available.

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

I can't approve the Ready run for 9a4b5871. Dong-bot (authorized delegate) can only post reviews and comments, and has no access to Actions or workflow approval. A maintainer with Actions approval rights needs to click "Approve and run" on run 44891405026. It shows action_required with zero jobs because this is a fork PR. Please route it to whoever handles those approvals.

Review state on 9a4b5871 (base 9466bc8d)

  • There are three formal APPROVED reviews on the current head: Dong1017 (posted by Dong-bot), @amy-why-3459 and @zengchuang-hw. No CHANGES_REQUESTED is outstanding.
  • Only two of them open with an exact LGTM line: the Dong1017/Dong-bot review and @amy-why-3459's. @zengchuang-hw's review still opens with PASS, and they declined to resubmit it. So the two first-line LGTM approvals are Dong-bot's and @amy-why-3459's, not @zengchuang-hw's. Check that against the merge rules rather than assume it.

Diff check

  • I re-read the full diff (diffComplete: true) and found no blockers.
  • has_iterative_protocol is task_resources(name).joinpath("optimization.py").is_file(). It never imports task code.
  • benchmark_compare._validate_config now calls that helper, and its error message is unchanged. The CLI and comparison share one check.
  • benchmarks list --iterative filters the already-sorted builtin_tasks() result. Plain list and session start are unchanged.
  • The tests cover the empty match, the directory named optimization.py, the import-raising file, help text, the packaged list, and the device-library import guard.
  • I did not run the tests or CI. The pytest and ruff results come from the PR description.

Next steps

@Inference1

Copy link
Copy Markdown
Contributor Author

@jiangkuaixue123, Dong-bot asked me to route the fork workflow approval. Could you approve Ready CI for 9a4b5871 when you have a chance? The run is waiting to start. Thanks!

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technical / third-party verify (cursor App permission none — not write+ / dual / merge-ready).

Tip already has Dong1017 always-on + amy + zengchuang Approves — no additional tip Approve from cursor this round (dual-clean tip unchanged).

Spot-checked tip diff Refs #549:

  • has_iterative_protocol is file-presence only (optimization.py.is_file()), no task import / device probe
  • CLI --iterative filters builtin_tasks() through that helper; compare config shares the same check
  • Tests cover empty filter, directory-named optimization.py rejected, and help text

Fork Ready lane was stuck in action_required; tip Ready run was approved so required checks can complete. Await Ready green before steward merge. Cursor does not merge (permission none).

@Inference1

Copy link
Copy Markdown
Contributor Author

Ready CI finished for 9a4b5871. Linux 3.10 and Linux 3.11 each reported 5,898 passed, 31 skipped and one failure.

Both failed in test_recovered_attempt_does_not_trust_a_rewritable_declaration. Its callback imports digest from the retired inferrsi.infra.artifacts module before it can raise the expected KeyboardInterrupt. The same test failed in another PR on the same base, and upstream is tracking it in #555. Both installed-package jobs, the dashboard browser test, the PR references check and both advisory host checks passed.

I’ll follow the upstream fix in #556 and verify this PR against the corrected base once it lands.

Share the packaged protocol-file check with comparison configuration so both commands discover the same tasks without loading task code.

Agent-assisted-by: OpenAI Codex (implementation, tests and documentation)
Signed-off-by: Inference1 <3617237115@qq.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technical / third-party tip COMMENT (cursor App permission none — not re-Approve; tip already dual-clean Dong + amy + zeng).

Ready blocker: same mainline residue — Ready full-check FAIL on test_recovered_attempt_does_not_trust_a_rewritable_declaration / retired inferrsi.infra.artifacts.digest (run 44891405026). Unrelated to --iterative filter change.

Unblock: #556 MERGED. Rebase/update onto main + Ready re-run, then write+ steward merge when green.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head d9e36ebea5fcd3e9021e86d216df63d2347fde6d (tip moved after prior COMMENT; re-verified).

Technical / third-party verify only (cursor App collaborator permission none — not write+ eligible / dual-tip / merge-ready).

Checked

  • Shared has_iterative_protocol = packaged root optimization.py file presence; used by benchmarks list --iterative and compare config validation (no task-code import).
  • Tests: directory-named optimization.py excluded; file presence lists sorted tasks without executing protocol; engineering list returns moe/qwen only; discovery still avoids device libs.
  • Docs: operations / engineering / optimization-benchmarks updated for --iterative.
  • Base includes merged #556 digest fix; Ready full-check still pending at review time — steward merge after green.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

Incremental review: 9a4b5871ac6220f86207e6be8a5b5335e7fb2952 → d9e36ebea5fcd3e9021e86d216df63d2347fde6d. Previous review. The two heads have the same tree and common base 9466bc8d; all nine PR files are unchanged. The target main moved to 10ba7a55, but this head does not contain that tip. This approval covers the #549 code change, not integration with current main.

I reran focused CPU checks using the pinned CLI branch and registry helper: the default list is unchanged; an empty match returns []; filtering preserves registry order, excludes a directory named optimization.py, and includes a file that would raise if imported without executing it. I also verified that comparison uses the same helper and that the parser declares --iterative. No new code blockers or non-blocking findings. I did not rerun the installed-wheel check or full suite, inspect CI, or test integration with current main.

@Inference1

Copy link
Copy Markdown
Contributor Author

All eight Ready CI jobs passed. The checkout step in both Linux logs confirms that CI tested 7bede3a2, the merge of PR head d9e36ebe with main at 10ba7a55, which includes #556. Python 3.10 and Python 3.11 each reported 5,905 passed and 31 skipped.

The nine-file feature diff is unchanged. I also reran the three benchmark test files and the previously failing recovery test against that combined tree locally: 74 passed. Could a maintainer squash-merge this PR when convenient? Thanks for the reviews and CI approval.

@Inference1

Copy link
Copy Markdown
Contributor Author

Ready CI passed on d9e36eb against main at 10ba7a5, which includes the recovery-test import fix from #556. GitHub tested merge commit 7bede3a.

Both Linux full-check jobs reported 5,905 passed and 31 skipped. All 10 PR checks passed. Python 3.10 · Python 3.11

The signed commit uses the file tree and parent from the reviewed feature commit. GitHub marks it Verified, and its sign-off is Signed-off-by: Inference1 <3617237115@qq.com>. The current head has qualifying LGTM approvals from Dong1017 and amy-why-3459, and GitHub reports APPROVED and CLEAN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants