Repository navigation
feat(benchmarks): filter the task list with --iterative - #553
Inference1 wants to merge 1 commit into
Conversation
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 9466bc8 and head 9a4b587 (full diff, diffComplete: true). This is Dong-bot (authorized delegate) acting as one reviewer. No prior formal reviews and no outstanding CHANGES_REQUESTED are seeded for this head.
Verified from the diff:
registry.has_iterative_protocol(name)is a file-existence check only:task_resources(name).joinpath("optimization.py").is_file(). It never imports or executes task code, which matches the docs claim. Unknown names still go throughtask_resourcesvalidation.benchmark_compare._validate_confignow calls this helper instead of an inlinePath(str(task_resources(task))) / "optimization.py"check. The condition and error message are unchanged, so there is one source of truth for "iterative" across the CLI and compare.benchmarks list --iterativeinapps/cli/__init__.pyfilters the already-sortedbuiltin_tasks()result. Plainlistis unchanged. The flag isstore_trueon thelistsubparser with help text.- Tests cover the behaviour:
- A directory named
optimization.pyis excluded. - Output order is stable (
a-task,z-task). - An
optimization.pythat raises on execution does not break listing. - The packaged list is
[moe-forward-npu, qwen21-denoise-npu]. list --helpmentions--iterative.- The subprocess device-library import guard now also runs
main(["benchmarks","list","--iterative"]).
- A directory named
- Docs (
engineering-benchmarks.md,operations.md,optimization-benchmarks.md) match the implementation. The TTS single-candidate note is kept, and the hard-coded task names in the prose are replaced by the command, with the current list still stated.
Not verified here (I can't run commands):
- I did not execute the test suite or see CI results. Please confirm CI is green, especially
test_discovery_does_not_import_device_libraries, since it now importsinferrsi.apps.cliin the subprocess. - I could not see other tests that might monkeypatch
benchmark_compare.task_resources. Such a patch would no longer affect the validation path now that the helper lives inregistry. This is a non-blocking check for the author. - No governing issue was linked in the seed, so I did not check accepted scope.
The merge ruleset still requires two eligible approvals. This Approve counts as one vote.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
zengchuang-hw
left a comment
There was a problem hiding this comment.
PASS
首轮独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30 → 9a4b5871ac6220f86207e6be8a5b5335e7fb2952。实现符合 #549 的约定范围:CLI 与 comparison 共用 registry 的文件存在判断,默认列表仍按 builtin_tasks() 排序,session start 不变。文件存在并不代表协议或设备可用。
独立执行了当前提交的 registry、参数定义和列表分支,用临时资源验证默认排序、空结果、排除同名目录、包含导入即报错的文件(未执行该文件)和帮助说明。核对了已打包的 MoE、Qwen denoise 和 TTS 入口。没有独立重建 wheel、运行完整测试集或查看 CI。此批准针对代码改动,不替代 #549 的最终验收。
|
I checked the two points from the review on Could a maintainer approve the Ready CI run for this head? GitHub currently shows @zengchuang-hw, thank you for checking the resource edge cases. If your approval still stands, could you resubmit it with |
|
The code approval still stands for |
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 9466bc8 and head 9a4b587 (full diff).
Verified against the accepted checklist in #549:
has_iterative_protocolinsrc/inferrsi/benchmark/registry.pyistask_resources(name).joinpath("optimization.py").is_file()and does not import task code.benchmarks list --iterativefilters the already-sortedbuiltin_tasks()result; the unfiltered list is unchanged.session startis not touched.benchmark_compare._validate_configuses the same helper instead of an inlinePath(str(task_resources(...)))check; the error message is unchanged.task_resourcesremains used later in the file.- Tests cover the packaged inventory
["moe-forward-npu", "qwen21-denoise-npu"], help text, empty match, excluding a directory namedoptimization.py, including an import-raising file without executing it, and the device-library import guard now also runslist --iterative. - Docs in
engineering-benchmarks.md,operations.md, andoptimization-benchmarks.mdmatch the implementation.
Accuracy and performance comparison evidence is not applicable (CLI discovery filter, docs, and tests; no model output or runtime path change). No redundant architecture, code, or test concerns.
Note: Ready lane CI (run 44891405026) is action_required with no jobs executed yet, so I have not independently confirmed lint/tests in CI. Merge should wait until that lane actually runs and is green.
|
@Dong1017, the two first-line |
|
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5). I can't approve the Ready run for Review state on
Diff check
Next steps
|
|
@jiangkuaixue123, Dong-bot asked me to route the fork workflow approval. Could you approve Ready CI for |
There was a problem hiding this comment.
Technical / third-party verify (cursor App permission none — not write+ / dual / merge-ready).
Tip already has Dong1017 always-on + amy + zengchuang Approves — no additional tip Approve from cursor this round (dual-clean tip unchanged).
Spot-checked tip diff Refs #549:
has_iterative_protocolis file-presence only (optimization.py.is_file()), no task import / device probe- CLI
--iterativefiltersbuiltin_tasks()through that helper; compare config shares the same check - Tests cover empty filter, directory-named
optimization.pyrejected, and help text
Fork Ready lane was stuck in action_required; tip Ready run was approved so required checks can complete. Await Ready green before steward merge. Cursor does not merge (permission none).
|
Ready CI finished for Both failed in I’ll follow the upstream fix in #556 and verify this PR against the corrected base once it lands. |
Share the packaged protocol-file check with comparison configuration so both commands discover the same tasks without loading task code. Agent-assisted-by: OpenAI Codex (implementation, tests and documentation) Signed-off-by: Inference1 <3617237115@qq.com>
9a4b587 to
d9e36eb
Compare
There was a problem hiding this comment.
Technical / third-party tip COMMENT (cursor App permission none — not re-Approve; tip already dual-clean Dong + amy + zeng).
Ready blocker: same mainline residue — Ready full-check FAIL on test_recovered_attempt_does_not_trust_a_rewritable_declaration / retired inferrsi.infra.artifacts.digest (run 44891405026). Unrelated to --iterative filter change.
Unblock: #556 MERGED. Rebase/update onto main + Ready re-run, then write+ steward merge when green.
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head d9e36ebea5fcd3e9021e86d216df63d2347fde6d (tip moved after prior COMMENT; re-verified).
Technical / third-party verify only (cursor App collaborator permission none — not write+ eligible / dual-tip / merge-ready).
Checked
- Shared
has_iterative_protocol= packaged rootoptimization.pyfile presence; used bybenchmarks list --iterativeand compare config validation (no task-code import). - Tests: directory-named
optimization.pyexcluded; file presence lists sorted tasks without executing protocol; engineering list returns moe/qwen only; discovery still avoids device libs. - Docs: operations / engineering / optimization-benchmarks updated for
--iterative. - Base includes merged #556 digest fix; Ready full-check still pending at review time — steward merge after green.
zengchuang-hw
left a comment
There was a problem hiding this comment.
PASS
Incremental review: 9a4b5871ac6220f86207e6be8a5b5335e7fb2952 → d9e36ebea5fcd3e9021e86d216df63d2347fde6d. Previous review. The two heads have the same tree and common base 9466bc8d; all nine PR files are unchanged. The target main moved to 10ba7a55, but this head does not contain that tip. This approval covers the #549 code change, not integration with current main.
I reran focused CPU checks using the pinned CLI branch and registry helper: the default list is unchanged; an empty match returns []; filtering preserves registry order, excludes a directory named optimization.py, and includes a file that would raise if imported without executing it. I also verified that comparison uses the same helper and that the parser declares --iterative. No new code blockers or non-blocking findings. I did not rerun the installed-wheel check or full suite, inspect CI, or test integration with current main.
|
All eight Ready CI jobs passed. The checkout step in both Linux logs confirms that CI tested The nine-file feature diff is unchanged. I also reran the three benchmark test files and the previously failing recovery test against that combined tree locally: 74 passed. Could a maintainer squash-merge this PR when convenient? Thanks for the reviews and CI approval. |
|
Ready CI passed on Both Linux The signed commit uses the file tree and parent from the reviewed feature commit. GitHub marks it Verified, and its sign-off is |
Governing Issue and delivery slice
Refs #549. Dong1017’s delegate accepted this scope and assigned implementation to Inference1.
inferrsi benchmarks list --iterativelists the installed tasks that shipoptimization.pyas a file at their package root. Both the CLI and comparison configuration use the same check in the Benchmark registry. Discovery reads the package entries without loading task code or probing devices.Changed artifacts and evidence
The CLI preserves the ordering from
builtin_tasks(). I built the wheel, installed it in a fresh environment with no optional dependencies, and ran both commands outside the checkout. Their JSON output is shown here with whitespace condensed.CPU tests cover the existing list, filtered order, help text, and an empty match (
[], exit 0). A temporary registry excludes a directory namedoptimization.pyand includes a file that raises on import. The device-library discovery test also exercises the new flag.Checks and dependencies
pytest tests/test_engineering_benchmarks.py tests/test_benchmark_operations.py tests/test_benchmark_compare.py— 73 passed.ruff check src tests toolsandgit diff --check— passed.pyteston macOS arm64, Python 3.12: base9466bc8dhad 5,776 passed, 118 failed and 33 skipped. The feature tree had 5,780 passed, 117 failed and 33 skipped. Every failing test also failed on the base. Most failures occur in test-service process cleanup on this host.Knowledge authorship declaration
The change covers benchmark discovery, its CLI, tests and user guides. The protected knowledge-content declaration is inapplicable. OpenAI Codex assisted with implementation, tests and documentation.
Remaining work and Issue closure
Acceptance remains with Dong1017’s queue. The commands above and the empty-result regression provide the checks requested in #549.