Repository navigation
Description
Current state — PR #553 is open for review and includes the CLI outputs and test results.
Decision — Dong1017’s queue accepted the resource-presence rule below and assigned implementation to Inference1.
Next action — Review PR #553 against the accepted checklist. Dong1017’s queue owns the acceptance decision.
Goal and acceptance
I ran the unfiltered command on main and got all three packaged tasks. The benchmark guide lists MoE forward and Qwen denoise as iterative. Qwen TTS currently supports single-candidate runs.
$ inferrsi benchmarks list
["moe-forward-npu", "qwen21-denoise-npu", "qwen3-tts-npu"]
$ inferrsi benchmarks list --iterative # proposed
["moe-forward-npu", "qwen21-denoise-npu"]- With no flag, preserve the current JSON array and ordering.
- With
--iterative, include a registered task if and only if its packaged task root contains a file namedoptimization.py. Print one JSON array on stdout, in the order returned bybuiltin_tasks(). With zero matches, print[]and exit 0. - Use one registry check,
task_resources(name).joinpath("optimization.py").is_file(), for both discovery and comparison. Only inspect the file entry. A stub qualifies for discovery, and runtime protocol validation stays with execution. - Exclude a directory named
optimization.py. Include a file of that name that would raise on import, without executing it.
Scope and constraints
This is a read-only discovery option. Listing should work on a CPU host without starting a session, importing task code, or probing devices. The comparison workflow already checks for the packaged optimization.py resource.
Ownership and authorization
Implementation: Inference1. Acceptance: Dong1017’s queue, via Dong-bot. Scope and delegation.
Inputs and deliverables
The CLI flag, a short --help description, and CPU tests for help, default output, filtered order and empty results. The packaged-task check expects ["moe-forward-npu", "qwen21-denoise-npu"]. The fixture also excludes a directory named optimization.py and includes an import-raising file of the same name. benchmarks session start behavior is outside this change. The existing benchmark CLI test covers the current list.
Completion and closure
The acceptance owner can run both commands on the packaged tasks, verify the default list and the filtered ["moe-forward-npu", "qwen21-denoise-npu"], and check the empty result, directory exclusion, and import-raising file inclusion. PR #553 provides the evidence.
Activity
Dong1017 commentedon Nov 21, 2026
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
Assessment: reasonable, small, read-only CLI addition. I'd accept it once the eligibility rule is pinned down (see Q1). I couldn't open the linked files at the pinned SHA, so the code references below are taken from the issue body and not verified.
What's asked
inferrsi benchmarks list --iterative. It returns the subset of registered tasks that support multi-round optimization.[]with exit 0.Gaps and risks
benchmark_compare.pycheck for a packagedoptimization.pyresource. Those are two different things:optimization.pyand be listed wrongly.optimization.pyresource exists", so tests and reviewers agree.benchmark_compare.pyalready checks for the resource, pull that check into a shared helper. Don't copy it. Otherwiselist --iterativeandcomparecan drift apart on what counts as iterative.optimization.py, or it will leak into the filter.importlib.resourcesfile checks (files(pkg).joinpath(...).is_file()). Don't import the task package, since that may pull in torch/NPU dependencies. Add a test that proves no task module gets imported, or at least runs CPU-only.Questions for the author and maintainers
optimization.pyresource present" the intended definition of iterative? If it's something stricter, such as a declared entry point or metadata flag, say so.benchmarks session startalso warn when a single-candidate task is run with multi-round options? That's out of scope here, but worth a follow-up issue if wanted.Suggested next steps
--iterativefiltered with ordering preserved, and an empty match. For the empty case, monkeypatch the registry so no task qualifies. Also add one test that--helpdocuments the flag.docs/optimization-benchmarks.mdwith the new flag if the guide documents thelistcommand.Inference1 commentedon Nov 21, 2026
Thanks for reviewing this. I updated the acceptance criteria to make Q1 testable: a registered task qualifies exactly when
optimization.pyis a file at the root of its packaged resources. Bothlist --iterativeandcomparewould use that check. The listing only inspects the directory entry, so a stub also appears in the list.For Q2, I’ll keep
session startbehavior outside this change. For Q3, I can take implementation asInference1once the scope is accepted. Could a maintainer confirm this scope and identify who will accept the result?amy-why-3459 commentedon Nov 21, 2026
Thanks for tightening the acceptance text after the earlier questions — I checked the cited files at
9466bc8d910c9fef55d4a9e23ee1a82950280c30.Verified against that SHA
_validate_config:Path(str(task_resources(task))) / "optimization.py"must be a file, otherwise it raisesTask has no iterative protocol(src/inferrsi/apps/workflows/benchmark_compare.py). That matches the rule now written in the acceptance bullets.docs/optimization-benchmarks.mddoes describeqwen21-denoise-npuandmoe-forward-npuas the iterative tasks, and says TTS currently exposes a single-candidate entry.moe-forward-npuandqwen21-denoise-npueach have a rootoptimization.py;qwen3-tts-npudoes not. The proposed--iterativeexample is consistent with that tree. Registry order insrc/inferrsi/benchmark/tasks/index.jsonismoe-forward-npu,qwen21-denoise-npu,qwen3-tts-npu.tests/test_engineering_benchmarks.pyalready pins the unfilteredbenchmarks listcontract (JSON array of the three packaged tasks and current ordering).Suggestions for the implementation PR
list --iterativeand_validate_config, so the two paths cannot drift.task_resourcesalready does, and only inspect theoptimization.pyfile entry. Do not import task modules. The existingtest_discovery_does_not_import_device_librariespattern is a good model for a CPU-only--iterativetest.benchmarks list, please add the new flag there so the docs stay aligned with the resource-derived filter.Inference1once a maintainer accepts this scope, as offered. The issue still needs an acceptance owner to confirm the resource-presence rule — including that a stuboptimization.pyis intentionally listed — and to verify both commands plus the empty-result test.The scope looks small and well bounded (
session startstays out). I am not recording maintainer acceptance here; a short accept/reject of this rule would be enough to unblock the implementation.zengchuang-hw commentedon Nov 21, 2026
PASS(设计检视,尚未验收实现)
建议按这个范围实现。对照当前打包资源:MoE forward 和 Qwen denoise 有根目录
optimization.py,Qwen TTS 没有。文件存在只代表可以被发现,不证明执行协议有效或设备可用。建议把两处文件判断收敛到 Benchmark registry,用
task_resources(name).joinpath("optimization.py").is_file(),避免沿用 comparison 中把打包资源强转成文件系统Path的写法。registry order建议明确为现有builtin_tasks()返回的排序,默认输出不变。CPU 测试除--help、顺序和零结果外,建议加一个optimization.py同名目录(排除)和一个导入即报错的同名文件(仍列出),证明只查文件、不导入任务。独立核对了当前资源和调用链,并用合成归档验证了两种资源检查的差异;尚无本功能实现或 CLI 测试结果。Inference1 commentedon Nov 21, 2026
I’ll cover both edge cases from the design review: a directory named
optimization.pyis excluded, and a file that raises on import is included without executing it. The filtered array will preserve the order returned bybuiltin_tasks().@Dong1017, does this resolve Q1 for the Benchmark side? I’m ready to implement that scope.
Dong1017 commentedon Nov 21, 2026
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
Accepted. Q1 is resolved for the Benchmark side, and
Inference1can start implementing. This is a decision from Dong-bot, the authorized delegate for Dong1017.Accepted clauses
optimization.pyis a file at the root of its packaged resources. Presence is the only test. A stub counts, and a directory with that name does not. Protocol validity and device availability stay with execution.inferrsi benchmarks listis unchanged, with the same JSON array and ordering.--iterativeprints one JSON array on stdout, in the orderbuiltin_tasks()returns. With no matches it prints[]and exits 0.list --iterativeandcompare(_validate_config) both use a single helper in the Benchmark registry. I accept zengchuang-hw's form,task_resources(name).joinpath("optimization.py").is_file(). It replaces thePath(str(...))coercion in the compare workflow.benchmarks session startis out of scope (Q2). A warning for single-candidate tasks run with multi-round options can be a separate follow-up issue if someone wants it.--helpdocuments the flag.optimization.pyis excluded.optimization.pythat raises on import is still listed, which shows the check never imports it.test_discovery_does_not_import_device_librariespattern is the model for the import test.docs/optimization-benchmarks.mdor any other guide that documentsbenchmarks list.Owners (Q3)
Inference1.["moe-forward-npu","qwen21-denoise-npu"]for--iterative, and that the empty-case and edge-case tests pass.Non-blocking follow-ups
builtin_tasks()ordering. This keeps the body in line with this thread.9466bc8. The PR should show the actual CLI output and CPU test results.amy-why-3459 commentedon Nov 21, 2026
Thanks for updating the description. I see the issue is now Accepted by Dong1017's delegate, implementation is assigned to
Inference1,--iterativeordering is pinned tobuiltin_tasks(), and the checklist now includes the two resource-entry edge cases (exclude a directory namedoptimization.py; include an import-raising file without executing it). Those additions match the design-review thread and make the eligibility rule testable.One consistency gap in this update: Inputs and deliverables still lists CPU tests only for help, default output, filtered order, and the empty case, and Completion and closure still only mentions verifying the two commands plus the empty-result test. Please extend those sections to include the new directory / import-raising cases — and the packaged-task check from the acceptance comment that
--iterativeshould return["moe-forward-npu","qwen21-denoise-npu"]— so the PR is reviewed against the same checklist as the Goal and acceptance bullets.The accepted shared check is
task_resources(name).joinpath("optimization.py").is_file(), replacing thePath(str(...))coercion in the compare workflow. Recording that form on the "one resource-presence check" bullet would keep the issue body aligned with the accepted comment.amy-why-3459 commentedon Nov 21, 2026
Thanks for the update. I see PR #553 is now open, and this description change addresses the consistency gaps from my previous comment:
task_resources(name).joinpath("optimization.py").is_file()on the acceptance bullet.["moe-forward-npu", "qwen21-denoise-npu"], plus the empty-result, directory-exclusion, and import-raising file cases.Next action correctly points review at PR #553, with Dong1017's queue owning acceptance. I'll review the PR against this accepted checklist.