Skip to content

task: add --iterative to inferrsi benchmarks list #549

Open

Description

@Inference1

Current state — PR #553 is open for review and includes the CLI outputs and test results.
Decision — Dong1017’s queue accepted the resource-presence rule below and assigned implementation to Inference1.
Next action — Review PR #553 against the accepted checklist. Dong1017’s queue owns the acceptance decision.

Goal and acceptance

I ran the unfiltered command on main and got all three packaged tasks. The benchmark guide lists MoE forward and Qwen denoise as iterative. Qwen TTS currently supports single-candidate runs.

$ inferrsi benchmarks list
["moe-forward-npu", "qwen21-denoise-npu", "qwen3-tts-npu"]

$ inferrsi benchmarks list --iterative  # proposed
["moe-forward-npu", "qwen21-denoise-npu"]
  • With no flag, preserve the current JSON array and ordering.
  • With --iterative, include a registered task if and only if its packaged task root contains a file named optimization.py. Print one JSON array on stdout, in the order returned by builtin_tasks(). With zero matches, print [] and exit 0.
  • Use one registry check, task_resources(name).joinpath("optimization.py").is_file(), for both discovery and comparison. Only inspect the file entry. A stub qualifies for discovery, and runtime protocol validation stays with execution.
  • Exclude a directory named optimization.py. Include a file of that name that would raise on import, without executing it.

Scope and constraints

This is a read-only discovery option. Listing should work on a CPU host without starting a session, importing task code, or probing devices. The comparison workflow already checks for the packaged optimization.py resource.

Ownership and authorization

Implementation: Inference1. Acceptance: Dong1017’s queue, via Dong-bot. Scope and delegation.

Inputs and deliverables

The CLI flag, a short --help description, and CPU tests for help, default output, filtered order and empty results. The packaged-task check expects ["moe-forward-npu", "qwen21-denoise-npu"]. The fixture also excludes a directory named optimization.py and includes an import-raising file of the same name. benchmarks session start behavior is outside this change. The existing benchmark CLI test covers the current list.

Completion and closure

The acceptance owner can run both commands on the packaged tasks, verify the default list and the filtered ["moe-forward-npu", "qwen21-denoise-npu"], and check the empty result, directory exclusion, and import-raising file inclusion. PR #553 provides the evidence.

Activity

  1. Dong1017 commented on Nov 21, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    Assessment: reasonable, small, read-only CLI addition. I'd accept it once the eligibility rule is pinned down (see Q1). I couldn't open the linked files at the pinned SHA, so the code references below are taken from the issue body and not verified.

    What's asked

    • Add inferrsi benchmarks list --iterative. It returns the subset of registered tasks that support multi-round optimization.
    • The default output (JSON array and ordering) is unchanged. An empty match returns [] with exit 0.
    • Eligibility comes from packaged task resources, with no hardcoded name list. It must not import task code, start a session, or probe devices.

    Gaps and risks

    1. The eligibility signal is under-specified. The acceptance text says "supply the iterative protocol". The constraints section points to the benchmark_compare.py check for a packaged optimization.py resource. Those are two different things:
      • Resource presence says a file ships. It doesn't say the file implements the protocol.
      • A task could ship a stub or a differently shaped optimization.py and be listed wrongly.
      • Please state the exact rule, for example "a packaged optimization.py resource exists", so tests and reviewers agree.
    2. Reuse the existing predicate. If benchmark_compare.py already checks for the resource, pull that check into a shared helper. Don't copy it. Otherwise list --iterative and compare can drift apart on what counts as iterative.
    3. Check against the guide. The issue says the guide lists MoE forward and Qwen denoise as iterative and Qwen TTS as single-candidate. The acceptance test should assert the filtered output against the resource-derived result, and a docs-consistency check would be good. Qwen TTS must not ship an optimization.py, or it will leak into the filter.
    4. Resource lookup must not import. Use importlib.resources file checks (files(pkg).joinpath(...).is_file()). Don't import the task package, since that may pull in torch/NPU dependencies. Add a test that proves no task module gets imported, or at least runs CPU-only.
    5. Output contract. Confirm the output stays a single JSON array on stdout with no extra text. Reject unknown combinations with a clear error if more filters are added later.

    Questions for the author and maintainers

    • Q1: Is "packaged optimization.py resource present" the intended definition of iterative? If it's something stricter, such as a declared entry point or metadata flag, say so.
    • Q2: Should benchmarks session start also warn when a single-candidate task is run with multi-round options? That's out of scope here, but worth a follow-up issue if wanted.
    • Q3: Who is the implementation owner and who is the acceptance owner? The body leaves both open.

    Suggested next steps

    1. The author updates the acceptance bullet with the exact eligibility rule (Q1).
    2. A maintainer confirms the scope. A short "accepted" comment is enough.
    3. Implement with a shared resource-check helper.
    4. Add CPU tests for three cases: default output unchanged, --iterative filtered with ordering preserved, and an empty match. For the empty case, monkeypatch the registry so no task qualifies. Also add one test that --help documents the flag.
    5. Update docs/optimization-benchmarks.md with the new flag if the guide documents the list command.
  2. Inference1 commented on Nov 21, 2026

    @Inference1
    ContributorAuthor

    Thanks for reviewing this. I updated the acceptance criteria to make Q1 testable: a registered task qualifies exactly when optimization.py is a file at the root of its packaged resources. Both list --iterative and compare would use that check. The listing only inspects the directory entry, so a stub also appears in the list.

    For Q2, I’ll keep session start behavior outside this change. For Q3, I can take implementation as Inference1 once the scope is accepted. Could a maintainer confirm this scope and identify who will accept the result?

  3. amy-why-3459 commented on Nov 21, 2026

    @amy-why-3459
    Collaborator

    Thanks for tightening the acceptance text after the earlier questions — I checked the cited files at 9466bc8d910c9fef55d4a9e23ee1a82950280c30.

    Verified against that SHA

    • The comparison workflow already uses a resource-presence check in _validate_config: Path(str(task_resources(task))) / "optimization.py" must be a file, otherwise it raises Task has no iterative protocol (src/inferrsi/apps/workflows/benchmark_compare.py). That matches the rule now written in the acceptance bullets.
    • docs/optimization-benchmarks.md does describe qwen21-denoise-npu and moe-forward-npu as the iterative tasks, and says TTS currently exposes a single-candidate entry.
    • In the packaged tree, moe-forward-npu and qwen21-denoise-npu each have a root optimization.py; qwen3-tts-npu does not. The proposed --iterative example is consistent with that tree. Registry order in src/inferrsi/benchmark/tasks/index.json is moe-forward-npu, qwen21-denoise-npu, qwen3-tts-npu.
    • tests/test_engineering_benchmarks.py already pins the unfiltered benchmarks list contract (JSON array of the three packaged tasks and current ordering).

    Suggestions for the implementation PR

    1. Extract that existing compare predicate into a shared helper and use it from both list --iterative and _validate_config, so the two paths cannot drift.
    2. Keep discovery read-only: resolve the packaged root the same way task_resources already does, and only inspect the optimization.py file entry. Do not import task modules. The existing test_discovery_does_not_import_device_libraries pattern is a good model for a CPU-only --iterative test.
    3. If the user-facing guides document benchmarks list, please add the new flag there so the docs stay aligned with the resource-derived filter.
    4. Implementation ownership can sit with Inference1 once a maintainer accepts this scope, as offered. The issue still needs an acceptance owner to confirm the resource-presence rule — including that a stub optimization.py is intentionally listed — and to verify both commands plus the empty-result test.

    The scope looks small and well bounded (session start stays out). I am not recording maintainer acceptance here; a short accept/reject of this rule would be enough to unblock the implementation.

  4. zengchuang-hw commented on Nov 21, 2026

    @zengchuang-hw
    Contributor

    PASS(设计检视,尚未验收实现)

    建议按这个范围实现。对照当前打包资源:MoE forward 和 Qwen denoise 有根目录 optimization.py,Qwen TTS 没有。文件存在只代表可以被发现,不证明执行协议有效或设备可用。

    建议把两处文件判断收敛到 Benchmark registry,用 task_resources(name).joinpath("optimization.py").is_file(),避免沿用 comparison 中把打包资源强转成文件系统 Path 的写法。registry order 建议明确为现有 builtin_tasks() 返回的排序,默认输出不变。CPU 测试除 --help、顺序和零结果外,建议加一个 optimization.py 同名目录(排除)和一个导入即报错的同名文件(仍列出),证明只查文件、不导入任务。独立核对了当前资源和调用链,并用合成归档验证了两种资源检查的差异;尚无本功能实现或 CLI 测试结果。

  5. Inference1 commented on Nov 21, 2026

    @Inference1
    ContributorAuthor

    I’ll cover both edge cases from the design review: a directory named optimization.py is excluded, and a file that raises on import is included without executing it. The filtered array will preserve the order returned by builtin_tasks().

    @Dong1017, does this resolve Q1 for the Benchmark side? I’m ready to implement that scope.

  6. Dong1017 commented on Nov 21, 2026

    @Dong1017
    Collaborator

    Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

    Accepted. Q1 is resolved for the Benchmark side, and Inference1 can start implementing. This is a decision from Dong-bot, the authorized delegate for Dong1017.

    Accepted clauses

    • Eligibility rule: a registered task is iterative if and only if optimization.py is a file at the root of its packaged resources. Presence is the only test. A stub counts, and a directory with that name does not. Protocol validity and device availability stay with execution.
    • Default output: inferrsi benchmarks list is unchanged, with the same JSON array and ordering.
    • Filtered output: --iterative prints one JSON array on stdout, in the order builtin_tasks() returns. With no matches it prints [] and exits 0.
    • One shared check: list --iterative and compare (_validate_config) both use a single helper in the Benchmark registry. I accept zengchuang-hw's form, task_resources(name).joinpath("optimization.py").is_file(). It replaces the Path(str(...)) coercion in the compare workflow.
    • No imports or probing: the listing doesn't import task code, start a session or probe devices.
    • Scope: benchmarks session start is out of scope (Q2). A warning for single-candidate tasks run with multi-round options can be a separate follow-up issue if someone wants it.
    • Tests (CPU-only):
      • --help documents the flag.
      • Default output is unchanged.
      • Filtered output keeps registry order.
      • An empty result is covered, by monkeypatching the registry.
      • A directory named optimization.py is excluded.
      • A file named optimization.py that raises on import is still listed, which shows the check never imports it.
      • The test_discovery_does_not_import_device_libraries pattern is the model for the import test.
    • Docs: add the flag to docs/optimization-benchmarks.md or any other guide that documents benchmarks list.

    Owners (Q3)

    • Implementation: Inference1.
    • Acceptance: Dong1017's queue, via Dong-bot. The acceptance check is that both commands run against the packaged tasks, giving ["moe-forward-npu","qwen21-denoise-npu"] for --iterative, and that the empty-case and edge-case tests pass.

    Non-blocking follow-ups

    • Please update the issue body's acceptance checklist to include the two edge cases (directory excluded, import-raising file included) and the builtin_tasks() ordering. This keeps the body in line with this thread.
    • I haven't run any code. This rests on the issue body and on the checks amy-why-3459 and zengchuang-hw reported against 9466bc8. The PR should show the actual CLI output and CPU test results.
  7. amy-why-3459 commented on Nov 21, 2026

    @amy-why-3459
    Collaborator

    Thanks for updating the description. I see the issue is now Accepted by Dong1017's delegate, implementation is assigned to Inference1, --iterative ordering is pinned to builtin_tasks(), and the checklist now includes the two resource-entry edge cases (exclude a directory named optimization.py; include an import-raising file without executing it). Those additions match the design-review thread and make the eligibility rule testable.

    One consistency gap in this update: Inputs and deliverables still lists CPU tests only for help, default output, filtered order, and the empty case, and Completion and closure still only mentions verifying the two commands plus the empty-result test. Please extend those sections to include the new directory / import-raising cases — and the packaged-task check from the acceptance comment that --iterative should return ["moe-forward-npu","qwen21-denoise-npu"] — so the PR is reviewed against the same checklist as the Goal and acceptance bullets.

    The accepted shared check is task_resources(name).joinpath("optimization.py").is_file(), replacing the Path(str(...)) coercion in the compare workflow. Recording that form on the "one resource-presence check" bullet would keep the issue body aligned with the accepted comment.

  8. amy-why-3459 commented on Nov 21, 2026

    @amy-why-3459
    Collaborator

    Thanks for the update. I see PR #553 is now open, and this description change addresses the consistency gaps from my previous comment:

    • The shared check is now recorded as task_resources(name).joinpath("optimization.py").is_file() on the acceptance bullet.
    • Inputs and deliverables and Completion and closure now include the packaged-task expectation ["moe-forward-npu", "qwen21-denoise-npu"], plus the empty-result, directory-exclusion, and import-raising file cases.

    Next action correctly points review at PR #553, with Dong1017's queue owning acceptance. I'll review the PR against this accepted checklist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind:taskBounded delivery task with a delegated slicestatus:proposedWaiting for the decision named in the status block

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions