Skip to content

docs(cli): name the probe and the per-call cap on the first-run surfaces - #1212

Merged
Birfy merged 1 commit into
mainfrom
agent/1201-first-run-caps
Nov 21, 2026
Merged

Birfy merged 1 commit into
mainfrom
agent/1201-first-run-caps

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1201.

Base: 1aa816e0e155c2f59db1686e404a7ebae739d94b
Tip SHA: c6649930fd9bc39b3f0f8af09c595e755e44a79d

What changed

#1166 / #1184 shipped a per-call cap (evolve --max-calls / --max-usd, MCP plan / start fields max_calls / max_usd). docs/start.md documents it, but the surfaces a first-run user or host agent actually follows did not mention it:

  • agentdescent/integrations/SKILL.md (the shared host skill, installed identically on claude-code, dsh, codex, opencode and grok-bot): the plan step now tells the host to give a first real run a ceiling that holds before each call — max_calls, and max_usd with usd_per_call, on MCP plan / start; --max-calls / --max-usd on the CLI — and keeps --budget described as the between-round bound that can overshoot. --probe stays the first-model preflight.
  • docs/plugin-quickstart.md "Two habits worth keeping" and the budget paragraph of docs/plugins.md: the same cap, with the overshoot warning kept and a pointer to docs/start.md#real-model.
  • algorithms run's next: is now agentdescent plan <spec> --probe, then agentdescent evolve <spec> --max-calls N, where N is the plan's own estimate.agent_calls_upper_bound (a placeholder <N> only when the plan reports no positive bound). It stays one string for both the text line and --json.next, and it does not start a run.
  • demo --algorithm's footer is now four steps — algorithms run … --out spec.json, plan spec.json --probe, evolve spec.json --max-calls <N>, doctor. --out spec.json gives the two consuming steps a real path; the cap is <N> because the demo has no estimate. One list still feeds both the footer and --json.next.
  • The two checked-in rendered copies of the skill (integrations/claude-code/skills/agentdescent/SKILL.md and integrations/dsh-agentdescent/lib/index.js) were regenerated with render_claude_plugin / render_dsh_plugin, not hand-edited; tests/test_integrations.py still owns the drift check, so the new test file carries no duplicate equality row.

tests/test_demo.py's old "--out spec.json" not in out guard becomes a positive assertion: the demo does not claim a spec exists, and the comment now says so — the #1145 rule that demo must not point at agentdescent.<algo>.spec.json is kept verbatim.

Verification at the tip

Tip SHA: c6649930fd9bc39b3f0f8af09c595e755e44a79d

Ran:

  • pytest -q -o addopts='' tests/test_first_run_caps.py → 7 passed
  • pytest -q -o addopts='' test_first_run_caps + test_cli_algorithms_run + test_demo + test_plan_text_output → 80 passed
  • pytest -q -o addopts='' tests/test_integrations.py (both rendered copies drift-checked) → 105 passed, 2 skipped
  • the new test file alone on base 1aa816e0 → 7 failed
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 15 changelog fragment(s) OK

Not run:

  • full pytest -q — the CI matrix is the line for this docs/CLI-text change
  • mkdocs build --strict — the social plugin fetches fonts.google.com on this offline host; a local copy with social cards off builds rc 0 (25.60s)

Red first

At the base 1aa816e0, clean worktree, only tests/test_first_run_caps.py added:

python3 -m pytest -o addopts='' -p no:cacheprovider -q tests/test_first_run_caps.py
7 failed in 4.68s

exactly the seven rows: test_shared_host_skill_names_the_per_call_cap,
test_plugin_quickstart_habits_name_the_per_call_cap,
test_plugins_budget_paragraph_names_the_per_call_cap,
test_algorithms_run_next_names_the_probe_then_the_cap[gepa|archive_selection],
test_demo_algorithm_footer_names_the_probe_then_the_cap[gepa|archive_selection].

Green at the tip

tests/test_first_run_caps.py                                        7 passed in 6.78s
test_first_run_caps + test_cli_algorithms_run + test_demo
  + test_plan_text_output                                          80 passed in 43.88s
tests/test_integrations.py (both rendered copies drift-checked)    105 passed, 2 skipped in 20.29s
python3 -m tools.gen_api_docs --check        # docs/api.md is up to date
python3 -m tools.changelog_fragments check   # 15 changelog fragment(s) OK

Full suite and docs at the tip

Full suite, same 8-chunk split at c6649930 -> 7606 tests, 21 failed, 10 errors, 56 skipped; the failing-id set is identical to base 1aa816e0 (comm empty both ways), so 0 new red.

mkdocs build --strict -> rc 0 in 21.29 s with the social plugin excluded; the unmodified config hangs on this host at material/social post_page, which is a host limitation, and CI's build job runs the unmodified config.

Not run: no real model call -- every changed line is a printed string, a doc paragraph, or a test that reads them.

Scope

Five docs/CLI text surfaces, the two re-rendered plugin copies, one updated demo guard, the algorithms run next-step string, the new test file and one changelog.d/ fragment. No new flag, no behaviour change: plan's own default next: (cli.py:2489) is deliberately untouched and wants its own card. T2 by path and size.

@github-actions github-actions Bot added risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller agent-pr Pull request opened by an agent labels Nov 21, 2026
@HugoXK HugoXK added area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) area:docs Documentation site (docs/, mkdocs.yml) and README area:plugins Host plugins and installers: Claude Code, Codex, DSH, OpenCode, MCP server documentation Improvements or additions to documentation labels Nov 21, 2026
…ces (#1201)

#1166/#1184 added `--max-calls` / `--max-usd`, and docs/start.md documents
them, but the lines a new user or host agent follows did not: the shared host
skill, the plugin-quickstart habits, the budget paragraph in docs/plugins.md,
and the `next:` line `algorithms run` / `demo --algorithm` print. A first real
run is started from those lines, so the preflight and the cap belong there --
at the plan's own `agent_calls_upper_bound` for `algorithms run`, and as `<N>`
in the demo footer, which runs before a spec exists.

`--out spec.json` joins the demo footer so the `plan` / `evolve` steps it adds
have a path to name; the `agentdescent.<algo>.spec.json` claim from #1145
stays gone.

The two rendered plugin copies were regenerated with render_claude_plugin /
render_dsh_plugin, so the drift tests keep them in step instead of a hand
edit.

Fixes #1201.
@HugoXK
HugoXK marked this pull request as draft November 21, 2026 18:20
@HugoXK

HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

State change: draft -> ready for review.

This PR was opened ready for review. It was converted to draft with no commit attached and no owner for the change; the head is the same commit it was opened at, CI is green on it, and nothing in the diff changed. Restoring the opened state so the review lane can see it.

@HugoXK
HugoXK marked this pull request as ready for review November 21, 2026 18:26
@HugoXK

HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Advisory Verifier report; .github/agent/review-policy.json counts only Birfy, so this is evidence, not a counting PASS.

Verdict: PASS (advisory)
Tip SHA: c664993

Ran (fresh clone at refs/pull/1212/head; base worktree 1aa816e0):

  • full offline suite at this tip, one command, -n 8 -o addopts="" -> 21 failed / 7972 passed / 51 skipped / 10 errors in 296.53s; the same command at base 1aa816e0 -> 21 failed / 7963 passed / 53 skipped / 10 errors in 291.31s. The 31 red ids (21 failed + 10 errors) are identical between the two ends (comm empty both ways): 0 new red; they are host reds (no Docker daemon, wheel/exec-bit/sandbox, era-srbench data), none in a file this diff touches.
  • red at base 1aa816e0 with only this branch's tests/test_first_run_caps.py added -> 7 failed, exactly the seven rows named in the body; the branch's four changed test files on that base -> 14 failed / 66 passed.
  • pytest -q -o addopts="" tests/test_first_run_caps.py at the tip -> 7 passed (6.05s); the four changed files -> 80 passed (30.15s); tests/test_integrations.py (both rendered copies drift-checked) -> 105 passed / 2 skipped (20.13s).
  • python -m tools.gen_api_docs --check -> docs/api.md is up to date.
  • python -m tools.changelog_fragments check -> 15 changelog fragment(s) OK.
  • mkdocs build --strict at this tip with only the social plugin removed (local config copy) -> rc 0, 14.17s; the unmodified config did not finish on this host (social-card generation, killed after >8 min), so I do not count an unmodified local build.
  • CI at this tip (check-run API) -> build, test (3.9)–test (3.13), otel-wire all success; deploy skipped. mergeable_state: clean; head is still this sha.

Not run:

  • unmodified mkdocs build --strict -> the social plugin hangs on this host; the same tree builds rc 0 with only that plugin off, and CI's build job ran the unmodified config green at this tip.
  • risk_gate.py -> no gh binary here; the tier below is computed from .github/agent/risk-rules.json.
  • real provider / Docker -> nothing in the diff calls either; Docker is the same 10 errors at both ends.

Summary

Closes #1201. Five docs/CLI text surfaces, the two re-rendered skill copies, four test files and one changelog fragment: the shared host skill now names the per-call cap (max_calls / --max-calls, plus max_usd / --max-usd) beside the --probe preflight and keeps --budget as the between-round bound; the quickstart habits and the docs/plugins.md budget paragraph repeat it with the overshoot warning; algorithms run's next: is plan <spec> --probe, then evolve <spec> --max-calls N with N from estimate.agent_calls_upper_bound (the test pins the real number, e.g. 11, not a placeholder); the demo --algorithm footer gains --out spec.json, plan spec.json --probe and evolve spec.json --max-calls <N>.

The evidence is honest: the new file alone fails 7/7 at base, one row per surface, the four modified test files also fail there (14 rows), and at the tip the seven rows, the 80-row focused set and the rendered-copy drift check are green. The names the skill tells a host to pass are real on both MCP tools (agentdescent/mcp.py:712, :765), the docs/start.md#real-model anchor exists, and docs/api.md / CHANGELOG.md are untouched.

Blocking

None.

Non-blocking

  • The body's closing line says "T2 by path and size". The diff touches agentdescent/integrations/** and integrations/**, which .github/agent/risk-rules.json lists as T3, and the controller set risk:T3 — body text only; the label is right.
  • docs/plugin-quickstart.md still heads that list "Two habits worth keeping" while it now holds four bullets (it already held three at base); the added bullet is in the right place.
  • Issue docs/plugins: first-run skill and algorithms-run next omit --max-calls and --probe after #1184 #1201 is Birfy-authored and was claimed with agent-working without agent-ready ever set (issue events: the four labels added at creation, none removed). The scope in the issue is clear and the code is unaffected; the claim basis is not the repo's agent-ready path — Lead to confirm.
  • Labels are documentation + three area:* + agent-pr + risk:T3; agent-pr is the controller's, not this branch's.

Risk / merge

T3 by path (agentdescent/integrations/**, integrations/**; matches risk:T3). Danyang's PASS and explicit merge click.

No UI surface: CLI next: / demo-footer strings and Markdown docs; nothing is rendered. Before -> after: next: agentdescent evolve <spec> -> next: agentdescent plan <spec> --probe, then agentdescent evolve <spec> --max-calls N; the demo footer's 2 steps -> 4 (adds --out spec.json, plan spec.json --probe, evolve spec.json --max-calls <N>).

No merge action from QA: this report is advisory, the merge gate is the repo's review decision on the head sha, and the merge itself is executed by the repo's merge process.

@Birfy

Birfy commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: c664993
Ran:

  • .github/scripts/risk_gate.py --repo ThinkFlowLab/agentdescent --pr 1212 → T3 (214 lines / 11 files; path agentdescent/integrations/SKILL.md, integrations/**)
  • pytest -o addopts="" -p no:cacheprovider -q tests/test_first_run_caps.py tests/test_cli_algorithms_run.py tests/test_demo.py tests/test_plan_text_output.py at this tip → 80 passed
  • python -m tools.changelog_fragments check → 15 fragments OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • Shared SKILL.md is byte-identical to the Claude Code copy; the DSH CONTENT string includes the new per-call cap paragraph
  • Required CI at this tip → green (build, test (3.9)–test (3.13); otel-wire also green)
    Not run:
  • Full offline suite locally — the CI test matrix is the line
    Findings:
  • No actionable defects. algorithms run next: now names plan --probe, then evolve --max-calls N at the plan's agent_calls_upper_bound. The demo --algorithm footer adds --out spec.json, the probe, and --max-calls <N>. The shared skill and plugin docs point at the same cap beside --budget.
  • Nit (non-blocking): docs/plugin-quickstart.md still titles the list "Two habits worth keeping" after a fourth bullet was added.
    Verdict: PASS

@Birfy
Birfy merged commit 0d77201 into main Nov 21, 2026
14 checks passed
@Birfy
Birfy deleted the agent/1201-first-run-caps branch November 21, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent area:cli The agentdescent command line (cli.py, doctor, plan/evolve/status/apply) area:docs Documentation site (docs/, mkdocs.yml) and README area:plugins Host plugins and installers: Claude Code, Codex, DSH, OpenCode, MCP server documentation Improvements or additions to documentation risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs/plugins: first-run skill and algorithms-run next omit --max-calls and --probe after #1184

2 participants