Skip to content

fix(typing): bind sixteen annotation names at module scope - #1207

Merged
Birfy merged 5 commits into
mainfrom
agent/1203-annotation-names
Nov 21, 2026
Merged

Birfy merged 5 commits into
mainfrom
agent/1203-annotation-names

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #1203.

What changed

Sixteen type names were used in this package's annotations but never bound at module scope, so a static reader could not resolve them (the same blind spot as Tuple in agents.py, #1202). This change binds them, and it is import-only: no annotation text, no signature, no generated API text, no runtime path.

  • Union (agent_society.py), Optional (evaluation.py), Sequence (scheduler.py) join the existing typing import lines; selection.py gets a plain import random (the name is used as a value there, not only in annotations).
  • Twelve cycle-prone cross-module names go behind if TYPE_CHECKING: — seven new blocks (async_evolve.py, audit/wiring.py, budget.py, evolution.py, proofs.py, evolvespec.py, selfplay.py) and one extended (evolve_settings.py): EvolvingArtifact, BudgetSettings, RuntimeSettings, ObservabilitySettings, VerifierBoost, Completion, TrajectoryStore, ParallelStrategy, CallLimit, VersionVector, Invariant, RoundInfo.
  • tests/test_annotation_names.py (306 lines) parses the package with ast and fails when an annotation name has no module-scope binding in its own module. It prints every offending row, not just the first per object, and carries two rule controls: a function-local import must not be credited, and a TYPE_CHECKING-only import must be.
  • Runtime resolution is deliberately unchanged for the twelve TYPE_CHECKING names — that block does not execute, so typing.get_type_hints() on those objects still raises. The issue scopes the fix as AST-level for exactly that reason (acceptance 4): roughly fifty pre-existing annotations reference TYPE_CHECKING-only names, and a get_type_hints test would stay red after a correct fix and push real runtime imports into the modules that avoid them.

Rows (24 sites / 16 names / 12 modules at main 1aa816e0; line numbers move with main):

module object name
agent_society.py society_evolve Union
async_evolve.py async_evolve EvolvingArtifact, BudgetSettings, RuntimeSettings, ObservabilitySettings
audit/wiring.py Audit, attach VerifierBoost
budget.py budgeted_completion Completion
evaluation.py EvaluationReport Optional
evolution.py _Runtime, evolve TrajectoryStore; evolve also ParallelStrategy, CallLimit, BudgetSettings, RuntimeSettings, ObservabilitySettings
evolution.py _record_rollout VersionVector
evolve_settings.py BudgetSettings CallLimit
evolvespec.py build_call_limit, _limit_from_knobs CallLimit
proofs.py __init__ Invariant
scheduler.py pop_for Sequence
selection.py __init__ random
selfplay.py on_round RoundInfo

All twelve TYPE_CHECKING import paths were checked against the target module at this tip (budget.CallLimit, agents.Completion, evolution.EvolvingArtifact, evolution.RoundInfo, evolvable.VersionVector, parallel.ParallelStrategy, trajectory.TrajectoryStore, invariants.Invariant, audit.boost.VerifierBoost, evolve_settings.BudgetSettings/RuntimeSettings/ObservabilitySettings). The sweep pins names, not paths — a wrong module would need a hand check, and this PR records the one it had.

The branch's two commits (6403221 test, 691b00f fix) are unchanged; the tip is a plain merge of main at 1aa816e0 (the branch had fallen two merges behind) plus the changelog fragment and its wording fix. No rebase, no force-push.

Verification at the tip

Tip SHA: e3d9475c47cca456e86d8d989b16b00cc8493808

Ran:

  • pytest -q -o addopts='' tests/test_annotation_names.py → 3 passed
  • focused eleven files (annotation_names, agent_society, async_evolve, budget, evaluation, evolution, evolve_settings, evolvespec, proofs, selection, selfplay) → 251 passed, 8 warnings
  • the new test file alone on base 1aa816e0 → 1 failed, 2 passed, printing all 24 rows
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 15 changelog fragment(s) OK

Not run:

  • full pytest -q — the offline suite stalls on this host once the ~/.agentdescent rows hit a read-only $HOME; the CI matrix is the line
  • mkdocs build --strict — no docs/ page or mkdocs.yml changes in this diff

Scope

Import lines and if TYPE_CHECKING: blocks across the twelve modules above, one new test file, one changelog.d/ fragment. No runtime code path, no signature text, no generated docs.

Every module here defers annotation evaluation, so a name the module never
binds at module scope costs nothing at import time and nothing in CI. The new
test resolves the property statically: annotation names are read out of the
AST (string annotations and nested subscripts included) and must be bound at
module scope, where get_type_hints resolves them.

Module scope only, by construction: a function-local import binds that
function's locals, never the module globals. Two controls pin that -- one
synthetic, one on async_evolve.py, whose function-local EvolvingArtifact
import must not count -- and a TYPE_CHECKING import must count.

Fails on main at e43e847 with the 24 rows this issue lists.
Sixteen names over twenty-four annotated objects in twelve modules were never
bound at module scope, so resolving any of those annotations raised NameError
while the suite stayed green. Nothing here changes runtime behaviour: every
module already defers annotation evaluation, and the cross-module names stay
behind TYPE_CHECKING because the cycles are why those annotations are strings.

Three typing lines gain a name (Union, Optional, Sequence), one existing
TYPE_CHECKING block gains CallLimit, seven modules gain a TYPE_CHECKING block,
and selection.py takes a plain module-level import random -- stdlib, no cycle.
@github-actions github-actions Bot added risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller agent-pr Pull request opened by an agent labels Nov 21, 2026
@HugoXK

HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Advisory Verifier report; .github/agent/review-policy.json counts only Birfy, so this is evidence, not a counting PASS.

Verdict: CHANGES REQUESTED
Tip SHA: 15a9218

Ran:

  • pytest -q tests/test_annotation_names.py at this tip -> pass (3 passed)
  • pytest -q on the 11-file focused set named in the body -> pass (251 passed, 8 warnings)
  • the same new test file on base 1aa816e0 (only that file added) -> fail (1 failed, 2 passed; 24 rows)
  • python -m tools.gen_api_docs --check -> pass (docs/api.md is up to date)
  • python -m tools.changelog_fragments check -> pass (15 changelog fragment(s) OK)
  • local risk_gate.py --pr 1207 -> T3 (362 changed lines, 14 files)
    Not run:
  • pytest -q (full suite) -> stalls on this host; the PR body does not claim it either
  • mkdocs build --strict -> no mkdocs in the review venv; the diff changes no docs/ page or mkdocs.yml

Summary

Import-only change across 12 modules: Union, Optional, Sequence join existing typing lines and selection.py gets a plain import random, while 12 cross-module names sit behind new (7) or extended (1) if TYPE_CHECKING: blocks; plus tests/test_annotation_names.py (306 lines, AST sweep with two rule controls) and one changelog fragment. Fixes #1203; no runtime path, signature, docs/api.md or CHANGELOG.md change.

The sweep is well built and its red is real: 24 rows at base 1aa816e0 with the test file alone (agentdescent/selection.py:__init__ -> random, from the function-local import, included), 0 rows at the tip, and deleting one import re-reds exactly that row. Function-local imports are not credited, if TYPE_CHECKING: counts as module scope by the stated rule, class bodies and nested functions resolve against the right namespaces. All 12 TYPE_CHECKING import paths resolve to a real definition at the tip.

Blocking

  1. changelog.d/1203-annotation-names.md:2-4 claims a runtime fix the change does not deliver. "…so typing.get_type_hints() on those objects no longer raises NameError" is false for the 12 names imported only under if TYPE_CHECKING:: that block never runs, so the name is still absent from the module globals get_type_hints resolves against. Measured at this tip on the reachable annotated objects: 5 resolve, 10 still raise NameError — evolution.evolve: name 'ParallelStrategy' is not defined; evolve_settings.BudgetSettings: name 'CallLimit' is not defined; proofs.ProofRunner.__init__: name 'Invariant' is not defined; the same for async_evolve.async_evolve, audit.wiring.Audit/attach, budget.budgeted_completion, evolution._Runtime, evolvespec.build_call_limit/_limit_from_knobs — and the two nested-function rows (selfplay.on_round -> RoundInfo, evolution._record_rollout -> VersionVector) name values absent from their module's runtime globals. 20 of the 24 rows still raise; at base all 24 did. The issue itself says "Nothing here changes runtime behaviour", so the fragment contradicts it.

    Fix direction (one file, no other code needed): state what the change does — module-scope bindings a static reader resolves, with the cycle-prone names kept behind if TYPE_CHECKING: and runtime resolution explicitly unchanged. If the runtime property is wanted where no cycle exists, import for real: selfplay.py:48 already imports .evolution at runtime, so RoundInfo can join that line (evolve_settings/evolvespec need a cycle check first). Do not add a get_type_hints test: the issue's acceptance 4 rules it out for the ~50 pre-existing TYPE_CHECKING-only references.

  2. The PR body has no verification block. .github/pull_request_template.md:8-13 requires ## Verification at the tip with Tip SHA:, Ran: and Not run: (AGENTS.md pipeline step 4; CODE_STANDARDS.md:9-18), and the body has none of the three: "Green at the tip" never names 15a9218c…, and nothing says the full pytest -q and mkdocs build --strict were not run. Body-only; add the block with the sha, the three commands that did run, and those two under Not run:.

Non-blocking

  • tests/test_annotation_names.py:146-184 pins names, not import paths: a from .wrong_module import Name would satisfy the sweep, and a TYPE_CHECKING import never executes, so the suite cannot catch a bad path. I checked all 12 by hand at this tip and each exists (budget.CallLimit:89, agents.Completion:40, evolution.EvolvingArtifact:380, evolution.RoundInfo:1323, evolvable.VersionVector:51, parallel.ParallelStrategy:203, trajectory.TrajectoryStore:282, invariants.Invariant:45, audit/boost.VerifierBoost:147, evolve_settings.BudgetSettings:95 / RuntimeSettings:127 / ObservabilitySettings:161).
  • _annotation_sites skips class-attribute annotations on a class nested inside a function (tests/test_annotation_names.py:173: in_function short-circuits AnnAssign), and _scope_imports (:36-65) has no match case. Both are 0 occurrences in the package today, so no row is missed; a comment would keep the sweep honest if it stays evergreen.
  • "24 sites" counts (module, object, name) pairs; the same rows occur in 31 annotation expressions at base, and the two selection.py:__init__ methods share one row. Cosmetic.

Risk / merge

risk_gate -> T3 (362 changed lines, 14 files; agentdescent/async_evolve.py and agentdescent/evolution.py are high-risk paths). Fix both items on this branch; a new tip voids this report. After a PASS at the tip, T3 still needs Danyang's PASS and explicit merge click.

@HugoXK

HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Both blocking items from the advisory report are fixed at tip e3d9475c47cca456e86d8d989b16b00cc8493808:

  1. changelog.d/1203-annotation-names.md no longer claims a runtime fix. It now states what the change does — module-scope bindings a static reader resolves, twelve cycle-prone names kept behind if TYPE_CHECKING:, and runtime get_type_hints() explicitly unchanged for those names. The measurement is right: the four typing-line/random rows resolve at runtime, the TYPE_CHECKING rows do not, which is why the card scopes this as AST-level.
  2. The PR body now has ## Verification at the tip with Tip SHA:, the five Ran: commands and the two Not run: lines.

No code change beyond the fragment wording; the sweep, the two rule controls and the import paths are unchanged.

@HugoXK

HugoXK commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator Author

Advisory Verifier report; .github/agent/review-policy.json counts only Birfy, so this is evidence, not a counting PASS.

Verdict: PASS (advisory)

Tip SHA: e3d9475c47cca456e86d8d989b16b00cc8493808

Ran (fresh clone, refs/pull/1207/head):

  • full offline suite at this head, one command, -n 8 -o addopts="" -> 21 failed / 7956 passed / 63 skipped / 10 errors in 290.30s. The same command at base 1aa816e0 -> 22 failed / 7952 passed / 63 skipped / 10 errors in 400.10s. The tip's 31 red ids are all base ids: 0 new. One base-only red, tests.test_evalcache::test_concurrent_callers_for_one_key_compute_once — load-flaky (1 failed / 2 passed over three isolated base runs) and it touches no file in this diff.
  • red at base with only tests/test_annotation_names.py added -> 1 failed, 2 passed, the failure printing 24 module:object -> name pairs; the two rule controls pass by design. My own independent AST sweep (not the PR's) at base: 31 annotation expressions / 24 pairs / 16 names / 12 modules; at this tip: 0.
  • focused eleven-file set named in the body at this head -> 251 passed, 8 warnings in 75.23s.
  • all twelve touched modules import at the tip, and all 40 TYPE_CHECKING names package-wide resolve to a real definition (AST check) — the sweep pins names, not paths, so this is the path check.
  • python -m tools.gen_api_docs --check -> docs/api.md is up to date; python -m tools.changelog_fragments check -> 15 changelog fragment(s) OK. No docs/ page or mkdocs.yml change, so mkdocs build --strict is out of scope for this diff.

Runtime reach, matching the corrected fragment: get_type_hints at this head resolves the four names bound for real (agent_society.society_evolve/Union, evaluation.EvaluationReport/Optional, scheduler.ResumeQueue.pop_for/Sequence, selection.ParetoFrontier.__init__/random — every one a NameError at base), and still raises for proofs.ProofRunner.__init__/Invariant and budget.budgeted_completion/Completion, the TYPE_CHECKING-only shape issue #1203 scopes out (acceptance 4).

Head move: my earlier read at 15a9218c93 is void — the head moved to e3d9475c while I was measuring. The only delta is changelog.d/1203-annotation-names.md (+10/−4), and it now states the static property and the runtime limit correctly. Both items the earlier advisory report raised are closed: the fragment no longer claims a runtime fix, and the body carries ## Verification at the tip with Tip SHA: equal to this head.

Risk: T3 by path (agentdescent/async_evolve.py, agentdescent/evolution.py) and by size (14 files, 362 changed lines) — matches risk:T3.

Not run:

  • docker sandboxes (no daemon on this box — the same 10 errors at both ends)
  • wheel smoke (no packaging or release path in the diff)
  • real provider (offline box; nothing in the diff calls one)
  • risk_gate.py (no gh here; the tier is computed from .github/agent/risk-rules.json)

Findings: none blocking.

Record (non-blocking):

  • Labels are agent-pr + risk:T3 only — no nature label and no area:* (issue fix(typing): sixteen annotation names in the package are never bound at module scope #1203 carries bug, chore).
  • The body's Not run: says the full suite "stalls on this host"; it does not — it completes in 290s here, so the head now has a full-suite measurement instead of "the CI matrix is the line".
  • mergeable_state read blocked because test (3.9)–test (3.13) were still in progress at this read; build, otel-wire, risk tier and reviewed tip are green.

No UI surface: import lines, if TYPE_CHECKING: blocks, one AST test and one changelog fragment; nothing is rendered.

No merge action from QA: this report is advisory, the merge gate is the repo's review decision on the head sha, and the merge itself is executed by the repo's merge process.

@HugoXK

HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS
Tip SHA: e3d9475

Ran (fresh clone; base worktree at 1aa816e0):

  • red: base 1aa816e0 + only this branch's tests/test_annotation_names.py -> 1 failed, 2 passed; the sweep prints all 24 rows, including agentdescent/async_evolve.py:async_evolve -> EvolvingArtifact and agentdescent/selection.py:__init__ -> random; the two rule-control tests pass on base by design.
  • pytest -o addopts="" -q -p no:cacheprovider tests/test_annotation_names.py at the tip -> 3 passed.
  • focused eleven files (annotation_names, agent_society, async_evolve, budget, evaluation, evolution, evolve_settings, evolvespec, proofs, selection, selfplay) -> 251 passed, 8 warnings.
  • independent AST scan using issue fix(typing): sixteen annotation names in the package are never bound at module scope #1203's own reproducer (not the new test's helper): 31 annotation occurrences / 24 (object, name) pairs at base, 0 at the tip.
  • python3 -m tools.gen_api_docs --check -> docs/api.md is up to date.
  • python3 -m tools.changelog_fragments check -> 15 changelog fragment(s) OK.
  • all twelve TYPE_CHECKING import paths resolve against their target modules (12/12).
  • git diff -U0 1aa816e0..HEAD -- agentdescent/: every added line is an import or an if TYPE_CHECKING: header; no annotation text, signature, __all__ or generated file changed.
  • full pytest -q at 15a9218c (the only delta to this tip is changelog.d/) -> 20 failed, 7969 passed, 61 skipped in 2420.63s; 19 of the 20 fail identically at base 1aa816e0 (test_era_srbench.py 6, test_genesis_example.py 1, test_sandbox_container.py 12 -- host Docker/bench data, not this diff), the 20th is test_async_evolve.py::test_shutdown_grace_is_respected_and_warns, a timing test that passes 3/3 standalone here and 2/2 at base.
  • CI at e3d9475c: 10 checks, build / test 3.9-3.13 / otel-wire all success, deploy skipped; report-only risk tier and reviewed tip success (test 3.9 completed 2026-11-21T18:14:51Z).

Not run:

  • mkdocs build --strict -> no docs/ page or mkdocs.yml change in the diff.
  • wheel smoke -> no packaging/release path change; Docker/real-model checks -> not applicable to an import-only change.

Summary

Fixes #1203. 14 files +356/-12: twelve package modules gain only imports (three join runtime typing lines, selection.py a plain import random, twelve names behind TYPE_CHECKING), plus tests/test_annotation_names.py with two rule controls, one changelog fragment. The sweep is module-scope-only and deduped per (module, name), the red names every row, and the type-checking-only credit/deny controls pin the rule so it cannot silently weaken. The body and the changelog text at this tip state the runtime limit correctly: typing.get_type_hints() still raises NameError for 20 of the 24 pairs because those twelve names are TYPE_CHECKING-only -- I re-probed and got exactly that, so the new wording is true. The earlier 15a9218c sentence ("no longer raises") was wrong and the follow-up commit fixes it; this report judges e3d9475c.

Blocking

None.

Non-blocking

  • "24 sites" in the body, the fragment and the test message is 24 distinct (object, name) pairs over 31 annotation occurrences (EvaluationReport/Optional x4, budgeted_completion/Completion x2, selection.__init__/random x3, society_evolve/Union x2). Nothing depends on the distinction, but "sites" reads as occurrences.
  • selection.py keeps three function-local import random (lines 438, 598, 1031) that the new module-level import makes redundant; harmless and consistent with the "no reordering beyond the added name" non-goal.
  • PR labels are agent-pr + risk:T3 only; the repo asks for one type and one or more area:* on PRs (author side, not code).
  • The body lists full pytest -q under Not run: because of the read-only $HOME here; the host result above is the line for that gap, and the CI matrix is the merge-gate evidence.

Risk / merge

risk:T3 (14 files, package code). Advisory only: no merge action from QA -- this report is advisory, the merge gate is the repo's review decision on the head sha, and the merge itself is executed by the repo's merge process.

No UI surface: typing-import-only change; no CLI, MCP, docs page or rendered UI text changes (the changelog fragment is release text, not a surface).

@HugoXK

HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Review tip: e3d9475c47cca456e86d8d989b16b00cc8493808

Blocking: none. Both items from 15a9218c are fixed at this tip:

  • changelog.d/1203-annotation-names.md:1-11 now scopes the change to static readers and states that runtime typing.get_type_hints() is unchanged for the twelve if TYPE_CHECKING: names. That matches the runtime probe: 5 annotated objects resolve, 10 still raise NameError (base 1aa816e0: 24 of 24 raise). No get_type_hints test was added.
  • The PR body has ## Verification at the tip with Tip SHA: e3d9475c…, five Ran: lines and Not run: for the full suite and mkdocs, per .github/pull_request_template.md:8-13.

The code is unchanged from 15a9218c — the only diff between the two tips is the fragment (git diff --stat 15a9218c..e3d9475c = 1 file), so the earlier code assessment stands.

Ran at this tip (serial, HOME=/home/xuke PYTHONPATH=., from a clean clone at e3d9475c):

  • tests/test_annotation_names.py → 3 passed
  • focused eleven files (annotation_names, agent_society, async_evolve, budget, evaluation, evolution, evolve_settings, evolvespec, proofs, selection, selfplay) → 251 passed, 8 warnings
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • python -m tools.changelog_fragments check → 15 changelog fragment(s) OK
  • base 1aa816e0 with only this tip's test file → 1 failed, 32 passed (the test file is identical at both tips)

Not run: full pytest -q (8050 tests; 15 minutes reached 34% on this host, and the only failure in that window is the pre-existing tests/test_dir_evolution.py::test_write_to_backs_up_before_overwriting — OSError: [Errno 30] Read-only file system: /home/xuke/.agentdescent — which reproduces at 1aa816e0); mkdocs build --strict (not installed).

Non-blocking:

  • changelog.d/1203-annotation-names.md:6 lists "a schema generator" among the static readers. A runtime schema generator still fails on the twelve TYPE_CHECKING names; line 8's carve-out is what keeps the claim true, so naming the AST-based tools/gen_api_docs instead would be tighter.
  • changelog.d/1203-annotation-names.md:2 is 81 columns; changelog.d/README.md asks for about 80.

Advisory only: .github/agent/review-policy.json counts only Birfy, so this is not a counting PASS and no approval is recorded here.

@Birfy

Birfy commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: e3d9475
Ran:

  • .github/scripts/risk_gate.py --repo ThinkFlowLab/agentdescent --pr 1207 → T3 (368 lines / 14 files; path agentdescent/async_evolve.py, agentdescent/evolution.py)
  • pytest -o addopts="" -p no:cacheprovider -q tests/test_annotation_names.py at this tip → 3 passed
  • python -m tools.changelog_fragments check → 15 fragments OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • Required CI at this tip → green (build, test (3.9)–test (3.13); otel-wire also green)
    Not run:
  • Full offline suite locally — the CI test matrix is the line
    Findings:
  • No actionable defects. The earlier CHANGES REQUESTED at 15a9218c is addressed at this tip: the changelog now scopes the change to static readers and states that runtime typing.get_type_hints() is unchanged for the twelve names behind if TYPE_CHECKING:. Union / Optional / Sequence join existing typing imports, selection.py gets import random, and the twelve cycle-prone names sit behind TYPE_CHECKING. The new sweep test covers the 24 sites.
    Verdict: PASS

@Birfy
Birfy merged commit b511505 into main Nov 21, 2026
10 checks passed
@Birfy
Birfy deleted the agent/1203-annotation-names branch November 21, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(typing): sixteen annotation names in the package are never bound at module scope

2 participants