Skip to content

fix(python): update bare-verify tests for the default gate set - #499

Merged
dmealing merged 1 commit into
mainfrom
fm/mo-main-ci-red
Oct 28, 2026
Merged

dmealing merged 1 commit into
mainfrom
fm/mo-main-ci-red

Conversation

@dmealing

@dmealing dmealing commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

What Changed

  • test_cli_config_verify.py: test_verify_codegen_bare_defaults_to_codegen now copies the canonical prompts/ directory into the temp project. Bare verify also runs the template gate against the default prompts/ root, and the fitness model's template.prompt refs resolve there. Before this change the test exited 1.
  • test_cli_verify_requirements.py: the "no requirement, no generators selected" test now expects two stderr lines instead of one. The first is the default-gate-set note, which now reads running --codegen --templates (default).. The second is the template gate's skip note, because no --prompts root is named. Before this change the single-value unpack raised ValueError.
  • Test-only change, no product code touched. It brings both Python tests in line with the bare-verify gate set that 71e9cd7c6 introduced. Those two tests were the failing Python gate on main.
Validation

Risk Assessment

✅ Low: Test-only edits that match _cmd_verify's current behavior, which runs the template gate (prompts/ root) and prints the skip note. No source behavior changes.

Testing

The change only edits test files, since the CLI code was already right. I ran the two target tests at the target commit and both pass. With the old test files from the base commit they fail, so the red CI is reproduced and the fix is confirmed. I then ran the full Python suite, split into the non-integration part and the Docker-backed integration part as ci-local.sh does: 3812 passed, plus 420 passed and 11 skipped. The TypeScript baseline had already passed. No product code changed, and the worktree is clean.

  • Live validation: ✅ go - 4 of 4 scenarios driven live against the product
Scenario Result Live Evidence
Bare metaobjects verify over a model with no requirements and no generators exits 0 and prints exactly the two expected stderr lines (default gate-set note, then template-gate skip note) ✅ pass live test_cli_verify_requirements.py::test_a_model_with_no_requirement_and_no_generators_selected_still_exits_0_and_says_nothing_new passes at HEAD and fails at base (before-fix.txt)
Bare metaobjects verify on a config project with prompts present exits 0 after gen (codegen + template gate) ✅ pass live test_cli_config_verify.py::test_verify_codegen_bare_defaults_to_codegen passes at HEAD. At base it failed with 'ref fitness/coach-note did not resolve under templates'.
Adversarial: the base-commit versions of both tests still fail, so the fix is what turns them green ✅ pass live before-fix.txt: 2 failed, 34 passed with the old test files
Full Python regression suite stays green (unit and integration) ✅ pass live python-full.txt: 3812 passed. Integration run with docker: 420 passed, 11 skipped.
Evidence: Before fix: base-commit tests fail (2 failed)
----------------------------- Captured stderr call -----------------------------
metaobjects verify — running --codegen --templates (default). Explicit subverbs: --codegen (codegen drift), --templates (template/prompt drift), --db (schema drift — not supported in Python).
error: [coachNote] ref 'fitness/coach-note' did not resolve under templates (missing template).
metaobjects verify --templates: 1 drift error(s).
=========================== short test summary info ============================
FAILED tests/codegen/test_cli_verify_requirements.py::test_a_model_with_no_requirement_and_no_generators_selected_still_exits_0_and_says_nothing_new
FAILED tests/codegen/test_cli_config_verify.py::test_verify_codegen_bare_defaults_to_codegen
2 failed, 34 passed in 37.54s
Python full non-integration suite: 3812 passed.

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 4 of 4 scenarios driven live against the product
Scenario Result Live Evidence
Bare metaobjects verify over a model with no requirements and no generators exits 0 and prints exactly the two expected stderr lines (default gate-set note, then template-gate skip note) ✅ pass live test_cli_verify_requirements.py::test_a_model_with_no_requirement_and_no_generators_selected_still_exits_0_and_says_nothing_new passes at HEAD and fails at base (before-fix.txt)
Bare metaobjects verify on a config project with prompts present exits 0 after gen (codegen + template gate) ✅ pass live test_cli_config_verify.py::test_verify_codegen_bare_defaults_to_codegen passes at HEAD. At base it failed with 'ref fitness/coach-note did not resolve under templates'.
Adversarial: the base-commit versions of both tests still fail, so the fix is what turns them green ✅ pass live before-fix.txt: 2 failed, 34 passed with the old test files
Full Python regression suite stays green (unit and integration) ✅ pass live python-full.txt: 3812 passed. Integration run with docker: 420 passed, 11 skipped.
  • scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains
  • uv run --frozen pytest tests/codegen/test_cli_verify_requirements.py tests/codegen/test_cli_config_verify.py -q at target commit (36 passed)
  • Same command with the two test files checked out from base fb2db7822 (2 failed, 34 passed), then restored to HEAD
  • uv run --extra integration pytest tests/ --ignore=tests/integration -q -n 8 (3812 passed)
  • uv run --extra integration pytest tests/integration -q -n 8 with docker up (420 passed, 11 skipped)
  • Baseline scripts/ci-local.sh --only ts-fast --only ts-unit --strict-toolchains (ran before this phase, passed)
⏭️ **Document** - skipped

Step was skipped.

⚠️ **Lint** - 1 warning
  • ⚠️ linter found issues (exit code 1)
✅ **Push** - passed

✅ No issues found.

#475 made a bare verify run the template gate as well as the codegen gate. Two tests
still described the old behaviour: one asserted a single stderr line, the other ran a
config project with no prompt templates. They now match docs/features/cli.md and the
Unreleased CHANGELOG entry; the code was right.
@dmealing
dmealing merged commit 7fade2b into main Oct 28, 2026
1 check passed
@dmealing
dmealing deleted the fm/mo-main-ci-red branch October 28, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant