[Adrian Madrid (aemadrid-twx)]

The loop guard's VERIFY_RE knows rspec, but not the way most Ruby projects run minitest or test-unit: ruby given a test file (ruby -Itest test/models/user_test.rb), rake test or bin/rails test. And runner_verdict knows rspec's summary, but not minitest's 8 runs, 9 assertions, 1 failures, 0 errors, 0 skips. So in a minitest repository the guard sees no verification at all. A turn whose last test run failed still counts as ended green and feeds the envelope, and the judge is told "No test or verification run has been seen in this turn."

This PR adds to VERIFY_RE:

• rake test and rake spec, with NAME=value arguments before them and a namespaced task (rake test:models)
• rails test, which with the optional directories already in the pattern covers bin/rails test
• ruby given a file named like a test (test_*.rb, *_test.rb, *_spec.rb), with the options it is spelled with (-Ilib, -I lib, -w, -W0, --disable-gems). A one-off ruby -e '...', a syntax check (ruby -wc x.rb, ruby -c x_test.rb) and a script that is not a test (ruby repro.rb) still don't count.

bundle exec was already passed over by _PREFIX_RE. runner_verdict gets minitest's and test-unit's summary for a failed and a passed run. state.TEST_RUN_RE and ERR_RE are unchanged: minitest's failure count already matches ERR_RE.

The tests are in tests/test_live_core.py, next to the launcher and runner-summary tests:

• 15 Ruby test commands that count as a verification, so a failing run after an edit means the turn did not end green
• 12 Ruby commands that don't count (ruby -e, syntax checks, scripts, rake db:migrate, rails server)
• a piped run (ruby test/a_test.rb 2>&1 | tail -5) judged by minitest's summary, failed and then passed
• four rows in the runner-summary table

How it was produced: Claude Code, an AI coding agent, found the gap. A probe of the plugin on a small Ruby repository recorded test_runs: 0 in its envelope although every session ran the tests. Claude Code then wrote the fix and the tests. I directed the work.

What was checked, on macOS 26.6 with Python 3.13, on main at 81e32e5:

• Without the fix, the 20 new test cases fail. With it, python -m pytest tests reports 927 passed, 16 skipped and 0 failed.
• experiments/tests: 6 tests in test_model_spelling.py and test_tooling.py fail with and without this change. They check against the Codex and Copilot CLIs installed on this machine, whose versions are outside the tested range.
• Following the repository's guidance, I replayed 90 recorded Claude Code sessions through TurnGuard, the way history.replay_envelope folds them. They are headless runs on small Ruby tickets, of two kinds: 45 where the tests were in the repository and 45 where they were held out.
  • Before the fix, no session had a verification. After it, all 45 sessions with tests in the repository have at least one, with 158 runs in all.
  • None of the 45 held-out sessions gets one. There the agent's Ruby commands were ruby -Ilib -e '...' checks, syntax checks and cat test/....
  • In 2 sessions the last verification reads failed while the ticket itself passed. Both are correct: the agent's last command ran other tickets' test files, not yet implemented, and they failed.
  • The only test runs still missed are 4 loops (for f in test/*.rb; do ruby $f; done), whose file the pattern can't see. That is the same limit as for any runner given a variable.

Known unknowns:

• A bare rake whose default task runs the tests is not counted, since rake alone may run anything.
• ruby test/test_helper.rb would count, since it is named like a test file. I didn't see it in practice.
• Not tested against a Rails application's sessions, only plain minitest.

🤖 Generated with Claude Code

[Adrian Madrid (aemadrid-twx)]

@microsoft-github-policy-service agree company="Timeworxs"

[Muhammad Abdullah Butt (1AbdullahButt)]

One of the negative cases the description gives, a syntax check (ruby -wc x.rb), still counts as a verification when the file is named like a test. VERIFY_RE starts with (?i), so the option alternatives in the new ruby branch (turn.py:48) are case-insensitive:

• -W\S* also matches -w…, so it consumes -wc (and -Wc) whole.
• -[IrC](?:\S+|...) also matches -c followed by attached letters, so it consumes -cw.

I copied the ruby branch of the pattern (with its (?i) and the trailing lookahead) into a standalone snippet:

ruby -wc test/a_test.rb     True
ruby -Wc test/a_test.rb     True
ruby -cw test/a_test.rb     True
ruby -c test/a_test.rb      False   (the case in the new negative test)
ruby -wc lib/retry.rb       False   (not test-named)

So an agent that syntax-checks the test file it just edited gets a verification, and with exit 0 and output Syntax OK, runner_verdict returns None. A turn can then look verified without any test having run.

Scoping the option group with (?-i:...) fixes it and keeps every positive case in the PR. Something like:

r"ruby(?-i:\s+(?:-[Ir](?:\S+|\s+[^\s\-]\S*)|-C\s*\S+|-W[0-2]?(?::\S+)?|-w|--[\w\-]+(?:=\S+)?))*\s+"

With that, the five lines above all give False, and ruby -w -Ilib test/slugify_test.rb, ruby -I lib test/x_test.rb, ruby -W0 --disable-gems test/test_parser.rb, ruby -Itest test/models/user_test.rb, ruby spec/foo_spec.rb and ruby -W:no-deprecated test/a_test.rb still match. It drops the bundled-flags alternative -[abdf-z]+. If you want to keep that, put it inside the (?-i:...) group without the c. Adding "ruby -wc test/a_test.rb" and "ruby -cw test/a_test.rb" to test_ruby_that_runs_no_test_file_is_not_a_verification would pin it.

I also read the rake/rails alternatives, the minitest and test-unit summary regexes, and _PREFIX_RE (which strips bundle exec). Those look right. git merge-tree against main is clean. I didn't run the test suite, only the regex itself.

Reviewed with help from Claude Code.

[Adrian Madrid (aemadrid-twx)]

Thanks, you're right. Under the pattern's (?i), -W\S* took -wc and -[IrC] took -cw, and I had only tried -wc on a file not named like a test.

The second commit scopes the options with (?-i:...) as you suggested, with -C\s*\S+ and -W[0-2]?(?::\S+)?. I kept the bundled-flags alternative inside the group, without c and e, so ruby -wv test/a_test.rb still counts.

The negative test now has ruby -wc, -Wc, -cw and -we on a test-named file, and the positive one has -W:no-deprecated, -wv and -C /w -Itest. python -m pytest tests reports 934 passed, 16 skipped and 0 failed, and the replay of the 90 recorded sessions gives the same counts as before.