Repository navigation
Run CHASM scheduler property tests in a no-retry CI tier - #72
Open
chaptersix wants to merge 5 commits into
Open
chaptersix wants to merge 5 commits into
chaptersix wants to merge 5 commits into
Conversation
The scheduler property test landed inside schedulertest, which the Makefile globs into UNIT_TEST_DIRS. That tier runs through test-runner with --max-attempts=3, and test-runner exits with the exit code of its last attempt only. Each retry re-runs the test in a fresh process where rapid draws a new random seed, so a retried property test checks a different set of inputs than the one that failed: a real counterexample is replaced by a likely-passing run and the job goes green having found a bug. Give property tests their own tier instead: - PROPERTY_TEST_DIRS is derived from a directory naming convention (any package directory ending in "proptest"), so a new property test package is enrolled, and excluded from the retrying unit tier, automatically rather than by a hand-maintained list whose failure mode is silent. - property-test and property-test-coverage run the tier. The latter goes through test-runner to keep the JUnit report, coverage and flaky-test reporting, but with --max-attempts=1. PROPERTY_TEST_MAX_ATTEMPTS is assigned with := so the MAX_TEST_ATTEMPTS environment variable set in the CI workflow cannot re-enable retries for this tier. - A Property test job runs the tier on every PR at rapid's default 100 checks and uploads rapid's minimized counterexample as an artifact on failure. - A nightly Property Test Soak workflow runs the same tier at 100k checks. It is deliberately not a required check: a counterexample it finds is a real bug, but on an input that no particular PR introduced. Move the property test to chasm/lib/scheduler/schedulerproptest, and promote the configWith helper it shared with ledger_test.go to schedulertest.ConfigWith. A dedicated directory is required because the tier subtracts whole directories from UNIT_TEST_DIRS; schedulertest returns to the unit tier, where retries are appropriate for its unit tests. Document the tier in docs/development/testing.md, including when and where to commit a counterexample: rapid replays every fail file under testdata/rapid/<TestName>/ ahead of the random cases, so one belongs with the fix that resolves it and must not be committed for an unfixed bug, which would fail every PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The deferred-start stranding that temporalio#13042 documented was already fixed when that PR merged. temporalio#13069 ("Keep CHASM schedule buffers progressing after discarded or failed starts") added the reenableDeferredStarts call in Invoker.addTasks, which returns deferred starts to Attempt == 0 when no pending or running start remains so the next ProcessBuffer pass reconsiders them. That commit is an ancestor of temporalio#13042, landing roughly seven hours and four commits earlier, so the skip and the invariant filter were stale on arrival — presumably written against an older base and rebased without re-checking. Both suppressions pass silently once the bug is gone, which is why they survived: nothing fails when the condition they describe stops happening. The filter was also broader than the bug it named, discarding every ledger-lost-start violation where nothing was running and all pending starts were deferred, not just the stranding case. - Un-skip TestLedger_DeferredStartsStrandedByPausedCompletion, and rewrite its comment, which described the broken behaviour as current, to explain the gap and name reenableDeferredStarts as the mechanism that closes it. - Delete checkInvariantsIgnoringKnownBugs, whose own comment asked for exactly this, and use schedulertest.CheckInvariantsHook so the property test now asserts the full invariant set. Verified with 300k generated cases (200k in one run plus five independent seeds at 20k each) and with the un-skipped test under -race -shuffle on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sions Picked up from #71, which built a property test tier independently and got two things right that this branch had wrong. The property-test job was not in the test-status aggregate gate's needs list, so it ran but could not block a merge: test-status reported success regardless of its result. Add it. More importantly, a rapid fail file is not a durable regression test, so testdata/rapid/ is now gitignored instead of committed. A fail file replays a raw sequence of draws, so adding, removing or reordering a draw in a generator makes the stored sequence decode to different values. Verified against rapid v1.3.0: with the original generator a stored fail file reproduces immediately ("failed after 0 tests"); after inserting one draw ahead of the existing one, the same file no longer reproduces and the test simply passes. rapid reports nothing — the protection disappears without anything failing to say so, which is the same silent-rot failure as the stale t.Skip removed in the previous commit. Replace the "commit the fail file" guidance in the docs, the Makefile and the soak workflow with pinning the case as a hand-written deterministic test, and add a "Deferring a fix" section that spells out why skips and invariant filters rot, using the temporalio#13042/temporalio#13069 sequence as the worked example. Also narrow the counterexample artifact glob to *.fail, matching #71. Deliberately not taken from #71, to keep this change small: the production scheduler fixes and their regressions, the chasmtest.TaskRunner harness extraction, the rapid state-machine rewrite of the property test, the migration work and its proto field, and the mutation-test tool. Its rapid-import discovery for PROPERTY_TEST_DIRS was also left out: it pairs with #71 leaving property packages in the unit tier, whereas this tier subtracts them, where auto-discovery would pull an entire production package's unit tests out of the retrying tier. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Raise the per-PR property tier to 1000 generated cases, and take the chasmtest.TB change from #71 without the driver rewrite that PR builds on top of it. chasmtest.Engine took a *testing.T, so a driver built inside a rapid property had to be handed the outer *testing.T. Anything the harness itself reported — a gomock expectation, an engine assertion, an error-level test log — then failed the enclosing test instead of the generated case, unshrunk and unattributed. Only the explicit invariant checks, which were reported through the *rapid.T, could shrink. Declare chasmtest.TB as the subset of testing.TB the engine, gomock and testlogger actually need, and take it in NewEngine, Engine.t and InvariantCheck. *testing.T still satisfies it, so no caller changes; *rapid.T satisfies it too, including FailNow transitively through zaptest.TestingT, so testify's require keeps working. Widen the three scheduler invariant helpers and the handful of InvariantCheck literals that the narrower signature pinned to *testing.T. schedulertest.NewDriver, NewDriverFromMigration, newDriverScaffold and Driver.T take chasmtest.TB as well — the driver only ever used Helper and Fatalf — and the property test now builds its driver on rt. Verified by failing through the driver's own T inside a property: the case shrinks to the same minimal draw as an identical property with no driver (401 of 1..1000, over four consecutive runs). Harness nondeterminism can still occasionally leave a case unminimized, since start RPCs run on concurrent goroutines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Empty commit to start a workflow run now that Actions are re-enabled on the repository. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stephanos
reviewed
Nov 17, 2026
|
|
||
| Because a sibling directory can only reach exported API, the package under test needs an exported | ||
| harness — see [schedulertest](../../chasm/lib/scheduler/schedulertest), which is a non-test package | ||
| precisely so property tests can import its driver. |
There was a problem hiding this comment.
^ Please edit and cut down verbosity heavily before you merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
CHASM scheduler property tests move into their own CI tier that runs without retries, the test engine is built on an interface so rapid can own harness failures, and two stale suppressions are removed.
PROPERTY_TEST_DIRSis derived from a directory naming convention: any package directory ending inproptestjoins the tier and is filtered out ofUNIT_TEST_DIRS.property-testandproperty-test-coveragerun the tier. The coverage target goes throughtest-runnerwith--max-attempts=1.Property testjob runs the tier on every PR at 1000 generated cases and joins thetest-statusgate. A nightlyProperty Test Soakruns the same tier at 100k cases and deliberately does not gate.chasm/lib/scheduler/schedulerproptest;schedulertestkeeps the harness and its own unit tests and returns to the unit tier.chasmtest.TBreplaces*testing.Tin the test engine and the scheduler driver.TestLedger_DeferredStartsStrandedByPausedCompletionis un-skipped andcheckInvariantsIgnoringKnownBugsis deleted.docs/development/testing.mdgains a "Property tests" section.Why?
A property test needs something from its runner that a unit test does not
test-runnerexits with the exit code of its last attempt (testrunner.go:450), and each retry re-runs the failed test in a fresh process. A fresh process means rapid draws a new seed, so the retry evaluates a different set of inputs than the one that failed. The counterexample is not re-examined; it is replaced by a likely-passing run, and the job goes green.This gets worse the better the test works. A property test earns its keep on rare counterexamples, and the rarer the input that triggers one, the more reliably three attempts bury it.
The natural fix is to scope retries off for that one package, and that turns out not to be expressible.
--max-attemptsis a single argument to a singletest-runnerinvocation covering all 248 unit packages, and there is no allowlist, annotation or per-test hook anywhere intools/testrunner. "No retries here" can only be said by making a separate invocation — which is why this is a tier rather than a flag.The tier still goes through test-runner
leak-testsets the precedent for a hermetic test that must not be retried, and it bypassestest-runnerentirely with a plaingo test. That also costs it the JUnit report, coverage and flaky-test reporting. We do not have to pay that:test-runnerrejects only--max-attempts=0, so=1is valid and keeps the whole reporting path while dropping the retries.Two details follow from putting the tier in CI rather than only in the Makefile:
PROPERTY_TEST_MAX_ATTEMPTSis assigned with:=, not?=.run-tests.ymlsetsMAX_TEST_ATTEMPTSas a workflow-level environment variable, and environment values beat a Makefile?=, so a?=knob would silently inherit 3 in CI while looking correct locally.MAX_TEST_ATTEMPTS: 1for itself, becausepost-test-reporting's crash heuristic isjunit_count -lt MAX_TEST_ATTEMPTS. A single-attempt job produces one JUnit file, so leaving the workflow default at 3 would emit a spurious crash report on every property failure.Property tests get their own directories
The tier subtracts whole directories from
UNIT_TEST_DIRS, because that glob is what sweeps new test packages into the retrying tier in the first place. Directory granularity then decides where property tests can live: one placed inside the package it exercises forces a choice between pulling that package's entire unit suite into the no-retry tier — forservice/matching, 229 test functions across 35 files — or leaving the property test where retries erase its counterexamples.So property tests live in a sibling directory. That is also the established shape here: roughly twenty
<pkg>testpackages already exist (chasmtest,nexustest,hsmtest,historytest,persistencetest).schedulertestwas built for it, its package doc noting it "deliberately lives in a non-test package so that property tests (pgregory.net/rapid) ... can import the driver". The constraint this imposes is worth stating plainly: a sibling directory reaches only exported API, so a package joining the tier needs an exported harness seam.Membership is a convention, not a list
An explicit
PROPERTY_TEST_DIRSlist would work, and it would need updating in two places, and forgetting it would leave a property test in the retrying tier with no symptom. That is the original problem wearing a different hat, so the list is derived instead:$(filter %proptest/,$(TEST_DIRS)). Creating the directory enrolls it and excludes it from the unit tier, with nothing to remember. Aproptestdirectory with no test file is correctly not enrolled, sinceTEST_DIRSonly contains directories holding a_test.go.Seeds stay random, and there are two tiers
Hypothesis ships a
ciprofile that setsderandomize=True, and pinning the seed is a legitimate way to stop a retry re-rolling the inputs. We get that determinism from--max-attempts=1instead, which leaves the randomness free to do what it is for: a fresh seed every run means input coverage accumulates across commits, rather than re-testing one frozen set forever.That splits naturally into a gating tier at 1000 cases (8.1s including compilation, cheap enough to sit in
test-status) and a non-gating nightly soak at 100k. The soak does not gate on purpose — a counterexample it finds is a real bug, but it is found on an input no particular PR introduced, so it must not block unrelated merges.Cases are set through
RAPID_CHECKSrather than-rapid.checksbecause the flag is only defined in test binaries that link rapid; passing it to any other package fails with "flag provided but not defined". The environment variable is simply ignored by binaries that do not read it.Fail files are reproduction aids, not regression tests
rapid writes a minimized reproducer and replays every stored file ahead of the random cases, which makes committing them look like a free permanent regression test. It is not. A fail file replays a raw sequence of draws, so adding, removing or reordering a draw in a generator makes the stored sequence decode to different values — and it then stops reproducing the bug silently, with no error and no warning.
Verified against rapid v1.3.0 in a scratch module. With the original generator, a stored file reproduces immediately (
failed after 0 tests). After inserting a single draw ahead of the existing one, the same file no longer reproduces and the test simply passes.So
testdata/rapid/is gitignored, CI uploads fail files as artifacts for local reproduction, and the documented workflow is to pin the case as a hand-written deterministic test. The docs also cover deferring a fix, because at.Skipor an invariant filter has the same silent-decay property and this repository has already been bitten by it (below).Why the CHASM test engine changed
chasmtest.Enginetook a concrete*testing.T, so a driver built inside a rapid property had to be handed the enclosing test'sT. The property test said so in a comment: "The outer*testing.Towns the engine scaffolding (it requires a realtesting.TB)."The consequence is that only explicit invariant violations shrank. Anything the harness itself reported — a gomock expectation, an engine assertion, an error-level test log — failed the enclosing test instead of the generated case: unshrunk, unattributed, and in a shape rapid could not minimize. For a tier whose whole purpose is turning a rare failure into a small reproducer, that is a hole in the most valuable half of the output.
chasmtest.TBis the subset the engine, gomock and testlogger actually need (testlogger.CleanupCapableT, plusErrorfandContext). It is backward compatible: every existing caller passes*testing.T, which satisfies it, andmake build-testsconfirms the repository still compiles.*rapid.Tsatisfies it too, includingFailNowtransitively viazaptest.TestingT, which is what keeps testify'srequireworking in the widened invariant helpers.Widening
InvariantCheckcascades, and the cascade is bounded: three scheduler invariant helpers and a handful of function literals pinned to*testing.T. Worth knowing why it stayed small — #71 avoids it by deletinginvariant_test.goand relocating those invariants intoschedulertest, which is the larger refactor this PR does not take.Why the suppressions go in the same PR
The tier is only worth having if the property actually asserts the full invariant set.
checkInvariantsIgnoringKnownBugsdiscarded everyledger-lost-startviolation where nothing was running and all pending starts were deferred — broader than the single bug it named — so it could swallow exactly the class of accounting failure the ledger invariant exists to catch. Building a no-retry tier around a property that filters its own findings would be theatre.Both suppressions also turned out to be stale on arrival. temporalio#13069 fixed the stranding by adding
reenableDeferredStartstoInvoker.addTasks, andgit merge-base --is-ancestorconfirms that commit is an ancestor of temporalio#13042, landing about seven hours and four commits earlier. They survived because nothing fails when the condition a skip describes stops happening — the skip's own comment asked for its deletion.Relationship to #71
#71 built a property tier independently and got two things right that this branch had wrong, both taken here: adding the job to the
test-statusgate, and treating fail files as disposable.chasmtest.TBis also from #71, without the driver rewrite built on top of it, and the artifact glob is narrowed to*.failto match.Deliberately not taken, to keep this reviewable: the production scheduler fixes and their regressions, the
chasmtest.TaskRunnerextraction, the rapid state-machine rewrite of the property test, the migration work and its proto field, and the mutation-test tool.Its rapid-import discovery for
PROPERTY_TEST_DIRSis left out as well. It works in #71 because that PR leaves property packages in the unit tier too; this tier subtracts them, so discovery by import would pull an entire production package's unit tests out of the retrying tier the moment someone added a rapid import.How did you test it?
The behaviours this PR exists to produce were verified directly rather than assumed.
Retries are actually disabled. Injected a deliberately failing property and ran the real target with
MAX_TEST_ATTEMPTS=3in the environment.test-runnerreceived--max-attempts=1, loggedstarting test attempt #1, thenexiting with failure after running 1 attempt(s), and produced exactly one JUnit file.Fail file staleness. The rapid v1.3.0 experiment described above.
Harness failures shrink. Failed through the driver's own
Tinside a property, and compared against an identical property with no driver. Both shrink to the same minimal draw (401 of 1..1000) across four consecutive runs, so the driver does not impede shrinking.Tier membership.
PROPERTY_TEST_DIRSauto-derives; a throwawayservice/matching/matchingproptestwas enrolled and excluded from the unit tier while./service/matching/stayed in it; aproptestdirectory with no test file was correctly not enrolled.Suppression removal. 300k generated cases with the suppression gone (200k in one run plus five independent seeds at 20k), and the un-skipped test green three times under
-race -shuffle on.Suites and tooling.
make build-testspasses repo-wide,./chasm/...passes under-race -shuffle on,make lint-code-fastreports 0 issues, and actionlint, yamlfmt,gofmtandmake fmt-importsare clean. TheProperty testjob was run end to end underact.Potential risks
The
Property testjob has never run on a hosted runner. Underactin an arm64 container,make property-test-coveragepassed with cgo and the race detector andpost-test-reportingconsumed its JUnit; only the artifact and Codecov uploads failed, which act cannot serve. That does not exercise the hosted runner image or the cache actions. Because the job is intest-status, a failure specific to hosted CI would block merges repo-wide rather than only this PR. The gate is a single entry intest-status'sneedslist and is trivial to revert.The naming convention is load-bearing. A property test in a directory not ending in
proptestruns in the unit tier with retries and its counterexamples are discarded. Documented, but a convention rather than an enforced check.chasmtest.TBwidens a shared signature.NewEngine,Engine.t,InvariantCheckand the scheduler's three invariant helpers now take an interface used by all CHASM libraries. Every caller passes*testing.Tand the repository compiles, but the blast radius is wider than this package.schedulertestreturns to the unit tier, so its 12 unit tests are subject to retries again. Correct for deterministic tests, but a change frommain.The gating tier uses a random seed. A genuinely flaky property would now block merges rather than being retried away. That is the intended trade; the nightly soak is non-gating so broader exploration cannot.
🤖 Generated with Claude Code