Repository navigation
Strengthen CHASM scheduler lifecycle verification and fix progress bugs - #71
Draft
chaptersix wants to merge 22 commits into
Draft
chaptersix wants to merge 22 commits into
chaptersix wants to merge 22 commits into
Conversation
Squashed from temporalio#13042 Source head: 33ffb9e Original commits: 40c3ec5 chasm/chasmtest: expose node, backend and transaction close for test drivers de4543e schedulertest: scheduler lifecycle driver harness 216ee9f schedulertest: fix gofmt and importas lint violations 19b8e23 schedulertest: scheduler invariants and deterministic tables 40f1bfa schedulertest: add buffer-bound and action-budget invariants fa8215d schedulertest: rapid property test for scheduler invariants 6984f62 schedulertest: add default case to property-test op switch dc098f1 schedulertest: port driver to current chasmtest engine 74c592c scheduler: stamp InvokerExecuteTask start time from an injectable clock a801336 schedulertest: add a start ledger invariant 6495d8a schedulertest: property test lagging side effects and schedules that end 33ffb9e schedulertest: apply go fix modernization
Separate external acceptance, RPC responses, callbacks, and local result recording. Check accepted-start accounting, logical task validity, idle routes, retry identity, and occurrence accounting; retain deterministic regressions and mutation checks. Preserve pending ALLOW_ALL starts when callbacks precede recording, release deferred work after selected starts are discarded, and admit backfills when positive shared capacity divides below one. Build on temporalio#13042. Reuse and reference #35, #41, #42, #43, #44, #45, #51, #52, #53, #55, #56, #57, #59, #62, and #70. Keep unresolved migration counterexamples opt-in and document the remaining reservation and persistence gaps.
Require TEMPORAL_RUN_MIGRATION_COUNTEREXAMPLES=1 exactly in every gated test and share one skip message. Fail fast when a controlled start RPC is left unconsumed or answered twice instead of blocking until the test timeout. Release the held create in the delete race test on early failure. Assert the rollback conversion output rather than its input fixture, echo the request ID actually sent in the rollback retry mock, and drop misleading seed labels, a debug log, and wall-clock fixtures. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The unit-test job retries failures, and each retry draws a new Rapid seed, so a rare property violation can be retried away. Add a property-test target that runs more generated cases with no retries, and a CI job that uploads the minimized failing cases. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
V1 to CHASM migration dropped the V1 backfill cursor, so the backfiller re-included the already-consumed start point. Migrated backfillers now carry the V1 StartTime as LastProcessedTime, and processBackfill treats a non-nil LastProcessedTime (including the Unix epoch) as recorded progress instead of treating a zero timestamp as unset. Migrated backfillers also got an immediate task before SchedulerCallbacksTask reconciled imported running workflows, so they could buffer starts against stale state. When callbacks are needed, the backfillers now start without a task and SchedulerCallbacksTask schedules them after reconciliation. CHASM to V1 rollback only overrode StartTime when Attempt > 0, so a fresh [T,T] backfill lost its inclusive boundary on V1's exclusive cursor. Rollback now uses the recorded cursor if present, else shifts the range start back 1ms. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Rolling a CHASM schedule back to V1 silently dropped buffered starts that were ready or retrying (Attempt > 0, no run recorded) and pending cancel/terminate requests, because V1 InternalState has no place for them. Add Invoker.canMigrateToWorkflow and check it in MigrateToWorkflow and in the migrate task. Both return the retryable ErrMigrationActionsPending (Unavailable, like ErrMigrationPending) until the invoker drains, after which migration succeeds. No persisted state or proto changes. The schedulertest ledger test now asserts refusal then success after draining; the migration-untransferable invariant is kept as a defence. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The rollback migrate task sent a fresh random request ID on every attempt and treated any WorkflowExecutionAlreadyStarted as success. An unrelated workflow holding the V1 schedule's workflow ID was accepted as the destination and the CHASM source was closed. Persist a request_id on WorkflowMigrationState (new field 3) and send it on every start attempt. On AlreadyStarted, accept only when StartRequestId matches; otherwise walk to the chain's first run (the V1 workflow may have continued as new) and require its create request ID to match. A foreign chain or a missing first run returns an error and the source stays open. An Unavailable before the destination commits stays retryable. Compatibility: MigrateToWorkflow generates an ID when none is supplied, as does the admin handler. A migration persisted before this field has an empty ID; the task assigns and persists one before its first start attempt and never accepts AlreadyStarted on it without chain verification. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
processBuffer computed isRunning only from starts with a RunId, so a start selected for execution (Attempt > 0) but not yet recorded did not block a second ProcessBuffer pass from selecting another start under single-action overlap policies. Count tracked, uncompleted starts with Attempt > 0 as running. Because a dropped selected start now defers later starts without producing a completion, and a recorded RunId makes it cancelable, re-enable deferred starts in recordExecuteResult when a start is dropped or newly started. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Record the production fixes and their default-run regressions in the coverage document, along with the dedicated property-test job. Name the four still-gated migration cases with one TestMigrationCounterexample prefix and state why each needs a handoff protocol. Drop the gate helper that no longer has callers in the scheduler package. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add chasmtest.TB, the subset of testing.TB that the test engine, gomock and testlogger need, which both *testing.T and *rapid.T satisfy. The engine, InvariantCheck and the schedulertest driver take it, and the property tests build their driver from the *rapid.T so harness failures are shrunk and recorded in rapid's failfile. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add chasmtest.TaskRunner, which steps an execution through its own pure and side-effect tasks with a controllable clock, with hooks for component-specific behaviour. The scheduler driver now delegates to it and keeps only scheduler operations, snapshots and invariants. The next pure-task time is read from the tree's pure-task metadata instead of the mock backend, so Engine.NodeForRef, BackendForRef and CloseTransaction are removed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ckages property-test now discovers every package whose tests import pgregory.net/rapid instead of hardcoding one directory, since -rapid.checks is undefined in other test binaries. The mutation check moves from Python to cmd/tools/mutationtest, runs via make mutation-test, and fails if any edit anchor does not match exactly once. The committed rapid failfile is removed (the same bug is pinned by TestLedger_FinalStartPendingAtIdle) and testdata/rapid is ignored. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…by environment The four counterexamples are known failures that need an unbuilt handoff protocol. An unconditional t.Skip naming the missing piece replaces the TEMPORAL_RUN_MIGRATION_COUNTEREXAMPLES gate, and the coverage plan now describes mutation-test and the uncommitted failfile. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Remove the always-true IdleRouteChecked flag and AcceptedStarts nil guard, share one violation accumulator across the checks, and report every violation from CheckInvariantsHook instead of only the first. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
TestSchedulerProperties now runs as a rapid state machine over step, pure, sideEffect, complete, pause, unpause, trigger, backfill, update, failed starts and lost start responses, with small drawn buffer and action limits and the CANCEL_OTHER and TERMINATE_OTHER overlap policies. Harness fixes the wider generator needed: the retry policy is now free of random jitter so rapid can replay and shrink, the task runner no longer jumps the clock over work its BeforeAdvance hook schedules earlier, and the start ledger accepts a start replaced by a CANCEL_OTHER or TERMINATE_OTHER start. Two production bugs it found are recorded as skipped reproducers. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…art phases InvokerExecuteTask passed its per-task context by value, so each phase spent MaxActionsPerExecution from its own copy and one execution could take up to three times the budget. Pass it by pointer. The widened scheduler property found this. Its reproducer is now unskipped and the generator draws every overlap policy with a small budget again. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Stop drawing MaxBufferSize, the generator reserve and MaxActionsPerExecution; only idle time is varied. Drop the reproducer for backfill overfilling a MaxBufferSize below 20, which only occurs at a limit nobody configures, and rewrite the action-budget regression to fail at the default budget. Document the harness structure. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e harness schedulertest is now the one home of the default schedule and config, the spec processor, the library wiring and a SchedulerEngine wrapper with typed read and update helpers. helper_test.go keeps its existing names as thin wrappers over them, and the Driver builds its engine through NewEngine. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The stuck and unbacked-IdleCloseTime checks now live in schedulertest as StateInvariant. The engine asserts it after every committed transition for both the unit tests and the Driver, and testEnv uses it through AllowStuck. The idle check now runs armed idle tasks through the handler's Validate, which subsumes the Driver's idle-route rule, so that rule and validIdleTask go. The Driver keeps its no-stuck rule: it ignores the framework visibility task and counts only valid side-effect tasks, which the shared count cannot. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Driver now embeds SchedulerEngine for the engine, registry, clock and typed read and update helpers, and keeps only what is lifecycle specific. The fake frontend and history services move to fakes.go, Snapshot to snapshot.go, and Patch, Update, CompleteWorkflow, MigrateToWorkflow and DeliverCompletion go through the shared update helper. Public method names are unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
4 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
Add a scheduler lifecycle driver and extend it to test external acceptance, RPC response delivery, callbacks, and local result recording as separate events. Check accepted-start identity and exactly-once accounting, valid logical tasks and idle deadlines, retry identity, occurrence accounting, and bounded backfill progress.
Production fixes, each with a default-run regression (these will be split into their own PRs later):
SchedulerCallbacksTaskreconciles imported running state.MigrateToWorkflowand the migrate task return the retryableErrMigrationActionsPendingwhile a selected start is unrecorded or cancel/terminate requests are queued.WorkflowMigrationState.request_id(new field 3) is persisted and sent on every start attempt. An already-started destination is accepted only if its start request, or the create request of its chain's first run, matches. A foreign chain or missing first run leaves the source open.InvokerExecuteTaskpassed its per-task context by value, so the terminate, cancel and start phases each spentMaxActionsPerExecutionseparately and one execution could take up to three times the budget. It is now passed by pointer. Found by the widened property test.Also adds a
make property-testtarget and aProperty testCI job. The unit-test job retries failures and each retry draws a new Rapid seed, so a rare violation could be retried away; the dedicated job runs 5,000 cases with no retries and uploads minimized failfiles.The harness is now reusable by other CHASM components. The stepping loop lives in
chasmtest.TaskRunner;schedulertestnow holds the single copy of the scheduler's test fixtures, engine wrapper and stuck/idle-deadline invariants, shared by the existing unit tests (helper_test.godelegates to it;invariant_test.gois gone) and the lifecycle driver, which adds only the fake services, operations and ledger snapshot. The engine, runner and driver takechasmtest.TB, which*rapid.Tsatisfies, so harness failures shrink.TestSchedulerPropertiesruns as a Rapid state machine and now covers backfill, update, failed starts, lost start responses, and the CANCEL_OTHER and TERMINATE_OTHER policies.make property-testdiscovers the packages that import Rapid, and the mutation check is a Go tool behindmake mutation-test.David Porter's work from temporalio#13042 at
33ffb9eis consolidated intoaa47350a3, preserving authorship and the exact source tree. Everything after it is ours.Why?
The original driver could not deliver a callback before local start recording, treated an RPC request as sufficient evidence of a start, and suppressed a broad class of closed/deferred states. These gaps could conceal real accounting and progress failures.
Reuses regression evidence and fixes from #35, #41, #51/#52, #42/#43, #53/#55, #59/#62, #44/#45, #47, #56, #57 and #70. The coverage document maps each reference to local tests and records assumptions.
How did you test it?
Passed locally with Go 1.27.3 and
CGO_ENABLED=0:Potential risks and incomplete work
DeleteSchedulecommits afterwards. All need a handoff protocol that fences source admission and identifies the migration at the destination. This command is expected to fail. Each is an unconditionalt.Skipnaming the missing handoff protocol; remove the skip to see the assertion fail.request_idexisted is assigned one before its next attempt. If an older binary had already created its destination under a random ID, ownership cannot be proven; the task errors, the source stays open and an operator must intervene.Property testjob was run locally withactin an arm64 container and passed (5,000 cases under the race detector in about 4 minutes, against a 14-minute test timeout). That does not exercise the hosted runner image, the cache actions, or thetest-statusgate.buffer-boundcannot fire there.Notes for reviewers
git diff aa47350a3..HEADisolates our additions. Each production fix is its own commit.Reproduction details and harness constraints
The new catchup-expiry regression uses a fixed clock at 2020-01-01 UTC, a 10-second interval ending after 31 seconds, BUFFER_ALL, a 60-second catchup window, a 10-minute idle timeout, and a 5-minute workflow-duration bound. Its event sequence is
step, step, step, pure.Rapid failfiles are not committed (
testdata/rapid/is ignored); the CI job uploads them as artifacts on failure.Controlled tests use
BeginSideEffect,NextStart, externalAccept,Respond,NextRecord,Record, andFinishSideEffect. Mutate/read the engine during an in-flight handler only when all controlled RPCs are blocked or the handler is blocked at the recording gate. These helpers deliberately serialize engine access; they do not establish arbitrary concurrent engine safety. A recording gate invokes the real handler's component-update closure, not a durable storage commit.Callback tests cover duplicate delivery before the RPC response, after the response but before recording, and after recording, for ALLOW_ALL, SKIP, and BUFFER_ALL. Ignoring an early ALLOW_ALL callback preserves the pending start until result recording; ALLOW_ALL does not need that callback to release an overlap slot. Callback omission from #70 is intentionally not imported.