Skip to content

Strengthen CHASM scheduler lifecycle verification and fix progress bugs - #71

Draft
chaptersix wants to merge 22 commits into
mainfrom
test/chasm-scheduler-lifecycle
Draft

chaptersix wants to merge 22 commits into
mainfrom
test/chasm-scheduler-lifecycle

Conversation

@chaptersix

@chaptersix chaptersix commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

What changed?

Add a scheduler lifecycle driver and extend it to test external acceptance, RPC response delivery, callbacks, and local result recording as separate events. Check accepted-start identity and exactly-once accounting, valid logical tasks and idle deadlines, retry identity, occurrence accounting, and bounded backfill progress.

Production fixes, each with a default-run regression (these will be split into their own PRs later):

  • Early ALLOW_ALL callbacks lose start accounting. A callback arriving before the start response could remove the pending entry before its run identity and action count were recorded. The entry is preserved until recording.
  • Discarded selections strand deferred work. Paused completion or catchup expiry could discard a selected start while leaving the remaining BUFFER_ALL work deferred. Deferred work is reconsidered when a discard leaves no running or in-flight start.
  • Backfills stall despite positive capacity. Dividing capacity among many backfillers could round every allocation to zero. At least one is admitted when capacity exists.
  • Selected starts did not hold the overlap slot. A start selected for execution but not yet acknowledged was not counted as running, so a second BUFFER_ALL/BUFFER_ONE/SKIP start could be selected. It now reserves the slot, and deferred starts are reconsidered when a start records its run or is dropped, so a failed start cannot strand the buffer and TERMINATE_OTHER/CANCEL_OTHER still act.
  • Migrated backfills replayed the V1 cursor point. A migrated backfiller now carries the V1 exclusive cursor as its watermark; watermark presence (not value) marks recorded progress, so an epoch cursor is honoured. No schema change.
  • Migrated backfillers ran before callback reconciliation. They now get their first task only after SchedulerCallbacksTask reconciles imported running state.
  • Rollback of a fresh backfill lost its first point. The range start is shifted back 1ms onto V1's exclusive cursor.
  • Rollback dropped invoker execution phases. MigrateToWorkflow and the migrate task return the retryable ErrMigrationActionsPending while a selected start is unrecorded or cancel/terminate requests are queued.
  • Rollback accepted any existing workflow as its destination. WorkflowMigrationState.request_id (new field 3) is persisted and sent on every start attempt. An already-started destination is accepted only if its start request, or the create request of its chain's first run, matches. A foreign chain or missing first run leaves the source open.
  • Action budget was spent once per phase. InvokerExecuteTask passed its per-task context by value, so the terminate, cancel and start phases each spent MaxActionsPerExecution separately and one execution could take up to three times the budget. It is now passed by pointer. Found by the widened property test.

Also adds a make property-test target and a Property test CI job. The unit-test job retries failures and each retry draws a new Rapid seed, so a rare violation could be retried away; the dedicated job runs 5,000 cases with no retries and uploads minimized failfiles.

The harness is now reusable by other CHASM components. The stepping loop lives in chasmtest.TaskRunner; schedulertest now holds the single copy of the scheduler's test fixtures, engine wrapper and stuck/idle-deadline invariants, shared by the existing unit tests (helper_test.go delegates to it; invariant_test.go is gone) and the lifecycle driver, which adds only the fake services, operations and ledger snapshot. The engine, runner and driver take chasmtest.TB, which *rapid.T satisfies, so harness failures shrink. TestSchedulerProperties runs as a Rapid state machine and now covers backfill, update, failed starts, lost start responses, and the CANCEL_OTHER and TERMINATE_OTHER policies. make property-test discovers the packages that import Rapid, and the mutation check is a Go tool behind make mutation-test.

David Porter's work from temporalio#13042 at 33ffb9e is consolidated into aa47350a3, preserving authorship and the exact source tree. Everything after it is ours.

Why?

The original driver could not deliver a callback before local start recording, treated an RPC request as sufficient evidence of a start, and suppressed a broad class of closed/deferred states. These gaps could conceal real accounting and progress failures.

Reuses regression evidence and fixes from #35, #41, #51/#52, #42/#43, #53/#55, #59/#62, #44/#45, #47, #56, #57 and #70. The coverage document maps each reference to local tests and records assumptions.

How did you test it?

  • Covered by existing tests
  • Added new unit tests and deterministic regressions
  • 5,000 randomized lifecycle cases
  • Four deliberate mutations detected: remove the callback guard, deferred rearming, positive-capacity admission, or the selected-start reservation
  • Schedule functional tests against in-process SQLite
  • Race detector on the scheduler, frontend and worker scheduler packages and on the 5,000-case property run: no data races
  • Fast lint

Passed locally with Go 1.27.3 and CGO_ENABLED=0:

export CGO_ENABLED=0
go test -tags test_dep ./chasm/... ./service/frontend ./service/worker/scheduler -count=1
make property-test TEST_RACE_FLAG=off
CGO_ENABLED=1 go test -race -tags test_dep ./chasm/chasmtest/... ./chasm/lib/scheduler/... ./service/frontend ./service/worker/scheduler -count=1
CGO_ENABLED=1 make property-test
make mutation-test
go test -tags test_dep ./tests -run Schedule -count=1 -args -persistenceType=sql -persistenceDriver=sqlite
make lint-code-fast GOLANGCI_LINT_BASE_REV=03ecd1d228d2b8d9d433da55785afc9aeeead918 GOLANGCI_LINT_FIX=false

Potential risks and incomplete work

  • Four migration failures remain unresolved and skipped. A trigger accepted during the migration snapshot is dropped; forward migration accepts an unrelated existing CHASM schedule; the V1 source keeps executing after an ambiguous create response; a create held across DeleteSchedule commits afterwards. All need a handoff protocol that fences source admission and identifies the migration at the destination. This command is expected to fail. Each is an unconditional t.Skip naming the missing handoff protocol; remove the skip to see the assertion fail.
  • Rollback in flight across a deploy fails closed. A migration persisted before request_id existed is assigned one before its next attempt. If an older binary had already created its destination under a random ID, ownership cannot be proven; the task errors, the source stays open and an operator must intervene.
  • Ownership failures retry forever. A foreign chain or missing first run leaves the schedule paused for migration until an operator acts.
  • Backfill watermark compatibility. Any non-nil watermark now counts as progress; previously an explicit zero timestamp did not. No current writer produces one.
  • Reservation costs one extra ProcessBuffer pass per newly started workflow while a deferred start exists.
  • CI has not run on this PR in GitHub Actions. The Property test job was run locally with act in an arm64 container and passed (5,000 cases under the race detector in about 4 minutes, against a 14-minute test timeout). That does not exercise the hosted runner image, the cache actions, or the test-status gate.
  • No durability claim. Serialized restart, physical timer recovery, durable commit behavior, History admission fencing, and cross-cluster failover need separate integration coverage.
  • Harness limits. Rollback migration is not a property operation. Shrunk sequences can be long because start RPCs run on concurrent goroutines. All tweakables except idle time stay at their defaults in the property, so buffer-bound cannot fire there.

Notes for reviewers

  • git diff aa47350a3..HEAD isolates our additions. Each production fix is its own commit.
  • Keep subsequent commits signed.

Reproduction details and harness constraints

The new catchup-expiry regression uses a fixed clock at 2020-01-01 UTC, a 10-second interval ending after 31 seconds, BUFFER_ALL, a 60-second catchup window, a 10-minute idle timeout, and a 5-minute workflow-duration bound. Its event sequence is step, step, step, pure.

CGO_ENABLED=0 go test -tags test_dep ./chasm/lib/scheduler/schedulertest -run 'TestExpiredSelectionReleasesRemainingBuffer|TestLedger_DeferredStartsStrandedByPausedCompletion' -count=1

Rapid failfiles are not committed (testdata/rapid/ is ignored); the CI job uploads them as artifacts on failure.

Controlled tests use BeginSideEffect, NextStart, external Accept, Respond, NextRecord, Record, and FinishSideEffect. Mutate/read the engine during an in-flight handler only when all controlled RPCs are blocked or the handler is blocked at the recording gate. These helpers deliberately serialize engine access; they do not establish arbitrary concurrent engine safety. A recording gate invokes the real handler's component-update closure, not a durable storage commit.

Callback tests cover duplicate delivery before the RPC response, after the response but before recording, and after recording, for ALLOW_ALL, SKIP, and BUFFER_ALL. Ignoring an early ALLOW_ALL callback preserves the pending start until result recording; ALLOW_ALL does not need that callback to release an overlap slot. Callback omission from #70 is intentionally not imported.

davidporter-id-au and others added 22 commits November 11, 2026 14:52
Squashed from temporalio#13042
Source head: 33ffb9e

Original commits:
40c3ec5 chasm/chasmtest: expose node, backend and transaction close for test drivers
de4543e schedulertest: scheduler lifecycle driver harness
216ee9f schedulertest: fix gofmt and importas lint violations
19b8e23 schedulertest: scheduler invariants and deterministic tables
40f1bfa schedulertest: add buffer-bound and action-budget invariants
fa8215d schedulertest: rapid property test for scheduler invariants
6984f62 schedulertest: add default case to property-test op switch
dc098f1 schedulertest: port driver to current chasmtest engine
74c592c scheduler: stamp InvokerExecuteTask start time from an injectable clock
a801336 schedulertest: add a start ledger invariant
6495d8a schedulertest: property test lagging side effects and schedules that end
33ffb9e schedulertest: apply go fix modernization
Separate external acceptance, RPC responses, callbacks, and local result recording. Check accepted-start accounting, logical task validity, idle routes, retry identity, and occurrence accounting; retain deterministic regressions and mutation checks.

Preserve pending ALLOW_ALL starts when callbacks precede recording, release deferred work after selected starts are discarded, and admit backfills when positive shared capacity divides below one.

Build on temporalio#13042. Reuse and reference #35, #41, #42, #43, #44, #45, #51, #52, #53, #55, #56, #57, #59, #62, and #70. Keep unresolved migration counterexamples opt-in and document the remaining reservation and persistence gaps.
Require TEMPORAL_RUN_MIGRATION_COUNTEREXAMPLES=1 exactly in every gated test and share one skip message. Fail fast when a controlled start RPC is left unconsumed or answered twice instead of blocking until the test timeout. Release the held create in the delete race test on early failure.

Assert the rollback conversion output rather than its input fixture, echo the request ID actually sent in the rollback retry mock, and drop misleading seed labels, a debug log, and wall-clock fixtures.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The unit-test job retries failures, and each retry draws a new Rapid seed, so a rare property violation can be retried away. Add a property-test target that runs more generated cases with no retries, and a CI job that uploads the minimized failing cases.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
V1 to CHASM migration dropped the V1 backfill cursor, so the backfiller
re-included the already-consumed start point. Migrated backfillers now
carry the V1 StartTime as LastProcessedTime, and processBackfill treats
a non-nil LastProcessedTime (including the Unix epoch) as recorded
progress instead of treating a zero timestamp as unset.

Migrated backfillers also got an immediate task before
SchedulerCallbacksTask reconciled imported running workflows, so they
could buffer starts against stale state. When callbacks are needed, the
backfillers now start without a task and SchedulerCallbacksTask
schedules them after reconciliation.

CHASM to V1 rollback only overrode StartTime when Attempt > 0, so a
fresh [T,T] backfill lost its inclusive boundary on V1's exclusive
cursor. Rollback now uses the recorded cursor if present, else shifts
the range start back 1ms.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Rolling a CHASM schedule back to V1 silently dropped buffered starts that
were ready or retrying (Attempt > 0, no run recorded) and pending
cancel/terminate requests, because V1 InternalState has no place for them.

Add Invoker.canMigrateToWorkflow and check it in MigrateToWorkflow and in
the migrate task. Both return the retryable ErrMigrationActionsPending
(Unavailable, like ErrMigrationPending) until the invoker drains, after
which migration succeeds. No persisted state or proto changes.

The schedulertest ledger test now asserts refusal then success after
draining; the migration-untransferable invariant is kept as a defence.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The rollback migrate task sent a fresh random request ID on every attempt
and treated any WorkflowExecutionAlreadyStarted as success. An unrelated
workflow holding the V1 schedule's workflow ID was accepted as the
destination and the CHASM source was closed.

Persist a request_id on WorkflowMigrationState (new field 3) and send it
on every start attempt. On AlreadyStarted, accept only when StartRequestId
matches; otherwise walk to the chain's first run (the V1 workflow may have
continued as new) and require its create request ID to match. A foreign
chain or a missing first run returns an error and the source stays open.
An Unavailable before the destination commits stays retryable.

Compatibility: MigrateToWorkflow generates an ID when none is supplied,
as does the admin handler. A migration persisted before this field has an
empty ID; the task assigns and persists one before its first start attempt
and never accepts AlreadyStarted on it without chain verification.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
processBuffer computed isRunning only from starts with a RunId, so a
start selected for execution (Attempt > 0) but not yet recorded did not
block a second ProcessBuffer pass from selecting another start under
single-action overlap policies.

Count tracked, uncompleted starts with Attempt > 0 as running. Because a
dropped selected start now defers later starts without producing a
completion, and a recorded RunId makes it cancelable, re-enable deferred
starts in recordExecuteResult when a start is dropped or newly started.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Record the production fixes and their default-run regressions in the coverage document, along with the dedicated property-test job. Name the four still-gated migration cases with one TestMigrationCounterexample prefix and state why each needs a handoff protocol. Drop the gate helper that no longer has callers in the scheduler package.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add chasmtest.TB, the subset of testing.TB that the test engine, gomock
and testlogger need, which both *testing.T and *rapid.T satisfy. The
engine, InvariantCheck and the schedulertest driver take it, and the
property tests build their driver from the *rapid.T so harness failures
are shrunk and recorded in rapid's failfile.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add chasmtest.TaskRunner, which steps an execution through its own pure
and side-effect tasks with a controllable clock, with hooks for
component-specific behaviour. The scheduler driver now delegates to it
and keeps only scheduler operations, snapshots and invariants. The next
pure-task time is read from the tree's pure-task metadata instead of the
mock backend, so Engine.NodeForRef, BackendForRef and CloseTransaction
are removed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ckages

property-test now discovers every package whose tests import
pgregory.net/rapid instead of hardcoding one directory, since
-rapid.checks is undefined in other test binaries.

The mutation check moves from Python to cmd/tools/mutationtest, runs via
make mutation-test, and fails if any edit anchor does not match exactly
once. The committed rapid failfile is removed (the same bug is pinned by
TestLedger_FinalStartPendingAtIdle) and testdata/rapid is ignored.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…by environment

The four counterexamples are known failures that need an unbuilt handoff
protocol. An unconditional t.Skip naming the missing piece replaces the
TEMPORAL_RUN_MIGRATION_COUNTEREXAMPLES gate, and the coverage plan now
describes mutation-test and the uncommitted failfile.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Remove the always-true IdleRouteChecked flag and AcceptedStarts nil guard,
share one violation accumulator across the checks, and report every
violation from CheckInvariantsHook instead of only the first.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
TestSchedulerProperties now runs as a rapid state machine over step, pure,
sideEffect, complete, pause, unpause, trigger, backfill, update, failed
starts and lost start responses, with small drawn buffer and action limits
and the CANCEL_OTHER and TERMINATE_OTHER overlap policies.

Harness fixes the wider generator needed: the retry policy is now free of
random jitter so rapid can replay and shrink, the task runner no longer
jumps the clock over work its BeforeAdvance hook schedules earlier, and the
start ledger accepts a start replaced by a CANCEL_OTHER or TERMINATE_OTHER
start. Two production bugs it found are recorded as skipped reproducers.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…art phases

InvokerExecuteTask passed its per-task context by value, so each phase
spent MaxActionsPerExecution from its own copy and one execution could
take up to three times the budget. Pass it by pointer.

The widened scheduler property found this. Its reproducer is now
unskipped and the generator draws every overlap policy with a small
budget again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Stop drawing MaxBufferSize, the generator reserve and
MaxActionsPerExecution; only idle time is varied. Drop the reproducer
for backfill overfilling a MaxBufferSize below 20, which only occurs at
a limit nobody configures, and rewrite the action-budget regression to
fail at the default budget. Document the harness structure.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e harness

schedulertest is now the one home of the default schedule and config, the
spec processor, the library wiring and a SchedulerEngine wrapper with typed
read and update helpers. helper_test.go keeps its existing names as thin
wrappers over them, and the Driver builds its engine through NewEngine.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The stuck and unbacked-IdleCloseTime checks now live in schedulertest as
StateInvariant. The engine asserts it after every committed transition for
both the unit tests and the Driver, and testEnv uses it through AllowStuck.
The idle check now runs armed idle tasks through the handler's Validate, which
subsumes the Driver's idle-route rule, so that rule and validIdleTask go.

The Driver keeps its no-stuck rule: it ignores the framework visibility task
and counts only valid side-effect tasks, which the shared count cannot.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Driver now embeds SchedulerEngine for the engine, registry, clock and typed
read and update helpers, and keeps only what is lifecycle specific. The fake
frontend and history services move to fakes.go, Snapshot to snapshot.go, and
Patch, Update, CompleteWorkflow, MigrateToWorkflow and DeliverCompletion go
through the shared update helper. Public method names are unchanged.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants