Repository navigation
Fix skipped mutable side effect replay recovery - #3164
Merged
Merged
Conversation
Quinn-With-Two-Ns
marked this pull request as ready for review
October 28, 2026 17:11
GregoryTravis
approved these changes
Nov 3, 2026
Resolve pending mutable side effect invocations at the original workflow-task boundary without changing legacy replay timing. Gate recovery behind a first-release-disabled SDK flag and register opt-in use at the mutableSideEffect call site. Add cache-eviction, callback-time, and nondeterminism detection replay coverage for issue #3163.
Capture recovery eligibility when each replay invocation is created. A flag first recorded on a later workflow task must not retime earlier unflagged continuations. Remove the task-start recovery hook and add a late-opt-in regression that also verifies new flagged calls complete at their own task time. Leave skip-count matching and the default-off rollout unchanged.
Quinn-With-Two-Ns
force-pushed
the
fix/mutable-side-effect-replay-3163
branch
from
November 10, 2026 17:56
e1cf1c5 to
44ad431
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 44ad431. Configure here.
Advance currentSkipCount before releasing a recovered skipped invocation. Otherwise a later marker with skipCount >= 2 is treated as another skip and its payload is lost. Leave the legacy history-event matching path unchanged. Cover two recovered skips within one task and across tasks, full replay, replay-to-live marker attributes, and cache-eviction result correctness on the test and real servers.
Use a completed update as a persisted barrier before evicting the workflow cache in the direct-query logging test. A constant query result did not prove the activity-failure-handling task was complete and could race cache eviction against an in-flight workflow task. Keep the single-log and replay-count assertions unchanged, without adding sleeps. Validate on Java 11 with the dev server, including 25 consecutive executions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

What changed?
MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY, disabled by default for its first release. Register opt-in usage at themutableSideEffectcall site and honor recorded flags during replay.Skip-count calculation and marker matching are unchanged.
Why?
Refs #3163.
A skipped mutable side effect has no corresponding marker event. During full replay, its internal placeholder can remain pending when there is no later command event to trigger cleanup, and an empty
RecordMarkercommand can reach the server.Unconditionally notifying the pending invocation at the next task start changes the workflow time observed by its continuation and can break replay. The added timing regression covers that case; flagged cleanup instead runs at the completed task's boundary, before the next task advances time or delivers signals.
Breaking changes?
None to public APIs. The new SDK flag remains disabled by default (
Enabled: pending) for rollout compatibility; tests opt in explicitly. Unflagged histories retain their existing replay behavior.Server PR
None required.
Validation
./gradlew --offline spotlessApplyadf0fc8) on the test and real servers. Baseline replay passed first, followed by 40 patched replay cases comparing exact callback values, order, and workflow times with opt-in off/on. The bulk corpus and generator are intentionally not included in this PR.Note
Medium Risk
Changes workflow replay state machine and timing for mutable side effects; flag is off by default but incorrect behavior could affect determinism when enabled.
Overview
Fixes replay for skipped
Workflow.mutableSideEffectcalls when a workflow task emits no commands (e.g. after cache eviction or worker restart), which could leave invalid pending marker state or spuriousRecordMarkercommands.Introduces opt-in SDK flag
MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY(disabled by default). When enabled, replay tracks eligible invocations in order and, at workflow task completed—after that task’s history events are applied—resolves still-pending skipped replays vianotifySkippedReplaying()so callbacks run at the original task’s workflow time instead of at the next task start.Flag use is recorded at the
mutableSideEffectcall site so later opt-in does not retime earlier unflagged invocations. CHANGELOG documents the fix; state-machine, integration, and JSON history replay tests cover commandless tasks, skip counts, callback timing, and nondeterminism cases. The query logging test gains an update barrier so cache eviction happens after activity failure is handled.Reviewed by Cursor Bugbot for commit 8ded721. Bugbot is set up for automated code reviews on this repo. Configure here.