Skip to content

Fix skipped mutable side effect replay recovery - #3164

Merged
Quinn-With-Two-Ns merged 4 commits into
mainfrom
fix/mutable-side-effect-replay-3163
Nov 10, 2026
Merged

Quinn-With-Two-Ns merged 4 commits into
mainfrom
fix/mutable-side-effect-replay-3163

Conversation

@Quinn-With-Two-Ns

@Quinn-With-Two-Ns Quinn-With-Two-Ns commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What changed?

  • Track replaying mutable-side-effect invocations separately from the command queue and resolve skipped invocations after their workflow task's command events have been applied.
  • Reuse the existing cancellation/cached-result transition, preserving callback workflow time without sweeping unrelated commands.
  • Introduce MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY, disabled by default for its first release. Register opt-in usage at the mutableSideEffect call site and honor recorded flags during replay.
  • Add cache-eviction, callback-time, flag-registration, and negative nondeterminism-detection tests. Retain only the commandless-task and callback-time history fixtures.

Skip-count calculation and marker matching are unchanged.

Why?

Refs #3163.

A skipped mutable side effect has no corresponding marker event. During full replay, its internal placeholder can remain pending when there is no later command event to trigger cleanup, and an empty RecordMarker command can reach the server.

Unconditionally notifying the pending invocation at the next task start changes the workflow time observed by its continuation and can break replay. The added timing regression covers that case; flagged cleanup instead runs at the completed task's boundary, before the next task advances time or delivers signals.

Breaking changes?

None to public APIs. The new SDK flag remains disabled by default (Enabled: pending) for rollout compatibility; tests opt in explicitly. Unflagged histories retain their existing replay behavior.

Server PR

None required.

Validation

  • ./gradlew --offline spotlessApply
  • Retained state-machine and mutable-side-effect tests: 14 passed.
    ./gradlew --offline :temporal-sdk:test \
      --tests io.temporal.internal.statemachines.MutableSideEffectStateMachineTest \
      --tests io.temporal.workflow.MutableSideEffectReplayTest \
      --tests io.temporal.workflow.MutableSideEffectTest
  • Real-server configuration: 9 passed.
    ./gradlew --offline -PtestServer=dev-server :temporal-sdk:test \
      --tests io.temporal.workflow.MutableSideEffectReplayTest \
      --tests io.temporal.workflow.MutableSideEffectTest
  • One-off compatibility validation generated 20 histories with the unmodified baseline SDK (adf0fc8) on the test and real servers. Baseline replay passed first, followed by 40 patched replay cases comparing exact callback values, order, and workflow times with opt-in off/on. The bulk corpus and generator are intentionally not included in this PR.
  • Broader focused state-machine, side-effect, versioning, query, sticky-worker, and SDK-flag tests passed during development. The full repository suite has not been run.

Note

Medium Risk
Changes workflow replay state machine and timing for mutable side effects; flag is off by default but incorrect behavior could affect determinism when enabled.

Overview
Fixes replay for skipped Workflow.mutableSideEffect calls when a workflow task emits no commands (e.g. after cache eviction or worker restart), which could leave invalid pending marker state or spurious RecordMarker commands.

Introduces opt-in SDK flag MUTABLE_SIDE_EFFECT_REPLAY_RECOVERY (disabled by default). When enabled, replay tracks eligible invocations in order and, at workflow task completed—after that task’s history events are applied—resolves still-pending skipped replays via notifySkippedReplaying() so callbacks run at the original task’s workflow time instead of at the next task start.

Flag use is recorded at the mutableSideEffect call site so later opt-in does not retime earlier unflagged invocations. CHANGELOG documents the fix; state-machine, integration, and JSON history replay tests cover commandless tasks, skip counts, callback timing, and nondeterminism cases. The query logging test gains an update barrier so cache eviction happens after activity failure is handled.

Reviewed by Cursor Bugbot for commit 8ded721. Bugbot is set up for automated code reviews on this repo. Configure here.

@Quinn-With-Two-Ns
Quinn-With-Two-Ns marked this pull request as ready for review October 28, 2026 17:11
@Quinn-With-Two-Ns
Quinn-With-Two-Ns requested a review from a team as a code owner October 28, 2026 17:11

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Resolve pending mutable side effect invocations at the original workflow-task boundary without changing legacy replay timing. Gate recovery behind a first-release-disabled SDK flag and register opt-in use at the mutableSideEffect call site.

Add cache-eviction, callback-time, and nondeterminism detection replay coverage for issue #3163.
Capture recovery eligibility when each replay invocation is created. A flag first recorded on a later workflow task must not retime earlier unflagged continuations.

Remove the task-start recovery hook and add a late-opt-in regression that also verifies new flagged calls complete at their own task time. Leave skip-count matching and the default-off rollout unchanged.
@Quinn-With-Two-Ns
Quinn-With-Two-Ns force-pushed the fix/mutable-side-effect-replay-3163 branch from e1cf1c5 to 44ad431 Compare November 10, 2026 17:56

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 44ad431. Configure here.

Advance currentSkipCount before releasing a recovered skipped invocation. Otherwise a later marker with skipCount >= 2 is treated as another skip and its payload is lost. Leave the legacy history-event matching path unchanged.

Cover two recovered skips within one task and across tasks, full replay, replay-to-live marker attributes, and cache-eviction result correctness on the test and real servers.
Use a completed update as a persisted barrier before evicting the workflow cache in the direct-query logging test. A constant query result did not prove the activity-failure-handling task was complete and could race cache eviction against an in-flight workflow task.

Keep the single-log and replay-count assertions unchanged, without adding sleeps. Validate on Java 11 with the dev server, including 25 consecutive executions.
@Quinn-With-Two-Ns
Quinn-With-Two-Ns merged commit 11cc574 into main Nov 10, 2026
36 of 38 checks passed
@Quinn-With-Two-Ns
Quinn-With-Two-Ns deleted the fix/mutable-side-effect-replay-3163 branch November 10, 2026 23:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants