Repository navigation
fix(core): keep SDK flags pending until recorded in history - #1815
hikaru-212 wants to merge 4 commits into
Conversation
|
|
||
| type CapturedCompletion = (Vec<Command>, WorkflowTaskCompletedMetadata); | ||
|
|
||
| /// Builds the tasks Temporal Server 1.32.2 handed out in sdk-rust#1797 for a workflow started with |
There was a problem hiding this comment.
Referring to the server version & exact PR is overly specific
There was a problem hiding this comment.
Agreed. I've removed the specific Server version and issue reference. The comment now describes the general behavior modeled by the fixture.
| /// Mirrors how the Python SDK (1.36.0) decides whether to use a default-enabled lang flag on each | ||
| /// activation, returning whether the flag is used and whether it must be reported to Core. |
There was a problem hiding this comment.
Shouldn't refer to specifically Python. This would apply to any lang.
There was a problem hiding this comment.
Agreed. I've renamed the helper to default_enabled_lang_flag and updated the comments to describe the language-agnostic behavior.
| } else if act.is_replaying { | ||
| (false, false) |
There was a problem hiding this comment.
always returning false when replaying doesn't seem right here? If the flag is in the activation, is should count as used, regardless of if we're replaying or not.
There was a problem hiding this comment.
The helper already checks available_internal_flags before is_replaying, so an available flag is used even during replay. The false case only applies when the flag is absent during replay. I've clarified this in the comment.
What was changed
Fixes #1797.
WorkflowTaskCompletedhistory, instead of treating them as recorded when assembling a completion request.Why?
When an Update is rejected during a workflow's start delay, the Server can discard the rejection-only speculative Workflow Task completion without persisting a
WorkflowTaskCompletedevent.Previously,
gather_for_wft_complete()marked the sent Core and language SDK flags as recorded immediately. If that completion was discarded, a cached workflow would omit those flags from subsequent completions even though they were absent from History.In the reported case, a later
patched()call produces a patch marker and aTemporalChangeVersionupsert. Cold replay does not find theUpsertSearchAttributeOnPatchflag in History, skips the upsert, and fails withTMPRL1100at theUpsertWorkflowSearchAttributesevent.Language SDK flags can be lost through the same mechanism. A cached run may expose an unrecorded flag through
available_internal_flags, causing a language SDK to stop reporting it. Replay can then select different flag-dependent behavior.The fix removes the premature bookkeeping updates. Flags remain pending until
add_from_complete()observes them in History, consistent with the existing SDK name/version bookkeeping.In the observed first-task discard, the Server returned
reset_history_event_id = 0, which does not distinguish the discard from an ordinary completion. Relying on History observation avoids requiring a separate rollback signal. On the normal path, previously completed tasks are reflected in History before subsequent activations, so already-recorded flags are not resent.How was this tested?
The original failure was reproduced before the fix against Temporal Server 1.32.2 with Python SDK 1.36.0, including captured completion metadata and the replay error.
Two new Core regressions use a mock Server sequence based on the observed Workflow Tasks:
patch_after_discarded_first_task_rejection_replaysverifies that the next persisted completion recordsUpsertSearchAttributeOnPatch, and that its History replays through the accepted Update and patch without nondeterminism.lang_flag_selection_after_discarded_first_task_rejection_matches_replaymodels Python SDK 1.36.0's default-enabled language flag behavior and verifies that live execution and replay select the same flag-dependent behavior.Restoring either premature flag-bookkeeping update causes the corresponding regression to fail.
cargo buildcargo test --features experimentalcargo integ-testcargo lint/cargo test-lintcargo +nightly fmt --all --checkcargo changelog-checkThe full default local integration suite passed. Six tests reported as passed returned early because optional external infrastructure was unavailable.
Scope and limitations