Skip to content

fix(core): keep SDK flags pending until recorded in history - #1815

Open
hikaru-212 wants to merge 4 commits into
temporalio:mainfrom
hikaru-212:investigate/rust-1797
Open

hikaru-212 wants to merge 4 commits into
temporalio:mainfrom
hikaru-212:investigate/rust-1797

Conversation

@hikaru-212

Copy link
Copy Markdown

What was changed

Fixes #1797.

  • Keep Core and language SDK flags pending until they are observed in persisted WorkflowTaskCompleted history, instead of treating them as recorded when assembling a completion request.
  • Add two deterministic Core regressions covering patch replay and language SDK flag selection after a discarded speculative Workflow Task completion.
  • Update the Rust SDK and Core changelogs.

Why?

When an Update is rejected during a workflow's start delay, the Server can discard the rejection-only speculative Workflow Task completion without persisting a WorkflowTaskCompleted event.

Previously, gather_for_wft_complete() marked the sent Core and language SDK flags as recorded immediately. If that completion was discarded, a cached workflow would omit those flags from subsequent completions even though they were absent from History.

In the reported case, a later patched() call produces a patch marker and a TemporalChangeVersion upsert. Cold replay does not find the UpsertSearchAttributeOnPatch flag in History, skips the upsert, and fails with TMPRL1100 at the UpsertWorkflowSearchAttributes event.

Language SDK flags can be lost through the same mechanism. A cached run may expose an unrecorded flag through available_internal_flags, causing a language SDK to stop reporting it. Replay can then select different flag-dependent behavior.

The fix removes the premature bookkeeping updates. Flags remain pending until add_from_complete() observes them in History, consistent with the existing SDK name/version bookkeeping.

In the observed first-task discard, the Server returned reset_history_event_id = 0, which does not distinguish the discard from an ordinary completion. Relying on History observation avoids requiring a separate rollback signal. On the normal path, previously completed tasks are reflected in History before subsequent activations, so already-recorded flags are not resent.

How was this tested?

The original failure was reproduced before the fix against Temporal Server 1.32.2 with Python SDK 1.36.0, including captured completion metadata and the replay error.

Two new Core regressions use a mock Server sequence based on the observed Workflow Tasks:

  • patch_after_discarded_first_task_rejection_replays verifies that the next persisted completion records UpsertSearchAttributeOnPatch, and that its History replays through the accepted Update and patch without nondeterminism.
  • lang_flag_selection_after_discarded_first_task_rejection_matches_replay models Python SDK 1.36.0's default-enabled language flag behavior and verifies that live execution and replay select the same flag-dependent behavior.

Restoring either premature flag-bookkeeping update causes the corresponding regression to fail.

Check Result
cargo build Passed
cargo test --features experimental 1,375 passed, 6 ignored
cargo integ-test 620 passed, 3 ignored
cargo lint / cargo test-lint Passed
cargo +nightly fmt --all --check Passed
cargo changelog-check Passed

The full default local integration suite passed. Six tests reported as passed returned early because optional external infrastructure was unavailable.

Scope and limitations

  • The regressions use mock Server sequences based on the observed behavior. The real Server reproduction was performed before the fix, not against the patched Core.
  • The language SDK regression checks flag metadata and selection consistency, not Python command equivalence. A separate Python runtime nondeterminism failure has not been demonstrated.
  • This prevents future missing-flag Histories; it does not repair Histories already persisted without the required flags.
  • Related chunking work (Add opt-in workflow task chunking shutoff #1659, fix(core): preserve workflow task boundaries with versioned chunking #1629, and Workflow Task Chunking: v2 #1509) introduces first-completion-only flags. Those flags require their own eligibility rules when integrating with History-authoritative bookkeeping, so they cannot be recorded on a later task after the first durable completion has already selected a different mode.

@hikaru-212
hikaru-212 requested a review from a team as a code owner November 18, 2026 04:24

type CapturedCompletion = (Vec<Command>, WorkflowTaskCompletedMetadata);

/// Builds the tasks Temporal Server 1.32.2 handed out in sdk-rust#1797 for a workflow started with

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Referring to the server version & exact PR is overly specific

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I've removed the specific Server version and issue reference. The comment now describes the general behavior modeled by the fixture.

Comment on lines +751 to +752
/// Mirrors how the Python SDK (1.36.0) decides whether to use a default-enabled lang flag on each
/// activation, returning whether the flag is used and whether it must be reported to Core.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't refer to specifically Python. This would apply to any lang.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. I've renamed the helper to default_enabled_lang_flag and updated the comments to describe the language-agnostic behavior.

Comment on lines +756 to +757
} else if act.is_replaying {
(false, false)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

always returning false when replaying doesn't seem right here? If the flag is in the activation, is should count as used, regardless of if we're replaying or not.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The helper already checks available_internal_flags before is_replaying, so an available flag is used even during replay. The false case only applies when the flag is absent during replay. I've clarified this in the comment.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Update rejection during initial Workflow Task delay loses SDK flags and breaks patched() replay

2 participants