Skip to content

Split an update from a command-less first WFT on uncached workers - #1772

Closed
SBrouck wants to merge 1 commit into
temporalio:mainfrom
SBrouck:fix/live-update-after-empty-first-wft
Closed

SBrouck wants to merge 1 commit into
temporalio:mainfrom
SBrouck:fix/live-update-after-empty-first-wft

Conversation

@SBrouck

@SBrouck SBrouck commented Nov 9, 2026 •

Copy link
Copy Markdown

Fixes #1775.

What changed and why

An update request delivered live to a worker that doesn't have the workflow cached, right after a first workflow task that issued no commands, was dispatched in the same activation as InitializeWorkflow. find_end_index_of_next_wft_seq took the first task for a WFT heartbeat (WFTStarted → WFTCompleted → WFTScheduled, no command) and merged it into the update's task.

On replay the UpdateAccepted event ends the first sequence instead (see update_accepted_after_empty_wft), so lang receives [InitializeWorkflow] then [DoUpdate]. The Python SDK runs update handlers before the main workflow function within one activation (needed for update-with-start), so live it ran the handler before run, on replay after it, and every later task failed with a nondeterminism error.

HistoryUpdate now carries the sequencing event IDs of the WFT's pending update requests, set by WorkflowMachines from its protocol messages. When a heartbeat-looking WFT is the closest one started before such an ID, the sequence ends there, the same cut the accepted event produces on replay. Histories without pending update requests chunk exactly as before.

Validation

  • core_tests::updates::update_after_empty_first_task_on_uncached_worker: the live counterpart of replay_with_empty_first_task. Fails before the change (first activation is [InitializeWorkflow, DoUpdate], not replaying).
  • history_update unit tests: update_request_after_empty_wft, and update_request_after_heartbeats_ends_at_closest_wft, which checks that heartbeats before the update stay grouped with the first task (same cut as replay) and that the same history without a pending request still merges.
  • cargo test -p temporalio-sdk-core --lib (558 passed), cargo lint, cargo test-lint, cargo fmt --check, cargo changelog-check.
  • End to end: sdk-python main built against this branch. The repro from the issue now completes (3/3 runs) with identical activity IDs live and on replay; tests/worker/test_workflow.py -k "update or signal" (82 passed), test_update_with_start.py, test_replayer.py, test_update_core.py (55 passed).

Changelog: Core Fixed entry and changelog/fixed/unhurried-update-otter.md.

An update request delivered live to a worker without the workflow cached,
after a first workflow task that issued no commands, was dispatched in the
same activation as InitializeWorkflow: find_end_index_of_next_wft_seq took
the first task for a WFT heartbeat and merged it into the update's task.
On replay the UpdateAccepted event ends the first sequence instead, so
lang sees init and the update in separate activations. SDKs that run
update handlers before the main workflow function (Python) therefore ran
them in a different order live and on replay, which failed with a
nondeterminism error (temporalio/sdk-python#2080).

Pass the sequencing event IDs of the WFT's pending update requests to the
chunking logic, and end the sequence at the WFT started just before them,
the same cut the accepted event produces on replay.
@SBrouck
SBrouck requested a review from a team as a code owner November 9, 2026 10:04
@CLAassistant

CLAassistant commented Nov 9, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Sushisource

Copy link
Copy Markdown
Member

I am fairly sure this will get fixed by #1659 which is in progress. If you could check your problematic case against that PR and close this if it fixes it that would be great.

@SBrouck

SBrouck commented Nov 13, 2026

Copy link
Copy Markdown
Author

Thanks! I ran the repro from #1775 against #1659. I merged it onto ff30d403 locally so the sdk-python bridge builds.

  • With disable_workflow_task_chunking enabled, the live run applies the update after the first activation, the same as replay, and the workflow completes.
  • With the feature off, which is the default in Add opt-in workflow task chunking shutoff #1659, it still fails with the same NDE (Activity id of scheduled event … does not match activity id of activity command …).

So #1659 fixes this case once the feature is on. Is the plan to make the unchunked path the default, or to have the lang SDKs enable it? If so, I'm happy to close this in favour of #1659. If chunking stays the default for a while, this PR fixes the chunked path on its own, but I understand if you'd rather not change the chunker.

Copy link
Copy Markdown
Member

The plan is to make it the default after a few releases with it being non-default to allow for rollback. So yes, I think the move would be to close this and rely on that. Thanks!

@SBrouck

SBrouck commented Nov 16, 2026

Copy link
Copy Markdown
Author

Thanks, that makes sense. Closing in favour of #1659.

@SBrouck SBrouck closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update sent to an uncached Worker after a command-less first Workflow Task runs before the run method, then fails replay

3 participants