Repository navigation
Conversation
An update request delivered live to a worker without the workflow cached, after a first workflow task that issued no commands, was dispatched in the same activation as InitializeWorkflow: find_end_index_of_next_wft_seq took the first task for a WFT heartbeat and merged it into the update's task. On replay the UpdateAccepted event ends the first sequence instead, so lang sees init and the update in separate activations. SDKs that run update handlers before the main workflow function (Python) therefore ran them in a different order live and on replay, which failed with a nondeterminism error (temporalio/sdk-python#2080). Pass the sequencing event IDs of the WFT's pending update requests to the chunking logic, and end the sequence at the WFT started just before them, the same cut the accepted event produces on replay.
|
|
|
I am fairly sure this will get fixed by #1659 which is in progress. If you could check your problematic case against that PR and close this if it fixes it that would be great. |
|
Thanks! I ran the repro from #1775 against #1659. I merged it onto
So #1659 fixes this case once the feature is on. Is the plan to make the unchunked path the default, or to have the lang SDKs enable it? If so, I'm happy to close this in favour of #1659. If chunking stays the default for a while, this PR fixes the chunked path on its own, but I understand if you'd rather not change the chunker. |
|
The plan is to make it the default after a few releases with it being non-default to allow for rollback. So yes, I think the move would be to close this and rely on that. Thanks! |
|
Thanks, that makes sense. Closing in favour of #1659. |
Fixes #1775.
What changed and why
An update request delivered live to a worker that doesn't have the workflow cached, right after a first workflow task that issued no commands, was dispatched in the same activation as
InitializeWorkflow.find_end_index_of_next_wft_seqtook the first task for a WFT heartbeat (WFTStarted → WFTCompleted → WFTScheduled, no command) and merged it into the update's task.On replay the
UpdateAcceptedevent ends the first sequence instead (seeupdate_accepted_after_empty_wft), so lang receives[InitializeWorkflow]then[DoUpdate]. The Python SDK runs update handlers before the main workflow function within one activation (needed for update-with-start), so live it ran the handler beforerun, on replay after it, and every later task failed with a nondeterminism error.HistoryUpdatenow carries the sequencing event IDs of the WFT's pending update requests, set byWorkflowMachinesfrom its protocol messages. When a heartbeat-looking WFT is the closest one started before such an ID, the sequence ends there, the same cut the accepted event produces on replay. Histories without pending update requests chunk exactly as before.Validation
core_tests::updates::update_after_empty_first_task_on_uncached_worker: the live counterpart ofreplay_with_empty_first_task. Fails before the change (first activation is[InitializeWorkflow, DoUpdate], not replaying).history_updateunit tests:update_request_after_empty_wft, andupdate_request_after_heartbeats_ends_at_closest_wft, which checks that heartbeats before the update stay grouped with the first task (same cut as replay) and that the same history without a pending request still merges.cargo test -p temporalio-sdk-core --lib(558 passed),cargo lint,cargo test-lint,cargo fmt --check,cargo changelog-check.mainbuilt against this branch. The repro from the issue now completes (3/3 runs) with identical activity IDs live and on replay;tests/worker/test_workflow.py -k "update or signal"(82 passed),test_update_with_start.py,test_replayer.py,test_update_core.py(55 passed).Changelog: Core
Fixedentry andchangelog/fixed/unhurried-update-otter.md.