Skip to content

WHN impl 2: store worker control queue when workflow task starts - #12992

Open
feiyang3cat wants to merge 2 commits into
fx/whn-implementation-inmem-1from
fx/whn-implementation-inmem-2
Open

feiyang3cat wants to merge 2 commits into
fx/whn-implementation-inmem-1from
fx/whn-implementation-inmem-2

Conversation

@feiyang3cat

@feiyang3cat feiyang3cat commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

What changed?

  • tracks the WHN-capable worker’s control queue for the current task, including normal and eager starts.

Why?

  • need to know which worker control queue to send the WHN

How did you test it?

  • built
  • run locally and tested manually
  • covered by existing tests
  • added new unit test(s) for metadata persistence and cleanup
  • added new functional test(s)

@feiyang3cat
feiyang3cat added this pull request to stack #12987 November 9, 2026 09:00
@feiyang3cat feiyang3cat changed the title Track started workflow task metadata for WHN WHN impl 2: Track started workflow task metadata for WHN Nov 9, 2026
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from f0d570c to 9785e58 Compare November 10, 2026 05:19
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from 9785e58 to 6b58851 Compare November 10, 2026 05:20
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from 6b58851 to cb28997 Compare November 10, 2026 05:39
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch 2 times, most recently from 64d0457 to 9b8183f Compare November 10, 2026 07:33
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from 9b8183f to f7612e9 Compare November 10, 2026 17:46
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from f7612e9 to 59205ae Compare November 13, 2026 05:15
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from 59205ae to 2dae008 Compare November 14, 2026 18:10
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from 2dae008 to e6ff69f Compare November 17, 2026 09:09
@feiyang3cat feiyang3cat changed the title WHN impl 2: Track started workflow task metadata for WHN WHN impl 2: store started workflow task metadata for WHN Nov 18, 2026
@feiyang3cat feiyang3cat changed the title WHN impl 2: store started workflow task metadata for WHN WHN impl 2: store started workflow task metadata when a WT starts for WHN Nov 18, 2026
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from 891953d to 8d45750 Compare November 18, 2026 07:09
@feiyang3cat
feiyang3cat requested a balanced review from Copilot November 18, 2026 09:35
@feiyang3cat
feiyang3cat marked this pull request as ready for review November 18, 2026 09:35
@feiyang3cat
feiyang3cat requested review from a team as code owners November 18, 2026 09:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Inline follow-up workflow tasks lose the worker control queue, leaving them without a heartbeat-notification destination.

1 open issue
3 files not reviewed
  • api/persistence/v1/executions.pb.go: Generated file
  • client/matching/client_gen.go: Generated file
  • service/history/interfaces/mutable_state_mock.go: Generated file
What changed in this PR

This PR records the control queue of workers that support workflow heartbeat notifications, providing a routing destination for the current workflow task.

Changes:

  • Preserve worker capabilities through matching poll forwarding.
  • Store control queues for normal and eager starts, with mutable-state lifecycle and replication support.
  • Add unit coverage and update the Temporal API dependency.
File Description
service/​matching/​pri_forwarder.go Forward workflow-task capabilities.
service/​matching/​matching_engine.go Capture capabilities in poll metadata.
service/​matching/​forwarder_test.go Verify worker metadata survives forwarding.
service/​history/​workflow/​workflow_task_state_machine.go Store, retrieve, and clear the control queue.
service/​history/​workflow/​mutable_state_impl.go Expose queue assignment and support replication.
service/​history/​workflow/​mutable_state_impl_test.go Cover queue lifecycle, speculative state, and replication.
service/​history/​workflow_heartbeat_notification_worker_control_queue_test.go Test normal and eager-start queue persistence.
service/​history/​interfaces/​workflow_task_info.go Add the task’s worker control queue.
service/​history/​interfaces/​mutable_state.go Declare the queue setter.
service/​history/​interfaces/​mutable_state_mock.go Add the generated setter mock.
service/​history/​api/​recordworkflowtaskstarted/​api.go Record the queue for capable polling workers.
service/​history/​api/​create_workflow_util.go Record the queue for capable eager-start workers.
proto/​internal/​temporal/​server/​api/​persistence/​v1/​executions.proto Define the persisted control-queue field.
go.sum Add API dependency checksums.
go.mod Update the Temporal API dependency.
client/​matching/​client_gen.go Preserve capabilities in matching requests.
api/​persistence/​v1/​executions.pb.go Generate persistence bindings for the new field.

Review effort: Balanced

Comment on lines +179 to +181
if req.PollRequest.GetCapabilities().GetSupportsWorkflowHeartbeatNotification() {
mutableState.SetWorkflowTaskWorkerControlQueue(req.PollRequest.GetWorkerControlTaskQueue())
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

▎ Good catch, fixed. Both inline-start paths in RespondWorkflowTaskCompleted (normal and speculative) now carry over currentWorkflowTask.WorkerControlQueue to the new task. That's safe for two reasons:
▎ - The inline task always goes back to the caller, which must hold the token of the currently started task, so the caller is the worker that registered the queue.
▎ - The stored queue was only set when that worker reported SupportsWorkflowHeartbeatNotification.
▎
▎ I didn't use RespondWorkflowTaskCompletedRequest.WorkerControlTaskQueue, because that request's capabilities have no WHN flag. The speculative path only sets it in memory, so it doesn't add a persistence write.
▎
▎ Added TestInlineWorkflowTaskKeepsWorkerControlQueue. It covers normal and speculative inline tasks, each with and without a queue.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Copilot did you check my changes?

@feiyang3cat
feiyang3cat requested a review from simvlad November 18, 2026 17:46
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch 2 times, most recently from 9f6d49b to 8573d52 Compare November 18, 2026 19:17
@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from 8573d52 to 56016b5 Compare November 19, 2026 04:33
}

func (ms *MutableStateImpl) SetWorkflowTaskWorkerControlQueue(queue string) {
ms.workflowTaskManager.SetWorkflowTaskWorkerControlQueue(queue)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only sets in the memory, and if set by speculative workflow task, it won't change all in-mem behavior

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The lifecycle subtests use fatal assertions bound to the parent test, so a failure can abort remaining cases instead of being reported against the individual subtest.

2 open issues
3 files not reviewed
  • api/persistence/v1/executions.pb.go: Generated file
  • client/matching/client_gen.go: Generated file
  • service/history/interfaces/mutable_state_mock.go: Generated file

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the worker control-queue propagation and persistence changes.

1 open issue
1 resolved since last review
3 files not reviewed
  • api/persistence/v1/executions.pb.go: Generated file
  • client/matching/client_gen.go: Generated file
  • service/history/interfaces/mutable_state_mock.go: Generated file

Review effort: Balanced

@feiyang3cat
feiyang3cat force-pushed the fx/whn-implementation-inmem-2 branch from 2a48b97 to fee10ef Compare November 19, 2026 05:38
When the worker that starts a workflow task supports workflow heartbeat
notifications, store its worker control task queue in ExecutionInfo as
workflow_task_worker_control_queue. It comes from PollWorkflowTaskQueue, or
from StartWorkflowExecution for an eager workflow task, and is cleared with
the other workflow task fields. Matching now keeps poll capabilities when
forwarding a workflow task poll.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the worker control-queue changes.

1 open issue
3 files not reviewed
  • api/persistence/v1/executions.pb.go: Generated file
  • client/matching/client_gen.go: Generated file
  • service/history/interfaces/mutable_state_mock.go: Generated file

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants