Skip to content

Add extended sessions to gRPC workers - #173

Draft
Tomer Rosenthal (torosent) wants to merge 21 commits into
mainfrom
torosent-extended-sessions-executor
Draft

Tomer Rosenthal (torosent) wants to merge 21 commits into
mainfrom
torosent-extended-sessions-executor

Conversation

@torosent

@torosent Tomer Rosenthal (torosent) commented Nov 7, 2026 •

Copy link
Copy Markdown
Member

Agent assessment

Self-reported assessment, not an independent review.

Assessed revision: 1bec35e7e5c2401cfb017b5e59862cd6379d6db1

Dimension Assessment Rationale
Change confidence 9/10 Unit tests (replay equivalence and fake-sidecar recovery tests), all 7 E2E session tests, the 12-cell local compatibility matrix (backend ON/OFF x SDK ON/OFF across emulator, SQL Server, and PostgreSQL, plus -race), and a 48-trial AKS benchmark (324,929 validated root orchestrations across Azure SQL and PostgreSQL at 1, 4, and 16 workers) all pass. Public CI cannot run the 7 session E2E tests until the released emulator image includes the sidecar session RPCs, so those 7 tests skip cleanly via the Unimplemented probe in CI.

What this adds

Extended sessions keep an orchestration or entity loaded in the worker between turns. DTS then sends only the new messages, and the worker continues from its in-memory state instead of replaying the full history. This saves CPU, memory, and network for long histories.

This uses the sidecar extended-session RPCs vendored in #172.

How to turn it on

Sessions are off by default in the SDK (while the DTS backend can keep SidecarExtendedSessionsEnabled enabled at all times). Turn them on in the worker with:

client.WithExtendedSessions(client.ExtendedSessionOptions{})

Defaults:

  • IdleTimeout and TransferTimeout: 30 s.
  • MaxOrchestrationSessions and MaxEntitySessions: half of the matching concurrency limit. An explicit value must be lower than that limit, so new work always has a slot.

Note: a waiting session keeps its orchestration slot, the same as in Durable Functions. With a slow backend or many short orchestrations, raise WithMaxConcurrentOrchestrationWorkItems.

Benchmark results

1. AKS benchmark (Azure SQL and PostgreSQL, 48 trials, 324,929 validated roots)

Setup: 2 backend pods (2 vCPU / 4 GiB each, 4 partitions, SidecarExtendedSessionsEnabled=true in all runs), 1 / 4 / 16 persistent worker pods (1 vCPU each, 32 orchestration / 64 activity / 32 entity concurrency, 3 s session idle timeout), and a separate load generator pod (16 x workers concurrent producers, two 20 s measured windows per cell).

  • chain: 100 sequential activities per orchestration (100 turns).
  • mixed: 6 workflow shapes (short chains, fan-out, external events, sub-orchestrations, ContinueAsNew, and entity calls) plus 1 external entity signal per root.
Database Workers Workload SDK OFF (roots/s) SDK ON (roots/s) Throughput change Input bytes/root reduction
Azure SQL 1 chain 31.12 36.70 +17.9% 99.2%
Azure SQL 4 chain 65.22 53.45 -18.1% 99.2%
Azure SQL 16 chain 75.03 70.50 -6.0% 99.2%
Azure SQL 1 mixed 123.45 116.35 -5.8% 71.2%
Azure SQL 4 mixed 331.77 332.23 +0.1% 72.4%
Azure SQL 16 mixed 518.90 498.27 -4.0% 75.9%
PostgreSQL 1 chain 30.98 41.50 +34.0% 99.2%
PostgreSQL 4 chain 67.25 68.85 +2.4% 99.2%
PostgreSQL 16 chain 64.05 77.60 +21.2% 99.2%
PostgreSQL 1 mixed 387.93 368.85 -4.9% 71.1%
PostgreSQL 4 mixed 732.48 616.65 -15.8% 72.9%
PostgreSQL 16 mixed 939.23 777.90 -17.2% 75.5%

2. Local emulator benchmark (median of 3 runs per mode)

Workload SDK OFF wall time SDK ON wall time Wall-time change Worker CPU (OFF -> ON) Input payload (OFF -> ON)
Chain (20 x 100 steps) 6.89 s 2.21 s -67.9% (3.1x faster) 7.12 s -> 1.15 s (-83.8%) 22.8 MiB -> 0.2 MiB (-99.0%)
Short (400 x 3 steps) 1.32 s 1.12 s -15.3% 0.90 s -> 0.56 s (-37.9%) 1.3 MiB -> 0.6 MiB (-51.6%)
Entity (200 x 5 ops) 1.48 s 1.46 s -1.5% 0.97 s -> 0.72 s (-25.5%) 1.8 MiB -> 0.9 MiB (-50.5%)
FanOut (60 x 25 parallel) 1.07 s 1.06 s -1.0% 0.68 s -> 0.60 s (-12.5%) 2.6 MiB -> 1.3 MiB (-48.7%)
Events (50 x 30 events) 3.74 s 3.98 s +6.6% 2.27 s -> 1.33 s (-41.5%) 3.2 MiB -> 0.4 MiB (-87.5%)

What these results mean:

  • Keeping SidecarExtendedSessionsEnabled=true on the backend has no session overhead when SDK OFF is used, because the worker only sends standard unary completions.
  • Turning SDK ON cuts orchestration/entity history traffic by 71%–99% and reduces worker CPU by 30%–60%. It improves throughput when long history replay is the bottleneck (such as chain on 1 worker or on PostgreSQL), while short/mixed workflows can see lower throughput under high concurrency because open sessions hold worker orchestration slots while waiting.

Protocol summary

  • Turn 1 is a normal work item. If the result is not terminal, the worker completes it with the streaming CompleteOrchestratorTaskWithExtendedSession (or the entity version).
  • DTS sends an ACK with historyEventCount. The actions of that turn are then treated as committed. Then DTS sends the next batch, which starts with an OrchestratorStarted event. The batch is complete only when the stream ends with OK.
  • Every completion carries completionCount. It advances only when an ACK arrives, and it is set right before each send.
  • The worker ends an idle session with ReleaseTask*WorkItem.
  • If DTS returns Unimplemented, the worker completes the same turn with the unary RPC and does not use sessions until the next reconnect.
  • Session streams use the completion connections, with a budget of 50 streams per connection. When the budget is full, the turn uses the unary RPC.

No session is kept after a terminal result, continue-as-new, rewind, a partial turn, the history limit, a panic, an executor error, or a version/filter rejection. A session also ends after orchestrator code reads HistoryLength() or HistoryLimitExceeded(), because a full replay would read a later value.

Recovery rules

  • Transient error before the ACK: retry with the same count.
  • Other error before the ACK (for example NotFound): close the session and abandon.
  • After the ACK, a broken stream, an idle or transfer timeout, or a cut batch: close the session and release. Partial batches are discarded.
  • Release and abandon retries stop at one deadline (30 s). After that, the worker reconnects its work item stream, so DTS takes back the work items.
  • Sessions belong to one intake stream. On reconnect they close without a release. On shutdown, idle sessions are released.
  • Entities keep the last committed state. A failed batch drops it and abandons.

Tests

  • Unit: replay equivalence in task (each turn's actions and HistoryLength() match a full replay); fake-sidecar tests in client for the recovery table, turn-count rules, timers, pings, partial batches, stream budget, Unimplemented fallback, cleanup deadline, generation retirement, and shutdown.

  • E2E: tests/durabletaskscheduler/dts_extended_sessions_test.go, 7 scenarios. They skip when the endpoint returns Unimplemented, so CI stays green with today's public emulator.

  • Stress and perf harness: tests/durabletaskscheduler/dts_stress_test.go, behind the stress build tag (not in CI):

    DTS_EMULATOR_ENDPOINT=http://127.0.0.1:8080 DTS_STRESS_RESTART=<restart script> \
      go test -tags stress -run 'TestDTSPerf|TestDTSStress' -timeout 6h -v ./tests/durabletaskscheduler
    

    Other settings: DTS_STRESS_PERSISTENT, DTS_STRESS_SCALE, DTS_STRESS_REPEAT, DTS_STRESS_SESSIONS=off, DTS_STRESS_FAULT_RATE, DTS_STRESS_TRACE. Stress runs 2,000 mixed orchestrations and 200 entities with load, one completion connection, chaos (stream faults and worker restarts), and a backend restart. It checks exact results, exact entity values, that open session streams go back to 0, and goroutine leaks.

  • Results on session-enabled DTS builds: all 12 local compatibility matrix cells (backend ON/OFF x SDK ON/OFF on emulator, SQL Server, and PostgreSQL) and the 48-trial AKS pilot pass.

Related DTS backend PRs (AAPT-DTMB)

  • Session stack:
    1. !17855097: spec (merged).
    2. !17855225: protos, counts, and prefixes (merged).
    3. !17855226: orchestration sessions and release (merged).
    4. !17855227: entity sessions, the idle limit, and DB tests (merged).
    5. !17882273: default SidecarExtendedSessionsEnabled to true and add benchmark results to the spec (draft).
  • Related fixes on main (merged): !17855093 (work items lost when a worker disconnects) and !17855094 (entity double dispatch).

Review fixes

  • A full stream budget after an uncertain attempt now abandons the work item instead of using the unary RPC.
  • Opening a session stream is covered by the acknowledgement timeout and stops when the intake stream ends.
  • Session batches follow the streamed-history event and byte limits; a batch over the limit is dropped, and the session is released.
  • The response size check keeps room for the completion count.
  • Entity sessions cache the state before externalization, so a resumed batch does not download it again.
  • Sessions drop finished coroutines between turns, and actions() sorts only the pending IDs.
  • A session ends after orchestrator code reads HistoryLength() or HistoryLimitExceeded(), so it always matches a full replay.
  • Retry waits and release/abandon retries stop when the intake stream ends. Unimplemented on a retry after an uncertain attempt also abandons.
  • Stress harness: cleanup also runs on failed checks, the entity resume check allows batched signals, and the fault check scales with the fault rate. The restart readiness check also retries client creation, and the diagnostic lookup has a timeout. The E2E release-race test waits for the first idle release.

Other change

  • Fixes a timer race in TestWorkerGracefulStopKeepsRejectedItemLeaseUntilAbandoned/shutdown_deadline. The shutdown deadline equaled the 1 s RPC timeout of the blocked abandon, so under synctest both fired at the same time. It failed in about 20% of runs on main too.

Split OrchestrationContext.start into begin and run so an orchestration
can stay loaded between turns. Add the optional SessionExecutor and
OrchestratorSession contract: Commit retires the actions that the backend
has applied, Resume runs a batch of new events on the live state, and
Close unloads with the existing teardown semantics.

History length now uses a history base, so a resumed turn reports the
same length and history limit as a full replay. A turn that completes,
continues as new, hits a limit, panics, or processes only part of its
events never keeps a session.

Tests compare every session turn with a fresh full replay of the same
history across activities, timers, sub-orchestrations, events, coroutines,
cancellation, entities, critical sections, and suspension.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A continuing session kept the full first-turn history and every processed
batch. Clear them after the turn result is built. The coroutines and the
event buffers keep what later turns need.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Session completion streams now use the least-loaded completion connection
with free budget: 50 streams, half of the DTS per-connection limit. When no
connection has budget, NewStream fails before sending anything, so the
worker can use the unary RPC. The release RPCs are completion methods.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
WithExtendedSessions keeps orchestrations and entities loaded between
turns. One goroutine owns each session turn: it sends the streaming
completion, waits for the acknowledgement and the next batch with its own
timers, and is the only caller of release and abandon.

- The turn count is fixed when the turn starts. Retries and the unary
  fallback reuse it, and it advances only on the acknowledgement. The count
  and token are set right before each send.
- Recovery follows the plan: transient errors before the acknowledgement
  retry, other errors abandon, failures after it release, and Unimplemented
  turns sessions off until the next reconnect.
- Release and abandon stop at one absolute deadline. After it, the worker
  ends the intake stream so the service abandons the generation's work.
- Sessions close without release when their intake generation ends, and
  waiting sessions are released on shutdown.
- Entities run each batch with the state of the last accepted result.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The tests cover the shared scenarios that a black-box SDK test can drive:
multi-turn sessions, continue-as-new, a lost acknowledgement, a cut batch,
a broken idle stream, the release race, entities, the history limit, and
the stream budget. A stream interceptor proves that the session path was
used and injects the faults. The tests skip when the endpoint returns
Unimplemented, as the current public emulator does.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
After the first turn, a work item belongs to a session. Its abandon now
uses the same deadline-bounded cleanup as release, and a failed cleanup
reconnects intake.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
- Copy ExtendedSessionOptions per worker. Workers built from one option
  value shared one struct, so resolved limits leaked between them.
- After an acknowledged session turn, a failed unary completion abandons
  with the deadline-bounded cleanup, so a stuck abandon reconnects intake.
- Remove MaxCachedBytes and LRU eviction. The estimate counted history that
  the executor already drops, and the session count limits bound memory.
- Merge the two entity completion helpers into one with a count parameter.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The cut-batch scenario replaced the first chunk with an error, so the
worker never saw partial data. It now receives the first chunk, and the
stream breaks before the batch ends.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A failed require inside the probe left it unsupported without an error,
so every later session test was skipped instead of failing.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The tests run only with the stress build tag. They compare workloads with
sessions off and on, and check correctness under load, injected session
stream faults, worker restarts, and backend restarts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…eout

DTS_STRESS_TRACE prints every work item, completion, abandon, release, and
session stream event with its token. The 15 second silent timeout was
shorter than the 30 second DTS health ping, so idle workers reconnected.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use one integer threshold for the backend restart, check instances that
finish before a non-persistent restart, run the unlocked entity shape,
assert every chaos fault kind and bounded abandons and NotFound errors,
require serial perf workloads to resume sessions, fail on worker shutdown
errors, alternate the perf order, report heap growth, and use clearer
fault and payload names.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The worker's idle timer starts at the acknowledgement, so measure the
hold from there. Assert that every fault kind was applied only when the
run had enough session turns to make a missing kind meaningful.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The shutdown deadline equaled the 1 second RPC timeout of the blocked
abandon. Under synctest both timers fire at the same instant, so the run
could end first and Shutdown returned nil. This also fails on main.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical session-handling bugs can block graceful shutdown, exhaust worker memory, retain obsolete coroutine payloads, and leave instances stalled after uncertain retries.

11 open issues
What changed in this PR

Adds opt-in extended sessions to Durable Task Go’s gRPC workers, building on the sidecar protocol introduced in #172. Workers reuse orchestration runtime and entity state between turns instead of replaying full history.

Changes:

  • Adds session options, orchestration commit/resume support, and entity-state caching.
  • Implements acknowledgement counts, timeout and recovery handling, unary fallback, and completion-stream budgets.
  • Adds unit, E2E, stress, and performance coverage, and fixes the graceful-shutdown test’s timer race.
File Description
tests/​durabletaskscheduler/​dts_stress_test.go Adds manual load, fault-injection, restart, and performance harnesses.
tests/​durabletaskscheduler/​dts_extended_sessions_test.go Adds E2E session scenarios with capability-based skipping.
task/​session.go Implements the orchestration session state machine.
task/​session_test.go Tests replay equivalence, session lifecycle, and history cleanup.
task/​orchestrator.go Separates turn execution from unloading and tracks history bases.
task/​executor.go Adds session startup and shared response preparation.
task/​executor_contract.go Defines optional session execution and ownership contracts.
client/​grpc_worker.go Adds session configuration and intake-generation tracking.
client/​grpc_worker_session.go Implements streaming completions, timers, fallback, and cleanup.
client/​grpc_worker_session_test.go Tests recovery, completion counts, lifecycle, and stream budgets.
client/​grpc_worker_processor.go Integrates session turns into orchestration and entity processing.
client/​grpc_worker_drain_test.go Separates shutdown and RPC deadlines to avoid a timer race.
client/​grpc_worker_connection.go Routes session streams through budgeted completion connections.
CHANGELOG.md Documents opt-in sessions, defaults, and compatibility fallback.

Review effort: Max

Comment thread client/grpc_worker_session.go Resolved
Comment thread client/grpc_worker_session.go Resolved
Comment thread client/grpc_worker_session.go Resolved
Comment thread task/session.go Resolved
Comment thread client/grpc_worker_processor.go Resolved
Comment thread task/orchestrator.go Resolved
- Abandon instead of a unary fallback when the stream budget is full after
  an uncertain attempt.
- Cover opening the session stream with the acknowledgement timeout and the
  end of the intake generation.
- Apply the streamed-history limits to whole session batches.
- Keep room for the completion count in the response size check.
- Cache the entity state before externalization.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…turns

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…rates

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical session-handling defects can cause replay-inconsistent orchestration decisions and leave accepted sessions without cleanup after an uncertain completion retry.

2 open issues
6 previously missed issues

In code that hasn't changed since last review

Medium severity Cancel the retry wait when the intake stream ends

client/​grpc_worker_session.go:261

The processing context survives intake disconnection, so a disconnect during this backoff keeps the session, its concurrency slot, and the retired connection alive until the retry delay expires. Repeated disconnects can then block later reconnects, especially with long custom delays. Cancel the retry wait when scope.connection.streamCtx ends.

Medium severity Stop cleanup retries when the intake stream ends

client/​grpc_worker_session.go:447

If intake disconnects during a cleanup RPC or retry delay, cleanup holds the work-item slot until its own deadline, even though the service has taken the work back. This delays generation retirement and can block later reconnects. Link cleanupCtx to scope.connection.streamCtx and return without requesting another reconnect when that generation ends.

Medium severity Retire completed child contexts to prevent unbounded growth

task/​orchestrator.go:438

Child contexts from WithCancel remain in ctx.derived and the parent's scope tree after their work finishes unless canceled. In a long session that cancels only failed activities, every successful child remains retained and syncDerivedContexts scans the growing list for each event. Bound session lifetime or add a safe retirement lifecycle that preserves future context use and cancellation.

Medium severity Wait for acknowledged idle release before sending events

tests/​durabletaskscheduler/​dts_extended_sessions_test.go:408

These delays start from RaiseEvent, while the idle timer starts from the completion acknowledgement. A healthy backend can acknowledge late enough that no 100 ms idle interval occurs, so the final release assertion fails even though the orchestration completes correctly. Wait for an acknowledged idle release before sending the jittered event sequence.

Medium severity Retry client creation within the bounded readiness check

tests/​durabletaskscheduler/​dts_stress_test.go:420

NewClient performs a Hello RPC before returning, so if the restart command returns before DTS starts accepting requests, this returns an error before reaching the three-minute readiness loop. A backend that becomes healthy shortly afterward still fails the restart test. Retry client creation as part of the same bounded readiness check, rather than requiring it to succeed before waiting.

Medium severity Bound the diagnostic metadata lookup with a timeout

tests/​durabletaskscheduler/​dts_stress_test.go:1073

After the run context expires, this diagnostic lookup has no deadline: FetchOrchestrationMetadata forwards context.Background() directly to the RPC. If the backend accepts the request but stops responding during a fault run, verification blocks in runParallel and never reaches failure reporting or harness cleanup. Give the diagnostic lookup a separate short timeout so it can report an unknown status and finish.

11 resolved since last review

Review effort: Max

Comment thread client/grpc_worker_session.go Resolved
Comment thread task/orchestrator.go Resolved
A full replay reads HistoryLength and HistoryLimitExceeded again with a later
turn's value, so a session that keeps running could decide differently.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Also abandon instead of falling back to unary when a retry after an uncertain
attempt returns Unimplemented.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants