Skip to content

feat: let Core read a Stream for Workflow code - #1819

Draft
moetemp wants to merge 2 commits into
moe/streams-directfrom
moe/streams-direct-read
Draft

moetemp wants to merge 2 commits into
moe/streams-directfrom
moe/streams-direct-read

Conversation

@moetemp

@moetemp moetemp commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

What was changed

This PR lets Core read a Stream itself, for Workflow code. Workflow code gets each page of Records in an activation, as it arrives.

It's a prototype, created for the purpose of getting your feedback on the design. Its base is #1801, which has the append and the close. The Python SDK's side is temporalio/sdk-python#2132.

  • Commands and jobs. StartStreamRead and CloseStreamRead are commands for Core, not for the server. ResolveStreamRead gives each page and the end of a read.
  • A read keeps its Workflow Task open, as a Local Activity does, with the same heartbeat. When only reads are open and the task has a command for the server, Core completes the task immediately.
  • Pages. A read wakes Workflow code as soon as it has Records and no activation is pending. A page has at most read_page_records, 256 by default. read_min_gap is an opt-in least time between two batches, for fewer activations on a slow Stream. A read holds at most read_ahead_bytes of Records that wait, 4 MiB by default.
  • The marker. A read has its range of offsets, a count, a digest, and each page's Record count and activation. With records_in_history the marker has the Records, about 1 MiB of them.
  • Replay. Core reads the Records of a read from the Store again and checks the count and the digest. Workflow code gets each page in the same activation as in the first run.
  • A Store outage. A read is tried again for up to 30 s before Workflow code gets the failure. A replay whose Store stays down fails only the task. A replay whose Records are gone fails the task too, or the Workflow with WorkflowErrorType::StreamReplay.
  • A graceful shutdown ends a task that only reads hold open and asks the server for a new one, so another Worker takes the run.
  • Metrics for reads and for the reads of a replay, with their failures.
  • The task duration warning leaves out the time in which only Stream calls hold a task open.

Why?

A library that iterates a Stream in Workflow code needs each Record while its Producer still runs. ADK is an example, since its own loop reads the model through an async generator. A Local Activity for each batch gives one marker for each batch. With this PR, an iteration costs one marker for each Workflow Task that it spans.

These points are open.

  • With no read_min_gap, a read wakes Workflow code for each Record on a Stream that isn't busy, which costs CPU. Is an opt-in gap the right setting for it?
  • While a read holds a Workflow Task open, a Signal or an Activity result reaches the Workflow at the heartbeat at the latest.

Checklist

  1. How was this tested: the tests in crates/sdk-core/src/core_tests/streams/reads.rs drive Core with a mock history and the test Store of feat: let Core append to and close a Stream for Workflow code #1801. The integration tests that need a Temporal server have no Stream case yet.
  2. Any docs updates needed? No. records_digest documents the encoding of the digest, and a unit test pins it.

Workflow code can read a Stream through Core. Core reads the Store and gives each page of Records
to Workflow code. The protos, the call code and Core's side are one commit, since none compiles
alone.
The tests drive a Worker on a mock history and a test Store. They cover the pages, the caps, the
idle time, the replay of Records, the shutdown and the metrics.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant