Skip to content

feat: add temporalio-streams with the Stream runtime and shared Store types - #1817

Open
moetemp wants to merge 12 commits into
moe/streams-crate-protosfrom
moe/streams-crate-types
Open

moetemp wants to merge 12 commits into
moe/streams-crate-protosfrom
moe/streams-crate-types

Conversation

@moetemp

@moetemp moetemp commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

This PR adds the experimental temporalio-streams crate. It has the Stream runtime that Core and the SDK bridges call, and the definitions that every Stream Store shares. Core's Worker doesn't use it, and #1801 adds that.

  • StreamStore has the five operations of the Stream service and the limits of one batch. A bridge implements it for a Store in its own language.
  • StreamRuntime checks each request, calls the Store, and checks the answer, with the same rules for every Store.
  • StreamError and Reason give each failed call one gRPC code and one StreamErrorDetails value, so every SDK raises the same error for the same failure.
  • epoch::epoch_key encodes a Producer's epoch as 40 bytes that compare in order. A Store that fences does it by that order.
  • record has the stored form of a Record, and BatchLimits for the size limits of one batch. ids has the size limits of Stream ids, Producer ids and offsets.

The crate defines no protos. It re-exports the Stream protos of #1733 as temporalio_streams::proto.

This PR includes the runtime of #1799, which is closed.

Why?

Core and each SDK bridge need one Store interface and one set of checks, so every Store refuses the same writes. A Store persists the epoch key bytes as its fence, and #1801 hashes them and the ProducerKind numbers into a digest that History keeps, so unit tests pin those bytes. The Stores are in temporal-community/sdk-streams#9.

Checklist

  1. How was this tested:

    • cargo test -p temporalio-streams. The runtime tests use a Store that returns the answers that a test sets.
  2. Any docs updates needed?

    • No.

@moetemp
moetemp marked this pull request as ready for review November 18, 2026 17:54
@moetemp
moetemp requested a review from a team as a code owner November 18, 2026 17:54
@moetemp
moetemp marked this pull request as draft November 18, 2026 18:02
@moetemp
moetemp added this pull request to stack #1818 November 18, 2026 18:05
@moetemp
moetemp force-pushed the moe/streams-crate-types branch from b543267 to dc75996 Compare November 18, 2026 18:10
@moetemp
moetemp marked this pull request as ready for review November 18, 2026 18:11
Comment thread crates/streams/README.md Outdated
Comment on lines +3 to +30
Experimental. What every Store of a Temporal Stream shares: the error of a failed call, the order
of a Producer's epochs, and the stored form of a Record.

## Errors

A failed call gives a `StreamError` with a `Reason`. `StreamError::code` gives its gRPC code and
`StreamError::details` its encoded `StreamErrorDetails`, so every SDK raises the same error for
the same failure.

## The order of epochs

`epoch::epoch_key` gives the epoch of a Producer as 40 bytes that compare in order. A Store fences
by this order, so every Store refuses the same writes. `testdata/epochs.json` has the shared
vectors. `examples/gen_vectors.rs` writes them from a second implementation that shares no code
with the crate. Run it with `cargo run -p temporalio-streams --example gen_vectors`.

## The stored form of a Record

`record::data_record` gives the bytes that a Store keeps for one Record. They have the Producer of
the Record and its body, and no offset. `record::BatchLimits` has the limits of one batch: its
Records, its bytes, and the bytes of one Record. The default is `MAX_BATCH_RECORDS`,
`MAX_BATCH_BYTES` and `MAX_RECORD_BYTES`, and a Store can set less. The sizes are of the stored
form.

## Protos

The crate defines no protos. `temporalio-protos` generates them, and `temporalio_streams::proto`
re-exports them, so a value needs no conversion between the crates.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very LLMy and just restating stuff you could find in the API docstrings. This can be reduced to just a basic summary

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6891b5a cuts it down to the short summary.

Comment thread crates/streams/src/lib.rs Outdated
Comment on lines +3 to +4
//! Experimental. What every Store of a Temporal Stream shares: the error of a failed call, the
//! order of a Producer's epochs, and the stored form of a Record.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//! Experimental. What every Store of a Temporal Stream shares: the error of a failed call, the
//! order of a Producer's epochs, and the stored form of a Record.
//! Experimental. Contains shared definitions for Temporal Stream Store implementations.

Generally speaking I'm pretty anal about us trying to keep our API docstrings sound not so LLMy. I'm not going to comment on every single one but it would be good to take a pass over all of them and fix them up so they make sense contextually and read well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took your wording as is. In 6891b5a I also rewrote the other docstrings that sounded stiff. I'll do another pass when the stack is close to merge.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I folded #1799 into this PR, so the crate doc mentions the runtime now too.

Comment thread crates/streams/examples/gen_vectors.rs Outdated
Int(i128),
Arr(Vec<Json>),
Obj(Vec<(&'static str, Json)>),
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't need a whole independent JSON implementation just for this. We aren't testing serde_json.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8bcf405 drops the custom writer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d5d4e11 drops the generator and the vectors too. Nothing outside the crate computes epoch keys, so a unit test in epoch.rs checks the key bytes.

The generator stays a second implementation through its own key math. A custom JSON writer
added nothing to that.
The docs read as plain descriptions now, and the README is a short summary like the other
crates. The removed tests restated constants or repeated what other tests and the vectors check.
@moetemp
moetemp requested a review from Sushisource November 18, 2026 19:44
Nothing outside the crate computes epoch keys. The Redis Store calls `epoch_key` and only compares
the bytes, so the generator and the JSON vectors had no reader.
The docs read as plain descriptions now. The removed tests repeated other tests or only checked
that values pass through to the Store.
The Redis Store keeps the kind number in its Producer keys, so a changed number breaks its data.
@moetemp moetemp changed the title feat: add temporalio-streams with the types that every Stream Store shares feat: add temporalio-streams with the Stream runtime and shared Store types Nov 18, 2026
@moetemp
moetemp removed this pull request from stack #1818 November 18, 2026 20:21
The comments didn't say that each read sets its own wait and page size, so the two
constants read as fixed settings.
A Store doesn't have to refuse a lower epoch of a Producer.
Each Record has what a reader needs to tell the attempts apart.
@moetemp
moetemp added this pull request to stack #1820 November 19, 2026 05:46
@moetemp
moetemp removed this pull request from stack #1820 November 19, 2026 07:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants