Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -226,8 +226,9 @@ jobs:
- run: uv tool install poethepoet
- run: uv sync --all-extras
- run: cargo install --locked nexgen --version 0.2.8 --features advanced --force
# The Stream package needs the read, read pages and read page methods of a poll, which only
# a nexgen fork has.
# The Stream package needs `@nexus.helper` with `sync=true`, which only a nexgen fork has.
# This commit of the fork also has the poll iterators of the Workflow target and the read,
# read pages and read page methods of a poll.
- run: cargo install --locked --git https://github.com/moetemp/nexgen --rev 07b565b6d4339dc8a3f67ce2e9e72ba4925d7f9c --features advanced --root "$RUNNER_TEMP/nexgen-stream" nexgen
- run: echo "NEX_GEN_STREAM_BIN=$RUNNER_TEMP/nexgen-stream/bin/nexgen" >> "$GITHUB_ENV"
- run: poe build-develop
Expand Down
2 changes: 1 addition & 1 deletion changelog/added/chattering-ledger-otter.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
Add the experimental Streams API. Client and Activity code create a Stream, append Records to it, read it and close it. Workflow code can't call a Stream in this version. A client takes its Stream Stores in `stream_config=StreamConfig(stores=[...])`, such as `RedisStreamStore` of the `temporalio-streams-redis` package, or `MemoryStreamStore` of the `temporalio-streams-memory` package in tests.
Add the experimental Streams API. Client and Activity code create a Stream, append Records to it, read it and close it. A client takes its Stream Stores in `stream_config=StreamConfig(stores=[...])`, such as `RedisStreamStore` of the `temporalio-streams-redis` package, or `MemoryStreamStore` of the `temporalio-streams-memory` package in tests.
Add `temporalio.contrib.streams.current_producer` and `as_producer`. An Activity can store its Producer in its heartbeat details and pass it to the appends of its retry. The retry's Records then continue the first attempt's epoch, so a reader does not take them as a replacement of the first attempt's Records.
A Workflow outbound interceptor sees a Stream operation only in its operation-specific method, such as `start_append_records`, and not in `start_system_nexus_operation`. An unknown Stream operation raises `ValueError`.
11 changes: 11 additions & 0 deletions changelog/added/patient-river-heron.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
Workflow code creates a Stream, appends Records to it, reads it, describes it and closes it, with `workflow.create_stream` and `workflow.get_stream`. A create and a describe run in the Store's Local Activity, with the options of `StreamConfig.workflow_call_options`. Core makes a read, an append and a close itself, with no Activity. It calls the Store as a task on the Worker's event loop, never on a Workflow thread. For Workflow code, a Store waits up to a read's `wait`, ends that wait on a cancel, gives offsets of 1 to 1024 bytes that compare in order, writes a resent batch once, and raises `StreamError` with a reason. A Store can refuse a lower epoch of a Producer with `STREAM_ERROR_REASON_FENCED`, and it doesn't have to. Core tries a call again for up to 30 s only on `STREAM_ERROR_REASON_STORAGE_UNAVAILABLE`. Any other exception reaches Workflow code on the first try, as a non-retryable `STREAM_ERROR_REASON_STORAGE_CORRUPT` with its type in the message. `Replayer` takes the `stream_config` of the Worker's client.
The Workflow `Stream` has `read`, `read_pages` and `read_page`, so Workflow code reads a Stream with the same calls as a client. Core reads the Store inside the Workflow Task. A page that Workflow code gets in one activation has at most 256 Records, and the rest come in the next activation with no wait. A loop that asks for its next page in the same activation keeps one read open, and a read that no code waits on closes before the activation completes, so it never holds a Workflow Task open. With a `wait`, a quiet Stream gives empty pages, so a loop on a Stream that can be quiet for long uses `read_pages` and leaves after an empty page. A call that waits on a quiet Stream holds its Workflow Task open, with one Workflow Task about every 8 s. The calls of one Workflow Task write one marker in History. `StreamConfig.workflow_read_replay_from` decides whether the marker holds the Records (`StreamReplaySource.HISTORY`) or only their offsets and a digest, which a replay reads from the Store again (`StreamReplaySource.STORE`, the default). With `HISTORY`, one Workflow Task's marker holds at most 1 MiB of Records, and the read goes on in the next task.
A Workflow's append runs in Core inside the Workflow Task, with no Activity. It writes as the Workflow, with its Workflow Task as the epoch. A `producer=` of another kind raises `StreamError` with the reason `STREAM_ERROR_REASON_INVALID`.
A Workflow's close runs in Core, with no Activity, and spans two Workflow Tasks. The task that calls `close` records the intent, Core closes the Stream after the server took that task, and the next task gives the result, so a failed task closes nothing. Await the close. A graceful shutdown of the Worker waits at most 10 s for a close that the code doesn't await, and a Worker that dies when the Workflow ends loses it. An append and a close of Workflow code always carry the Workflow Task as their epoch. An append is at-least-once. After a Worker crash or an eviction, the rerun of its task can write a second copy with the epoch of the new task.
Core records metrics for the Direct calls of a Stream, such as `temporal_stream_read_total`, `temporal_stream_read_records` and `temporal_stream_close_latency`, each with the attribute `stream_store`, and `temporal_stream_close_abandoned` for a close that a graceful shutdown gave up.
Add `StreamConfig.workflow_read_ahead_bytes`, the most bytes of one read's Records that wait in the Worker for Workflow code, 4 MiB by default.
Add `StreamConfig.workflow_read_page_records`, the most Records of one read that Workflow code gets in one activation. It is 256 by default and must be 1 to 1000.
Add `StreamConfig.workflow_read_min_gap`, the least time between two batches of one read. It has no gap by default and must be at most 1 s. A gap trades latency for fewer activations of Workflow code.
Add `temporalio.exceptions.StreamReplayError`. With it in `workflow_failure_exception_types`, a replay that can't read the Records of a read from the Store again fails the Workflow, and not only the Workflow Task.
Core tries a read, an append or a close of Workflow code again while its Store is unavailable, from 100 ms up to 2 s apart, for at most 30 s. Only then does Workflow code get the retryable `StreamError`. A replay tries its Store for 5 s. If the Store stays down, only the Workflow Task fails, with `StreamReplayUnavailable`, also with `StreamReplayError` in `workflow_failure_exception_types`.
In Workflow code, `temporalio.contrib.streams.current_producer` gives the Workflow's Producer. `BufferedAppender` is for client and Activity code, and it raises `RuntimeError` in Workflow code.
7 changes: 4 additions & 3 deletions scripts/gen_nexus_system_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,10 @@
workflow_init_path = base_dir / "temporalio" / "workflow" / "__init__.py"
NEX_GEN_VERSION = "0.2.8"
# The Stream package needs a newer nexgen, which rejects the other WITs. One pin is enough
# again once the api WITs and a nexgen release agree. The read, read pages and read page methods
# of a poll are only in commit 07b565b6 of the fork https://github.com/moetemp/nexgen. So
# NEX_GEN_STREAM_BIN points at a build of that commit.
# again once the api WITs and a nexgen release agree. The Stream WIT uses `@nexus.helper` with
# `sync=true` and `@nexus.support-type`, which only the fork https://github.com/moetemp/nexgen
# has. Its commit 07b565b6 also has the poll iterators of the Workflow target and the read, read
# pages and read page methods of a poll. So NEX_GEN_STREAM_BIN points at a build of that commit.
NEX_GEN_STREAM_VERSION = "0.2.12"
proto_files = [
gen_protos.api_proto_dir
Expand Down
8 changes: 5 additions & 3 deletions temporalio/api/stream/v1/message_pb2.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -167,8 +167,8 @@ global___ActivityEpoch = ActivityEpoch
@typing.final
class WorkflowProducer(google.protobuf.message.Message):
"""Experimental. Might change in the future.
A Workflow run Producer. On the server's Streams, a Workflow writes as this Producer in its
Workflow Task. On a Store, Workflow code writes through a Local Activity, so its writes carry a
A Workflow run Producer. Workflow code that writes in its Workflow Task writes as this
Producer, with that task as the epoch. A write that it makes through a Local Activity has a
Local Activity Producer.
"""

Expand All @@ -180,7 +180,9 @@ class WorkflowProducer(google.protobuf.message.Message):
"""Required. The run id of the Workflow run, not of its chain. Each run is its own Producer."""
@property
def epoch(self) -> global___WorkflowTaskEpoch:
"""The Workflow Task of the write."""
"""The Workflow Task of the write. Workflow code leaves it empty, and the SDK sets it before
the write goes to the Store.
"""

def __init__(
self,
Expand Down
82 changes: 81 additions & 1 deletion temporalio/bridge/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions temporalio/bridge/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,8 @@ temporalio-sdk-core = { version = "0.11.0", path = "./sdk-core/crates/sdk-core",
"envconfig",
"prometheus",
] }
# The Stream runtime, from the submodule.
temporalio-streams = { path = "./sdk-core/crates/streams" }
tokio = "1.26"
# Matches the sdk-core client crate's rustls stack; used to build the custom
# server certificate verifier for ClientTlsConfig.verification_server_name.
Expand Down
59 changes: 53 additions & 6 deletions temporalio/bridge/_visitor.py
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,38 @@ async def _visit_coresdk_workflow_activation_ResolveNexusOperation(
if o.HasField("result"):
await self._visit_coresdk_nexus_NexusOperationResult(fs, o.result)

async def _visit_temporal_api_stream_v1_StreamRecord(
self, fs: VisitorFunctions, o: Any
):
if o.HasField("body"):
await self._visit_temporal_api_common_v1_Payload(fs, o.body)

async def _visit_coresdk_workflow_activation_StreamReadEnd(
self, fs: VisitorFunctions, o: Any
):
if o.HasField("failure"):
await self._visit_temporal_api_failure_v1_Failure(fs, o.failure)

async def _visit_coresdk_workflow_activation_ResolveStreamRead(
self, fs: VisitorFunctions, o: Any
):
for v in o.records:
await self._visit_temporal_api_stream_v1_StreamRecord(fs, v)
if o.HasField("end"):
await self._visit_coresdk_workflow_activation_StreamReadEnd(fs, o.end)

async def _visit_coresdk_workflow_activation_ResolveStreamAppend(
self, fs: VisitorFunctions, o: Any
):
if o.HasField("failed"):
await self._visit_temporal_api_failure_v1_Failure(fs, o.failed)

async def _visit_coresdk_workflow_activation_ResolveStreamClose(
self, fs: VisitorFunctions, o: Any
):
if o.HasField("failed"):
await self._visit_temporal_api_failure_v1_Failure(fs, o.failed)

async def _visit_coresdk_workflow_activation_WorkflowActivationJob(
self, fs: VisitorFunctions, o: Any
):
Expand Down Expand Up @@ -353,6 +385,18 @@ async def _visit_coresdk_workflow_activation_WorkflowActivationJob(
await self._visit_coresdk_workflow_activation_ResolveNexusOperation(
fs, o.resolve_nexus_operation
)
elif o.HasField("resolve_stream_read"):
await self._visit_coresdk_workflow_activation_ResolveStreamRead(
fs, o.resolve_stream_read
)
elif o.HasField("resolve_stream_append"):
await self._visit_coresdk_workflow_activation_ResolveStreamAppend(
fs, o.resolve_stream_append
)
elif o.HasField("resolve_stream_close"):
await self._visit_coresdk_workflow_activation_ResolveStreamClose(
fs, o.resolve_stream_close
)

async def _visit_coresdk_workflow_activation_WorkflowActivation(
self, fs: VisitorFunctions, o: Any
Expand Down Expand Up @@ -492,6 +536,11 @@ async def _visit_coresdk_workflow_commands_ScheduleNexusOperation(
if o.HasField("input"):
await self._visit_nexus_operation_input_payload(fs, o.input)

async def _visit_coresdk_workflow_commands_AppendToStream(
self, fs: VisitorFunctions, o: Any
):
await self._visit_payload_container(fs, o.bodies)

async def _visit_coresdk_workflow_commands_WorkflowCommand(
self, fs: VisitorFunctions, o: Any
):
Expand Down Expand Up @@ -551,6 +600,10 @@ async def _visit_coresdk_workflow_commands_WorkflowCommand(
await self._visit_coresdk_workflow_commands_ScheduleNexusOperation(
fs, o.schedule_nexus_operation
)
elif o.HasField("append_to_stream"):
await self._visit_coresdk_workflow_commands_AppendToStream(
fs, o.append_to_stream
)

async def _visit_coresdk_workflow_completion_Success(
self, fs: VisitorFunctions, o: Any
Expand Down Expand Up @@ -681,12 +734,6 @@ async def _visit_temporal_api_streamservice_v1_ReadRecordsRequest(
) -> None:
pass

async def _visit_temporal_api_stream_v1_StreamRecord(
self, fs: VisitorFunctions, o: Any
):
if o.HasField("body"):
await self._visit_temporal_api_common_v1_Payload(fs, o.body)

async def _visit_temporal_api_streamservice_v1_ReadRecordsResponse(
self, fs: VisitorFunctions, o: Any
):
Expand Down
2 changes: 2 additions & 0 deletions temporalio/bridge/proto/common/__init__.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
from .common_pb2 import (
ExternalStorageMetrics,
NamespacedWorkflowExecution,
StreamReadStop,
VersioningIntent,
WorkerDeploymentVersion,
)

__all__ = [
"ExternalStorageMetrics",
"NamespacedWorkflowExecution",
"StreamReadStop",
"VersioningIntent",
"WorkerDeploymentVersion",
]
8 changes: 5 additions & 3 deletions temporalio/bridge/proto/common/common_pb2.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading