Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
171ecdc
Added the Stream Stores and the hooks of the generated Streams API.
moetemp Nov 14, 2026
514998a
Moved Core to the Stream WIT with a Workflow append and close.
moetemp Nov 14, 2026
6be1259
Ran a Workflow's Stream append in the Store's Local Activity.
moetemp Nov 14, 2026
b2f8071
Ran a Workflow's Stream close in an Activity after its Workflow Task.
moetemp Nov 14, 2026
92dcedf
Let a Stream config hold several Stores with no selector.
moetemp Nov 14, 2026
fc4909c
Failed the Workflow for a Stream call with no Stream config.
moetemp Nov 14, 2026
ad825d3
Let BufferedAppender batch the appends of Workflow code.
moetemp Nov 14, 2026
b53f74c
Moved BufferedAppender to temporalio.contrib.streams.
moetemp Nov 14, 2026
c89f7d3
Added a README to temporalio.contrib.streams.
moetemp Nov 14, 2026
f8d19e7
Ran a Workflow's Stream close in a Local Activity.
moetemp Nov 17, 2026
af3640a
Added current_producer to temporalio.contrib.streams.
moetemp Nov 17, 2026
a3b1a2c
Added as_producer for a Producer that an Activity stored.
moetemp Nov 17, 2026
38c8284
Linked the docs of as_producer to the Producer of the Streams API.
moetemp Nov 17, 2026
ac04e4a
Moved the Stream crates and renamed last_write_ttl to producer_idle_ttl.
moetemp Nov 17, 2026
3f67ec8
Wrote every append and close of a Workflow as the Workflow Producer.
moetemp Nov 17, 2026
688209c
Moved Core to the commit with the idle TTL's name in a comment.
moetemp Nov 17, 2026
abaf7a0
Removed the Stream calls of Workflow code and their Local Activities.
moetemp Nov 17, 2026
17b70ac
Said in the changelog that Workflow code can't call a Stream.
moetemp Nov 17, 2026
a377a3f
Scheduled the Stream calls of Workflow code without the generic hook.
moetemp Nov 18, 2026
1b73059
Fitted the Stream reads to the new names of a polled read.
moetemp Nov 18, 2026
d4e05f4
Moved the Redis Stream Store out of the bridge.
moetemp Nov 18, 2026
20b53dd
Ran the Stream tests on a Store written in Python.
moetemp Nov 18, 2026
8ddc7ff
Removed the memory Stream Store and the Stream runtime from the bridge.
moetemp Nov 18, 2026
f93c959
Added the batch limits of a Stream Store.
moetemp Nov 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,10 @@ jobs:
- run: uv tool install poethepoet
- run: uv sync --all-extras
- run: cargo install --locked nexgen --version 0.2.8 --features advanced --force
# The Stream package needs the read, read pages and read page methods of a poll, which only
# a nexgen fork has.
- run: cargo install --locked --git https://github.com/moetemp/nexgen --rev 07b565b6d4339dc8a3f67ce2e9e72ba4925d7f9c --features advanced --root "$RUNNER_TEMP/nexgen-stream" nexgen
- run: echo "NEX_GEN_STREAM_BIN=$RUNNER_TEMP/nexgen-stream/bin/nexgen" >> "$GITHUB_ENV"
- run: poe build-develop
- run: poe gen-protos
- name: Check generation unchanged
Expand Down
3 changes: 3 additions & 0 deletions changelog/added/chattering-ledger-otter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
Add the experimental Streams API. Client and Activity code create a Stream, append Records to it, read it and close it. Workflow code can't call a Stream in this version. A client takes its Stream Stores in `stream_config=StreamConfig(stores=[...])`, such as `RedisStreamStore` of the `temporalio-streams-redis` package, or `MemoryStreamStore` of the `temporalio-streams-memory` package in tests.
Add `temporalio.contrib.streams.current_producer` and `as_producer`. An Activity can store its Producer in its heartbeat details and pass it to the appends of its retry. The retry's Records then continue the first attempt's epoch, so a reader does not take them as a replacement of the first attempt's Records.
A Workflow outbound interceptor sees a Stream operation only in its operation-specific method, such as `start_append_records`, and not in `start_system_nexus_operation`. An unknown Stream operation raises `ValueError`.
5 changes: 5 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -173,12 +173,17 @@ exclude = [
'temporalio/api',
'temporalio/bridge/proto',
'temporalio/nexus/system/workflow_service',
'temporalio/nexus/system/stream_service',
]

[[tool.mypy.overrides]]
module = "temporalio.nexus.system.workflow_service.*"
ignore_errors = true

[[tool.mypy.overrides]]
module = "temporalio.nexus.system.stream_service.*"
ignore_errors = true

[tool.pydocstyle]
convention = "google"
# https://github.com/PyCQA/pydocstyle/issues/363#issuecomment-625563088
Expand Down
82 changes: 67 additions & 15 deletions scripts/gen_nexus_system_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,15 @@
)
wit_deps_dir = wit_input_dir / "deps"
workflow_output_dir = base_dir / "temporalio" / "nexus" / "system" / "workflow_service"
stream_output_dir = base_dir / "temporalio" / "nexus" / "system" / "stream_service"
notification_output_dir = base_dir / "temporalio" / "nexus" / "notifications"
workflow_init_path = base_dir / "temporalio" / "workflow" / "__init__.py"
NEX_GEN_VERSION = "0.2.8"
# The Stream package needs a newer nexgen, which rejects the other WITs. One pin is enough
# again once the api WITs and a nexgen release agree. The read, read pages and read page methods
# of a poll are only in commit 07b565b6 of the fork https://github.com/moetemp/nexgen. So
# NEX_GEN_STREAM_BIN points at a build of that commit.
NEX_GEN_STREAM_VERSION = "0.2.12"
proto_files = [
gen_protos.api_proto_dir
/ "temporal"
Expand All @@ -36,10 +42,22 @@
gen_protos.api_proto_dir
/ "temporal"
/ "api"
/ "streamservice"
/ "v1"
/ "request_response.proto",
gen_protos.api_proto_dir
/ "temporal"
/ "api"
/ "notificationservice"
/ "v1"
/ "request_response.proto",
]
# The Workflow functions of these generated packages are exported by temporalio.workflow.
# They are in import order, so that sorting the imports keeps them between the markers.
workflow_export_packages = [
("temporalio.nexus.system.stream_service.workflow", stream_output_dir / "workflow"),
("temporalio.nexus.system.workflow_service", workflow_output_dir),
]


def nex_gen_command() -> list[str]:
Expand All @@ -66,6 +84,21 @@ def nex_gen_command() -> list[str]:
return ["nexgen"]


def nex_gen_stream_command() -> list[str]:
bin_path = os.environ.get("NEX_GEN_STREAM_BIN")
if not bin_path:
raise RuntimeError(
"NEX_GEN_STREAM_BIN must point at a nexgen build of commit 07b565b6 "
"of https://github.com/moetemp/nexgen"
)
version = subprocess.check_output([bin_path, "--version"], text=True).strip()
if version != f"nexgen {NEX_GEN_STREAM_VERSION}":
raise RuntimeError(
f"NEX_GEN_STREAM_BIN is {version}, not nexgen {NEX_GEN_STREAM_VERSION}"
)
return [bin_path]


def build_descriptor_set(descriptor_path: Path) -> None:
subprocess.check_call(
[
Expand All @@ -85,14 +118,15 @@ def generate_package(
output_dir: Path,
descriptor_path: Path,
*,
native_api: bool,
flags: list[str],
) -> None:
args = [*command, "python", str(wit_input_dir / wit_name), str(wit_deps_dir)]
if native_api:
args.append("--native-api")
subprocess.check_call(
[
*args,
*command,
"python",
str(wit_input_dir / wit_name),
str(wit_deps_dir),
*flags,
"--system-nexus",
"--support-package",
"temporalio.nexus.system._support",
Expand Down Expand Up @@ -132,17 +166,26 @@ def replace_marker_block(


def generate_workflow_exports() -> None:
exports = generated_exports(workflow_output_dir / "__init__.py")
package_exports = [
(module, generated_exports(package_dir / "__init__.py"))
for module, package_dir in workflow_export_packages
]
import_block = [
"# BEGIN GENERATED NEXUS SYSTEM EXPORTS\n",
"from temporalio.nexus.system.workflow_service import (\n",
*[f" {export},\n" for export in exports],
")\n",
*[
line
for module, exports in package_exports
for line in [
f"from {module} import (\n",
*[f" {export},\n" for export in exports],
")\n",
]
],
"# END GENERATED NEXUS SYSTEM EXPORTS\n",
]
all_block = [
" # BEGIN GENERATED NEXUS SYSTEM __ALL__\n",
*[f' "{export}",\n' for export in exports],
*[f' "{export}",\n' for _, exports in package_exports for export in exports],
" # END GENERATED NEXUS SYSTEM __ALL__\n",
]
content = workflow_init_path.read_text()
Expand All @@ -162,6 +205,7 @@ def generate_workflow_exports() -> None:
def generate_nexus_system_api() -> None:
required_paths = [
wit_input_dir / "workflow-service.wit",
wit_input_dir / "stream-service.wit",
wit_input_dir / "notification-service.wit",
wit_deps_dir,
*proto_files,
Expand All @@ -173,23 +217,31 @@ def generate_nexus_system_api() -> None:
descriptor_path = Path(temp_dir) / "temporal_api.bin"
build_descriptor_set(descriptor_path)
command = nex_gen_command()
for wit_name, output_dir, native_api in (
("workflow-service.wit", workflow_output_dir, True),
("notification-service.wit", notification_output_dir, False),
stream_command = nex_gen_stream_command()
for wit_name, output_dir, package_command, flags in (
("workflow-service.wit", workflow_output_dir, command, ["--native-api"]),
(
"stream-service.wit",
stream_output_dir,
stream_command,
["--targets", "workflow,client"],
),
("notification-service.wit", notification_output_dir, command, []),
):
shutil.rmtree(output_dir, ignore_errors=True)
output_dir.parent.mkdir(parents=True, exist_ok=True)
generate_package(
command,
package_command,
wit_name,
output_dir,
descriptor_path,
native_api=native_api,
flags=flags,
)
workflow_output_dir.parent.joinpath("__init__.py").touch()
generate_workflow_exports()
format_paths = [
str(workflow_output_dir),
str(stream_output_dir),
str(notification_output_dir),
str(workflow_init_path),
]
Expand Down
34 changes: 22 additions & 12 deletions scripts/gen_payload_visitor.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,20 +24,30 @@


def discover_system_nexus_roots() -> list[Descriptor]:
module_path = base_dir / "temporalio" / "nexus" / "system" / "workflow_service"
spec = spec_from_file_location(
"temporalio_nexus_system_workflow_service",
module_path / "__init__.py",
submodule_search_locations=[str(module_path)],
)
if spec is None or spec.loader is None:
raise RuntimeError(f"Cannot load generated system service from {module_path}")
module = module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
services: list[type] = []
system_path = base_dir / "temporalio" / "nexus" / "system"
for module_path in sorted(p.parent for p in system_path.glob("*/services.py")):
spec = spec_from_file_location(
f"temporalio_nexus_system_{module_path.name}",
module_path / "__init__.py",
submodule_search_locations=[str(module_path)],
)
if spec is None or spec.loader is None:
raise RuntimeError(
f"Cannot load generated system service from {module_path}"
)
module = module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
services.extend(
value
for value in vars(module.services).values()
if isinstance(value, type) and nexusrpc.get_service_definition(value)
)

annotations = [a for service in services for a in get_type_hints(service).values()]
roots: list[Descriptor] = []
for annotation in get_type_hints(module._services.WorkflowService).values():
for annotation in annotations:
if get_origin(annotation) is not nexusrpc.Operation:
continue
for operation_type in get_args(annotation):
Expand Down
4 changes: 2 additions & 2 deletions temporalio/api/enums/v1/stream_pb2.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ class _StreamErrorReasonEnumTypeWrapper(
STREAM_ERROR_REASON_NOT_FOUND: _StreamErrorReason.ValueType # 1
"""Nobody created the Stream. NOT_FOUND."""
STREAM_ERROR_REASON_EXPIRED: _StreamErrorReason.ValueType # 2
"""The Stream's Retention or its last-write TTL passed. NOT_FOUND."""
"""The Stream's Retention or its Producer idle TTL passed. NOT_FOUND."""
STREAM_ERROR_REASON_CLOSED: _StreamErrorReason.ValueType # 3
"""The Stream is closed. FAILED_PRECONDITION."""
STREAM_ERROR_REASON_FENCED: _StreamErrorReason.ValueType # 4
Expand Down Expand Up @@ -139,7 +139,7 @@ STREAM_ERROR_REASON_UNSPECIFIED: StreamErrorReason.ValueType # 0
STREAM_ERROR_REASON_NOT_FOUND: StreamErrorReason.ValueType # 1
"""Nobody created the Stream. NOT_FOUND."""
STREAM_ERROR_REASON_EXPIRED: StreamErrorReason.ValueType # 2
"""The Stream's Retention or its last-write TTL passed. NOT_FOUND."""
"""The Stream's Retention or its Producer idle TTL passed. NOT_FOUND."""
STREAM_ERROR_REASON_CLOSED: StreamErrorReason.ValueType # 3
"""The Stream is closed. FAILED_PRECONDITION."""
STREAM_ERROR_REASON_FENCED: StreamErrorReason.ValueType # 4
Expand Down
10 changes: 10 additions & 0 deletions temporalio/client/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,12 @@
ScheduleUpdate,
ScheduleUpdateInput,
)
from ._stream_store import (
StreamBatchLimits,
StreamConfig,
StreamSelectContext,
StreamStore,
)
from ._worker_versioning import (
BuildIdOp,
BuildIdOpAddNewCompatible,
Expand Down Expand Up @@ -350,6 +356,10 @@
"TaskReachabilityType",
"CloudOperationsClient",
"Plugin",
"StreamBatchLimits",
"StreamConfig",
"StreamSelectContext",
"StreamStore",
"Callback",
"_ClientImpl",
"_apply_headers",
Expand Down
24 changes: 23 additions & 1 deletion temporalio/client/_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import temporalio.api.workflowservice.v1
import temporalio.common
import temporalio.converter
import temporalio.nexus.system.stream_service.client
import temporalio.runtime
import temporalio.service
import temporalio.workflow
Expand Down Expand Up @@ -94,6 +95,7 @@
ScheduleBackfill,
ScheduleHandle,
)
from ._stream_store import StreamConfig, _refuse_store_plugins
from ._worker_versioning import (
BuildIdOp,
TaskReachabilityType,
Expand All @@ -114,7 +116,9 @@
from ._plugin import Plugin


class Client:
class Client(
temporalio.nexus.system.stream_service.client.StreamServiceClientExtension
):
"""Client for accessing Temporal.

Most users will use :py:meth:`connect` to create a client. The
Expand Down Expand Up @@ -156,6 +160,7 @@ async def connect(
grpc_compression: GrpcCompression = GrpcCompression.GZIP,
payload_limits: PayloadLimitsConfig = PayloadLimitsConfig(),
header_codec_behavior: HeaderCodecBehavior = HeaderCodecBehavior.NO_CODEC,
stream_config: StreamConfig | None = None,
) -> Self:
"""Connect to a Temporal server.

Expand Down Expand Up @@ -225,7 +230,18 @@ async def connect(
payload_limits: Warning thresholds for outbound payload/memo sizes. Over-threshold
fields are logged but still sent. Set a threshold to 0 to disable it.
header_codec_behavior: Encoding behavior for headers sent by the client.
stream_config: The Stream Stores of this client and the selector that
picks one for a new Stream, as
:py:class:`temporalio.converter.ExternalStorage` holds drivers and a
selector. A client with it sends each Stream operation to one of its
Stores. This release has no Streams on the server, so with ``None`` a
Stream operation raises
:py:class:`temporalio.exceptions.StreamConfigError`.

.. warning::
This API is experimental and subject to change.
"""
_refuse_store_plugins(plugins)
connect_config = temporalio.service.ConnectConfig(
target_host=target_host,
api_key=api_key,
Expand Down Expand Up @@ -262,6 +278,7 @@ def make_lambda(
default_workflow_query_reject_condition=default_workflow_query_reject_condition,
header_codec_behavior=header_codec_behavior,
plugins=plugins,
stream_config=stream_config,
)

def __init__(
Expand All @@ -275,11 +292,13 @@ def __init__(
default_workflow_query_reject_condition: None
| (temporalio.common.QueryRejectCondition) = None,
header_codec_behavior: HeaderCodecBehavior = HeaderCodecBehavior.NO_CODEC,
stream_config: StreamConfig | None = None,
):
"""Create a Temporal client from a service client.

See :py:meth:`connect` for details on the parameters.
"""
_refuse_store_plugins(plugins)
# Store the config for tracking
config = ClientConfig(
service_client=service_client,
Expand All @@ -289,6 +308,7 @@ def __init__(
interceptors=interceptors,
default_workflow_query_reject_condition=default_workflow_query_reject_condition,
header_codec_behavior=header_codec_behavior,
stream_config=stream_config,
)
self._initial_config = config.copy()

Expand Down Expand Up @@ -3029,6 +3049,7 @@ class ClientConnectConfig(TypedDict, total=False):
grpc_compression: GrpcCompression
payload_limits: PayloadLimitsConfig
header_codec_behavior: HeaderCodecBehavior
stream_config: StreamConfig | None


class ClientConfig(TypedDict, total=False):
Expand All @@ -3043,3 +3064,4 @@ class ClientConfig(TypedDict, total=False):
temporalio.common.QueryRejectCondition | None
]
header_codec_behavior: Required[HeaderCodecBehavior]
stream_config: Required[StreamConfig | None]
Loading
Loading