Skip to content

[NO NOT MERGE] Direct Nexus calls - #720

Open
bergundy wants to merge 4 commits into
temporalio:mainfrom
bergundy:direct-nexus-calls
Open

bergundy wants to merge 4 commits into
temporalio:mainfrom
bergundy:direct-nexus-calls

Conversation

@bergundy

@bergundy bergundy commented Feb 13, 2026 •

Copy link
Copy Markdown
Member

What changed?

Added two RPCs for making direct Nexus requests using namespace credentials: StartNexusOperation and CancelNexusOperation.

Unlike standalone Nexus operations (StartNexusOperationExecution, etc.), direct calls do not create a durable caller-side operation. They go directly to the handler registered for the given endpoint. Initially, only the system Nexus endpoint (__temporal_system) will be supported.

  • The response mirrors temporal.api.nexus.v1.StartOperationResponse. It is either a sync result, an async start, or a Temporal Failure with ApplicationFailureInfo or CanceledFailureInfo. Links use temporal.api.common.v1.Link.
  • Handler errors are returned as gRPC errors. The status code is derived from the handler error type, and a new NexusHandlerFailure error detail carries the handler error. Clients should honor the handler error's retry behavior rather than retrying by status code.
  • namespace only applies to the system Nexus endpoint, where it is the namespace the operation runs in. It is passed in the request body, not in the HTTP route.
  • Added Callback.Ephemeral, which is set via StartNexusOperationRequest.completion_callback.
    • An ephemeral callback is never persisted. It turns the request that carries it into a long poll.
    • If the operation completes before the long-poll deadline, the outcome is returned synchronously.
    • Otherwise, the response is an async start, and the caller re-issues the request with the same request ID to keep waiting.
    • Other APIs reject this variant for now.
  • Added temporal.api.activityservice.v1.ActivityService, a Nexus service exposed on the system Nexus endpoint. It is not served over gRPC.
    • Its GetResult operation waits for a standalone activity to complete. Its output is the activity's result payload, as encoded by the activity. If the activity fails, times out, or is terminated, the operation fails with the activity's failure as the cause. If the activity is canceled, the operation is canceled.
    • This is the same completion that a callback attached to the activity delivers, so workflows can call the operation as well as direct callers.
    • The operation runs in the caller's namespace, so the request has no namespace field.
    • Added nexus/activity-service.wit, and added the service to the generated nexus-rpc YAML.
  • Exempted the direct-call RPCs and ActivityService.GetResult from the temporal-resource-id lint. The direct calls' routing target is derived from the operation input (start) or the operation token (cancel).

The HTTP routes (/nexus-requests/endpoints/{endpoint}/services/{service}/operations/{operation}[/cancel]) qualify these APIs as "requests" (nexus-requests) to distinguish them from standalone Nexus operation routes (nexus-operations).

Why?

This lets callers outside of a workflow invoke system Nexus operations with minimal overhead, including long-running ones. An async system operation called from a workflow completes via a durable callback. Called from outside a workflow, the same operation is awaited with an ephemeral callback.

A separate service per archetype keeps each operation's implementation in one place on the server, instead of several libraries implementing operations of WorkflowService.

Notes

  • GetResult returns temporal.api.common.v1.Payload directly, so buf.yaml ignores RPC_RESPONSE_STANDARD_NAME for activityservice/v1/service.proto. api-linter.yaml disables the HTTP annotation and Get method rules for the service.
  • The nexus-rpc YAML generator now derives type refs from the file that defines each message, so operations can use messages from other packages.
  • The WIT was generated with nexgen 0.2.12. Generated system bindings currently decode a payload output as a system-encoded Payload proto. For GetResult, they need to decode the output with the user's data converter instead.
  • With released nexgen versions, make system-nexus-wit fails on the existing notification-service.wit (0.2.8 lacks add-message --file, and 0.2.12 rejects @nexus.doc on an interface). The ActivityService step runs after it, so I ran that step separately. Its output doesn't change when regenerated.

Server PR

Not yet.

@bergundy
bergundy requested review from a team as code owners February 13, 2026 22:50

// An operation was started asynchronously. The token can be used for cancellation.
message AsyncSuccess {
string operation_token = 1;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was under the impression that operation token was only for handler-server and caller-server side, not caller client side which operates on operation IDs

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for exposing direct calls to a handler, bypassing the durable caller machinery. The semantics are different.

@cretz cretz Feb 18, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand. Where are the RPC calls for standalone Nexus operations from a client that are durable and take operation IDs and such? The name of these RPCs makes it seem like those. Or are standalone Nexus operations as being designed internally not occurring as designed?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#685 see also PR description.

@cretz cretz Feb 18, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So both PRs have StartNexusOperation and mean different things, though both PR descriptions acknowledge the ambiguity.

I would recommend not using the term Execution to differentiate. I do think the outbound caller form should use Execution, but I think the "direct" form here should have a more qualifying name. I assume this is mostly for server-to-server because most people will want durable and therefore even if they are calling a handler on their own cluster, they'll use the durable/visibility-based form from the other PR, correct?

You're going to have to have an adjective to refer to this form of API regardless, so I recommend baking that adjective into the RPC names as you will with docs or anything else. Direct may be ok, but other options may make sense too, but we need some term to refer to these in both the RPC name and in conversation/docs IMO.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use cases are:

  • Exposing system endpoint Nexus APIs to SDK clients (allows us to go all in on Nexus for public APIs, each API would automatically work within and outside of a workflow)
  • Building interfaces with workers as a backend (e.g. for the UI and CLI)

I intentionally used the API names from the Nexus handler specification since there's a 1-1 mapping with those APIs.

@cretz cretz Feb 18, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need a term for this approach and the RPCs should include that term to avoid ambiguity with the other ones. Execution isn't enough IMO to differentiate (and won't be in CLI, docs, or anywhere else).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not planning on exposing these methods to the CLI or docs BTW. This is an internal detail that would be used inside the SDK. We have been using the terms Durable and Direct or Execution and Request.

I'm not sure what to call this. Maybe:

  • StartDirectNexusOperation?
  • CancelDirectNexusOperation?

Got better ideas?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatives:

  • SendStartNexusOperationRequest
  • SendCancelNexusOperationRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like StartNexusOperation vs StartNexusOperationExecution: clear and elegant IMO.

Add StartNexusOperation and CancelNexusOperation to the WorkflowService for invoking Nexus operations directly,
without a durable caller-side operation.

- Responses mirror temporal.api.nexus.v1.StartOperationResponse: sync result, async start, or a Temporal failure.
  Links use temporal.api.common.v1.Link.
- Add Callback.Ephemeral. An ephemeral callback is never persisted and turns the request that carries it into a
  long poll. The outcome is returned synchronously if the operation completes before the long-poll deadline;
  otherwise the request returns as an async start and the caller re-issues it with the same request ID.
- Expose PollActivityExecution as a system Nexus operation. The request namespace must be empty when invoked via
  Nexus.
- Exempt the direct-call RPCs from the temporal-resource-id annotation requirement.
// Also exposed as an asynchronous operation on the system Nexus endpoint. When invoked via Nexus, the request's
// namespace must be empty; the caller's namespace is used.
rpc PollActivityExecution (PollActivityExecutionRequest) returns (PollActivityExecutionResponse) {
option (nexusannotations.v1.operation).tags = "exposed";

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This must be reverted before committing. I'm not proposing to add this API as a Nexus API now, I just needed an "async" API for the prototype.

…Failure

- The StartNexusOperation and CancelNexusOperation HTTP routes no longer include the namespace, which is only
  applicable to the system Nexus endpoint and is passed in the body.
- Handler errors are returned as gRPC errors with a NexusHandlerFailure detail carrying the handler error,
  whose retry behavior clients should honor.
…yService.GetResult

ActivityService is a new Nexus-only service on the system Nexus endpoint. Its GetResult
operation outputs the activity's result payload as is, and fails or is canceled with the
activity's failure. That's the same completion a durable callback attached to the activity
delivers, so workflows can call the operation too, not just direct callers.

PollActivityExecution is no longer exposed via Nexus.

The Nexus RPC YAML generator now derives type refs from the file that defines each message,
so operations can use messages from other packages, e.g. GetResult outputs a common.v1.Payload.

@bergundy bergundy left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

activity service was added for the demo, not to be merged.

bergundy added a commit to temporalio/temporal that referenced this pull request Nov 18, 2026
## What changed?
Added `chasm.ComponentFieldEphemeral()`, a component field option that
marks a component, and everything under it, as in-memory only.

- Ephemeral nodes are never serialized, persisted, snapshotted, or
replicated. They only live while the execution's mutable state is
loaded, and they're lost when it's reloaded.
- A map that was never persisted isn't persisted while it only holds
ephemeral items. Adding an ephemeral item to an empty map doesn't
persist the map, and removing the item leaves no tombstone. The map is
persisted once it holds a durable item.
- Pending component metadata, e.g. request links, isn't applied to
ephemeral nodes.
- Adding tasks to an ephemeral component, including immediate pure
tasks, returns an internal error (via softassert) instead of silently
dropping the tasks.
- Replacing a field's value deletes the replaced node's subtree. Paths
the new node writes are rewritten rather than deleted. This also fixes
durable replacements, where descendants the new value didn't have were
left behind and came back on reload.
- A durable data pointer to data inside an ephemeral subtree fails the
transaction, since it would dangle after a reload.
- A replicated node at or under an ephemeral node replaces the ephemeral
node and its subtree.

## Why?
This is the first step towards direct Nexus calls (temporalio/api#720).
A caller outside of a workflow attaches an ephemeral completion callback
to the execution it waits on, turning its request into a long poll. The
callback is a regular CHASM `Callback` component, but it belongs to an
in-flight request, so it must never be persisted.

A follow-up PR will skip persisting update transactions that make no
durable changes, so that attaching an ephemeral component doesn't cost a
write.

## How did you test it?
- [x] built
- [ ] run locally and tested manually
- [x] covered by existing tests
- [x] added new unit test(s)
- [ ] added new functional test(s)

New `chasm/tree_test.go` tests cover:
- An ephemeral item and its subtree are excluded from mutations and
snapshots, including when the item is mutated or has request links set.
- The ephemeral item keeps its in-memory state across transactions.
- A map that only held ephemeral items is persisted once a durable item
is added.
- Removing the ephemeral item leaves no tombstone, while removing the
durable item does.
- Emptying a never-persisted map removes it without a tombstone.
- Adding tasks to an ephemeral component is rejected.
- Replacing a durable item with an ephemeral one deletes the durable
item and its subtree.
- Replacing a durable component with a durable one deletes only the
descendants the new component doesn't have.
- A replicated durable item replaces an ephemeral item at the same path,
while other ephemeral items keep their in-memory state.
- A durable data pointer to ephemeral data is rejected, while pointers
within the ephemeral subtree are allowed.

## Potential risks
- Components that use ephemeral fields must treat them as a cache. Their
state disappears when the mutable state is reloaded, evicted, or moved
to another shard.
- `syncSubComponents` now always serializes a collection that was never
persisted. Without ephemeral fields, that's only a newly created
collection, which was already serialized before this change.
Handler errors returned by StartNexusOperation and CancelNexusOperation carry a
temporal-nexus-retryable trailer, which reflects the handler error's retry behavior since the
status code doesn't, and a temporal-nexus-failure-source trailer for errors that originated
outside the server. Operation failures stay in the response.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants