Skip to content

feat: read codec-encoded external storage references - #1814

Open
jmaeagle99 wants to merge 1 commit into
mainfrom
extstore-encode-refs
Open

jmaeagle99 wants to merge 1 commit into
mainfrom
extstore-encode-refs

Conversation

@jmaeagle99

Copy link
Copy Markdown
Contributor

What was changed

  • Read: before retrieval, every payload marked with __temporal_external_storage_reference: "true" is decoded with the payload codec, one reference at a time, using the same serialization context as the rest of the message. A marked payload that doesn't decode to a reference fails with the new TMPRL1106 (ExternalStorageError::UndecodableReference). One with no storage configured fails with TMPRL1105.
  • Write, off by default: after storing, each reference is encoded, marked, and keeps its external_payloads. This is switched on by a hidden ExternalStorage::with_encoded_references, which defaults to false.
  • store_external_payloads and retrieve_external_payloads take a &SerializationContextData. Each worker and client call site builds one context and passes it to both its codec calls and its storage calls.

Why?

A reference written in plain form puts its claim data, such as bucket and key names, into history unencrypted, even when a codec is configured. Encoding references fixes that. Readers have to ship first, because a worker that can't decode an encoded reference can't find it at all. That's why writing ships here but stays off: it can be turned on in a later release by changing the default, once workers have upgraded.

Checklist

  1. How was this tested:

Unit tests for the encode and decode passes, and integration tests covering an encoded round trip read by a client with writing off, signals, queries and memos, and a worker without the codec failing with TMPRL1106.

  1. Any docs updates needed?

Rustdoc only. The changelog fragment is included.

@jmaeagle99
jmaeagle99 marked this pull request as ready for review November 18, 2026 00:46
@jmaeagle99
jmaeagle99 requested a review from a team as a code owner November 18, 2026 00:46

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant