Skip to content

feat(extstore): decode extstore references that were encoded by a codec. - #671

Open
cconstable wants to merge 3 commits into
mainfrom
extstore-encode-refs
Open

cconstable wants to merge 3 commits into
mainfrom
extstore-encode-refs

Conversation

@cconstable

@cconstable cconstable commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

What was changed

Workers and clients now read external storage references that were encoded by the payload codec and marked with __temporal_external_storage_reference metadata. Each marked reference is decoded with the configured codec, then retrieved from storage as before. This release only reads encoded references. Writing them stays disabled until a later release.

  • No storage configured: a marked reference raises ExternalStorageNotConfiguredError (TMPRL1105) before anything is decoded.
  • Marked payload that doesn't decode to a reference: raises the new ExternalStorageReferenceDecodeError (TMPRL1106).

No configuration changes. Encoded references are decoded with the codec you already have:

data_converter = Temporalio::Converters::DataConverter.new(
  payload_codec: EncryptionCodec.new, # now also decodes encoded references
  external_storage: Temporalio::Converters::ExternalStorage.new(drivers: [s3_driver])
)

Why

References hold driver claim data such as bucket names and object keys. Until now they were written to history unencoded, even when a codec encrypted everything else.

Spec: https://github.com/temporalio/features/blob/encoding-payload-refs/features/external_storage/spec.md#wire-format

@cconstable
cconstable marked this pull request as ready for review November 16, 2026 22:10
@cconstable
cconstable requested a review from a team as a code owner November 16, 2026 22:10

@jmaeagle99 jmaeagle99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve from an external storage point of view. Would like to see if @chris-olszewski has any feedback.


def decode(payloads)
payloads.map do |p|
next p unless p.metadata['encoding'] == 'test/base64'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This automatically weakens the strict nature of this codec for all tests. Consider subclassing or adding a optional parameter to allow passthrough when encoding doesn't match with a default of false.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants