Repository navigation
[HSTACK] feat(datafusion): configurable planning-metrics serialization - #26
Merged
Merged
Conversation
…n in Delta physical codecs
Add `DeltaPhysicalCodecConfig { serialize_planning_metrics }` (default: true)
to `DeltaPhysicalCodec` and `DeltaNextPhysicalCodec`. When disabled, planning
time scan count metrics are neither written nor restored on decode, so plans
shipped to execution-only workers (e.g. datafusion-distributed tasks) do not
carry coordinator-owned metrics that would otherwise be counted once per task.
Also drop the `created_at=planning` label from decoded metrics: its only
consumer was a datafusion-distributed fork patch that is reverted in favor of
this config (datafusion-contrib/datafusion-distributed#756).
Breaking: build the codecs with `::default()` / `::new(config)` instead of
`DeltaPhysicalCodec {}` / `DeltaNextPhysicalCodec`.
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
radustoenescu
pushed a commit
that referenced
this pull request
Oct 22, 2026
…zation (#24, #26) Preserve planning-time count metrics when serializing classic and next Delta scans, with a shared configuration controlling both encoding and decoding. Keep serialization enabled by default; disable it for execution-only worker plans so coordinator-owned counters are not multiplied across tasks. Introduce the final codec implementation directly, without the intermediate created_at=planning label mechanism. Consolidates b9b71be and ebd748a; the resulting repository tree is unchanged from ebd748a.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DeltaPhysicalCodecConfig { serialize_planning_metrics }to the classic and next physical codecs; it defaults to preserving current serialization behavior.created_at=planninglabel: execution-only workers no longer need copies of coordinator-owned planning counters.This allows Cobweb to retain planning metrics through Flight tickets without sending copies to every DFD task. No DFD change is required.
Related discussion: datafusion-contrib/datafusion-distributed#756.
API change
Construct the codecs with
::default()or::new(config)instead of unit/braced literals. Consumers such as OneAdobe/datafusion-python must update constructor calls when adopting this revision.Scope
Based on
main-1.0-stats. Flag coverage is one parameterized test: classic/next × enabled/disabled. Cross-configuration decoding and distributed metric ownership are covered in Cobweb.