Skip to content

Prevent extracted-component overwrites with content-addressed filenames - #71

Merged
Volv-G merged 1 commit into
masterfrom
piforge/tangle-pipeline-crud/dehydrator-content-addressed-com-024377c
Sep 29, 2026
Merged

Volv-G merged 1 commit into
masterfrom
piforge/tangle-pipeline-crud/dehydrator-content-addressed-com-024377c

Conversation

@Volv-G

@Volv-G Volv-G commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

(AI-assisted)

What

Dehydration named extracted component files after the component name alone while deduplicating on digest. Two different components sharing a name therefore wrote to one path: the first spec was silently overwritten, and both tasks were redirected onto the survivor.

This is the parent for the upcoming YAML→Python decompiler, which reuses this same extraction path and cannot offer a distinct-component guarantee without it.

Evidence

Reproduced on master with two different specs that share the name train model:

task a: {'url': 'file://./components/train_model.yaml'}
task b: {'url': 'file://./components/train_model.yaml'}
files written: ['train_model.yaml']   # payload 'first' destroyed; both tasks point at 'second'

After the change: two files, payloads ['first', 'second'], distinct URLs.

A supplied digest is not proof of content. The hydrator digests a component's raw source text (pipeline_hydrator.py:860), so two byte-different files holding the same component carry different valid digests — a leading comment is enough:

specs equal: True
carried digests equal: False | both valid sha256: True
spec-derived digests equal: True

The converse also occurred: an edited spec keeps the digest of what it used to be, so two different specs could share a stale digest, collapse onto one file, and bypass the overwrite check entirely.

How

  • Extracted filenames are <stem>-<sha256-of-spec><extension>, and the dedup key is that same address, computed with the canonical utils.compute_spec_digest(spec) — never componentRef.digest. The ref digest is a locator; the file's content is the spec.
  • Collision safety is the full 64-character address, not a short prefix: two specs can only share a filename if they share a digest, in which case sharing the file is correct. _safe_filename folds - to _, so the separator cannot occur inside the stem.
  • The stem is truncated on UTF-8 bytes, not code points — _safe_filename keeps non-ASCII alphanumerics, and a CJK stem is three bytes per character, so '界'*62 produced a 256-byte name under a code-point cap. Truncation lands on a character boundary.
  • The address and extension are reserved first; the separator is charged to the stem so an extension that fits by exactly one byte is not rejected. When nothing fits, the name is <address><extension> with no leading dash.
  • An extension too long to form any valid name raises ComponentFilenameTooLongError naming the field and byte count, instead of surfacing as OSError: [Errno 63] File name too long from inside the writer.

Failure modes

  • Extracted component filenames change. dehydrate FILE/AUTO output is <name>-<digest>.yaml rather than <name>.yaml. Refs are rewritten consistently in the same pass and stay relative to the output file, so a dehydrate→hydrate round trip is unaffected; previously extracted files on disk are not migrated.
  • Filenames are now stable across re-export and independent of source formatting and traversal order, where they previously tracked the component name and the order tasks were walked.
  • All other choices (DIGEST, NAME, URL, KEEP, AUTO's canonical-URL and library-digest preferences), URI I/O, relative-ref rewriting and subgraph extraction are unchanged.
  • Subgraph filenames still use <name>_<counter> and remain traversal-order dependent. They write to a separate directory and are unique within a run, so there is no data-loss bug there; deliberately out of scope.

Testing

  • Full suite 1983 passed on Python 3.10 and 1983 on 3.12.
  • Three regressions folded into tests/test_pipeline_dehydrator.py: same-name/different-spec (sharing a stale locator digest) keeps both payloads; identical specs with differing, "unknown" and missing digests write once and share a URL named from the canonical spec digest; and a byte-limit test covering a CJK name, the exact address + extension == 255 boundary, and one byte over rejected without writing.
  • Mutation-tested throughout; two survivors led to real fixes — a digest-case test that was passing only because macOS is case-insensitive and would have failed on Linux CI, and a byte-vs-character budget that is indistinguishable unless the extension is multi-byte.

Residual gaps: tests/test_packaging.py could not run locally (pkgs.shopify.io returns 400 for uv-build; identical on pristine master), and the byte-limit tests prove encoded length and successful creation on APFS only — native Linux was unavailable. Both are covered by CI.

Review focus

  • The behaviour change is the filename shape; confirm no consumer outside this repo hard-codes components/<name>.yaml. No in-tree caller does, and the known downstream wrapper overrides only the constructor and prompt, not _save_component_to_file (whose signature lost its now-unused digest parameter).
  • ComponentFilenameCollisionError is unreachable while the address is a full digest; it is kept as a loud guard so any future shortening fails instead of silently overwriting.

Dehydration named extracted component files after the component name alone
while deduplicating on digest, so two different components sharing a name
wrote to one path: the first spec was silently overwritten and both tasks
were redirected onto the survivor.

Filenames are now <stem>-<sha256-of-spec><extension>, addressed by the spec
being written rather than componentRef.digest -- the hydrator digests a
component's raw source text, so equal specs can carry different valid
digests and an edited spec can still carry a stale one.

The stem is truncated on UTF-8 bytes, not code points, and is dropped
entirely when the address and extension fill the 255-byte limit. An
extension too long to name at all is reported instead of surfacing as
OSError from the writer.
@Volv-G
Volv-G requested a review from Ark-kun as a code owner September 29, 2026 06:38
@Volv-G
Volv-G merged commit 9b831f0 into master Sep 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant