Archiver 2GB byte-array ceiling fix

[ehurheap]

Problem

HoodieTimelineArchiver OOMs at HoodieAvroDataBlock.serializeRecords while wrapping clean instants into an archive log block. Root cause: HoodieCleanerPlan.filePathsToBeDeletedPerPartition and HoodieCleanMetadata.partitionMetadata each carry per-partition maps that, at 6.7M partitions, produce serialized instants whose individual Avro block exceeds Integer.MAX_VALUE bytes. Lowering the archival batch size doesn't help — a single clean instant is already >2GB.

Changes

Two opt-in features + one CLI plumbing fix, all default-off so behaviour is unchanged unless explicitly enabled.

1. hoodie.archive.trim.clean.action.metadata (HoodieArchivalConfig) — new boolean, default false. When true, strips per-partition file lists from HoodieCleanerPlan. Retention boundary, policy, timestamps, summary counts, and extraMetadata are preserved.
2. MetadataConversionUtils.createMetaWrapper — new 3-arg overload accepting trimCleanActionMetadata. Old 2-arg signature kept as a delegating shim. HoodieTimelineArchiver.convertToAvroRecord reads the config and passes it through. In-place mutation on the freshly-deserialized objects avoids allocating a rebuilt copy of the very maps we're shrinking.
3. ArchiveExecutorUtils.archive — added .withProperties(System.getProperties()) to the HoodieWriteConfig builder. Previously the ARCHIVE CLI's config was built from only the five positional args and silently ignored any -Dhoodie.*=… passed via spark.driver.extraJavaOptions. Now fork-side configs (this trim flag, existing options like hoodie.commits.archival.batch.size) can be tuned per-invocation without a code change.

Verified

Manual spark-submit of the ARCHIVE command against events-v1 with -Dhoodie.archive.trim.clean.action.metadata=true completes in ~57 min through the previously-failing Feb 2023 clean instants; nine archive-file rollovers, ~1GB each, well under the ceiling. Prior runs OOM'd at the same fourth clean instant every time.

Compatibility

• No default behaviour change: trim is off, existing createMetaWrapper callers unchanged, existing CLI arg positions unchanged.
• Non-clean action types (commit, deltacommit, replacecommit, rollback, savepoint, compaction) untouched.
• Semantic trade-off when trim is enabled: archived clean instants no longer carry per-file replay detail. Downstream verified there are no restore/DR paths in this deployment that read this from the archived timeline.

[snyk-io]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.