Skip to content

Tune Edge compaction and TsFile defaults - #18566

Merged
JackieTien97 merged 2 commits into
apache:masterfrom
JackieTien97:ty-tune-edge-compaction-defaults
Sep 1, 2026
Merged

Tune Edge compaction and TsFile defaults#18566
JackieTien97 merged 2 commits into
apache:masterfrom
JackieTien97:ty-tune-edge-compaction-defaults

Conversation

@JackieTien97

@JackieTien97 JackieTien97 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • tune Edge-only compaction queue, aligned-series batching, and inner/cross-space candidate limits
  • use a 32 MiB target with eight first-level candidates, based on an Edge-specific 1-device x 100-measurement workload
  • use smaller TsFile chunk and page targets for the constrained Edge profile
  • assert every packaged Edge default in the dedicated integration test

Rationale

This follows the parameter suggestions from the Edge distribution mailing-list discussion, with the file-size policy adjusted using an empirical Edge workload instead of applying every suggested value verbatim.

The Edge process uses a 224 MiB maximum heap. With the current memory ratios, one active aligned MemTable with 100 DOUBLE measurements flushed every 25,000 rows in the probe. Each flush contained 20,300,000 bytes of logical MemTable data.

With deterministic data calibrated to an observed 5.08x compression ratio, each natural flush produced an approximately 4.02 MB TsFile. A 32 MiB compaction target paired with eight candidate files therefore creates an approximately 32 MB stable first-level file.

The inner and cross-space input-size limits remain at 50 MiB. This intentionally prevents two approximately 32 MB files from being compacted again into one approximately 64 MB file for a single-device workload, avoiding an additional write-amplification level while still bounding every task. The aligned-series batch of 2 and the smaller chunk/page targets bound temporary memory during compaction and file writing. All overrides remain isolated to the Edge archive.

Empirical validation

Workload: one aligned device, 100 DOUBLE measurements using GORILLA and LZ4, 400,000 rows and 40,000,000 values in total.

  • observed compression ratio: 5.08x
  • natural flush boundary: 25,000 rows and 2,500,000 values
  • logical data per flush: 20,300,000 bytes
  • natural TsFile size: approximately 4.02 MB
  • each inner compaction selected 8 files, approximately 30 MiB total input
  • estimated memory per compaction task: 3.64 MiB
  • output sizes: 32,115,699 bytes and 32,115,029 bytes
  • task times: 2.86 seconds and 2.84 seconds
  • a later scheduler pass kept both approximately 32 MB files, confirming the 50 MiB cap prevented a 64 MB second-level merge
  • post-compaction count remained 400,000 rows
  • sampled combined Edge process RSS stayed below 425 MiB

Build and integration validation

  • mvn spotless:apply -P with-integration-tests -pl integration-test
  • mvn clean package -DskipTests -pl distribution -am
  • verified all six binary distribution ZIPs are still produced
  • inspected the Edge ZIP and verified all proposed values are packaged
  • mvn verify -DskipUTs -Dit.test=IoTDBEdgeBasicIT -DfailIfNoTests=false -Dfailsafe.failIfNoSpecifiedTests=false -pl integration-test -am -P EdgeIT -P with-integration-tests

@JackieTien97
JackieTien97 merged commit 05a97b9 into apache:master Sep 1, 2026
39 checks passed
@JackieTien97
JackieTien97 deleted the ty-tune-edge-compaction-defaults branch September 1, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant