Skip to content

Implement max disk utilization check - #452

Merged
kralka merged 2 commits into
google:mainfrom
kralka:max_disk_utilization
Nov 9, 2026
Merged

kralka merged 2 commits into
google:mainfrom
kralka:max_disk_utilization

Conversation

@kralka

@kralka kralka commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #444

@coveralls

coveralls commented Nov 5, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 42595082615

Coverage increased (+0.2%) to 90.934%

Details

  • Coverage increased (+0.2%) from the base build.
  • Patch coverage: 1 uncovered change across 1 file (125 of 126 lines covered, 99.21%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
src/sedpack/io/dataset_filler.py 22 21 95.45%
Total (5 files) 126 125 99.21%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 4081
Covered Lines: 3711
Line Coverage: 90.93%
Coverage Strength: 0.91 hits per line

💛 - Coveralls

@kralka
kralka marked this pull request as ready for review November 8, 2026 09:45
@kralka
kralka requested a review from jmichelp November 8, 2026 09:45
DiskFullError: If a new shard needs to be opened and disk
utilization is at or above `max_disk_utilization`. When used inside
a `DatasetFiller` context manager, all shards opened so far are
still closed and saved to the dataset on context exit (`__exit__`)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC now shards live in RAM and are written to disk when closing them. If the disk is already full, current shards won't succeed at being written to the disk unless you take some margin to serialize what's in RAM and not yet on disk, correct?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this might be a misleading docstring. Intended use-case is max_disk_utilization=0.8 plus whatever the currently in RAM shards are (should be under 1GB).

The longer version:

The user controls examples_per_shard aiming for decent size shards (10MB to 100MB rule of thumb). The fun part is that the user does not really know how much compression there is (for Magika use-case different filetypes have vastly different compression ratios). The user also controlls concurrency (arg of DatasetFiller.__init__) which tells how many in RAM shards can be open at once -- thus limiting the saving at the end.

I am also not 100% happy with this design but it is enough for automated benchmarking and at the same time not overly complicated.

Alternatives considered:

  • Keeping statistics of closed shards
  • Dropping data (not an option)

@kralka
kralka requested a review from jmichelp November 9, 2026 09:20
@kralka
kralka enabled auto-merge November 9, 2026 09:48
@kralka
kralka added this pull request to the merge queue Nov 9, 2026
Merged via the queue into google:main with commit 24af758 Nov 9, 2026
110 of 113 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants