Repository navigation
Implement max disk utilization check - #452
Conversation
Coverage Report for CI Build 42595082615Coverage increased (+0.2%) to 90.934%Details
Uncovered Changes
Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
| DiskFullError: If a new shard needs to be opened and disk | ||
| utilization is at or above `max_disk_utilization`. When used inside | ||
| a `DatasetFiller` context manager, all shards opened so far are | ||
| still closed and saved to the dataset on context exit (`__exit__`) |
There was a problem hiding this comment.
IIRC now shards live in RAM and are written to disk when closing them. If the disk is already full, current shards won't succeed at being written to the disk unless you take some margin to serialize what's in RAM and not yet on disk, correct?
There was a problem hiding this comment.
Yes this might be a misleading docstring. Intended use-case is max_disk_utilization=0.8 plus whatever the currently in RAM shards are (should be under 1GB).
The longer version:
The user controls examples_per_shard aiming for decent size shards (10MB to 100MB rule of thumb). The fun part is that the user does not really know how much compression there is (for Magika use-case different filetypes have vastly different compression ratios). The user also controlls concurrency (arg of DatasetFiller.__init__) which tells how many in RAM shards can be open at once -- thus limiting the saving at the end.
I am also not 100% happy with this design but it is enough for automated benchmarking and at the same time not overly complicated.
Alternatives considered:
- Keeping statistics of closed shards
- Dropping data (not an option)
Part of #444