Skip to content

Defer shard closing to different process - #450

Merged
kralka merged 2 commits into
google:mainfrom
kralka:simplify_writing
Nov 5, 2026
Merged

kralka merged 2 commits into
google:mainfrom
kralka:simplify_writing

Conversation

@kralka

@kralka kralka commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Allow full compression and file writing in another process. Simplify the writing logic.

Part of #444

Allow full compression and file writing in another process. Simplify the
writing logic.

Part of google#444
@kralka
kralka marked this pull request as ready for review November 4, 2026 17:20
@kralka
kralka requested a review from jmichelp November 4, 2026 17:20
@coveralls

coveralls commented Nov 4, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 41998013006

Coverage increased (+0.1%) to 90.687%

Details

  • Coverage increased (+0.1%) from the base build.
  • Patch coverage: 12 uncovered changes across 3 files (315 of 327 lines covered, 96.33%).
  • 11 coverage regressions across 4 files.

Uncovered Changes

File Changed Covered %
src/sedpack/io/merge_shard_infos.py 43 36 83.72%
src/sedpack/io/dataset_filler.py 82 78 95.12%
tests/io/test_write_multiprocessing.py 141 140 99.29%
Total (6 files) 327 315 96.33%

Coverage Regressions

11 previously-covered lines in 4 files lost coverage.

File Lines Losing Coverage Coverage
src/sedpack/io/shard/shard.py 5 73.08%
src/sedpack/io/tfrec/tfdata.py 4 76.56%
src/sedpack/io/dataset_filler.py 1 95.45%
src/sedpack/io/merge_shard_infos.py 1 85.96%

Coverage Stats

Coverage Status
Relevant Lines: 3962
Covered Lines: 3593
Line Coverage: 90.69%
Coverage Strength: 0.91 hits per line

💛 - Coveralls

@kralka
kralka added this pull request to the merge queue Nov 5, 2026
Merged via the queue into google:main with commit 5c461fe Nov 5, 2026
61 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants