Skip to content
#

memmap

Here are 9 public repositories matching this topic...

Language: All
Filter by language

Pretraining data for language models: tokenize a text corpus once into resumable, sha256-verified binary shards, move it to a GPU server, and read random training windows via numpy.memmap. Weighted source mixing, PyTorch DataLoader adapters, CLI. Pretraining only - not for SFT or chat data.

  • Updated Oct 3, 2026
  • Python

Add this topic to your repo

To associate your repository with the memmap topic, visit your repo's landing page and select "manage topics."

Learn more