{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreibicvy36shto53aorsn2kovrdkcmvza7lwm5z7whk3ci73mgajar4",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mnykpvhz7ly2"
  },
  "path": "/t/datasets-streaming-bottlenecks-storage-client-memory-gc-pressure-vs-sharding-cache-request-amplification/176485#post_3",
  "publishedAt": "2026-06-11T04:32:49.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "@John6666"
  ],
  "textContent": "Thanks for your dedicate reply! @John6666 this is a better map of the problem than my original post, and I’ll take the framing: the useful thing is a topology-aware data plane, not a faster downloader. Your reply helped me narrow scope considerably, so let me be precise about where I’ve landed (disclosure: I’m building a Rust storage substrate in this space, single-node prototype, so judge accordingly).\n\nMeasured against your six-item minimum integration, I’d score a storage-layer substrate like mine 0/6 today, and more importantly, I now think items 1, 3, and 4 (shard planning, bounded prefetch, DataLoader interaction) are _dataloader-layer_ problems that MosaicML Streaming and WebDataset already solve well on top of dumb storage. I don’t think a filesystem should compete there. Item 2 (node-local NVMe cache) is shipped today by JuiceFS/Alluxio/Mountpoint-S3, so the bar is parity, not novelty.\n\nWhat I think is genuinely underserved is the _storage-side correctness_ slice your reply touches but doesn’t dwell on:\n\n  1. **Transactional artifact lifecycle.** Checkpoints written through POSIX/NFS/FUSE can be observed half-written; raw S3 gives you atomic PUTs but no atomic rename-replace of a `latest` pointer with GC of replaced generations. A publish primitive like stage bytes to object storage, then commit name+attrs+manifest in one metadata transaction, with atomic promote and snapshot-aware cleanup is write-path, so it has no ingestion barrier, and none of the dataloader-layer tools touch it.\n  2. **Frozen dataset views for long runs.** An O(1) snapshot pin over an immutable, generation-keyed namespace gives every rank an identical, deterministic listing for the whole run, and fails loudly (rather than silently reading new bytes) if a file is replaced mid-epoch. This only matters if datasets actually mutate during training, which is one of my questions below.\n\n\n\nAnd to be explicit about the two hard problems your reply implies for anything storage-side: existing Hub/S3 datasets would have to be _ingested_ into an owned layout (so Hub/Xet “support” degenerates into exactly the ingest pipeline you warned about), and a FUSE mount without readahead exhibits the exact range-amplification anti-pattern you flagged, I’d rather name that myself than have someone discover it.\n\nThree scope-narrowing questions, since you’ve clearly seen more of these failure modes than I have:\n\n  1. In the painful cases you’ve seen, are datasets **frozen for the duration of a run** , or actively curated/refreshed mid-training? (If frozen-by-convention is near-universal, snapshot/generation semantics lose most of their value and immutable shards + a static index already suffice.)\n  2. Is **half-written / wrongly-promoted checkpoint state** a pain people actually hit in practice (especially around `latest` promotion and cleanup of old checkpoints), or is “write to tmp, rename, hope” good enough on the filesystems people use?\n  3. For the diagnostics list you gave: which of those do you think belong to the **storage layer** (per-mount request counts, bytes, cache hit/miss) vs. the dataloader (skipped samples, per-rank throughput)? I want to avoid building counters nobody can consume.\n\n\n\nIf the answers point where I suspect, my minimum useful integration is much smaller than my original post implied: an fsspec/Python binding over a read-plan API (exact block-range reads with request/cache counters), a byte-budgeted local cache, and the transactional checkpoint path( with shard planning explicitly ceded to Streaming/WebDataset on top).\n\nEnjoy your weekend ahead!",
  "title": "Datasets streaming bottlenecks: storage client memory/GC pressure vs sharding/cache/request amplification?"
}