Datasets streaming bottlenecks: storage client memory/GC pressure vs sharding/cache/request amplification?

Thanks for your dedicate reply! @John6666 this is a better map of the problem than my original post, and I’ll take the framing: the useful thing is a topology-aware data plane, not a faster downloader. Your reply helped me narrow scope considerably, so let me be precise about where I’ve landed (disclosure: I’m building a Rust storage substrate in this space, single-node prototype, so judge accordingly).

Measured against your six-item minimum integration, I’d score a storage-layer substrate like mine 0/6 today, and more importantly, I now think items 1, 3, and 4 (shard planning, bounded prefetch, DataLoader interaction) are dataloader-layer problems that MosaicML Streaming and WebDataset already solve well on top of dumb storage. I don’t think a filesystem should compete there. Item 2 (node-local NVMe cache) is shipped today by JuiceFS/Alluxio/Mountpoint-S3, so the bar is parity, not novelty.

What I think is genuinely underserved is the storage-side correctness slice your reply touches but doesn’t dwell on:

1. Transactional artifact lifecycle. Checkpoints written through POSIX/NFS/FUSE can be observed half-written; raw S3 gives you atomic PUTs but no atomic rename-replace of a latest pointer with GC of replaced generations. A publish primitive like stage bytes to object storage, then commit name+attrs+manifest in one metadata transaction, with atomic promote and snapshot-aware cleanup is write-path, so it has no ingestion barrier, and none of the dataloader-layer tools touch it.
2. Frozen dataset views for long runs. An O(1) snapshot pin over an immutable, generation-keyed namespace gives every rank an identical, deterministic listing for the whole run, and fails loudly (rather than silently reading new bytes) if a file is replaced mid-epoch. This only matters if datasets actually mutate during training, which is one of my questions below.

And to be explicit about the two hard problems your reply implies for anything storage-side: existing Hub/S3 datasets would have to be ingested into an owned layout (so Hub/Xet “support” degenerates into exactly the ingest pipeline you warned about), and a FUSE mount without readahead exhibits the exact range-amplification anti-pattern you flagged, I’d rather name that myself than have someone discover it.

Three scope-narrowing questions, since you’ve clearly seen more of these failure modes than I have:

1. In the painful cases you’ve seen, are datasets frozen for the duration of a run , or actively curated/refreshed mid-training? (If frozen-by-convention is near-universal, snapshot/generation semantics lose most of their value and immutable shards + a static index already suffice.)
2. Is half-written / wrongly-promoted checkpoint state a pain people actually hit in practice (especially around latest promotion and cleanup of old checkpoints), or is “write to tmp, rename, hope” good enough on the filesystems people use?
3. For the diagnostics list you gave: which of those do you think belong to the storage layer (per-mount request counts, bytes, cache hit/miss) vs. the dataloader (skipped samples, per-rank throughput)? I want to avoid building counters nobody can consume.

If the answers point where I suspect, my minimum useful integration is much smaller than my original post implied: an fsspec/Python binding over a read-plan API (exact block-range reads with request/cache counters), a byte-budgeted local cache, and the transactional checkpoint path( with shard planning explicitly ceded to Streaming/WebDataset on top).

Enjoy your weekend ahead!