{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreif5k72a7kmttt2ml6yp647n4sioag6c2to2zvvj23aunpgb4uj72u",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3moecktz3cok2"
  },
  "path": "/t/downloading-datasets/176816#post_3",
  "publishedAt": "2026-06-15T21:49:32.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "garrying/VMD-D",
    "garrying/VMD-D/tree/main",
    "huggingface_hub issue #4058: Xet downloads barely report progress",
    "hf download / file download docs",
    "Datasets streaming docs",
    "datasets issue #8169: Streaming dataset hangs consistently",
    "huggingface_hub issue #4178: Streaming dataset hangs consistently",
    "Datasets loading docs",
    "huggingface_hub environment variables",
    "datasets issue #8129: load_dataset() hangs when hf_xet is enabled",
    "xet-core issue #407: Cannot download file from XET hosted repo using CLI",
    "xet-core issue #789: hf_xet downloads stalling",
    "huggingface_hub issue #3868: HTTP download fails for files >50GB",
    "datasets #8129",
    "Datasets cache management"
  ],
  "textContent": "Yeah. Waiting can definitely be the right answer here, especially if the real transfer has not started yet. But when a download appears to be stuck, there are a few other layers worth separating besides “slow network” or a temporary network / Hugging Face-side issue:\n\n* * *\n\nFrom the screenshot alone, I would not immediately assume this is an authentication problem. It may be, but a stuck-looking progress bar can also come from the Hub transfer layer, Xet-backed file transfer, `datasets` materialization/cache, local disk/cache state, or simply progress reporting that is too quiet.\n\nFor this particular dataset, I would debug it in layers.\n\n`garrying/VMD-D` is not a tiny text-only dataset. It is an image/mask dataset stored as Parquet / optimized Parquet, with train/test splits and around 15k rows total:\n\n  * dataset page: garrying/VMD-D\n  * files tab: garrying/VMD-D/tree/main\n\n\n\nSo the first materialized load can legitimately take some time. Still, if it stays at `0%` for a long time with no network or disk activity, I would test these paths separately.\n\n## 1. First, check whether it is only the visible progress bar\n\nA progress bar that looks frozen does not always mean the transfer is actually dead. There are recent Xet-related reports where large downloads barely updated the `tqdm` progress bar, making the process look hung even though bytes were still moving:\n\n  * huggingface_hub issue #4058: Xet downloads barely report progress\n\n\n\nSo before changing code, I would check:\n\n  * Is network traffic still active?\n  * Is disk usage increasing?\n  * Is CPU active?\n  * Does the cache directory size keep changing?\n  * Does it eventually jump from `0%` to a later percentage?\n\n\n\nIf yes, waiting may indeed be the right answer.\n\nIf there is no visible network/disk activity for several minutes, then I would move to layer-by-layer tests.\n\n## 2. Check versions and auth state\n\nThis is low-cost and helps others understand what path you are using.\n\n\n    hf auth whoami\n    hf --version\n\n\nThen check the Python-side versions:\n\n\n    python - <<'PY'\n    import platform\n    print(\"python\", platform.python_version())\n\n    try:\n        import datasets\n        print(\"datasets\", datasets.__version__)\n    except Exception as e:\n        print(\"datasets import failed:\", repr(e))\n\n    try:\n        import huggingface_hub\n        print(\"huggingface_hub\", huggingface_hub.__version__)\n    except Exception as e:\n        print(\"huggingface_hub import failed:\", repr(e))\n\n    try:\n        import hf_xet\n        print(\"hf_xet\", getattr(hf_xet, \"__version__\", \"unknown\"))\n    except Exception as e:\n        print(\"hf_xet unavailable:\", repr(e))\n    PY\n\n\nIf `hf auth whoami` fails, fix authentication first. But if authentication is fine, I would not keep focusing only on the token. The symptom can still be caused by transfer/cache/materialization layers.\n\n## 3. Test the Hub transfer layer directly\n\nEven if your final code uses `load_dataset()`, testing the Hub download path directly is useful because it removes some of the `datasets` preparation/cache logic from the picture.\n\nStart with a dry run:\n\n\n    hf download garrying/VMD-D --repo-type dataset --dry-run\n\n\nThis should tell you which files would be downloaded and how much is already cached.\n\nThen test the actual Hub transfer:\n\n\n    hf download garrying/VMD-D --repo-type dataset\n\n\nInterpretation:\n\nResult | What it suggests\n---|---\n`hf download --dry-run` works | Repo metadata and file listing are probably reachable.\n`hf download` works | The Hub transfer path is probably okay. Then the issue may be more specific to `datasets.load_dataset()` materialization/cache.\n`hf download` also hangs | The problem may be in Hub transfer, Xet, cache, disk, route/proxy/VPN, or runtime environment.\n`hf auth whoami` fails | Auth/token should be fixed first.\nSmall files download but large Parquet shards hang | That points more toward transfer/cache/Xet/large-file behavior than a simple token issue.\n\nOfficial docs for this path:\n\n  * hf download / file download docs\n\n\n\n## 4. Test `datasets` streaming separately\n\nThen test whether `datasets` can at least open and iterate the dataset without fully materializing it locally:\n\n\n    from datasets import load_dataset\n\n    ds = load_dataset(\"garrying/VMD-D\", split=\"train\", streaming=True)\n    print(next(iter(ds)))\n\n\n`streaming=True` is useful because it lets you work with a dataset without downloading/materializing the whole dataset first:\n\n  * Datasets streaming docs\n\n\n\nInterpretation:\n\nResult | What it suggests\n---|---\nStreaming works | Basic repo access, split resolution, and first-sample reading probably work. If normal `load_dataset()` hangs, the issue may be in download/materialization/cache.\nStreaming also hangs | It may still be a Hub/client/runtime issue, not necessarily auth. Streaming itself can have hang modes too.\nStreaming returns a sample quickly but normal load hangs | This is a strong signal to inspect materialized loading, cache, Parquet handling, Xet, and disk.\n\nImportant caveat: streaming is a diagnostic, not a perfect proof. There are also recent reports where `streaming=True` itself can hang:\n\n  * datasets issue #8169: Streaming dataset hangs consistently\n  * huggingface_hub issue #4178: Streaming dataset hangs consistently\n\n\n\nSo I would use streaming as a useful split, not as a final verdict.\n\n## 5. Test the normal materialized path with an explicit split\n\nAfter that, test the normal path, but with an explicit split:\n\n\n    from datasets import load_dataset\n\n    ds = load_dataset(\"garrying/VMD-D\", split=\"train\")\n    print(ds)\n\n\nSpecifying `split=\"train\"` makes the test easier to reason about than loading the whole `DatasetDict` first.\n\nThe normal `load_dataset()` path may involve several steps:\n\n  * resolving the dataset repository\n  * discovering the data files\n  * downloading files through `huggingface_hub`\n  * reading Parquet files\n  * preparing/cacheing an Arrow-backed `Dataset`\n  * returning the selected split\n\n\n\nOfficial loading docs:\n\n  * Datasets loading docs\n\n\n\nThis is why I would not debug the screenshot as if it represented only one simple HTTP request.\n\n## 6. Try isolating the Xet path\n\nBecause the dataset files are Xet-backed, and because recent Hub transfers now commonly use `hf_xet`, I would test once with Xet disabled.\n\nFor CLI:\n\n\n    HF_HUB_DISABLE_XET=1 hf download garrying/VMD-D --repo-type dataset\n\n\nFor Python, set it before importing `datasets` or `huggingface_hub`:\n\n\n    HF_HUB_DISABLE_XET=1 python - <<'PY'\n    from datasets import load_dataset\n\n    ds = load_dataset(\"garrying/VMD-D\", split=\"train\")\n    print(ds)\n    PY\n\n\nOr inside a notebook, restart the runtime/kernel, then run this before importing `datasets`:\n\n\n    import os\n    os.environ[\"HF_HUB_DISABLE_XET\"] = \"1\"\n\n    from datasets import load_dataset\n\n    ds = load_dataset(\"garrying/VMD-D\", split=\"train\")\n    print(ds)\n\n\nI would treat this as a diagnostic switch, not necessarily a permanent fix.\n\nResult | Interpretation\n---|---\nWorks with `HF_HUB_DISABLE_XET=1` | The Xet transfer path is probably involved.\nStill hangs with `HF_HUB_DISABLE_XET=1` | The issue may be elsewhere: network route, proxy/VPN, cache, disk, timeout, runtime, or another bug.\nWorks without disabling Xet after upgrading `hf_xet` | The issue may have been fixed in a newer transfer client.\nXet-disabled HTTP fallback fails for very large files | That does not necessarily mean Xet was innocent; HTTP fallback is not always a universal replacement for Xet.\n\nRelated docs and issues:\n\n  * huggingface_hub environment variables\n  * datasets issue #8129: load_dataset() hangs when hf_xet is enabled\n  * xet-core issue #407: Cannot download file from XET hosted repo using CLI\n  * xet-core issue #789: hf_xet downloads stalling\n  * huggingface_hub issue #3868: HTTP download fails for files >50GB\n\n\n\nThe closest diagnostic pattern I found is datasets #8129: `load_dataset()` hangs when `hf_xet` is enabled, while `streaming=True` and direct Hub access work, and `HF_HUB_DISABLE_XET=1` changes the behavior. I am not saying this dataset is definitely hitting the same bug, but the split is close enough that the same diagnostic structure is useful.\n\n## 7. Update the transfer-related packages\n\nIf this is a fresh environment or notebook, I would also try updating the relevant packages:\n\n\n    pip install -U datasets huggingface_hub hf_xet\n\n\nThen restart the Python process/kernel and retry.\n\nThis matters because `huggingface_hub`, `datasets`, and `hf_xet` can interact in the download path. Updating only one of them may leave you with a mixed stack.\n\nIf you are in a notebook environment such as Colab or Kaggle, restarting the runtime after upgrade is important. Otherwise, Python may keep using already-imported modules.\n\n## 8. Check cache and disk state\n\nAlso check free disk space and cache locations. `datasets` can involve more than one cache:\n\n  * Hub cache: files downloaded from the Hub\n  * Datasets cache: processed Arrow-formatted datasets\n  * Xet cache: Xet chunks/ranges\n\n\n\nUseful docs:\n\n  * Datasets cache management\n  * huggingface_hub environment variables\n\n\n\nUseful commands:\n\n\n    df -h\n    hf cache ls\n\n\nIf you want to test with a separate clean cache, use a temporary location:\n\n\n    mkdir -p /tmp/hf-test-cache\n    HF_HOME=/tmp/hf-test-cache python - <<'PY'\n    from datasets import load_dataset\n\n    ds = load_dataset(\"garrying/VMD-D\", split=\"train\")\n    print(ds)\n    PY\n\n\nIf that works, your previous cache may be involved. If it still hangs, the issue is probably not only a stale cache entry.\n\nYou can also separate Hub and Datasets cache variables, but for a quick test `HF_HOME` is often simpler because it moves the general Hugging Face cache root.\n\n## 9. Optional: make the test more focused on the first file/split\n\nIf the goal is only to verify access, avoid loading more than needed at first.\n\nFor example, use streaming and take one example:\n\n\n    from datasets import load_dataset\n\n    ds = load_dataset(\"garrying/VMD-D\", split=\"train\", streaming=True)\n    first = next(iter(ds))\n    print(first.keys())\n\n\nOr use a split-level normal load:\n\n\n    from datasets import load_dataset\n\n    train = load_dataset(\"garrying/VMD-D\", split=\"train\")\n    print(train)\n\n\nOnce that works, then move on to the full workflow.\n\n## 10. A compact decision table\n\nHere is how I would read the outcomes:\n\nTest | If it works | If it hangs/fails\n---|---|---\n`hf auth whoami` | Token is probably visible to CLI. | Fix login/token first.\n`hf download ... --dry-run` | Metadata and file list are reachable. | Metadata/access/client/network issue.\n`hf download ...` | Hub transfer path probably works. | Hub transfer / Xet / cache / disk / network route issue.\n`load_dataset(..., streaming=True)` | Basic dataset access and first-sample path probably work. | Still may be Hub/client/runtime; not automatically auth.\n`load_dataset(..., split=\"train\")` | Normal materialized path works. | Materialization/cache/Parquet/Xet/disk path may be involved.\n`HF_HUB_DISABLE_XET=1 ...` | If behavior changes, Xet path is probably involved. | Not only Xet; inspect network/cache/disk/runtime.\nFresh `HF_HOME=/tmp/...` works | Old cache may be involved. | Cache is less likely to be the only cause.\nPackage upgrade fixes it | Version interaction was likely involved. | Continue with transfer/runtime diagnostics.\n\n## 11. If you want to report it upstream\n\nIf it still hangs after the tests above, I would include this information in a follow-up issue or forum reply:\n\n\n    Dataset: garrying/VMD-D\n    Environment: local / Colab / Kaggle / server / etc.\n    OS:\n    Python:\n    datasets:\n    huggingface_hub:\n    hf_xet:\n    Command that hangs:\n    Does `hf download ... --repo-type dataset --dry-run` work?\n    Does `hf download ... --repo-type dataset` work?\n    Does `load_dataset(..., streaming=True)` work?\n    Does `load_dataset(..., split=\"train\")` work?\n    Does `HF_HUB_DISABLE_XET=1` change the behavior?\n    Free disk space:\n    Cache location:\n    Proxy/VPN/corporate network:\n    Last visible progress line:\n    Whether network/disk activity continues while the bar is stuck:\n\n\nThat makes it much easier to distinguish:\n\n  * slow-but-normal first download\n  * misleading `tqdm` progress\n  * Xet-backed transfer stall\n  * Datasets materialization/cache issue\n  * stale/partial cache issue\n  * auth/access issue\n  * route/proxy/VPN/runtime-specific issue\n\n\n\n## Short version\n\nI would try this order:\n\n\n    hf auth whoami\n    hf download garrying/VMD-D --repo-type dataset --dry-run\n    hf download garrying/VMD-D --repo-type dataset\n\n\nThen:\n\n\n    from datasets import load_dataset\n\n    ds = load_dataset(\"garrying/VMD-D\", split=\"train\", streaming=True)\n    print(next(iter(ds)))\n\n\nThen:\n\n\n    from datasets import load_dataset\n\n    ds = load_dataset(\"garrying/VMD-D\", split=\"train\")\n    print(ds)\n\n\nAnd as an Xet isolation test:\n\n\n    HF_HUB_DISABLE_XET=1 hf download garrying/VMD-D --repo-type dataset\n\n\nIf disabling Xet changes the behavior, the transfer path is likely involved. If streaming works but normal `load_dataset()` hangs, the issue may be in materialization/cache/Parquet handling rather than basic access. If everything hangs, I would look at network route, disk/cache, package versions, and runtime environment next.",
  "title": "Downloading datasets"
}