{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreigtw5rnjvcwcqdn35sn3tayzkgd6soxvbloqzh2gqdm4622qcrcla",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mnm5xhihrxr2"
  },
  "path": "/t/why-is-uploading-to-bucket-so-slow/176575#post_2",
  "publishedAt": "2026-06-06T06:56:57.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "Storage Buckets",
    "Hugging Face Hub Python Library: Buckets",
    "HF_XET_HIGH_PERFORMANCE",
    "huggingface_hub environment variables",
    "HF_HUB_DISABLE_XET",
    "Migrating the Hub from Git LFS to Xet",
    "Xet is on the Hub",
    "Rearchitecting Hugging Face Uploads and Downloads",
    "huggingface/xet-core",
    "Upload speeds extremely slow / stalling since April 1st",
    "xet-core issue: raise default HF_XET_CLIENT_READ_TIMEOUT",
    "datasets issue involving cas::upload_xorb",
    "xet-core releases",
    "Azure AzCopy issue with pending/stuck upload-like behavior",
    "Google Cloud Storage Python client timeout behavior on slow uploads",
    "rclone / MinIO over TLS stall-like behavior"
  ],
  "textContent": "Hmm… That speed drop is so extreme that I would not treat it as a normal cap or a generic “server/network is slow” situation:\n\n* * *\n\n## Short answer\n\nFor an **87.5 MB** file, **39 KB/s** is unusually slow.\n\nThe first thing I would clarify is the upload method:\n\n> Was this uploaded through the **web UI / browser** , or through the **CLI**?\n\nThat matters because a browser Bucket upload, `hf buckets cp`, and a normal `hf upload` may not use exactly the same transfer path.\n\nIf you prefer to stay in the **web UI only** , I would try the browser-side workarounds below first. If those do not help, then the CLI tests are the best next step because they give clearer logs and separate the browser path from the lower-level transfer path.\n\n## GUI-only things to try first\n\nIf you uploaded through the web UI, I would try these before assuming the whole service is slow.\n\nTry this in the browser/UI | Why it helps\n---|---\nRetry in a fresh private/incognito window | Removes extension/session/cached-state effects\nTry a different browser | Separates browser-specific upload behavior\nDisable browser extensions temporarily | Ad blockers, privacy tools, script blockers, antivirus extensions, and download/upload helpers can interfere\nTurn off VPN/proxy if possible | Checks whether the route to HF transfer endpoints is bad\nTry a different network, such as phone tethering | Quickly separates local network/ISP route from HF-side behavior\nRe-login to Hugging Face and reopen the Bucket page | Clears stale auth/session state\nTry renaming the file to a simple ASCII filename | Avoids edge cases around path/filename handling\nTry uploading a same-size dummy file through the UI | Checks whether the problem is file-specific or transfer-path-specific\nTry uploading at a different time | Useful if the problem is temporary route or backend degradation\nWatch the browser DevTools Network tab | HTTP errors, retries, stalled requests, 401/403/5xx, or very long pending requests are useful clues\n\nA same-size dummy file can be created without changing the original file:\n\n\n    python - <<'PY'\n    from pathlib import Path\n    import os\n\n    Path(\"test-90mb.bin\").write_bytes(os.urandom(90 * 1024 * 1024))\n    PY\n\n\nIf you do not want to use CLI at all, just create any roughly 90 MB test file locally and try uploading it through the same web UI.\n\nInterpretation:\n\nBrowser-only result | What it suggests\n---|---\nIncognito/different browser is much faster | Browser session, extension, or local browser state\nDifferent network is much faster | Route, VPN, proxy, ISP, or corporate network issue\nSame-size dummy file is also ~39 KB/s | Not specific to `.pkl`; transfer path is more likely\nSame-size dummy file is fast, original file is slow | File-specific or local inspection behavior becomes more plausible\nDevTools shows repeated retries / 5xx / long pending requests | Useful evidence for HF maintainers\nDevTools shows 401/403 | Auth/session/permission issue\nDevTools shows no real network activity but UI says uploading | Browser/UI progress reporting may be misleading\n\n## Best practical workaround: use the Bucket CLI as a baseline\n\nIf GUI-only attempts do not help, the most useful next test is the Bucket CLI:\n\n\n    hf buckets cp ./model.pkl hf://buckets/<namespace>/<bucket>/model.pkl\n\n\nThis is not just a workaround. It is also a diagnostic split:\n\nResult | What it suggests\n---|---\nWeb UI is slow, `hf buckets cp` is fast | Browser/UI/session path is likely the problem\nWeb UI and `hf buckets cp` are both slow | Lower-level transfer path, route, Xet/CAS, proxy/VPN, or client path\nCLI gives a clear error | Much more actionable than a browser progress bar\n\nDocs:\n\n  * Storage Buckets\n  * Hugging Face Hub Python Library: Buckets\n\n\n\n## If you already used CLI, try Xet high-performance mode\n\n\n    HF_XET_HIGH_PERFORMANCE=1 \\\n    hf buckets cp ./model.pkl hf://buckets/<namespace>/<bucket>/model.pkl\n\n\nFor a normal Hub repo upload instead of a Bucket upload:\n\n\n    HF_XET_HIGH_PERFORMANCE=1 \\\n    hf upload <namespace>/<repo> ./model.pkl model.pkl\n\n\nThe current `huggingface_hub` docs describe `HF_XET_HIGH_PERFORMANCE=1` as the high-performance mode for Xet transfers.\n\nDocs:\n\n  * HF_XET_HIGH_PERFORMANCE\n  * huggingface_hub environment variables\n\n\n\n## If you are uploading to a normal Hub repo, try disabling Xet as a diagnostic\n\n\n    HF_HUB_DISABLE_XET=1 \\\n    hf upload <namespace>/<repo> ./model.pkl model.pkl\n\n\nI would treat this as a **diagnostic** , not necessarily the best long-term mode.\n\nResult | What it suggests\n---|---\n`HF_HUB_DISABLE_XET=1` is much faster | Xet/CAS transfer path is suspicious\n`HF_HUB_DISABLE_XET=1` is slower but completes | Xet path may be problematic, but fallback is not optimized\nNo change | Browser path, route, proxy/VPN, auth, or something else may be the issue\n\nDocs:\n\n  * HF_HUB_DISABLE_XET\n\n\n\nImportant: if you are testing through Python code, set environment variables **before** importing `huggingface_hub`, `datasets`, `transformers`, or anything that may import them.\n\n## First split: which upload path is slow?\n\nIf you used… | Try next… | Why\n---|---|---\nWeb UI / browser upload | GUI-only checks, then `hf buckets cp` | Separates browser path from CLI path\n`hf buckets cp` already | `HF_XET_HIGH_PERFORMANCE=1 hf buckets cp ...` | Checks whether Xet high-performance mode helps\n`hf upload` to a repo | `HF_XET_HIGH_PERFORMANCE=1` and `HF_HUB_DISABLE_XET=1` variants | Checks whether Xet/CAS is involved\nUnsure | Start by identifying whether it was web UI or CLI | The next useful test depends on that\n\n## Why I do not think this is a normal cap\n\nA rough scale check:\n\nSpeed | Approx. time for 87.5 MB\n---|---\n39 KB/s | ~37 minutes\n100 KB/s | ~15 minutes\n1 MB/s | ~1.5 minutes\n8 MB/s | ~11 seconds\n\nSo this is not just “a bit slow”. If your usual upload speed is around 8–9 MB/s, then 39 KB/s is a very large drop.\n\nA normal bandwidth cap, rate limit, or temporary server slowdown would not be my first guess. I would instead suspect that the upload is technically still alive, but the transfer stack has entered a bad low-throughput / retry / backoff state.\n\n## Why the transfer path matters\n\nModern Hugging Face uploads are not always just one simple HTTP upload.\n\nHugging Face has been moving Hub storage from Git LFS-era storage toward Xet-backed storage:\n\n  * Migrating the Hub from Git LFS to Xet\n  * Xet is on the Hub\n  * Rearchitecting Hugging Face Uploads and Downloads\n  * huggingface/xet-core\n\n\n\nIn simple terms, the path may look more like this:\n\n\n    your file\n      -> browser or hf CLI\n      -> huggingface_hub\n      -> hf-xet\n      -> chunking / hashing / local transfer work\n      -> CAS / xorb transfer endpoint\n      -> backend object storage\n      -> final Hub or Bucket metadata update\n\n\nSo “uploading to a bucket is slow” can mean several different things.\n\nLayer | Possible issue\n---|---\nBrowser upload | Web UI, session, extensions, progress display, signed upload path\nCLI path | Local client, auth, version, config\nXet path | Transfer mode, cache, concurrency, timeout\nCAS/xorb path | Transfer endpoint issue\nNetwork route | Bad route to the HF transfer endpoint\nProxy/VPN/corporate network | TLS inspection or traffic shaping\nBackend storage | Possible, but not the first thing I would assume\n\n## Similar failure class\n\nThis exact case may be new, but the general failure shape is not new.\n\nThere are recent Hugging Face / Xet-era reports where upload or transfer performance becomes extremely slow or stalls:\n\n  * Upload speeds extremely slow / stalling since April 1st\n  * xet-core issue: raise default HF_XET_CLIENT_READ_TIMEOUT\n  * datasets issue involving cas::upload_xorb\n  * xet-core releases\n\n\n\nThe point is not that your case is definitely one of those exact issues. The point is that **Xet/CAS/xorb transfer behavior is a real layer to test** , especially when the symptom is “very slow but not cleanly failed”.\n\nThere are also older object-storage-client issues outside Hugging Face with similar “still alive but barely progressing” behavior:\n\n  * Azure AzCopy issue with pending/stuck upload-like behavior\n  * Google Cloud Storage Python client timeout behavior on slow uploads\n  * rclone / MinIO over TLS stall-like behavior\n\n\n\nSo I would not think of this as simply “the internet is slow”. I would think of it as a transfer-path issue until proven otherwise.\n\n## Minimal info to post back\n\nIf you want others or HF maintainers to help, I would post these results:\n\n\n    Upload method:\n    - Web UI / browser Bucket upload?\n    - hf buckets cp?\n    - hf upload?\n    - something else?\n\n    GUI-only checks:\n    - Incognito/private window result\n    - Different browser result\n    - Extensions disabled result\n    - Different network result\n    - Same-size test file result\n    - Browser DevTools Network errors, if any\n\n    Environment, if CLI is used:\n    - OS\n    - hf --version\n    - output of hf env\n    - huggingface_hub version\n    - hf_xet version\n    - logged in or anonymous?\n    - VPN/proxy/corporate network?\n\n    CLI tests, if possible:\n    - hf buckets cp speed\n    - HF_XET_HIGH_PERFORMANCE=1 result\n    - HF_HUB_DISABLE_XET=1 result, if using normal Hub upload\n    - same-size test-90mb.bin result\n\n    Logs:\n    - any CAS / xorb / hf_xet / timeout / retry / 401 / 403 / 5xx messages\n\n\nYou can collect CLI version info with:\n\n\n    hf --version\n    hf env\n    hf auth whoami\n\n    python - <<'PY'\n    import os\n    import platform\n\n    print(\"python:\", platform.python_version())\n    print(\"platform:\", platform.platform())\n    print(\"HF_TOKEN present:\", bool(os.environ.get(\"HF_TOKEN\")))\n\n    try:\n        import huggingface_hub\n        print(\"huggingface_hub:\", huggingface_hub.__version__)\n    except Exception as e:\n        print(\"huggingface_hub unavailable:\", repr(e))\n\n    try:\n        import hf_xet\n        print(\"hf_xet:\", getattr(hf_xet, \"__version__\", \"unknown\"))\n    except Exception as e:\n        print(\"hf_xet unavailable:\", repr(e))\n    PY\n\n\n## My practical guess\n\nMy practical guess is:\n\n\n    not a normal bandwidth cap\n    not generic \"the server is slow\"\n    probably an unhealthy upload transfer path\n    possibly browser Bucket upload, Xet/CAS/xorb, or the route to those endpoints\n\n\nIf you want to stay GUI-only first, I would try:\n\n\n    private window\n    different browser\n    extensions disabled\n    VPN/proxy off\n    different network\n    same-size dummy file\n    browser DevTools Network tab\n\n\nIf those do not change anything, then the most useful next baseline is:\n\n\n    hf buckets cp ./model.pkl hf://buckets/<namespace>/<bucket>/model.pkl\n\n\nAnd if CLI is already being used, I would compare:\n\n\n    HF_XET_HIGH_PERFORMANCE=1 \\\n    hf buckets cp ./model.pkl hf://buckets/<namespace>/<bucket>/model.pkl\n\n\nThose comparisons should quickly tell whether this is a browser-only issue, a Bucket-specific issue, or a lower-level Xet/CAS transfer-path issue.",
  "title": "Why is uploading to bucket so slow?"
}