{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreigpbfeicqvylat3skj6vdakxprvj34tugmrrhw5577eymqw45efzy",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mk4viaje5dd2"
  },
  "path": "/t/hf-zerogpu-space-hangs-no-output-in-the-logs/175410#post_4",
  "publishedAt": "2026-04-23T00:25:20.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "huggingface.co",
    "github.com",
    "docs.pytorch.org",
    "@spaces.GPU"
  ],
  "textContent": "I’ve also done a bit of experimenting here with Zero GPU and `torchscript`. Before I knew it, the default PyTorch version for Zero GPU Spaces had been updated to 2.11.0 (in cases like this, the documentation is often written after the fact, so it doesn’t always match the actual behavior). I suspect that compatibility between this version and `torchscript`—in terms of how the weights actually behave, rather than just in theory—is quite questionable. It seems to work in some cases… but if it can’t use the GPU and falls back to the CPU, it’ll be too slow to fit within the `duration` and will likely time out.\n\nSince I don’t have the actual code, this is purely speculation, but:\n\n* * *\n\nAt this point, the likely causes and the practical solutions are much clearer.\n\n## The short answer\n\nThe most likely cause is **not** a generic ZeroGPU failure.\n\nIt is more likely one of these:\n\n  1. **something specific about your real TorchScript artifact** ,\n  2. **how that artifact is placed on CUDA before callback entry** ,\n  3. **an interaction between that artifact and your older runtime stack** ,\n  4. or a combination of those three.\n\n\n\nThat is the cleanest reading of the pattern now.\n\n## Why this is the right frame\n\nThe pattern you described points away from ordinary inference slowness and toward a **worker-boundary problem** :\n\n  * the UI responds,\n  * the click is registered,\n  * the request enters the ZeroGPU path,\n  * but control never reaches the first line of the decorated function.\n\n\n\nHF’s ZeroGPU docs still matter for the semantics here: `@spaces.GPU` is the hosted ZeroGPU entry mechanism, the decorator is effect-free outside ZeroGPU, and HF explicitly says ZeroGPU can have **limited compatibility** compared with standard GPU Spaces. That means a container can behave correctly locally while still failing at the hosted ZeroGPU worker transition. (huggingface.co)\n\nSo the real question is no longer “why is inference slow?” It is:\n\n> **what is making the hosted ZeroGPU worker unhappy before the callback body starts?**\n\nThat broader runtime-contract lens is still the right one here.\n\n* * *\n\n## Most likely causes\n\n## 1. Your real TorchScript artifact is the top suspect\n\nThis is now the strongest explanation.\n\nAt this point, the observed evidence points away from a blanket “TorchScript never works on ZeroGPU” interpretation and much more toward **something specific about your actual`.pt` file**.\n\nThat “something specific” could be:\n\n  * graph complexity,\n  * operator set,\n  * custom classes or custom ops,\n  * serialization-time assumptions,\n  * export-time environment differences,\n  * or behavior that appears only on the real forward path.\n\n\n\nI am not claiming which one without seeing the artifact. But the evidence now supports “artifact-specific” much more strongly than “platform-wide.”\n\n### Why this matters\n\nThis shifts the debugging target from the platform to the model artifact itself.\n\nThat is a big difference. It means the likely fix is not “tune the queue” or “increase duration.” It is more likely:\n\n  * change how the artifact is loaded,\n  * change where it is placed,\n  * re-export it,\n  * or run it on a newer baseline.\n\n\n\n* * *\n\n## 2. Module-level CUDA placement may be the real trigger\n\nThis is the second most likely cause.\n\nThere is an important difference between:\n\n  * loading a TorchScript model at startup, and\n  * placing that model on CUDA at startup before callback entry.\n\n\n\nThose are not the same thing operationally.\n\nThe symptom you described — failure before the first line inside `@spaces.GPU` — is very consistent with a problem that happens **before** the model’s useful forward path starts. One very plausible way to get that is:\n\n  * model exists at module scope,\n  * model is moved to CUDA too early for the hosted ZeroGPU path,\n  * then worker entry breaks before user code inside the callback begins.\n\n\n\nSo I would now treat this as a central hypothesis:\n\n> **the real problem may be startup-time CUDA placement of the real TorchScript model, not TorchScript loading by itself.**\n\nThat would explain why a normal local process can work and the hosted worker still fails.\n\n* * *\n\n## 3. Your older stack may be amplifying the issue\n\nThis is still a serious suspect.\n\nYour failing stack is older than the current template-style ZeroGPU baseline. That matters because public issue history shows `@spaces.GPU` behavior can be sensitive to Gradio/runtime version changes. There is a Gradio issue where the decorator path itself appears to be involved in model-loading failure behavior on ZeroGPU. (github.com)\n\nSo the older stack is probably not just a neutral background detail. It may be:\n\n  * making the artifact problem easier to trigger,\n  * or exposing a boundary condition that the newer baseline handles better.\n\n\n\nI would now think of your runtime versions as part of the problem surface, not just as static facts.\n\n* * *\n\n## 4. Basic `torch.jit.load(...)` is probably not the main problem anymore\n\nI would move this lower on the list.\n\nPyTorch’s docs describe `torch.jit.load` as the standard way to load a saved ScriptModule, with normal file-based behavior and `map_location` support. That basic API path is not, by itself, the most suspicious thing now.\n\nSo I would separate these two ideas clearly:\n\n  * **basic TorchScript file loading** → probably not the central issue\n  * **your real artifact’s behavior after or around load** → still highly suspicious\n\n\n\nThat distinction matters because it changes the solution strategy.\n\n* * *\n\n## 5. `torch.jit.optimized_execution(False)` is probably not the main fix\n\nThis flag is real and useful, but I no longer think it is central.\n\nThere is real PyTorch issue history around first-call TorchScript optimization overhead, and `torch.jit.optimized_execution(False)` is relevant to that class of problem.\n\nBut your failure pattern is earlier than that:\n\n  * it happens before the useful callback body begins.\n\n\n\nSo my updated read is:\n\n  * this flag may help with runtime cost or first-pass overhead,\n  * but it is probably **not** the reason the hosted callback boundary fails.\n\n\n\nIt is a secondary control, not the main solution.\n\n* * *\n\n## 6. TorchScript’s current ecosystem status raises the risk of edge cases\n\nThis is background rather than the root cause, but it matters.\n\nPyTorch’s current docs mark TorchScript as **deprecated** and recommend `torch.export` going forward. That does not mean your artifact should fail today, but it does mean TorchScript is no longer the most future-facing or highest-priority path in the ecosystem. (docs.pytorch.org)\n\nSo if you are hitting a complex edge case involving:\n\n  * hosted runtime behavior,\n  * worker-boundary timing,\n  * and a real scripted artifact,\n\n\n\nthat is no longer surprising in the way it would have been when TorchScript was the clear primary path.\n\n* * *\n\n## What is probably **not** the cause anymore\n\nThese are now lower-probability primary causes:\n\n### Not the main cause: UI complexity\n\nYou already reduced the UI enough that this should not be first on the list.\n\n### Not the main cause: logging blind spots\n\nYou used flushes and durable writes, and both point to the same boundary.\n\n### Not the main cause: hidden setup inside the callback body\n\nThe callback body never gets control in the failing pattern.\n\n### Not the main cause: generic ZeroGPU cannot run callbacks\n\nThe broader evidence now points away from that.\n\n### Not the main cause: generic TorchScript incompatibility\n\nThe broader evidence now points away from that too.\n\n* * *\n\n## Solutions\n\nNow that the likely causes are narrower, the solutions are much more concrete.\n\n## Solution 1: Use the newer baseline as your reference environment\n\nThis is the most important practical move.\n\nDo not treat the older failing Space as the only truth source anymore.\n\nInstead, treat the **newer/current-style ZeroGPU baseline** as the control environment and compare your real model against that.\n\n### Why this is the right move\n\nBecause it removes a whole class of ambiguity:\n\n  * if the real model fails there too, the artifact becomes the prime suspect;\n  * if the real model works there, the older stack becomes the stronger suspect.\n\n\n\nThat is much more informative than continuing to debug in the older environment alone.\n\n* * *\n\n## Solution 2: Separate CPU-load from CUDA-placement\n\nThis is probably the single most important diagnostic and architectural split now.\n\nYou should think in two stages:\n\n### Stage A: can the real TorchScript artifact be loaded and kept on CPU at startup?\n\nIf not, the artifact itself is the main suspect.\n\n### Stage B: what changes when it is placed on CUDA before callback entry?\n\nIf CPU-load is fine but startup CUDA placement breaks hosted callback entry, then the fix is likely to be about **when and where** you move the model to CUDA.\n\n### Practical consequence\n\nIf startup CUDA placement is the trigger, the likely short-term fix is:\n\n  * keep the model on CPU earlier,\n  * only move/use it inside the ZeroGPU-managed path as needed for debugging or a revised serving design.\n\n\n\nThat is not a performance claim. It is a stability-first debugging move.\n\n* * *\n\n## Solution 3: If the real artifact works on the newer baseline, migrate the original app toward that baseline\n\nIf the real model behaves on the newer setup, then the root problem is probably not the artifact alone.\n\nAt that point, the best solution is:\n\n> **move the original app toward the newer runtime path instead of continuing to preserve the older one.**\n\nThat means:\n\n  * newer Gradio path,\n  * newer `spaces` behavior,\n  * current template-style structure,\n  * and then reintroducing your app logic carefully.\n\n\n\nIn that scenario, trying to preserve the older runtime as the “real” environment just slows you down.\n\n* * *\n\n## Solution 4: If the real artifact still fails on the newer baseline, treat it as an export/serialization problem\n\nIf the real `.pt` artifact fails even in the newer reference setup, then the likely solutions shift toward the artifact itself:\n\n  * re-save or re-export it under a newer PyTorch stack,\n  * simplify or isolate the problematic graph,\n  * identify unusual/custom components,\n  * or consider moving away from TorchScript if it is becoming a long-term maintenance liability.\n\n\n\n### Long-term direction\n\nBecause TorchScript is deprecated, `torch.export` becomes the most natural long-term direction if the scripted artifact turns out to be the recurring source of hosted-runtime pain. (docs.pytorch.org)\n\nThat is not a recommendation to rewrite everything immediately. It is the likely strategic path if the artifact itself turns out to be the core issue.\n\n* * *\n\n## Solution 5: Trust live build/runtime evidence for exact defaults, and docs for semantics\n\nThis is a practical rule rather than a code change, but it matters.\n\nFor exact current defaults, live build/runtime behavior is often more reliable than prose docs that may lag. For high-level semantics — like what `@spaces.GPU` does, how ZeroGPU differs from standard GPU Spaces, and what config keys matter — the docs still matter. (huggingface.co)\n\nThat is the right way to combine the two sources of truth.\n\n* * *\n\n## The clearest overall diagnosis\n\nIf I had to state the diagnosis as plainly as possible:\n\n> Your failure is most likely caused by **your real TorchScript artifact or its startup/device-placement behavior** , with your older runtime stack acting as a likely amplifier.\n\nThat is where I would put my confidence now.\n\n* * *\n\n## The clearest overall solution path\n\nIf I had to state the solution path as plainly as possible:\n\n  1. **use the newer baseline as the reference**\n  2. **test the real model there**\n  3. **split CPU-load from startup CUDA placement**\n  4. **if needed, migrate the old app toward the newer baseline**\n  5. **if needed, re-export / modernize the artifact path**\n\n\n\nThat is the highest-leverage path now.\n\n* * *\n\n## Final ranking\n\n### Most likely causes\n\n  1. **real TorchScript artifact**\n  2. **module-level CUDA placement of that artifact**\n  3. **older runtime stack interaction**\n  4. **artifact-specific runtime/operator behavior**\n  5. **generic ZeroGPU problem** as a distant possibility\n\n\n\n### Most likely solutions\n\n  1. **move the real model into the newer baseline**\n  2. **separate CPU-load from CUDA-placement**\n  3. **migrate toward the newer stack if the real model works there**\n  4. **re-export or modernize the artifact if it still fails there**\n  5. **treat`optimized_execution(False)` as secondary, not primary**\n\n\n\n* * *\n\n## Final takeaway\n\nThe best framing now is **not** :\n\n> “Why does ZeroGPU hang?”\n\nThe better framing is:\n\n> **“Why does my real TorchScript artifact, or its startup/device placement, fail on my older hosted path when a simple TorchScript artifact can work on the current baseline?”**\n\nThat is the actual problem now.\n\nAnd that is a much more solvable problem.\n\nThe high-level rule remains: solve this as a **runtime-contract and artifact-boundary issue** , not as a generic inference-speed issue.",
  "title": "HF ZeroGPU Space Hangs, No Output in the logs"
}