{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreicfuik2p3celvqbe56tf2o3k2dxvxjmtryjni45zfcnxwtcpvhlxe",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mg7pehhb2nd2"
  },
  "path": "/t/docker-space-runs-normally-but-run-logs-only-show-application-startup-banner-no-container-stdout-stderr/173988#post_4",
  "publishedAt": "2026-03-04T05:31:30.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "bin29-manimcat.hf.space",
    "Hugging Face Forums",
    "GitHub",
    "Hugging Face"
  ],
  "textContent": "The following primarily covers troubleshooting methods.\nHmm… It’s certainly possible that some default setting on Hugging Face’s end changed and triggered this issue.\n\n* * *\n\n## What your current signals say\n\nYour Space is not “half-started” or wedged:\n\n  * `GET /health` returns `{\"status\":\"ok\", ... \"services\":{\"redis\":true,\"queue\":true,\"openai\":true}}`, so the container is serving traffic and the app is executing code paths successfully. (bin29-manimcat.hf.space)\n  * Build logs are complete, so the build pipeline and image push are fine (your observation).\n  * The only anomaly is that **run logs show only** `===== Application Startup ... =====`.\n\n\n\nThat pattern is consistent with **(a)** stdout/stderr not receiving anything after startup (logs going to files / another sink), **(b)** output buffering (especially Python), or **(c)** a logs UI/SSE retrieval problem. The “container logs are what your app prints to stdout/stderr” mental model is explicitly reflected in common debugging guidance for Spaces. (Hugging Face Forums)\n\n* * *\n\n## Background: why “run logs” can be blank even when the app is healthy\n\n### Build logs vs run logs\n\n  * **Build logs** : what happens while building your image and pushing it.\n  * **Run logs** : what the running container prints to **stdout/stderr** (what “docker logs” would show in a normal Docker environment). (Hugging Face Forums)\n\n\n\nThe platform banner line (`===== Application Startup ... =====`) is not your app output; it’s the runtime marker that the container was started. Many threads show that exact marker as the first/only visible line when something else prevents useful stdout/stderr from appearing. (Hugging Face Forums)\n\n* * *\n\n## Likeliest causes in your specific situation (healthy endpoints, empty run logs)\n\n### 1) Your app is logging to files (or a process manager is)\n\nA common real-world pattern on Spaces is: startup banner appears, then a library announces it is “writing logs to …/something.log”. After that, the logs UI looks “empty” because nothing goes to stdout/stderr. (Hugging Face Forums)\n\nTypical culprits in Docker Spaces:\n\n  * File-based logging configs (e.g., Winston/Pino to file, Python logging to `logs/*.log`)\n  * Reverse proxy defaults (nginx access/error logs to `/var/log/...`)\n  * Process managers (PM2/supervisord) routing logs to their own log files\n\n\n\n**Why this matches your case:** a service can be perfectly healthy (responding on `/health`, returning 401/4xx on business endpoints) while still emitting nothing to stdout/stderr.\n\n### 2) Output buffering (especially if any Python is involved)\n\nBuffering can make the stream appear blank even though your code ran. This is discussed directly in Spaces runtime logging contexts; the common mitigation is `PYTHONUNBUFFERED=1` / running Python unbuffered. (Hugging Face Forums)\n\n### 3) Multi-process container where PID 1 is not your “chatty” process\n\nIf PID 1 is a shell script that backgrounds the actual server, or a supervisor that doesn’t forward child output, stdout/stderr can be quiet while the server itself is running.\n\n### 4) Less common: log streaming/viewer path issue\n\nThere are recurring reports where people see the startup banner and little else, sometimes “fixed” by rebuild/reupload/restart, suggesting a mix of infra churn and observability limitations. (Hugging Face Forums)\n\n* * *\n\n## The fastest discriminator: does _anything_ you print reach run logs?\n\nMake a temporary change that **must** show up if log forwarding works.\n\n### A. Force a BOOT line to stdout _and_ stderr at container start\n\nIn your Dockerfile `CMD` (or entrypoint script), wrap your actual start command:\n\n\n    bash -lc 'echo BOOT_STDOUT; echo BOOT_STDERR 1>&2; exec /path/to/your/start-command'\n\n\nInterpretation:\n\n  * If `BOOT_STDOUT` / `BOOT_STDERR` appear in run logs: forwarding works → your app/process/logging config is the cause.\n  * If they do **not** appear: either you are not looking at the correct log stream endpoint, or the log streaming path for that runtime instance is broken.\n\n\n\n### B. Add a periodic heartbeat to stderr\n\nEven if your app is quiet, this proves continuous forwarding:\n\n\n    bash -lc 'while true; do echo \"HEARTBEAT $(date -Is)\" 1>&2; sleep 15; done & exec /path/to/start'\n\n\n* * *\n\n## Ensure you’re using the “correct” streaming method (JWT + `api.hf.space`)\n\nEven if `https://huggingface.co/api/spaces/.../logs/run` is quiet, verify via the method documented in `huggingface_hub` issue #2667:\n\n  1. fetch a **Space JWT** from `/api/spaces/{space_id}/jwt`\n  2. stream from `https://api.hf.space/v1/{space_id}/logs/run` (SSE) (GitHub)\n\n\n\nIf logs show up there but not in the UI/Hub endpoint, that’s strong evidence of a **viewer/endpoint issue** rather than a container stdout/stderr issue.\n\n* * *\n\n## Docker Spaces config checks that still matter (even though your `/health` works)\n\n  * Confirm your Space exposes the expected port via `app_port` in `README.md` YAML (default `7860`). (Hugging Face)\n(Your `/health` already implies external routing is correct, but it’s still worth validating `app_port` hasn’t drifted.)\n\n\n\n* * *\n\n## Practical “fix patterns” once you know which bucket you’re in\n\n### If forwarding works but your app is silent\n\nDo one (or more) of these:\n\n  * **Route logs to stdout/stderr**\n\n    * Node: ensure your logger uses a console transport (not file-only)\n    * Python: attach a `StreamHandler(sys.stdout)` / `StreamHandler(sys.stderr)`\n    * nginx: configure access/error logs to `/dev/stdout` and `/dev/stderr` (or symlink)\n  * **Make PID 1 the real server**\n\n    * Avoid `&`, `nohup`, daemon modes; use `exec` so the main server owns stdout/stderr.\n  * **If Python involved:** set `PYTHONUNBUFFERED=1` (or run `python -u ...`) to avoid buffered output. (Hugging Face Forums)\n\n\n\n\n### If forwarding does _not_ work even for BOOT lines\n\n  * Try the JWT + `api.hf.space` stream first. (GitHub)\n\n  * If that is also blank:\n\n    * push a trivial commit to force a fresh container schedule (new runtime instance)\n    * (optional) use Dev Mode to inspect where logs are going from inside the container (files vs stdout). Dev Mode is explicitly intended for debugging/monitoring a running Space. (Hugging Face)\n\n\n\n* * *\n\n## Direct answers to your 4 questions\n\n### 1) Is this a known issue?\n\n“Startup banner only” is a recurring symptom people report, but the **most common underlying causes** are app-side (logging-to-file, buffering, process manager / backgrounding). There are also occasional platform-side logging/viewer failures, but they’re less frequent than the app-side causes. (Hugging Face Forums)\n\n### 2) Docker Space configs that cause banner-only logs?\n\nYes—anything that prevents meaningful stdout/stderr:\n\n  * file-only logging\n  * nginx/supervisord/PM2 defaults writing to files\n  * daemonizing/backgrounding the server so PID 1 is quiet\n  * buffered output (Hugging Face Forums)\n\n\n\n### 3) “Log channel reset” other than factory reboot?\n\nNo dedicated “reset log forwarder” control is documented. The practical equivalents are:\n\n  * force a new runtime instance (commit that changes startup behavior)\n  * switch hardware and switch back\n  * duplicate the Space as a control\n\n\n\n### 4) Can HF check backend log forwarder status?\n\nOnly HF staff can. The most actionable escalation bundle is:\n\n  * timestamps (UTC) when you streamed logs\n  * confirmation `/health` returns OK (include body)\n  * results of the BOOT stdout/stderr probe\n  * whether JWT + `api.hf.space` stream shows anything (GitHub)\n\n\n\n* * *\n\n## Suggested next move (highest leverage)\n\n  1. Add the **BOOT stdout/stderr** lines in `CMD` and redeploy.\n  2. Stream logs using **JWT +`api.hf.space`** once, in parallel with the UI. (GitHub)\n  3. If BOOT appears: fix logging/buffering/process model. If BOOT does not appear anywhere: escalate with the evidence bundle above.\n\n",
  "title": "Docker Space Runs Normally but Run Logs Only Show “Application Startup” Banner, No Container stdout/stderr"
}