{
  "$type": "site.standard.document",
  "canonicalUrl": "https://justingarrison.com/blog/2017-02-13-how-to-debug-a-running-docker-container-from-a-separate-container",
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreifgwoqbccqlij6y4yysxqggos6bxdieuh6wln4dthw2clgkh6vrly"
    },
    "mimeType": "image/png",
    "size": 45289
  },
  "description": "Debugging a stripped down Docker container from a separate container",
  "path": "/blog/2017-02-13-how-to-debug-a-running-docker-container-from-a-separate-container",
  "publishedAt": "2017-02-13T00:00:00.000Z",
  "site": "at://did:plc:p7uix7mresfq4nfzxp3klgfa/site.standard.publication/3mmdn7mg2qm2d",
  "textContent": "Alternate title: How-to debug freaking go binary containers\n\nContainers are great for shipping software, but sometimes you can go too far when stripping down your container to make it as small as possible. There’s a fine balance between a “no-frills” image and something impossible to debug (I’m looking at you, single binary go containers).\n\nThe normal way I see people debug a running container is to docker exec -it $CONTAINER sh and install their debug tools on demand in the container. But what if your container doesn’t have /bin/sh? What if it doesn’t have a package manager? You could copy utilities into the container with docker cp and then exec into the running container but that’s also a hassle.\n\nSo instead of trying to debug from within the container, a friend recently asked how you could debug from a different container. I’m not that smart so I asked much smarter people online and got a good answer back. Thanks again Justin Cormack for the reply.\n\nLet’s create a stripped down container with only caddy.\n\nFirst download/extract the caddy binary\n\nThen create a Dockerfile to copy the binary into a scratch container.\n\nBuild the container and run caddy.\n\nNow run the container.\n\nNow caddy is running publishing port 2015 (currently giving a 404 page because there’s no content that but doesn’t matter). How do you debug the container? Not that you’d ever need to, caddy doesn’t have bugs. :) But for hypothetical reasons.\n\nMany people suggested using --link but that only puts the containers on the same network. Not the same namespace, but connected to each other on the same virtual network.\n\nOthers suggested using --volumes-from but that doesn’t let you mount your tools into an existing running container unless that running container is exporting a volume and that volume is already in the $PATH.\n\nInstead we’re going to build a separate container with all the tools we need (in this case strace) and run it in the same pid and network namespace as the original container.\n\nFirst create a debug container with strace\n\nBuild the container\n\nNow run your strace container in the same pid and network namespace.\n\nThis attached strace to the caddy process and is following it as it executes.\n\nThat’s great but we also can get to the root filesystem (not that there’s much of one) of the remote container. This time we’ll just use the alpine image and launch a shell, again in the same pid and network namespace.\n\nWe can now see caddy running\n\nThe caddy container file system is available in /proc/1/root\n\nWith this container attached to the original we can do more debugging. You can still debug the network but make sure you use localhost because your new sh process is running in the same network namespace\n\nAll your standard debugging tools should work from this 2nd container without tainting the original container. If you run into errors make sure you check your kernel permissions (notice how strace needed --cap-add sys_ptrace but the sh container only needed sys_admin)\n\nThis can obviously be helpful for go containers or any other container that you just need to bring some extra debugging tools into without changing the container itself.\n\nLet me know if you’ve found any other helpful tricks for debugging containers on the fly.\n\nYour best option to get a response or feedback is on twitter. https://twitter.com/rothgar",
  "title": "How-to Debug a Running Docker Container from a Separate Container"
}