{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/misc/tinkering-with-unix-domain-socket/",
  "description": "Build Unix domain socket servers and clients with Python and socat. Access Docker API via UDS, create HTTP servers, and optimize inter-process communication.",
  "path": "/misc/tinkering-with-unix-domain-socket/",
  "publishedAt": "2023-03-11T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python",
    "Shell",
    "Unix",
    "Networking"
  ],
  "textContent": "I've always had a vague idea about what Unix domain sockets are from my experience working\nwith Docker for the past couple of years. However, lately, I'm spending more time in\nembedded edge environments and had to explore Unix domain sockets in a bit more detail. This\nis a rough documentation of what I've explored to gain some insights.\n\nThe dry definition\n\nUnix domain sockets (UDS) are similar to TCP sockets in a way that they allow two processes\nto communicate with each other, but there are some core differences. While TCP sockets are\nused for communication over a network, Unix domain sockets are used for communication\nbetween processes running on the same computer.\n\nA Unix domain socket is a way for programs to exchange data in a fast and efficient way\nwithout having to worry about the overhead of network protocols like TCP/IP or UDP. It works\nby creating a special file on the file system called a socket, which acts as a bidirectional\ndata channel between the processes. The processes can send and receive data through the\nsocket just like they would with a network socket. Also, just like TCP/UDP sockets, Unix\ndomain sockets can also be either stream-based (TCP equivalent) or datagram-based (UDP\nequivalent).\n\nUnix domain sockets are commonly used in server-client applications, such as web servers,\ndatabases, and email servers, where they provide a secure and efficient way for processes on\nthe same machine to communicate with each other. They're also used in many other types of\nprograms where different parts of the program need to work together or share data. Another\ncool thing about them is that you can control access to your server just by tuning the\npermission of the socket file on the system.\n\nPrerequisites\n\nI'm running these experiments on an M-series Macbook pro. However, any Unix-y environment\nwill work as long as you can run the following tools:\n\n- socat: To create the socket servers and clients.\n- curl: To make HTTP requests to a supported socket server.\n- jq: To pretty print JSON payloads.\n- lsof: To display currently listening socket server processes.\n\nInspecting Unix domain sockets in your system\n\nMost likely, there are currently multiple processes listening on different sockets in your\nsystem. You can explore them using lsof with the following command:\n\nThis will return a list of all Unix domain socket files and the server process PIDs that are\ncurrently listening on them:\n\nYou can also filter out the socket files by their process names. Docker processes listen on\na few socket files:\n\nThis will return:\n\nCreating a Unix domain socket\n\nRunning the following command on your terminal will create a stream-based Unix domain\nsocket:\n\nThis process listens on the /tmp/stream.sock and prints the incoming data to the stdout.\nThe fork portion on the command ensures that multiple clients can be connected to the\nserver process and they'll be served by forking child processes.\n\nFrom another console, you can try to send data to the socket file as a client:\n\nRunning this command will send the hello world string to the /tmp/stream.sock file and\nthe server process will print it on the standard output stream.\n\nSimilarly, you can also create a datagram-based socket server with socat like this:\n\nNow send data to the server with this:\n\nConnecting to Docker engine via a Unix domain socket\n\nBy default, Docker runs through a non-networked UNIX socket. It can also optionally\ncommunicate using SSH or a TLS (HTTPS) socket. On MacOS, the socket file can be found in\n~/.docker/run/docker.sock. We can make HTTP requests against the listening socket server\nand use Docker engine's RESTful API suite.\n\nChecking the engine's version number: The following command uses curl to spawn a\nclient process and send a request against the Docker engine running in my local system.\n\nThis returns (truncated output for readability):\n\nListing the containers: This command lists all the running containers on my machine.\n\nListing the images:\n\nDownloading a container: This allows you to programmatically download the hello-world\nimage from Dockerhub:\n\nListening for docker events: This API call lets you listen for all incoming events from\nthe docker engine. You can run the following command on one terminal and send events from\nanother:\n\nHere, the --no-buffer flag is necessary for instructing curl to send the output events\nto the input stream of jq without doing any buffering. This allows jq to pretty-print\nthe outputs in real-time. Now from another console if you run the following command, you'll\nsee events pouring into the console that's listening for them:\n\nThe complete list of APIs can be found in the [Docker engine API documentation].\n\nWriting a Unix domain socket server in Python\n\nYou can quickly write a simple server that allows clients to connect to it via Unix domain\nsockets. If the clients exist on the same machine then, a UDS server has the advantage of\nhaving lower overhead than its networked TCP counterpart.\n\nThe following server uses Python's socketserver module to create a stream-based echo\nserver:\n\nHere, socketserver.ThreadingUnixStreamServer enables us to create a server that allows\nmultiple clients to be connected to it via Unix domain sockets. The server spins up a new\nthread to serve each new client and does bi-directional communication via UDS. The client\ncode is quite similar to a TCP client:\n\nThe client connects to the server through the /tmp/stream.sock socket and sends a static\nhello world string to it. The server then sends that data back and the client sends it to\nthe stdout stream.\n\nRunning the server and client as two separate processes will yield the following output:\n\n![Terminal split view showing Unix domain socket server and client exchanging hello world messages][image_1]\n\nExposing an HTTP application via a Unix domain socket\n\nWebservers usually allow you to expose HTTP applications via Unix domain sockets. In Python,\nthe [uvicorn] ASGI server lets you do this quite easily. This can come as handy whenever you\nneed to spin up a local server and all the clients are running on the same machine or you're\nrunning your server behind a proxy. Here's an example of a simple webserver built with\n[starlette] and served with uvicorn.\n\nYou can expose this server through a UDS like this:\n\nCalling this API with curl from another console will return the HTML content in the\nresponse:\n\nIf you want to access this server from a browser, you'll need to make sure that your reverse\nproxy server (Nginx / Apache / Caddy) is configured to relay the incoming request from the\nnetwork to the UDS server. For a quick and dirty approach, you can use socat to proxy the\nrequest from a HOST:PORT pair to the UDS server like this:\n\nThe uvicorn command spins up a webserver in the background as before and listens on the\nsocket file /tmp/stream.sock. Then we're using socat to create a forking TCP server that\nhandles the incoming HTTP requests from the network and relays them to the webserver via\nUDS. It also relays the server's responses back to the client - doing the work of a reverse\nproxy.\n\nYou can then head over to your browser and go to http://localhost:9999. This will display\nthe HTML page:\n\n![Browser displaying Hello World HTML page served via Unix domain socket with socat proxy][image_2]\n\nFurther reading\n\n- [Understanding sockets - Digital Ocean]\n\n- [Fun with Unix domain sockets - Simon Willison]\n\n\n\n\n[docker engine api documentation]:\n    https://docs.docker.com/engine/api/latest/\n\n[uvicorn]:\n    https://www.uvicorn.org/\n\n[starlette]:\n    https://www.starlette.io/\n\n[understanding sockets - digital ocean]:\n    https://www.digitalocean.com/community/tutorials/understanding-sockets\n\n[fun with unix domain sockets - simon willison]:\n    https://simonwillison.net/2021/Jul/13/unix-domain-sockets/\n\n[image_1]:\n    https://blob.rednafi.com/static/images/tinkering_with_unix_domain_socket/img_1.png\n\n[image_2]:\n    https://blob.rednafi.com/static/images/tinkering_with_unix_domain_socket/img_2.png",
  "title": "Tinkering with Unix domain sockets"
}