{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/python/multithreaded-socket-server-signal-handling/",
  "description": "Gracefully shutdown Python's ThreadingTCPServer with signal handlers for SIGINT, SIGTERM handling and client notification on server shutdown.",
  "path": "/python/multithreaded-socket-server-signal-handling/",
  "publishedAt": "2023-02-26T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python",
    "Networking",
    "Concurrency"
  ],
  "textContent": "While working on a multithreaded socket server in an embedded environment, I realized that\nthe default behavior of Python's socketserver.ThreadingTCPServer requires some extra work\nif you want to shut down the server gracefully in the presence of an interruption signal.\nThe intended behavior here is that whenever any of SIGHUP, SIGINT, SIGTERM, or\nSIGQUIT signals are sent to the server, it should:\n\n- Acknowledge the signal and log a message to the output console of the server.\n- Notify all the connected clients that the server is going offline.\n- Give the clients enough time (specified by a timeout parameter) to close the requests.\n- Close all the client requests and then shut down the server after the timeout exceeds.\n\nHere's a quick implementation of a multithreaded echo server and see what happens when you\nsend SIGINT to shut down the server:\n\nHere's the client code:\n\nHere, the server logs and echoes back whatever the client sends and the client just sends\nthe string hello world continuously in a while loop. This is pretty much the canonical\nmultithreaded server-client example that's found in the socketserver docs. In the client\ncode, the only thing that's a little different is that within the while loop, a\ntime.sleep(1) function was added to simulate the client performing some processing tasks.\nAlso, without the sleep, the server would've flooded the stdout with the client message\nlogs and made the demonstration difficult.\n\nLet's run the server and the client in two separate processes and then send a SIGINT\nsignal to the server by clicking Ctrl + C on the server console:\n\n![Terminal showing multi-threaded socket server with client connection and SIGINT handling][image_1]\n\nAt first, the server just ignores the signal, and clicking Ctrl + C multiple times crashes\nthe server down with this nasty traceback (full traceback trimmed for brevity):\n\nMultithreaded socket server with graceful shutdown\n\nWhat we want here is that whenever the server gets SIGHUP, SIGINT, SIGTERM, or\nSIGQUIT, it should notify the clients and gracefully shut itself down. I played around\nwith the socketserver.ThreadingTCPServer API for a while to come up with a solution that\nworked nicely for my use case. Here's the full server implementation:\n\nApart from a few extra methods that perform logging and signal handling, the overall\nstructure of this server is similar to the vanilla multithreaded server from the previous\nsection. In the RequestHandler, we have defined a custom\nnotify_clients_when_server_is_interrupted method that notifies all clients whenever the\nserver receives an interruption signal. This is a custom method that's not defined in the\nBaseRequestHandler class. The notify method logs the status of the interruption signal and\nthen sends a SHUTDOWN message to the clients. Afterward, it closes the client connection.\n\nThe setup method extends the eponymous method from the BaseRequestHandler class and\ncalls the notify_clients_when_server_is_interrupted method. This ensures that whenever the\nserver is shutting down, it refuses any new client connections. Within the handle method, in\nthe data processing while loop, we check the value of the _is_interrupted flag on the\nserver instance. If the value is True, we call the notify method. The value of this flag\nis managed by the SocketServer class. Calling the notify method from within the data\nprocessing loop will notify all currently connected clients.\n\nNext, we define a new server class called SocketServer that inherits from the\nsocketserver.ThreadingTCPServer class. The reuse_address, daemon_threads, and\nblock_on_close class variables override the default values inherited from the base\nThreadingTCPServer class. Here are the explanations for each:\n\n1. reuse_address: This variable determines whether the server can reuse a socket that's\n   still in the [TIME_WAIT] state after a previous connection has been closed. If this\n   variable is set to True, the server can reuse the socket. Otherwise, the socket will be\n   unavailable for a short period of time after it's closed.\n\n2. daemon_threads: This variable determines whether the server's worker threads should be\n   daemon threads. Daemon threads are threads that run in the background and don't prevent\n   the Python interpreter from exiting when they are still running. If this variable is set\n   to True, the server's worker threads will be daemon threads. I found that daemon\n   threads work better when I need to shut down the server that's connected to multiple\n   long-running clients.\n\n3. block_on_close: This variable determines whether the server should block until all\n   client connections have been closed before shutting down. If this variable is set to\n   True, the server will block until all client connections have been closed. Otherwise,\n   the server will shut down immediately, even if there are still active client connections.\n   We want to set it to False since we'll handle the graceful shutdown in a custom signal\n   handler method on the server class.\n\nGoing forward, the SocketServer class overrides the server_activate, get_request,\nshutdown_request, and shutdown methods from the base class. All of them just log a few\nkey pieces of information to the console and calls the methods from the parent class\nverbatim. The interesting part happens in the custom handle_signal method. When an\ninterruption signal is sent to the server, the handle_signal method is activated. The\nmethod takes an integer parameter timeout which specifies how many seconds the server\nshould wait before shutting down after receiving the signal.\n\nThe method then returns the actual signal handler function that takes two parameters: an\ninteger signum representing the signal number and a FrameType object which represents\nthe current stack frame. The function is responsible for handling the signal by making the\nserver wait for timeout seconds before shutting it down gracefully.\n\nFirst, the method sets a variable _is_interrupted to True to indicate that the server\nhas received an interruption signal. Then, the method enters a while loop that continues\nuntil the current time exceeds the deadline time, which is calculated by adding the\ntimeout to the current monotonic time. During each iteration of the while loop, the\nmethod logs a message to the console to indicate that the signal has been received and the\nserver will be closed in a certain number of seconds. The delta variable is calculated as\nthe difference between the deadline and the current monotonic time, plus 1. This ensures\nthat the logging message displays an accurate countdown of the remaining time until the\nserver shuts down.\n\nOnce the deadline exceeds and the while loop completes, the method calls\nserver_close() and shutdown() methods of the server to close the requests and shut\nitself down gracefully. The server_close() method closes the listening socket and stops\naccepting new client connections, while the shutdown() method stops all active client\nconnections and waits for them to finish processing their current requests. However, in this\ncase, since we are giving the clients enough time to close the connections and using daemon\nthreads to process the requests, calling shutdown() will immediately close all the client\nrequests and bring down the server.\n\nFinally, in the __main__ section, we instantiate the SocketServer class and register the\nRequestHandler. Then we register the signal handler with a timeout of 5 seconds. This\nmeans, upon receiving the interruption signal, the server will wait 5 seconds before\nshutting itself down. Notice, how we're running the server.serve_forever method in a new\nthread. That's because our custom signal handler explicitly calls the shutdown of the\nserver instance and the shutdown method can only be called when the serve_forever loop\nis running in a different thread. From the [shutdown documentation]:\n\n> Tell the serve_forever() loop to stop and wait until it does. shutdown() must be called\n> while serve_forever() is running in a different thread otherwise it will deadlock.\n\nNow that the server is coded to shut down gracefully, we also expect the client to behave\nproperly. That means, whenever the client receives the SHUTDOWN message, it should\nimmediately close the connection. Here's a slightly modified version of the vanilla socket\nclient code that we've seen before:\n\nThe only difference between this and the previous client is that this client will break out\nof the process loop when it encounters the SHUTDOWN message from the server. Now to see\nthe whole thing in action, you can fire up the server and the client from two different\nterminals. Once both the server and client are running, try sending a SIGINT or any of the\nthree other handled signals. You see that the server acknowledges the interruption signal,\ngives the clients enough time to disconnect, then shut itself down in a graceful manner:\n\n![Terminal showing graceful shutdown with signal handling and client notifications][image_2]\n\nFurther reading\n\n- [socketserver]\n\n\n\n\n[time_wait]:\n    https://totozhang.github.io/2016-01-31-tcp-timewait-status/\n\n[shutdown documentation]:\n    https://docs.python.org/3/library/socketserver.html#socketserver.BaseServer.shutdown\n\n[socketserver]:\n    https://docs.python.org/3/library/socketserver.html\n\n[image_1]:\n    https://blob.rednafi.com/static/images/multithreaded_socket_server_signal_handling/img_1.png\n\n[image_2]:\n    https://blob.rednafi.com/static/images/multithreaded_socket_server_signal_handling/img_2.png",
  "title": "Signal handling in a multithreaded socket server"
}