{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/python/contextmanager/",
  "description": "Deep dive into Python context managers. Learn to write custom managers with __enter__/__exit__, use contextlib decorators, and manage resources elegantly.",
  "path": "/python/contextmanager/",
  "publishedAt": "2020-03-26T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python"
  ],
  "textContent": "Python's context managers are great for resource management and stopping the propagation of\nleaked abstractions. You've probably used it while opening a file or a database connection.\nUsually it starts with a with statement like this:\n\nIn the above case, file.txt gets automatically closed when the execution flow goes out of\nthe scope. This is equivalent to writing:\n\nWriting custom context managers\n\nTo write a custom context manager, you need to create a class that includes the __enter__\nand __exit__ methods. Let's recreate a custom context manager that will execute the same\nworkflow as above.\n\nYou can use the above class just like a regular context manager.\n\nFrom generators to context managers\n\nCreating context managers by writing a class with __enter__ and __exit__ methods isn't\ndifficult. However, you can achieve better brevity by defining them using\ncontextlib.contextmanager decorator. This decorator converts a generator function into a\ncontext manager. The blueprint for creating context manager decorators goes something like\nthis:\n\nWhen you use the context manager with the with statement:\n\nIt roughly translates to:\n\nThe setup code goes before the try..finally block. Notice the point where the generator\nyields. This is where the code block nested in the with statement gets executed. After the\ncompletion of the code block, the generator is then resumed. If an unhandled exception\noccurs in the block, it's re-raised inside the generator at the point where the yield\noccurred and then the finally block is executed. If no unhandled exception occurs, the\ncode gracefully proceeds to the finally block where you run your cleanup code.\n\nLet's implement the same CustomFileOpen context manager with contextmanager decorator.\n\nNow use it just like before:\n\nWriting context managers as decorators\n\nYou can use context managers as decorators also. To do so, while defining the class, you\nhave to inherit from contextlib.ContextDecorator class. Let's make a RunTime decorator\nthat'll be applied on a file-opening function. The decorator will:\n\n- Print a user provided description of the function.\n- Print the time it takes to run the function.\n\nYou can use the decorator like this:\n\nUsing the function like this should return:\n\nYou can also create the same decorator via contextlib.contextmanager decorator.\n\nNesting contexts\n\nYou can nest multiple context managers to manage resources simultaneously. Consider the\nfollowing dummy manager:\n\nNotice the order they're closed. Context managers are treated as a stack, and should be\nexited in reverse order in which they're entered. If an exception occurs, this order\nmatters, as any context manager could suppress the exception, at which point the remaining\nmanagers will not even get notified of this. The __exit__ method is also permitted to\nraise a different exception, and other context managers then should be able to handle that\nnew exception.\n\nCombining multiple context managers\n\nYou can combine multiple context managers too. Let's consider these two managers.\n\nNow combine these two using the decorator syntax. The following function takes the above\ndefine managers a and b and returns a combined context manager ab.\n\nThis can be used as:\n\nIf you have variable numbers of context managers and you want to combine them gracefully,\ncontextlib.ExitStack is here to help. Let's rewrite context manager ab using\nExitStack. This function takes the individual context managers and their arguments as\ntuples and returns the combined manager.\n\nExitStack can be also used in cases where you want to manage multiple resources\ngracefully. For example, suppose, you need to create a list from the contents of multiple\nfiles in a directory. Let's see, how you can do so while avoiding accidental memory leakage\nwith robust resource management.\n\nUsing context managers to create SQLAlchemy session\n\nIf you are familiar with SQLALchemy, Python's SQL toolkit and Object Relational Mapper, then\nyou probably know the usage of Session to run a query. A Session basically turns any\nquery into a transaction and make it atomic. Context managers can help you write a\ntransaction session in a very elegant way. A basic querying workflow in SQLAlchemy may look\nlike this:\n\nThe excerpt above creates an in memory SQLite connection and a session_scope function\nwith context manager. The session_scope function takes care of committing and rolling back\nin case of exception automatically. The session_scope function can be used to run queries\nin the following way:\n\nAbstract away exception handling monstrosity with context managers\n\nThis is my absolute favorite use case of context managers. Suppose you want to write a\nfunction but want the exception handling logic out of the way. Exception handling logics\nwith sophisticated logging can often obfuscate the core logic of your function. You can\nwrite a decorator type context manager that will handle the exceptions for you and decouple\nthese additional code from your main logic. Let's write a decorator that will handle\nZeroDivisionError and TypeError simultaneously.\n\nNow use this in a function where these exceptions occur.\n\nYou can see that the errhandler decorator is doing the heavylifting for you. Pretty neat,\nhuh?\n\nThe following one is a more sophisticated example of using context manager to decouple your\nerror handling monstrosity from the main logic. It also hides the elaborate logging logic\nfrom the main method.\n\nThis will return\n\nPersistent parameters across HTTP requests with context managers\n\nAnother great use case for context managers is making parameters persistent across multiple\nHTTP requests. Python's requests library has a Session object that will let you easily\nachieve this. So, if you're making several requests to the same host, the underlying TCP\nconnection will be reused, which can result in a significant performance increase. The\nfollowing example is taken directly from the official docs of the [requests] library. Let's\npersist some cookies across requests.\n\nThis should show:\n\nRemarks\n\nTo avoid redundencies, I have purposefully excluded examples of nested with statements and\nnow deprecated contextlib.nested function to create nested context managers.\n\nFurther reading\n\n- [Python contextlib documentation]\n- [Python with context manager by Jeff Knupp]\n- [SQLAlchemy session creation]\n- [Scipy lectures on context managers]\n- [Merging context managers]\n\n\n\n\n[requests]:\n    https://docs.python-requests.org/en/latest/user/advanced/#session-objects\n\n[python contextlib documentation]:\n    https://docs.python.org/3/library/contextlib.html\n\n[python with context manager by jeff knupp]:\n    https://jeffknupp.com/blog/2016/03/07/python-with-context-managers/\n\n[sqlalchemy session creation]:\n    https://docs.sqlalchemy.org/en/20/core/engines.html\n\n[scipy lectures on context managers]:\n    https://scipy-lectures.org/advanced/advanced_python/index.html#context-managers\n\n[merging context managers]:\n    https://stackoverflow.com/a/45681273/8963300",
  "title": "The curious case of Python's context manager"
}