{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/python/decorators/",
  "description": "Complete guide to Python decorators from first principles. Learn closures, higher-order functions, decorator patterns, and advanced techniques.",
  "path": "/python/decorators/",
  "publishedAt": "2020-05-13T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python"
  ],
  "textContent": "_Updated on 2022-02-13_: _Change functools import style._\n\nWhen I first learned about Python decorators, using them felt like doing voodoo magic.\nDecorators can give you the ability to add new functionalities to any callable without\nactually touching or changing the code inside it. This can typically yield better\nencapsulation and help you write cleaner and more understandable code. However, _decorator_\nis considered as a fairly advanced topic in Python since understanding and writing it\nrequires you to have command over multiple additional concepts like first class objects,\nhigher order functions, closures etc. First, I'll try to introduce these concepts as\nnecessary and then unravel the core concept of decorator layer by layer. So let's dive in.\n\nFirst class objects\n\nIn Python, basically everything is an object and functions are regarded as first-class\nobjects. It means that functions can be passed around and used as arguments, just like any\nother object (string, int, float, list, and so on). You can assign functions to variables\nand treat them like any other objects. Consider this example:\n\nThe above example demonstrates how Python treats functions as first class citizens. First, I\ndefined two functions, func_a and func_b and then func_c takes them as parameters.\nfunc_c runs the functions taken as parameters and prints the results. Then we assign\nfunc_c to variable main_func. Finally, we run main_func and it behaves just like\nfunc_c.\n\nHigher order functions\n\nPython also allows you to use functions as return values. You can take in another function\nand return that function or you can define a function within another function and return the\ninner function.\n\nNow you can assign the result of higher to another variable and execute the output\nfunction.\n\nLet's look into another example where you can define a nested function within a function and\nreturn the nested function instead of its result.\n\nNotice how the nested function inner was defined inside the outer function and then the\nreturn statement of the outer function returned the nested function. After definition, to\nget to the nested function, first we called the outer function and received the result as\nanother function. Then executing the result of the outer function prints out the message\nfrom the inner function.\n\nClosures\n\nYou saw examples of inner functions at work in the previous section. Nested functions can\naccess variables of the enclosing scope. In Python, these non-local variables are read only\nby default and we must declare them explicitly as non-local (using nonlocal keyword) in\norder to modify them. Following is an example of a nested function accessing a non-local\nvariable.\n\nNow run the function,\n\nWell, that's unusual.\n\nThe burger function was called with the string deli and the returned function was bound\nto the name ingr. On calling ingr(), the message was still remembered and used to derive\nthe outcome although the outer function burger had already finished its execution.\n\nThis technique by which some data (\"deli\") gets attached to the code is called closure in\nPython. The value in the enclosing scope is remembered even when the variable goes out of\nscope or the function itself is removed from the current namespace. Decorators uses the idea\nof non-local variables multiple times and soon you'll see how.\n\nWriting a basic decorator\n\nWith these prerequisites out of the way, let's go ahead and create your first simple\ndecorator.\n\nBefore using the decorator, let's define a simple function without any parameters.\n\nTreating the functions as first-class objects, you can use your decorator like this:\n\nIn the above two lines, you can see a very simple decorator in action. Our deco function\ntakes in a target function, manipulates the target function inside a wrapper function and\nthen returns the wrapper function. Running the function returned by the decorator, you'll\nget your modified result. To put it simply, decorators wraps a function and modifies its\nbehavior.\n\n> The decorator function runs at the time the decorated function is imported/defined, not\n> when it is called.\n\nBefore moving onto the next section, let's see how we can get the return value of target\nfunction instead of just printing it.\n\nIn the above example, the wrapper function returns the result of the target function and the\nwrapper itself. This makes it possible to get the result of the modified function.\n\nCan you guess why the return value of the decorated function appeared in the last line\ninstead of in the middle like before?\n\nThe @ syntactic sugar\n\nThe way you've used decorator in the last section might feel a little clunky. First, you\nhave to type the name ans three times to call and use the decorator. Also, it becomes\nharder to tell apart where the decorator is actually working. So Python allows you to use\ndecorator with the special syntax @. You can apply your decorators while defining your\nfunctions, like this:\n\nSometimes the above syntax is called the pie syntax and it's just a syntactic sugar for\nfunc = deco(func).\n\nDecorating functions with arguments\n\nThe naive decorator that we've implemented above will only work for functions that take no\narguments. It'll fail and raise TypeError if your try to decorate a function having\narguments with deco. Now let's create another decorator called yell which will take in a\nfunction that returns a string value and transform that string value to uppercase.\n\nCreate the target function that returns string value.\n\nFunction hello takes a name:string as parameter and returns a message as string. Look\nhow the yell decorator is modifying the original return string, transforming that to\nuppercase and adding an extra ! sign without directly changing any code in the hello\nfunction.\n\nSolving identity crisis\n\nIn Python, you can introspect any object and its properties via the interactive shell. A\nfunction knows its identity, docstring etc. For instance, you can inspect the built in\nprint function in the following ways:\n\nThis introspection works similarly for functions that you defined yourself. I'll be using\nthe previously defined hello function.\n\nNow what's going on there. The decorator yell has made the function hello confused about\nits own identity. Instead of reporting its own name, it takes the identity of the inner\nfunction wrapper. This can be confusing while doing debugging. You can fix this by using\nbuiltin wraps decorator from the functools module. This will make sure that the original\nidentity of the decorated function stays preserved.\n\nIntrospecting the hello function decorated with modified decorator will give you the\ndesired result.\n\nDecorators in the wild\n\nBefore moving on to the next section let's see a few real world examples of decorators. To\ndefine all the decorators, we'll be using the following template that we've perfected so\nfar.\n\nTimer\n\nTimer decorator will help you time your callables in a non-intrusive way. It can help you\nwhile debugging and profiling your functions.\n\nIn the above way, we can introspect the time it requires for function dothings to complete\nits execution.\n\nException logger\n\nJust like the timer decorator, we can define a logger decorator that will log the state of\na callable. For this demonstration, I'll be defining a exception logger that will show\nadditional information like timestamp, argument names when an exception occurs inside of the\ndecorated callable.\n\nLet's invoke ZeroDivisionError to see the logger in action.\n\nThe decorator first prints a few info regarding the function and then raises the original\nerror.\n\nValidation & runtime checks\n\nPython's type system is strongly typed, but very dynamic. For all its benefits, this means\nsome bugs can try to creep in, which more statically typed languages (like Java) would catch\nat compile time. Looking beyond even that, you may want to enforce more sophisticated,\ncustom checks on data going in or out. Decorators can let you easily handle all of this, and\napply it to many functions at once.\n\nImagine this: you have a set of functions, each returning a dictionary, which (among other\nfields) includes a field called \"summary.\" The value of this summary must not be more than\n30 characters long; if violated, that's an error. Here is a decorator that raises a\nValueError if that happens:\n\nRetry\n\nImagine a situation where your defined callable fails due to some I/O related issues and\nyou'd like to retry that again. Decorator can help you to achieve that in a reusable manner.\nLet's define a retry decorator that will rerun the decorated function multiple times if an\nHTTP error occurs.\n\nApplying multiple decorators\n\nYou can apply multiple decorators to a function by stacking them on top of each other. Let's\ndefine two simple decorators and use them both on a function.\n\nThe decorators are called in a bottom up order. First, the decorator greet gets applied on\nthe result of getname function and then the result of greet gets passed to the flare\ndecorator. The decorator stack above can be written as flare(greet(getname(name))). Change\nthe order of the decorators and see what happens!\n\nDecorators with arguments\n\nWhile defining the retry decorator in the previous section, you may have noticed that I've\nhard coded the number of times I'd like the function to retry if an error occurs. It'd be\nhandy if you could inject the number of tries as a parameter into the decorator and make it\nwork accordingly. This isn't a trivial task and you'll need three levels of nested functions\nto achieve that.\n\nBefore doing that let's cook up a trivial example of how you can define decorators with\nparameters.\n\nThe decorator joinby takes a single parameter called delimiter. It splits the string\noutput of the decorated function by a single space and then joins them using the user\ndefined delimiter specified in the delimiter argument. The three layer nested definition\nlooks scary but we'll get to that in a moment. Notice how you can use the decorator with\ndifferent parameters. In the above example, I've defined three different functions to\ndemonstrate the usage of joinby. It's important to note that in case of a decorator that\ntakes parameters, you'll always need to pass something to it and even if you don't want to\npass any parameter (run with the default), you'll still need to decorate your function with\ndeco() instead of deco. Try changing the decorator on the goodbye function from\njoinby() to joinby and see what happens.\n\nTypically, a decorator creates and returns an inner wrapper function but here in the\nrepeat decorator, there is an inner function within another inner function. This almost\nlooks like a _dream within a dream_ from the movie Inception.\n\nThere are a few subtle things happening in the joinby() function:\n\n- Defining outer_wrapper() as an inner function means that repeat() will refer to a\n  function object outer_wrapper.\n\n- The delimiter argument is seemingly not used in joinby() itself. But by passing\n  delimiter a closure is created where the value of delimiter is stored until it will be\n  used later by inner_wrapper()\n\nDecorators with & without arguments\n\nYou saw earlier that a decorator specifically designed to take parameters can't be used\nwithout parameters; you need to at least apply parenthesis after the decorator deco() to\nuse it without explicitly providing the arguments. But what if you want to design one that\ncan used both with and without arguments. Let's redefine the joinby decorator so that you\ncan use it with parameters or just like an ordinary parameter-less decorator that we've seen\nbefore.\n\nHere, the _func argument acts as a marker, noting whether the decorator has been called\nwith arguments or not:\n\nIf joinby has been called without arguments, the decorated function will be passed in as\n_func. If it has been called with arguments, then _func will be None. The \\ in the\nargument list means that the remaining arguments can't be called as positional arguments.\nThis time you can use joinby with or without arguments and function hello and greet\nabove demonstrate that.\n\nA generic pattern\n\nPersonally, I find it cumbersome how you need three layers of nested functions to define a\ngeneralized decorator that can be used with or without arguments. David Beazley in his\n[Python Cookbook] shows an excellent way to define generalized decorators without writing\nthree levels of nested functions. It uses the built in functools.partial function to\nachieve that. The following is a pattern you can use to define generalized decorators in a\nmore elegant way:\n\nLet's redefine our retry decorator using this pattern.\n\nIn this case, you don't have to write three level nested functions and the\nfunctools. partial takes care of that. Partials can be used to make new derived functions\nthat have some input parameters pre-assigned.Roughly partial does the following:\n\nThis eliminates the need to write multiple layers of nested factory function get a\ngeneralized decorator.\n\nDefining decorators with classes\n\nThis time, I'll be using a class to compose a decorator. Classes can be handy to avoid\nnested architecture while writing decorators. Also, it can be helpful to use a class while\nwriting stateful decorators. You can follow the pattern below to compose decorators with\nclasses.\n\nLet's use the above template to write a decorator named Emphasis that will add bold tags\n<b></b>to the string output of a function.\n\nThe __init__() method stores a reference to the function num_calls and can do other\nnecessary initialization. The __call__() method will be called instead of the decorated\nfunction. It does essentially the same thing as the wrapper() function in our earlier\nexamples. Note that you need to use the functools.update_wrapper() function instead of\n@functools.wraps.\n\nBefore moving on, let's write a stateful decorator using classes. Stateful decorators can\nremember the state of their previous run. Here's a stateful decorator called Tally that'll\nkeep track of the number of times decorated functions are called in a dictionary. The keys\nof the dictionary will hold the names of the functions and the corresponding values will\nhold the call count.\n\nA few more examples\n\nCaching return values\n\nDecorators can provide an elegant way of memoizing function return values. Imagine you have\nan expensive API and you'd like call that as few times as possible. The idea is to save and\ncache values returned by the API for particular arguments, so that if those arguments appear\nagain, you can serve the results from the cache instead of calling the API again. This can\ndramatically improve your applications' performance. Here I've simulated an expensive API\ncall and provided caching with a decorator.\n\nYou'll see that running this function takes roughly 3 seconds. To cache the result, we can\nuse Python's built in functools.lru_cache to save the result against an argument in a\ndictionary and serve that when it encounters the same argument again. The only drawback here\nis, all the arguments need to be hashable.\n\nLeast Recently Used (LRU) Cache organizes items in order of use, allowing you to quickly\nidentify which item hasn't been used for the longest amount of time. In the above case, the\nparameter max_size refers to the maximum numbers of responses to be saved up before it\nstarts deleting the earliest ones. While you run the decorated function, you'll see first\ntime it'll take roughly 3 seconds to return the result. But if you rerun the function again\nwith the same parameter it'll spit the result from the cache almost instantly.\n\nUnit Conversion\n\nThe following decorator converts length from SI units to multiple other units without\npolluting your target function with conversion logics.\n\nLet's use that on a function that returns the area of a rectangle.\n\nUsing the convert decorator on the area function shows how it prints out the transformation\nunit before returning the desired result. Experiment with other conversion units and see\nwhat happens.\n\nFunction registration\n\nThe following is an example of registering logger function in Flask framework. The decorator\nregister_logger doesn't make any change to the decorated logger function. Rather it\ntakes the function and registers it in a list called logger_list every time it's invoked.\n\nIf you run the server and hit the http://localhost:5000/ url, it'll greet you with a\nHello World! message. Also you'll able to see the printed method and path of your HTTP\nrequest on the terminal. Moreover, if you inspect the logger_list, you'll find the\nregistered logger there. You'll find a lot more real life usage of decorators in the Flask\nframework.\n\nFurther reading\n\n- [Primer on Python decorator - Real Python]\n- [Decorators in Python - Datacamp]\n- [5 reasons you need to write python decorators]\n\n\n\n\n[python cookbook]:\n    https://realpython.com/asins/1449340377/\n\n[primer on python decorator - real python]:\n    https://realpython.com/primer-on-python-decorators/\n\n[decorators in python - datacamp]:\n    https://www.datacamp.com/community/tutorials/decorators-python\n\n[5 reasons you need to write python decorators]:\n    https://www.oreilly.com/content/5-reasons-you-need-to-learn-to-write-python-decorators/",
  "title": "Untangling Python decorators"
}