{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/python/lru-cache-on-methods/",
  "description": "Avoid memory leaks when caching instance methods with lru_cache by making cache containers local to instances instead of global.",
  "path": "/python/lru-cache-on-methods/",
  "publishedAt": "2022-01-15T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python",
    "TIL",
    "Performance"
  ],
  "textContent": "Recently, fell into this trap as I wanted to speed up a slow instance method by caching it.\n\n> When you decorate an instance method with functools.lru_cache decorator, the instances\n> of the class encapsulating that method never get garbage collected within the lifetime of\n> the process holding them.\n\nLet's consider this example:\n\nHere, I've created a simple SlowAdder class that accepts a delay value; then it sleeps\nfor delay seconds and calculates the sum of the inputs in the calculate method. To avoid\nthis slow recalculation for the same arguments, the calculate method was wrapped in the\nlru_cache decorator. The __del__ method notifies us when the garbage collection has\nsuccessfully cleaned up instances of the class.\n\nIf you run this program, it'll print this:\n\nYou can see that the lru_cache decorator is doing its job. The second call to the\ncalculate method with the same argument took noticeably less time compared to the first\none. In the second case, the lru_cache decorator is just doing a simple dictionary lookup.\nThis is all good but the instances of the ShowAdder class never get garbage collected in\nthe lifetime of the program. Let's prove that in the next section.\n\nGarbage collector can't clear up the affected instances\n\nIf you execute the above snippet with an -i flag, we can interactively prove that no\ngarbage collection takes place. Let's do it:\n\nHere on the REPL, you can see that I've reassigned slow_adder to None and then explicitly\ntriggered the garbage collector. However, we don't see the message in the __del__ method\nprinted here and the output of gc.collect() is 0. This implies that something is holding a\nreference to the slow_adder instance and the garbage collector can't clear up the object.\nLet's inspect who has that reference:\n\nThe cache_info() is showing that the cache container keeps a reference to the instance\nuntil it gets cleared. When I manually cleared the cache and reassigned the variable\nslow_adder to None, only then did the garbage collector remove the instance. By default,\nthe size of the lru_cache is 128 but if I had applied lru_cache(maxsize=None), that\nwould've kept the cache forever and the garbage collector would wait for the reference count\nto drop to zero but that'd never happen within the lifetime of the process.\n\nThis can be dangerous if you create millions of instances and they don't get garbage\ncollected naturally. It can overflow your working memory and cause the process to crash! I\naccidentally did it where the infected class was being instantiated millions of times via\nHTTP API requests.\n\nThe solution\n\nTo solve this, we'll have to make the cache containers local to the instances so that the\nreference from cache to the instance gets scraped off with the instance. Here's how you can\ndo that:\n\nThe only difference here is - instead of decorating the method directly, I called the\ndecorator function on the _calculate method just as a regular function and saved the\nresult as an instance variable named calculate. The instances of this class get garbage\ncollected as usual.\n\nNotice that this time, clearing out the cache wasn't necessary. I had to call gc.collect()\nto invoke explicit garbage collection. That's because this shenanigan creates cyclical\nreferences and the GC needs to do some special magic to clear the memory. In real code,\nPython interpreter will clean this up for you in the background without you having to call\nthe GC.\n\nThe self dilemma\n\nEven after applying the solution above, a weird thing happens in the case of instance\nmethods. Let's run the src_2.py script interactively to demonstrate that:\n\nHere, I've created another instance of the SlowAdder class and called calculate with the\nsame arguments. But whenever I called the calculate method on the slow_adder_2 instance\nwith the same parameters as before, the first time, it recalculated it instead of returning\nthe result from the cache. How come!\n\nUnderneath, the lru_cache decorator uses a dictionary to cache the calculated values. A\n[hash function is applied] to all the parameters of the target function to build the key of\nthe dictionary, and the value is the return value of the function when those parameters are\nthe inputs. This means, the first argument self also gets included while building the\ncache key. However, for different instances, this self object is going to be different and\nthat makes the hashed key of the cache different for every instance even if the other\nparameters are the same.\n\nBut what about class methods & static methods\n\nClass methods and static methods don't suffer from the above issues as they don't have any\nties to their respective instances. In their case, the cache container is local to the\nclass, not the instances. Here, you can stack the lru_cache decorator as usual. Let's\ndemonstrate that for classmethod first:\n\nYou can inspect the garbage collection behavior here:\n\nStatic methods behave exactly the same. You can use the lru_cache decorator in similar\nfashion as below:\n\nFurther reading\n\n- [functools.lru_cache - Python Docs]\n- [Don't lru_cache methods! (intermediate) anthony explains #382]\n- [Python LRU cache in a class disregards maxsize limit when decorated with a staticmethod\n  or classmethod decorator]\n\n\n\n\n[hash function is applied]:\n    https://github.com/python/cpython/blob/8882b30dab237c8b460cb8d18cecc8b8d031da25/Lib/functools.py#L448\n\n[functools.lru_cache - Python Docs]:\n    https://docs.python.org/3/library/functools.html#functools.lru_cache\n\n[don't lru_cache methods! (intermediate) anthony explains #382]:\n    https://www.youtube.com/watch?v=sVjtp6tGo0g\n\n[python lru cache in a class disregards maxsize limit when decorated with a staticmethod or classmethod decorator]:\n    https://stackoverflow.com/questions/70409673/python-lru-cache-in-a-class-disregards-maxsize-limit-when-decorated-with-a-stati",
  "title": "Don't wrap instance methods with 'functools.lru_cache' decorator in Python"
}