{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/python/escape-template-pattern/",
  "description": "Replace inheritance-based template pattern with composition and Protocol types to create cleaner, testable Python code without namespace pollution.",
  "path": "/python/escape-template-pattern/",
  "publishedAt": "2023-07-01T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python",
    "Design Patterns"
  ],
  "textContent": "Over the years, I've used the [template pattern] across multiple OO languages with varying\ndegrees of success. It was one of the first patterns I learned in the primordial hours of my\nsoftware engineering career, and for some reason, it just feels like the natural way to\ntackle many real-world code-sharing problems. Yet, even before I jumped on board with the\n[composition over inheritance] camp, I couldn't help but notice how using this particular\ninheritance technique spawns all sorts of design and maintenance headaches as the codebase\nstarts to grow.\n\nAn epiphany\n\nThis isn't an attempt to explain why you should prefer composition over inheritance\n(although you should), as it's a vast topic and much has been said regarding this. Also,\nonly after a few years of reading concomitant literatures and making enough mistakes in\nreal-life codebases, it dawned on me that opting for inheritance as the default option leads\nto a fragile design. So I won't even attempt to tackle that in a single post and will refer\nto a few fantastic prior arts that proselytized me to the composition cult.\n\nThe goal of this article is not to focus on the wider spectrum of how to transform\nsubclass-based APIs to use composition but rather to zoom in specifically on the template\npattern and propose an alternative way to solve a problem where this pattern most naturally\nmanifests. In the first portion, the post will explain what the template pattern is and how\nit gradually leads to an intractable mess as the code grows. In the latter segments, I'll\ndemonstrate how I've designed a real-world service by adopting the obvious and natural path\nof inheritance-driven architecture. Then, I'll explain how the service can be refactored to\nescape the quagmire that I've now started to refer to as the template pattern hellscape.\n\nOnly a few moons ago, while watching Hynek Schlawack's Python 2023 talk aptly titled\n[Subclassing, Composition, Python, and You] and reading his fantastic blog post [Subclassing\nin Python Redux], the concept of adopting composition to gradually phase out\nsubclass-oriented design from my code finally clicked for me. However, it's not always\nobvious to me how to locate inheritance metastasis and exactly where to intervene to make\nthe design better. This post is my attempt to distill some of my learning from those\nresources and focus on improving only a small part of the gamut.\n\nThe infectious template pattern\n\nYou're consciously or subconsciously implementing the template pattern when your API design\nfollows these steps:\n\n- You have an Abstract Base Class (ABC) with abstract methods.\n- The ABC also includes one or more concrete methods.\n- The concrete methods in the ABC depend on the concrete implementation of the abstract\n  methods.\n- API users are expected to inherit from the ABC and provide concrete implementations for\n  the abstract methods.\n- Users then utilize the concrete methods defined in the ABC class.\n\nThis pattern enables the sharing of concrete method implementations with subclasses.\nHowever, the concrete methods of the baseclass are only valid when the user inherits from\nthe base and implements the abstract methods. Attempting to instantiate the baseclass\nwithout implementing the abstract methods will result in a TypeError. Only the subclass\ncan be initialized once all the abstract methods have been implemented.\n\nObserve this example:\n\nThis is how you use it:\n\nHere, the abstract Base class is defined by inheriting from the abc.ABC class. Inside\nBase, there's a concrete_method that relies on an abstract_method. The\nconcrete_method is defined to call abstract_method, expecting that subclasses will\nprovide their own implementation of abstract_method. If a subclass of Base fails to\nimplement abstract_method, calling concrete_method on an instance of that subclass will\nraise a NotImplementedError.\n\nThe snippet also provides an example subclass called Sub, which inherits from Base.\nSub overrides the abstract_method and provides its own implementation. In this case, it\njust prints a statement. By subclassing Base and implementing abstract_method, Sub\nbecomes a concrete class that can be instantiated. The purpose of this design pattern is to\ndefine a common interface through the Base class, with the expectation that subclasses\nwill implement specific behavior by overriding the abstract methods, while still providing a\nway to call those methods through the concrete methods defined in the baseclass. This\nseemingly innocuous and often convenient bi-directional relationship between the base and\nsub class tends to become infectious and introduces complexity into all the subclasses that\ninherit from the base.\n\nThe dark side of the moon\n\nTemplate pattern seems like the obvious way of sharing code and it almost always is one of\nthe first things that people learn while familiarizing themselves with how OO works in\nPython. Plus, it's used extensively in the standard library. For example, in the\ncollections.abc module, there are a few ABCs that you can subclass to build your own\ncontainers. I [wrote about this] a few years back. Here's how you can subclass\ncollections.abc.Sequence to implement a tuple-like immutable datastructure:\n\nYou'd use the class as such:\n\nWe're inheriting from the Sequence ABC and implementing the required abstract methods.\nHere's the first issue: how do we even know which methods to implement and which methods we\nget for free? You can consult the [Sequence documentation] and learn that __getitem__ and\n__len__ are the abstract methods that subclasses are expected to implement. In return, the\nbase Sequence class gives you __contains__, __iter__, __reversed__, index, and\ncount as mixin methods. You can also print out the abstract methods by accessing the\nSequence.__abstractmethod__ attribute. Sure, you're getting a lot of concrete methods for\nfree, but suddenly you're dependent on some out-of-band information to learn about the\nbehavior of your specialized CustomSequence class.\n\nThe following three sections will briefly explore the issues that deceptively creep up on\nyour codebase when you opt for the template pattern.\n\nElusive public API\n\nYou've already seen the manifestation of this issue in the CustomSequence example. The\nsubclass-oriented code-sharing pattern like this makes it difficult to discover the public\nAPI of your specialized class because many of its functionalities come from the concrete\nmixin methods provided by the base Sequence class. Now, this isn't too terrible for a tool\nin the standard library since they're usually quite well-documented, and you can always\nresort to inspecting the subclass instance to learn about the abstract and concrete methods.\n\nNot all subclass-driven design is bad, and the standard library makes judicious use of the\ntemplate pattern. However, in an application codebase that you might be writing, this\nelusive nature of the public API can start becoming recalcitrant. Your code may not be as\nwell-documented as the standard library, or instantiating the subclass may be expensive,\nmaking introspection difficult. You're basically trading off readability for writing\nergonomics. There's nothing wrong in doing that as long as you're aware of the tradeoffs.\nAll I'm trying to say is that it's a non-ideal default.\n\nNamespace pollution\n\nIf you introspect the previously defined subclass with dir(CustomSequence), you'll get the\nfollowing result. I've removed the common attributes that every class inherits from object\nfor brevity and annotated the abstract and mixin method names for clarity.\n\nFrom the above list, it's evident that all the methods from the baseclass and the subclass\nlive in the same namespace. The moment you're inheriting from some baseclass, you have no\ncontrol over what that class is bringing over to your subclass's namespace and effectively\npolluting it. It's like a more sneaky version of from foo import .\n\nThis flat namespacing makes it hard to understand which method is coming from where. In the\nabove case, without the annotations, you'd have a hard time discerning between the methods\nthat you implemented and the alien methods from the baseclass. This isn't a cardinal sin in\nthe Python realm if that's what you want, but it's certainly a suboptimal default.\n\nSRP violation & goose chase program flow\n\nThe biggest complaint I have against the template pattern is how it encourages the baseclass\nto do too many things at once. I can endure poor discoverability of public APIs and\nnamespace pollution to some extent, but when a class tries to do too many things\nsimultaneously, it eventually exhibits the tendency to give birth to [God Objects];\nbreaching the SRP (Single Responsibility Principle).\n\nIntentionally violating the SRP rarely fosters good results, and in this case, the baseclass\ndefines both concrete and abstract methods. Not only that, the base expects the subclasses\nto implement those abstract methods so that it can use them in its concrete method\nimplementation. Just reading back this sentence is giving me a headache. If you design your\nAPIs in this manner, you'll have to carefully read through both the sub and the base class\nimplementations to understand how this intricate bi-directional thread is woven into your\nprogram flow. This seems easy enough in a simple example where you can see both the base and\nthe sub class in a single snippet, but it quickly gets out of hand when large base and sub\nclasses are scattered across multiple modules. You'll need to perform the mental gymnastics\nof tracking this back-and-forth logic, aka the abominable goose chase program flow.\n\nThe disease and the cure\n\nLet's examine a specific design problem and observe how it can be modeled using the template\npattern. Then, we'll explore an alternative solution that replaces the inheritance-driven\ndesign with composition.\n\nDesigning with template pattern\n\nThe following code snippet mimics a real-world [webhook] dispatcher that takes a message and\nposts it to a callback URL via HTTP POST request. First, we'll commit the cardinal sin of\nmodeling the domain with the template pattern and then we'll try to find a way out of the\nquandary. Here it goes:\n\nHere's how you'll orchestrate the classes:\n\nWe start by defining an immutable Message container to store our webhook message. Next, we\nwrite an abstract BaseWebhook class that inherits from abc.ABC. This class serves as a\ntemplate for the webhook functionality and declares two abstract methods: get_message()\nand get_url(). Type annotations are used to indicate the return types of these methods.\nAny subclasses derived from BaseWebhook must implement these abstract methods. The\nsend() method, implemented in the baseclass, uses the concrete implementations of the\nabstract methods to perform webhook dispatching. In this case, we simulate the HTTP POST\nfunctionality by printing the message and destination URL.\n\nThe Webhook class is a subclass of BaseWebhook and provides concrete implementations of\nthe abstract methods. It accepts a single Message object as a parameter in its\nconstructor. The get_url() method returns a fixed URL, while the get_message() method\nconverts the Message object into a serializable dictionary representation using\ndataclasses.asdict().\n\nIn this structure, the user of the Webhook class only needs to initialize the class and\ncall the send() method on the instance. The send() method, however, lives in the\nBaseWebhook class, not the specialized Webhook subclass. It utilizes the concrete\nimplementations of abstract methods to deliver the send() functionality. In the following\nsection, we'll explore a method to avoid this weird back-and-forth program flow.\n\nFinding salvation in strategy pattern\n\nThere are multiple ways and conflicting opinions on how to get out of the hole we've dug for\nourselves. Some even like to spend more time prattling around the philosophy of how OO is\nterrible and how, if it weren't for Java's huge influence on Python, we wouldn't be in this\nmess, rather than attempting to solve the actual problem. So instead of trying to cover\nevery possible solution under the sun, I'll go through the one that has worked for me fairly\nwell.\n\nWe'll refactor the code in the previous section to take advantage of composition and\n[structural subtyping] support in Python. Long story short, structural subtyping refers to\nthe ability to ensure type safety based on the structure or shape of an object rather than\nits explicit inheritance hierarchy. This allows us to define and enforce contracts based on\nthe presence of specific attributes or methods, rather than relying on a specific class or\ninheritance relationship.\n\nThis is achieved through the use of the typing.Protocol class introduced in Python 3.8. By\ndefining a protocol using the typing.Protocol class, we can specify the expected\nattributes and methods that an object should have to satisfy the protocol. Any object that\nmatches the structure defined by the protocol can be treated as if it conforms to that\nprotocol, enabling more flexible and dynamic type-checking in Python. This conformity is\nusually checked by a type-checking tool like [mypy]. If you want to learn more, check out\nGlyph's post titled [I Want a New Duck]. Here's how I refactored it:\n\nThe classes can be wired together as follows:\n\nWe've agreed that the BaseWebhook class tries to do too many things at once. The first\nstep to disentangling a class is to identify its responsibilities and create multiple\ncomponent classes where each new class will only have one responsibility. Here, the base\nclass retrieves the necessary data and dispatches the webhook using that data at the same\ntime. The Retriever and Dispatcher protocol classes will formalize the shape and\nstructure of those component classes. These protocols work like the ABCs, but you don't need\nto inherit from them to ensure interface conformity; the type checker will do it for you.\n\nThe Retriever class has two methods: get_message and get_url, which fetch message and\nURL data respectively. Similarly, the Dispatcher protocol has only a dispatch method\nthat sends the webhook. In either case, the protocol methods don't implement anything; they\nwork just like the abstract methods of the ABCs, and the protocol classes themselves can't\nbe instantiated. Then the HookRetriever and HookDispatcher components implicitly\nimplement the protocol classes. Notice that neither of the components inherits from the\nprotocol classes. The type checker will ensure that they conform to the defined protocols.\n\nThe question is, how does the type checker know which class is supposed to conform to which\nprotocol? The answer lies in the final Webhook class. We define a final dataclass that\ntakes instances of the Message, Retriever, and Dispatcher classes in the constructor.\nNotice that while adding type hints to the retriever and dispatcher parameters of the\ndataclass constructor, we're using the protocol classes instead of the concrete ones. This\nis how the type checker knows that whatever instance is passed to the retriever and\ndispatcher parameters must conform to the Retriever and Dispatcher protocols,\nrespectively. Note that we've completely eliminated subclassing from our public API.\nInjecting dependencies in this manner is also known as the [strategy pattern].\n\nThe Webhook class now has a hierarchical namespace instead of a flat one, unlike our\ninheritance-based friend. You'll have to be explicit about where a method is coming from\nwhen calling it. So if you need to access the fetched URL, you'll need to explicitly call\nself.retriever.get_url(). The self namespace has only one user-defined public method,\n.send(), which can be called to dispatch the webhook from a Webhook instance. This also\nmeans you no longer have to deal with goose chase program flow since all the dependencies\nflow towards the final Webhook class.\n\nOn the flip side, you'll need to do more work while initializing the Webhook class. The\nMessage, HookRetriever, and HookDispatcher classes need to be instantiated first and\nthen passed explicitly to the constructor of the Webhook class to instantiate it. You're\nbasically trading writing ergonomics for readability. Instantiating the template subclass\nwas a lot easier for sure.\n\nTradeoffs\n\nOpting in for composition isn't free, as it usually leads to more verbose code\norchestration. If you're passing all the dependencies explicitly, as shown above, wiring the\ncode together will be more complex. However, in return, you get a more readable and testable\ndesign substrate. So, I'm more than happy to make the tradeoff. Additionally, avoiding\nnamespace pollution means that one attribute access has now turned into two or more\nattribute accesses, which can cause performance issues in tight conditions.\n\nMoreover, you can't just take your inheritance-heavy API and suddenly turn it into a\ncomposable one. It usually requires planning and designing from the ground up, where you\nmight decide that the ROI isn't good enough to justify the effort of refactoring. Plus, in a\nlanguage like Python, you can't always escape inheritance, nor should you try to do so.\n\n> Yet behold, it need not be the customary stratagem that thou graspest at each moment thine\n> heart yearns to commune code amidst classes.\n\nFurther reading\n\n- [End of object inheritance - Augie Fackler, Nathaniel Manista]\n\n\n\n\n[template pattern]:\n    https://refactoring.guru/design-patterns/template-method/python/example\n\n[composition over inheritance]:\n    https://python-patterns.guide/gang-of-four/composition-over-inheritance/\n\n[subclassing, composition, python, and you]:\n    https://www.youtube.com/watch?v=k8MT5liCQ7g\n\n[subclassing in python redux]:\n    https://hynek.me/articles/python-subclassing-redux/\n\n[wrote about this]:\n    /python/mixins/\n\n[sequence documentation]:\n    https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence\n\n[god objects]:\n    https://blog.devgenius.io/code-smell-14-god-objects-b84b75b702\n\n[webhook]:\n    https://zapier.com/blog/what-are-webhooks/\n\n[structural subtyping]:\n    /python/structural-subtyping/\n\n[mypy]:\n    https://mypy-lang.org/\n\n[i want a new duck]:\n    https://blog.glyph.im/2020/07/new-duck.html\n\n[strategy pattern]:\n    https://refactoring.guru/design-patterns/strategy/python/example\n\n[end of object inheritance - augie fackler, nathaniel manista]:\n    https://www.youtube.com/watch?v=3MNVP9-hglc",
  "title": "Escaping the template pattern hellscape in Python"
}