Deciphering Python's metaclasses
Updated on 2023-09-11: Fix broken URLs.
In Python, metaclass is one of the few tools that enables you to inject metaprogramming capabilities into your code. The term metaprogramming refers to the potential for a program to manipulate itself in a self referential manner. However, messing with metaclasses is often considered an arcane art that's beyond the grasp of the plebeians. Heck, even Tim Peters advises you to tread carefully while dealing with these.
Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't (the people who actually need them know with certainty that they need them, and don't need an explanation about why).
Metaclasses are an esoteric OOP concept, lurking behind virtually all Python code. Every Python class that you create is attached to a default metaclass and Python cleverly abstracts away all the meta-magics. So, you're indirectly using them all the time whether you are aware of it or not. For the most part, you don't need to be aware of it. Most Python programmers rarely, if ever, have to think about metaclasses. This makes metaclasses exciting for me and I want to explore them in this post to formulate my own judgement.
Metaclasses
A metaclass is a class whose instances are classes. Like an "ordinary" class defines the behavior of the instances of the class, a metaclass defines the behavior of classes and their instances.
Metaclasses aren't supported by every object oriented programming language. Those programming language, which support metaclasses, considerably vary in way they implement them. Python provides you a way to get under the hood and define custom metaclasses.
Understanding type and class
In Python, everything is an object. Classes are objects as well. As a result, all classes must have corresponding types. You deal with built in types like int, float, list etc all the time. Consider this example:
In the above example, variable a is an instance of the built in class int. Type of a is int and the type of int is type. User defined classes also show similar behavior. For example:
Here, I've defined another class named Foo and created an instance a of the class. Applying type on instance a reveals its type as main.Foo and applying type on class Foo reveals the type as type. So here, we can use the term class and type interchangeably. This brings up the question:
What on earth this type (function? class?) thing actually is and what is the type of type?
Let's apply type on type:
Weird. The type of any class (not instance of a class) in Python is type and the type of type is also type. By now, you've probably guessed that type is a very special class in Python that can reveal the type of itself and of any other class or object. In fact, type is a metaclass and all the classes in Python are instances of it. You can inspect that easily:
The the last line of the above code snippet demonstrates that type is also an instance of metaclass type. Normally, you can't write self referential classes like that in pure Python. However, you can circumvent this limitation by subclassing from type. This enables you to write custom metaclasses that you can use to dictate and mutate the way classes are created and instantiated. From now on, I'll be referring to the instance class of a metaclass as target class. Let's create a custom metaclass that just prints the name of the target class while creating it:
Despite the fact that we haven't called class A or created an instance of it, the new method of metaclass PrintMeta was executed and printed the name of the target class. In the return statement of new method, super() was used to call the new method of the base class (type) of the metaclass PrintMeta.
Special methods used by metaclasses
Type type, as the default metaclass in Python, defines a few special methods that new metaclasses can override to implement unique code generation behavior. Here is a brief overview of these "magic" methods that exist on a metaclass:
- new: This method is called on the Metaclass before an instance of a class based on the metaclass is created
- init: This method is called to set up values after the instance/object is created
- prepare: Defines the class namespace in a mapping that stores the attributes
- call: This method is called when the constructor of the new class is to be used to create an object
These are the methods to override in your custom metaclass to give your classes behaviors different from that of type. The following example shows the default behaviors of these special methods and their execution order.
Some people immediately think of init, and I've occasionally called it "the constructor" myself; but in actuality, as its name indicates, it's an initializer and by the time it's invoked, the object has already been created, seeing as it's passed in as self. The real constructor is a far less famous function: new. The reason you might never hear about it or use it - is that allocation doesn't mean that much in Python, which manages memory for you. So if you do override new, it'd be just like your init - except you'll have to call into Python to actually create the object, and then return that object afterward.
Pay attention to the execution order of the special methods of the custom metaclass ExampleMeta. The prepare method is called first and is followed by new, init and call respectively. Only after that the first method init of the target class A is called. This is important since it'll help you to decide how you'll mutate and change the behavior of your target class.
Metaclass conflicts
Note that the metaclass argument is singular – you can't attach more than one metaclass to a class. However, through multiple inheritance you can accidentally end up with more than one metaclass, and this produces a conflict which must be resolved.
Class First and Second are attached to different metaclasses and class Third inherits from both of them. Since metaclasses trickle down to subclasses, class Third is now effective attached to two metaclasses (FirstMeta and SecondMeta). This will raise TypeError. Attachment with only one metaclass is allowed here.
Examples in the wild
In this section, I'll go through a few real life examples where metaclasses can provide viable solutions to several tricky problems that you might encounter during software development. The solutions might appear over-engineered in some cases and almost all of them can be solved without using metaclasses. However, the purpose is to peek into the inner wirings of metaclasses and see how they can offer alternative solutions.
Simple logging with metaclasses
The goal here is to log a few basic information about a class without directly adding any logging statements to it. Instead, you can whip up a custom metaclass to perform some metaprogramming and add those statements to the target class without mutating it explicitly.
In the above example, I've created a metaclass called LittleMeta and added the necessary logging statements to record the information about the target class. Since the logging statements are residing in the new method of the metaclass, these informations will be logged before the creation of the target class. In the target class Point, LittleMeta replaces the default type metaclass and produces the desired result by mutating the class.
Returning class attributes in a custom list
In this case, I want to dynamically attach a new attribute to the target class called attrs_ordered. Accessing this attribute from the target class (or instance) will give out an alphabetically sorted list of the attribute names. Here, the prepare method inside the metaclass AttrsListMeta returns an OrderDict instead of a simple dict - so all attributes gathered before the new method call will be ordered. Just like the previous example, here, the new method inside the metaclass implements the logic required to get the sorted list of the attribute names.
You can access the attrs_ordered attribute from both class A and an instance of class A. Try removing the sorted() function inside the new method of the metaclass and see what happens!
Creating a singleton class
In OOP term, a singleton class is a class that can have only one object (an instance of the class) at a time.
After the first time, if you try to instantiate a Singleton class, it will basically return the same instance of the class that was created before. So any modifications done to this apparently new instance will mutate the original instance since they're basically the same instance.
In the above example, at first, I've created a singleton class A by attaching the Singleton metaclass to it. Secondly, I've instantiated class A and assigned the instance of the class to a variable a. Thirdly, I've instantiated the class again and assigned variable a b to this seemingly new instance. Checking the identity of the two variables a and b reveals that both of them are actually the same object.
Implementing a class that can't be subclassed
Suppose you want to create a base class where the users of your class won't be able to create any subclasses from the base class. In that case, you can write a metaclass and attach that your base class. The base class will raise RuntimeError if someone tries to create a subclass from it.
Disallowing multiple inheritance
Multiple inheritance can be fragile and error prone. So, if you don't want to allow the users to use a base class with any other base classes to form multiple inheritance, you can do so by attaching a metaclass to that target base class.
Timing classes with metaclasses
Suppose you want to measure the execution time of different methods of a class. One way of doing that is to define a timer decorator and decorating all the methods to measure and show the execution time. However, by using a metaclass, you can avoid decorating the methods in the class individually and the metaclass will dynamically apply the timer decorator to all of the methods of your target class. This can reduce code repetition and improve code readability.
Registering plugins with metaclasses
Suppose a specific single class represents a plugin in your code. You can write a metaclass to keep track of all of the plugins so than you don't have to count them manually.
Debugging methods with metaclasses
Debugging a class often involves inspecting the individual methods and adding extra debugging logic to those. However, this can get tedious if you've do this over an over again. Instead, you can write an inspection decorator and use a metaclass to dynamically apply the decorator to all of the methods of your target class. Later on, you can simply detach the metaclass once you're done with debugging and don't want the extra logic in your target class.
Exception handling with metaclasses
Sometimes you need to handle exceptions in multiple methods of a class in a generic manner. That means all the methods of the class have the same exception handling, logging logic etc. Metaclasses can help you avoid adding repetitive exception handling and logging logics to your methods.
Abstract base classes
An abstract class can be regarded as a blueprint for other classes. It allows you to provide a set of methods that must be implemented within any child classes built from the abstract class. Abstract classes usually house multiple abstract methods. An abstract method is a method that has a declaration but does not have an implementation.
When you want to provide a common interface for different implementations of a component, abstract classes are the way to go. You can't directly initialize or use an abstract class. Rather, you've to subclass the abstract base class and provide concrete implementations of all the abstract methods. Python has a dedicated abc module to help you create abstract classes. Let's see how you can define a simple abstract class that provides four abstract methods:
Although it seems like interface ICalc is simply inheriting from the class ABC, in fact, ABC is attaching a metaclass ABCMeta to ICalc. This metaclass transforms the ICalc class into an abstract class. You can see that the class ICalc gives TypeError when you take an attempt to initialize it. The only way to use this interface is via creating subclasses from ICalc base class and implementing all the abstract methods there. The snippet below shows that:
Metaclasses & dataclasses
Data classes were introduced to python in version 3.7. Basically they can be regarded as code generators that reduce the amount of boilerplate you need to write while generating generic classes. Dataclasses automatically create init, repr, eq, gt, lt etc methods without you having to add them explicitly. This can be very handy when you need to create custom collections for your data. You can create dataclasses in the following manner:
Creating multiple dataclasses
Avoiding dataclass decorator with metaclasses
Now, one thing that I find cumbersome while creating multiple dataclasses is having to attach the @dataclasses.dataclass decorator to each of the dataclasses. Also, the decorator takes multiple arguments to customize the dataclass behavior and it can quickly get cumbersome when you've to create multiple dataclasses with custom behavior. Moreover, this goes against the DRY (Don't Repeat Yourself) principle in software engineering.
To avoid this, you can write a metaclass that will automatically apply the customized dataclass decorator to all of the target classes implicitly. All you have to do is to attach the metaclass to a base dataclass and inherit from it in the later dataclasses that need to be created.
Should you use it?
Almost all of the problems you've encountered above can be solved without using metaclasses. Decorators can also be exclusively used to perform metaprogramming in a more manageable and subjectively cleaner way. One case where you absolutely have to use metaclasses is to avoid applying decorators to multiple classes or methods repetitively.
Also, metaclasses can easily veer into the realm of being a "solution in search of a problem". If the problem at hand can be solved in a simpler way, it probably should be. However, I still think that you should at least try to understand how metaclasses work to have a better grasp on how Python classes work in general and can recognize when a metaclass really is the appropriate tool to use.
Remarks
Wrapping your mind around metaclasses can be tricky. So, to avoid any unnecessary confusion, I've entirely evaded any discussion regarding the behavioral difference between old style classes and new style classes in Python. Also, I've intentionally excluded mentioning the differences between type in Python 2 and type in Python 3 entirely. Python 2.x has reached its EOL. Save yourself some trouble and switch to Python 3.x if you already haven't done so.
This article assumes familiarity with decorators, dataclasses etc. If your knowledge on them is rusty, checkout these posts on decorators and dataclasses.
Further reading
Discussion in the ATmosphere