Python is a language that is used for a variety of purposes, and its USP is the simplicity of use by a programmer to solve all types of problems that come up. Around 99.99% of such problems can be solved by using various simple features of the language. But there will be that 0.01% of problems that would need special handling.
Please note that by “simple features” I mean features that are “Pythonic” in nature, and these features conform to the philosophy of the language. Hence, decorators, generators, iterators, classes, etc. come under the head of “simple features” since they all conform to the philosophy of python, which, in short, may be stated as follows:
Philosophy of Python
Python is a language that is easy to read, and the code itself may serve as technical documentation for the (technical) person reading the code.” Hence python code is easily maintainable. If you want to know more about the philosophy of Python, please take a look at this page: https://www.python-course.eu/python3_history_and_philosophy.php.
When I started in the software industry some fourteen years back, I used to do a lot of Perl and C coding. While C/C++ programs are moderately maintainable (that is if the coder had left enough comments on critical and complex parts of the codebase), maintaining and debugging Perl programs were a nightmare. Most of the time, what I used to do was when I was given a Perl code that doesn’t work, I used to figure out (from whoever gave me the code) what the code is supposed to do, and I used to write that part from scratch. I worked on Perl for about 8 years, and then I started getting exposed to Python development services, and it was love at first sight. (Nevertheless, I still do Perl and C/C++ intermittently nowadays).
However, whether meta-programming (and metaclasses) conform to that philosophy is highly debatable. It is because metaprogramming, in particular, manipulates written code, and the meaning (and purpose) of that code is dynamically determined. Hence, reading such code does not provide the reader with an immediate understanding of what the code is trying to do. And the reader has to understand it by looking at it from various contexts.
Well, enough talking, let’s see how we can use meta-programming in python. I will start with simple examples (sample codes), and as we go, we will see more complex scenarios. But before that, let us list down some facts that we need to remember while doing meta-programming and implementing meta-classes:
Examples and Notes
- In python, almost everything is an object (Please note I wrote “almost”. So we will take a look at what other types of entities we have in python).
- To implement meta-classes, you need to derive that class from ‘type’, which is not an object. You may also derive your class from another class that is derived from ‘type’, and you would successfully create a meta-class.
- Quite a lot of things we would be doing using meta-classes can also be done using decorators. However, as I had mentioned above, there would be a few actions that would need to be implemented using meta-classes.
- Meta classes do not perform magic. However, it is a truly powerful feature in python, and hence it should be used in cases that ‘normal’ python code cannot do.
- Meta-programming involves introspection, and this can be easily implemented using the ‘type’ function. A built-in function that helps with introspection is the ‘dir’ function, which lists down all attributes/methods of an object in python. I understand that some of you will argue about it, but if you look closely at the definition of introspection. Then you would certainly realize that the ‘dir()’ function satisfies the conditions laid down by the definition of introspection. Hence I will always consider it as a tool for introspection.
- In Python 2.4+, all classes must inherit from some other class. If there is no class that you can inherit from, then you must inherit from the class named “object”. In the case of python 3.x, classes are derived from “classobj” by default. In our examples, we will be considering Python2.7.x, but if you are using python 3, then please replace the first line in the code with class MyCrawler(classobj):
Example #1:
The results are as follows:
The results are as follows:
The first 2 outputs are expected and possibly need no explanation. The ‘type’ of ‘crawlobj’ is an instance of the class we have defined. In the second statement, we checked if ‘MyCrawler’ is a class or not. Turned out to be a class. No surprises there. But, hey, what does the third statement show?
Remember I wrote, “In python, nearly all is an object”? Well, actually in python, everything is an object some of them are instances of classes, some rare instances of metaclasses, except for type.
The question that arises now is what exactly is ‘type’. To find that out, we will be using a method of the ‘inspect’ module named ‘isclass()’. ‘isclass’ returns True if its argument is a class, and False otherwise. So let us see what happens when we do that to the last line in the previous code sample:
inspect.isclass(type(type( crawlobj)))
If you execute this at the python command-line interpreter (or in a codebase), the result you would get is ‘True’. So ‘type’ itself is a class, but it is not one of our standard classes, it is a bit special. ‘Type’ is a built-in metaclass in Python from which you can create your metaclasses. So to create your metaclass, you would do something like the following:
class MyOwnMetaClass(type):
# Do whatever you want to do.
Now, we have already seen that if we call ‘type’ with one argument, it returns us the type of that object passed in as a parameter to ‘type’. However, ‘type’ may even call with 3 arguments. In that case, it creates a class on the fly. The class creates with the 3 parameters provided to the ‘type’ function.
The 3 parameters are:1) The name of the class to create (a string value).
2) A list of base classes from which the class will inherit attributes (this may be empty in case the class doesn’t need to inherit attributes/methods from other classes).
3) A dictionary containing the names of all attributes and methods to include in the class. Please note that you would still need to implement the methods provided as part of the third parameter. You may also leave the third parameter empty.
Here are examples of the calls to ‘type’ and their outputs. We will also show you the corresponding class that gets created by the call to the ‘type’ function with 3 parameters.
MyCrawler = type('MyCrawler', (), {})
In the above code, the first argument is the name of the metaclass we are creating. The second and third arguments are empty in the example. The second argument is a list of base classes (which is empty in our example, so our class MyCrawler doesn’t inherit from any base class). And the third argument, a dictionary, holds all the fields and methods the class might have. The above code could simply be written as:
class MyCrawler:
pass
Now let’s consider something more realistic. After all, you won’t be writing code to create classes that do nothing. Consider the following code:
Example #2:
The above code is equivalent to writing the following python code:
At first look, the meta-class implementation might look a bit cryptic and difficult to understand what is happening. However, as you begin to use this notation (met classes), you will begin to get used to it. And you will be able to write flexible and concise code. However, as I have stated before, these things should be used only in cases where they are the only solution.
For instance, if you are a ‘professional ethical hacker’ (excuse me here, but there is no such thing called ‘ethical hacker’ since the term ‘hacking’ means breaking into people’s private spaces, which in itself is a crime. I have worked with organizations that do this and believe me, the biggest clients of such organizations are not corporates, but government security agencies. If necessary, they can get the exact information about the clicks made by you and the keys pressed by you in a given time. Sorry for digressing.), you would possibly be using this type of logic in your everyday work. A web application developer or a REST API framework developer might also use this. But that would be only in one or two places, not all over the entire codebase.
Meta-programming is also performed when you create and use decorators. So let us put a rapid show at it.
Decorators as metaprogramming tool
If you have used and created decorators, then you already know what it is and how it works. However, for those who haven’t created a decorator, I will quickly go through the steps of creating one.
First, the decorator is a calibrator, who takes the function as an argument, and performs certain actions before and (perhaps) after calling the function from within. Here is an example of a decorator I wrote for a certain project, and it checks if a session in a web application is valid or not.
Example #3:
Since we will be taking a look at the structure here, there is no need to follow the logic in it. “is_session_valid” is a usual python function, but preferable to getting a scalar or a list/tuple/dict as an argument, it takes a function as an argument. Next, once inside the “is_session_valid” function, we define another function named “sessioncheck” which takes an HTTP request object as an argument. For our purposes here, you may ignore where this request object comes from. Inside “sessioncheck”, it does some processing and eventually calls the function that was passed in as an argument. Finally, “is_session_valid” returns the internal function “sessioncheck”.
Now, this is an example of a simple function decorator, and in fact, it doesn’t have much to do with metaprogramming directly. However, what if we were to do this to an entire class? (Remember, when we defined the term decorator, we mentioned the word “callable”, and it means an object for which the function call is defined. So that brings classes into the fray too). For a single class, you could make a class decorator (I won’t be explaining how to create a class decorator here as that is beyond the scope of our discussion here.
Create a metaclass
However, if you want to know how to create one, there are excellent tutorials on the internet that can teach you how to create one), and wrap it with it. But what if we are considering multiple classes? What if we do not know in advance how many classes need to wrap, with our decorator? The solution is to create a metaclass.
To write a metaclass, we normally modify and/or define methods like new, prepare and call. We might also mess with init, but we might not want to do that since init is called when an object has already been created, and we want to initialize it. The creation of the object happens inside new, and hence our focus is normally on that method. Let’s see some code that creates a metaclass using some of the above-mentioned magic methods (Please try using python3.x):
Example #4:
The output of the above code would be:
100
Let’s try to explain what happened above. It is simple: MyClass recognized the attrib attribute automatically from the metaclass MetaDemo.
__init_subclass__ in Python >= 3.6 is glorious. So much metaclass complexity you can now avoid for equivalent functionality: https://t.co/wEjUcMLzk2 From PEP 487 https://t.co/mLXqWV47W2
— Brett Slatkin (@haxor) January 31, 2018
A metaclass can be thought of as a class factory, in the same way, we consider a class to be an object factory.
Wrapping Up:
Well, what we have been able to touch here is the tip of the proverbial iceberg. Metaprogramming and metaclasses are vast and deep and a fascinating topic, and I think one can write a 250-page book on it, that will be able to delve into these topics at a moderate depth. This write-up is basically for inculcating interest in this topic, and I sincerely hope I have been able to do justice in that regard. We have left out topics like introspection in python, which go hand in hand with metaprogramming.
Specifically, one should look at how __getattribute__ and its brothers and sisters (namely __getattr__, etc) work. I would request the reader to research such things. There are many wonderful documents and tutorials on the internet that take a look at one aspect of metaprogramming and examine that aspect in great depth. So it would take time to learn it. But once learned, it will be a wonderful tool for the programmer who has to tackle problems that need very dynamic and unique solutions. So it is definitely worth the effort.
