Decorators from Scratch

A guide to master decorators with Python.

Hello, Hypers!

Stephen Gruppetta is the authority when it comes to Python. If you don’t know his newsletters about Python and technical writing then you have to change something in your life. But today, if he allows me :), I will write about one of my favorite Python features: decorators.

It has been a while since I wrote a post about programming languages, and it’s nice to be back on this topic. If you are new to Under the Hype and you like articles about programming languages, then I recommend you to read my two previous articles about Lisp:

• Learn the True Polymorphism

• A New Language for Every New Problem

By the way, those are the most-read articles in Under the Hype.

But today it is all about decorators, so let’s get started!

Note: This email has a lot of images so your client will probably cut part of it. Don’t worry, you will have the option to continue reading. Also, if you prefer, you can read this issue online in Under the Hype website.

Decorating

The first thing is to define what a decorator is. Is it a feature that is exclusive to Python? No! A decorator is a basic concept for any programming language with at least a slight Functional flavor.

A decorator is a function that takes another function as a parameter and returns a modified version of that original function. These are cases in which mathematical notation is handy (like always).

Suppose we have a function f: A → B. That is a function that receives an element from the set A as an argument and returns an element from the set B as a result. Then a decorator D(f) returns a modified version of the function f. In general, we have that D(f): A → C. That means that this modified version of the function keeps receiving the same arguments but could return a different set of values.

The only thing that a language should have to support decorators is the possibility to treat functions as first-class citizens. The first-class citizens of a programming language are all the elements that can be passed as arguments to functions, assigned to variables, and returned from functions. When functions are first-class citizens of a language, we say that the language is functional. Actually, I prefer to say that the language has some functional features since not all languages have the same functional flavor.

For example, in C# we have delegates. These are abstractions for functions that allow us to give them first-class treatment. Even in a non-object-oriented language like C, we can use pointers to get the benefits of some functional behaviors. On the other hand, we have pure functional languages like Haskell. Here, functions are the most basic elements. So in Haskell, we don’t need any abstraction or weird tricks to treat functions as first-class citizens.

Then, what’s so special about Python decorators?

The Importance of the “How”

Python has always leaned towards functionality. Functions have always been first-class citizens and there is no extra friction to pass functions as arguments, assign them to variables, or return them. It comes naturally.

Also, Python has other extra functional features like lambda functions, list comprehension, tuple management, etc. It has even added some sort of pattern matching recently. All this makes Python a decent functional language considering that it was not designed to be a pure functional language.

However, it is not clear how all those features make Python’s decorators so special. For example, Javascript doesn’t add any friction when it comes to treating functions as first-class citizens and we never hear about Javascript decorators. Why?

Let’s answer it with an example. Suppose we have the classic recursive Fibonacci function. It receives an integer n and returns the n-th term of the Fibonacci’s succession. But, there is a problem. When n is too large, the function has to perform too many calculations. It could take a lot of time to get a result. So let’s create a decorator that prevents the function execution when n is greater than 10. Let’s do it in both Python and Javascript.

This is the Python version:

And this is the JS one:

They look very similar! What is all the fuss about Python decorators? Well, let’s see how they are applied.

In Javascript we would have something like this:

We explicitly apply the decorator to the original function to get the modified version. We can do the same in Python, but we have also a beautiful alternative:

This Python code does the same as the previous JS code. Note that we don’t need to explicitly call the decorator. By adding @limit_10 right before the function definition we automatically modify the original function. Python makes the decorator call for us, which makes the code way more readable.

This simple change in how to use decorators makes all the difference. Almost at the end of this article, I will show you how in Python we can fully leverage the power of decorators when combining them with classes. This is another big plus for the Python version of decorators. But first, I want to give more practical examples of these funny guys.

Decorators with Arguments

The previous decorator works great but we can do better. Instead of having a hardcoded limit for n, we could make this limit a parameter so I can decorate different functions with different limits. This is how we can do it:

It is a little trickier but not that much. Now the decorator should receive the arguments we want to pass to it. In this case, we want to pass the limit for n. Then, like before, we need to define a function that receives another function and returns a modified version of that function.

I use wrapper as the name of the modified version. Now we can use it like this:

Note that we are using it in two different functions with two different limits. For fib, we are limiting n to 10 and for factorial the limit is 20.

The convenient notation and the use of parameters make decorators a powerful tool in Python. And if you are not so convinced yet, check the following example.

Zero-effort Dynamic Programming

As we said before, the recursive version of the Fibonacci function is too slow. The reason is that it repeats the same calculations too many times. But there is a known solution for this problem: Dynamic Programming.

Dynamic Programming is an algorithm paradigm that is very helpful for some kinds of problems. When a problem can be split into many simpler subproblems, we can solve those simpler subproblems and store the results. This way, whenever we need to solve the same subproblem again, we have a pre-calculated solution.

Usually, when learning to code, students learn recursion. Then they discover how inefficient some recursive algorithms are, and apply Dynamic Programming in the form of Memoization.

Memoization is the easiest way to apply Dynamic Programming if you have already mastered recursion. For example, the Fibonacci function with memoization looks like this:

We only need to add a cache to store the result of the subproblems. If we already have the solution in the cache, we return it immediately. Otherwise, we apply the same recursive function as before. We only need to make sure to store the result in the cache before returning it.

Note: Here we initialize the cache as an array of 1000 elements, all of them equal to -1. The -1 value means that the result for that value of n has not been pre-calculated.

As you can imagine, we can make this modification with a decorator instead of creating a new version of the function:

Now the cache is created inside the decorator and we don’t need to take care of it. As you can see, with this modification we can calculate even the 50-th element in the Fibonacci’s succession. You can even create a new version of this decorator to parameterize things like the size of the cache or the default value (-1).

But there's more. We can write a Universal Memoization Decorator. It will use a dictionary as the cache. Here is a possible implementation:

And just like that, we can apply memoization to any recursive function that requires it without modifying the function. Now that’s a cool application!

Note: Having to write and read from a dictionary adds a little overhead to the runtime. But most of the time it will still be a great improvement compared with the recursive version. Also, an optimized version of this decorator is implemented in the functools library.

Classes as Decorators

Decorators have more surprises. I think the full power of Python decorators resides in the capability to combine them with Python classes and their magical methods.

Python is an Object-Oriented programming language. We can define classes that are abstractions of real-life objects, relationships, and basically anything. Also, Python includes what we call magic methods. These are special methods that allow us to manage the behavior of a class. We can make a custom class and the instances of that class behave like numbers, different data structures, functions, or anything else. For the purpose of this article, we are going to use only two magic methods: __init__ and __call__.

The __init__ method is probably the most famous magic method. It is used to initialize an instance of a class. On the other hand, the __call__ method allows us to use the instances of a class as functions. For example:

In the last two lines, we first initialize an instance of the class, which triggers the __init__ method, and then use the instance as a function, which triggers the __call__ method.

These methods allow us to use classes as decorators and to decorate classes. So, in Python, we can decorate either functions or classes but also we can use either functions or classes as decorators.

Before finishing, I’ll give you some examples of decorators involving classes.

For example, decorators are a great option to add logs to your code. You write the decorator once and use it everywhere. Look at this example of a logger for class initialization. This is an example of using a function to decorate a class.

Now we’ll use a class to decorate a function. In this case, we are interested in counting how many recursive calls are required to obtain the expected result.

This converts the fib function into a class with a counter property. After executing the function, we get the total number of recursive calls in the counter property.

Finally, a class decorating another class:

In this case, we use the Decorator class to add a new method to any class. For class decorators, we should have an __init__ method that receives the decorated class. Then we need to implement a __call__ method that will modify the __init__ method of the decorated class. Here we also use the property __class__ that is present in any object instance in Python to add the additional method.

Conclusions

This post was a little bit longer (or maybe trickier) than usual. When I started writing it, I was hesitating between a practical post and a more theoretical one. I think it came out very practical with a little bit of theory at the beginning. If you want a more theoretical one (for example, one about what happens with decorators behind the scenes), then let me know by replying.

Python decorators are a powerful tool that allows us to write better code. With them, we can recycle a lot of logic without losing readability. But, as it happens with any programming tool, you need to practice if you want to understand them deeply and make the best use of them. I hope the code snippets in this article help you to practice.

I also hope you have enjoyed this post. If you want me to write more about this stuff, please let me know. Thank you very much for another week.

See you next Tuesday!