Table of Contents
Decorators can be a daunting topic when first encountered. While The Zen of Python states “There should be one– and preferably only one –obvious way to do it”, there are many, equally valid ways to implement the same decorator. These different methods can be categorized as either function-based, class-based, or a hybrid of both. In this post I will explain the design and behavior of Python decorators and provide examples of decorators that I frequently use in my own code.
What is a Decorator?
In Python, absolutely everything is an object, including functions. Since functions are objects, they can be passed as arguments to another function, they can be the return value of a function, and they can be assigned to a variable. If you understand these concepts then you have everything you need to understand decorators.
A decorator is any callable object that takes a function as an input parameter. I specifically said “callable object” rather than “function” since Python allows you to create other types of callable objects. This interesting language feature is what allows us to create class-based decorators, as we will see shortly.
A Simple Decorator
Decorators are wrappers that allow you to execute code before and after the “wrapped” (or “decorated”) function is executed. By manually constructing a decorator function this “wrapping” effect can easily be demonstrated. Consider the
decorators.basic module given below:
simple_decoratoraccepts a function as a parameter, making it a decorator.
simple_decorator) allows us to execute code before and after executing the wrapped function.
Line 5: This print statement will be executed before the wrapped function.
Line 6: The wrapped function is executed within
Line 7: This print statement will be executed after the wrapped function
Line 9: This is probably the most confusing part.
function_wrapperis the return value of the decorator function (
simple_decorator). At this point, the wrapped function (
function_to_decorate) HAS NOT been executed.
Line 12: We will use this function to demonstrate how
Open an interactive Python shell and execute
undecorated_function() to see the behavior before applying any decorator. Next, we pass
undecorated_function as a parameter to
simple_decorator, and store the return value in
(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1188.8.131.52)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from decorators.basic import * >>> undecorated_function() I FORBID you from modifying how this function behaves! >>> decorated_function = simple_decorator(undecorated_function)
Finally, we execute
>>> decorated_function() Preparing to execute: undecorated_function I FORBID you from modifying how this function behaves! Finished executing: undecorated_function
This is the result we expect after applying
simple_decorator. However, wouldn’t it be better if we could permanently alter the behavior of
undecorated_function? We can easily do this if we replace the reference to
undecorated_function with the function returned by
>>> undecorated_function = simple_decorator(undecorated_function) >>> undecorated_function() Preparing to execute: undecorated_function I FORBID you from modifying how this function behaves! Finished executing: undecorated_function
This is exactly what happens when you decorate a function using Python’s
@decorator syntax. To reinforce this point, we can modify our code to use the normal decorator syntax. Note that we are applying the decorator to
undecorated_function in Line 12:
Let’s verify that the behavior of the function has been modified:
(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1184.108.40.206)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from decorators.basic import * >>> undecorated_function() Preparing to execute: undecorated_function I FORBID you from modifying how this function behaves! Finished executing: undecorated_function
As you can see,
@simple_decorator is just syntactic sugar for
undecorated_function = simple_decorator(undecorated_function).
This type of decorator is nice, but what if our original function has one or more input parameters? It wouldn’t be possible to provide values for these parameters with the current implementation of
simple_decorator. What can we do to fix this?
Passing Arguments to the Wrapped Function
Fixing this is very easy. Remember, when we execute the “wrapped” function, we are really executing the function returned by the decorator function (
function_wrapper in the previous example). Any values we provide to this function can be passed on to the wrapped function.
However, we need to accommodate all possible combinations of input parameters. We can do this by modifying
function_wrapper to accept
*args, **kwargs and passing them on to
function_to_decorate. This has been implemented in a new module,
Let’s test this version with the two decorated functions,
(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1220.127.116.11)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from decorators.better import * >>> greeting_anonymous() Preparing to execute: greeting_anonymous Hello anonymous website viewer! Finished executing: greeting_anonymous >>> greeting_personal() Preparing to execute: greeting_personal Hello Aaron! Finished executing: greeting_personal >>> greeting_personal(name="Jerry") Preparing to execute: greeting_personal Hello Jerry! Finished executing: greeting_personal
This works exactly the way we need it to, the value we passed to
greeting_personal is used because it was passed from
function_to_decorate. Also, we did not break any existing functionality since the function that does not take any input parameters (
greeting_anonymous) is also decorated and behaves as expected.
Passing Arguments to the Decorator
The decorators we have created so far are certainly useful for many different scenarios, but they are also limited since there is no way to pass arguments to the decorator itself. How is this different than the last decorator we created? A decorator that accepts arguments would have the form shown below:
@foo(baz, fiz=buz) def bar(): ...
If you remember back to the beginning of this post, I mentioned that, in general, there are two different ways to implement Python decorators: function-based and class-based. The examples we have seen so far have all been function-based. Unless you need to create a decorator that accepts arguments, you should use these function-based designs since they are more readable and require less nesting/indentation than the equivalent class-based design.
However, when you need to pass arguments to a decorator, there isn’t an advantage to using either the function-based or class-based design. Let’s take a look at the function-based design first since it is a natural progression from the decorators we have already examined.
The generic form of a function-based decorator that accepts arguments is given below:
In order to pass arguments to the decorator, we add another wrapper function, called
decorator_factory (Line 2). I am calling it a factory because the return value is the actual function decorator (
decorator_with_args is where
function_to_decorate is passed in as an argument).
In order to understand how this works you need to realize that Line 15 (where we are applying the decorator to a function) is actually a function call to
decorator_factory("foo", "bar") which returns
decorator_with_args is the actual decorator but in order to pass the params “foo” and “bar” to the decorator, we had to call the factory method that creates the actual decorator which accepts the function
special_greeting as an argument and wraps it.
Contrast this with the other decorators we have created: when a function is decorated with
@a_better_decorator these are not function calls (notice that there are no parentheses after the decorator name).
Let’s fire up the REPL and test this decorator. We will also verify that we are still able to use the decorated function as expected by passing arguments that modify its behavior:
(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-118.104.22.168)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from decorators.function_based import * >>> special_greeting() First argument provided to decorator: foo Allow me to give a very special welcome to Dennis! Second argument provided to decorator: bar >>> special_greeting(name="Harold") First argument provided to decorator: foo Allow me to give a very special welcome to Harold! Second argument provided to decorator: bar
As you can see, the arguments that were passed into the decorator (
"bar") were used by the wrapper function, and the value passed into the decorated function (
"Harold") is also used when the wrapper function is executed.
Functions, Classes and Callables
Are you familiar with the built-in
callable accepts a single argument and returns a bool value:
True if the object provided appears to be callable and
False if it does not. If you think that functions are the only “callable” that exists you might consider this function rather useless or unnecessary. However, classes are also callable since this is how new instances are created (e.g.,
object = MyObject()).
The call syntax,
(...), can call functions or create class instances as we have just seen. But Python has a unique feature that objects other than functions can also be called. Adding the
__call__ method to any class will make instances of that class callable. This allows us to create decorators that are implemented using classes.
The same decorator can be implemented using a callable class instance:
The main difference between the function-based and class-based designs is how the arguments passed to the decorator are handled. In the function-based approach, the arguments are available to
function_wrapper as local variables. In the class-based design, the arguments are provided to the
__init__ method and assigned to instance variables which can be accessed from
We can confirm that this decorator behaves in exactly the same way as the function-based version:
(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-122.214.171.124)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from decorators.class_based import * >>> special_greeting() First argument provided to decorator: foo Allow me to give a very special welcome to Dennis! Second argument provided to decorator: bar >>> special_greeting(name="Hank") First argument provided to decorator: foo Allow me to give a very special welcome to Hank! Second argument provided to decorator: bar
Which Design Is Better?
Is there any advantage to using either decorator design? In my opinion, the class-based design is flatter and easier to read, making it the more Pythonic choice. However, I acknowledge that the function-based design is more conventional since the idea of a callable object that is an instance of a class (rather than a function) is not what most people expect when they encounter the concept of decorators.
Other than that, there are no obvious benefits to choosing one design over the other. You should use the design that makes the most sense to you.
I have intentionally left out something very important from these decorator examples. You should ALWAYS decorate
function_wrapper (or whatever name is used in your application) with the
functools.wraps decorator (located in the standard library’s
functools module). Why is this important? Consider the example below:
(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1126.96.36.199)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from decorators.class_based import * >>> import inspect >>> special_greeting.__name__ 'function_wrapper' >>> inspect.signature(special_greeting) <Signature (*args, **kwargs)>
When we inspect the name and signature of the
special_greeting function, we instead receive the name and signature of the decorator that was applied to it. While confusing, it becomes even more of a headache if you need to debug this code. This is easily fixed with the
functools.wraps decorator (Lines 2,11):
Now, if we inspect
special_greeting we will see the correct name and signature:
(venv) decorators $ python Python 3.7.6 (default, Jan 19 2020, 06:08:58) [Clang 11.0.0 (clang-1188.8.131.52)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from decorators.class_based import * >>> import inspect >>> special_greeting.__name__ 'special_greeting' >>> inspect.signature(special_greeting) <Signature (name='Dennis')>
Hopefully you have a better understanding of how decorators are designed and how they behave in Python. The remainder of this post will contain examples of decorators that I frequently use in my Python projects along with pytest functions that demonstrate the intended usage.
Example: Function Timeout
If you have ever written code that interacts with an outside service, you have probably encountered a situation where your program becomes stuck waiting for a response with no way to abort the function call. One way to get un-stuck is with the
It’s very easy to use. Simply decorate any function with
@timeout(seconds=X) and specify the number of seconds to wait before aborting the function call as shown below (Line 7). If the function completes before the specified number of seconds have elapsed, your program will continue executing. However if the function has not completed after the specified number of seconds have elapsed, a
TimeoutError will be raised, aborting the function call.
In the simple test scenario below, the
sleep function waits for two seconds when called. Since it is decorated with
TimeoutError will be raised one second after it is called. The
test_timeout function verifies that the correct error is raised (Lines 13-14).
Example: Retry Function
The next example is similar to the
@timeout decorator since both are designed to handle functions that are unreliable. The
@retry decorator adds retry logic to the decorated function.
To use it, you specify a set of
exceptions that can trigger a failed attempt, the number of failed attempts that can occur before aborting the function call (
max_attempts), the number of seconds to delay after each failed attempt before trying again (
delay) and an optional handler method to be called whenever an exception is raised (for logging, etc).
The actual decorator definition begins on Line 26 below. Before that, the custom Exception
RetryLimitExceededError is defined (Lines 6-13). This is the exception raised after
max_attempts to call the function have failed. The
handle_failed_attempt function (Lines 16-23) is provided as an example of what could be provided to the
We can use the
@timeout decorator to demonstrate and test the
@retry decorator. Since we need to know the type of
Exception that we expect to occur when we call the decorated function, we specify
exceptions=(TimeoutError,) since this is the error raised by the
With the code below, we will attempt to call
retry_with_timeout a maximum of two times. Since the only thing this function does is wait for two seconds and we have decorated it with
@timeout(seconds=1), calling it will always raise a
TimeoutError. Therefore, after the second failed attempt, the
@retry decorator will raise a
test_retry_with_timeout function verifies that the
RetryLimitExceededError is in fact raised after calling the
Example: Log Call Signature and Execution Time
The most common application of decorators might be logging. It’s easy to see why, having the ability to run code immediately before and after a function is called allows you to report information and capture metrics.
This decorator uses the class-based design, and allows the user to provide a custom logger. If none is provided, a logger will be created based on the module that contains the wrapped function.
Whenever the function is called, the
@LogCall() decorator adds an
info level entry the the log with the following data:
- Timestamp when the function was called
- function name and values of all arguments provided to the function (i.e. the call signature)
- Time elapsed while executing the function
The code below tests the
@LogCall() decorator with a custom logger and with the default logger. With the default logger, we expect the name of the logger to be the name of the module containing the decorated function,
tests.test_log_call (Line 30). When a custom logger is provided, we expect the name to match the value we specified when the logger was created,
custom_log (Lines 10, 38).
A nice feature of this decorator is that the function call signature contains the names and values of all keyword arguments, even if a default value was used or if the name was not given when the call occurred. For example, the call to the decorated function in Line 28 is
save_values("Aaron", "Charlie", "Ollie"), but the call signature that is logged contains the names of all three arguments,
save_values(a=Aaron, b=Charlie, c=Ollie) (Line 32).
Similarly, the call to the decorated function in Line 36 is
rand_time(max=4, add_random=True), which only provides two arguments. The call signature that is logged includes the default value of the missing argument,
rand_time(min=1, max=4, add_random=True) (Line 40).
If we run
pytest for these decorator examples, all of the tests pass:
(venv) decorators $ pytest tests/test_* ================================================================= test session starts ================================================================== platform darwin -- Python 3.7.6, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 -- /Users/aaronluna/Desktop/vigorish/venv/bin/python cachedir: .pytest_cache rootdir: /Users/aaronluna/Desktop/vigorish, inifile: pytest.ini plugins: clarity-0.3.0a0, black-0.3.8, mypy-0.5.0, dotenv-0.4.0, flake8-1.0.4, cov-2.8.1 collected 10 items tests/test_log_call.py::FLAKE8 PASSED [ 10%] tests/test_log_call.py::BLACK PASSED [ 20%] tests/test_log_call.py::test_default_logger PASSED [ 30%] tests/test_log_call.py::test_custom_logger PASSED [ 40%] tests/test_retry.py::FLAKE8 PASSED [ 50%] tests/test_retry.py::BLACK PASSED [ 60%] tests/test_retry.py::test_retry_with_timeout PASSED [ 70%] tests/test_timeout.py::FLAKE8 PASSED [ 80%] tests/test_timeout.py::BLACK PASSED [ 90%] tests/test_timeout.py::test_timeout PASSED [100%] ================================================================== 10 passed in 6.63s ==================================================================
If you would like to download all or some of the code from this post, you can easily do so from the Github gist linked below:
I hope this introduction to decorators in Python was helpful and easy to understand. If you have any questions, criticism or feedback please leave a comment. Thanks!