Last time I explored a number of useful Python decorators. The post was a revisit to a topic I first posted about five years ago [see Python Decorators, part 1 and Python Decorators, part 2]. Back then I didn’t really know what to do with decorators, but I could see they were useful.

Since then, I’ve found many applications for them, hence the revisit to the topic. In this post, we’ll continue looking at useful applications. If nothing else, they may provide some ideas for decorators of your own.

Jumping right in, the last post featured decorators that logged any arguments passed to the decorated function. Those logged anything found in *args (positional arguments) and anything found in **kwargs (keyword arguments). Which works for general debugging, but what if your interest is in only one argument? This might be nice if the function takes lots of arguments or if its arguments can be especially long. You might not want to log that much data.

Here’s a decorator the logs just one argument:

The decorator takes a parameter, which, that determines which parameter of the decorated function to log. If which is an integer, it indexes a positional argument (lines #17 to #20). If it’s a string, it names a keyword argument (lines #22 to #25).

As usual, a decorator taking one or more parameters is double-nested. The DataRecorder function returns the inner decorator function (line #38), and that function returns the inner-inner wrapper function (line #35).

As in previous examples, this decorator stores the logging stats in attributes it creates — count:int and data:list — on the wrapper function object.

We use the decorator like this:

When run, this prints:

DataRecorder(2)
@DataRecorder: function1
DataRecorder(y)
@DataRecorder: function2

function1[1]: () {}
function1[2]: (1,) {}
function1[3]: (1, 2) {}
function1[4]: (1, 2, 3) {}
function1[5]: (4, 5, 6) {}
function1[6]: (7, 8, 9) {}

function2[1]: () {}
function2[2]: () {'x': 1, 'y': 2}
function2[3]: () {'x': 2, 'z': 9}
function2[4]: (3,) {'y': 5}
function2[5]: () {'x': 7, 'y': 8, 'z': 9}

function1: 6 calls
values: [4]3, [5]6, [6]9

function2: 5 calls
values: [2]2, [4]5, [5]8

The number in square brackets is the number of the call. At the end (last five lines), the logged parameter values are listed with the call number prepended.

With a bit more code, we can improve the decorator to handle multiple arguments. And rather than logging parameters, let’s actually change them. The requirement here is for a decorator that allows applying an arbitrary function to any input(s), positional or keyword:

This decorator has three input parameters: a list of positional argument indexes (arg_indexes), a list of keyword argument names (kw_names), and a list of functions to apply to the selected arguments (ftn_list).

There must be as many functions in ftn_list as there are indexes and names combined in the first two lists. The functions must expect and return a single parameter, which should be the same as the argument type (unless the function is converting types — see next decorator).

There can be one additional function in ftn_list. If so, it is applied to the decorated function’s return value.

The code should be fairly self-explanatory. Lines #15 to #19 process positional arguments, and lines #21 to #25 process keyword arguments. Lines #30 to #33 process the return value. Note that in all cases, we supply the argument to the handling function and pass its return value to the decorated function.

We use the decorator like this:

Note how we’re using two occurrences of the StringHandler decorator to apply two sets of argument transformations (lines #8 and #9). In one case, we have a function that strips whitespace from strings in positional indexes 0, 1, and 2, in the other, functions that force the case of strings for two positional arguments (1 and 2) as well as two keyword arguments (“x” and “y”).

The commented-out line #7 shows how to apply a function to just the return value.

When run, this prints:

@StringHandler: string_function
@StringHandler: StringHandler.<locals>.decorator.<locals>.wrapper

a: dog
b: CAT
c: HORSE
x: STEAK
y: EGGS
z: bacon

rv=dog-cat-horse:steak-eggs-bacon

a: apple
b: ORANGE
c: GRAPE
x: BURGER
y: FRIES
z: coke

rv=apple-orange-grape:burger-fries-coke

Note how, despite the forced uppercase of some inputs, the return value is forced to lowercase, not by the commented-out line but by the extra function passed in line #9.

Here’s a variation, a pair of decorators that allow converting str to bytes and vice-versa:

As with DataRecorder above, we support modifying only a single argument. Additionally, we’ll only handle positional arguments. Most situations where this conversion is relevant involve just one parameter that very likely to be positional (if not, the code above shows how to include keyword arguments).

The two decorators are identical except for the str or bytes conversion. If you’ve ever had to deal with functions that expect bytes when the code generally deals with str (or in rare cases vice versa), this is a handy technique.

We use the decorator like this:

When run, this prints:

String2Byte(2, utf8)
@String2Byte: function1
Byte2String(3, utf8)
@Byte2String: function2
String2Byte(0, utf8)
@String2Byte: function3
Byte2String(0, utf16)
@Byte2String: function4

b'\xc4\x86\xc3 … \x8b\xc4\xa5' (bytes)

Hello (str)

b'\xec\xa7\x88 … \xed\x8f\x92' (bytes)

World (str)

(Long strings trimmed to fit.).

Note that, if we want to alter both an input argument and the return value, we can stack two decorators, like this:

When run, this prints:

String2Byte(1, utf8)
Byte2String(0, utf8)
@Byte2String: bytes_handling_function
@String2Byte: Byte2String.<locals>.decorator.<locals>.wrapper

bytes_handling_function(13 bytes)
data = 48.65.6c.6c.6f.2c.20.57.6f.72.6c.64.21

retv = 'Hello, World!'

The function bytes_handling_function takes and returns bytes, but in our code, thanks to the decorators, we can treat it as if it takes and returns strings.

Last time we saw a decorator for timing a function. It recorded the total time spent in the decorated function as well as logging each call with a timestamp and arguments list. That decorator acted on each function individually, but suppose we wanted to time a group of functions?

Here’s a decorator that lets us declare “group” names under which timing stats are gathered:

This is very similar to the timing decorator from last time, so I won’t spend much time describing the code. The key difference is the addition of the FunctionTimes dictionary (line #3). This dictionary has (user-supplied) group names as keys. The values are themselves dictionaries with the decorated function names as keys. Note that in line #44, we set the data attribute on the FunctionTimer function object to reference this dictionary.

The decorator takes a group name, so we have a double-nested set of functions. The outer function (lines #5 to #42) checks FunctionTimes to see if the group name exists as a key. If not, it adds a new entry (lines #9 to #11). Then it just returns the nested decorator function (line #42).

The decorator function (lines #13 to #39) gets the decorated function’s name (line #15) and checks to see if that name is in the group. If not, it adds a new entry (lines #18 and #19). Then it returns the nested wrapper function (line #39).

The wrapper function (lines #21 to #36) times the decorated function (lines #25 to #28) and logs the data (lines # to #).

We use the decorator like this:

Sorry for the long bit of code here. I wanted to have enough functions and enough calls to them to make it interesting.

There’s nothing special about the functions. They use time.sleep function to give them varying delay times. Each function takes a t (time) and a d (delta) parameter that determine the minimum time plus a random additional time. The FAST global variable, if True, allows speeding through the test for development or debugging.

When run, this prints:

@FunctionTimer(main)
@FunctionTimer.decorator(fast_function)
@FunctionTimer(group1)
@FunctionTimer.decorator(slow_function)
@FunctionTimer(group1)
@FunctionTimer.decorator(another_slow_function)
@FunctionTimer(group2)
@FunctionTimer.decorator(slower_function)
@FunctionTimer(group2)
@FunctionTimer.decorator(another_slower_function)
@FunctionTimer(group3)
@FunctionTimer.decorator(slowest_function)

Running some slow functions... (be patient)

FunctionTimer[fast_function]
[fast_function]: 1118.0416 msecs
FunctionTimer[slow_function]
[slow_function]: 2927.4383 msecs
FunctionTimer[another_slow_function]
[another_slow_function]: 3236.9994 msecs
FunctionTimer[fast_function]
[fast_function]: 1176.5464 msecs
FunctionTimer[slower_function]
[slower_function]: 3172.3967 msecs
FunctionTimer[another_slower_function]
[another_slower_function]: 3657.9977 msecs
FunctionTimer[fast_function]
[fast_function]: 1031.5982 msecs

...Hang in there...

FunctionTimer[fast_function]
[fast_function]: 789.7108 msecs
FunctionTimer[slow_function]
[slow_function]: 3817.0942 msecs
FunctionTimer[another_slow_function]
[another_slow_function]: 2811.3104 msecs
FunctionTimer[fast_function]
[fast_function]: 1086.5883 msecs
FunctionTimer[slower_function]
[slower_function]: 3194.0699 msecs
FunctionTimer[another_slower_function]
[another_slower_function]: 4089.8889 msecs
FunctionTimer[fast_function]
[fast_function]: 1027.6462 msecs

...Got a bit more to go...

FunctionTimer[fast_function]
[fast_function]: 1042.563 msecs
FunctionTimer[slow_function]
[slow_function]: 3778.7955 msecs
FunctionTimer[another_slow_function]
[another_slow_function]: 2739.5931 msecs
FunctionTimer[fast_function]
[fast_function]: 934.2696 msecs
FunctionTimer[slower_function]
[slower_function]: 4429.907 msecs
FunctionTimer[another_slower_function]
[another_slower_function]: 3640.9476 msecs
FunctionTimer[fast_function]
[fast_function]: 1028.4622 msecs
FunctionTimer[slowest_function]
[slowest_function]: 8002.107 msecs
FunctionTimer[fast_function]
[fast_function]: 1088.5107 msecs
FunctionTimer[slowest_function]
[slowest_function]: 7987.3595 msecs
FunctionTimer[fast_function]
[fast_function]: 959.5968 msecs

...Done!

[main] (1 function)
:fast_function::
11.283534, 1.118042, 1.176546, 1.031598, 0.789711, 1.086588,
1.027646, 1.042563, 0.934270, 1.028462, 1.088511, 0.959597

[group1] (2 functions)
:another_slow_function::
8.787903, 3.236999, 2.811310, 2.739593

:slow_function::
10.523328, 2.927438, 3.817094, 3.778796

[group2] (2 functions)
:another_slower_function::
11.388834, 3.657998, 4.089889, 3.640948

:slower_function::
10.796374, 3.172397, 3.194070, 4.429907

[group3] (1 function)
:slowest_function::
15.989467, 8.002107, 7.987360

Which also goes on for a bit. Sorry!

Note that, in the listing at the end, the first number listed for each function is the total time (in seconds) spent in the function. The numbers that follow are the times spent on individual calls (in the order called).

The decorators so far have all been for functions. Nearly identical decorators can be created for class instance methods — just remember to include the self parameter [see the previous post for more on class instance method decorators].

Class decorators are a bit different because they wrap a class object rather than a function object. The decorator protocol is the same but wrapping a (newly created!) class offers opportunities to modify the class.

For instance, suppose you intend to create a number of different (but related) classes, and you plan to always implement the same simple __repr__ method:

Which, for the example above, would print as:

<ExampleClass @2783de82900>

We can use the following class decorator to automatically add such a function to any class it decorates:

Note that, unlike all other decorators so far, this decorator does not return a wrapper around the function or, in this case, class — it returns the class itself. But first it modifies the class by installing the __repr__ method (line #8).

Obviously, a decorator like this can make many different modifications to classes it decorates. This represents a horizonal form of class hierarchy for cases where different classes can’t share a parent that implements common methods.

We use the decorator like this:

The example assumes that, for whatever reason, these five classes can’t share a common ancestor but should have a common method (in this case __repr__).

When run, this prints:

@AddSimpleReprMethod(EgClass1)
@AddSimpleReprMethod(EgClass2)
@AddSimpleReprMethod(EgClass3)
@AddSimpleReprMethod(EgClass4)
@AddSimpleReprMethod(EgClass5)

c1: <EgClass1 @24f6b5c2a50>
c2: <EgClass2 @24f6b5c2ba0>
c3: <EgClass3 @24f6b5c2cf0>
c4: <EgClass4 @24f6b5c2e40>
c5: <EgClass5 @24f6b5c2f90>

Demonstrating that the defined __repr__ function does exist in each class.

Here’s another version:

This one creates a __repr__ method that lists all the instance’s attributes.

Use it like this:

When run, this prints:

Change repr for foobar
Change repr for tasbot

foobar(name='Fred', count=0, timer=0.0)
foobar(name='Barney', count=0, timer=0.0)

tasbot(x=0.0, y=0.0)
tasbot(x=4.2, y=2.1)

Note that this form of output can be used to recreate the instances:

The built-in eval function treats the text as declaring an object instance.

When run, this prints:

Change repr for faxbun

faxbun(name='Sam', x=0.0, y=0.0)
faxbun(name='Judy', x=4.2, y=2.1)

faxbun(name='Sam', x=0.0, y=0.0) [type=faxbun]
faxbun(name='Judy', x=4.2, y=2.1) [type=faxbun]

A common use for such a __repr__ method is to save object instances and their state to a text file and then restoring them from that file later. In such cases, you definitely want any object you intend to save to have this __repr__ method.

Last time, I mentioned the built-in method decorators staticmethod and classmethod. Both modify the method signature. The former to have no reference to the enclosing class (nor an instance of that class), the latter to receive a reference to the class but not to any instance.

For this one, I think it makes the most sense to start with a use case:

Above is a simple 2D point class with a static method (lines #13 to #17) and a class method (lines #19 to #24). Note that we’re using StaticMethod and ClassMethod decorators rather than Python’s built-in staticmethod and classmethod. The former two are defined below.

Static methods take an ordinary set of parameters — there is no self or cls parameter. Static methods can know about the class because they’re defined inside the class (and its namespace), but they otherwise function as ordinary functions. Class methods take a cls parameter, which is a reference to the class rather than a specific instance, as with self.

When run, this prints:

@StaticMethod: distance
@ClassMethod: setdent

distance([0.000 x 0.000], [-5.000 x 5.000])
7.0710678118654755
distance([0.000 x 0.000], [-5.000 x 5.000])
7.0710678118654755

setdent(point, 27)
27
setdent(point, 31)
31

We need something a bit different in method decorators that change the signature of the wrapped function. The usual technique works fine on instances, but not on classes. To implement static or class methods, we need to use a Python descriptor:

See Python Descriptors, part 1 and Python Descriptors, part 2 for details on how descriptors work. The short version is that they provide much more control over how the decorated method gets accessed.

When client code uses a static or class method, it invokes the descriptor’s __get__ method, which returns a wrapper function, proxy. The client code, in calling the method, calls the proxy, which invokes the actual method.

Something to be aware of in Python is that it’s hard to entirely protect classes from client code. Python has no sense of private or protected data, as some languages do. Client code can overwrite or modify these decorated methods.

Here’s an example:

When run, this prints:

@StaticMethod: barfoo
<StaticMethodProperty @293c066e900> init
@ClassMethod: foobar

<StaticMethodProperty @293c066e900> get: <null>
TestClass.barfoo(21,42,63)=55566
<ClassMethodProperty @293c066eba0> get: <null>
TestClass.foobar(21,42)=63

g=<TestClass @293c066ecf0>
<StaticMethodProperty @293c066e900> get: <TestClass @293c066ecf0>
g.barfoo(63,84,108)=571536
<ClassMethodProperty @293c066eba0> get: <TestClass @293c066ecf0>
g.foobar(63,84)=147

You may not modify barfoo!
You may not delete barfoo!
You may not modify foobar!
You may not delete foobar!

<StaticMethodProperty @293c066e900> get: <null>
TestClass.barfoo=[1, 2, 3]
hasattr(TestClass,"barfoo")=False
<ClassMethodProperty @293c066eba0> get: <null>
TestClass.foobar=[1, 2, 3]
hasattr(TestClass,"foobar")=False

Using descriptors to implement the property decorator is left as a reader exercise.

Lastly, all the decorators in these two posts are functions, which raises the question of whether a decorator can be a class. It can, though as we’ll see, using a decorator class requires a slight change in usage.

Here’s a basic decorator class (for decorating functions):

Note that we’re also using an inner class (lines #4 to #17) to create wrapper instances.

Using this decorator class requires making instances of it:

Note that line #3 and line #7 create instances of the decorator. They are not calls to a decorator function to set parameters, as we’ve used in some previous decorator functions.

When run, this prints:

@Decorator()
@DecoratorFunction(<function function0 at 0x000001F02EB2E160>)
@Decorator()
@DecoratorFunction(<function function1 at 0x000001F02EB2DA80>)

Decorator(0 args, 0 kwargs)
retv=(21, 42, 63)

Decorator(5 args, 0 kwargs)
retv=(280.7495567531091, 'Hello, World!')

This has run longer than usual (especially considering the first one did, too), but it’s a meaty topic with a lot of potential. I hope these examples have opened doors to new ways to use Python.

Link: Zip file containing all code fragments used in this post.

∅