This series of posts is for those who have used a programming language before but are not familiar with Python. The first six posts introduced the language; the seventh discussed how to download, install, and start using Python.

The last post began an exploration of user-defined data type (aka classes). This post picks up where we left off.

When we last saw our 3D Point class, we’d given its instance objects the following:

1. The data attributes x, y, and z.
2. The ability to initialize the data attributes when creating new instances.
3. An “informal” print string (accessible via the built-in str function).
4. A representation or debug string (accessible via the built-in repr function).
5. A True/False value (accessible via the built-in bool function).
6. Iteration over the three data attributes (accessible via the built-in iter function).
7. Implementing the addition operators (+ and +=) for point objects.

Line item #6 lets us treat Point objects like list objects:

When run, this prints:

Point: [3.000, 1.000, 4.000]

for-loop:
point[0] = 3.0
point[1] = 1.0
point[2] = 4.0

list: [3.0, 1.0, 4.0]
tuple: (3.0, 1.0, 4.0)

iter: [3.0, 1.0, 4.0]
enumerate: [(0, 3.0), (1, 1.0), (2, 4.0)]
sorted: [1.0, 3.0, 4.0]

any: True
all: True
min: 1.0
max: 4.0
sum: 8.0

We can use Point objects in the above contexts because we defined the dunder iter method in our Point class (see last post).

Line #1 imports the Point definition from the examples module (which is a Python file named examples.py in the local directory). This makes the Point class available to the code below.

Line #3 defines a new Point object with values; line #5 prints its “informal” value.

Lines #9 and #10 implement a for loop — which invokes a list context: a list-like object is expected. In this case one that iterates over the elements (x, y, and z). The enumerate function (see below) gives us index numbers (0, 1, and 2).

Lines #13 and #14 use the list and tuple constructors to create a list and a tuple respectively. Both constructors expect a list-like object. [See part 2 and part 3 to review list-like objects.]

Lines #17 to #20 demonstrate the built-in functions iter, enumerate, and sorted. Of note is that these return special list-like objects that are not exactly lists [see part 4 for more on these functions]. They are usually used in for loops. Here we pass them to the list constructor to make them actual list objects.

Lines #22 to #26 demonstrate the built-in functions any, all, min, max, and sum (see the Built-in Functions documentation for the full list). The first two are logical functions that return True if, respectively, any items or all items of the input list evaluate as true. The last three are basic math functions that return, respectively, the minimum, maximum, and sum of a list. These require the list contain numeric items, of course. Our iterator returns the x, y, and z elements, which are numeric, so we can use these functions without error.

So, giving the Point class a dunder iter method allowed our Point objects to be list-like in many contexts. Generally, we can pass these objects to any function expecting something list-like.

But not all:

When run, this prints:

Point: [3.000, 1.000, 4.000]

Try indexing:
OOPS: TypeError
'Point' object is not subscriptable

Try getting the length:
OOPS: TypeError
object of type 'Point' has no len()

Try reversing:
OOPS: TypeError
'Point' object is not reversible

So, our Point objects can’t be indexed (lines #10 to #12), and we can’t even get their length (line #21).

Lastly, the built-in reversed function (line #30) doesn’t work even though the seemingly similar enumerate and sorted functions did.

Let’s fix this by adding the appropriate dunder methods for the class:

The above is a code fragment with only the new class methods shown. We defined dunder iter in the last post but include it here because it’s so closely related to the new methods.

Lines #12 to #14 define the dunder len method (“length”). The built-in len function calls this method to obtain the object’s length. Python calls it internally when it needs to know a list-like object’s length.

Lines #16 to #23 define the dunder getitem method. Python calls this when we use the index notation — object[index] — to access a list item.

Lines #25 to #32 define the dunder setitem method. Python calls this when we use the index notation to change a list item (e.g. object[index] = new_value).

Lines #34 to #36 define the dunder delitem method. Python calls this when we use the del statement and index notation to delete a list item (e.g. del object[index]).

Now we have a more fully featured list-like object:

When run, this prints:

Point: [3.000, 1.000, 4.000]

length: 3

Getting values by index:
[0] 3.0
[1] 1.0
[2] 4.0
OOPS: IndexError
Invalid index: 3

Changing values by index:
[0] 2.0
[1] 7.0
[2] 1.0
OOPS: IndexError
Invalid index: 3

reversed: [1.0, 7.0, 2.0]

Try deleting:
OOPS: NotImplementedError
Point elements may not be deleted!

Note that lines #25, #27, #29, and #31 invoke the dunder setitem method to change the value while lines #26, #28, #30, and #32 (as well as lines #13 to #16) invoke the dunder getitem method to access it.

Note also how we control the Exception types and content. This allows including relevant information in the content as we did with the index errors.

As an aside, let’s dig a little deeper into the reversed function:

Line #3 creates a new Point object as usual. Line #8 passes it to the reversed function and stores the returned object in the variable rv. Line #9 uses the type function to give us that returned object’s data type. Line #10 prints the object.

Line #14 tries to get its length, and line #22 attempts to index a list item. Then, lines #29 and #30 both convert the object to a list.

Lines #34 and #35 exercise the reversed function in a for loop. Note this results in a new reversed object for the loop.

When run, this prints:

Point: [2.110, 4.220, 6.330]

reversed(pt) returned a <class 'reversed'>
<reversed object at 0x000002CDD8D43A00>

OOPS: TypeError
object of type 'reversed' has no len()

OOPS: TypeError
'reversed' object is not subscriptable

make it a list: list(rv)=[6.33, 4.22, 2.11]
try that again: list(rv)=[] ??

for-loop:
6.33
4.22
2.11

The reversed function, as with many built-in functions that return lists, actually returns an iterator object that isn’t a list, but which acts like one in most list contexts. It’s a kind of proxy object that hands us list items when asked. Importantly, it never contains the list items — it has no length. It can only return the next list item or signal the end of the list.

This is why the type function (from line #9) tells us the returned object is a <class ‘reversed’> — an instance of some class named “reversed” — and printing the object gives us the default Python object representation string. Clearly this is not a list. We can’t use index notation (the rv[1] on line #22 raises an Exception), and it has no length (the len(rv) on line # 14 also raises an Exception).

We can make it a list, as we do on line #29, but when we try that a second time, we get an empty list. Iterator objects provide only one trip through their list. Asked beyond that, we get nothing (or a StopIteration Exception depending on context).

In this regard, the reversed function is indeed similar to other functions such as sorted, enumerate, and many others. They all return some type of iterator rather than an actual list. The reason is simple: there’s no reason to duplicate a list in the iterator. Better to have it merely traverse the list.

The difference can be substantial:

Note that we import the getsizeof function from the sys module (line #2) so we can determine how much memory our objects require.

Line #5 creates a list of one million numbers, starting at zero. The built-in range function is one of those functions that returns an iterator, so we pass its returned value to list to be converted to an actual list.

Line #16 gets an iterator for big_list using the built-in iter function.

When run, this prints:

list length: 1,000,000 items
list size: 8,000,056 bytes

first 3 list items: [0, 1, 2]
last 3 list items: [999997, 999998, 999999]

iterator size: 48 bytes

iter item: 0
iter item: 1
iter item: 2
...

The list requires over eight million bytes. Its iterator only 48. (The range object is also 48 bytes.)

The reversed function differs from these other functions in three ways:

Firstly, for reversed to work an object must provide dunder len and dunder getitem. As we saw, this differs from enumerate and sorted which require only dunder iter. Any function that iterates forwards only needs an iterator from the list-like object. But reversed iterates backwards, so it needs to know the length and how to access the list items in reverse sequence — which it can only do by indexing them.

Secondly, there is an alternate way to get a reversed version of a list. Recall that the index notation allows three parameters: start, stop, and step. Since list[:] defaults to the entire list, list[::-1] gives us a reversed copy of the list:

When run, this prints:

forward: Hello, World!
reverse: !dlroW ,olleH

forward: [1, 2, 3, 'a', 'b', 'c']
reverse: ['c', 'b', 'a', 3, 2, 1]

Our Point class makes no allowances for this notation. We assume indexing only in the form pt[n] where n must be 0, 1 or 2. We could modify our dunder getitem and dunder setitem methods to handle this if we wanted, but it doesn’t make sense for our simple example class. The key point is that we can make our class as sophisticated (or not) as a given project requires.

Thirdly, there is a dunder reversed special method that lets us control exactly what reversed returns:

If implemented, reversed invokes this method, otherwise it defaults to using dunder len and dunder getitem. If none of these are provided (or even just one of the latter two), the reversed doesn’t work.

Lines #8 to #17 define our dunder reversed. It’s a method (function) containing a function named generator (lines #10 to #14). Line #17 calls generator and returns what it returns.

The generator function is special because it contains the yield statement. It is this statement that makes the function a generator. The yield statement is similar to the return statement, but it doesn’t exit the function the way return does. Instead, yield provides a value and then sleeps until called again to return another value.

As implemented here, the generator yields three values, the point elements z, y, and x, in that order. As with any iterator object, the built-in next function manually causes a generator to yield items. In list contexts, generators act like any iterator, providing list items one by one.

Generators are an advanced topic for later. This is just an introduction. [See Python Generators, part 1, part 2, and part 3, if you want to dig into the details now.]

Note that we can return an actual list:

If we test it with:

When run, this prints:

Point: [2.110, 4.220, 6.330]

reversed(pt) returned a <class 'list'>
[6.33, 4.22, 2.11]

length: len(rv)=3

index: rv[0]=6.33
index: rv[1]=4.22
index: rv[2]=2.11

make it a list: list(rv)=[6.33, 4.22, 2.11]
try that again: list(rv)=[6.33, 4.22, 2.11] !!

for-loop:
6.33
4.22
2.11

This works, but we’re returning an actual list, so reversed doesn’t work quite as usual and doesn’t follow expectations (which, in coding, is bad, m’kay).

It would be better to do it like this:

Which returns an iterator as expected (and if very like how we implemented the dunder iter method last week). However, the first method above (with yield) is cleaner and doesn’t require the intermediate list or the call to the iter function.

The bottom line of this digression is that reversed is a bit special compared to other seemingly similar built-in functions. If an object has a dunder reversed method, then the reversed function calls it and returns whatever it returns. Failing that, if the object has both the dunder len and dunder getitem methods, then the reversed function uses those internally something like this:

Note the use of the yield statement, which makes the function a generator.

When run, this prints:

<generator object the_reversed_function at 0x0000018C5B88A4D0>
type(rv)=<class 'generator'>
list(rv)=[11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
list(rv)=[]

Which is essentially the behavior we saw above for a proper reversed object.

That was a deeper dive than I intended. Hopefully it provided some insight to how Python works with objects.

We ended last time by giving our Point class the ability to handle the + and += operators. This allowed us to add and add-to Point objects in three basic cases:

Point₃ = Point₁ + Object₂
Point₁ += Object₂
Point₃ = Object₂ + Point₁

In the first two cases, Object₂ may or may not be a Point object. Dispatching the addition to the Point dunder methods depends only on Point₁. In the third case, Object₂ is definitely not a Point and doesn’t now know how to add one. In this case, Python dispatches the addition to the Point. (If that also fails, Python raises an Exception.)

Note that these three forms extend to other operators, such as minus, multiply, divide, and many more (see Operators).

These points apply to any numeric data type we implement. Particular to our Point design, however, is the issue of member-wise addition versus single-value (aka scalar) addition. There are two possibilities. Firstly, adding two points:

Secondly, adding a Point to a scalar value:

We’d like our Point objects to handle both, so here’s a more sophisticated version of the three dunder add methods:

This implementation of dunder add (lines #8 to #26) checks to see if other is an integer or float object (line #12) and, if so, adds that single value to x, y, and z (lines #13 to #15) and uses the new values to create and return a new Point object (line #16). Failing the first check, it checks to see if other is a Point object (line #19). If so, it does a member-wise addition (lines #20 to #22) and, as above, uses the sums to create and return a new Point object.

Failing both checks, other isn’t an expected data type, so we fall to line #26 and raise ValueError.

The dunder iadd (lines #28 to #46) is identical in structure but we apply the sums to self and return it.

The dunder radd only gets called when the left-hand object is not a Point and doesn’t know how to add one. In fact, it raises an Exception (probably ValueError), which Python catches and tries again with the right-hand object. If that also raises an Exception, Python considers the add a failure and re-raises the Exception.

Note that we could just call dunder add:

Rather than do the check for integer or float objects here. The dunder add method does the same check, so this works fine.

Because addition commutes, we can even do this:

Which makes dunder radd an alias for dunder add.

Important: note that subtraction and division do not commute, so dunder rmul and dunder rtruediv need thicker implementations. (Likewise, some of the other operators with right-hand dunder methods. See the Emulating numeric types documentation for details.)

That’s enough for now. The ZIP file below includes a full example of the Point class. Next week we’ll finish our design with some (non-special) methods of our own.

Link: Zip file containing all code fragments used in this post.

∅