Python has a feature called a generator — it’s a way of writing a function that allows that function to trade control back and forth with a calling function. It’s not something one needs often, but it can very useful in certain situations.

Create a Python generator, firstly, in using yield instead of return and, secondly, by designing the function a little differently from one that generates and then returns a list of items.

To illustrate how yield works consider the following simple function:

def gen0 ():
    yield 21
    yield 42
    yield 63
    yield 84
#

Because it contains the yield keyword, this function is a generator. Note that a function is allowed to contain more than one yield statement, but using at least one makes the function a generator. When called, such a function returns a generator object; I’ll come back to that.

First let’s consider two ways of using this generator function:

# Use the generator in a for-loop...
for x in gen0():
    print(x)
print()

# Use the generator to create a list...
numbers = list(gen0())
print(numbers)
print()
#

When we run this, we get:

21
42
63
84

[21, 42, 63, 84]

The generator looks like an iterable and is providing those numbers — one-by-one, according to the for-loop. The list constructor takes an iterable and returns a list, so again it appears a generator is an iterable object.

Let’s take a closer look at what’s going on. Let’s go back to that generator object returned when we call the function:

g = gen0()
print('#1 next(g) = %d' % next(g))
print('#2 next(g) = %d' % next(g))
print('#3 next(g) = %d' % next(g))
print('#4 next(g) = %d' % next(g))
print('#5 next(g) = %d' % next(g))
#

The output of this is:

#1 next(g) = 21
#2 next(g) = 42
#3 next(g) = 63
#4 next(g) = 84
Traceback (most recent call last):
  File "generators.py", line 21, in <module>
    print('next(g) = %d' % next(g))
StopIteration

Each call to next returns a new number until the fifth one. That one causes a StopIteration error. That’s what we’d expect for an iterating object that’s reached the end of its items.

This explains what’s happening in the for loop and in the list constructor. A generator object is an iterable object. Cool. But what value does one have over a plain old list or function that returns one? Why would we write a generator rather than:

def not_a_generator ():
    return [21, 42, 63, 84]
#

(Note that, if we made this function a generator by replacing the return with yield, that generator would return a single item: the entire list.) If any generator is just passed to the list constructor (as in the second use case above), there is, as you’ll see, not much value in writing one.

Their value becomes apparent when they’re used in for-loops.

§

To see that, let’s repeat that simple generator (we’ll get more complicated later), but this time let’s instrument the heck out of it:

def gen1 (x):
    '''A generator function with three items.'''
    print('  start')
    print('  yield')
    yield x+21
    print('  resume')
    yield x+42
    print('  resume')
    yield x+63
    print('  resume')
    yield x+84
    print('  return')
    return -1

# Use the generator in a for-loop...
print('Loop Begin')
for n in gen1(37):
    print(n)
print('Loop End')
print()

# Now use it to create a list...
print('List Begin')
g = list(gen2(3))
print(g)
print('List Begin')
print()
#

I made it slightly fancy by adding a feature: this generator takes a parameter, called x, that it uses in generating return values. (It just adds x to the list of canned values.) I added one other wrinkle: the generator returns a value.

Let’s see what happens when we run the code:

Loop Begin
  start
  yield
58
  resume
79
  resume
100
  resume
121
  return
Loop End

List Begin
  start
  yield
  resume
  resume
  resume
  return
[24, 45, 66, 87]
List Begin

Here’s the crucial point; here’s why generators have value. Consider the chain of events that occurs during the for-loop:

1. The loop begins. The for-loop mechanism calls gen1(37) to kick things off and gets back a generator object.
2. The for-loop calls next on the returned object. This is the first call, so the generator starts, hits the first yield (line #5), and returns 37+21=58.
3. The for-loop regains control, taking 58 as the first value of the iteration.
4. The for-loop body prints the number 58.
5. The for-loop asks the generator for the next object, which transfers control back to the generator.
6. The generator resumes at line #6, then hits the second yield (line #7). This time it returns the value 37+42=79.
7. Back to the for-loop, which uses 79 as the next iteration value.
8. Print 79.
9. Another call to next, and resume the generator, this time at line #8.
10. … And so on; you get the idea.

The big deal is that a generator provides its values piecemeal, not as a single list. Consider the same list returned from a regular function:

def also_not_a_generator (x):
    nums = [21, 42, 63, 84]
    return [x+n for n in nums]
#

The function generates the entire list before returning the whole thing. A generator calculates and returns its items one-by-one on demand (and not at all if demand stops). In certain situations, that’s a very big value.

Providing the generator to a list constructor has no benefit because the list constructor internally invokes and exhausts the generator. There is no back-and-forth between your code and the generator.

A generator is also useful for infinite data streams (for instance a list of prime numbers). Such a generator can’t be used in a for loop, because it’ll run forever. Or rather, it can be used so long as something in the for-loop body breaks out of the loop. We’ll get to that.

One last thing: that return value doesn’t seem to have done anything. It did, but it’s hidden so far. We’ll get to that, too.

§ §

That’s the basic generator behavior. Let’s do something a little more interesting than four fixed numbers:

def gen2 (nbr_chars, base_char='A'):
    print('  start')
    b = ord(base_char)
    # Return a sequence N chars long...
    for n in range(nbr_chars):
        print('  yield')
        yield chr(b+n)
        print('  resume')
    print('  exit')

# Use the generator in a for-loop...
for b in gen2(8,'N'):
    print(b)
print()

# Use the generator to create a list...
g = list(gen2(26))
print(g)
print()
#

This is a trivial example, but it serves to show how more substantial ones might work. It generates a list of N characters, starting at a base character (‘A’ by default).

When run, it prints something similar to the example above, except with characters instead of numbers. In the for-loop, the print statements are intermingled between the generator and the loop. In the list constructor those statements happen together before the print statement.

I suspect that most list generation needs don’t benefit from being implemented as a generator, but consider something like:

def gen_remote_query (**kwargs):
    '''Makes internet queries to a database.'''
    # ... setup code ...
    for qry in queries:
        # ...connect to server...
        # ...query server...
        # ...format results...
        yield results
#

There could be a clear benefit in spreading the work across the for-loop consuming the results. It might also benefit from processing incrementally in case the connection is lost or a query crashes.

Bottom line, there are good reasons for using a generator rather than a regular function. In situations with no benefit, it’s harmless.

§

One place generators shine is infinite lists.

Suppose I wanted something like the built-in range function, but I wanted it to be infinite — it should keep giving me the next number so long as I keep asking. To make it a little fancy, let’s have it use floats instead of integers as the range function does. We’ll call it a stepper:

def stepper (start=0.0, step=1.0):
    '''Generator an infinite list of numbers.'''
    curr = float(start)
    incr = float(step)

    # Loop forever...
    while True:
        rval = yield curr
        # Resume (possible received value)...
        if rval is not None:
            # Received value is new step value...
            incr = float(rval)
            continue
        # Next increment...
        curr += incr
#

This generator takes two arguments, a starting number (default zero) and a step value (default 1.0). If called with no arguments, it will act like the range function went float and lost its mind. It’ll return floats [0.0, 1.0, 2.0,…] until — and after — the cows come home.

One difference between this generator and the previous ones is that I’m taking the return value of the yield. And it does have a potential return value. One can send a message to a running generator function. This shows up as a return value to the yield. I’ll return to this later.

One has to be careful with generators like this. The examples I used previously would both be a problem:

# Infinite generator in a for-loop; BAD IDEA!...
for n in stepper(step=0.25):
    print(n)
print()

# Infinite generator to create a list; BAD IDEA!...
#
g = list(stepper(-3.14, step=0.01))
print(g)
print()
#

Eventually Python blows up for lack of something; memory, disk, sanity. Using an infinite generator requires a different approach. There are two common ones: firstly, limit the loop to N iterations; secondly, break the loop on some trigger value.

Here’s an example of both:

# Generate [0.000, 0.125, 0.250,... 10.000]...
g1 = stepper(step=0.125)
for ix in range(81):
    n = next(g1)
    print('%4d: %5.3f' % (ix+1,n))
print()

# Generate [1.00, 0.96, 0.92,... 0.04] (approx)...
g2 = stepper(start=1, step=-0.04)
counter = next(g2)
while 0.0 < counter:
    print(counter)
    counter = next(g2)
#

In the first case, the for-loop is limited to 81 iterations. In the second, the returned value must be positive for the while loop to continue.

[A while loop should always raise your hackles a little. They require a run-time condition to terminate, and it’s up to you to make sure one happens. It’s easy to either, [a] forget to include something like an increment statement, and then the loop is eternal, or [b] change the code in a way that breaks the condition. For instance, changing to counting up from counting down, but your break condition requires going below some value — often the one you started counting up from.]

§ §

I’ll pick things up next time to talk about the yield return value. It’s a meaty enough topic to deserve its own post. And I haven’t even mentioned the yield from statement (except right there just now). I’ll get to that next time, too.

In the meantime, here’s another infinite generator example. This one returns an infinite list of random 2D points, all in some given X-Y limit. The same cautions and care apply as with any infinite generator.

def points (xmin=-1.0,xmax=1.0, ymin=-1.0,ymax=1.0):
    '''Generate infinite random X-Y points.'''
    xspan = abs(xmax - xmin)
    yspan = abs(ymax - ymin)
    while True:
        x = xmin + (random() * xspan)
        y = ymin + (random() * yspan)
        yield (x, y)

pgen = points(0.0,10.0, 0.0,10.0)
for ix in range(25):
    pt = next(pgen)
    print('[%2d]: Next point is %s' % (ix+1,pt))
print()
#

Note that it returns a tuple. It can easily be expanded to create 3D points by adding a Z member. The defaults to the generator cause it to return points in the -1.0 to +1.0 range for both X and Y.

Next time I’ll explore some more useful applications, get into the yield return value, and introduce the yield from statement. Until then, keep on coding.

[Sample code for all three posts: examples.zip]

Ø