Python generators are pluggable

By Vasudev Ram

Generator image attribution

While working on a Python project, it crossed my mind that generators could be of use in it. A little research made me realize that generators are pluggable, i.e. they can be passed to functions, and then be used within those functions. This is because generators are a kind of Python object, and any Python object can be passed as an argument to a function.

This in turn is because almost everything in Python is an object (including generators), similar to how almost everything in Unix is a file. Both those concepts can enable some powerful operations.

Here is a program that demonstrates passing generator objects as arguments to another function, and then using those generators inside it:

# Program to show that generators are pluggable, i.e.,
# can be passed as function arguments, and then used 
# inside those functions to which they are passed.
# Author: Vasudev Ram - http://jugad2.blogspot.com
# Copyright 2016 Vasudev Ram

def gen_squares(fro, to):
    '''A generator function that returns a generator 
    that returns squares of values in a range.'''
    for val in range(fro, to + 1):
        yield val * val

def gen_cubes(fro, to):
    '''A generator function that returns a generator 
    that returns cubes of values in a range.'''
    for val in range(fro, to + 1):
        yield val * val * val

def use(gen):
    print "In use() function:"
    print "Using:", gen
    print "Items:",
    for item in gen:
        print item,
    print

print "Pluggable Python generators.\n"
print "In main module:"
print "type(use): ", type(use)
print "use:", use
print
print "type(gen_squares): ", type(gen_squares)
print "gen_squares: ", gen_squares
print "type(gen_squares(1, 5)): ", type(gen_squares(1, 5))
print "gen_squares(1, 5): ", gen_squares(1, 5)
print
print "type(gen_cubes): ", type(gen_cubes)
print "gen_cubes: ", gen_cubes
print "type(gen_cubes(1, 5)): ", type(gen_cubes(1, 5))
print "gen_cubes(1, 5): ", gen_cubes(1, 5)
print
for gen_obj in (gen_squares(1, 5), gen_cubes(1, 5)):
    use(gen_obj)
    print

Run the program with:

python pluggable_generators.py

Here is the output:

Pluggable Python generators.

In main module:
type(use):  <type 'function'>
use: <function use at 0x0202C3B0>

type(gen_squares):  <type 'function'>
gen_squares:  <function gen_squares at 0x0207BF30>
type(gen_squares(1, 5)):  <type 'generator'>
gen_squares(1, 5):  <generator object gen_squares at 0x020869B8>

type(gen_cubes):  <type 'function'>
gen_cubes:  <function gen_cubes at 0x0207BFB0>
type(gen_cubes(1, 5)):  <type 'generator'>
gen_cubes(1, 5):  <generator object gen_cubes at 0x020869B8>

In use() function:
Using: <generator object gen_squares at 0x020869B8>
Items: 1 4 9 16 25

In use() function:
Using: <generator object gen_cubes at 0x020869E0>
Items: 1 8 27 64 125

As you can see, I've printed both type(obj) and obj for many of the objects shown, to make it more clear what is going on. Also, a generator function and a generator object (the result of calling a generator function), are two different things, so they are printed separately as well.

A few points about generators and their use:

They can potentially lead to less memory usage, since values are only generated on demand, i.e. evaluation is lazy.

They can help with separation of concerns, a key technique that leads to program modularity; the code for the actual generator functions like gen_squares and gen_cubes does not have to be embedded in the use() function, which makes both the generators and the use() function more reusable.

Someone could say here that we could write gen_squares and gen_cubes as regular functions instead of as generator functions, and then just call them from use(), so their code still does not have to be embedded in the use() function, and that would be right. But in that case, the calls to them would return lists, and if the lists were very large, that would use a lot of memory, and maybe crash or slow down the program. Those issues will not happen with generators, though, because each item is generated just before it is used, and then it is thrown away, not stored. So the memory needed is not proportional to the number of items generated.

Here are some links about Python generators:

Generators - Python Wiki

Stack Overflow - Understanding generators in Python

The image at the top of the post is a Ferranti two-phase AC generator set.

- Vasudev Ram - Online Python training and programming

Signup to hear about new products and services I create.

Posts about Python  Posts about xtopdf

My ActiveState recipes

Share |

Follow up #1 on "regular" functions versus generator functions in Python

By Vasudev Ram

Two blog posts ago, in:

Exploring "regular" functions versus generator functions in Python,

I said that I would describe the points of interest that I found about the two versions, one that used a regular function and the other that used a generator function.

Here are some of those points:

1. The generator feature of the Python language was originally developed to enable programmers to create functions that could generate a series of values, not just a single value.

Note: "a series of values" does not mean just a function that can return multiple values (all at the same time). That could be done, trivially, in Python, like this, before generators were added to the language:

>>> def foo():
...     return 1, 2, 3
...
>>> a = foo()
>>> a
(1, 2, 3)

This is just a function whose return value consists of more than one item. Actually, since the value returned is really a tuple, you can argue that there is only one return value:

>>> a = foo()
>>> type(a)

Even if you do this:

>>> a, b, c = foo()
>>> a
1
>>> b
2
>>> c
3

what is happening is that the function returns a single value, a tuple, and then tuple unpacking is used to assign the items of the returned tuple to a, b, and c.

But generators can "return" (or rather, "yield", using the yield keyword), a series of values over time, on demand.

And that is what the lazy_text_proc.py program does; it yields a series of processed lines, on demand, in the for loop. So far, so good - nothing new that the Python docs don't say.

2. Now, coming to the differences between the two programs:

I initially thought (correctly, as it turned out), that the generator version would use less memory than the non-generator version. That seems to be right, on studying the code of both versions, since the non-generator version builds up a list of processed lines in memory as the input file is read and processed, and only then returns the list to its caller, while the generator version does not build up any list, it only returns each processed line to its caller, in the for loop, on each iteration.

But the non-generator version of this program does not necessarily have to return a list of all the processed lines to its caller (for printing or other further processing). Instead, the code that the caller would use to print or process the lines further, can simply be put in the non-generator function, below the line:

new_line = process_line(line, old_pat, new_pat)

and eliminate the use of the list, ending up with this code:

# Process a text file, calling process_line on each line.
def regular_text_proc(filename, old_pat, new_pat):

    with open(filename) as fp:
        for line in fp:
            new_line = process_line(line, old_pat, new_pat)
            result = process_line_more(new_line) # where process_line_more could just be a print,
            # or could be something else.

With this change, the non-generator version would take about the same amount of memory as the generator version.

So what is the advantage of generators in this case?

None, practically (*), except for the possibly clearer code resulting from the separation of concerns of the reading and processing stages.

The moral of the story seems to be that one should not apply language features blindly, but check whether they really are useful and achieve the desired result, and also whether that result can be achieved more simply.

This is not to say that generators are not useful at all, obviously (**); it is just that the use of generators in this example does not seem to be of much benefit (if compared to the modified non-generator version).

(*) I think I would still use the generator version in this case, due to the benefit of separation of concerns - the code just seems somewhat cleaner / easier to understand and maintain in this way.

(**) For example, here is an example where the use of generators may improve performance while still simplifying the code:

Use generators for fetching large db record sets (Python recipe). I had seen an example similar to this in the 2nd Edition of the Python Cookbook by O'Reilly Media, but don't have the book handy right now. This example seems to be roughly the same as that cookbook one, though, IIRC.

I have only a basic understanding of generators myself, and wrote these posts to explore them, hence the title of the first post. Comments are welcome.

- Vasudev Ram - Dancing Bison Enterprises

Contact me

Share |

Exploring "regular" functions versus generator functions in Python

By Vasudev Ram

This post is about "regular" functions versus generator functions in Python. I'm using the term "regular" functions for lack of a better word; what I mean by that is non-generator functions.

Consider this text file, test1.txt:

this is a line with a foo and another foo and one more foo.
the foo brown foo jumped over the lazy foo
foo are you. you are foo.

Here is a program, with a "regular" function, to process all the lines in that text file:

# regular_text_proc.py

import string

# Replace instances of the string old_pat with new_pat in line.
def process_line(line, old_pat, new_pat):
    return line.replace(old_pat, new_pat)

# Process a text file, calling process_line on each line.
def regular_text_proc(filename, old_pat, new_pat):

    new_lines = []
    with open(filename) as fp:
        for line in fp:
            new_line = process_line(line, old_pat, new_pat)
            new_lines.append(new_line)
    return new_lines

def main():

    newlines = regular_text_proc("test1.txt", "foo", "bar")

    print "new file:"
    for line in newlines:
        print line,

main()

This command:

python regular_text_proc.py

gives this output:

new file:
this is a line with a bar and another bar and one more bar.
the bar brown bar jumped over the lazy bar
bar are you. you are bar.

Here is a program, with a generator function, to do the same kind of processing of the same file:

# lazy_text_proc.py
# Lazy text processing with Python generators.

import string

# Replace instances of the string old_pat with new_pat in line.
def process_line(line, old_pat, new_pat):
    return line.replace(old_pat, new_pat)

# Process a text file lazily, calling process_line on each line.
def lazy_text_proc(filename, old_pat, new_pat):
    with open(filename) as fp:
        for line in fp:
            new_line = process_line(line, old_pat, new_pat)
            yield new_line

def main():
    newlines = lazy_text_proc("test1.txt", "foo", "bar")
    print "type(newlines) =", type(newlines)
    # Line below will give error if uncommented, because
    # newlines is not a list, it is a generator.
    #print "len(newlines) =", len(newlines)
    print "new file:"
    for lin in newlines:
        print lin,

main()

This command:

python lazy_text_proc.py

gives the same output as the regular_text_proc.py program, except for the type(newlines) output, which I added, to show that the variable called 'newlines', in this program, is not a list but a generator. (It is a list in the regular_text_proc.py program.)

I found the difference between these two programs, one with a regular function and the other with a generator function, to be interesting in a few ways. I'll discuss that in my next blog post.

The Wikipedia article on generators is of interest.

- Vasudev Ram - Dancing Bison Enterprises

Contact me

Share |