Created a Bitbucket repository for pipe_controller

By Vasudev Ram


I created a Bitbucket repository for my pipe_controller Python project (referred to in my earlier posts as PipeController, but since that is the Python class name, I'm calling it by the Python module name, from now on, since that is more appropriate).

The pipe_controller repository is here.

pipe_controller is released under the BSD 3-Clause license.

- Vasudev Ram - Dancing Bison Enterprises

Share |

Using PipeController to run a pipe incrementally

By Vasudev Ram


In my earlier post about PipeController v0.1 (my experimental tool to simulate pipe-like functionality in Python), I had said that it had some interesting properties (that I did not design up front, but discovered after using it a bit). Here is one of them; it needed some code changes.

The main changes in PipeController v0.2 (download link) are:

- adding support for input and output filenames as arguments to the PipeController class's __init__ method, and also to be able to set those filenames via setter methods;

- two functions, open_for_read() and open_for_write(), to open filenames if given;

- calling of the functions to open files is now done in the run_pipe() method, not the __init__() method; this is a change that enables running pipes incrementally;

- a set_input_source() and a set_output_dest() method; this is also a change that enables running pipes incrementally;

- an add_processors() method to add a list of processors instead of just one as add_processor() did;

And this is the main code fragment (from file test_pipe_controller_03.py) that enables incremental processing (together with the above changes and other misc. stuff):

pc = PipeController(input_source = in_filename)
 processors = [ oto0, eto3, upcase, delspace ]
 ctr = 1

 for processor in processors:
  out_filename = out_filename_prefix + str(ctr).zfill(3)
  tmp_str = "Run #%d: out_filename = %s" % (ctr, out_filename)
  debug(tmp_str)
  pc.set_output_dest(out_filename) 
  pc.add_processor(processor)
  debug("before pc.runpipe()")
  pc.run_pipe()
  ctr += 1
 debug("exiting main")

You can download PipeController v0.2 here.

With these changes, it is possible to run PipeController pipes incrementally.

Refer to the Python functions used in my earlier post, while reading the description below:

By "incrementally", I mean: first run a pipe with only one component, say oto0; then run it with oto0 piped to eto3; then with oto0 piped to eto3 piped to upcase; and so on.

In each run, the output can go to a different file. This can help to debug the pipe's logic: run it incrementally, and check the output generated by each stage.

Another use for this feature, is that it saves the intermediate outputs, each of which may be useful in their own right. This can commonly occur in business (or scientific or other) data-processing situations.

An example with a UNIX analogy: with PipeController v0.2, you can do the Python equivalent of this command sequence:

$ cat it1 | ot0 > ot1-001

$ cat it1 | oto0 | eto3 > ot1-002

$ cat it1 | oto0 | eto3 | upcase > ot1-003

$ cat it1 | oto0 | eto3 | upcase | delspace > ot1-004

(assuming that you have UNIX commands equivalent to the functions oto0, eto3, upcase and delspace mentioned in my previous post; and such commands are trivial to write using commands like tr, etc.)

The above 4 commands result in each incremental run of the pipe (with an additional command tacked on the end each time), generating its output in a different file.

You can do the same sort of thing with PipeController, but with a single program, like this (using the same Python functions and same input file as in the original post about v0.1):

$ python test_pipe_controller_03.py it1 ot1-

The last argument in the above line is actually "ot1-", not a typo; it is a prefix to the output filenames that will be generated by the program. And test_pipe_controller_03.py is a new test program, part of the v0.2 release.

For the input file it1 (same one as in the original post),

$> cat it1
1  some lowercase text
2  more lowercase text
3  even more lowercase text
4  yet more lowercase text

the incremental outputs from the above program run are as follows:

$> cat ot1-001
1  s0me l0wercase text
2  m0re l0wercase text
3  even m0re l0wercase text
4  yet m0re l0wercase text

$> cat ot1-002
1  s0m3 l0w3rcas3 t3xt
2  m0r3 l0w3rcas3 t3xt
3  3v3n m0r3 l0w3rcas3 t3xt
4  y3t m0r3 l0w3rcas3 t3xt

$> cat ot1-003
1  S0M3 L0W3RCAS3 T3XT
2  M0R3 L0W3RCAS3 T3XT
3  3V3N M0R3 L0W3RCAS3 T3XT
4  Y3T M0R3 L0W3RCAS3 T3XT

$> cat ot1-004
1  S0M3L0W3RCAS3T3XT
2  M0R3L0W3RCAS3T3XT
3  3V3NM0R3L0W3RCAS3T3XT
4  Y3TM0R3L0W3RCAS3T3XT

The final output here is the same as in the original example in my previous post, but now you also have the intermediate results generated, for debugging or other uses.

Some notes:

1. I've renamed the file containing the PipeController class, from the earlier name pipes.py (in v0.1) to pipe_controller.py, to avoid a clash with the pipes module in the standard Python library. Also, pipe_controller.py is now a module; i.e., you can now do "from pipe_controller import PipeController" in your own Python program to use the PipeController class; see the file test_pipe_controller_03.py as an example of that.

2. I've also renamed the v0.2 zip file from the earlier name unix-pipes.zip, to pipe_controller-v0.2.zip, because a reader on the comp.lang.python newsgroup, rightly commented that calling it unix-pipes was a bit misleading (though unintentional), since PipeController does not enable IPC between programs, as UNIX pipes do; it only enables a sort of pipelined communication between functions in a single Python program.

- Vasudev Ram - Dancing Bison Enterprises

Share |

PipeController v.01 released - simulating UNIX-style pipes in Python

By Vasudev Ram


Some time ago, I had written a post about ways of doing UNIX-style pipes in Python. It had links to some different tools that enable you to do that.

More recently, I wrote another post about one more such tool, Plumbum:

Plumbum, UNIX shell-like library and tool in Python

Recently, I worked on implementing something on the same lines myself. I've tentatively named it PipeController, for lack of a better name. I did google for shorter and better-sounding names, mostly variations on the word "pipe", but most of them were already taken by other software products or other stuff. So I've settled on this name for now.

PipeController is a tool to experiment with a simple, sequential, synchronous simulation of UNIX-style pipes in Python. It's the first release, and has only a little functionality as of now.

The main source file is pipes.py. Apart from class PipeController, it has a main function that does a simple test of setting up and running a pipe.

How to use PipeController:

Each component of the "pipe" is to be implemented by the calling program as a Python function. Each function should take one input and return one output, which should be the result of processing the input.

The tool (basically, the class PipeController), takes care of setting up the pipe and running it.

You have to create an instance of PipeController and calls its methods to make that happen. The main methods to be used are:

- PipeController.add_processor(func_name)
- PipeController.run_pipe()

The add_processor() method just adds the given function name (which should be passed as a bare name, without trailing parentheses), to a list in the instance.

The run_pipe() method actually runs the pipe. It loops over the lines of input (text, for now) in sys.stdin, and passes them to each of the functions in turn, using an inner loop:

def run_pipe(self):
        item = self._pipe_input.readline()
        while item != '': # while not EOF
            result = item
            for processor in self._processors:
                result = processor(result)
            self._pipe_output.write(result)
            item = self._pipe_input.readline()
        self._pipe_output.close()

You can call add_processor() any number of times. Each call will append one function to the pipeline. The functions will be called on the input in the order they are added to the instance.

The default input for the whole pipeline is sys.stdin and the default output is sys.stdout. As of now there is no support or usage example for changing the default input source or output destination programmatically, although the PipeController.__init__() method's signature indicates that you can. I put that in so I can work on it later.

The run_pipe() method loops over the lines in the input, passes each line to the chain of functions, one by one, with each function's output becoming the next function's input (as in the case of UNIX pipes, except there we have commands instead of functions), and writes the final result for each line to the output.

Sample usage (some code omitted):

pc = PipeController()
pc.add_processor(oto0)
pc.add_processor(eto3)
pc.add_processor(upcase)
pc.add_processor(delspace)
pc.run_pipe()

where the Python functions oto0, eto3, upcase and delspace convert occurrences of "o" to "0", "e" to "3", letters to uppercase, and delete spaces (from their input), respectively. (See the code in the PipeController source zip file linked to below, for those function definitions and the rest of the code).

If that was all that PipeController provided, it could be replaced by just nesting / composing function calls, with the innermost call taking the input string (line). E.g. Instead of the for loop in the run_pipe() method, we could use:

f(g(h(item)))

or, to use the example functions in the code:

delspace(upcase(eto3(ot0(item)))

But while experimenting with the code after first writing it, I discovered that it has a few interesting properties (and hence potential uses), some of which may not be as convenient to achieve using the functional composition method. I'll write about that in an upcoming post or two.

You can download PipeController v0.1 here.

It has a few input files that are meant to be used with the existing main function that tests the PipeController class.

Example usage (where it1 is one of the input text files):

On UNIX/Linux:

$ cat it1 | python pipes.py

or

$ python pipes.py < it1

On Windows:

C:> type it1 | python pipes.py

or

C:> python pipes.py < it1

The input file it1 contains:

1    some lowercase text
2    more lowercase text
3    even more lowercase text
4    yet more lowercase text

After the above commands run (any one), the output is:

1    S0M3L0W3RCAS3T3XT
2    M0R3L0W3RCAS3T3XT
3    3V3NM0R3L0W3RCAS3T3XT
4    Y3TM0R3L0W3RCAS3T3XT

which is the result of processing the input using the pipeline.

To summarize, again:

it1 is an input text file containing a few lines of text. The above command transforms the contents in the following ways, via the pipeline: converts occurrences of 'o' to '0', then occurrences of 'e' to '3', then upper-cases letters, and finally deletes spaces.

Each of these is conversions is done by a corresponding Python function in the program (the functions named above). The whole pipeline is setup and run by an instance of the PipeController class.

You can define any functions of your own (and any number of them) and use them to create a pipeline for your own purposes. The only requirement is that each function should take a string (representing a line of text) as input (i.e. its argument), and return the processed string.

Enjoy, and please give your feedback, if any.

- Vasudev Ram - Dancing Bison Enterprises

Share |