Decoupling producers and consumers of iterables with generators in Python

Generators can help you decouple the production and consumption of iterables - making your code more readable and maintainable. I learned this trick a few years back from David Beazley's Generator tricks for systems programmers slides. Consider this example:

Now, how'd you decouple the print statement from the infinite_counter? Since the function never returns, you can't collect the outputs in an iterable, return the container, and print the elements of the iterable in another function. You might be wondering why would you even need to do it. I can think of two reasons:

• The infinite_counter function is the producer of the numbers and the print function is consuming them. These are two separate responsibilities tangled in the same function which violates the Single Responsibility Principle.

• What'd you do if you needed a version of the infinite counter where the consumer had different behavior?

One way the second point can be addressed is - by accepting the consumer function as a parameter and applying that to the produced value.

You can override the value of consumer with any callable and make the function more flexible. However, applying multiple consumers will still be hairy. Doing this with generators is cleaner. Here's how you'd transform the above script to take advantage of generators:

The infinite_counter returns a generator that can lazily be iterated to produce the numbers and you can call any arbitrary consumer on the generated result without coupling it with the producer.

Writing a workflow that mimics 'tail -f'

In a UNIX system, you can call tail -f | grep to print the lines of a file in real-time where the lines match a specific pattern. Running the following command on my terminal allows me to tail the syslog file and print out any line that contains the word xps:

If you look carefully, the above command has two parts. The tail -f returns the new lines appended to the file and grep consumes the new lines to look for a particular pattern. This behavior can be mimicked via generators as follows:

Here, the tail_f continuously yields the logs, and the grep function looks for the pattern xps in the logs. Replacing grep with any other processing function is trivial as long as it accepts a generator. The tail_f function doesn't know anything about the existence of grep or any other consumer function.

Continuously polling a database and consuming the results

This concept of polling a log file for new lines can be extended to databases and caches as well. I was working on a microservice that polls a Redis queue at a steady interval and processes the elements one by one. I took advantage of generators to decouple the function that collects the data and the one that processes the data. Here's how it works:

You'll need to run an instance of Redis server and Redis CLI to test this out. If you've got Docker installed in your system, then you can run docker run -it redis to quickly spin up a Redis instance. Afterward, run the above script and start the CLI. Print the following command on the CLI prompt:

The above script should print the following:

This allows you to define multiple consumers and run them in separate threads/processes without the producer ever knowing about their existence at all.