{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/python/decouple-with-generators/",
  "description": "Learn how Python generators decouple data production from consumption, enabling cleaner code for streaming, polling, and pipeline patterns.",
  "path": "/python/decouple-with-generators/",
  "publishedAt": "2022-04-03T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python"
  ],
  "textContent": "Generators can help you decouple the production and consumption of iterables - making your\ncode more readable and maintainable. I learned this trick a few years back from David\nBeazley's [Generator tricks for systems programmers] slides. Consider this example:\n\nNow, how'd you decouple the print statement from the infinite_counter? Since the function\nnever returns, you can't collect the outputs in an iterable, return the container, and print\nthe elements of the iterable in another function. You might be wondering why would you even\nneed to do it. I can think of two reasons:\n\n- The infinite_counter function is the producer of the numbers and the print function is\n  consuming them. These are two separate responsibilities tangled in the same function which\n  violates the [Single Responsibility Principle].\n\n- What'd you do if you needed a version of the infinite counter where the consumer had\n  different behavior?\n\nOne way the second point can be addressed is - by accepting the consumer function as a\nparameter and applying that to the produced value.\n\nYou can override the value of consumer with any callable and make the function more\nflexible. However, applying multiple consumers will still be hairy. Doing this with\ngenerators is cleaner. Here's how you'd transform the above script to take advantage of\ngenerators:\n\nThe infinite_counter returns a generator that can lazily be iterated to produce the\nnumbers and you can call any arbitrary consumer on the generated result without coupling it\nwith the producer.\n\nWriting a workflow that mimics 'tail -f'\n\nIn a UNIX system, you can call tail -f <filename> | grep <pattern> to print the lines of a\nfile in real-time where the lines match a specific pattern. Running the following command on\nmy terminal allows me to tail the syslog file and print out any line that contains the\nword xps:\n\nIf you look carefully, the above command has two parts. The tail -f <filename> returns the\nnew lines appended to the file and grep <pattern> consumes the new lines to look for a\nparticular pattern. This behavior can be mimicked via generators as follows:\n\nHere, the tail_f continuously yields the logs, and the grep function looks for the\npattern xps in the logs. Replacing grep with any other processing function is trivial as\nlong as it accepts a generator. The tail_f function doesn't know anything about the\nexistence of grep or any other consumer function.\n\nContinuously polling a database and consuming the results\n\nThis concept of polling a log file for new lines can be extended to databases and caches as\nwell. I was working on a microservice that polls a Redis queue at a steady interval and\nprocesses the elements one by one. I took advantage of generators to decouple the function\nthat collects the data and the one that processes the data. Here's how it works:\n\nYou'll need to run an instance of [Redis] server and [Redis CLI] to test this out. If you've\ngot [Docker] installed in your system, then you can run docker run -it redis to quickly\nspin up a Redis instance. Afterward, run the above script and start the CLI. Print the\nfollowing command on the CLI prompt:\n\nThe above script should print the following:\n\nThis allows you to define multiple consumers and run them in separate threads/processes\nwithout the producer ever knowing about their existence at all.\n\n\n\n\n[generator tricks for systems programmers]:\n    https://www.dabeaz.com/generators/Generators.pdf\n\n[single responsibility principle]:\n    https://en.wikipedia.org/wiki/Single-responsibility_principle\n\n[redis]:\n    https://redis.io/\n\n[redis cli]:\n    https://redis.io/docs/ui/cli/\n\n[docker]:\n    https://www.docker.com/",
  "title": "Decoupling producers and consumers of iterables with generators in Python"
}