{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/python/switch-between-multiple-datastreams/",
  "description": "Poll multiple data sources in a single thread using Python generators with itertools.cycle to alternate between infinite data streams efficiently.",
  "path": "/python/switch-between-multiple-datastreams/",
  "publishedAt": "2023-02-19T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python",
    "Database",
    "TIL"
  ],
  "textContent": "I was working on a project where I needed to poll multiple data sources and consume the\nincoming data points in a single thread. In this particular case, the two data streams were\ncoming from two different Redis lists. The correct way to consume them would be to write two\nseparate consumers and spin them up as different processes.\n\nHowever, in this scenario, I needed a simple way to poll and consume data from one data\nsource, wait for a bit, then poll and consume from another data source, and keep doing this\nindefinitely. That way I could get away with doing the whole workflow in a single thread\nwithout the overhead of managing multiple processes.\n\nHere's what I'm trying to do:\n\nOne way is to poll the data sources in two generator functions and yield the result. Then in\nthe consumer, we'll have to alternate between the generators to fetch the next result like\nthis:\n\nLet's make a concrete example out of the pesudocode:\n\nThe code above defines two generator functions, stream_even() and stream_odd(), that use\nthe count() function from the itertools module to generate an infinite sequence of even\nand odd integers respectively.\n\nThe consume() function creates a tuple containing the two generator objects, and enters an\ninfinite loop. On each iteration of the loop, it iterates over the tuple using a for loop;\neffectively alternating between the two streams. In each iteration, it waits for 1 second\nusing the time.sleep() function and then uses the next() function to retrieve the next\nitem from the current stream. If the result is not None, it prints a message to the\nconsole indicating which stream it came from and what the value was. Else, it loops back to\nthe beginning of the iteration.\n\nRunning the snippet will print the folling output to the console:\n\nThe consumer infinite loop can be written in a more concise manner with itertools.cycle.\nInstead of using the while loop, we can use this function to indefinitely cycle between\nthe elements of an iterable.\n\nHere, the finalized executable script:\n\nFurther reading\n\n- [itertools.cycle]\n\n\n\n\n[itertools.cycle]:\n    https://docs.python.org/3/library/itertools.html#itertools.cycle",
  "title": "Switching between multiple data streams in a single thread"
}