Async Batch Requests in Python

As a data engineer, one of the most common tasks I perform is getting data from an API. For a long time, I've been using the requests library to make these requests. However, I recently discovered the httpx library, which has a built-in support for asynchronous requests. At the same time, I've worked on a couple of projects that required a smarter approach than just making sequential requests, and I worked on abatcher to abstract away some of the complexity. Let's go through multiple examples of doing 100 requests with different approaches. Sequential Requests Doing 100 sequential requests with the httpx library looks like this: Async Requests The same thing can be done asynchronously with httpx's AsyncClient: Now, doing this to a random API might not be super friendly to the API provider. In most cases, APIs have a limit on the number of requests per minute. Async Requests with Batching The easiest way to do this is to use the httpx.AsyncClient with a semaphore. This will limit the number of concurrent requests to the API at any given time. This works pretty well and might cover most of your use cases. However, there are places where you'll be rate limited by the API provider allowing only a certain number of requests per minute. Async Requests with Batching and Rate Limiting To handle this, we can use the aiometer library which allows us to limit the number of concurrent requests. The same 100 requests we did before, but with rate limiting looks like this (extracted from the aiometer example in their README): You can tweak the maxatonce and maxpersecond options to fine-tune concurrency! Conclusion The httpx library combined with the aiometer library is a great addition to your toolbelt if you're doing a lot of API requests. I've also made (alongside Cursor) a small and probably buggy Python package, abatcher, with this functionality abstracted away behind a simple interface. Check out abatcher on GitHub! Here's how you can use it: Let me know if you have any feedback!