DuckDB and BigQuery Storage API
David Gasquez
April 5, 2024
BigQuery has a not so well known API, Storage API, that let's you grab a result set or table as Arrow datasets. It is cheaper than the standard query costs and integrates with all the rest of the Arrow ecosystem, like DuckDB.
You can test it out with this code:
Gives you a result like this:
The code is also available as a Google Colab Notebook!
Discussion in the ATmosphere