Hf sync hangs on uploading to bucket

I’m running hf sync on a bucket with a few hundred thousand files. I’m using version 1.14.0, and from my experience, the issue seems to have become worse in newer versions.

I even wrote a custom script that downloads files in batches of 200, but the utility still freezes from time to time. It seems that all those issues are connected to hidden limit on the number of files that can be transferred per minute. To work around that, you may need to package the data into a format like WebDataset.