mirror of
https://github.com/ubicloud/ubicloud.git
synced 2025-10-08 15:51:57 +08:00
5e34c5a809
We previously used curl to download images from presigned URLs. This worked well for our internal MinIO cluster, but with R2 presigned URLs, curl performs poorly since it downloads with a single connection. First, I benchmarked aria2 [^1], which supports multiple connections. However, Daniel recommended using a memory-safe tool instead and suggested htcat [^2], which his team wrote about 10 years ago. After a few small improvements, htcat now works well for our use case and delivers performance comparable to aria2. If the URL is from R2, we now use htcat to download it with multiple connections. Since htcat pipes to stdout, we calculate the sha256 hash while downloading similar to how we did with curl. I will download the htcat binary on existing hosts. [^1]: https://aria2.github.io/ [^2]: https://github.com/htcat/htcat |
||
|---|---|---|
| .. | ||
| common | ||
| host | ||
| inference_endpoint | ||
| kubernetes | ||
| minio/bin | ||
| postgres | ||
| victoria_metrics/bin | ||
| .rubocop.yml | ||
| Gemfile | ||
| Gemfile.lock | ||