It looks like performance tuning guideline of MinIO suggests disabling IPv6 for the instance all together. This is weird but I will still give the whole guideline a shot (https://github.com/minio/minio/blob/master/docs/tuning/tuned.conf) which first requires me to switch the existing cluster to IPv4 from public IPv6.