Yesterday, our runners started hitting the DockerHub rate limits. After some investigation, we discovered that DockerHub limits IPv6 by the first 64 bits of the address instead of the full 128 bits. This means that all runners on the same host share the same rate limit. DockerHub support also confirmed that they began enforcing this limit yesterday, which explains why we started encountering it then. They limit IPv4 addresses by the full address, not by prefix bits, allowing us to pull more images using IPv4. To quickly address this issue, we completely disabled IPv6 on the runner virtual machines at d9074187dca46efd7a94b9803db85cb243f7b20. However, disabling IPv6 only for DockerHub requests would be a better solution, as runners can still use IPv6 for other requests. The `address=/.docker.io/::` line instructs dnsmasq to return a NULL IPv6 address (::) for all AAAA queries for it, effectively filtering out AAAA records. dnsmasq man: An address specified as '#' translates to the NULL address of 0.0.0.0 and its IPv6 equivalent of :: so --address=/example.com/# will return NULL addresses for example.com and its subdomains. This is partly syntactic sugar for --address=/example.com/0.0.0.0 and --address=/example.com/:: but is also more efficient than including both as separate configuration lines. Note that NULL addresses normally work in the same way as localhost, so beware that clients looking up these names are likely to end up talking to themselves. |
||
---|---|---|
.. | ||
common | ||
host | ||
inference_endpoint | ||
minio/bin | ||
postgres/bin | ||
Gemfile | ||
Gemfile.lock |