Skip to main content

DockerHub and Quay.io returning 429 due to rate limits

Daniel Kane
Updated

quay.io returning 429 and rate limits:

Redhat owns quay.io, and this page explains under what conditions that quay.io can return limits.

https://access.redhat.com/articles/5531191

"The best defense against the Docker Hub rate limiting is to move any base images needed to quay.io directly. Quay.io does not restrict the number of public repositories for any user (either free or paid). In addition, quay.io does not restrict anonymous pulls against its repositories (either public or private) and only rate limits in the most severe circumstances to maintain service levels (e.g. tens of requests per second from the same IP address)."

This other vague page may show on Google Searches:

https://docs.quay.io/issues/429.html

dockerhub.com rate limits

Pricing showing rate limits: https://www.docker.com/pricing/

Announcement: https://www.docker.com/increase-rate-limits

FAQ: https://www.docker.com/pricing/resource-consumption-updates

Doc: https://docs.docker.com/docker-hub/download-rate-limit/

  • A pull request is defined as up to two GET requests on registry manifest URLs (/v2/*/manifests/*).
  • A normal image pull makes a single manifest request.
  • A pull request for a multi-arch image makes two manifest requests.
  • HEAD requests are not counted.
  • Limits are applied based on the user doing the pull, and not based on the image being pulled or its owner.
  • The “overall rate limit” will return a simple "toomanyrequests: 429 Too Many Requests" response. The pull limit returns a longer error message that includes a link to this page.

Client side error message (manifest unknown)

$ docker pull nexus-test.local/alpine/git:v2.34.2
Error response from daemon: manifest for nexus-test.local/alpine/git:v2.34.2 not found: manifest unknown: manifest unknown

 

How to reduce the outgoing requests to https://registry-1.docker.io:

  • Set an appropriate "Maximum metadata age" (eg: 1440)
    As the rate limit is counted by the GET requests to /v2/*/manifests/* path, it is important to set not too short value for "Maximum metadata age".
  • Set an appropriate "Maximum component age" (eg: -1)
    The example value "-1" prevents the Nexus docker proxy repository for checking for changes to already existing images.
  • (Optional) Set an appropriate "Not found cache TTL" (eg: 1440)
    As the Docker web site says "the total number of pulls that can be performed", 401/404 responses wouldn't be counted, but just in case, recommended to set "Not found cache".
  • For Docker group repositories, set the docker hub proxy repository to the end of the Group Members list
    Ref: https://help.sonatype.com/en/repository-management.html#managing-repositories-and-repository-groups

 

How to set Docker Hub username and password:

From Administration => Repository => Repositories, open your docker proxy repository, then scroll down to the HTTP section like blow:

mceclip0.png

Then type your Docker Hub username and password, then click [ Save ] button.

 

How to check the rate limits from the nexus.log

From Administration => Support => Logging page, click [ Create Logger ] button:

mceclip1.png

Type "org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl" and select "DEBUG".
If NewDB (PostgreSQL or H2) is used, "org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerProxyFacetImpl"

mceclip2.png

 

After that, when user request an image to Docker hub, you will see the logging, which contains "RateLimit-Limit" and "RateLimit-Remaining", like below:

//--------------------------------------------------------------------------------------

2020-11-03 01:29:37,882+0000 DEBUG [qtp1720842924-2989] node-nxrm-ha1.standalone.localdomain anonymous org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl - Response: HttpResponseProxy{HTTP/1.1 200 OK [Content-Length: 2621, Content-Type: application/vnd.docker.distribution.manifest.v2+json, Docker-Content-Digest: sha256:d6d62d888f917e50ce65e60b9b6f0af32cd26cc519eafb14ad5836f5337ef6f1, Docker-Distribution-Api-Version: registry/2.0, Etag: "sha256:d6d62d888f917e50ce65e60b9b6f0af32cd26cc519eafb14ad5836f5337ef6f1", Date: Tue, 03 Nov 2020 01:29:37 GMT, Strict-Transport-Security: max-age=31536000, RateLimit-Limit: 5000;w=21600, RateLimit-Remaining: 4999;w=21600] ResponseEntityProxy{[Content-Type: application/vnd.docker.distribution.manifest.v2+json,Content-Length: 2621,Chunked: false]}}

--------------------------------------------------------------------------------------//

After checking the rate limit, please restore the logging by clicking [ Reset to Default Level ] button.

Ref: https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/

 

Overall rate limits

https://docs.docker.com/docker-hub/download-rate-limit/#other-limits

The overall rate limits is not logged in above DEBUG logging. So please use "org.sonatype.nexus.httpclient.outbound" DEBUG logging, which logs the status code 429 like below:

//--------------------------------------------------------------------------------------

2022-06-17 00:20:32,417+0000 DEBUG [qtp2127697935-203564] anonymous org.sonatype.nexus.httpclient.outbound - https://auth.docker.io/token?service=registry.docker.io&scope=repository%3Aalpine%2Fgit%3Apull < HTTP/1.1 429 Too Many Requests @ 37.20 ms

--------------------------------------------------------------------------------------//

Also, nexus.log might contain WARN like below:

//--------------------------------------------------------------------------------------

2022-06-17 00:17:57,017+0000 WARN [qtp2127697935-203567] some-user org.apache.http.impl.auth.HttpAuthenticator - BEARER [complete=true] authentication error: Could not retrieve token from https://auth.docker.io/token. Status code: 429

--------------------------------------------------------------------------------------//

Related articles