Glossary
HTTP 429 Too Many Requests
HTTP 429 Too Many Requests tells the client it has sent more requests than the server is willing to accept in a given window. It is a temporary refusal, not a rejection of the request itself: the same request may succeed later. What decides whether a client handles it well is the Retry-After header, which turns a guess about when to try again into a number the client can act on.
Why Retry-After matters more than the status
Without it, a client has to guess. The common guess is exponential backoff, which works and wastes time when the real window is short.
The less common guess is retrying immediately, which turns a rate limit into a loop and can keep the limit permanently exhausted.
With it, the client waits exactly as long as needed and the next request succeeds.
The header accepts either a number of seconds or an HTTP date, and the two are equivalent to a client that reads them.
Publishing budget headers alongside — a limit, a remaining count, a reset — lets a well-behaved client slow down before reaching the refusal at all.
What a crawler does with one
Search crawlers treat sustained 429 responses as a signal to reduce their crawl rate, which is the intended behaviour and usually desirable.
A site returning 429 to a crawler across many pages for a long period will be crawled less, and pages already indexed can eventually be dropped.
A short burst is handled gracefully and has no lasting effect, which is the difference between protecting a server and throttling discovery.
Which is why rate limiting aimed at abusive traffic should be narrow enough not to catch legitimate crawlers doing ordinary work.
Where it is commonly used wrongly
Returning 503 instead, which says the whole service is unavailable rather than that this client has exceeded a quota.
Returning 403, which tells the client the request will never be accepted and stops it retrying at all.
Returning 429 without Retry-After, which is valid and leaves every client guessing.
Applying the limit per address behind a shared network, which refuses an entire office because of one client.
How it is observed
Rate-limit headers are read from the responses a scan already received. Nothing is sent to locate or exercise a limit, because doing so would be a load test.
A 429 encountered during a crawl is recorded as what it is, and the affected pages are reported as not verified rather than as clean.
Absence of rate-limit headers is reported as a lack of published information, not as an absent control — plenty of gateways limit without surfacing counters.
A crawl that receives sustained 429 responses reports reduced coverage, which is what keeps the rest of the findings honest.
Frequently asked questions
- Is 429 bad for search ranking?
- A short burst is handled gracefully. Sustained 429 responses across many pages reduce crawl rate and, over a long enough period, can cause indexed pages to be dropped.
- Should I use 503 instead for rate limiting?
- No. 503 says the service is unavailable to everyone; 429 says this client exceeded a quota. Clients and crawlers act on the difference.
- Is Retry-After required on a 429?
- The specification describes it as an option rather than a requirement, and omitting it leaves every client guessing. It is the single most useful thing to add.
Sources
Related
VeriFixScan crawls a site and applies its checks to every page it reaches, keeping the evidence behind each finding. Scanning one website is free.
Scan a website