How rate limiting works
A website, API gateway, CDN or web application firewall counts requests against a quota. The client may be identified by its IP address, API key, account, cookie or a combination of signals. Limits can also apply to a specific endpoint, such as /api/search or /login.
When the quota is exhausted, the service usually returns 429 Too Many Requests. It may include a Retry-After header showing when to try again:
HTTP/1.1 429 Too Many Requests
Retry-After: 60Common counting methods include:
- Fixed window: resets the counter at set intervals, such as every minute.
- Sliding window: counts requests during the immediately preceding period.
- Token bucket: adds tokens over time, allowing short bursts while limiting the average rate.
Why proxies still hit limits
A proxy changes the source IP that a website sees. It doesn’t change an API key, account, cookie, browser fingerprint or request pattern. Rotation may distribute independent requests across IPs, but it won’t reset limits tied to another identity.
Shared IP addresses can have the opposite effect. Activity from several users may count toward the same per-IP quota, causing you to receive a 429 sooner than expected. Repeatedly switching addresses to ignore an explicit limit can also trigger stricter bot controls.
How to handle rate limits
- Check the published rules. Read the API documentation, terms and robots.txt before automating requests.
- Honor
Retry-After. Pause for the stated number of seconds or until the stated HTTP date. - Reduce request pressure. Lower concurrency, cache repeated responses and queue work instead of sending bursts.
- Back off between retries. Increase the delay after each 429 and add small random variation so workers don’t retry together.
- Request more capacity. Use an official API plan or ask the site owner for a higher quota when your workload is authorized.
Rate limiting vs IP banning
| Rate limiting | IP banning | |
|---|---|---|
| Purpose | Control request volume | Refuse an address |
| Duration | Usually temporary | May be long-term |
| Typical response | 429 with a retry period | Often 403 or a dropped request |
An IP ban normally persists after a short pause. A rate limit usually clears when its counter resets.
FAQ
Is rate limiting the same as throttling?
Not always. Rate limiting usually caps or rejects requests, while throttling may slow them down or place them in a queue. Services often use the terms interchangeably.
Can proxies prevent rate limiting?
No. Proxies only change the visible IP. Limits based on accounts, API keys, cookies or behavior still apply, and you should not rotate IPs to evade a site’s stated rules.
How long does rate limiting last?
It depends on the service and quota window. Follow Retry-After when present; otherwise consult the API documentation and retry with increasing delays.