Handling Gemini 429 Rate Limits with Circuit Breakers

What a 429 actually is

When you exceed any free-tier dimension, Google returns the same error over and over:

google.api_core.exceptions.ResourceExhausted: 429 You exceeded your current quota...
* Quota exceeded for metric: generativelanguage.googleapis.com/generate_content_free_tier_requests, limit: 20, model: gemini-3-flash

The important detail: a daily-limit 429 will not clear by waiting a few seconds. Retrying it a fixed number of times just burns wall-clock time on a request you know will fail.

Step 1 — Exponential backoff with jitter for burst limits

For transient RPM/TPM bursts, exponential backoff with jitter is the standard fix. Back off with min(60, 2^n) seconds plus a small random jitter so concurrent requests don’t synchronize into waves, honor Retry-After when the API returns it, and always cap the number of attempts. This protects you from retry storms against a limit that resets soon.

Step 2 — Circuit breaker with per-model cooldowns for daily limits

Backoff is not enough when a model’s daily quota is exhausted. The classic fallback chain (try model A, then B, then C) has no memory: every request still calls A, waits for its 429 across the network, and only then moves to B. That accumulated latency is what pushed this project to 8.29 s under load.

The fix is a circuit breaker adapted to model calls — when a model says it is exhausted, put it in cooldown and skip it until the penalty expires:

class GeminiLLMService(BaseLLMService):
    def __init__(self, ...):
        self.model_cooldowns = {}
 
    def _is_on_cooldown(self, model_name: str) -> bool:
        cooldown_until = self.model_cooldowns.get(model_name, 0.0)
        return time.time() < cooldown_until

Before every API call, drop every model that is on cooldown so requests jump straight to a healthy fallback:

def _get_active_models(self) -> list:
    models_to_try = [self.model_name] + self.fallback_models
    active_models = [m for m in models_to_try if not self._is_on_cooldown(m)]
 
    # Prevent deadlock: if everything is on cooldown, try the full list anyway.
    if not active_models:
        logger.warning("All models are on cooldown. Trying all of them.")
        return models_to_try
 
    return active_models

And when a ResourceExhausted (429) comes back, penalize that model for 60 seconds:

def _apply_cooldown(self, model_name: str, exception: Exception):
    ex_name = type(exception).__name__
    ex_msg = str(exception)
 
    if "ResourceExhausted" in ex_name or "429" in ex_msg or "quota" in ex_msg.lower():
        self.model_cooldowns[model_name] = time.time() + 60.0

The real telemetry

Running the same benchmark (25 complex questions back to back) before and after the change:

• Before: 8.29 s per query — every request waited on an exhausted model’s 429 before falling back.
• After: 0.93 s per query — the client skips cooldown models instantly.
• Reduction: ~88.7 % average response time.

The system stopped making useless network trips to exhausted endpoints, and the frontend noticed nothing when the primary model ran out of quota.

Takeaways

1. Treat quota as part of the system design, not an edge case. On free tiers it is the normal operating mode.
2. Backoff handles bursts; circuit breakers handle persistent exhaustion. Use backoff for transient 429s and a circuit breaker with cooldowns for daily limits.
3. The best performance optimization is often not making the call. Skipping a request to a quota-exhausted model is worth more than making it faster.