Backpressure and Rate Limits: Making Agent Tool Calls Safe to Fan Out

A support-analytics agent turn comes back with eight tool calls: four search_tickets, three get_usage, one list_customers. The harness fans them all out in parallel — that is the point of parallel execution — against a third-party API whose key is rated ten requests per second. The burst lands as eight requests inside forty milliseconds. Three come back 429. The retry policy fires with the backoff tuned for gateway blips, two hundred milliseconds, all three retries landing at once. Meanwhile the model, told "tool failed" on two of the calls, does the reasonable thing: it rephrases into narrower searches and issues six more. Within four seconds the key has exhausted its rolling window — and the key is shared. The production sync job starts failing too. Nothing crashed. Every component followed its design.

This is not a model bug. Parallel calls are correct behavior, retries are correct behavior, and rephrasing a failed query is exactly what you want a model to do — when the failure is a failure. The bug is the contract: the harness treats a rate limit as an obstacle to retry around, when it is information to comply with. Retries got idempotency keys; ordering got write serialization; deadlines got timeouts. This article is the load-domain chapter: how many tool calls may be in flight, at what rate, and what the model deserves to be told when the answer is "not now."

Agents are the worst kind of client

Rate limiting is usually written about from the server's side. But agents invert the assumptions a server makes about clients:

• Bursty by construction. A model turn emits its calls as a clump, after a thinking pause. The traffic shape is a sawtooth — long silence, then N requests at once — which is precisely the shape token buckets punish.
• Correlated retries. When three parallel calls fail together, three retries fire together. Without jitter, every retry wave lands as a synchronized burst, and the failure that triggered them recurs on schedule.
• Adaptive amplification. A human who hits a rate limit slows down. A model handed a generic "tool failed" does the opposite: it splits the query, tries a different tool, issues follow-up lookups. Presenting a 429 as an ordinary error converts a throttled client into a heavier one.
• Shared credentials. The agent rarely has its own API key. It shares the org's quota with production traffic, so its bursts have victims beyond itself.

None of these are exotic. Any harness with parallel execution and a retry policy has all four on day one.

Two limits, often confused

"Rate limiting" names two different constraints, and a correct client enforces both:

• Concurrency — how many requests may be in flight at once. A semaphore. It protects your connection pool and the server's per-connection work.
• Rate — how many requests may start per second. A token bucket. It protects quotas and rolling windows.

They interact through latency. A decent first cut of the concurrency cap comes from Little's law: concurrency ≈ rate × expected_latency. Ten requests per second against a 400 ms API gives a cap of four in flight. Note what each cap alone fails to catch: with only a concurrency cap, a latency spike to four seconds quietly drops your achieved rate — safe. But with only a rate cap, a slow API accumulates in-flight requests until sockets exhaust — unsafe. And with neither, the eight-call burst at the top of this article is not even a spike, it is a normal turn.

The client-side token bucket

The bucket lives in the harness, scoped per (tool, credential) — per credential, because that is the unit the server throttles. One shared bucket for all tools on one key, a separate bucket for a different key, regardless of which tool uses it:

import time

class TokenBucket:
    def __init__(self, rate_per_s: float, burst: int):
        self.rate = rate_per_s
        self.capacity = float(burst)
        self.tokens = float(burst)
        self.updated = time.monotonic()

    def take(self, now: float) -> float:
        """Reserve one call; return seconds to wait (0 = go now)."""
        self.tokens = min(self.capacity,
                          self.tokens + (now - self.updated) * self.rate)
        self.updated = now
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return 0.0
        return (1.0 - self.tokens) / self.rate

    def on_429(self, wait_s: float, now: float) -> None:
        """Drain tokens when the server says we went too fast."""
        self.updated = now
        self.tokens = min(self.tokens, 0.0) - wait_s * self.rate

The discipline that makes this work: the executor calls take() before dispatching every request, and the wait it returns comes out of the call's deadline budget — if the wait is longer than the call can afford, do not sleep; return rate_limited immediately. A bucket that blocks past its caller's deadline converts throttling into the same hung calls the timeouts chapter exists to eliminate.

Honor the server's numbers

Servers tell you their limits, and most clients ignore them. Parse the standard headers:

• Retry-After — seconds or an HTTP-date. Highest precedence: it is the server's explicit instruction.
• RateLimit-Remaining / RateLimit-Reset (the IETF draft) and the older X-RateLimit-Remaining / X-RateLimit-Reset — when remaining hits zero, wait until reset before the next call, not until the next 429.
• Scope headers some vendors add (X-RateLimit-Organization, per-endpoint buckets) — map each to its own bucket so one exhausted scope does not stall calls governed by another.

Two parsing rules save real incidents. Prefer relative seconds over absolute dates — HTTP-dates depend on clock sync, and a skewed clock turns "wait 2 s" into "wait 6 hours" or "wait in the past." And clamp: a Retry-After: 3600 against a call with a 30-second deadline is the server saying stop, not an instruction to sleep. Surface it to the caller; never let a header silently extend a bounded call.

A 429 is a response, not an error

What the model sees when a call is throttled decides the next thirty seconds of system behavior. A generic error invites rephrasing, splitting, and alternative tools — more load. A structured result lets the model plan:

{
  "status": "rate_limited",
  "scope": "api_key",
  "retry_after_ms": 4200,
  "hint": "Concurrent calls on this key are deferred; batch or wait."
}

With that, "wait, then re-issue the same call" becomes the obviously correct plan, and models take it — deferral is not failure, and the arguments were fine. This is the response-design principle applied to load: errors are responses too, and the shape of this one carries the decision the model needs to make.

Draw the line at what stays invisible. Short waits — up to a couple of seconds — should be retried inside the harness, below the transcript, so micro-stalls never interrupt the model's flow at all. Beyond that, surface. The threshold is a judgment call; having an explicit one is not.

Backoff with jitter, at the harness layer

When the harness does retry, it owns two responsibilities the model cannot see. First, jitter: when N calls fail together, their retries must decorrelate. Full jitter is the standard answer:

import random

def backoff(attempt: int, base_s: float = 0.5, cap_s: float = 20.0) -> float:
    return random.uniform(0, min(cap_s, base_s * 2 ** attempt))

Sleeping a uniform random value in [0, min(cap, base·2ⁿ)] spreads each retry wave across the whole interval instead of its edge, and it degrades gracefully when the server's recovery time is unknown. Equal jitter (random.uniform(b/2, b)) is a fine alternative for latency-sensitive paths; the property that matters is that no two retries from the same wave fire at the same instant.

Second, the deadline: every wait is charged against the call's remaining deadline budget. A backoff longer than what the call can afford aborts the call with timed_out, it does not sleep past the boundary and hope. Retries without deadline accounting are how a throttled API turns into a pile of abandoned-but-running requests.

Shed load instead of queueing it

Between the model's fan-out and the executor there must be a bounded queue, and when it is full the harness sheds — it returns rate_limited immediately with the queue's estimated drain time, rather than accepting the work. The temptation is the opposite: accept everything, queue the excess, let calls wait their turn. Unbounded queues feel polite and kill turns: every queued call accumulates latency, the deadline clock is running the whole time, and the steady state is twelve calls that all time out instead of four that succeed and eight the model knows are deferred.

Shedding is not a degradation of the fan-out; it is what keeps it honest. The model issued twelve calls, the system can run four now — telling it exactly that beats pretending all twelve are in flight.

Coalesce what you can

The cheapest request is the one never sent. Two mechanisms collapse fan-outs without changing model behavior:

• Single-flight. When two parallel calls in the same turn request the same (tool, arguments), execute one HTTP request and share the result. Reads only — writes to the same resource go through serialization, a different mechanism for a different hazard. In practice models dedupe surprisingly often, especially in re-planning loops.
• A per-turn micro-cache. Identical read calls within one turn — sometimes across steps, as the model re-grounds itself — hit a TTL cache measured in seconds, scoped to the turn. Not a caching layer, just an acknowledgment that a 3-second-old read is current enough for the conversation that is still happening.

Batch endpoints, where the API offers them, convert N calls into one — but only if the model is told the batch tool exists and when to prefer it. That is schema design, not backpressure; the harness cannot invent it.

What this does not fix

• Cross-process budgets. Two agent processes each running a local bucket still sum to twice the key's rate. A local bucket prevents self-inflicted bursts; org-wide contention needs a shared limiter — a Redis-backed bucket, or sub-allocations granted per process.
• Server-side fairness. How a server should apportion a shared quota among clients is the server's problem, with its own literature. This article is about not being the client that makes it urgent.
• Genuinely empty tanks. A daily quota that is exhausted does not respond to backoff. Surface it to the operator, stop the run or degrade it, and do not let the retry policy spend the night asking.
• The model's choice of plan. Told "not now," a model can still choose to hammer a different tool. The structured result makes the situation legible; it does not make the model obedient. Approval gates and budgets are the controls for that.

Test it with a throttling proxy

Backpressure code fails silently: the bucket that never gets consulted, the header parse that never matches, the jitter that is always zero. The test that catches all of them is a fake downstream that throttles like a real one — first k requests return 429 with Retry-After, then it lets traffic through while recording timestamps:

def test_throttled_fanout_respects_window(fake_api):
    fake_api.set_rate_limit(allowed=3, retry_after_s=0.2)
    results = harness.execute(
        [call("search_tickets", q) for q in ["a", "b", "c", "d", "e"]]
    )
    # requests 4 and 5 must not start before the window elapses
    starts = fake_api.request_starts()
    assert starts[3] - starts[2] >= 0.2
    assert starts[4] - starts[2] >= 0.2
    # nothing exploded into retries: exactly one request per logical call
    assert len(starts) == 5
    # short inline waits stayed invisible to the model
    assert all(r.status != "rate_limited" for r in results)

Assert on timestamps and counts, not just outcomes — a client that fires everything and catches 429s can still pass a test that only checks final results. Add one test where Retry-After exceeds the call's deadline and assert the call returns rate_limited or timed_out without the request being sent, and one where two processes share a key to document, in CI, exactly what the local bucket does not prevent.

The checklist

• A client-side token bucket and a concurrency semaphore, both, scoped per (tool, credential).
• Parse Retry-After and rate-limit headers; prefer relative seconds; clamp absurd values instead of honoring them into deadlock.
• Throttled calls surface as structured rate_limited results with retry_after_ms — never as generic errors.
• Harness-level retries stay invisible for short waits, always jittered, always charged against the deadline.
• A bounded executor queue that sheds with an honest drain estimate rather than queueing a turn into the ground.
• Single-flight and a per-turn read cache collapse the fan-out before it becomes load.
• A throttling-proxy test asserts request timestamps and counts — the only witness that can't be fooled.

An agent that can be told "not now" and trusted to plan around it is a better client than most human-driven systems. Backpressure is how you buy that, for the price of one bucket, one semaphore, and one honest error shape.