Concurrency Bulkheads: One Slow Tool Must Not Starve the Rest

The vector-search vendor deploys a bad build at 14:03. Nothing errors — queries that answered in 40ms now take 29 seconds, just under the harness timeout. The agent, mid-investigation, has already fanned out its usual turn: one search_docs, two read_ticket, a list_deployments, a fetch_runbook. The harness has one concurrency limit — max_concurrent_tools = 8 — and one shared pool of worker slots behind it.

The search_docs call takes a slot and holds it for 29 seconds. The parallel calls take theirs and finish in milliseconds. Then the model reads the results, emits the next fan-out — and now two of eight slots are pinned by search_docs retries. By the third turn, every fan-out contains at least one search, and the searches always consume their slots for the full timeout window. The five cheap tools — 5-millisecond database reads — now queue behind them. list_deployments that used to return before the model finished its next token now take twenty seconds, not because the database slowed down, but because it shares a waiting line with something that did.

Every component is healthy. The database passes its checks, the runbook fetcher is fine, the vendor is degraded, not down — precisely the shape that trips no circuit breaker for a while. What's missing is a wall between dependencies: the naval-architecture insight that a ship with one big hold sinks when the hold floods, and a ship with compartments sinks much slower. Concurrency bulkheads are that wall, and for agent harnesses they are one afternoon of work.

Why agents hit this harder than every other client

A human-driven client makes one request at a time; when the search box is slow, one human waits. An agent is a fan-out machine: a single turn can emit five to fifteen tool calls, dispatched in parallel the moment the model stops talking. That arrival pattern turns one slow dependency into a slot-eating machine governed by Little's Law:

residents = arrival_rate × residence_time

Residence time for a healthy call is its latency. Residence time for a degraded call is the timeout — the slot is occupied for the full 29 seconds regardless of whether the call eventually fails. So a dependency that degraded from 40ms to 29s holds roughly 725× more slot-seconds per call, and every one of those slot-seconds is charged against a pool that your fast tools share. Two slow searches in flight per turn, an 8-slot pool, and your p99 for everything is now your timeout.

Retries make it worse before they make it better: each retry re-enters the same queue, behind the sibling calls it is starving. And the model experiences the queue as tool latency, which it cannot distinguish from tool failure — the same confusion that makes agents rephrase arguments at dead tools. The model will happily conclude list_deployments is broken and route around it.

The bulkhead contract

Partition concurrency by dependency, not by session, not by tool name alone. Every external backend the harness talks to — or every tool with meaningfully different latency and failure behavior — gets its own bounded partition:

• A semaphore capping in-flight calls to that dependency.
• A bounded admission decision when the semaphore is full: fail fast, don't queue forever.
• A structured busy result the model can route around, with a retry_after_ms hint.
• Telemetry per partition: in-flight gauge, wait time, rejection count.

The defining property is failure isolation: the slow dependency can saturate its own partition completely, for minutes, and the fast tools never feel it. Cost is a little idle capacity — a fast tool can't borrow the search partition's unused slots — which is the entire trade, and it's a good one. You are trading a few percent of theoretical throughput for the guarantee that the 5ms tools keep answering in 5ms while the 29s tool burns.

One naming note: partition by backend or credential class, not by tool name, when several tools share a dependency. If search_docs, search_code, and summarize_page all hit the same vector-store deployment, they share a partition — otherwise you've built three doors into one room and the room still floods.

Sizing a partition

Little's Law does the arithmetic; your telemetry supplies the numbers. For each dependency:

partition_size ≈ peak_calls_per_second × p99_latency_seconds × headroom

A search serving 2 calls/second at a 1-second p99 needs ~2 slots; with 2× headroom for bursts and retries, 4. Cheap database reads at 20 calls/second and 10ms p99 need well under one slot of steady-state work — a partition of 2–4 exists purely to absorb fan-out bursts. Expensive one-shot operations (a report generation, a batch export) get partitions of 1–2 because that's all they can usefully overlap.

Three sizing disciplines matter more than the arithmetic:

• Size by p99, not the mean. A pool sized to average latency starves exactly when latency degrades — the moment a bulkhead matters.
• Keep a fast lane. If one tool family answers in single-digit milliseconds and another in seconds, they never belong in the same partition, even on the same backend. Read replicas and primary writes are two dependencies pretending to be one until you partition them.
• Re-size from telemetry, not incidents. A partition that rejects (busy) during normal load is mis-sized; record it and fix the number, don't tuning-theatre the timeout.

What a full partition does

The admission decision is where implementations go wrong. The wrong answer is an unbounded queue: it converts saturation into latency that spreads back through the agent's turn deadline, which is the failure you built the bulkhead to prevent. The right answer is a bounded wait — a short one, tens of milliseconds, riding on the expectation that fast partitions cycle quickly — and then an honest rejection:

class Busy(Exception):
    def __init__(self, retry_after_ms):
        self.retry_after_ms = retry_after_ms

async def call_partitioned(tool, args, partitions):
    part = partitions.of(tool)          # keyed by backend class
    try:
        # bounded wait: don't inherit the turn deadline as queue time
        slot = await asyncio.wait_for(part.acquire(), timeout=0.05)
    except asyncio.TimeoutError:
        raise Busy(retry_after_ms=part.estimate_free_ms())
    try:
        return await part.run(tool, args)   # the real call, with its own timeout
    finally:
        part.release(slot)

The harness catches Busy and returns it to the model as a structured result — same discipline as a circuit_open verdict:

{
  "status": "busy",
  "tool": "search_docs",
  "retry_after_ms": 4000,
  "detail": "search concurrency limit reached; other tools are unaffected"
}

Note what the busy result is not: it is not an error. The model should not rephrase arguments, split the query, or try a sibling tool — the correct behavior is to do the other work in the turn (which is unaffected, because the partitions held) and retry the search when the hint expires. Tell it that in the result's wording; models follow plain language better than they infer from error codes.

Shedding is safe because of the earlier chapters in this series: idempotency keys make the eventual retry a no-op if the shed call actually landed, and write serialization means the operations worth protecting were already ordered. A bulkhead assumes retries exist; it is a bad fit for systems where every rejection is a permanent loss.

The physical layer is the real bulkhead

The semaphore partition is bookkeeping; the resource underneath it is what actually floods. Audit three of them:

• HTTP connection pools. A shared httpx.Client or requests.Session with max_connections=32 is one big hold. Give each backend its own client with its own pool sized to its partition — otherwise your semaphores admit calls that then queue on a shared socket pool, and you've rebuilt the starvation one layer down.
• Database pools. Every tool that touches the primary database shares its pool. Partition by role (primary writes vs. replica reads), and make sure the pool size is at least the sum of the partitions that sit above it.
• Subprocess and worker slots. Local tools — linters, sandboxed shells, file converters — occupy real CPU and memory. A ThreadPoolExecutor(16) shared by every local tool floods exactly like a socket pool when one converter hangs.

The rule: no shared, unpartitioned execution resource may sit underneath two tools with different latency profiles. The semaphore decides who gets to start; the pool decides who actually runs. Both walls, or neither works.

It composes with the rest of the series

Bulkheads are the containment chapter, and they interact with everything around them:

• With circuit breakers: the breaker opens on the dependency inside its partition. An open breaker drains the partition's queue instantly — rejected calls never even contest the semaphore — and the two together mean a dead dependency costs you one probe per cooldown instead of a full partition of timeout-length residencies.
• With timeouts: the timeout is what bounds a slot's residence time, which bounds how long a partition can stay saturated by one bad wave. Timeout × partition size is the worst-case drain time of a flooded partition; make that product smaller than your agent's patience.
• With backpressure and rate limits: a 429-aware client already sheds its own load; the partition is the second wall for when the server is slow rather than rejecting. Size partitions to the server's stated limits when it publishes them — a partition larger than the vendor's concurrency quota just moves the queue to their front door.
• With idempotency keys: shedding + retrying is only sound when the retry can't double-execute. The key is what makes busy a cheap answer instead of a gamble.

Ordering matters at dispatch time: acquire the partition slot before minting the attempt's idempotency key state, and release it the moment the response bytes land — don't hold a slot through response parsing or model-visible logging.

A bulkhead you can test

The contract is one test, and it belongs in CI next to the fault-injecting proxies from the earlier chapters:

async def test_hanging_tool_does_not_starve_fast_tools(fake_backends):
    # search hangs just under its timeout; the DB stays at 5ms
    fake_backends.search.hang(seconds=29.0)

    for _ in range(20):                      # saturate the search partition
        asyncio.create_task(harness.call("search_docs", {"q": "x"}))
        await asyncio.sleep(0)               # let them all enter

    start = time.monotonic()
    result = await harness.call("list_deployments", {"limit": 10})
    elapsed_ms = (time.monotonic() - start) * 1000

    assert result["status"] == "ok"
    assert elapsed_ms < 250                  # was 5ms; 250 is the fence, not the goal
    assert fake_backends.search.in_flight <= SEARCH_PARTITION_SIZE

Three assertions, three regressions they catch: the fast tool's latency fence (starvation), the in-flight cap (partition leak — a release on an early-exit path that isn't taken), and the saturation precondition itself. Then the negative test: temporarily point list_deployments at the search partition — the fence must go red. A bulkhead test that can't fail with the wall removed is not a test.

Add two cheap ones: a test that a busy result carries a plausible retry_after_ms and no stack trace, and a test that the hung partition eventually frees its slots after the timeout fires (partitions must drain; a slot leak is a slow-motion flood).

What this does not fix

The slow dependency is still slow — a bulkhead contains the blast radius, it doesn't heal the wound. It does nothing for resources you can't partition (one shared CPU, one process-wide file lock). It can't enforce fairness across agent runs if each run mints its own partitions — for multi-tenant harnesses, a global per-dependency partition shared by all sessions is the honest shape, with per-session fairness handled at admission. And if your agent makes exactly one tool call at a time, there's nothing to isolate yet — the fan-out is what makes the wall worth building.

The checklist

• One partition per dependency class (backend/credential), not per session and not per tool name when tools share a backend.
• Semaphore + bounded wait + structured busy result with retry_after_ms; never an unbounded queue, never a generic error.
• Size by p99 × peak rate with headroom; keep a fast lane for single-digit-millisecond tools.
• No shared execution resource (HTTP pool, DB pool, executor) underneath tools with different latency profiles — partition the physical layer too.
• Timeouts bound slot residence; breaker drains the partition from inside; idempotency keys make shedding retryable.
• Per-partition telemetry: in-flight, wait, rejections — a busy rate during normal load is a sizing bug, not load.
• CI hangs one tool and fences the fast tools' latency; the negative test removes the wall and requires red.

An agent that queues its 5-millisecond reads behind its 29-second searches isn't slow because the work is slow — it's slow because the compartments were never built. Bulkheads are the cheapest wall in the series: a semaphore per dependency, an honest busy result, and one test that hangs a tool to prove the rest of the ship still floats.