Fallback Chains: What the Agent Does When the Tool Is Down

The search vendor dies at 14:03, and every layer you built from this series does its job. The timeout caps each attempt, the circuit breaker opens after the fourth failure, the bulkhead keeps the dead dependency from touching the database tools. By 14:04 the agent's search_docs calls return in two milliseconds with a structured circuit_open result. Containment worked.

Then the model improvises. It rephrases the query and tries again (the breaker result told it not to, but the turn's plan says find the runbook). It tries read_ticket on a guessed ID because search would have found the real one. Eventually it answers the user's question anyway, citing "the search results" — which never ran. The ticket that comes out of this contains a confident paragraph of fiction, and the postmortem will note, uncomfortably, that every reliability control reported green.

This is the gap the containment chapters leave open. Timeouts decide when a call ends. Breakers decide when a dependency stops being called. Bulkheads decide how far the damage spreads. None of them decide what the agent does with the turn it is still holding — and that decision, left unmade by you, gets made by the model at runtime, under time pressure, with a bias toward appearing helpful.

An error is a prompt; a fallback is a decision

Hand a model a raw failure and you have delegated a product decision to the least deterministic component in your system. Models do not treat errors as dead ends; they treat them as feedback. Watch a few transcripts and the improvisation patterns repeat: rephrase the arguments, guess a sibling identifier, substitute a different tool with different semantics, or quietly drop the failed step from the plan and present the rest as complete. Every one of those is a fallback chain — just one the model authored on the fly, untested, invisible to telemetry, and inconsistent across runs.

The fix is to move that decision to design time. For every tool in your manifest, answer one question while calm: when this tool is unavailable, what is the best honest answer the agent can still give? The answer is an ordered list of fallbacks with decreasing fidelity — a ladder — and the harness, not the model, walks it. The model never sees the primary's failure; it sees the best rung that answered.

This is graceful degradation, the same discipline load balancers and CDNs have practiced for decades, applied at the tool-call boundary. The novel part for agents is that the consumer of a degraded answer is a model that will reason over whatever provenance you attach — so provenance becomes load-bearing.

The ladder

A rung is a callable with the same result shape as the primary, plus a claim about fidelity. Order rungs by fidelity, not by speed:

FALLBACK_LADDERS = {
    "search_docs": [
        Rung(call=search_replica,        fidelity="fresh",    budget_ms=400),
        Rung(call=cached_search,         fidelity="stale",    budget_ms=50,
             max_age=Duration(hours=6)),
        Rung(call=grep_local_index,      fidelity="partial",  budget_ms=800),
        # implicit terminal rung: a structured refusal
    ],
    "create_refund": [
        Rung(call=refund_primary,        fidelity="exact",    budget_ms=2000),
        Rung(call=queue_for_replay,      fidelity="deferred", budget_ms=100),
    ],
}

async def call_with_ladder(tool, args):
    for rung in ladder_for(tool):
        try:
            result = await with_timeout(rung.call(args), rung.budget_ms)
            return degraded_if(rung, result)      # annotate unless fidelity == "exact"
        except (Timeout, Rejected, BreakerOpen):
            continue
    return structured_refusal(tool, args)         # the ladder ran out

Three details do the real work. Each rung has its own timeout — a fallback inheriting the primary's 29-second timeout re-creates the stall you were escaping. Each rung has its own breaker — when the cache is the thing that died, rung two must fail in milliseconds and the ladder must fall through to rung three, not take the whole chain down with it. And the ladder ends in a structured refusal, not an exception: a final rung that says search unavailable, answer without it or ask the user is a legitimate, testable outcome.

Size every rung's budget against its job, and keep fallbacks dramatically cheaper than the primary. A fallback that costs as much as the primary is not a fallback — it is a second outage wearing a hat.

Provenance the model can honor

The rung's result must carry what happened, in language the model will obey. A degraded result is a first-class structured result, same discipline as circuit_open and busy from the earlier chapters:

{
  "status": "ok",
  "tool": "search_docs",
  "degraded": true,
  "fidelity": "stale",
  "source": "search_cache",
  "age_minutes": 214,
  "detail": "live search is unavailable; these results are from a cache captured 3.6 hours ago. They may not reflect changes since then. Do not claim results are current. The local grep tool is available for exact-string lookups."
}

The detail wording is not decoration — it is the steering surface. "Do not claim results are current" changes model behavior measurably where an error code changes nothing. Say what happened, what the model may assert, and what it may still do. A degraded answer the model presents as fresh is worse than no answer: it converts a dependency outage into a trust outage.

Reads degrade, writes queue

The ladder's shape depends on what the tool does. Reads degrade: an older or narrower answer is often within contract, so the ladder is replica → cache → substitute → refusal, gated by a staleness budget per tool class. Search six hours stale is fine for a runbook lookup and unacceptable for a stock price — encode that budget on the rung (max_age), and when the cache exceeds it, the rung declines and the ladder falls through. Freshness-dependent correctness degrades to refusal, never to stale-on-the-sly.

Writes never substitute. A refund executed by a "fallback refund tool" is a different product decision smuggling itself in during an outage. Writes degrade to deferral: a durable queue — a table, an outbox, anything that survives restarts — with the idempotency key minted up front. The rung's result is honest about the state change:

{
  "status": "queued",
  "tool": "create_refund",
  "idempotency_key": "7f9d2c1e-...",
  "detail": "refund processor is down; the request is durably queued and will replay with the same idempotency key. It has NOT executed yet. Tell the user it is pending, not completed."
}

When the processor recovers, replay carries the original key, so a partial execution before the outage cannot double-refund — the idempotency chapter is what makes deferral safe. The agent's ledger entry says queued, and if the model tells the user "your refund is done" against this result, that is a wording failure you can catch in review because the result text pinned the truth.

Don't encode the ladder in the prompt

The tempting shortcut is a system-prompt sentence: "if search fails, use the local index." Resist it. Prompt-encoded fallbacks fail four ways at once: the model applies them to the wrong failure class (falling back on a busy result that asked for a retry); it skips them nondeterministically; you cannot unit-test a sentence; and telemetry never sees which rung served the answer. The ladder belongs in the dispatcher, where it is ordered, budgeted, monitored, and testable — and the prompt only ever sees the rung's result.

It composes with the rest of the series

Fallbacks are the answer layer on top of the containment stack, and the interactions matter:

• With circuit breakers: the open breaker is what arms the ladder — the primary's rung fails fast (BreakerOpen), and the ladder walks down without burning the timeout. Per-rung breakers keep a dead cache from cascading.
• With bulkheads: busy is not down. A partition-full result carries retry_after_ms; falling back on it serves stale data while the primary was a retry away. Only breaker-open, timeout-exhausted, and hard-failure rungs advance the ladder.
• With idempotency keys: they are what make the write queue and the read-retry safe. Fallback design assumes the key contract from chapter one.
• With resource budgets: budget exhaustion degrades along the same shape — a cost-aware rung (cheaper model, cheaper search tier) is just another step down the fidelity ladder, and both share the provenance discipline.
• With observability: every rung fires a counter. fallback_served{tool, rung} is the metric that tells you how much of your product is running on degraded mode right now — and how long the primary has been down while users kept getting answers.

A ladder you can test

Fault-inject the primary and assert the ladder's contract, in CI next to the breaker and bulkhead batteries:

async def test_dead_primary_serves_degraded_with_provenance(fake_backends):
    fake_backends.search.kill()                        # breaker opens fast

    result = await harness.call("search_docs", {"q": "runbook restore"})

    assert result["status"] == "ok"
    assert result["degraded"] is True
    assert result["fidelity"] == "stale"
    assert result["age_minutes"] <= 360                 # the staleness budget held
    assert "do not claim" in result["detail"].lower()   # steering wording intact
    assert result["elapsed_ms"] < 500                   # the ladder answered fast

Then the negative tests, because a fallback suite that cannot fail vouches for nothing: strip the degraded annotation and the provenance assertions must go red; exceed max_age and the rung must decline rather than serve; kill every rung and the terminal result must be a structured refusal — never an unhandled exception, never an empty success. For write ladders, kill the processor, assert the queued result carries the idempotency key, restart it, and assert the replay executes exactly once.

What this does not fix

A ladder serves answers that already exist somewhere. It cannot conjure data nobody captured, and a stale cache is still stale — if the task's correctness depends on freshness, the honest terminal rung is refusal, and refusing well is part of the design. Fallbacks also mask outages: users keep getting answers while the primary burns, so a rung-usage rate above noise must page a human, or your dependency can be down for a week and your dashboards will show only a subtle fidelity shift. Finally, rungs are a maintenance surface — a canned answer or a local index that never gets exercised rots silently, so schedule the ladder test above into CI, not into memory.

The checklist

• Every tool's manifest entry answers, at design time, "what is the honest answer when this is down?" — an ordered ladder, terminal rung = structured refusal.
• Ladders live in the dispatcher, never in the prompt; the model only ever sees rung results.
• Per-rung timeout and per-rung breaker; fallbacks dramatically cheaper than primaries.
• Degraded results carry degraded, fidelity, source, age and steering wording that says what the model may claim.
• Reads gate on staleness budgets per tool class; writes queue durably with the original idempotency key and surface queued, never done.
• busy does not advance the ladder; only breaker-open, timeout-exhausted, and hard failure do.
• Rung counters in metrics, alerting on sustained fallback usage — silent degradation is an outage with better manners.
• CI kills the primary, asserts provenance and latency, and the negative tests strip annotations and require red.

Containment bought you a harness that survives a dependency dying. Fallback chains buy the thing that survival was for: an agent that keeps giving honest, correctly-caveated answers while the pager is still ringing. Decide the ladder now, while everything is up — the model's improvisation under fire is the one fallback you never chose.