Model Fallback

LLM endpoints fail. Rate limits, model outages, context-overflow errors, and content filters all surface as request errors that would otherwise stall an agent loop. Model fallback lets you pass a ranked list of models in a single request — BitRouter walks the list until one succeeds, then returns that response.

This is a body-level extension to the OpenAI, Anthropic, and Google protocol surfaces. No SDK required — set one field.

Quick example

curl http://127.0.0.1:4356/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "models": [
      "openai/gpt-4o",
      "anthropic/claude-sonnet-4-6",
      "google/gemini-2.5-pro"
    ],
    "messages": [{"role": "user", "content": "Summarize the Iliad in one sentence."}]
  }'

The model field stays the primary for billing and routing semantics. The models array overrides it as an ordered preference list — first one that returns successfully wins.

What triggers a fallback

BitRouter falls through to the next model on errors that are upstream-side and likely transient, and surfaces 4xx errors caused by your request directly to the caller.

Fallback is single-pass. BitRouter attempts each model at most once per request, in order. There is no exponential backoff between attempts — the assumption is that you'd rather retry on a different model immediately than wait on a failing one.

Inspecting which model answered

For every settled request BitRouter emits a request finished log line carrying the upstream that actually answered — provider, model, and, for a multi-account provider, the account that served it (which reflects any failover hop) — alongside token counts and latency. Read it from the gateway's log output (~/.bitrouter/bitrouter.log for a local install) to confirm which provider and model handled a request, including after a fallback fell through to a later model.

The response body's model field echoes the model you requested first, so it does not by itself reveal which fallback served the response — use the log line for that.

Cost and latency tradeoffs

Each fallback attempt is a fresh upstream request. Practical advice:

• Lowest expected cost: order by cheapest first, accept higher tail latency under load.
• Lowest expected latency: order by most reliable first, accept higher per-token cost.
• For long-running agent loops: bias toward reliability. The cost of a stalled loop is much higher than the marginal cost difference between two frontier models.

For declarative cost or latency optimization across providers of a single model, see Provider Selection. Fallback and provider selection compose: BitRouter picks the best provider for each model in your models array, falling through to the next model only after the chosen provider for the current one has exhausted its retry budget.

Anthropic and Google surfaces

The models field works identically on /v1/messages (Anthropic Messages) and /v1beta/models/{model}:generateContent (Google Generative AI). On Anthropic, the field is read alongside the existing model field; on Google, BitRouter accepts it as an extension to the request body — the upstream :generateContent path is rewritten per attempt.

Limits

• Maximum 8 entries in models.
• Each model ID must resolve to a registered model in the registry. Unknown IDs return 400 before any upstream attempt.
• Streaming is supported. The first model that begins emitting tokens wins; later models are not attempted even if the stream fails after the first token.