Routing rules

How the lane picks an upstream for each call, and how to steer it.

Most model families are available from more than one upstream. The lane picks one per request so a call goes through even when a single upstream is slow or saturated. You can leave that choice to the lane or constrain it.

Default behaviour

• Upstreams are ranked by recent latency and error rate for that exact model.
• The price you pay is the posted rate, whichever upstream serves the call.
• If the first upstream fails before producing output, the next one is tried. You are charged once, for the call that answered.

Constraining a route

Pass an inferlane object alongside the usual body. Unknown keys are ignored, so the request stays valid for any OpenAI client.

{
  "model": "meta-llama/llama-3.3-70b-instruct",
  "messages": [{ "role": "user", "content": "Summarise this ticket." }],
  "inferlane": {
    "fallback_models": ["qwen/qwen3.8-flash"],
    "max_latency_ms": 4000,
    "allow_upstream_fallback": true
  }
}

• fallback_models: tried in order if the requested model is unavailable. The charge follows the model that answered.
• max_latency_ms: time to first token after which the lane moves to the next upstream.
• allow_upstream_fallback: set false to pin the first upstream and fail rather than switch.

Which route answered

The X-Inferlane-Route response header names the model family and upstream that served the call, and the model field in the body is the model that actually answered.