Failover & high availability
Automatic multi-vendor failover
Failover is ModelSite's core advantage over direct single-vendor connections — the same model can be backed by multiple upstream vendors, and the routing layer switches automatically when one fails, so a single vendor outage never interrupts your service.
How it works
- A model ID maps on the backend to one or more upstream vendors (e.g. Claude can come from Anthropic official + several proxy channels).
- On each request the routing layer picks an upstream by strategy (load, latency, health).
- If an upstream returns
5xx/ connection error / timeout, the router marks it for cooldown and immediately retries on the next available upstream. - The whole process is transparent to the caller — you only notice the occasional slightly higher time-to-first-token, never a failure.
Impact on callers
| Scenario | What you see |
|---|---|
| Occasional upstream 5xx | transparent retry, call succeeds (slightly higher latency) |
| Persistent upstream failure | auto-cooldown, switch to a healthy upstream |
| All upstreams fail | only then 5xx (rare) |
Billing consistency
No matter which upstream is hit, you're billed at the model's unified list price — the price you see doesn't change with backend switching. The actual upstream, latency, and token usage are recorded in the usage log.
Pairing with client-side retries
Platform failover already covers most upstream errors. Clients need only light backoff for 429 (rate limit) and the occasional 503 — no need to build complex retry logic for upstream failures.
For extra safety on critical workloads: keep reasonable concurrency to avoid maxing out a single key's RPM and triggering 429 (429 doesn't go through failover).