LLM Fallbacks

Fallback Chain Configuration

LLM fallback chains are defined as ordered arrays of model identifiers submitted in the model_chain parameter of the inference request body. Synapse's routing layer evaluates each model in sequence, selecting the first healthy provider that satisfies the request's latency budget and context window requirements. Chains support both Synapse-native models and external providers via standardized adapter layers, allowing heterogeneous chains that span multiple commercial API providers without requiring per-provider SDK integrations in client code.

Health Probing & Fault Detection

Each provider in the routing table is continuously evaluated by Synapse's synthetic probe fleet, which issues low-cost 16-token canary requests every 10 seconds from each point of presence. A provider is marked DEGRADED after two consecutive probe failures exceeding the P95 latency threshold, and FAULTED after five. Once faulted, the provider is removed from the active routing pool and all in-flight requests to it are reissued against the next chain member within 800 milliseconds. Recovery probes run on a 30-second cadence; a provider is reinstated after three successive successful probes.

Semantic Consistency Guarantees

When a fallback occurs mid-chain, Synapse re-evaluates the original request against the substitute model using an identical, canonicalized prompt envelope to maximize output consistency. For applications requiring strict output schema compliance, the output_schema parameter accepts a JSON Schema definition that is enforced via structured output mode on all capable models in the chain. If a fallback model does not support structured output natively, Synapse's constrained decoding layer enforces schema adherence at the token level, ensuring that chain transitions do not produce format regressions visible to downstream consumers.