Architecting a Zero-Latency Anycast Network for LLM Token Routing

Router v1 supported failover across providers, but configuration was static: customers defined a fixed priority order, and the router simply walked down the list when a provider failed health checks. That model worked for simple setups but broke down for customers running against multiple providers with genuinely different latency and reliability profiles depending on time of day, region, and model version. Router v2 replaces static priority lists with adaptive, continuously updated provider scoring.

Adaptive Provider Scoring

Each provider now receives a rolling score computed from a weighted combination of error rate, P95 latency, and token-stream stability over the trailing 5-minute window, recalculated every 10 seconds. Customers can configure the relative weighting of these factors; a customer running latency-sensitive autocomplete might weight P95 latency heavily, while a customer running batch summarization might weight error rate and ignore latency variance entirely.

{
  "routing_policy": "adaptive",
  "scoring_weights": {
    "error_rate": 0.5,
    "p95_latency": 0.3,
    "stream_stability": 0.2
  },
  "min_score_threshold": 0.85,
  "providers": ["provider_a", "provider_b", "provider_c"]
}

{
  "routing_policy": "adaptive",
  "scoring_weights": {
    "error_rate": 0.5,
    "p95_latency": 0.3,
    "stream_stability": 0.2
  },
  "min_score_threshold": 0.85,
  "providers": ["provider_a", "provider_b", "provider_c"]
}

{
  "routing_policy": "adaptive",
  "scoring_weights": {
    "error_rate": 0.5,
    "p95_latency": 0.3,
    "stream_stability": 0.2
  },
  "min_score_threshold": 0.85,
  "providers": ["provider_a", "provider_b", "provider_c"]
}

Automatic Model-Equivalence Mapping

Failing over between providers is only useful if the receiving provider can serve an equivalent model. Router v2 introduces a model-equivalence graph that maps customer-specified model identifiers to comparable models across providers, accounting for differences in context window, function-calling support, and output format. Customers can override automatic mappings explicitly, but the default graph removes the need to hand-maintain a provider-to-provider model mapping as new model versions ship.

Configurable Failover Policies

Beyond simple health-based failover, v2 supports policy-driven routing rules: route specific request types to specific providers regardless of score, cap the maximum number of consecutive failovers before surfacing an error to the caller rather than continuing to retry indefinitely, and define per-provider rate ceilings independent of the provider's own published rate limits, useful for customers managing cost exposure across multiple billing relationships.

What This Means for Existing Customers

Router v2 is backward compatible with v1 static priority-list configurations, which continue to function unchanged. Customers opting into adaptive scoring have seen an average 34% reduction in failover-triggered request failures during provider-side incidents, since the router now shifts traffic away from a degrading provider before it fails health checks outright, rather than reacting only after a hard failure.