Architecting a Zero-Latency Anycast Network for LLM Token Routing

Token streaming sounds simple until you trace a token's actual path from model output to client render: model server, inference gateway, edge proxy, client SDK. In our original implementation, three of those four hops buffered output, whether for batching efficiency, TCP_NODELAY misconfiguration, or simple oversight. The result was that tokens generated in 10-15ms intervals arrived at the client in bursts of 200-400ms, defeating the entire purpose of streaming.

Finding the Buffering: TCP_NODELAY and Nagle's Algorithm

The first and most impactful fix was embarrassingly simple. Our edge proxy, a custom Go service, was not setting TCP_NODELAY on outbound client connections, meaning Nagle's algorithm was coalescing small token writes into larger packets before transmission. For a streaming workload where individual tokens are often under 20 bytes, this is exactly the wrong default.

conn, err := net.Dial("tcp", clientAddr)
if err != nil {
    return err
}
if tcpConn, ok := conn.(*net.TCPConn); ok {
    tcpConn.SetNoDelay(true)
}

conn, err := net.Dial("tcp", clientAddr)
if err != nil {
    return err
}
if tcpConn, ok := conn.(*net.TCPConn); ok {
    tcpConn.SetNoDelay(true)
}

conn, err := net.Dial("tcp", clientAddr)
if err != nil {
    return err
}
if tcpConn, ok := conn.(*net.TCPConn); ok {
    tcpConn.SetNoDelay(true)
}

Removing Gateway-Level Batching

The inference gateway itself was batching tokens in groups of 4 before forwarding them downstream, a holdover from an earlier optimization aimed at reducing write syscalls under high concurrency. We replaced fixed-size batching with a hybrid approach: forward immediately if more than 8ms have elapsed since the last token, otherwise batch up to 3 tokens. This preserved most of the syscall-reduction benefit while keeping worst-case added latency under 8ms per token.

The TCP Handshake Tax on Cold Connections

For clients establishing a new connection per request rather than reusing a persistent connection, the TCP three-way handshake plus TLS negotiation added 60-150ms before the first byte of any response, streamed or not. We addressed this at the SDK level by defaulting to HTTP/2 connection reuse with a 90-second idle timeout, which eliminated handshake overhead for any client issuing more than one request per session, the overwhelming majority of production traffic.

Measured Outcome

Post-rebuild, median time between token generation and client delivery dropped to 38ms, with P99 under 50ms. The fix that mattered most by a wide margin was the TCP_NODELAY change; it alone accounted for roughly 70% of the total latency reduction, a reminder that infrastructure defaults are worth auditing before reaching for architectural rewrites.