Load balancing begins with eligibility.
Two endpoints with similar model names are not automatically interchangeable. Context window, tool schemas, system-message behavior, quantization, safety configuration, token accounting, and data retention can change the contract your application receives.
API Many keeps each provider route explicit. A route without credentials, contract approval, or an exact capability match stays out of the pool. Open health circuits are excluded; remaining routes participate in weighted selection using durable success, failure, and latency observations.
Customers receive the public model contract and request outcome, not supplier identities, route counts, topology, internal costs, or operational diagnostics. Model availability does not imply a particular infrastructure configuration.
How requests move
Filter
Ineligible, unconfigured, contract-blocked, or open-circuit routes never receive the request.
Rank
Weights combine with durable health and latency observations so traffic is distributed rather than pinned.
Stop safely
Only explicit retryable rejection moves to the next route. Timeouts and network ambiguity do not.