Skip to the pipeline

Scenarios

Each preset sets the controls below and runs one request. Start here.

Request

Or build one by hand and inject a fault to see how the pipeline reacts.

Identical text plus identical options produces the same cache fingerprint.

The model name selects an ordered provider chain from the routing table.

The error taxonomy decides whether retrying or failing over could possibly help.

Options

Idle. Pick a scenario, or send the request above.

Run

Stages append as they resolve. Provider breakers and totals persist until you reset.

Pipeline

one row per stage

    No request has run yet.

    Circuit breakers

    per provider

    Breakers are created on first use, as in the Python registry.

    Totals

    since reset
    Requests
    0served by the gateway
    Cache hit rate
    0%0 hit / 0 miss
    Tokens
    00 prompt / 0 completion
    Spend
    $0.00simulated
    Saved by cache
    $0.00avoided upstream cost

    Ledger

    one row per completed request
    Completed gateway requests, newest last
    Time Model Provider Tokens Cost (USD) Cache Outcome

    The ledger fills as requests complete.

    Response body

    application/json 
    // The normalised ChatResponse - or an RFC 7807 problem document - appears here.

    Structured log

    newest last

    One JSON line per event, as the Python logger emits.

    What this shows

    Each stage in the pipeline stands for a concern the production gateway has to answer.

    Authenticate
    The API key is resolved to a principal and its scopes. Keys are stored hashed and never written to a log line.
    Rate limit
    A per-key token bucket rather than a fixed window, because a fixed window lets a caller fire two windows' worth of traffic across the boundary instant. The default allowance is 60 requests per minute with a burst of 20; refusal returns 429 with Retry-After.
    Budget check
    An estimated token count projects the cost of the call before it is made, and it is checked against the key's remaining budget. Over budget returns 402 rather than an unexpected invoice.
    Cache lookup
    A SHA-256 fingerprint over only the fields that change the output - model, messages, tools, temperature, response format. A hit is the cheapest token the gateway will ever serve: nothing goes upstream and nothing is charged.
    Route
    The requested model name selects an ordered provider chain from the routing table, so callers name a capability and the operator decides who serves it.
    Circuit breaker
    Retries are only cheap when failures are rare. Once a provider has failed failure_threshold times in a row it is skipped outright until a recovery window elapses, then a couple of probe requests decide whether it is back. That turns a dead upstream from a full retry budget per request into an O(1) skip.
    Retry with backoff
    Retryable failures - timeout, upstream 429, 503 - wait an exponentially growing window with full jitter. The jitter is the point: without it, every client that failed at the same moment wakes at the same moment and the retry storm is indistinguishable from the outage that caused it. A provider-supplied Retry-After wins, capped so a hostile header cannot stall the request.
    Failover
    Errors marked failoverable move to the next provider in the chain. A rejected request does not: the next provider would reject it too, so failing fast is the honest answer.
    Tool calling
    When the model asks for a tool, the gateway executes it, appends the result as a tool message and calls again - bounded by max_tool_iterations so a loop cannot bill forever.
    Schema validation and repair
    With a JSON schema requested, the reply is parsed and validated. A failure is fed back to the model as a bounded repair loop before the request gives up with 422, because one malformed brace should not cost the caller the whole call.
    Usage and cost accounting
    Cost is computed from provider-reported token usage against an operator-supplied price book in USD per million tokens. Estimated counts guide decisions such as context trimming; they are never what gets charged.
    Structured logging
    One JSON line per request carrying the request id, route, every attempt with its outcome, latency and cost - enough to answer "what did this request actually do" without a debugger.

    Figures on this page are produced by the local stub, not measured against a model. The simulated prices match the repository's price book for the sim-* models so the arithmetic is real even though the tokens are not.