Skip to content
costhelm

Configuration files

These are the packaged defaults exactly as shipped in costhelm v0.1.0 — synced from the release, never retyped. Resolution order and override mechanics are covered in Configuration. The files are heavily annotated on purpose; the comments are part of the documentation.

costhelm/routing/routing.yaml
# costhelm — routing policy.
#
# v3 classified every prompt into TINY / LARGE / HUGE with a router LLM and
# then looked the tier up in a hardcoded TIER_TO_ORDER dict. Three gaps this
# file closes:
#
#   1. `role` was logged and then ignored. A "decision" call and a "perception"
#      call with the same token count got the same model. Roles now carry tier
#      floors and ceilings, so the role changes the model.
#   2. HUGE returned 503 with a note about an unimplemented summariser. It now
#      routes to whichever providers can actually hold the context.
#   3. Provider order per tier was a fixed list. It can now be sorted by cost,
#      by quality, or by a tradeoff dial between them, using the prices and
#      quality scores in pricing.yaml.
#
# Adding a role, a tier, or a provider preference is an edit here and nothing
# else. No Python knows the name of any role in this file.


version: 1


# ── Tiers ───────────────────────────────────────────────────────────────────
# `order` is the base failover ring. Rebuilt 2026-08 on the paid-only provider
# set (ollama, gemini, openrouter, openai) when the free tiers (github, groq,
# cerebras, nvidia) were retired; chat.py's TIER_TO_ORDER fallback mirrors
# TINY and LARGE verbatim and the two must move together. `min_ctx` filters
# candidates whose max_ctx cannot hold the estimate — that is what makes HUGE
# routable instead of a 503.
tiers:
  TINY:
    order: [openrouter, gemini, ollama]
    description: short, simple, factual
  LARGE:
    order: [gemini, openrouter, ollama]
    description: dense or long-context work
  HUGE:
    order: [gemini, ollama, openrouter, openai]
    min_ctx: 8000
    description: >
      Above the 8k classifier boundary. v3 refused these outright; v4 keeps
      only providers whose max_ctx actually covers the prompt (Gemini's 1M
      window is the reason this tier is servable at all, with OpenAI's 922k
      as the paid backstop) and refuses with a context-specific error if
      none qualify.


  # ── The cross-model capability ladder ──────────────────────────────────────
  # TINY/LARGE/HUGE above are the CLASSIFIER's ladder: they answer "how big is
  # this prompt", and their rings are v3's failover rings verbatim, so default
  # routing has not moved. What they were never able to answer is "how good does
  # this answer need to be", because a ring is a failover order and not a price
  # ladder — every rung of it could end up served by the same model.
  #
  # These three rungs are the price ladder. Each one names a DIFFERENT model, on
  # a different provider, and the rungs are ordered by the blended $/Mtok in
  # pricing.yaml. `strict: true` is what makes a rung a rung: without it
  # `order_for` appends every other configured provider as a last-resort tail,
  # and a CHEAP request could be answered by the frontier model. `objective:
  # order` pins the declared preference against the global cost/quality dial,
  # which would otherwise re-sort a one-model rung into a different model.
  #
  # The rungs were rebuilt 2026-08 on the paid-only provider set. The gemma4
  # number below was MEASURED against this gateway on 2026-07-30 (identical
  # 3-sentence prompt, max_tokens 512, temperature 0, `reasoning: "off"`).
  # gpt-oss-120b, the model that served STANDARD from Groq, did NOT survive
  # the host move: OpenRouter's gpt-oss endpoint answers `reasoning: "off"`
  # with HTTP 400 "Reasoning is mandatory for this endpoint" (measured
  # 2026-08-02), so the rung runs on llama-3.3 instead. See pricing.yaml for
  # the per-model rows.
  CHEAP:
    order: [ollama, openrouter]
    strict: true
    objective: order
    description: >
      Local weights first (gemma4:31b, $0.00/Mtok, measured 39.6 s for a
      379-token answer), then OpenRouter meta-llama/llama-3.3-70b-instruct
      ($0.13/$0.40 per Mtok) as the network failover. Nearly free in dollars,
      expensive in wall-clock on the local rung — which is the real tradeoff
      a cheap rung buys, and the reason latency belongs next to price.
  STANDARD:
    order: [openrouter, gemini]
    strict: true
    objective: order
    description: >
      OpenRouter meta-llama/llama-3.3-70b-instruct ($0.13/$0.40 per Mtok,
      verified live 2026-08-02) with Gemini gemini-3.1-flash-lite
      ($0.25/$1.50, measured 1.2 s) as the same-rung failover. Both are
      genuinely different models from the FRONTIER rung; openrouter also
      anchors CHEAP, so only a gemma4 answer makes the two rungs differ in
      practice.
  FRONTIER:
    order: [openai]
    strict: true
    objective: order
    description: >
      OpenAI gpt-5.6-terra ($2.00/$12.00 per Mtok), the dearest model this
      gateway reaches, alone on the rung now that the free frontier hosts
      (GitHub Models, Cerebras) are retired. Unlike its 8k-capped
      predecessors it takes 922k of input, but a prompt that size still
      belongs on HUGE — this ladder prices capability, the other one sizes
      context.


# The escalation ladder, cheapest first. Also defines tier ordering for the
# role floors and ceilings below, so there is one source of "which tier is
# bigger". Unchanged from v3's classifier vocabulary.
ladder: [TINY, LARGE, HUGE]


# ── Named secondary ladders ─────────────────────────────────────────────────
# `ladder` above stays the classifier's. A named ladder orders a different set
# of tiers, and rank / next_tier / prev_tier resolve within whichever ladder a
# tier belongs to — so escalation on the capability ladder walks CHEAP ->
# STANDARD -> FRONTIER and budget pressure walks it back down, while the size
# ladder keeps working exactly as it did. A tier may sit on only one ladder.
ladders:
  capability: [CHEAP, STANDARD, FRONTIER]


# ── Benched providers ───────────────────────────────────────────────────────
# Dropped from every tier, including the non-strict last-resort tail, and
# reported as a per-candidate rejection so the refusal names the reason. This is
# how a broken endpoint is taken out of rotation without a Python edit; delete
# the line to bring it back.
unavailable: {}
  # RESOLVED 2026-07-30. nvidia was benched here, and the entry is worth reading
  # even though it is now empty, because the diagnosis is the lesson.
  #
  # The symptom: every nvidia call sat for the client's full 180 s timeout and
  # failed with an EMPTY error string, three times out of three. That looked
  # like a dead provider. It was not. The deployment's NVIDIA_MODEL selected
  # deepseek-ai/deepseek-v4-pro, which accepts the connection and never answers;
  # nvidia/llama-3.1-nemotron-nano-8b-v1 behaved identically. The endpoint and
  # the key were fine the whole time — meta/llama-3.1-8b-instruct on the same
  # key answered in 1.37 s.
  #
  # So it was a bad model pin, not a dead provider, and the fix was one
  # NVIDIA_MODEL edit. Re-measured after that edit: 1 205 ms, 227 chars, and
  # nvidia is back in every ring it belongs to. Bench by name here when an
  # endpoint is genuinely broken; delete the line to bring it back.


# ── Cooldowns ───────────────────────────────────────────────────────────────
# Seconds a provider stays benched after each class of failure. Overrides the
# defaults in costhelm/routes/chat.py by name.
#
# `timeout` is the one that mattered: an httpx ReadTimeout stringifies to the
# EMPTY string, so the old matcher recognised nothing and handed back a ZERO
# cooldown — NVIDIA burned 180 s, was benched for no time at all, and was a
# candidate again on the very next request. Ten minutes is deliberately much
# longer than the 180 s it takes to notice.
backoff:
  timeout: 600
  queue: 15
  rate_limited: 30
  rpm: 60
  rpd: 3600
  upstream_5xx: 20
  auth: 600


# ── Roles ───────────────────────────────────────────────────────────────────
# The role a caller declares (`auto_route: <role>`) now shapes the tier:
#   default_tier  used when no classifier ran
#   min_tier      floor — clamp UP to at least this tier
#   max_tier      ceiling — clamp DOWN to at most this tier
#   escalate      may this role spend more to retry
#
# Role names are free-form. Add one here and it works; the gateway validates
# against this list and falls back to `default_role` for anything unknown.
default_role: perception


roles:
  perception:
    # Cheap, high-volume observation. Never worth a frontier model, so it is
    # capped: a wrong tier classification cannot escalate spend here.
    default_tier: TINY
    max_tier: LARGE
    escalate: true
  memory:
    # Retrieval and summarisation of stored context. Long inputs are normal,
    # so no ceiling, but no floor either.
    default_tier: TINY
    escalate: true
  decision:
    # The step whose mistakes are expensive. Floored at LARGE: a TINY
    # classification is overridden, which is the single clearest demonstration
    # that role changes routing.
    min_tier: LARGE
    default_tier: LARGE
    escalate: true
  # Example of adding a role with no code change: a cheap classifier role that
  # must never leave the smallest tier and must never escalate.
  triage:
    default_tier: TINY
    max_tier: TINY
    escalate: false


  # ── Roles on the capability ladder ────────────────────────────────────────
  # A role's `default_tier` is what selects its ladder, so these three route
  # across MODELS rather than across prompt sizes. The classifier still runs and
  # its verdict is still honoured — carried across by position, so a TINY
  # classification becomes CHEAP — and the floor/ceiling then override it, which
  # is the whole demonstration: the role, not the prompt length, decides which
  # model answers.
  bulk:
    #: High-volume grunt work. Capped at the free rung: a wrong classification
    #: cannot spend money here, and it may not escalate off it.
    default_tier: CHEAP
    max_tier: CHEAP
    escalate: false
  worker:
    #: The default working rung. May escalate one rung to FRONTIER when the
    #: cheap answer fails a structural check.
    default_tier: STANDARD
    escalate: true
  adjudicator:
    #: The step whose mistakes are expensive. Floored at FRONTIER, so even a
    #: TINY classification is answered by the frontier MODEL.
    min_tier: FRONTIER
    default_tier: FRONTIER
    escalate: false


# ── Cost / quality selection ────────────────────────────────────────────────
# Reorders the tier's candidate ring. Rates and quality scores are read from
# pricing.yaml, so a price change moves routing without touching this file.
#
#   objective: order | cost | quality | cost_quality
#     order         keep the tier's declared ring (v3 behaviour)
#     cost          cheapest blended $/Mtok first
#     quality       highest quality score first
#     cost_quality  score = tradeoff*quality - (1-tradeoff)*normalised_cost
#
#   tradeoff: 0.0 = pure cost, 1.0 = pure quality.
selection:
  objective: cost_quality
  tradeoff: 0.5
  #: Assumed output:input ratio when blending the two rates into one number.
  output_weight: 0.35
  #: Providers listed here are never reordered away from the front. Use for a
  #: local model you want tried first because it is free.
  pin_first: []


# ── Cascade escalation (FrugalGPT-style) ────────────────────────────────────
# Try the cheap tier; escalate only when the cheap answer looks bad. The point
# of the confidence gate is that escalating on *every* call is strictly worse
# than starting at the expensive tier — you pay for both.
escalation:
  enabled: true
  max_escalations: 1
  #: Answers scoring below this escalate one rung up the ladder.
  confidence_threshold: 0.5
  #: Which observations may trigger an escalation.
  triggers: [empty_response, provider_failure, low_confidence, schema_failure]
  #: Confidence is scored from structural signals only — no second LLM call,
  #: because paying for a judge to check a cheap answer often costs more than
  #: the expensive answer would have.
  confidence:
    empty_response: 0.0
    schema_failure: 0.1
    #: stop_reason == max_tokens. Scored just ABOVE the threshold on purpose:
    #: truncation almost always means the caller under-budgeted max_tokens, and
    #: a bigger model will hit the same cap. Escalating here would pay twice for
    #: the same truncated answer.
    truncated_response: 0.55
    #: Shorter than this many characters counts as a non-answer. DISABLED (0)
    #: on purpose. It was set to 8 during the build and immediately caused the
    #: failure this session is about: "Paris" — a correct one-word answer to a
    #: one-word question — scored 0.3, escalated to a bigger model, and made
    #: cost-per-resolved-task worse while cost-per-call looked unchanged.
    #: Emptiness is evidence that an answer is bad. Brevity is not. Set this
    #: only for a task class whose answers are known to be long.
    min_response_chars: 0
    short_response: 0.3
    #: Case-insensitive substrings that mark a hedge or refusal. EMPTY by
    #: default: a marker list is a use-case-specific guess, and a wrong one
    #: escalates good answers. Populate it for your own task class.
    hedge_markers: []
    hedge_penalty: 0.35
    default: 1.0


# ── Budget interaction ──────────────────────────────────────────────────────
# The budget controller is the authority on refusal; this is only a preference.
# When `prefer_affordable` is on, a tier whose cheapest provider is projected
# to breach the remaining allowance is deprioritised, so a run near its ceiling
# drifts down the ladder rather than slamming into a 402.
budget:
  prefer_affordable: true
  #: Refuse to escalate when the escalated tier would breach the allowance.
  block_escalation_on_breach: true


# ── Provider limits ─────────────────────────────────────────────────────────
# Per-provider pacing and context caps, merged over the built-in defaults in
# costhelm/routing/core.py at boot. Partial entries are allowed: raising one
# provider's rpm does not require restating the rest. The values below ARE the
# built-in defaults, restated as data so the numbers are editable without a
# code change (as of 2026-08: deliberate under-estimates of paid tiers — too
# low only throttles locally, too high buys 429s).
limits:
  ollama: {rpm: 9999, rpd: 9999999, tpm: 99999999, cooldown: 0, max_ctx: 32000}
  cerebras: {rpm: 30, rpd: 9999, tpm: 60000, cooldown: 2, max_ctx: 8000, tokens_per_day: 1000000}
  groq: {rpm: 30, rpd: 1000, tpm: 6000, cooldown: 2, max_ctx: 100000}
  nvidia: {rpm: 40, rpd: 9999, tpm: 100000, cooldown: 2, max_ctx: 100000}
  gemini: {rpm: 150, rpd: 10000, tpm: 1000000, cooldown: 1, max_ctx: 1000000}
  openrouter: {rpm: 60, rpd: 5000, tpm: 500000, cooldown: 1, max_ctx: 100000}
  github: {rpm: 10, rpd: 50, tpm: 99999999, cooldown: 6, max_ctx: 8000}
  # max_ctx is checked against the *prompt* estimate, so it is the model's
  # maximum input, not its full context window.
  openai: {rpm: 60, rpd: 9999, tpm: 200000, cooldown: 1, max_ctx: 922000}


# ── Agent pins ──────────────────────────────────────────────────────────────
# Map an agent name (the `agent` field on /v1/chat) to a preferred provider.
# A pin is a preference, not a hard binding: an explicit per-call `provider`
# still wins, and a pin to an unregistered provider is ignored. Ships empty.
#
#   planner: gemini
#   bulk_classifier: ollama
agent_pins: {}

Environment variables

Section titled “Environment variables”
Variable Meaning
COSTHELM_PORT / COSTHELM_HOST bind address (default 0.0.0.0:8111)
COSTHELM_CONFIG_DIR user-config directory (default ~/.costhelm)
COSTHELM_GATEWAY_DB ledger path (default <config dir>/gateway.sqlite)
COSTHELM_PRICING_YAML etc. per-file path overrides (_BUDGETS_, _ROUTING_, _CACHE_)
COSTHELM_ORDER logical provider failover order for non-routed calls
COSTHELM_ROUTER_ORDER router-pool order for the tier classifier
COSTHELM_API_KEYS comma-separated keys; setting it turns auth on for /v1/*
OTEL_EXPORTER_OTLP_ENDPOINT OTLP/HTTP collector base URL
COSTHELM_TRACE_UI trace UI base for dashboard deep links
COSTHELM_OTEL_CAPTURE_CONTENT opt-in prompt/completion capture on spans
provider keys OPENAI_API_KEY, GEMINI_API_KEY_1..16, OPEN_ROUTER_API_KEY, GROQ_API_KEY, CEREBRAS_API_KEY, NVIDIA_API_KEY, GITHUB_ACCESS_TOKEN, OLLAMA_MODEL + OLLAMA_URL
model overrides <PROVIDER>_MODEL, ROUTER_<PROVIDER>_MODEL