Architecture

costhelm is a FastAPI service organised around one idea: cost is a precondition, not a report.

The request pipeline

POST /v1/chat runs four stages (costhelm/pipeline/):

POST /v1/chat
  │
  1. build_context      normalise, estimate tokens, resolve principal, agent pin
  2. semantic cache     hit → return stored answer, bill $0, record tokens saved
  3. select_candidates  router-LLM tier classify → role clamp → cost/quality order
  4. dispatch           ┌ cascade loop ──────────────────────────────┐
                        │  for each candidate:                       │
                        │    budget admission (worst-case, pre-call) │──402
                        │    provider call (+1 retry, backoff)       │
                        │    structured-output validate/repair       │
                        │    confidence check → maybe escalate tier ─┘
                        │    meter → priced ledger row, OTel span
                        └ return text + cost/budget/cache envelopes

Each stage is inert unless configured: with the shipped config files, costhelm behaves like a plain failover gateway.

Invariants

Budget admission runs before the provider call. The controller projects the worst case (estimated input + full max_tokens at the model’s rates) and refuses with a 402 whose body carries the arithmetic. Nothing was sent, so the refusal costs $0. (economics/budget.py)

Spend is re-derived from the ledger. Every admission reads spend back out of the calls table — there is no parallel counter to drift. Migrations are additive-only, so an old ledger upgrades in place. (db.py)

Confidence is structural, never a judge LLM. The cascade escalates on empty text, schema failure, or provider failure — read off the response itself. Paying a judge to check a cheap answer often costs more than the expensive answer would have. (routing/policy.py)

Boot is fail-soft, and failures are surfaced. A broken pricing.yaml costs you dollar reports, not the gateway — but the error is recorded and shown, so a disarmed budget controller is never mistaken for a generous one. (main.py)

Refusals are observed at the HTTP boundary. A 402 writes no ledger row and opens no span — correct, and a blind spot — so an ASGI middleware keeps a bounded ring of refusals for /v1/refusals. (routes/observability.py)

Nothing model-specific is hardcoded. Roles, tiers, prices, limits, pins — all YAML. CI fails if a configured default model would price at $0.

Package map

costhelm/
├── main.py            boot wiring: fail-soft lifespan, middleware, static pages
├── config.py          3-layer config resolution
├── auth.py            optional API-key middleware for /v1/*
├── db.py              the priced ledger (SQLite)
├── schemas.py         the wire contract (Pydantic)
├── client.py          CosthelmClient, the reference consumer
├── providers/         one module per provider + the env-driven registry
├── pipeline/          context → semantic → tiering → dispatch
├── routing/           failover core + routing.yaml policy
├── economics/         pricing, meter, budget controller
├── cache/             semantic response cache + Gemini prompt cache
├── telemetry/         OTel spans (gen_ai.* + costhelm.*)
└── routes/            chat/batch/vision/embed, economics, observability