gate.cost: Spend correlation#

Correlate private model request IDs with explicit-confidence spend records from a LiteLLM gateway. Only exact request identifiers cross the local ledger-to-gateway boundary. The result distinguishes provider-billed, gateway-calculated, estimated, local-zero-marginal, and unavailable costs.

Data models are frozen dataclasses; monetary amounts are decimal.Decimal.

Exceptions#

exception gate.cost.CostLookupError#

A bounded spend lookup failed without exposing transport or secret details.

Session interface#

class gate.cost.SessionRecord(*args, **kwargs)#

Minimal structural session interface needed by cost functions.

The real SessionRecord lives in the corpus agentlog; we only need models and model_request_ids for cost correlation. Users who have the full session record can pass any object satisfying this protocol.

Spend correlation#

gate.cost.correlate_session(record: SessionRecord, snapshot: SpendSnapshot) → SessionCost#

Correlate one session only when every observed request has an exact spend row.

Parameters:

  • record – Structural agent session with private request IDs.

  • snapshot – Content-free results from exact LiteLLM spend queries.

Returns:

A complete gateway-calculated total, a local-zero-marginal result, or an unavailable result with coverage counts. Partial numeric totals are discarded.

gate.cost.approximate_session_cost(record: SessionRecord, window: SpendWindow, *, buffer_minutes: int = 5, model_min_score: float = 0.3) → SessionCost#

Correlate a session to gateway spend by time window + model proximity.

Matches every gateway row whose start_time falls within the session’s time span (padded by buffer_minutes) and whose model label has at least model_min_score similarity to one of the session’s recorded models.

Returns an estimated cost with observed/matched request coverage. This is the approximate join: not exact (no generation ID), but defensible as an estimate when the exact path is harness-blocked.

gate.cost.baseline_session_cost(record: SessionRecord) → SessionCost#

Return the best content-free cost known without any network access.

gate.cost.unavailable_cost(*, observed_requests: int = 0, matched_requests: int = 0, source: 'none' | 'litellm' | 'local-vllm' | 'openrouter' | 'openai' | 'anthropic' | 'modal' = 'none') → SessionCost#

Return an explicitly unavailable cost without inventing a numeric estimate.

Data models#

class gate.cost.SpendRecord(request_id: str, spend: Decimal = Decimal('0'))#

The only LiteLLM spend-log fields permitted past the network adapter.

class gate.cost.SpendSnapshot(requested_ids: set[str] = <factory>, rows: list[SpendRecord] = <factory>)#

Exact request IDs queried and the content-free rows returned for them.

class gate.cost.SessionCost(status: 'billed' | 'gateway_calculated' | 'estimated' | 'local_no_marginal_cost' | 'local_energy' | 'unavailable' = 'unavailable', amount_microusd: int | None = None, currency: 'USD' = 'USD', source: 'none' | 'litellm' | 'local-vllm' | 'openrouter' | 'openai' | 'anthropic' | 'modal' = 'none', observed_requests: int = 0, matched_requests: int = 0)#

One session’s cost with provenance and request-level coverage.

class gate.cost.GatewayRow(start_time: datetime, request_id: str = '', model: str = '', total_tokens: int = 0, spend: Decimal = Decimal('0'))#

A content-free LiteLLM spend row for time-window approximate correlation.

Only the fields needed for time + model + token approximate joins survive; request_id is kept for dedup but never used as an exact join key here.

class gate.cost.SpendWindow(rows: list[GatewayRow] = <factory>, start: datetime | None = None, end: datetime | None = None, count: int = 0, total_spend: Decimal = Decimal('0'))#

A bounded time window of content-free gateway rows for approximate cost join.

rows#

All gateway spend rows within the window (no request-level access).

Type:

list[GatewayRow]

start#

Window start (ISO date granularity).

Type:

datetime | None

end#

Window end.

Type:

datetime | None

count#

Number of rows.

Type:

int

total_spend#

Sum of all row spends in USD.

Type:

Decimal

Spend API#

gate.cost.fetch_spend_snapshot(request_ids: Sequence[str], *, base_url: str, token: str, max_requests: int = 256, timeout: float = 5.0) → SpendSnapshot#

Fetch exact LiteLLM spend rows with a bounded, caller-supplied scoped key.

The token should be a LiteLLM virtual key restricted to spend-read routes. The adapter intentionally avoids the gateway master key and never performs an unbounded all-history scan.

Parameters:

  • request_ids – Opaque IDs captured from local agent response metadata.

  • base_url – LiteLLM gateway root, such as http://127.0.0.1:4000.

  • token – A scoped spend-read virtual key.

  • max_requests – Hard cap on exact HTTP lookups in one attempt.

  • timeout – Per-request transport timeout in seconds.

Returns:

The queried IDs and content-free spend rows.

Raises:

CostLookupError – If bounds, identifiers, response shape, or transport fail.

gate.cost.fetch_spend_window(since: datetime, until: datetime, *, base_url: str = 'http://localhost:4000', token: str = '') → SpendWindow#

Fetch a bounded time window from the LiteLLM gateway (date-granular).

Returns content-free GatewayRow objects; raw content-bearing fields (messages, response, api_key, etc.) are dropped at the parse boundary. On any transport/parse failure returns an empty SpendWindow.

gate.cost.fetch_spend_summary(*, base_url: str, token: str, since: datetime | None = None, until: datetime | None = None) → SpendSummary#

Fetch a spend window and return both per-model and per-project aggregations.

Parameters:

  • base_url – LiteLLM gateway root, such as http://127.0.0.1:4000.

  • token – A scoped spend-read virtual key.

  • since – Window start (defaults to 24 hours ago).

  • until – Window end (defaults to now).

Returns:

A SpendSummary with per-model and per-project aggregations.

Energy estimation#

gate.cost.local_energy_cost(record: SessionRecord, node_name: str, *, electricity_rate_per_kwh: Decimal | None = None) → SessionCost#

Compute the electricity cost of a local-inference session.

Delegates to local_energy() for the canonical energy quantity, then derives USD from it via the electricity rate. Energy is the root; USD is the derived view.

This is a usable metric, not an audit-grade one: the wattage is a typical-load estimate from vendor specs, not a measured RAPL/NVML reading, and the rate is the published energy charge before riders. The user can override either input.

Parameters:

  • record – The agent session record (uses active duration, not wall).

  • node_name – Fleet node name from the compute ledger (e.g. 'salt').

  • electricity_rate_per_kwh – Override the rate; defaults to the Georgia Power seasonal marginal rate for Atlanta 30309.

Returns:

A local_energy SessionCost with the electricity cost in micro-USD, or unavailable if the node is unknown.

gate.cost.local_energy(record: SessionRecord, node_name: str) → EnergyRecord#

Compute local-inference energy from node wattage x active duration.

This is the time-based model: it uses the node’s typical sustained load wattage (from NODE_LOAD_WATTS) multiplied by the session’s active duration. It does not reflect batch size, utilization, or model load – it is a coarse proxy. When RAPL/NVML power sampling is available, a measured basis should replace this.

Parameters:

  • record – The agent session record (uses active duration).

  • node_name – Fleet node name (e.g. 'salt').

Returns:

An EnergyRecord with energy in microjoules, or unavailable.

gate.cost.energy_from_tokens(models: Sequence[str], *, prefill_tokens: int, decode_tokens: int, cache_read_tokens: int) → EnergyRecord#

Estimate energy from raw phase-split token counts (no SessionRecord needed).

Exposed for callers – like the quota tracker – that hold a windowed token tally rather than a whole session. The first matching model gives the point estimate; the band spans all matched models.

Parameters:

  • models – Model label(s); the first match drives the point estimate.

  • prefill_tokens – Compute-bound input tokens (uncached input + cache creation).

  • decode_tokens – Memory-bound generated tokens (output + reasoning).

  • cache_read_tokens – KV-cache hit tokens.

Returns:

An EnergyRecord in microjoules, or unavailable if no coefficients match.

class gate.cost.EnergyRecord(amount_microjoules: int | None = None, basis: 'measured' | 'node_wattage_time' | 'token_coefficient' | 'unavailable' = 'unavailable', method: 'time' | 'token' | None = None, source: str = '', coverage: float = 0.0, coefficient_version: str = '', amount_microjoules_low: int | None = None, amount_microjoules_high: int | None = None)#

One session’s energy consumption with provenance.

The canonical store is amount_microjoules (integer, like micro-USD). Joules, watt-hours, and kWh are derived properties. USD and CO2e are further derived views via electricity rate and grid carbon intensity.

class gate.cost.EnergyCoeff(prefill_mj: int = 0, decode_mj: int = 0, cache_read_mj: int = 0, source: str = '', version: str = '')#

Per-model energy coefficients in microjoules per token, by phase.

Prefill covers input-token processing (compute-bound, better GPU util). Decode covers output-token generation (memory-bound at batch 1). Cache-read covers KV-cache hits (memory read only, ~5-10% of prefill).

Aggregation#

class gate.cost.ModelSpendSummary(model: str, total_spend: Decimal = Decimal('0'), request_count: int = 0, total_tokens: int = 0, avg_cost_per_request: Decimal = Decimal('0'), first_seen: datetime | None = None, last_seen: datetime | None = None)#

Aggregated spend by model alias.

model#

Model name/alias.

Type:

str

total_spend#

Total spend in USD.

Type:

Decimal

request_count#

Number of requests.

Type:

int

total_tokens#

Sum of total_tokens across all rows.

Type:

int

avg_cost_per_request#

Total spend divided by request count.

Type:

Decimal

first_seen#

Earliest row timestamp.

Type:

datetime | None

last_seen#

Latest row timestamp.

Type:

datetime | None

class gate.cost.ProjectSpendSummary(project: str, total_spend: Decimal = Decimal('0'), request_count: int = 0, total_tokens: int = 0, avg_cost_per_request: Decimal = Decimal('0'), first_seen: datetime | None = None, last_seen: datetime | None = None, model_breakdown: dict[str, ~decimal.Decimal]=<factory>)#

Aggregated spend by project/metadata.

project#

Project name (or ‘unknown’ if metadata is absent).

Type:

str

total_spend#

Total spend in USD.

Type:

Decimal

request_count#

Number of requests.

Type:

int

total_tokens#

Sum of total_tokens across all rows.

Type:

int

avg_cost_per_request#

Total spend divided by request count.

Type:

Decimal

first_seen#

Earliest row timestamp.

Type:

datetime | None

last_seen#

Latest row timestamp.

Type:

datetime | None

model_breakdown#

Per-model spend within this project.

Type:

dict[str, Decimal]

class gate.cost.SpendSummary(by_model: dict[str, ~gate.cost.ModelSpendSummary]=<factory>, by_project: dict[str, ~gate.cost.ProjectSpendSummary]=<factory>, total_spend: Decimal = Decimal('0'), total_requests: int = 0, window_start: datetime | None = None, window_end: datetime | None = None)#

High-level spend summary aggregating per-model and per-project views.

by_model#

Spend aggregated by model alias.

Type:

dict[str, ModelSpendSummary]

by_project#

Spend aggregated by project/metadata.

Type:

dict[str, ProjectSpendSummary]

total_spend#

Sum of all spend across the window.

Type:

Decimal

total_requests#

Total number of requests.

Type:

int

window_start#

Start of the fetch window (if known).

Type:

datetime | None

window_end#

End of the fetch window (if known).

Type:

datetime | None

gate.cost.aggregate_by_model(rows: list[GatewayRow]) → dict[str, ModelSpendSummary]#

Group spend rows by model name into per-model summaries.

Parameters:

rows – Gateway spend rows to aggregate.

Returns:

Mapping from model name to ModelSpendSummary.

gate.cost.aggregate_by_project(rows: list[GatewayRow]) → dict[str, ProjectSpendSummary]#

Group spend rows by project/metadata into per-project summaries.

Since GatewayRow does not carry project metadata, all rows are grouped under ‘unknown’. Callers with enriched rows may post-process the result.

Parameters:

rows – Gateway spend rows to aggregate.

Returns:

Mapping from project name to ProjectSpendSummary.