# Evaluating a model or judge

`gate eval` scores a model against the checked-in, independently adjudicated ground truth in `resources/evalset/`: `cases.jsonl` (full-review recall/precision) and `judge-cases.jsonl` (judge-seat KEEP/REJECT accuracy).
It replaces four earlier, disjoint eval layers that never measured the right thing at once -- substring scoring with no precision metric, right-shaped metrics with zero callers, one-off scratch scripts, and a retired optimizer.

Every case's diff is embedded directly in its JSONL row.
Each run synthesizes a throwaway two-commit git repository straight from that diff -- `git init`, a base commit, a head commit -- and drives the real review pipeline against it, so scoring never depends on a historical SHA still existing in some other repository.

```sh
# Score gitlab-critic's full review pipeline against every positive/clean case, 3 runs each.
gate eval run --cases resources/evalset/cases.jsonl --model gitlab-critic

# Cap spend and only look at the first 5 cases; a hard stop between runs, never mid-run.
gate eval run --cases resources/evalset/cases.jsonl --model gitlab-critic \
  --max-cases 5 --max-cost 2.00

# Score a judge model's KEEP/REJECT accuracy against real/spurious findings.
gate eval judge --cases resources/evalset/judge-cases.jsonl --model qwen3.8-27b

# Full JSON report on stdout; always also written to --out (default ./eval-out/<timestamp>/).
gate eval run --cases resources/evalset/cases.jsonl --model gitlab-critic --json
```

## Review mode (`gate eval run`)

Each case is scored by a deterministic rubric with no LLM in the loop: a produced finding matches an expected defect iff it names the same file, its line falls within the expected line window (±5 by default), and any one of the case's mechanism keywords appears, case-insensitively, in the finding's title or body.

This matcher is a heuristic. A nearby finding can share a generic keyword while describing a different or nonexistent failure. Before changing a review route, inspect the findings against each expected mechanism as well as the aggregate scores; a keyword match alone does not establish correct defect detection.

From that:

- **effective recall** -- an expected defect counts as covered if any run raised a
  matching finding at all.
- **precision** -- matching findings over every finding raised, pooled across runs; on a
  `clean` case every finding is by definition a false positive.
- **determinism** -- mean pairwise Jaccard similarity of the finding-id set across runs.
- **availability** -- the fraction of runs that produced a usable review. Blocked and
  marker-only process outcomes are unavailable; real findings alongside a marker remain
  scoreable.

Runs repeat sequentially per case (`--runs`, default 3) -- the gateway behind `localhost:4000` rejects overlapping calls on one virtual key, so there is no concurrency dial to turn here.

## Judge mode (`gate eval judge`)

Calls `gate.review.judge.judge_findings` directly against batches of findings sharing one source diff (production's own shape -- a batch is what the judge actually sees), scoring:

- **real-keep rate** -- adjudicated-real findings the judge kept.
- **FP-rejection rate** -- adjudicated-spurious findings the judge rejected.
- **failed-call rate** -- batches where the judge failed closed (kept everything, per its
  own fail-open contract) rather than returning a validated verdict.

## Cost estimation

A report's `budget` block is honest about what it does not know: only models with a verified `$`/token rate in `evaluation.PRICES` get an estimated `spent_usd`, and only those models let `--max-cost` actually stop the sweep.
A model absent from `PRICES` reports `spent_usd: null` and `budget.enforced: false` rather than a silent `0.0` -- a free reading would be the same class of failure as a clean verdict manufactured from a review that never ran.

## The `ai` extra

Judge mode needs `pip install my-gate[ai]` (pydantic-ai); review mode does not, and fails with a plain `install my-gate[ai]` message rather than an import traceback when the extra is missing and a judge stage is requested anyway.