Evaluating a model or judge#

gate eval scores a model against the checked-in, independently adjudicated ground truth in resources/evalset/: cases.jsonl (full-review recall/precision) and judge-cases.jsonl (judge-seat KEEP/REJECT accuracy). It replaces four earlier, disjoint eval layers that never measured the right thing at once – substring scoring with no precision metric, right-shaped metrics with zero callers, one-off scratch scripts, and a retired optimizer.

Every case’s diff is embedded directly in its JSONL row. Each run synthesizes a throwaway two-commit git repository straight from that diff – git init, a base commit, a head commit – and drives the real review pipeline against it, so scoring never depends on a historical SHA still existing in some other repository.

# Score gitlab-critic's full review pipeline against every positive/clean case, 3 runs each.
gate eval run --cases resources/evalset/cases.jsonl --model gitlab-critic

# Cap spend and only look at the first 5 cases; a hard stop between runs, never mid-run.
gate eval run --cases resources/evalset/cases.jsonl --model gitlab-critic \
  --max-cases 5 --max-cost 2.00

# Score a judge model's KEEP/REJECT accuracy against real/spurious findings.
gate eval judge --cases resources/evalset/judge-cases.jsonl --model qwen3.8-27b

# Full JSON report on stdout; always also written to --out (default ./eval-out/<timestamp>/).
gate eval run --cases resources/evalset/cases.jsonl --model gitlab-critic --json

Review mode (gate eval run)#

Each case is scored by a deterministic rubric with no LLM in the loop: a produced finding matches an expected defect iff it names the same file, its line falls within the expected line window (±5 by default), and any one of the case’s mechanism keywords appears, case-insensitively, in the finding’s title or body.

This matcher is a heuristic. A nearby finding can share a generic keyword while describing a different or nonexistent failure. Before changing a review route, inspect the findings against each expected mechanism as well as the aggregate scores; a keyword match alone does not establish correct defect detection.

From that:

  • effective recall – an expected defect counts as covered if any run raised a matching finding at all.

  • precision – matching findings over every finding raised, pooled across runs; on a clean case every finding is by definition a false positive.

  • determinism – mean pairwise Jaccard similarity of the finding-id set across runs.

  • availability – the fraction of runs that produced a usable review. Blocked and marker-only process outcomes are unavailable; real findings alongside a marker remain scoreable.

Runs repeat sequentially per case (--runs, default 3) – the gateway behind localhost:4000 rejects overlapping calls on one virtual key, so there is no concurrency dial to turn here.

Judge mode (gate eval judge)#

Calls gate.review.judge.judge_findings directly against batches of findings sharing one source diff (production’s own shape – a batch is what the judge actually sees), scoring:

  • real-keep rate – adjudicated-real findings the judge kept.

  • FP-rejection rate – adjudicated-spurious findings the judge rejected.

  • failed-call rate – batches where the judge failed closed (kept everything, per its own fail-open contract) rather than returning a validated verdict.

Cost estimation#

A report’s budget block is honest about what it does not know: only models with a verified $/token rate in evaluation.PRICES get an estimated spent_usd, and only those models let --max-cost actually stop the sweep. A model absent from PRICES reports spent_usd: null and budget.enforced: false rather than a silent 0.0 – a free reading would be the same class of failure as a clean verdict manufactured from a review that never ran.

The ai extra#

Judge mode needs pip install my-gate[ai] (pydantic-ai); review mode does not, and fails with a plain install my-gate[ai] message rather than an import traceback when the extra is missing and a judge stage is requested anyway.