Evals run in CI, not by hand.
An eval you run by hand is one you skip under deadline. Wired into the pipeline, it blocks the bad merge.
The right scorer for each case.
Schema checks and exact matches are faster and steadier. The judge is kept for what is genuinely qualitative.
Cost and latency are results too.
A prompt that adds tokens or a model that doubles p95 is a regression, measured right next to correctness.
A judge with a rubric, not a vibe.
Explicit criteria make the judge's scores stable enough to gate on, and every failure explainable.
Verdict · LLM eval & release gate
Ship AI changes
without holding
your breath.
Run real evals against a golden set, deterministic checks and an LLM-as-judge, then gate the release on the result. Open any exhibit, edit the output, and watch the gate flip.
Runs in your browserNo sign-upNo API key


Docket
golden set · 8 exhibitsSample runIn the matter of the candidate v. the golden set
Order on release
Gate: pass rate ≥ 90%
Sample run of the app's own golden set and checks. Tokens are characters ÷ 4, priced at a sample $0.14 per 1k; every check is timed live in your browser. The judge is a keyword rubric in this demo.
§ 1What it does
Every model change stands trial.
A prompt tweak, a model swap or a new retrieval step is a change to production. Verdict tries it against a golden set of cases with known-good behavior and rules on whether it may ship.
Is it right?
Eight exhibits, each with one rule. Seven are checked exactly by code; one, tone, is scored by a judge against a written rubric.
What does it cost?
Every answer is counted in tokens and priced, per exhibit and per run, so a wordier prompt shows up as money.
How long does it take?
Every check is timed where it runs, so a slow instrument sits on the record right next to a wrong answer.
May it ship?
The pass rate meets the threshold and the release clears, or it falls short and the merge is held, like a failing CI check.
Rules of evidence
the seven instruments in the golden set| Instrument | What it proves | Passing | Regr. | Broken |
|---|---|---|---|---|
| containsanswers_questionThe output states a required fact, matched without regard to case. | The output states a required fact, matched without regard to case. | Passing run: passes | Regression: passes | Broken: fails |
| not-containsno_pii_leak, no_hallucinationA forbidden string never appears: leaked data, a stock disclaimer. | A forbidden string never appears: leaked data, a stock disclaimer. | Passing run: passes | Regression: passes | Broken: passes |
| regexorder_id_formatThe trimmed output matches a pattern end to end, such as an order ID. | The trimmed output matches a pattern end to end, such as an order ID. | Passing run: passes | Regression: passes | Broken: passes |
| citationcites_sourceThe answer cites its source with a bracketed reference like [1]. | The answer cites its source with a bracketed reference like [1]. | Passing run: passes | Regression: fails | Broken: fails |
| json-schemavalid_jsonThe output parses as JSON and every required key has the right type. | The output parses as JSON and every required key has the right type. | Passing run: passes | Regression: passes | Broken: fails |
| max-lengthsummary_under_160The output fits a hard character budget. | The output fits a hard character budget. | Passing run: passes | Regression: passes | Broken: passes |
| llm-judgetone_checkA rubric scores what code cannot check exactly, such as tone. | A rubric scores what code cannot check exactly, such as tone. | Passing run: passes | Regression: passes | Broken: fails |
Marks are computed from the app's own three sample candidates when this page is built: a solid check passes, a hatched cross fails.
§ 2How it works
The order of proceedings.
Every run follows the same five steps, in the same order, so two runs can be compared line by line.
File the golden set
Eight cases pin the inputs and what a good answer must have: a fact, a citation, a schema, a length, a tone.
Call the candidate
The change under test answers every case. Three sample candidates are on file, and you can rewrite any answer.
Examine each exhibit
Exact checks rule where the answer is structured. The judge scores only what code cannot, against its rubric.
Enter it in the ledger
Each answer is counted in tokens, priced and timed, and the whole run is totalled at the foot of the page.
Rule at the gate
At or above the threshold, the release clears. Below it, or with a must-pass exhibit failed, the merge is held.
In the docket above, the gavel falls once for each exhibit and twice for the verdict.
§ 3The architecture
Built like a test runner.
Datasets in, scored results out, a gate in the middle. In this demo every room of the courthouse runs inside your browser tab, from one pure module, lib/verdict.ts.
Your browser · no server · no API key
Golden set
8 cases, each one id, one check, one rule
CASESCandidate
3 sample outputs, plus every edit you make
CANDIDATESExaminer
6 exact check kinds; a judge for tone
runCheck · judgeScoreLedger
tokens = chars ÷ 4, cost at a sample rate, time via performance.now()
examine · timeCheckGate
pass rate ≥ threshold, must-pass exhibits, cost ceiling
decideVerdict
PASSED clears the release; BLOCKED holds the merge
RunRecord
Where each room goes in a production pipeline
- ExaminerThe judge becomes a model call with a rubric prompt and a fixed pass mark.
- LedgerToken counts and cost come from the provider's usage, not a character estimate.
- GateThe run is a CI step: a blocked verdict exits non-zero and the merge waits.
§ 4The stack
Entered into the record.
Small on purpose: the eval engine is a set of pure functions, so the landing, the console and the tests all run the same code.
- Framework
- Next.js 16, App Router
- Interface
- React 19, server-rendered
- Language
- TypeScript, strict
- Styles
- Tailwind CSS 4, inked SVG
- Type
- Fraunces, Franklin, Plex Mono
- Tests
- Vitest: checks, judge, gate
- Hosting
- Netlify
- Backend
- None, and no API key
export function runCheck(check: Check, out: string, opts: CheckOptions = {}): boolean {
const s = out ?? "";
switch (check.kind) {
case "contains":
return s.toLowerCase().includes(check.value.toLowerCase());
case "not_contains":
return !s.toLowerCase().includes(check.value.toLowerCase());
case "regex":
try {
return new RegExp(check.value).test(s.trim());
} catch {
return false;
}
case "citation":
return /\[\d+\]/.test(s);
case "max_length":
return s.length <= check.value;
case "judge":
return judgeScore(s) >= (opts.passMark ?? RUBRIC.passMark) - 1e-9;
The json-schema case continues in the file: parse, then check the type of every required key.
§ 5The case study
Four rulings from the brief.
The write-up on rikenpatel.work covers why Verdict exists, the decisions behind it, and what it changed about shipping AI features.
The console
Court is in session.
Pick a candidate, rewrite its testimony, set the gate and its must-pass exhibits, and keep a record of every run you make. It all happens in this tab.