Verdict · LLM eval & release gate

Ship AI changes
without holding
your breath.

Run real evals against a golden set, deterministic checks and an LLM-as-judge, then gate the release on the result. Open any exhibit, edit the output, and watch the gate flip.

Runs in your browserNo sign-upNo API key

Docket

golden set · 8 exhibitsSample run

In the matter of the candidate v. the golden set

Candidate under test

Order on release

Gate: pass rate ≥ 90%

Totals$0.006043 tok2/325%$0.0060 · 43 tok · 2/3 passed
Release gate drag the flagpass rate 25%
Awaiting the ruling

Sample run of the app's own golden set and checks. Tokens are characters ÷ 4, priced at a sample $0.14 per 1k; every check is timed live in your browser. The judge is a keyword rubric in this demo.

§ 1What it does

Every model change stands trial.

A prompt tweak, a model swap or a new retrieval step is a change to production. Verdict tries it against a golden set of cases with known-good behavior and rules on whether it may ship.

  1. Is it right?

    Eight exhibits, each with one rule. Seven are checked exactly by code; one, tone, is scored by a judge against a written rubric.

  2. What does it cost?

    Every answer is counted in tokens and priced, per exhibit and per run, so a wordier prompt shows up as money.

  3. How long does it take?

    Every check is timed where it runs, so a slow instrument sits on the record right next to a wrong answer.

  4. May it ship?

    The pass rate meets the threshold and the release clears, or it falls short and the merge is held, like a failing CI check.

Rules of evidence

the seven instruments in the golden set
Each check Verdict runs, the exhibits that use it, and whether each sample candidate passes it
InstrumentWhat it provesPassingRegr.Broken
containsanswers_questionThe output states a required fact, matched without regard to case.The output states a required fact, matched without regard to case.Passing run: passesRegression: passesBroken: fails
not-containsno_pii_leak, no_hallucinationA forbidden string never appears: leaked data, a stock disclaimer.A forbidden string never appears: leaked data, a stock disclaimer.Passing run: passesRegression: passesBroken: passes
regexorder_id_formatThe trimmed output matches a pattern end to end, such as an order ID.The trimmed output matches a pattern end to end, such as an order ID.Passing run: passesRegression: passesBroken: passes
citationcites_sourceThe answer cites its source with a bracketed reference like [1].The answer cites its source with a bracketed reference like [1].Passing run: passesRegression: failsBroken: fails
json-schemavalid_jsonThe output parses as JSON and every required key has the right type.The output parses as JSON and every required key has the right type.Passing run: passesRegression: passesBroken: fails
max-lengthsummary_under_160The output fits a hard character budget.The output fits a hard character budget.Passing run: passesRegression: passesBroken: passes
llm-judgetone_checkA rubric scores what code cannot check exactly, such as tone.A rubric scores what code cannot check exactly, such as tone.Passing run: passesRegression: passesBroken: fails

Marks are computed from the app's own three sample candidates when this page is built: a solid check passes, a hatched cross fails.

§ 2How it works

The order of proceedings.

Every run follows the same five steps, in the same order, so two runs can be compared line by line.

  1. File the golden set

    Eight cases pin the inputs and what a good answer must have: a fact, a citation, a schema, a length, a tone.

  2. Call the candidate

    The change under test answers every case. Three sample candidates are on file, and you can rewrite any answer.

  3. Examine each exhibit

    Exact checks rule where the answer is structured. The judge scores only what code cannot, against its rubric.

  4. Enter it in the ledger

    Each answer is counted in tokens, priced and timed, and the whole run is totalled at the foot of the page.

  5. Rule at the gate

    At or above the threshold, the release clears. Below it, or with a must-pass exhibit failed, the merge is held.

In the docket above, the gavel falls once for each exhibit and twice for the verdict.

§ 3The architecture

Built like a test runner.

Datasets in, scored results out, a gate in the middle. In this demo every room of the courthouse runs inside your browser tab, from one pure module, lib/verdict.ts.

Your browser · no server · no API key

  1. Golden set

    8 cases, each one id, one check, one rule

    CASES
  2. Candidate

    3 sample outputs, plus every edit you make

    CANDIDATES
  3. Examiner

    6 exact check kinds; a judge for tone

    runCheck · judgeScore
  4. Ledger

    tokens = chars ÷ 4, cost at a sample rate, time via performance.now()

    examine · timeCheck
  5. Gate

    pass rate ≥ threshold, must-pass exhibits, cost ceiling

    decide
  6. Verdict

    PASSED clears the release; BLOCKED holds the merge

    RunRecord

Where each room goes in a production pipeline

  • ExaminerThe judge becomes a model call with a rubric prompt and a fixed pass mark.
  • LedgerToken counts and cost come from the provider's usage, not a character estimate.
  • GateThe run is a CI step: a blocked verdict exits non-zero and the merge waits.
Data flows from the golden set to the candidate, the examiner, the ledger, the gate and the verdict, all inside the browser.

§ 4The stack

Entered into the record.

Small on purpose: the eval engine is a set of pure functions, so the landing, the console and the tests all run the same code.

Framework
Next.js 16, App Router
Interface
React 19, server-rendered
Language
TypeScript, strict
Styles
Tailwind CSS 4, inked SVG
Type
Fraunces, Franklin, Plex Mono
Tests
Vitest: checks, judge, gate
Hosting
Netlify
Backend
None, and no API key
Exhibit Alib/verdict.ts · runCheck
export function runCheck(check: Check, out: string, opts: CheckOptions = {}): boolean {
  const s = out ?? "";
  switch (check.kind) {
    case "contains":
      return s.toLowerCase().includes(check.value.toLowerCase());
    case "not_contains":
      return !s.toLowerCase().includes(check.value.toLowerCase());
    case "regex":
      try {
        return new RegExp(check.value).test(s.trim());
      } catch {
        return false;
      }
    case "citation":
      return /\[\d+\]/.test(s);
    case "max_length":
      return s.length <= check.value;
    case "judge":
      return judgeScore(s) >= (opts.passMark ?? RUBRIC.passMark) - 1e-9;

The json-schema case continues in the file: parse, then check the type of every required key.

§ 5The case study

Four rulings from the brief.

The write-up on rikenpatel.work covers why Verdict exists, the decisions behind it, and what it changed about shipping AI features.

  1. Evals run in CI, not by hand.

    An eval you run by hand is one you skip under deadline. Wired into the pipeline, it blocks the bad merge.

  2. The right scorer for each case.

    Schema checks and exact matches are faster and steadier. The judge is kept for what is genuinely qualitative.

  3. Cost and latency are results too.

    A prompt that adds tokens or a model that doubles p95 is a regression, measured right next to correctness.

  4. A judge with a rubric, not a vibe.

    Explicit criteria make the judge's scores stable enough to gate on, and every failure explainable.

The console

Court is in session.

Pick a candidate, rewrite its testimony, set the gate and its must-pass exhibits, and keep a record of every run you make. It all happens in this tab.

Launch the console Docket · Golden set · Checks · Runs · Gate