For solo builders and teams shipping AI faster than humans can review

An AI reviewer
for the code and content you ship.

Judge scores your code, PR diffs, and written content against a typed rubric. On every PR, edit, or prompt change, see exactly what got better, what got worse, and what to fix — before it reaches users.

  • What it scores: code · PR diffs · files & folders · generated AI output · docs & text.
  • How it judges: typed rubrics — boolean gates and 0–10 metrics, weighted. Deterministic, repeatable, audit-friendly.
  • Where it lives: GitHub App on every PR, CLI in CI, TypeScript SDK for code-first regression tests, MCP for agents, REST anywhere else.
Typed rubrics16 seededCustom rubrics·GitHub App · CI · MCP
judge.dashboard / score #314
Rubriccode-quality

Scoring src/auth.ts on acme/api

0.0
+0.0 improved
Type safety8.1 +0.6
Input validation6.4 +0.9
Error handling7.8 -0.2
Test signal6.9 +0.4
Last 8 iterationstrending
Triggered byMCP · claude-code
Try it

See it in 30 seconds. No signup.

What you actually type, what you actually get back. One Node CLI (npx judge), three real workflows. Every score is a typed row in your dashboard with cited evidence — not a chat blob.

JUDGE_API_KEY=jdg_…· mint on /connect· BYOK Anthropic key works too
Free account, no card·BYOK pays Anthropic at cost — one Sonnet 4.6 call per score (~$0.01–$0.05)·First score in <5 min — `npm i -g judge` → `judge init` → `judge run`
01

"Did my last edit make this file better — or worse?"

# Score what HEAD did to a file vs HEAD~1
$ judge score-edit src/auth.ts --judge code-quality

→ scoring before (HEAD~1)…  72.4
→ scoring after  (HEAD)  …  68.9

✗ verdict: REGRESSED   weighted Δ −3.5
  ↓ input_validation  0.9 → 0.6  (−0.3)
  ↓ error_handling    0.8 → 0.5  (−0.3)
  = type_safety       1.0 → 1.0   (·)

history: 12 runs → 71.6 → 71.8 → 72.4 → 68.9
scorecard: judge.tools/p/8a7…f2c (cited evidence)
exit 2  (--fail-on-regression)

The wedge: edit scoring. Diffs HEAD~1 → HEAD, scores both, returns a verdict with per-metric deltas. Plug it into a pre-commit hook with --fail-on-regression and you stop shipping silent quality drops.

02

"Did this PR diff regress quality?"

# Score a PR diff through a rubric
$ judge score --judge code-quality \
$   --pr acme/api#482 \
$   --target-kind REPO --target-key acme/api

→ fetch PR diff via GitHub
→ submit_score (forced tool-use)

● overall 74.0 / 100   △ +1.2 IMPROVED
  ✓ type_safety      0.90   (+0.10)
  ✓ error_handling   0.75   (+0.15)
  ✗ test_signal      0.40   (·)

rationale (cited):
  login.ts:42 — new guard validates token
  no tests added for the new branch

The GitHub PR source fetches the diff and a rubric scores it — the exact run the GitHub App does on every PR. Overall in 0–100, per-metric deltas vs the target's last score, and a cited rationale you can audit.

03

"Block regressions in CI without writing glue."

# Once: scaffold .judge/config.json + GH Action
$ judge init && judge install gh-actions

# On every PR (CI):
$ judge run --fail-on-regression

→ syncing 3 pipelines from .judge/config.json
✓ pr-diff:code     78.2  IMPROVED  (+2.1)
✓ auth:security    74.0  STABLE    (·)
✗ readme:doc       61.5  REGRESSED (−4.3)

1 regression → exit 1, build fails
PR comment posted with 3 scorecards

Check in .judge/config.json with named pipelines (PR diff, a file, README — anything text). One judge run in CI scores them all, fails the build on any REGRESSED, and the GitHub App posts the scorecards to the PR.

· Same engine via SDK (code-first regression tests), MCP (Claude Code, Cursor) and REST
Drop it into your loop

Three install paths. Same engine.

From zero-effort to full programmatic control. Most teams use all of them: GitHub App on every PR, the CLI / MCP / REST trio for CI and agents, and the TypeScript SDK for code-first regression tests against pinned baselines.

Zero installGitHub App

Score every PR automatically.

Connect once. Every PR that changes a watched repo gets scored against your chosen rubric. A typed score is posted back as a PR comment with cited evidence — block the merge on regression, or just observe.

  • One-click install on any repo or org
  • Typed score commented on the PR with cited evidence
  • Block the merge on regression — or just observe
Recent reviewsauto-refresh
code-qualitysrc/auth.ts
72.4 +4.1
securityPR #482 diff
64.2 -1.8
doc-clarityREADME.md
81.0 +2.3
One binaryCLI · MCP · REST

Wire it into CI, agents, anywhere.

`judge score` in your pipeline. MCP server for Claude Code and other agents to score their own output. REST endpoints if you want to call it from anywhere else. Same scores, same history.

  • CI: fail the deploy on regression with one config file
  • MCP: agents check their work before returning to the user
  • REST: score any artifact from your own scripts or schedulers
claude-code$
In codeTypeScript SDK

Regression tests for code & LLM output.

`compareToBaseline({ … })` — score a candidate against a pinned baseline and fail your test runner on overall or per-metric drop. Median over N runs, server is the source of truth, baselines content-addressed. Optional Vitest helpers (`regressionTest`, `evalTest`) wrap it; works just as well from any test runner or a plain script.

  • Pin a baseline scoreId, fail CI when iteration N+1 regresses
  • Vitest helpers: `regressionTest`, `evalTest` (5-min default timeout)
  • Zero runtime deps; same /mcp endpoint as CLI and agents
// auth.eval.ts
import { regressionTest } from "@inthemeantimeiwatch/judge-sdk/vitest";
import { sources } from "@inthemeantimeiwatch/judge-sdk";
regressionTest({
id: "auth/code-quality",
candidate: {
judge: "code-quality",
target: { kind: "COMPONENT", key: "src/auth.ts" },
source: () => sources.bundle({ paths: ["src/auth.ts"] }),
runs: 3,
},
fail: { onOverallDropBy: 0.3 },
});
✓ pass — Δ +0.4 vs baseline (pinned)
One scoring engine

Typed rubrics. Seeded or your own.

A judge is a rubric: a set of weighted, typed metrics the model fills in under forced tool-use. Start from the 16 seeded system rubrics or generate a custom one. Stack as many as you want — disagreements are signal, not noise.

System

16 system rubrics, seeded

Ready the moment you sign up: code quality, security, performance, test coverage, API design, maintainability, documentation, UX & accessibility, and more. Use as-is or fork one.

code-quality81.0
  • Type safety
  • Test signal

code quality · security · test coverage · doc clarity

Custom

Generate your own from one line

Describe what "good" means for your artifact and Judge drafts a typed rubric — boolean gates plus weighted 0–10 metrics. Edit the spec directly, or hand-author it. Restrict it to the target kinds it's meant for.

security64.2
  • Input validation
  • Authz checks

tone · brevity · factuality · brand fit · spec coverage

How it scores

Typed, weighted, repeatable

Tool-use forces the model to return a JSON-Schema-validated payload — never prose to parse. Each run yields an overall in 0–100, per-metric values, cited rationale, and a deterministic delta vs the previous run.

doc-clarity72.4
  • Onboarding path
  • Accuracy

boolean gates · 0–10 scales · weights · cited evidence

How it works

Output → Judge → Score → Delta → Decision.

  1. 01

    Point it at an output

    A PR diff, a code file, a folder of files, a generated report, a pasted document — anything text-bearing an LLM can read.

  2. 02

    Pick a judge

    Use a seeded system rubric, generate one from a one-line description, or stack several rubrics for cross-checked signal.

  3. 03

    Run it — PR, CI, agent, or hand

    GitHub App scores on every PR. `judge score` runs in CI. MCP tools fire from Claude Code. Click Run in the dashboard. Same engine, same scores.

  4. 04

    See what moved

    Every run is a typed Score row with per-metric deltas. IMPROVED / STABLE / REGRESSED is computed in code, not generated. The GitHub App comments the result on the PR.

Inputs

Point it at code, a diff, or text.

Three source kinds — GitHub PRs, GitHub files & folders, and raw text — fetch the artifact and hand it to the scoring engine. Code, docs, and generated AI output all read the same way. Scoring runs on Claude Sonnet 4.6.

GitHub PR diff
via GitHub App or PAT
GitHub repo file
any branch / tag / SHA
GitHub folder
multi-file, on any ref
Local file bundle
CLI / SDK, with sibling tests
Document
Markdown / text
Generated AI output
paste & score
Raw text
anything you can paste
Shell stdout
via SDK `sources.shell`
Why the score holds up

We publish our noise floor.

Every LLM judge has run-to-run variance. Most tools hide it. We measure it, render it, and refuse to call something IMPROVED unless it's actually above the band.

Same input → same verdict

Scored at temperature 0. Run it twice on identical input, get the same STABLE. Phantom regressions render inside a confidence band — never as a red trend chip.

Schema-validated output

Tool-use forces the LLM to call submit_score with a JSON-Schema-validated payload. No prose to parse, no malformed runs.

Cited evidence

Every metric comment quotes the artifact — the line, the phrase, the file — so you can verify the score didn't make it up.

Versioned judges

Editing a judge mints a new slug. Old scores stay honest; new ones compound on a fresh baseline. No silent recalibration.

Deterministic deltas

IMPROVED / STABLE / REGRESSED is computed in code from typed metric values, not generated by a second LLM call. No drift, no theater.

Open data model

Targets, Judges, Jobs, Runs, Scores, Metrics — everything is a typed Postgres row. Query directly, export, feed pipelines, BYOK or SaaS.

Where Judge sits

Eval tools (Braintrust, Promptfoo, Langfuse) target ML engineers with golden datasets. Code review tools (CodeRabbit, Greptile) target reviewers on diffs. UX tools target marketers on flows. Judge sits in the middle — typed quality scoring for any team that ships a thing, iterates on it, and needs to know whether the last change was a step forward.

Stop shipping changes you can't measure.

Paste an output. Get a score, a delta vs. last run, and a reason. 30 seconds, no card. Free account, all 16 system rubrics seeded — BYOK pays Anthropic at cost (~$0.02/score), no Judge markup. Connect GitHub when you're ready to score every PR.