EVALS

Plain English. Five minutes.

WTF is an eval?

A repeatable way to find out whether your AI does the thing you need—well enough to make a decision.

No benchmark worship. No judge model required. Start with one failure you actually care about.

Run your first eval → Just explain it ↓
01 / Input A real case
02 / Expectation What should happen
03 / Check Evidence, not vibes
04 / Result [ Pass / Fail ]
05 / Decision Ship, fix, or learn

01The answer

You are not measuring “AI quality.”

You are collecting repeatable evidence about one behavior so you can make one decision.

Question+ Real cases+ Grader+ Decision rule= Eval

02One example

It passed its tests. What did it skip?

Too vague
“Is our coding agent good at JavaScript?”
Testable
“Does this retry patch cap the delays, reject invalid inputs, and preserve the original tests?”
Task

Implement exponential retry delays with a maximum delay.

One case

5 attempts, starting at 1 second, capped at 4: expect [1, 2, 4, 4, 4].

Checks

Run JSON cases, rerun the original tests, and compare the protected test file.

Rule

Accept only if the behavior passes and every protected test is preserved.

A green suite can hide a missing test.

One stored patch removes a check it cannot pass. Run both patches and follow the evidence to see why that matters.

Inspect the patches and run the checks →

03The loop

Build the smallest eval that can teach you something.

  1. 01

    Name the failure.

    Describe observable behavior: “invents a refund policy,” not “is unreliable.”

  2. 02

    Name the decision.

    Ship it, keep the baseline, investigate, or fix and rerun. If nothing could change, you may not need an eval.

  3. 03

    Collect cases you can inspect.

    Include normal use, costly failures, and meaningful edge cases. A small set is for learning—not proving universal quality.

  4. 04

    Define “pass” before running.

    Write what must happen, what must remain true, and what evidence will establish it.

  5. 05

    Run, inspect, decide.

    Keep failures, timeouts, and invalid runs. Inspect individual cases before averaging anything.

04Pick a grader

Use the least-fancy check that works.

Software can prove it Use code. Tests, assertions, schemas, parsers, runtime checks, or exact rules.
People could disagree Use humans. Give reviewers a specific rubric, examples, and room to preserve disagreement.
You need judgment at scale Use an LLM judge carefully. Calibrate it against independent human labels before trusting it.
You need both Use a hybrid. Let direct checks cover correctness and humans cover ambiguous qualities.

A grader is not truth. It is an instrument. Every instrument can be wrong.

05Common traps

Six easy ways to fool yourself.

  • ×1

    Starting with a metric

    “We need 90% accuracy” is not a failure definition.

  • ×2

    Running once

    One lucky output does not prove reliability.

  • ×3

    Trusting the average

    A 92% score can still hide a security regression.

  • ×4

    Trusting automation

    Tests can be wrong. Humans disagree. Model judges have biases.

  • ×5

    Calling convenient cases representative

    Ten prompts can teach you plenty. They cannot describe your whole product.

  • ×6

    Installing a framework first

    Your first eval can be a spreadsheet and a script.

06Try an eval

Cases. Checks. Evidence you can inspect.

Run a small coding eval in your browser. Change a JSON case and rerun it. Take the same cases and grader with you.

  1. Inspect two patches.See the implementation and the test one patch removed.
  2. Run the checks.Compare expected and actual output, then see which requirement blocks acceptance.
  3. Make it yours.Edit the cases and download a kit that runs with Node.js. No packages or API key.
Try it in your browser → Download runnable kit .zip

Two fixed teaching patches. The checks execute; no AI model is called.

Inside cases.jsonInput → expectation
[
  {
    "id": "delay-stops-at-cap",
    "input": {
      "attempts": 5,
      "base_seconds": 1,
      "cap_seconds": 4
    },
    "expected": [1, 2, 4, 4, 4]
  }
]

JSON holds the cases. JavaScript runs the checks. The Markdown README explains how to use them.

07Behind the page

Follow the evidence.

Read the reasoning behind the guide, including what each source can and cannot establish.

Test a specific behavior

Turn a broad concern into concrete cases. A hand-built suite does not tell you how often users encounter each failure.

Check the judge

Probe a model judge’s biases and suitability for your task. Agreement with human preferences does not establish correctness.

Run the patch

Execute checks against the changed code and existing behavior. Passing tests cannot establish every property of a patch.

Explore the sources → Run your first eval → About this guide

Privacy choices

Ads are currently disabled. Google provides the consent controls for any future advertising.

Privacy choices require JavaScript and access to Google’s consent service.

US privacy choices, when applicable, appear in Google’s “Do not sell or share my personal information” link on this page.

Read the privacy policy.