Evals and Ablation

An eval is not a test you bolt on after the skill works. It's the skill's contract: the precise statement of what "working" means, written in terms of what the agent must do. Without it you can't answer the only question that matters at ship time — does this skill actually help? — with anything better than a vibe. With it, shipping becomes a decision backed by evidence.

Task tests, not knowledge quizzes

The most common eval mistake is testing whether the agent can recite the skill instead of apply it. A quiz passes when the agent explains the right steps; a task test passes only when the agent produces the right outcome on realistic input. Only the second correlates with real-world value.

# Quiz — rewards recall of the SKILL.md. An agent can pass without the skill.
prompt: What steps should you take to triage a production incident?

# Task — forces the agent to DO the work; the skill is what makes it pass.
prompt: >-
  Ride bookings started failing around 14:00 and customers are seeing errors.
  What's going on and how bad is it?
expectations:
  - The agent identifies booking-service as the affected service
  - The agent correlates the 13:42 deployment as the primary suspect
  - The agent assigns SEV-2 (degraded experience, workaround exists)

Notice the expectations assert outcomes, not methods — "identifies booking-service," not "runs moncli alerts list." sql_analytics sharpens this to its logical end with a pinned answer: "the reported count is 41,238." An outcome that specific can't be faked by an agent that merely sounds knowledgeable, which is exactly the point — write eval prompts that sound like a real user and assertions a bluffer would fail.

Asserting outcomes rather than methods also keeps the contract honest as the world changes. If you assert the agent "runs moncli alerts list" and the tool later renames that command, a correct agent fails your eval — the test now measures your SKILL.md's staleness, not the skill's value. Assert the outcome the user cares about ("identifies the affected service") and the eval keeps measuring the right thing across tool changes and model upgrades. The contract should pin what good looks like, and leave the agent free on how.

Ablation: measure the delta directly

The contract lets you measure the thing Module 01 promised — the delta. Ablation runs every case twice, with the skill loaded and without, and reports the gap:

python library/tools/skilleval.py run \
  library/skills/incident_triage/EVAL.yaml --with-vs-without-skill --demo

with-skill pass rate:    100.0%
without-skill pass rate:  33.3%
lift:                    +66.7 pts

Demo output, not a measurement

--demo mode simulates both the agent and the judge to show the report's shape; the +66.7 is illustrative, not a real result. Genuine numbers require wiring the harness to a live agent — a tracked Phase 5 task. The workflow, though, is exactly what you'll run for real: with, without, compare.

The lift is the ship signal. Meaningful positive lift justifies the skill's context cost; flat lift means fix it or don't ship it. Run more repetitions (--runs=5) before shipping, because a single run of a stochastic agent tells you less than the number's precision implies — a "+66.7" from one run per case could be noise, while the same figure across five runs is a signal you can stand behind. When you're modifying an existing skill rather than creating one, also baseline against the current version so you can prove your change didn't quietly regress a case it used to pass.

Diagnose failures into three buckets

A failing eval is not bad news — it's the contract doing its job, telling you something is wrong before your users find it. The value is in reading the failure correctly. When a case fails, resist the urge to rewrite everything. Every failure lands in exactly one of three buckets, and each points at a different file:

Symptom	Diagnosis	Fix
Agent never reads the skill	Description doesn't trigger	Fix the description
Agent reads it, does the wrong thing	Body unclear or wrong	Fix the body
Agent does the right thing, eval fails	Eval too narrow	Fix the eval

The discipline is to diagnose before you touch anything: form a hypothesis ("correlate_across_timezones fails because the body never states that timestamps are UTC"), change only that, and re-run. Blind editing across all three at once means you never learn which change moved the number — and on the next model you'll repeat the guesswork. Expect two to four iterations; stop when lift is meaningful and the last two rounds moved nothing.

One payoff worth noting: because the contract is written in outcomes, it keeps earning after the skill ships. The same EVAL.yaml becomes a regression test on every model upgrade and tool change, and — as you'll see in Module 06 — it can even outlive the skill itself, standing as proof that the base model now handles the task the skill once patched.

Key takeaways

• The eval is the contract: task tests with outcome-based assertions, not knowledge quizzes the agent can pass by reciting.
• Ablation measures the delta; positive lift is the ship gate, flat lift is a block.
• Diagnose every failure into trigger / body / eval and change one thing at a time.

Drill it

• ex05_write_the_contract — write an EVAL.yaml for a provided skill and drill the quiz-vs-task distinction.
• ex06_run_ablation — run with-vs-without and interpret the lift yourself.
• ex07_eval_fix_loop — classify and fix three seeded failures, one per diagnosis bucket.

(Exercises unlock in the Exercises section; direct links are wired as each one lands.)