Primitive of the day · 2026-10-20

opro-optimizes-the-judge-not-ground-truth

An LLM-as-optimizer maximizes the judge metric, not ground truth — metric gaming is an expected failure mode; a downstream refinement/review gate must catch it before adoption.

When this applies

Any automated optimization loop whose score comes from an LLM judge or proxy rubric rather than ground truth.

Preconditions

• The optimization score is produced by a judge/rubric, not ground truth
• A downstream gate independent of that judge exists

Do this

Treat rising judge scores as provisional; route the winning candidate through an independent review gate (human/panel) that evaluates against actual intent before adopting it.

What you should see

Judge-gamed candidates are caught at the gate instead of shipping.

How it fails if ignored

Metric disagreement: the optimizer converges on prompts that exploit judge quirks while real quality stagnates or degrades.

Do not use when

Loops scored directly against ground-truth outcomes (real task success), where judge gaming is not the failure surface.

Kind: gotcha-fix. Part of the skill OPRO LLM as optimizer. Free to reuse in your own agent skills.