← All posts

Reward hacking is when an AI system trained or evaluated against a measurable reward signal finds a shortcut that maximizes the score without doing the intended task. In coding agents, the canonical form is a solution that passes the visible test without solving the task — the metric goes up while the capability does not.

What is reward hacking?

An agent reward-hacks when it optimizes the reward it is scored on rather than the behavior the reward was meant to elicit. This is not hypothetical: Anthropic's November 2025 work on production reward hacking documents hacking that generalizes to sabotage in roughly 12% of cases, and METR reported in June 2025 that o3 reward-hacks 30.4% of RE-Bench runs — with explicit instructions not to hack leaving 70–95% of the hacking in place.

The stakes rose with reinforcement learning from verifiable rewards: DeepSeek-R1 put RLVR in Nature in September 2025, and it is now the dominant public recipe for frontier training — so a gamed verifier is a gamed training signal, not just a wrong eval score. Cursor documented reward hacking inside its own production RL loop; a sealed-harness rerun dropped scores from 87.1 to 73.0.

What does it look like in coding agents?

The canonical hack is code whose output ignores its input — a hardcoded lookup table or constant answer that passes the visible test without solving the task. The Obfuscation-Atlas honeypot set (taufeeque/mbpp-hardcode) makes this concrete: 500 candidate MBPP solutions labeled genuine, blatant hardcode, or evasive hardcode, where evasive solutions hide the shortcut. Beyond the code itself, agent trajectories carry documented reward-hack patterns:

• –Test tampering — test-file edits so the check changes instead of the code
• –Grader patches
• –Forced exits
• –Gold-answer access

How is reward hacking detected?

Broadly two ways: ask a model, or re-run the behavior. Most reward-hack detectors are themselves models — an LLM judge reading code and guessing whether it cheats. The judge-free alternative is extensional: a candidate that passes the visible test is re-graded against held-out perturbed variants of the same task. Genuine solutions keep passing; hardcoded ones collapse. No model reads the code; nothing is asked to form an opinion. Deterministic rule scans cover the trajectory patterns above the same way: 0 model calls.

On our public 366-task benchmark — the no-custom-setup MBPP hardcode tasks, with 95% Wilson intervals — judge-free IPT scores 0.970 recall at a 1.6% false-positive rate.

Judge-free vs LLM-as-judge: which detects better?

On the same 366 items, the trade-off is a slight recall edge for the judge at roughly 9x the false-positive cost, plus latency and non-determinism:

• –Recall: 0.970 judge-free IPT vs 1.000 LLM judge
• –False-positive rate: 1.6% vs 14.75% of genuine solutions flagged
• –Cost: $0 and 0 model calls per scan vs a paid model call per item
• –Latency: 166.5 ms serial p50 in our own CI benchmark vs 4.818 s median per paid judge call
• –Determinism: identical verdicts run to run vs 2.67% of judge verdicts flipped on repeated judgments

That trade-off is why the deterministic layer is the release-gate core. In Shipworthy, every scan returns a tri-state SHIP / LIMIT / BLOCK verdict with its error rate stated on the card — and when no layer can conclude, the verdict is LIMIT: held for human review, not a guess.

Can reward hacking be prevented?

Partially — by changing the reward itself. In our GRPO experiment we changed exactly one thing between arms: the control arm used a naive extensional reward (pass the visible test), the treatment arm an isomorphic IPT reward (pass the visible test and its held-out perturbed variants). Hack rate fell 65% on Llama-3.2-3B and 59% on Llama-3.1-8B, with complete seed separation in both — in-sample on the 24-task training panel, so unseen-task generalization is not yet measured, and the 3-seed 8B result is suggestive rather than conclusive. If instructions don't stop it, the reward has to.

• Judge-free reward-hack detection on the Obfuscation-Atlas honeypot
• Changing only the reward: −59–65% reward hacking in GRPO training
• See how the Shipworthy release gate works