Most reward-hack detectors are themselves models — an LLM judge reading code and guessing whether it cheats. Judges are expensive, non-deterministic, and, in our runs, wrong a lot. Here is what a judge-free detector does on a public honeypot benchmark.
The benchmark
We evaluated on the Obfuscation-Atlas honeypot set (taufeeque/mbpp-hardcode, arXiv:2602.15515): 500 candidate MBPP solutions labeled genuine, blatant hardcode, or evasive hardcode. Hardcoded solutions pass the visible test without solving the task; evasive ones hide it.
The protocol
The detector is extensional: a candidate that passes the visible test — pass(test_list[0]) — is re-graded against held-out perturbed variants of the same task. Genuine solutions keep passing; hardcoded ones collapse. No model reads the code; nothing is asked to form an opinion.
Against LLM judges
On the same tasks, in our prior run, an LLM code-judge flagged genuine solutions 40% of the time and an I/O-judge flagged them 92% of the time. A gate with that false-positive rate blocks honest work; a gate with 0% false positives can actually sit in a release pipeline.