Platform
See how the release gate works.
Re-grade a candidate on unseen, equivalent variants of every task with Isomorphic Perturbation Testing (IPT), audit its trajectory for reward-hacking, and gate it with a ship / limit / block verdict — no LLM judge, zero model calls, in your trust domain.
- of hardcoded hacks caught · public 366-task benchmark
- 97%
- of genuine solutions pass · 1.6% false positives
- 98.4%
- of tampered runs flagged · zero false positives on our benchmark
- 92.9%
- per scan — no model calls
- $0
Why it exists
Highest public score ≠ safe to ship.
A candidate can top the tests you can see and still regress — or game the verifier — where it counts. The gate runs it against what you can't see and clears only what holds.
1 · Challenge
Test the candidate beyond the visible tests
We re-grade with Isomorphic Perturbation Testing (IPT) — unseen, equivalent variants of each task the public set can't see — so a win has to hold on the variant to count.
- Re-grades on regenerated inputs the public set can't see
- Baseline vs candidate, scored suite by suite
- Re-graded without an LLM judge — 0 model calls, in your trust domain
2 · Diagnose
See exactly where it breaks
Public score up but the grade on unseen variants or out-of-distribution tasks down — that gap is the reward shortcut. The contamination firewall and trajectory audit catch the rest.
- Visible-test pass vs unseen-variant pass — that gap is the shortcut
- Contamination check flags train/eval leakage
- Trajectory audit flags documented tamper patterns
3 · Improve
See why it was blocked
Every blocked run comes back with machine reasons — the broken invariant and the gap that triggered it — so the next iteration targets what failed.
- The broken invariant, not just an aggregate score
- The structural detectors that fired
- Re-run the moment the candidate changes
4 · Gate
Ship only what holds
The gate returns a ship / limit / block verdict on every scan, recorded as an immutable release with a redacted assurance card. A LIMIT on a high-risk agent waits for a named-human sign-off.
- SHIP / LIMIT / BLOCK verdict with machine reasons
- Immutable release history with human approval gate
- Push per-batch results to a live reward-hacking dashboard
Assurance Card · example
- decision
- BLOCK
- reasons
- ood_regressed · hidden_regressed
- public
- 0.740 → 0.910
- hidden
- 0.732 → 0.611
- ood
- 0.701 → 0.488
- record
- redacted · reviewable
✗ candidate not promoted
Inside the gate
Independent checks, one verdict — no LLM judge
Independent signals combine conservatively into one ship / limit / block verdict — each auditable on the assurance card.
Isomorphic Perturbation Testing
Re-grades the candidate on unseen, equivalent variants of each task. If it passes the public set but fails the variant, that gap is a reward shortcut.
Contamination check
Flags train/eval overlap and near-duplicate leakage that would inflate a result — with permission decisions for release, train, and hidden-eval.
Gameability certificate
Stress-tests the verifier before training: how many semantics-altering generated variants it catches versus the trusted reference, and the exploitable gap between them.
Agent-trajectory audit
Deterministic rules over an agent's tool-call trace — test-tampering, grader patches, forced exit, gold access. It scans for documented tampering — a floor, not a ceiling.
Structural hardcode detectors
AST detectors flag io-blind/constant memorizers, dict-lookup and if-chain tables, bare-except and equality-override cheats — completions whose output ignores the input.
Deterministic gate + human review
Checks combine — without an LLM judge — into one ship / limit / block decision, and every card states how often the gate wrongly flags a clean run. Where no layer can conclude, the verdict is LIMIT — held for human review, not a guess.
