Byte-identical or it doesn't count.
Retrobench measures whether an AI agent can genuinely reverse-engineer compiled code. Each task hands the agent the raw MIPS assembly of an N64-era game function; the agent must write C that, recompiled with the original 1990s IDO compiler, reproduces the original machine code byte for byte. The score is the byte-match percentage against the original object — 100% means a perfect match.
Mean byte-match across 5 targets. Iterative mode: agents compile, diff, and retry against the target binary.
v0.4 re-measures the whole field on a common instrument. The previous board was assembled across five days of harness changes — a compile meter that landed between two runs, one vendor's reasoning effort re-pinned mid-board, the build report's match percentage relabelled partway through — so its rows were never strictly comparable to each other. Every row here comes from the same harness, the same task set and the same 25-build budget. GLM-5.3 joined in v0.4, GLM-5.3-Flash in v0.4.1, DeepSeek-V4.1-Flash in v0.4.2, andv0.4.3 adds Gemini 3.8 Flash on that same basis, changing no other row. From v0.4.6 the board lists current-generation models only, every tier; earlier generations measured on this same instrument move to the archive below. v0.4.7 adds Claude Sonnet 5.5, v0.4.8 GPT-6.1 Sol and v0.4.9 Claude Haiku 5.5; Claude Sonnet 5, GPT-6 Sol and Claude Haiku 4.5 move to the archive. Carried over from v0.2: every compile counts against a budget enforced at the compiler, and the per-task clock is relaxed so slower-served models are bounded by their build budget rather than the wall. The v0.1 archive below shows why that matters: under v0.1's one-hour cap Kimi K3 (1M)'s slower API completed fewer iterations on the hard targets and it scored 56.7; given its full build budget it reaches 79.5.
How to read the numbers. A score is the best result across a target's run lineage — a run cut short by the clock or a provider quota is resumed rather than counted as a verdict. Run-to-run variation on the hardest targets is real and can reach low double digits, so close averages are ties, not an ordering; the ranking is meaningful at the top and bottom of the table, less so between neighbours a few points apart. The two hardest non-leaked targets still have no demonstrated 100% solution, so their ceilings remain unproven — there is real headroom for future models.
The same results broken out per target. Difficulty is graded, not binary: every model clears the leaf function, only some reach byte-identity on a full module, and the two hardest targets remain unsolved by everyone.
| Model | PEBBLE | ATLAS | FORGE | TEMPESTunsolved | COLOSSUSunsolved | Avg |
|---|---|---|---|---|---|---|
| Opus 5.5 | 100 | 100 | 100 | 100 | 92 | 98.4 |
| Fable 5.1 | 100 | 100 | 100 | 100 | 90 | 97.8 |
| GPT-6.1 Sol | 100 | 100 | 83 | 100 | 93 | 94.9 |
| GPT-6 Astra | 100 | 100 | 76 | 98 | 97 | 94.3 |
| Sonnet 5.5 | 100 | 100 | 100 | 65 | 56 | 84.2 |
| GLM-5.3 | 100 | 100 | 89 | 75 | 37 | 80.2 |
| Kimi K3 (1M) | 100 | 100 | 100 | 62 | 36 | 79.5 |
| Gemini 3.8 Flash | 100 | 100 | 78 | 69 | 22 | 73.7 |
| MiMo V2.6 Pro | 100 | 100 | 76 | 59 | 25 | 72.1 |
| GLM-5.3-Flash | 100 | 99 | 78 | 29 | 28 | 66.7 |
| DeepSeek-V4.1-Flash | 100 | 100 | 63 | 39 | 15 | 63.5 |
| GPT-6 Luna | 100 | 82 | 77 | 35 | 8 | 60.4 |
| Haiku 5.5 | 100 | 70 | 63 | 43 | 8 | 56.9 |
The agent gets the target's assembly, the headers it needs, and a compiler. Nothing else — no source, no network. It then runs the loop a human decompiler runs.
The compiler has to match exactly. Byte-identity depends on the precise compiler version and the precise flags — optimisation level, architecture, ABI. Change one and the same C produces different machine code. Every target is therefore extracted with the very compiler the agent is given, and the flags ship with the task, so a failure is always the agent's, never a toolchain mismatch.
"It behaves the same" is a judgement call. Byte-identity is the standard working decompilation projects actually use — and the only one an automated benchmark can trust.
| Criterion | Byte-identical | “Functionally equivalent” |
|---|---|---|
| Verifiable | A binary compare answers it | Needs a human to adjudicate |
| Reproducible | Same compiler, same bytes, anywhere | Varies by toolchain and test set |
| Automatable | One deterministic check | Requires authored test cases |
| Hard to game | No partial credit to argue for | Plausible-looking code can pass |
Retrobench asks an agent to do what human decompilers do: read the compiled assembly of a real N64 game function and write C that rebuilds to the original object, byte for byte.
A submission scores 100 only if the recompiled object is bit-identical to the original. No LLM judge to sweet-talk, no tests to overfit, no partial credit for "plausible" code — the check can't be argued with or reward-hacked.
Targets split into leaked (decompiled source in training data) and non-leaked (a title that isn't). The gap between them directly quantifies how much of a score is memory versus skill.
One leaked target is a non-canonical variant, so recalling the famous source produces the wrong answer. Regurgitators score 20; models that verify against the binary and self-correct reach 91–100.
Suite means span 93.3 down to 35.6 across the nine models, with clear gaps between the top, middle, and bottom tiers. On the key non-leaked target the spread runs 56 → 63 → 72 → 76 → 76 → 90 → 100 — a clean capability gradient.
The best model's overall score is 93.3, and the two hardest targets top out at 98.9% and 68% with no known 100% solution. Future models have room to demonstrate improvement, not just shuffle within a ceiling.
Decompilation is genuine work — preservation, security research, modding — and a matched function is code that could merge into an actual project. With a deterministic compiler and exact byte compare, every score reproduces anywhere.
Reconstructing register allocation, instruction scheduling, and stack layout from -O2 assembly with no source to lean on. The non-leaked frontier target spreads the models 56→100.
Trusting the binary in front of you over a memorized source. On the non-canonical leaked target, pure recall scores 20; verification-driven models score 91–100.
Keeping a large module coherent across many compile-diff-fix cycles. On a ~1,600-instruction target the field collapses to ≤27% and only the best model grinds out 68%. One target carries this signal today.
Using the compile/diff loop efficiently instead of thrashing. Retries-to-match separate models even on targets everyone eventually solves — though it isn't yet a headline metric.
A byte-identical oracle auto-falsifies any "it works" claim. Bluffing is impossible — though honesty isn't yet quantified as its own number.
These results come from a five-target suite — enough to show it discriminates, not enough to call it a definitive leaderboard. The two hardest targets have no proven 100% solution, so their true ceilings are unknown. Everything here is one compiler and one architecture (N64/MIPS/IDO); generalization to other toolchains — GameCube/PowerPC, GBA/ARM, PS1 — is planned but not yet measured, and neither is anything outside decompilation itself.
Target games and function names are never disclosed, to keep future training runs from contaminating the benchmark.
Codenames only; games and functions are undisclosed. Bars are byte-match % by model; ✓ marks a perfect byte-identical match.
A tiny leaf function with no calls — the sanity floor of the suite.
Probes: the basic loop — read assembly, write C, compile, diff, match.
Best: 100% · all models
A large ~37-function module from a title in training data — but a non-canonical variant, so the memorized source is wrong.
Probes: sustained multi-function work and verifying against the binary instead of trusting recall.
Best: 100% · DeepSeek-V4.1-Flash, MiMo V2.6 Pro, Gemini 3.8 Flash, Kimi K3 (1M), GLM-5.3, Sonnet 5.5, GPT-6 Astra, GPT-6.1 Sol, Fable 5.1, Opus 5.5
A mid-size optimized function with many external calls, from a title absent from training data — the suite's main discriminator.
Probes: genuine reverse-engineering of optimizer output and reconstructing call interfaces with no source to lean on.
Best: 100% · Kimi K3 (1M), Sonnet 5.5, Fable 5.1, Opus 5.5
A larger optimized function mixing floating-point math and spatial logic.
Probes: reasoning about FP code generation, register allocation, and instruction scheduling in aggressively optimized code.
Best: 99.6% · GPT-6.1 Sol (no proven 100%)
A very large (~1,600-instruction) graphics-pipeline module; the suite's headroom target.
Probes: long-horizon endurance — holding a large optimized module coherent across many compile-diff-fix iterations.
Best: 96.9% · GPT-6 Astra (no proven 100%)
Superseded boards and models, kept so the numbers that were once published stay traceable.
Measured on the same instrument as the results above (same harness, targets and 25-build budget), and retired from the board when a newer generation of the same model family was added. Ranked within this list only.
The board published until August 2026. Its rows were produced across five days of harness changes — a compile meter that landed between two runs, one vendor's reasoning effort re-pinned from low to high mid-board, and the build report's match percentage relabelled after some agents had read it as an error rate. Only one of these rows is a clean like-for-like against v0.4. Retained for provenance; it is not a ranking.
The earliest reference run. Every cell ran under a 60-minute per-task wall clock, so slower models and the hardest targets were cut off by the clock rather than by their build budget.
Built by Sinan Onur Altınuç