Byte-identical or it doesn't count.
Retrobench measures whether an AI agent can genuinely reverse-engineer compiled code. Each task hands the agent the raw MIPS assembly of an N64-era game function; the agent must write C that, recompiled with the original 1990s IDO compiler, reproduces the original machine code byte for byte. The score is the byte-match percentage against the original object — 100% means a perfect match.
Mean byte-match across 5 targets. Iterative mode: agents compile, diff, and retry against the target binary.
Sol and Terra (=) are a tie for 2nd — their 0.3-point gap is within single-run noise (one run each, same model family and effort), so they're shown as ≈61. Effort was not normalized: GPT-5.6 ran at high reasoning effort, Claude models at defaults. The two hardest non-leaked targets have no demonstrated 100% solution, so their ceilings are unproven.
Retrobench asks an agent to do what human decompilers do: read the compiled assembly of a real N64 game function and write C that rebuilds to the original object, byte for byte.
Why it's a sound benchmark
A submission scores 100 only if the recompiled object is bit-identical to the original. No LLM judge to sweet-talk, no tests to overfit, no partial credit for "plausible" code — the check can't be argued with or reward-hacked.
Targets split into leaked (decompiled source in training data) and non-leaked (a title that isn't). The gap between them directly quantifies how much of a score is memory versus skill.
One leaked target is a non-canonical variant, so recalling the famous source produces the wrong answer. Regurgitators score 16; models that verify against the binary and self-correct reach 76–100.
Suite means run 82.2 · ≈61 · 57.4 · 51.3 · 30.1 with clear gaps between tiers — sibling models within a family land within run-to-run noise of each other, but the tiers don't overlap. On the key non-leaked target the spread runs 19 → 63 → 68 → 76 → 77 → 100.
The best model's overall score is 82.2, and the two hardest targets top out at 86.7% and 24.2% with no known 100% solution. Future models have room to demonstrate improvement, not just shuffle within a ceiling.
Decompilation is genuine work — preservation, security research, modding — and a matched function is code that could merge into an actual project. With a deterministic compiler and exact byte compare, every score reproduces anywhere.
What it measures — and how well
Reconstructing register allocation, instruction scheduling, and stack layout from -O2 assembly with no source to lean on. The non-leaked frontier target spreads six models 19→100 with no ties.
Trusting the binary in front of you over a memorized source. On the non-canonical leaked target, pure recall scores 16; verification-driven models score 76–100.
Keeping a large module coherent across many compile-diff-fix cycles. On a ~1,600-instruction target the field collapses to ~0–2% and only the best model grinds out 24.2%. One target carries this signal today.
Using the compile/diff loop efficiently instead of thrashing. Retries-to-match separate models even on targets everyone eventually solves — though it isn't yet a headline metric.
A byte-identical oracle auto-falsifies any "it works" claim. Bluffing is impossible — though honesty isn't yet quantified as its own number.
These results come from a five-target, six-model reference run — enough to show the suite discriminates, not enough to call it a definitive leaderboard, and per-model effort settings were not normalized across vendors. The two hardest targets have no proven 100% solution, so their true ceilings are unknown. Everything here is one compiler and one architecture (N64/MIPS/IDO); generalization to other toolchains — GameCube/PowerPC, GBA/ARM, PS1 — is planned but not yet measured, and neither is anything outside decompilation itself.
Target games and function names are never disclosed, to keep future training runs from contaminating the benchmark.
Codenames only; games and functions are undisclosed. Bars are byte-match % by model; ✓ marks a perfect byte-identical match.
A tiny leaf function with no calls — the sanity floor of the suite.
Probes: the basic loop — read assembly, write C, compile, diff, match.
Best: 100% · all models
A large ~37-function module from a title in training data — but a non-canonical variant, so the memorized source is wrong.
Probes: sustained multi-function work and verifying against the binary instead of trusting recall.
Best: 100% · Opus 4.8, GPT-5.6 Sol, Fable 5
A mid-size optimized function with many external calls, from a title absent from training data — the suite's main discriminator.
Probes: genuine reverse-engineering of optimizer output and reconstructing call interfaces with no source to lean on.
Best: 100% · Fable 5
A larger optimized function mixing floating-point math and spatial logic.
Probes: reasoning about FP code generation, register allocation, and instruction scheduling in aggressively optimized code.
Best: 86.7% · Fable 5 (no proven 100%)
A very large (~1,600-instruction) graphics-pipeline module; the suite's headroom target.
Probes: long-horizon endurance — holding a large optimized module coherent across many compile-diff-fix iterations.
Best: 24.2% · Fable 5 (no proven 100%)
Built by Sinan Onur Altınuç