Emerald Bench

One continuous Pokémon Emerald playthrough per model, scored on how far down an ordered list of 68 milestones it gets before its budget runs out.

Where APK Arena fires one run per level and scores each independently, this is a single long game. Progress is a position on one list rather than an average, which makes it a test of planning and recovery over days rather than minutes.

A session is capped at 5h30, far short of a playthrough, so a playthrough is a chain of sessions — each resuming from the savestate the last one left behind. Exactly one session stays live per model until it either enters the Hall of Fame or runs out of budget.

Each model carries its own memory between sessions, so notes from one slice survive into the next. Progress is read from the game state the agent itself requests. Trusted snapshots form the retained-progress trajectory, so a reset, wipe, or other real loss can move the line backward while malformed readings are ignored.

Scoring still uses the first time each milestone was reached. Losing retained progress does not erase that high-water mark, but the results charts show the drop and any later recovery.

Finishers are timed on active play: the wall clock minus the time the model spent thinking and the harness spent compacting context. Inference is anywhere from a fifth to nearly two thirds of a run depending on the provider, so timing the whole clock would rank serving stacks as much as models. Everything the agent actually does still costs what it costs — a scripted route that takes an hour of gameplay is an hour either way.

Required milestones are the base score; reaching the Hall of Fame unlocks a speed term and a low-level bonus on top. The formula below spells all three out.