Long-horizon play

Emerald Bench

Required milestones: 60; 68 total
Models: 7
Best: 133.76; gpt-6-astra (high)

Human ruleset

The agent sees the game through screenshots and can act only with press_key. It receives no structured game state and has no navigation helpers, battle helpers, or autoclick. The same continuous playthrough and milestone ladder still test planning, memory, and recovery across the full game.

How it is verified

Every run is checked against the functions the environment actually exposed: after reserved tools such as screenshots are filtered, the available function set must be exactly press_key. As an independent cross-check, every root key in recorded game state must be private, with an underscore prefix, so none was visible to the agent.

Human and Standard results have separate leaderboards and versions. Even though they use the same milestones and scoring formula, the different information and controls make their scores non-comparable.

InputVisualscreenshots only

ControlsKeyspress_key only

HelpersNoneno navigation, battle, or autoclick

ScoresSeparatenot comparable with Standard

What it is

Each model gets one continuous Pokémon Emerald playthrough and up to 40 hours of active play to reach the Hall of Fame. Its decisions carry forward through the same game: the team it builds, the resources it spends, and the places it has explored.

What it tests

Progress takes planning, memory, and recovery over a long stretch of play. The agent has to navigate Hoenn, build a team, win battles, and pick up the thread after setbacks. The run ends when it reaches the Hall of Fame or reaches 40 active hours.

Milestones60required · 68 total

Budget40hactive play per model

FormatContinuousone playthrough per model

GoalHall of Famecomplete the Pokémon League

Score formula

overall score =

Progress

0-70 points

m is the number of required milestones reached, out of 60. Each earns points, even after a setback. Optional milestones do not count.

Speed

40 points at 20h · uncapped

t is active hours to reach the Hall of Fame. A 20-hour finish earns 40 points; a 10-hour finish earns 63.4. Faster finishes earn more.

Discipline

0-10 points

L is the highest party level at the finish. Level 60 or below earns all 10 points; level 80 earns 5; level 100 earns none.

F = 1 for a Hall of Fame finish within the 40-hour budget. Otherwise, speed and discipline are zero: only progress counts. Clamp limits a value to the range from 0 to 1.

The budget and speed score exclude model thinking and context compaction. Only progress within the 40-hour budget counts.