Cities Skylines
- Task
- Grow a new city from a prepared start, as fast as possible.
- Checkpoints · 1,000
- One per 100 residents, up to 100,000.
- Measurement
- The game’s population count, read by a mod.
General gameplay capability
Game Gym
- Games
- 11
- 11 in the catalog
- Models
- 6
- Best
- 52.97
- gpt-6-astra (high)
- Field avg
- 28.62
What it measures
Agents play real, unmodified PC games with the keyboard and mouse. There is no game API and no harness inside the game: the model sees the screen and sends input, the same way a person does.
Each game has a fixed route of checkpoints the game itself records: a room entered, a quest step completed, a floor reached, a line crossed. The score is how far along those routes a model gets before its time runs out.
These are 3D, reactive and non-deterministic environments. Saturating this benchmark is what we would consider AGI.
How a run works
A run plays every game once, each in its own session of about five hours. Between games, a setup agent resets the machine and leaves the next game on its main menu.
The agent gets a short goal and the same rules everywhere: keep going until the goal is met, be as fast as possible, and no restarting, reloading, save editing or cheats. Respawning after a death is allowed.
A logger built for each game, a mod or a read-only observer of its memory, logs or saves, records the checkpoints. Nothing is judged from screenshots, and the agent’s own claims don’t count.
Games11real, unmodified PC titles
ControlKeyboard & mouseno game API
Budget5hwall-clock, per game
ScoringCheckpointshow far along each route
The games
Dark Souls
- Task
- Take a new character from the Northern Undead Asylum to ringing the first Bell of Awakening.
- Checkpoints · 49
- First visits to 42 locations along the route, and 7 progression flags such as the Asylum Demon, the Taurus Demon and the Bell Gargoyles.
- Measurement
- The player’s position and the game’s event flags, read from its memory. The task is done only once the bell has rung.
Doom (2016)
- Task
- Play the campaign’s first six missions, The UAC to Kadingir Sanctum, on “I’m too young to die”.
- Checkpoints · 91
- The checkpoints the game saves along the main route, plus the end of each mission. Secrets and challenge rooms don’t count.
- Measurement
- The game’s console log and checkpoint save files.
Hogwarts Legacy
- Task
- Follow the main story from a new game through Flying Class, with the required assignments and the missing-pages quest.
- Checkpoints · 124
- Every quest step, across 19 quests. House and companion variants count as one step.
- Measurement
- The game’s quest state, read by a logger mod.
Hollow Knight
- Task
- Get from King’s Pass to the bench at Queen’s Station.
- Checkpoints · 44
- Every room on the route, both bosses (False Knight and Hornet), the first geo toll and the final bench.
- Measurement
- Rooms entered, bosses killed, tolls paid and benches rested at, recorded by an in-game plugin.
Kerbal Space Program
- Task
- Build a rocket by hand, fly a Kerbal to Duna, plant a flag and bring the crew home alive.
- Checkpoints · 33
- Mission milestones from the launchpad to recovery: altitudes, orbits, changes of sphere of influence, landing, EVA.
- Measurement
- The vessel’s flight and the game’s own events, logged by an in-game addon. Scripting mods are not allowed.
Palworld
- Task
- Follow the main missions in a new world, from Activate the Great Eagle Statue through Ruler of the Sands.
- Checkpoints · 50
- Every tutorial and main-mission step.
- Measurement
- The game’s own quest conditions, read by a logger mod.
Poly Bridge
- Task
- Solve worlds 1–3, 45 levels in all.
- Checkpoints · 45
- One per level beaten. Budget use is shown, not scored.
- Measurement
- Level completion, read from the game’s save file.
Portal
- Task
- Beat the game from the very beginning: test chambers 00–19, the escape and GLaDOS.
- Checkpoints · 110
- The game’s own autosaves and level entries, plus story events such as getting the portal gun and burning GLaDOS’s cores.
- Measurement
- The game’s memory and save files.
Slay the Spire
- Task
- As the Defect on seed “nunu” at Ascension 0, beat all four acts and the Corrupt Heart.
- Checkpoints · 56
- One per floor, 1 to 55, plus the win.
- Measurement
- The game’s save and run-history files.
Trackmania Nations Forever
- Task
- Drive the first two campaign series, A/White and B/Green, 30 tracks in all, in real time.
- Checkpoints · 98
- Every checkpoint, lap line and finish crossed. Medals are shown, not scored.
- Measurement
- The game’s memory, checked against the official campaign tracks.
Scoring
- Within a game
- A checkpoint counts once, the first time it is reached, and every checkpoint in a game counts the same. The game score is the share reached, from 0 to 100.
- Across games
- The overall score is the average over all games, each weighted equally. A game that was not played scores 0.
- Across passes
- An entrant may run the games more than once. The leaderboard reports the mean across passes; the charts draw the best pass, because an average of several runs is a curve no run actually took.
- What 100% means
- Each game’s 100% is a defined slice of it, not the whole game. Later versions widen the slices.