- Levels
- 49
- Models
- 21
- Best
- 81.3%
- gpt-6-astra
- Field avg
- 58.5%
A suite of 49 individual phone-use levels across seven categories: Primitives, Vision, Memory, IQ, Tempospatial, Games, and Tasks. Each level is inspired by a real failure we encountered while running agents in production.
A gesture that kept missing, a game the agent played badly, a task it could not finish twice in a row. Turning those into levels gives us fast, repeatable readings on how models and harness changes hold up against the work we actually do.
Every run uses the nunu.ai harness: an agent loop wrapped around a code-execution sandbox that can capture screenshots and interact with the device. The agent takes screenshots, writes code to analyze what it sees, and then acts on the device through taps and swipes.
There is no structured state, no accessibility IDs and no text description of the screen at any point. Many games expose no accessibility tree at all, so the harness does not rely on one anywhere — which also means models are tested the way a person actually operates a phone, with no dev hooks to fall back on.
Primitives
Touchscreen control and fine motor accuracy — taps, swipes, complex gestures.
Vision
Reading the screen: counting, matching, visual search.
Memory
Spotting information that matters and recalling it across a long task.
IQ
Reasoning and rule induction, mostly puzzles with no instructions.
Tempospatial
Temporal and spatial reasoning under a clock.
Games
Multi-step games that need a strategy, not just a correct answer.
Tasks
Real workflows: driving phone UI, following multi-step instructions.
Games
Scored on in-game performance over a fixed time. The 50% baseline is a human playing the same game for the same length of time.
Vision, memory, IQ, primitives
Straight accuracy. Did you count the right number of objects? Did you tap the right target? Black and white.
Tasks
Scored programmatically on how much of the workflow completed, with penalties for specific mistakes.
Every level returns a score between 0 and 100%. What that number means depends on the level, which is the point — one figure per level, comparable across models and across changes to our own harness. A level is scored on the agent's best attempt within the run.