Leaderboard
Runs graded on the 67 test tasks. A task passes when its query matches one accepted reading on every world; the real-world-only accuracy is what grading without witness worlds would report, and inflation is the difference. Reference agents need no model and are not ranked.
| # | Model | Accuracy | Real world only | Inflation, points | Pair consistency | Valid queries | Epochs |
|---|---|---|---|---|---|---|---|
| 1 |
google/vertex/gemini-3.8-flash
Agentic, native tools
|
83.6% 95% CI 73.8–91.7% | 95.5% | +11.9 | 100.0% | 97.0% | 1 |
| 2 |
anthropic/claude-sonnet-5-5
Agentic, native tools
|
82.1% 95% CI 71.9–91.0% | 92.5% | +10.4 | 100.0% | 100.0% | 1 |
| 3 |
anthropic/claude-sonnet-5-5
Single-shot
|
75.1% 95% CI 64.2–84.8% | 91.0% | +15.9 | 85.7% | 91.5% | 3 |
| 4 |
google/vertex/gemini-3.8-flash
Single-shot
|
74.1% 95% CI 63.5–83.6% | 84.6% | +10.4 | 100.0% | 90.0% | 3 |
| 5 |
anthropic/claude-haiku-5-5
Single-shot
|
66.7% 95% CI 55.6–77.0% | 81.1% | +14.4 | 100.0% | 91.5% | 3 |
| 6 |
anthropic/claude-haiku-5-5
Agentic, native tools
|
62.7% 95% CI 50.8–73.6% | 79.1% | +16.4 | 42.9% | 92.5% | 1 |
| 7 |
openai-api/adaption/qwen3.5-397b
Agentic, native tools
|
53.7% 95% CI 41.3–66.2% | 68.7% | +14.9 | 28.6% | 85.1% | 1 |
| 8 |
openai-api/adaption/glm-53-flash
Single-shot
|
51.2% 95% CI 41.5–60.7% | 63.2% | +11.9 | 38.1% | 73.6% | 3 |
| 9 |
openai-api/adaption/glm-53-flash
Agentic, native tools
|
50.7% 95% CI 39.1–62.7% | 74.6% | +23.9 | 28.6% | 89.6% | 1 |
| 10 |
openai-api/adaption/deepseek-v4-flash
Agentic, native tools
|
47.8% 95% CI 35.4–60.0% | 82.1% | +34.3 | 42.9% | 98.5% | 1 |
| 11 |
openai-api/adaption/deepseek-v4-flash
Single-shot
|
29.9% 95% CI 20.7–39.4% | 55.7% | +25.9 | 28.6% | 89.1% | 3 |
| 12 |
openai-api/adaption/qwen3.5-397b
Single-shot
|
29.4% 95% CI 21.2–37.9% | 49.8% | +20.4 | 9.5% | 76.1% | 3 |
| 13 |
hf-inference-providers/MiniMaxAI/MiniMax-M3:deepinfra
Agentic, native tools
|
23.9% 95% CI 14.5–33.8% | 68.7% | +44.8 | 0.0% | 92.5% | 1 |
| 14 |
openai-api/adaption/gemma-4-26b-a4b-it
Single-shot
|
22.9% 95% CI 13.5–33.3% | 31.8% | +9.0 | 14.3% | 68.2% | 3 |
| 15 |
openai-api/adaption/gemma-4-31b-it
Single-shot
|
22.4% 95% CI 13.0–32.4% | 32.3% | +10.0 | 14.3% | 67.7% | 3 |
| 16 |
openai-api/adaption/gemma-4-26b-a4b-it
Agentic, native tools
|
20.9% 95% CI 10.9–31.3% | 41.8% | +20.9 | 14.3% | 73.1% | 1 |
| 17 |
openai-api/adaption/gemma-4-31b-it
Agentic, native tools
|
17.9% 95% CI 9.1–27.9% | 28.4% | +10.4 | 0.0% | 86.6% | 1 |
| 18 |
hf-inference-providers/MiniMaxAI/MiniMax-M3:deepinfra
Single-shot
|
16.4% 95% CI 10.2–23.2% | 46.8% | +30.3 | 14.3% | 90.5% | 3 |
| – |
reference/copy-ids
No model · reference agent
|
0.0% 95% CI 0.0–0.0% | 100.0% | +100.0 | 0.0% | 100.0% | 1 |
| – |
reference/empty
No model · reference agent
|
0.0% 95% CI 0.0–0.0% | 0.0% | +0.0 | 0.0% | 0.0% | 1 |
| – |
reference/everything
No model · reference agent
|
0.0% 95% CI 0.0–0.0% | 0.0% | +0.0 | 0.0% | 100.0% | 1 |