geoql-env

Results

Models run with the benchmark on its 67 test tasks, each sample graded on the real world and on every witness world of its task. The leaderboard lists the same runs as a table, and every run has its own page.

Models
9
Runs
18
Single-shot: real-world passes the witness worlds reject
27.6%of 1,078 samples right on the real world
Agentic: real-world passes the witness worlds reject
29.8%of 423 samples right on the real world

Accuracy on the real world alone and on every world

A query can return the right objects on the real map and still be wrong; the witness worlds expose it. The gap between the two dots is what grading on the real world alone would overstate.

Single-shot 3 epochs

  1. claude-sonnet-5-5 75% +15.9
  2. gemini-3.8-flash 74% +10.4
  3. claude-haiku-5-5 67% +14.4
  4. glm-53-flash 51% +11.9
  5. deepseek-v4-flash 30% +25.9
  6. qwen3.5-397b 29% +20.4
  7. gemma-4-26b-a4b-it 23% +9.0
  8. gemma-4-31b-it 22% +10.0
  9. MiniMax-M3 16% +30.3

Agentic 1 epoch

  1. gemini-3.8-flash 84% +11.9
  2. claude-sonnet-5-5 82% +10.4
  3. claude-haiku-5-5 63% +16.4
  4. qwen3.5-397b 54% +14.9
  5. glm-53-flash 51% +23.9
  6. deepseek-v4-flash 48% +34.3
  7. MiniMax-M3 24% +44.8
  8. gemma-4-26b-a4b-it 21% +20.9
  9. gemma-4-31b-it 18% +10.4
Show as a table
Accuracy on every world and on the real world alone
ModelModeEvery worldReal world onlyGap, points
anthropic/claude-sonnet-5-5 Single-shot 75.1% 95% CI 64.2–84.8% 91.0% +15.9 95% CI +7.7 to +25.4
google/vertex/gemini-3.8-flash Single-shot 74.1% 95% CI 63.5–83.6% 84.6% +10.4 95% CI +4.0 to +18.0
anthropic/claude-haiku-5-5 Single-shot 66.7% 95% CI 55.6–77.0% 81.1% +14.4 95% CI +7.5 to +22.1
openai-api/adaption/glm-53-flash Single-shot 51.2% 95% CI 41.5–60.7% 63.2% +11.9 95% CI +6.5 to +18.0
openai-api/adaption/deepseek-v4-flash Single-shot 29.9% 95% CI 20.7–39.4% 55.7% +25.9 95% CI +18.2 to +33.8
openai-api/adaption/qwen3.5-397b Single-shot 29.4% 95% CI 21.2–37.9% 49.8% +20.4 95% CI +13.0 to +28.0
openai-api/adaption/gemma-4-26b-a4b-it Single-shot 22.9% 95% CI 13.5–33.3% 31.8% +9.0 95% CI +3.2 to +15.5
openai-api/adaption/gemma-4-31b-it Single-shot 22.4% 95% CI 13.0–32.4% 32.3% +10.0 95% CI +4.4 to +16.2
hf-inference-providers/MiniMaxAI/MiniMax-M3:deepinfra Single-shot 16.4% 95% CI 10.2–23.2% 46.8% +30.3 95% CI +21.1 to +40.1
google/vertex/gemini-3.8-flash Agentic 83.6% 95% CI 73.8–91.7% 95.5% +11.9 95% CI +4.5 to +20.0
anthropic/claude-sonnet-5-5 Agentic 82.1% 95% CI 71.9–91.0% 92.5% +10.4 95% CI +4.1 to +18.5
anthropic/claude-haiku-5-5 Agentic 62.7% 95% CI 50.8–73.6% 79.1% +16.4 95% CI +8.1 to +25.4
openai-api/adaption/qwen3.5-397b Agentic 53.7% 95% CI 41.3–66.2% 68.7% +14.9 95% CI +6.2 to +24.6
openai-api/adaption/glm-53-flash Agentic 50.7% 95% CI 39.1–62.7% 74.6% +23.9 95% CI +14.3 to +34.3
openai-api/adaption/deepseek-v4-flash Agentic 47.8% 95% CI 35.4–60.0% 82.1% +34.3 95% CI +22.6 to +46.9
hf-inference-providers/MiniMaxAI/MiniMax-M3:deepinfra Agentic 23.9% 95% CI 14.5–33.8% 68.7% +44.8 95% CI +32.8 to +57.4
openai-api/adaption/gemma-4-26b-a4b-it Agentic 20.9% 95% CI 10.9–31.3% 41.8% +20.9 95% CI +11.8 to +30.8
openai-api/adaption/gemma-4-31b-it Agentic 17.9% 95% CI 9.1–27.9% 28.4% +10.4 95% CI +4.2 to +18.2

Execution match against the grade

Execution match runs a query and the reference program as written on the real world and accepts equal results that are non-empty and error-free, as OverpassNL's evaluation does. Each bar splits a run's samples by which of the two accepts them; the number is the share that execution match accepts and the grade rejects.

Single-shot

  1. claude-sonnet-5-5 15%
  2. gemini-3.8-flash 8%
  3. claude-haiku-5-5 10%
  4. glm-53-flash 9%
  5. deepseek-v4-flash 21%
  6. qwen3.5-397b 17%
  7. gemma-4-26b-a4b-it 7%
  8. gemma-4-31b-it 8%
  9. MiniMax-M3 19%

Agentic

  1. gemini-3.8-flash 10%
  2. claude-sonnet-5-5 7%
  3. claude-haiku-5-5 15%
  4. qwen3.5-397b 15%
  5. glm-53-flash 21%
  6. deepseek-v4-flash 31%
  7. MiniMax-M3 34%
  8. gemma-4-26b-a4b-it 18%
  9. gemma-4-31b-it 7%
Show as a table
Samples by whether execution match and the grade accept them
ModelModeBoth acceptExecution match onlyGrade onlyNeither
anthropic/claude-sonnet-5-5 Single-shot 140 69.7%31 15.4%11 5.5%19 9.5%
google/vertex/gemini-3.8-flash Single-shot 132 65.7%16 8.0%17 8.5%36 17.9%
anthropic/claude-haiku-5-5 Single-shot 115 57.2%20 10.0%19 9.5%47 23.4%
openai-api/adaption/glm-53-flash Single-shot 97 48.3%19 9.5%6 3.0%79 39.3%
openai-api/adaption/deepseek-v4-flash Single-shot 59 29.4%43 21.4%1 0.5%98 48.8%
openai-api/adaption/qwen3.5-397b Single-shot 57 28.4%34 16.9%2 1.0%108 53.7%
openai-api/adaption/gemma-4-26b-a4b-it Single-shot 32 15.9%14 7.0%14 7.0%141 70.1%
openai-api/adaption/gemma-4-31b-it Single-shot 40 19.9%17 8.5%5 2.5%139 69.2%
hf-inference-providers/MiniMaxAI/MiniMax-M3:deepinfra Single-shot 24 11.9%38 18.9%9 4.5%130 64.7%
google/vertex/gemini-3.8-flash Agentic 55 82.1%7 10.4%1 1.5%4 6.0%
anthropic/claude-sonnet-5-5 Agentic 52 77.6%5 7.5%3 4.5%7 10.4%
anthropic/claude-haiku-5-5 Agentic 41 61.2%10 14.9%1 1.5%15 22.4%
openai-api/adaption/qwen3.5-397b Agentic 35 52.2%10 14.9%1 1.5%21 31.3%
openai-api/adaption/glm-53-flash Agentic 33 49.3%14 20.9%1 1.5%19 28.4%
openai-api/adaption/deepseek-v4-flash Agentic 29 43.3%21 31.3%3 4.5%14 20.9%
hf-inference-providers/MiniMaxAI/MiniMax-M3:deepinfra Agentic 10 14.9%23 34.3%6 9.0%28 41.8%
openai-api/adaption/gemma-4-26b-a4b-it Agentic 14 20.9%12 17.9%0 0.0%41 61.2%
openai-api/adaption/gemma-4-31b-it Agentic 11 16.4%5 7.5%1 1.5%50 74.6%

Accuracy by family

Accuracy on every world by task family. Each family has only a few test tasks, so read these as rough.

Single-shot

Single-shot accuracy by family
ModelTag groundingElement typesAttribute filtersSet compositionContainmentProximityViewportCountingName matchingRelationsJudgment calls
claude-sonnet-5-5 83% 100% 56% 89% 100% 78% 100% 33% 72% 33% 89%
gemini-3.8-flash 67% 100% 67% 72% 94% 83% 100% 72% 67% 33% 67%
claude-haiku-5-5 78% 100% 39% 61% 83% 72% 94% 50% 61% 33% 67%
glm-53-flash 72% 72% 33% 44% 67% 39% 78% 50% 44% 14% 56%
deepseek-v4-flash 28% 39% 22% 17% 33% 50% 78% 11% 17% 14% 22%
qwen3.5-397b 39% 50% 17% 28% 22% 44% 72% 0% 28% 5% 22%
gemma-4-26b-a4b-it 61% 28% 6% 6% 33% 33% 33% 0% 39% 0% 17%
gemma-4-31b-it 22% 39% 11% 6% 72% 0% 22% 0% 44% 0% 33%
MiniMax-M3 11% 22% 6% 0% 28% 39% 39% 6% 6% 10% 17%

Agentic

Agentic accuracy by family
ModelTag groundingElement typesAttribute filtersSet compositionContainmentProximityViewportCountingName matchingRelationsJudgment calls
gemini-3.8-flash 67% 100% 83% 83% 100% 100% 100% 100% 67% 57% 67%
claude-sonnet-5-5 67% 100% 83% 100% 100% 83% 100% 67% 83% 43% 83%
claude-haiku-5-5 83% 83% 50% 17% 67% 83% 83% 33% 50% 57% 83%
qwen3.5-397b 100% 50% 33% 33% 50% 50% 83% 50% 50% 14% 83%
glm-53-flash 17% 67% 67% 50% 50% 67% 50% 50% 33% 29% 83%
deepseek-v4-flash 100% 50% 50% 67% 67% 50% 67% 17% 0% 29% 33%
MiniMax-M3 67% 0% 0% 0% 50% 67% 33% 17% 0% 14% 17%
gemma-4-26b-a4b-it 33% 67% 0% 0% 33% 50% 0% 17% 0% 0% 33%
gemma-4-31b-it 50% 17% 17% 0% 33% 0% 17% 17% 17% 0% 33%