OverpassQL benchmark
Map queries, graded on more than one world
Natural-language requests about OpenStreetMap, answered with OverpassQL. A query passes only when it returns the right objects on a frozen real-world map and on synthetic witness worlds built to expose near misses.
- Request
- Which hotels are in the parish of Canillo?
- Reference program
{{geocodeArea:"Canillo"}}->.a; nwr["tourism"="hotel"](area.a); out;- Graded on
- Andorra Witness world 1Witness world 2Witness world 3
- Tasks
- 72
- Real worlds
- 4
- Witness worlds
- 216
- Task families
- 11
Map questions, answered by a query
The open map, for anyone
OpenStreetMap is the open map behind countless apps, analyses and services, and the Overpass API, with its query language OverpassQL, is the standard way to query that data. A model that turns a plain request into a correct query puts the map in the hands of people who could never write one.
Meant to run on the device
Map assistants belong close to the map: on a laptop or a phone, offline and private. That means small models, and a benchmark has to show how far today's models are from good enough.
Wrong queries fail silently
A query that misses a tag, an element type or an area still returns plausible objects, and nothing tells the user. Comparing its text with a reference, or its answer on one map, lets exactly these queries through.
Right on the map is not right
18 runs of 9 models on the 67 test tasks. See every result.
- Single-shot: real-world passes the witness worlds reject
- 28%
- Agentic: real-world passes the witness worlds reject
- 30%
- Best run
- 84%gemini-3.8-flash, agentic; 96% on the real world alone
- Most overstated
- +44.8 pointsMiniMax-M3, agentic: 69% on the real world alone, 24% on every world
Every run scores lower once the witness worlds are counted, so grading on the real world alone overstates them all. The strongest models we ran pass about four tasks in five with tools and three in four without, and miss a third of the hard ones; open-weight models pass between a sixth and a half, and the smallest of them almost none of the hard tasks. Map assistants are meant to run on the device, on models smaller still: they are far from ready.
One request, one query, every world
-
1
A request about a place
Each task asks for objects in one region, such as the hotels of a parish or the bus stops of a line, with an answer type and, when needed, a map view.
-
2
A query that runs
The model replies with one OverpassQL query, directly or after using tools. It runs on the pinned Overpass engine against a frozen OpenStreetMap snapshot.
-
3
The same answer everywhere
The returned objects are compared with those of validated reference programs on the real world and on the task's witness worlds. The query passes only if it matches one accepted reading on all of them.
Right on one map is not enough
On a real map, a wrong query can return exactly the right objects: a tag filter it leaves out may exclude nothing there, an element type it ignores may not occur, or it may list ids it looked up. Witness worlds place probe objects inside, outside and on the edges of what a request means, so these queries give themselves away.
Built to be checked
Graded on every world
Each task has three witness worlds that place probe objects inside, outside and on the edges of what its request means. A query passes only if it matches an accepted reading on all of them and on the real world.
Validated references
Every accepted reading is a program checked before release: rewrites that mean the same must pass, and mutants that do not must fail.
Tasks chosen in advance
New tasks are drafted to a coverage checklist fixed beforehand from the constructs of OverpassNL's test queries: family, query constructs, difficulty and tag-frequency band. No task is dropped because a model failed it.
A second author and a reviewer
A second author writes its own query for every task, and a person reviews and decides every task before it is frozen.
Pinned end to end
The engine image is pinned by digest, the worlds by SHA-256 and the gold answers by hash. One command rebuilds every world and checks every answer.
Traceable scores
Every score file records the package version and the digests of the engine, worlds, tag documentation, tasks, gold answers, tag counts and prompt it was graded with.
From training a model to grading one
I trained GeoQL-4B, a 4-billion-parameter local model that writes OverpassQL from natural language, with supervised fine-tuning and GRPO reinforcement learning on OverpassNL. Its reinforcement learning stage raised a reward built from similarity to reference queries without improving the benchmark score. That taught me that a verifiable reward needs queries executed on an OpenStreetMap database, which is what this benchmark provides.
In my work, I build geospatial data pipelines with OpenStreetMap data, Mapbox layers, FAA NASR aeronautical data, VFR sectional charts and weather data, in GeoJSON.
Help review the tasks
I reviewed and decided every task here. Before the main study, independent annotators who know OpenStreetMap tagging will review every task again, without AI help, and audit a sample of graded model outputs.
If you know OpenStreetMap tagging and want to take part, open a discussion on this Space. See how a task is built.