About
How the tasks are built, how answers are graded, and what every score depends on.
Pipeline
A task is built and checked before any evaluated model sees it; a run is then graded on every world of every task.
One sample, graded
An illustration. The query runs on the real world and on the task's three witness worlds: filled squares are the objects it returns, and rings mark an accepted reading's answer.
This query leaves out the outdoor-seating filter. Every café in the real view has outdoor seating, so its answer is right there; witness world 2 adds a café without it, and the query returns that one too.
This query keeps the filter, so its answer equals an accepted reading's on every world.
Building a task
- Worlds
OpenStreetMap extracts of 4 regions, pinned by SHA-256, stripped of user metadata and loaded into a pinned Overpass engine.
- Checklist
A coverage checklist, fixed before drafting, gives each new task a family, the query constructs and difficulty, a tag-frequency band and a concept from the iD presets.
- Draft
A model drafts the request, every reading a careful reader could accept as a reference program with its OSM documentation, and probes for near misses.
- Validate
Rewrites that mean the same must pass and mutants that do not must fail, on the real world and on three witness worlds built from the probes.
- Second author
A second author writes its own query, graded on every world: a pass becomes a rewrite that must pass, a misreading the request does not support a mutant that must fail.
- Review
A person decides every task: accept, fix or add a reading, reword, or drop for reasons fixed in advance, never because a model failed. In this release, I reviewed all of them.
- Freeze
The task files and a hash of every gold answer are fixed before an evaluated model sees the task, and a canary string marks them.
Grading a run
- Request
The model gets the conventions, the request, the world, the answer type and, when the request names it, the map view.
- Model
Single-shot, it replies with one query. Agentic, it can run queries, resolve place names and look up tags before it submits one.
- Query
The query is taken from the reply, and placeholders such as the map view and named areas are filled in.
- Every world
It runs on the pinned engine against the real world and each witness world.
Real worldWitness 1Witness 2Witness 3 - Verdict
It passes only if its answers equal one accepted reading on every world; otherwise the verdict names what went wrong.
- Scores
Accuracy with a 95% interval, accuracy on the real world alone, the gap between them, and pair consistency.
The tasks
- Tasks
- 72
- Test
- 67
- Dev
- 5
- Real worlds
- 4
- Families
- 11
- Minimal pairs
- 7
Families range from tag grounding and element types to proximity, relations and judgment calls. The two tasks of a minimal pair differ in one phrase, which keeps or changes the answer. Dev tasks show their reference programs and notes; test tasks show only what a model gets, so that they stay useful for evaluation.
Review
I reviewed and decided every task in this release. Before the main study, independent annotators who know OpenStreetMap tagging will review every task again, without AI help, and audit a sample of graded model outputs.
Know OpenStreetMap tagging and want to help review? Open a discussion on this Space.
Conventions
Every prompt starts with these conventions. The instructions of the run's mode follow, either one query in a code block or tools and a final submit, and then the task's own part, shown on each task page.
You translate requests about OpenStreetMap data into OverpassQL queries.
Each request is about one world: a frozen OpenStreetMap snapshot of a region, queried with the Overpass API. The world has no edit history and no user metadata.
Conventions:
- Answers are the matching OSM objects (nodes, ways and relations). A feature counts whatever its element type, unless the request names a geometry; "mapped as points" means nodes.
- "In X" means inside the area that {{geocodeArea:"X"}} resolves to; "A, B" means A inside an area named B. "In this view" means inside the map view, {{bbox}}.
- "Within N m" means around:N.
- Names are matched on the name tag, exactly, unless the request says otherwise.
- Count requests count matching objects of any type; answer with out count; or by printing the objects.
- Untagged geometry (vertex nodes, member ways) does not count.
Queries:
- {{geocodeArea:"name"}} is the area of the world found by that place name; {{bbox}} and {{center}} are the map view, when the request has one. No other shortcuts are available.
- Output format, timeout and memory limits are set for you; any out statement works.
- History ([date:], [diff:], [adiff:], newer, changed, timeline, retro, compare), user filters (user, uid) and Overpass XML are not supported.Grading
The answer of a query on a world is the set of tagged OpenStreetMap objects it prints, as type and id, with areas mapped back to their relations. Untagged geometry, output format, order and duplicates do not matter. For a count task it is the total of the out count statements, or else the number of objects printed.
A response passes when its answers equal those of one accepted reading on every world of the task. Otherwise its verdict is one of these categories.
| Verdict | Meaning |
|---|---|
| pass | Equal to one accepted reading on every world of the task. |
| no_query | No query was found in the response. |
| rejected | Refused before running: Overpass XML, history or user filters, an unsupported shortcut, a place name with no area in the world, or a map view the task does not have. |
| query_error | The engine refused the query, for example for a syntax error. |
| runtime_error | The query failed while running. |
| timeout | The query ran past its time limit. |
| wrong_answer_type | Printed a count where the task asks for objects. |
| wrong_real | Its answer on the real world equals no accepted reading. |
| wrong_witness | Right on the real world but not on a witness world: a pass when grading on the real world alone. |
Scores
- Accuracy is the share of tasks and epochs that pass, with a 95% interval from a bootstrap that resamples intents, so that tasks written from one intent move together.
- Real-world-only accuracy is the share whose answer on the real world alone equals an accepted reading: what grading without witness worlds would report.
- Inflation is the real-world-only accuracy minus the accuracy.
- Pair consistency is the share of minimal pairs in which both tasks pass. The two tasks of a pair differ in one phrase, which keeps or changes the answer.
Worlds
Each real world is an OpenStreetMap extract pinned by SHA-256, stripped of user metadata and loaded into the pinned Overpass engine. Witness worlds are built from each task's witness spec; the places they rebuild keep their real ids and tags.
Reference runs
Agents that need no model, stored with their grading and re-graded by make reproduce: empty submits nothing, everything returns every tagged object of the world, and copy-ids lists by id the objects a reference program returns on the real world. Their scores are on the leaderboard.
Provenance
Every score file records these values. Scores graded against other tasks or gold answers are not comparable.
Licenses and attribution
- Map data, and everything built from it, including gold answers and their hashes: © OpenStreetMap contributors, under the Open Database License.
- Task files: CC BY 4.0. The OpenStreetMap ids, names and tags they contain are under the ODbL.
- Code: MIT.
- Tag documentation: the iD tagging schema, under the ISC license. Tag counts: taginfo, under the ODbL.
- Fonts: Inter and JetBrains Mono, under the SIL Open Font License 1.1 (Inter, JetBrains Mono).