reference/everything

Reference agent No model 67 test tasks 1 epoch
Accuracy
0.0%95% CI 0.0–0.0%
Real world only
0.0%without witness worlds
Inflation
+0.0points, 95% CI +0.0 to +0.0
Pair consistency
0.0%7 minimal pairs

Verdicts

Verdicts by category
VerdictCountShare
wrong_real 67

Valid queries: 100.0%. Mean F1 on the real world: 0.002.

Agreement per world

Share of answers equal to the closest accepted reading, per world
WorldEqual to the closest reading
Real world
Witness world 1
Witness world 2
Witness world 3

By family

Accuracy by task family
FamilyTasksAccuracy
Attribute filters 6
Containment 6
Counting 6
Element types 6
Judgment calls 6
Name matching 6
Proximity 6
Relations 7
Set composition 6
Tag grounding 6
Viewport 6

By tag frequency

Accuracy by how common the tags of a task are in OpenStreetMap
TagsTasksAccuracy
Common 21
Uncommon 24
Rare 22

A task counts in the bucket of its rarest tag: common from a million objects, uncommon from 100,000.

Tasks

Verdict per task
Task Verdict Real world Witness worlds F1
How many parishes does Andorra have?
ad-01
wrong_real Wrong 0 of 3 agree 0.00
How many mountain peaks in Andorra are higher than 2800 m?
ad-03
wrong_real Wrong 0 of 3 agree 0.00
Where are the wilderness huts in Andorra, the unstaffed huts where hikers can spend the night?
ad-04
wrong_real Wrong 0 of 3 agree 0.00
Which places of worship in Andorra have "Sant Joan" in their name?
ad-05
wrong_real Wrong 0 of 3 agree 0.00
Where are the chair lifts, gondolas and magic carpets in La Massana?
ad-06
wrong_real Wrong 0 of 3 agree 0.00
Where are the cycling tracks in Escaldes-Engordany? I mean sports tracks for cycling, not cycle paths.
ad-07
wrong_real Wrong 0 of 3 agree 0.00
Which government offices in this view are mapped as points?
ad-08
wrong_real Wrong 0 of 3 agree 0.00
Which shared cycle and foot paths in this view are two-way and less than 3 m wide?
ad-09
wrong_real Wrong 0 of 3 agree 0.00
Which Stolpersteine and memorial plaques in this view are outside Andorra la Vella?
ad-10
wrong_real Wrong 0 of 3 agree 0.00
Which unclassified roads are in Ordino?
ad-11
wrong_real Wrong 0 of 3 agree 0.00
Which social facilities are within 500 m of 42.511° N, 1.534° E?
ad-12
wrong_real Wrong 0 of 3 agree 0.00
Where are the fences in this view?
ad-13
wrong_real Wrong 0 of 3 agree 0.00
How many waterfalls and springs in Andorra have no name?
ad-14
wrong_real Wrong 0 of 3 agree 0.00
Find everything in Andorra that has "Meritxell" in its name.
ad-15
wrong_real Wrong 0 of 3 agree 0.00
Which roads do right-turn-only restrictions start from?
ad-16
wrong_real Wrong 0 of 3 agree 0.00
Which roads do right-turn-only restrictions end on?
ad-17
wrong_real Wrong 0 of 3 agree 0.00
What monuments are there in Andorra?
ad-18
wrong_real Wrong 0 of 3 agree 0.00
Which bicycle rental stations are within 400 m of the point at 49.6111 N, 6.1300 E?
lu-02
wrong_real Wrong 0 of 3 agree 0.00
Which roads and paths in the city of Luxembourg are named after Grand Duchess Charlotte?
lu-03
wrong_real Wrong 0 of 3 agree 0.00
Where are the embassies in the city of Luxembourg?
lu-04
wrong_real Wrong 0 of 3 agree 0.00
How many restaurants in the city of Luxembourg offer vegan food?
lu-05
wrong_real Wrong 0 of 3 agree 0.00
Which bicycle rental stations are within 0.4 km of the point at 49.6111 N, 6.1300 E?
lu-06
wrong_real Wrong 0 of 3 agree 0.00
Which streets does bus line 19 run along in either direction?
lu-07
wrong_real Wrong 0 of 3 agree 0.00
Where are the car-sharing stations in this view?
lu-08
wrong_real Wrong 0 of 3 agree 0.00
Which offices in Kirchberg are mapped as points?
lu-09
wrong_real Wrong 0 of 3 agree 0.00
Which vending machines in Belair take credit or debit cards?
lu-10
wrong_real Wrong 0 of 3 agree 0.00
Where are the silos in Hesperange?
lu-11
wrong_real Wrong 0 of 3 agree 0.00
Where are the silos in Hesper?
lu-12
wrong_real Wrong 0 of 3 agree 0.00
Show the deciduous broadleaved trees in this view, both individual trees and tree rows.
lu-13
wrong_real Wrong 0 of 3 agree 0.00
How many cardiologists are there in this view?
lu-14
wrong_real Wrong 0 of 3 agree 0.00
Which art galleries in Luxembourg have "art" or "galerie" in their name, regardless of capitalization?
lu-15
wrong_real Wrong 0 of 3 agree 0.00
What are the administrative centres of the communes and cantons?
lu-16
wrong_real Wrong 0 of 3 agree 0.00
Which buildings in Strassen are stadiums?
lu-17
wrong_real Wrong 0 of 3 agree 0.00
Which information points in Hesperange have a fixme tag or haven't been checked since the start of 2024?
lu-18
wrong_real Wrong 0 of 3 agree 0.00
How many ATMs are there in Monaco?
mc-01
wrong_real Wrong 0 of 3 agree 0.00
Which cafes in Monaco are mapped as points?
mc-02
wrong_real Wrong 0 of 3 agree 0.00
Which bus stops are within 300 m of the Casino de Monte Carlo?
mc-04
wrong_real Wrong 0 of 3 agree 0.00
Which restaurants in Monaco have no wheelchair information?
mc-05
wrong_real Wrong 0 of 3 agree 0.02
Which cafes are in Monaco?
mc-06
wrong_real Wrong 0 of 3 agree 0.00
Which restaurants in Monaco have wheelchair information?
mc-07
wrong_real Wrong 0 of 3 agree 0.00
Which bus stops does bus line 1 serve in Monaco?
mc-08
wrong_real Wrong 0 of 3 agree 0.01
Which taxi stands in Monaco are mapped as areas?
mc-09
wrong_real Wrong 0 of 3 agree 0.00
Where are the escalators and elevators in Monaco?
mc-10
wrong_real Wrong 0 of 3 agree 0.02
Which hotels are in La Condamine?
mc-11
wrong_real Wrong 0 of 3 agree 0.00
I use a wheelchair. Which accessible parking spaces, and which car parks with spaces reserved for disabled drivers, are within 250 m of a wheelchair-accessible public toilet?
mc-12
wrong_real Wrong 0 of 3 agree 0.00
Where are the sports pitches in this view?
mc-13
wrong_real Wrong 0 of 3 agree 0.00
How many crossings in this view have no road markings, not counting those with traffic lights?
mc-14
wrong_real Wrong 0 of 3 agree 0.00
What in Monaco has "Grace" in its name?
mc-15
wrong_real Wrong 0 of 3 agree 0.01
Show the train routes other than the TGV, together with the stops they call at.
mc-16
wrong_real Wrong 0 of 3 agree 0.00
Where can I park my bicycle in this view?
mc-17
wrong_real Wrong 0 of 3 agree 0.00
Where are the pharmacies in Malé?
mv-01
wrong_real Wrong 0 of 3 agree 0.00
Which mosques are in Hulhumalé?
mv-02
wrong_real Wrong 0 of 3 agree 0.00
Show the cafes in this view.
mv-04
wrong_real Wrong 0 of 3 agree 0.00
Which shops in Malé sell mobile phones or electronics?
mv-05
wrong_real Wrong 0 of 3 agree 0.00
Where are the Italian restaurants in the Maldives?
mv-07
wrong_real Wrong 0 of 3 agree 0.00
Which clinics in this view are mapped as points?
mv-08
wrong_real Wrong 0 of 3 agree 0.00
Which fountains in Malé have no information on whether their water is drinkable?
mv-09
wrong_real Wrong 0 of 3 agree 0.00
List the reefs and beaches in North Ari Atoll, except those that are tourist attractions.
mv-10
wrong_real Wrong 0 of 3 agree 0.01
Which convenience stores are there in Hulhumalé?
mv-11
wrong_real Wrong 0 of 3 agree 0.00
Which copy shops in Malé are within 200 m of a government office?
mv-12
wrong_real Wrong 0 of 3 agree 0.00
Show me the living streets in this view.
mv-13
wrong_real Wrong 0 of 3 agree 0.00
Show me the living streets in this view that are mapped as ways.
mv-14
wrong_real Wrong 0 of 3 agree 0.00
How many primary roads are there in Malé?
mv-15
wrong_real Wrong 0 of 3 agree 0.00
How many primary roads are there in Malé City?
mv-16
wrong_real Wrong 0 of 3 agree 0.00
Which walls have "Dhiraagu" or "Ooredoo" in their name, whatever the capitalization?
mv-17
wrong_real Wrong 0 of 3 agree 0.00
What are the members of the ferry route relations?
mv-18
wrong_real Wrong 0 of 3 agree 0.00
Which youth centres have a website?
mv-19
wrong_real Wrong 0 of 3 agree 0.00

Provenance

What this run was graded with.

Package
geoql-env 0.1.0
Engine image
wiktorn/overpass-api:v0.7.62.11@sha256:5f643f2aa500333f19458fe8be14457215d6e0b9f891ca5c5ed452b1f8882c47
Tasks
0741a2bbbe847920eb5f20125d108298c8ade1c36a73d9e9fda9309d76b557ba
Gold answers
172a3c16405238a3cad35ddb83983a1444af7de66412829cd7faa2875bf36703
Tag counts
94db7a7656e2c9458ffdf0c8f1e3e8410dd68fed0c702005a0f73ff388240e73
Tag documentation
sha512-z2nQgHLNwCjebDxlKBaERjg02UmcJyemAtsXExJFlXNNc4K9l7EdtfkGs064Y0IgzUltmUxM/FeUBRyhb4kv6w==
Prompt
a6c16acc42ae863c666bab94f8dc4a6770cdeaed830fd495b7b192e7f29af961

Files

The grading holds the summary, every verdict and the provenance; the predictions are the run's responses, one JSON object per line.