openai-api/adaption/gemma-4-26b-a4b-it

Single-shot 67 test tasks 3 epochs
Accuracy
22.9%95% CI 13.5–33.3%
Real world only
31.8%without witness worlds
Inflation
+9.0points, 95% CI +3.2 to +15.5
Pair consistency
14.3%7 minimal pairs

Verdicts

Verdicts by category
VerdictCountShare
wrong_real 73
pass 46
query_error 43
rejected 21
wrong_witness 18

Valid queries: 68.2%. Mean F1 on the real world: 0.332.

Agreement per world

Share of answers equal to the closest accepted reading, per world
WorldEqual to the closest reading
Real world
Witness world 1
Witness world 2
Witness world 3

By family

Accuracy by task family
FamilyTasksAccuracy
Attribute filters 6
Containment 6
Counting 6
Element types 6
Judgment calls 6
Name matching 6
Proximity 6
Relations 7
Set composition 6
Tag grounding 6
Viewport 6

By tag frequency

Accuracy by how common the tags of a task are in OpenStreetMap
TagsTasksAccuracy
Common 21
Uncommon 24
Rare 22

A task counts in the bucket of its rarest tag: common from a million objects, uncommon from 100,000.

Tasks

Verdict per task and epoch
Task Epoch Verdict Real world Witness worlds F1
How many parishes does Andorra have?
ad-01
1 wrong_real Wrong 0 of 3 agree 0.00
How many parishes does Andorra have?
ad-01
2 wrong_real Wrong 0 of 3 agree 0.00
How many parishes does Andorra have?
ad-01
3 wrong_real Wrong 0 of 3 agree 0.00
How many mountain peaks in Andorra are higher than 2800 m?
ad-03
1 query_error Wrong 0 of 3 agree 0.00
How many mountain peaks in Andorra are higher than 2800 m?
ad-03
2 query_error Wrong 0 of 3 agree 0.00
How many mountain peaks in Andorra are higher than 2800 m?
ad-03
3 query_error Wrong 0 of 3 agree 0.00
Where are the wilderness huts in Andorra, the unstaffed huts where hikers can spend the night?
ad-04
1 wrong_real Wrong 0 of 3 agree 0.27
Where are the wilderness huts in Andorra, the unstaffed huts where hikers can spend the night?
ad-04
2 wrong_real Wrong 0 of 3 agree 0.27
Where are the wilderness huts in Andorra, the unstaffed huts where hikers can spend the night?
ad-04
3 wrong_real Wrong 0 of 3 agree 0.00
Which places of worship in Andorra have "Sant Joan" in their name?
ad-05
1 pass Right 3 of 3 agree 1.00
Which places of worship in Andorra have "Sant Joan" in their name?
ad-05
2 wrong_real Wrong 0 of 3 agree 0.00
Which places of worship in Andorra have "Sant Joan" in their name?
ad-05
3 pass Right 3 of 3 agree 1.00
Where are the chair lifts, gondolas and magic carpets in La Massana?
ad-06
1 wrong_real Wrong 0 of 3 agree 0.00
Where are the chair lifts, gondolas and magic carpets in La Massana?
ad-06
2 wrong_real Wrong 0 of 3 agree 0.00
Where are the chair lifts, gondolas and magic carpets in La Massana?
ad-06
3 query_error Wrong 0 of 3 agree 0.00
Where are the cycling tracks in Escaldes-Engordany? I mean sports tracks for cycling, not cycle paths.
ad-07
1 pass Right 3 of 3 agree 1.00
Where are the cycling tracks in Escaldes-Engordany? I mean sports tracks for cycling, not cycle paths.
ad-07
2 pass Right 3 of 3 agree 1.00
Where are the cycling tracks in Escaldes-Engordany? I mean sports tracks for cycling, not cycle paths.
ad-07
3 wrong_witness Right 0 of 3 agree 1.00
Which government offices in this view are mapped as points?
ad-08
1 wrong_real Wrong 0 of 3 agree 0.55
Which government offices in this view are mapped as points?
ad-08
2 query_error Wrong 0 of 3 agree 0.00
Which government offices in this view are mapped as points?
ad-08
3 query_error Wrong 0 of 3 agree 0.00
Which shared cycle and foot paths in this view are two-way and less than 3 m wide?
ad-09
1 query_error Wrong 0 of 3 agree 0.00
Which shared cycle and foot paths in this view are two-way and less than 3 m wide?
ad-09
2 query_error Wrong 0 of 3 agree 0.00
Which shared cycle and foot paths in this view are two-way and less than 3 m wide?
ad-09
3 query_error Wrong 0 of 3 agree 0.00
Which Stolpersteine and memorial plaques in this view are outside Andorra la Vella?
ad-10
1 query_error Wrong 0 of 3 agree 0.00
Which Stolpersteine and memorial plaques in this view are outside Andorra la Vella?
ad-10
2 query_error Wrong 0 of 3 agree 0.00
Which Stolpersteine and memorial plaques in this view are outside Andorra la Vella?
ad-10
3 query_error Wrong 0 of 3 agree 0.00
Which unclassified roads are in Ordino?
ad-11
1 query_error Wrong 0 of 3 agree 0.00
Which unclassified roads are in Ordino?
ad-11
2 query_error Wrong 0 of 3 agree 0.00
Which unclassified roads are in Ordino?
ad-11
3 query_error Wrong 0 of 3 agree 0.00
Which social facilities are within 500 m of 42.511° N, 1.534° E?
ad-12
1 wrong_real Wrong 0 of 3 agree 0.03
Which social facilities are within 500 m of 42.511° N, 1.534° E?
ad-12
2 wrong_real Wrong 0 of 3 agree 0.03
Which social facilities are within 500 m of 42.511° N, 1.534° E?
ad-12
3 wrong_real Wrong 0 of 3 agree 0.03
Where are the fences in this view?
ad-13
1 wrong_real Wrong 0 of 3 agree 0.00
Where are the fences in this view?
ad-13
2 wrong_real Wrong 0 of 3 agree 0.00
Where are the fences in this view?
ad-13
3 wrong_real Wrong 0 of 3 agree 0.00
How many waterfalls and springs in Andorra have no name?
ad-14
1 wrong_real Wrong 0 of 3 agree 0.00
How many waterfalls and springs in Andorra have no name?
ad-14
2 query_error Wrong 0 of 3 agree 0.00
How many waterfalls and springs in Andorra have no name?
ad-14
3 wrong_real Wrong 0 of 3 agree 0.00
Find everything in Andorra that has "Meritxell" in its name.
ad-15
1 pass Right 3 of 3 agree 1.00
Find everything in Andorra that has "Meritxell" in its name.
ad-15
2 pass Right 3 of 3 agree 1.00
Find everything in Andorra that has "Meritxell" in its name.
ad-15
3 pass Right 3 of 3 agree 1.00
Which roads do right-turn-only restrictions start from?
ad-16
1 wrong_real Wrong 0 of 3 agree 0.00
Which roads do right-turn-only restrictions start from?
ad-16
2 wrong_real Wrong 0 of 3 agree 0.00
Which roads do right-turn-only restrictions start from?
ad-16
3 wrong_real Wrong 0 of 3 agree 0.00
Which roads do right-turn-only restrictions end on?
ad-17
1 wrong_real Wrong 0 of 3 agree 0.00
Which roads do right-turn-only restrictions end on?
ad-17
2 rejected Wrong 0 of 3 agree 0.00
Which roads do right-turn-only restrictions end on?
ad-17
3 rejected Wrong 0 of 3 agree 0.00
What monuments are there in Andorra?
ad-18
1 wrong_real Wrong 0 of 3 agree 0.00
What monuments are there in Andorra?
ad-18
2 wrong_real Wrong 0 of 3 agree 0.00
What monuments are there in Andorra?
ad-18
3 wrong_real Wrong 0 of 3 agree 0.00
Which bicycle rental stations are within 400 m of the point at 49.6111 N, 6.1300 E?
lu-02
1 pass Right 3 of 3 agree 1.00
Which bicycle rental stations are within 400 m of the point at 49.6111 N, 6.1300 E?
lu-02
2 pass Right 3 of 3 agree 1.00
Which bicycle rental stations are within 400 m of the point at 49.6111 N, 6.1300 E?
lu-02
3 pass Right 3 of 3 agree 1.00
Which roads and paths in the city of Luxembourg are named after Grand Duchess Charlotte?
lu-03
1 rejected Wrong 0 of 3 agree 0.00
Which roads and paths in the city of Luxembourg are named after Grand Duchess Charlotte?
lu-03
2 rejected Wrong 0 of 3 agree 0.00
Which roads and paths in the city of Luxembourg are named after Grand Duchess Charlotte?
lu-03
3 wrong_real Wrong 0 of 3 agree 0.00
Where are the embassies in the city of Luxembourg?
lu-04
1 rejected Wrong 0 of 3 agree 0.00
Where are the embassies in the city of Luxembourg?
lu-04
2 rejected Wrong 0 of 3 agree 0.00
Where are the embassies in the city of Luxembourg?
lu-04
3 rejected Wrong 0 of 3 agree 0.00
How many restaurants in the city of Luxembourg offer vegan food?
lu-05
1 rejected Wrong 0 of 3 agree 0.00
How many restaurants in the city of Luxembourg offer vegan food?
lu-05
2 wrong_real Wrong 0 of 3 agree 0.00
How many restaurants in the city of Luxembourg offer vegan food?
lu-05
3 rejected Wrong 0 of 3 agree 0.00
Which bicycle rental stations are within 0.4 km of the point at 49.6111 N, 6.1300 E?
lu-06
1 pass Right 3 of 3 agree 1.00
Which bicycle rental stations are within 0.4 km of the point at 49.6111 N, 6.1300 E?
lu-06
2 pass Right 3 of 3 agree 1.00
Which bicycle rental stations are within 0.4 km of the point at 49.6111 N, 6.1300 E?
lu-06
3 pass Right 3 of 3 agree 1.00
Which streets does bus line 19 run along in either direction?
lu-07
1 rejected Wrong 0 of 3 agree 0.00
Which streets does bus line 19 run along in either direction?
lu-07
2 rejected Wrong 0 of 3 agree 0.00
Which streets does bus line 19 run along in either direction?
lu-07
3 rejected Wrong 0 of 3 agree 0.00
Where are the car-sharing stations in this view?
lu-08
1 pass Right 3 of 3 agree 1.00
Where are the car-sharing stations in this view?
lu-08
2 pass Right 3 of 3 agree 1.00
Where are the car-sharing stations in this view?
lu-08
3 pass Right 3 of 3 agree 1.00
Which offices in Kirchberg are mapped as points?
lu-09
1 pass Right 3 of 3 agree 1.00
Which offices in Kirchberg are mapped as points?
lu-09
2 wrong_real Wrong 0 of 3 agree 0.08
Which offices in Kirchberg are mapped as points?
lu-09
3 wrong_witness Right 2 of 3 agree 1.00
Which vending machines in Belair take credit or debit cards?
lu-10
1 rejected Wrong 0 of 3 agree 0.00
Which vending machines in Belair take credit or debit cards?
lu-10
2 rejected Wrong 0 of 3 agree 0.00
Which vending machines in Belair take credit or debit cards?
lu-10
3 rejected Wrong 0 of 3 agree 0.00
Where are the silos in Hesperange?
lu-11
1 wrong_real Wrong 0 of 3 agree 0.00
Where are the silos in Hesperange?
lu-11
2 wrong_real Wrong 0 of 3 agree 0.00
Where are the silos in Hesperange?
lu-11
3 wrong_real Wrong 0 of 3 agree 0.00
Where are the silos in Hesper?
lu-12
1 wrong_real Wrong 0 of 3 agree 0.00
Where are the silos in Hesper?
lu-12
2 wrong_real Wrong 0 of 3 agree 0.00
Where are the silos in Hesper?
lu-12
3 wrong_real Wrong 0 of 3 agree 0.00
Show the deciduous broadleaved trees in this view, both individual trees and tree rows.
lu-13
1 wrong_real Wrong 0 of 3 agree 0.03
Show the deciduous broadleaved trees in this view, both individual trees and tree rows.
lu-13
2 wrong_real Wrong 0 of 3 agree 0.00
Show the deciduous broadleaved trees in this view, both individual trees and tree rows.
lu-13
3 wrong_real Wrong 0 of 3 agree 0.00
How many cardiologists are there in this view?
lu-14
1 wrong_real Wrong 0 of 3 agree 0.00
How many cardiologists are there in this view?
lu-14
2 wrong_real Wrong 0 of 3 agree 0.00
How many cardiologists are there in this view?
lu-14
3 wrong_real Wrong 0 of 3 agree 0.00
Which art galleries in Luxembourg have "art" or "galerie" in their name, regardless of capitalization?
lu-15
1 rejected Wrong 0 of 3 agree 0.00
Which art galleries in Luxembourg have "art" or "galerie" in their name, regardless of capitalization?
lu-15
2 rejected Wrong 0 of 3 agree 0.00
Which art galleries in Luxembourg have "art" or "galerie" in their name, regardless of capitalization?
lu-15
3 wrong_real Wrong 0 of 3 agree 0.00
What are the administrative centres of the communes and cantons?
lu-16
1 rejected Wrong 0 of 3 agree 0.00
What are the administrative centres of the communes and cantons?
lu-16
2 rejected Wrong 0 of 3 agree 0.00
What are the administrative centres of the communes and cantons?
lu-16
3 rejected Wrong 0 of 3 agree 0.00
Which buildings in Strassen are stadiums?
lu-17
1 query_error Wrong 0 of 3 agree 0.00
Which buildings in Strassen are stadiums?
lu-17
2 wrong_real Wrong 0 of 3 agree 0.00
Which buildings in Strassen are stadiums?
lu-17
3 wrong_real Wrong 0 of 3 agree 0.00
Which information points in Hesperange have a fixme tag or haven't been checked since the start of 2024?
lu-18
1 query_error Wrong 0 of 3 agree 0.00
Which information points in Hesperange have a fixme tag or haven't been checked since the start of 2024?
lu-18
2 query_error Wrong 0 of 3 agree 0.00
Which information points in Hesperange have a fixme tag or haven't been checked since the start of 2024?
lu-18
3 query_error Wrong 0 of 3 agree 0.00
How many ATMs are there in Monaco?
mc-01
1 wrong_witness Right 2 of 3 agree 1.00
How many ATMs are there in Monaco?
mc-01
2 wrong_witness Right 2 of 3 agree 1.00
How many ATMs are there in Monaco?
mc-01
3 query_error Wrong 0 of 3 agree 0.00
Which cafes in Monaco are mapped as points?
mc-02
1 wrong_witness Right 2 of 3 agree 1.00
Which cafes in Monaco are mapped as points?
mc-02
2 rejected Wrong 0 of 3 agree 0.00
Which cafes in Monaco are mapped as points?
mc-02
3 pass Right 3 of 3 agree 1.00
Which bus stops are within 300 m of the Casino de Monte Carlo?
mc-04
1 wrong_real Wrong 0 of 3 agree 0.00
Which bus stops are within 300 m of the Casino de Monte Carlo?
mc-04
2 wrong_real Wrong 0 of 3 agree 0.00
Which bus stops are within 300 m of the Casino de Monte Carlo?
mc-04
3 query_error Wrong 0 of 3 agree 0.00
Which restaurants in Monaco have no wheelchair information?
mc-05
1 query_error Wrong 0 of 3 agree 0.00
Which restaurants in Monaco have no wheelchair information?
mc-05
2 query_error Wrong 0 of 3 agree 0.00
Which restaurants in Monaco have no wheelchair information?
mc-05
3 wrong_witness Right 2 of 3 agree 1.00
Which cafes are in Monaco?
mc-06
1 wrong_witness Right 2 of 3 agree 1.00
Which cafes are in Monaco?
mc-06
2 wrong_witness Right 2 of 3 agree 1.00
Which cafes are in Monaco?
mc-06
3 wrong_witness Right 2 of 3 agree 1.00
Which restaurants in Monaco have wheelchair information?
mc-07
1 wrong_witness Right 2 of 3 agree 1.00
Which restaurants in Monaco have wheelchair information?
mc-07
2 wrong_witness Right 2 of 3 agree 1.00
Which restaurants in Monaco have wheelchair information?
mc-07
3 pass Right 3 of 3 agree 1.00
Which bus stops does bus line 1 serve in Monaco?
mc-08
1 query_error Wrong 0 of 3 agree 0.00
Which bus stops does bus line 1 serve in Monaco?
mc-08
2 wrong_real Wrong 0 of 3 agree 0.00
Which bus stops does bus line 1 serve in Monaco?
mc-08
3 query_error Wrong 0 of 3 agree 0.00
Which taxi stands in Monaco are mapped as areas?
mc-09
1 wrong_real Wrong 0 of 3 agree 0.00
Which taxi stands in Monaco are mapped as areas?
mc-09
2 wrong_real Wrong 0 of 3 agree 0.00
Which taxi stands in Monaco are mapped as areas?
mc-09
3 wrong_real Wrong 0 of 3 agree 0.00
Where are the escalators and elevators in Monaco?
mc-10
1 wrong_real Wrong 0 of 3 agree 0.06
Where are the escalators and elevators in Monaco?
mc-10
2 wrong_real Wrong 0 of 3 agree 0.06
Where are the escalators and elevators in Monaco?
mc-10
3 wrong_real Wrong 0 of 3 agree 0.06
Which hotels are in La Condamine?
mc-11
1 pass Right 3 of 3 agree 1.00
Which hotels are in La Condamine?
mc-11
2 pass Right 3 of 3 agree 1.00
Which hotels are in La Condamine?
mc-11
3 wrong_witness Right 2 of 3 agree 1.00
I use a wheelchair. Which accessible parking spaces, and which car parks with spaces reserved for disabled drivers, are within 250 m of a wheelchair-accessible public toilet?
mc-12
1 query_error Wrong 0 of 3 agree 0.00
I use a wheelchair. Which accessible parking spaces, and which car parks with spaces reserved for disabled drivers, are within 250 m of a wheelchair-accessible public toilet?
mc-12
2 query_error Wrong 0 of 3 agree 0.00
I use a wheelchair. Which accessible parking spaces, and which car parks with spaces reserved for disabled drivers, are within 250 m of a wheelchair-accessible public toilet?
mc-12
3 query_error Wrong 0 of 3 agree 0.00
Where are the sports pitches in this view?
mc-13
1 pass Right 3 of 3 agree 1.00
Where are the sports pitches in this view?
mc-13
2 pass Right 3 of 3 agree 1.00
Where are the sports pitches in this view?
mc-13
3 pass Right 3 of 3 agree 1.00
How many crossings in this view have no road markings, not counting those with traffic lights?
mc-14
1 wrong_real Wrong 0 of 3 agree 0.00
How many crossings in this view have no road markings, not counting those with traffic lights?
mc-14
2 query_error Wrong 0 of 3 agree 0.00
How many crossings in this view have no road markings, not counting those with traffic lights?
mc-14
3 query_error Wrong 0 of 3 agree 0.00
What in Monaco has "Grace" in its name?
mc-15
1 pass Right 3 of 3 agree 1.00
What in Monaco has "Grace" in its name?
mc-15
2 query_error Wrong 0 of 3 agree 0.00
What in Monaco has "Grace" in its name?
mc-15
3 pass Right 3 of 3 agree 1.00
Show the train routes other than the TGV, together with the stops they call at.
mc-16
1 query_error Wrong 0 of 3 agree 0.00
Show the train routes other than the TGV, together with the stops they call at.
mc-16
2 wrong_real Wrong 0 of 3 agree 0.00
Show the train routes other than the TGV, together with the stops they call at.
mc-16
3 query_error Wrong 0 of 3 agree 0.00
Where can I park my bicycle in this view?
mc-17
1 pass Right 3 of 3 agree 1.00
Where can I park my bicycle in this view?
mc-17
2 pass Right 3 of 3 agree 1.00
Where can I park my bicycle in this view?
mc-17
3 pass Right 3 of 3 agree 1.00
Where are the pharmacies in Malé?
mv-01
1 pass Right 3 of 3 agree 1.00
Where are the pharmacies in Malé?
mv-01
2 pass Right 3 of 3 agree 1.00
Where are the pharmacies in Malé?
mv-01
3 pass Right 3 of 3 agree 1.00
Which mosques are in Hulhumalé?
mv-02
1 wrong_real Wrong 0 of 3 agree 0.00
Which mosques are in Hulhumalé?
mv-02
2 query_error Wrong 0 of 3 agree 0.00
Which mosques are in Hulhumalé?
mv-02
3 pass Right 3 of 3 agree 1.00
Show the cafes in this view.
mv-04
1 pass Right 3 of 3 agree 1.00
Show the cafes in this view.
mv-04
2 pass Right 3 of 3 agree 1.00
Show the cafes in this view.
mv-04
3 pass Right 3 of 3 agree 1.00
Which shops in Malé sell mobile phones or electronics?
mv-05
1 wrong_real Wrong 0 of 3 agree 0.00
Which shops in Malé sell mobile phones or electronics?
mv-05
2 wrong_real Wrong 0 of 3 agree 0.00
Which shops in Malé sell mobile phones or electronics?
mv-05
3 pass Right 3 of 3 agree 1.00
Where are the Italian restaurants in the Maldives?
mv-07
1 pass Right 3 of 3 agree 1.00
Where are the Italian restaurants in the Maldives?
mv-07
2 pass Right 3 of 3 agree 1.00
Where are the Italian restaurants in the Maldives?
mv-07
3 pass Right 3 of 3 agree 1.00
Which clinics in this view are mapped as points?
mv-08
1 pass Right 3 of 3 agree 1.00
Which clinics in this view are mapped as points?
mv-08
2 pass Right 3 of 3 agree 1.00
Which clinics in this view are mapped as points?
mv-08
3 pass Right 3 of 3 agree 1.00
Which fountains in Malé have no information on whether their water is drinkable?
mv-09
1 query_error Wrong 0 of 3 agree 0.00
Which fountains in Malé have no information on whether their water is drinkable?
mv-09
2 wrong_real Wrong 0 of 3 agree 0.89
Which fountains in Malé have no information on whether their water is drinkable?
mv-09
3 query_error Wrong 0 of 3 agree 0.00
List the reefs and beaches in North Ari Atoll, except those that are tourist attractions.
mv-10
1 query_error Wrong 0 of 3 agree 0.00
List the reefs and beaches in North Ari Atoll, except those that are tourist attractions.
mv-10
2 wrong_real Wrong 0 of 3 agree 0.00
List the reefs and beaches in North Ari Atoll, except those that are tourist attractions.
mv-10
3 query_error Wrong 0 of 3 agree 0.00
Which convenience stores are there in Hulhumalé?
mv-11
1 pass Right 3 of 3 agree 1.00
Which convenience stores are there in Hulhumalé?
mv-11
2 pass Right 3 of 3 agree 1.00
Which convenience stores are there in Hulhumalé?
mv-11
3 pass Right 3 of 3 agree 1.00
Which copy shops in Malé are within 200 m of a government office?
mv-12
1 query_error Wrong 0 of 3 agree 0.00
Which copy shops in Malé are within 200 m of a government office?
mv-12
2 query_error Wrong 0 of 3 agree 0.00
Which copy shops in Malé are within 200 m of a government office?
mv-12
3 query_error Wrong 0 of 3 agree 0.00
Show me the living streets in this view.
mv-13
1 wrong_real Wrong 0 of 3 agree 0.00
Show me the living streets in this view.
mv-13
2 wrong_real Wrong 0 of 3 agree 0.00
Show me the living streets in this view.
mv-13
3 wrong_real Wrong 0 of 3 agree 0.00
Show me the living streets in this view that are mapped as ways.
mv-14
1 wrong_real Wrong 0 of 3 agree 0.00
Show me the living streets in this view that are mapped as ways.
mv-14
2 wrong_real Wrong 0 of 3 agree 0.00
Show me the living streets in this view that are mapped as ways.
mv-14
3 wrong_real Wrong 0 of 3 agree 0.00
How many primary roads are there in Malé?
mv-15
1 wrong_real Wrong 0 of 3 agree 0.00
How many primary roads are there in Malé?
mv-15
2 wrong_witness Right 0 of 3 agree 1.00
How many primary roads are there in Malé?
mv-15
3 wrong_witness Right 0 of 3 agree 1.00
How many primary roads are there in Malé City?
mv-16
1 wrong_real Wrong 0 of 3 agree 0.00
How many primary roads are there in Malé City?
mv-16
2 wrong_witness Right 0 of 3 agree 1.00
How many primary roads are there in Malé City?
mv-16
3 wrong_real Wrong 0 of 3 agree 0.00
Which walls have "Dhiraagu" or "Ooredoo" in their name, whatever the capitalization?
mv-17
1 wrong_real Wrong 0 of 3 agree 0.10
Which walls have "Dhiraagu" or "Ooredoo" in their name, whatever the capitalization?
mv-17
2 wrong_real Wrong 0 of 3 agree 0.10
Which walls have "Dhiraagu" or "Ooredoo" in their name, whatever the capitalization?
mv-17
3 wrong_real Wrong 0 of 3 agree 0.10
What are the members of the ferry route relations?
mv-18
1 query_error Wrong 0 of 3 agree 0.00
What are the members of the ferry route relations?
mv-18
2 wrong_real Wrong 0 of 3 agree 0.00
What are the members of the ferry route relations?
mv-18
3 wrong_real Wrong 0 of 3 agree 0.00
Which youth centres have a website?
mv-19
1 wrong_witness Right 0 of 3 agree 1.00
Which youth centres have a website?
mv-19
2 wrong_witness Right 0 of 3 agree 1.00
Which youth centres have a website?
mv-19
3 wrong_witness Right 0 of 3 agree 1.00

Provenance

What this run was graded with.

Package
geoql-env 0.1.0
Engine image
wiktorn/overpass-api:v0.7.62.11@sha256:5f643f2aa500333f19458fe8be14457215d6e0b9f891ca5c5ed452b1f8882c47
Tasks
0741a2bbbe847920eb5f20125d108298c8ade1c36a73d9e9fda9309d76b557ba
Gold answers
172a3c16405238a3cad35ddb83983a1444af7de66412829cd7faa2875bf36703
Tag counts
94db7a7656e2c9458ffdf0c8f1e3e8410dd68fed0c702005a0f73ff388240e73
Tag documentation
sha512-z2nQgHLNwCjebDxlKBaERjg02UmcJyemAtsXExJFlXNNc4K9l7EdtfkGs064Y0IgzUltmUxM/FeUBRyhb4kv6w==
Prompt
a6c16acc42ae863c666bab94f8dc4a6770cdeaed830fd495b7b192e7f29af961

Files

The grading holds the summary, every verdict and the provenance; the predictions are the run's responses, one JSON object per line.