geoql-env

Submit a run

Upload your model's predictions. They are graded on the 67 test tasks by the same grader as geoql-env score, and the run appears on the leaderboard with its own page. New here? Reproduce the benchmark with any model.

Predictions

One JSON object per line: {"task": id, "response": text}, optionally with "epoch" (1 to 3), "model", "mode" (single_shot or agentic) and "tools" (native or emulated). Every test task needs a prediction in every epoch. geoql-env export writes this file from an Inspect log.

Grading

The query in each response runs on the real world and on the task's witness worlds, after the server has checked its gold answers against the published hashes. Runs are graded one at a time; each person can have one run in grading.

What is published

The model name and link, your username, the predictions and every verdict, on the run's page and the leaderboard. A file can be submitted once.

Any model

Reproduce the benchmark

Everything this site runs is in the Space's repository: the package, the tasks, the gold hashes and the released runs. It needs Python 3.11 or newer, Docker, network access for the pinned downloads and about 2 GB of disk.

  1. Get the code

    Clone the Space and install the package with its pinned dependencies and the Inspect harness.

    git clone https://huggingface.co/spaces/Melikshah/geoql-env
cd geoql-env
python3 -m venv .venv
.venv/bin/pip install -c constraints.txt -e ".[inspect]"
  2. Build the worlds

    Start the pinned Overpass engine, download and build the 4 worlds and the tag documentation, then build every witness world and check every gold answer against the published hashes.

    .venv/bin/geoql-env engine up
.venv/bin/geoql-env worlds build
.venv/bin/geoql-env tagdocs build
.venv/bin/geoql-env gold
  3. Run your model

    Through Inspect, with any provider it supports and that provider's API key set: single-shot, one reply with one query, and agentic, at most 10 tool calls and then a submit, each for three epochs at the provider's default sampling.

    .venv/bin/inspect eval geoql_env/geoql_single_shot --model <provider>/<model> --epochs 3
.venv/bin/inspect eval geoql_env/geoql_agentic --model <provider>/<model> --epochs 3

    Another system can answer from each test task's prompt, shown on its task page with the conventions on the About page, or work as an agent with the same tools through geoql-env serve (HTTP) or geoql-env mcp <task> (MCP).

  4. Write the predictions

    One JSON line per task and epoch. geoql-env export writes them from an Inspect log; another system writes lines like {"task": "mv-01", "response": "...", "epoch": 1}.

    .venv/bin/geoql-env export logs/<log>.eval -o predictions.jsonl
  5. Check them

    Grade the file locally with the same grader as this site, on the real world and every witness world.

    .venv/bin/geoql-env score predictions.jsonl --split test
  6. Submit

    Sign in above and upload the file, one file per mode. It is graded on the test tasks and published with every verdict.