Resources / Datasets
Design selection replay: two trained campaigns and every selector tried
The records needed to score a design selector offline, in the shape of xRouteBench: every candidate design was trained once and the outcome kept, so a rule that picks which unexecuted design gets the GPU is evaluated by replay against the same rows as every previous rule, with no GPU and no model call. Two campaigns, 24 in-sample and 15 held-out designs, each row carrying what a selector may see (task, designer, static verifier score, parameter count) and what happened (validation accuracy or loss, GPU seconds). A verdicts file holds, per held-out pair and per frontier model that judged it from the exported code, which design the model picked and whether it was right. The board file is the current standing: the static score, the score with a size tie-break, smallest-first, largest-first, and two code-reading judges, each with pairwise accuracy, coverage, and the share of selection headroom captured.
Get it
| In-sample campaign rows | neurarch.com/selection-replay/in-sample.jsonl |
|---|---|
| Held-out campaign rows | neurarch.com/selection-replay/heldout.jsonl |
| Judge verdicts per pair | neurarch.com/selection-replay/judge-verdicts.jsonl |
| Selector board | neurarch.com/selection-replay/board.json |
| Hugging Face mirror (dataset viewer, load_dataset) | huggingface.co/datasets/neurarch-ai/arch-selection-replay |
curl -sO https://neurarch.com/selection-replay/heldout.jsonl
jq -s 'group_by(.task) | map({task: .[0].task, best: (max_by(.value) | .designer)})' heldout.jsonlWhat is in a row
What this dataset is not
Thirty-nine designs on eleven small tasks, trained for a few epochs on one T4; the papers this is measured against have 18,438 pairs. The graphs and the exported code are deliberately not included, so a code-reading judge cannot be re-run from here, only replayed from its recorded verdicts; a selector under the published contract (task, designer, score, params) can. The static verifier decides too few pairs on either campaign for its accuracy to be quoted, and out of sample the rule "always pick the larger design" matches one of the two frontier judges, so no row on the board is evidence that any method beats size on this data.
Licence and citation
Released under Creative Commons Attribution 4.0. Cite it as:
Neurarch. Design selection replay: two trained campaigns and every selector tried. https://neurarch.com/d/selection-replay.html