N Neurarch Architectures Checks Data Docs Open the app

Resources / Datasets

Structure, verdict and trained outcome triples

The corpus that pairs what a design looks like with what it did. Each row carries a structural fingerprint of a graph (task family, parameter count, component count), the static verifier verdict on it (score and blockers), and the outcome of actually training it: the full loss curve, best and final loss, area under the loss curve, step count, and whether it converged. Every graph was trained through the same managed-GPU path the product uses, so the outcome is measured rather than simulated.

80 trained graphs MIT Free to use Source repo On Hugging Face

Get it

Triplesraw.githubusercontent.com/neurarch-ai/neurarch-arch-bench/main/triples.jsonl
application/x-ndjson
Hugging Face mirror (dataset viewer, load_dataset)huggingface.co/datasets/neurarch-ai/arch-outcome-triples
text/html
 
curl -sO https://raw.githubusercontent.com/neurarch-ai/neurarch-arch-bench/main/triples.jsonl
jq -s 'map({score: .verifier.score, bestLoss: .outcome.bestLoss})' triples.jsonl

What is in a row

task family parameter count component count verifier score blockers loss curve best loss final loss steps converged

What this dataset is not

n is 80, and the headline finding is negative: across families the correlation between verifier score and achievable quality is about zero (rho -0.02), undefined inside four families whose graphs score identically, and positive only where the score varies. It is published because that is the result. The verifier tells you whether a graph runs, not how good it will get, and this is the file that shows it.

Licence and citation

Released under MIT License. Cite it as:

Neurarch. Structure, verdict and trained outcome triples. https://neurarch.com/d/outcome-triples.html

The rest of the set

Neurarch architecture corpus
36 neural network architectures kept as typed graphs rather than diagrams.
36 architectures · CC0-1.0
Neurarch structural check catalogue
The 41 structural checks Neurarch runs on a model graph, as data: id, severity, category, the condition that triggers it, why it costs something, and the fix.
41 checks · CC-BY-4.0
Verifier grounding study (264 graphs)
Clean reference architectures plus systematically corrupted variants (broken attention head divisibility, linear width mismatches, severed connections), each built as a real PyTorch model and run on a GPU.
264 graphs · MIT
Arch-Bench arena results
Frontier language models scored on design-from-spec tasks by a deterministic verifier rather than a human or an LLM judge.
18 model-split results · CC-BY-4.0
Arch-Bench task set
The task definitions behind the benchmark: design-from-spec and repair-and-extend instances for agents that build neural network architectures.
12 curated tasks, 8 fixtures · MIT
arch-design-sft: verified architecture-design SFT data
Supervised fine-tuning data for neural architecture design treated as structured graph editing.
3,010 verified examples · MIT
Verified architecture-design reasoning traces (Claude)
Spec to reasoning to design triples where the design is re-graded by the same deterministic verifier the benchmark uses, and only passing traces are kept.
306 verified traces · MIT
Verified architecture-design reasoning traces (Grok)
The same verified spec to reasoning to design triples as the Claude split, rejection-sampled from a different frontier model, so the two can be pooled for volume or held apart to see how much of the reasoning style is model-specific.
376 verified traces · MIT