{
  "_comment": "Rendered by /leaderboard.html. Add rows ONLY from actual harness runs (oss/neurarch-arch-bench/leaderboard.mjs with LEADERBOARD_OUT). Never hand-edit numbers. Fields per row: model, passed, total, avgScore, date (YYYY-MM-DD), oracle (bool, optional), failureCategories (optional object {category: count}, copy from the board entry in the output JSON). RUBRIC: rows carry the rubric version they were measured under. v2 and earlier included the task `scale-under-budget`, which was vacuous (its start graph already satisfied every constraint, so an empty plan passed it). v3 added the minParams floor that task always needed. A v2 row therefore includes one free pass and is NOT comparable to a v3 row; re-run before publishing them side by side.",
  "updated": "2026-08-21",
  "sections": [
    {
      "title": "Curated split (12 tasks)",
      "note": "Hand-authored design-from-spec and repair tasks. The oracle row replays a known-good reference solution per task and proves every task is solvable; it is the ceiling, not a model. Rows below were measured under rubric v2, which included one vacuous task (fixed in rubric v3); v2 and v3 rows are not directly comparable.",
      "rows": [
        {
          "model": "reference",
          "oracle": true,
          "passed": 12,
          "total": 12,
          "avgScore": 75,
          "date": "2026-07-01",
          "rubricVersion": 2
        },
        {
          "model": "claude-sonnet-4-6",
          "passed": 7,
          "total": 12,
          "avgScore": 76,
          "date": "2026-07-08",
          "failureCategories": {
            "missing-layer-type": 3,
            "connectivity": 2,
            "low-score": 1,
            "over-budget": 1
          },
          "rubricVersion": 2
        },
        {
          "model": "grok-4",
          "passed": 10,
          "total": 12,
          "avgScore": 71,
          "date": "2026-07-20",
          "failureCategories": {
            "missing-layer-type": 3,
            "too-shallow": 2
          },
          "rubricVersion": 2
        },
        {
          "model": "gpt-4o",
          "passed": 8,
          "total": 12,
          "avgScore": 57,
          "date": "2026-07-24",
          "failureCategories": {
            "shape-mismatch": 3,
            "low-score": 3,
            "over-budget": 2,
            "missing-layer-type": 1,
            "too-shallow": 1
          },
          "rubricVersion": 2
        },
        {
          "model": "anthropic/claude-opus-5",
          "passed": 12,
          "total": 12,
          "avgScore": 84,
          "date": "2026-07-31"
        },
        {
          "model": "deepseek/deepseek-v4-pro",
          "passed": 7,
          "total": 12,
          "avgScore": 58,
          "date": "2026-07-31",
          "failureCategories": {
            "missing-layer-type": 4,
            "shape-mismatch": 3,
            "low-score": 3,
            "too-shallow": 3
          }
        },
        {
          "model": "google/gemini-3.1-pro-preview",
          "passed": 8,
          "total": 12,
          "avgScore": 72,
          "date": "2026-07-31",
          "failureCategories": {
            "missing-layer-type": 3,
            "shape-mismatch": 1,
            "low-score": 1
          }
        },
        {
          "model": "moonshotai/kimi-k3",
          "passed": 11,
          "total": 12,
          "avgScore": 80,
          "date": "2026-07-31",
          "failureCategories": {
            "missing-layer-type": 1
          }
        },
        {
          "model": "openai/gpt-5.5",
          "passed": 9,
          "total": 12,
          "avgScore": 78,
          "date": "2026-07-31",
          "failureCategories": {
            "missing-layer-type": 3
          }
        },
        {
          "model": "qwen/qwen3.7-max",
          "passed": 6,
          "total": 12,
          "avgScore": 52,
          "date": "2026-07-31",
          "failureCategories": {
            "missing-layer-type": 2,
            "shape-mismatch": 3,
            "low-score": 4,
            "orphan-node": 1
          }
        },
        {
          "model": "x-ai/grok-4.5",
          "passed": 9,
          "total": 12,
          "avgScore": 63,
          "date": "2026-07-31",
          "failureCategories": {
            "shape-mismatch": 3,
            "low-score": 3,
            "over-budget": 1,
            "missing-layer-type": 1
          }
        }
      ]
    },
    {
      "title": "Generated split (seed 7, 120 tasks)",
      "note": "Procedurally generated, contamination-resistant tasks across 10 families including edit-in-place repair (replace_model forbidden). Same seed reproduces the identical split. Rows below were measured under rubric v2, which included one vacuous task (fixed in rubric v3); v2 and v3 rows are not directly comparable.",
      "rows": [
        {
          "model": "reference",
          "oracle": true,
          "passed": 24,
          "total": 24,
          "avgScore": 77,
          "date": "2026-07-03",
          "rubricVersion": 2
        },
        {
          "model": "grok-4",
          "passed": 88,
          "total": 120,
          "avgScore": 67,
          "date": "2026-07-20",
          "failureCategories": {
            "too-shallow": 15,
            "missing-layer-type": 13,
            "other": 9,
            "parse-error": 6,
            "connectivity": 4,
            "low-score": 2
          },
          "rubricVersion": 2
        },
        {
          "model": "anthropic/claude-opus-5",
          "passed": 117,
          "total": 120,
          "avgScore": 82,
          "date": "2026-07-31",
          "failureCategories": {
            "shape-mismatch": 3,
            "low-score": 3
          }
        }
      ]
    },
    {
      "kind": "grounded",
      "title": "Grounded split (6 tasks, trained on T4 through the production trainer)",
      "note": "Every statically-passing design plus the audited human reference is trained for real with identical per-task budgets and configs. \"vs reference\" is the trained metric relative to the reference design; it is NOT a SOTA comparison. Method and limits: the grounded report in the benchmark repo.",
      "rows": [
        {
          "model": "reference",
          "oracle": true,
          "trained": 6,
          "total": 6,
          "medianVsRef": 100,
          "worstVsRef": 100,
          "date": "2026-08-21"
        },
        {
          "model": "grok-3",
          "trained": 6,
          "total": 6,
          "medianVsRef": 103,
          "worstVsRef": 30,
          "date": "2026-08-21"
        },
        {
          "model": "claude-sonnet-4.6",
          "trained": 6,
          "total": 6,
          "medianVsRef": 100,
          "worstVsRef": 11,
          "date": "2026-08-21"
        },
        {
          "model": "gpt-oss-120b",
          "trained": 6,
          "total": 6,
          "medianVsRef": 102,
          "worstVsRef": 17,
          "date": "2026-08-21"
        }
      ]
    }
  ]
}