How fast, and how cheap, will your model serve?

Decode speed is a memory-bandwidth question, not a TFLOPS question: every generated token streams the weights and the KV cache through HBM. This page computes the numbers that actually decide your serving bill: KV cache per token (MHA vs GQA vs MLA), the critical batch size, and the latency-throughput tradeoff, from the same roofline math the Neurarch design canvas uses.

Attention Layers d_model Heads KV heads FFN dim Vocab Context Batch Precision GPUs / replica

KV cache per token: the number MLA exists to shrink

Decode throughput by accelerator

The latency-throughput tradeoff

Each point is a batch size (1, 2, 4, ... 512). Lines stop where the batch's KV cache no longer fits the accelerator. The ringed point is the critical batch, where the MLP side turns compute-bound and extra batching stops buying throughput. Doubling per-token latency can cut cost per token by an order of magnitude: that is the whole economics of serving in one curve.

These numbers come from your architecture's structure. Design the structure itself in Neurarch: the same roofline runs live on the canvas, per layer, as you edit, with GQA/MLA one parameter away.