Comparisons / Transformer Block vs LLaMA-3 Block
Transformer Block vs LLaMA-3 Block
What changed in the transformer block between 2017 and now.
LLaMA-3 Block has 695M more parameters than Transformer Block: 5 layers added, 3 removed, 2 changed.
Transformer Block
- Layers
- 6
- Parameters
- 7.1M
- Input
- 512 × 768
- Output
- 512 × 768
- Forward-passes
- yes
- Est. train cost
- $0.185
LLaMA-3 Block
- Layers
- 8
- Parameters
- 703M
- Input
- 1 × 2048
- Output
- 1 × 2048 × 4096
- Forward-passes
- yes
- Est. train cost
- $14.27
The deltas
Every number is LLaMA-3 Block relative to Transformer Block.
Which GPUs each one fits
Each side is measured at its own declared input (512 × 768 against 1 × 2048). Both columns are right about their own model; the difference between them is not a fact about the designs.
| GPU | Transformer Block | LLaMA-3 Block |
|---|---|---|
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |
Layer by layer
Aligned in topological order. 3 of 13 rows are the same layer with the same parameters.
Hide all 13 rows
| Transformer Block | LLaMA-3 Block | ||||||
|---|---|---|---|---|---|---|---|
| Layer | Params | Output | Layer | Params | Output | ||
| 1 | changed shape | Input Input | 512 × 768 | tokens Input | 1 × 2048 | ||
| 2 | removed | MultiHeadAttention Multi Head Attention | 2.4M | 512 × 768 | — | ||
| 3 | added | — | rope Rope | ||||
| 4 | added | — | embed Embedding | 525M | 1 × 2048 × 4096 | ||
| 5 | added | — | attn_norm Rms Norm | 4.1K | 1 × 2048 × 4096 | ||
| 6 | added | — | gqa Grouped Query Attention | 42M | 1 × 2048 × 4096 | ||
| 7 | same | Add_1 Add | 512 × 768 | residual_1 Add | 1 × 2048 × 4096 | ||
| 8 | changed type, normalizedShape | LayerNorm_1 Layer Norm | 512 × 768 | ffn_norm Rms Norm | 4.1K | 1 × 2048 × 4096 | |
| 9 | removed | FeedForward Feed Forward | 4.7M | 512 × 768 | — | ||
| 10 | added | — | swiglu_ffn Swiglu | 135M | 1 × 2048 × 4096 | ||
| 11 | same | Add_2 Add | 512 × 768 | residual_2 Add | 1 × 2048 × 4096 | ||
| 12 | removed | LayerNorm_2 Layer Norm | 512 × 768 | — | |||
| 13 | same | Output Output | 512 × 768 | hidden_state Output | 1 × 2048 × 4096 | ||
What this is not
- The two are priced at different declared inputs (512 × 768 against 1 × 2048), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.
Take it further
Open either graph in the editor, change it, and check it again: Transformer Block · LLaMA-3 Block
Compare any two models of your own, including anything on Hugging Face: the comparison tool.
Machine-readable: this page as markdown ·
the pair index ·
POST https://www.neurarch.com/api/v1/plan for a graph of your own.