Models / t5
t5-small
Reconstructed from its own config.json
with no weights read. 23.4M downloads on Hugging Face.
Our count against the checkpoint
The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.
Derived from structure
60.6M
60,576,000 parameters
Delta
+0.11%
What it costs to run
Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.
Layers
70
Will it forward-pass
Yes
Priced on
A10G (24GB)
Est. one run
$1.14
| Card | Memory | |
|---|---|---|
| T4 (16GB) | weights + activations | fits |
| A100 (40GB) | weights + activations | fits |
| H100 (80GB) | weights + activations | fits |
Structure
72 nodes. Output shapes are propagated from the input shape, batch dimension excluded.
| Layer | Type | Output shape | |
|---|---|---|---|
| 1 | Input | Input | 1 × 512 |
| 2 | Encoder_Embedding | Embedding | 1 × 512 × 512 |
| 3 | Relative_Position_Bias | Relative Position Bias | 1 × 512 × 512 |
| 4 | Enc_Attn_1 | Multi-Head Attention | 1 × 512 × 512 |
| 5 | Enc_Add_1 | Add | 1 × 512 × 512 |
| 6 | Enc_Norm_1 | LayerNorm | 1 × 512 × 512 |
| 7 | Enc_FFN_1 | Feed Forward | 1 × 512 × 512 |
| 8 | Enc_Attn_2 | Multi-Head Attention | 1 × 512 × 512 |
| 9 | Enc_Add_2 | Add | 1 × 512 × 512 |
| 10 | Enc_Norm_2 | LayerNorm | 1 × 512 × 512 |
| 11 | Enc_FFN_2 | Feed Forward | 1 × 512 × 512 |
| 12 | Enc_Attn_3 | Multi-Head Attention | 1 × 512 × 512 |
| 13 | Enc_Add_3 | Add | 1 × 512 × 512 |
| 14 | Enc_Norm_3 | LayerNorm | 1 × 512 × 512 |
| 15 | Enc_FFN_3 | Feed Forward | 1 × 512 × 512 |
| 16 | Enc_Attn_4 | Multi-Head Attention | 1 × 512 × 512 |
| 17 | Enc_Add_4 | Add | 1 × 512 × 512 |
| 18 | Enc_Norm_4 | LayerNorm | 1 × 512 × 512 |
| 19 | Enc_FFN_4 | Feed Forward | 1 × 512 × 512 |
| 20 | Enc_Attn_5 | Multi-Head Attention | 1 × 512 × 512 |
| 21 | Enc_Add_5 | Add | 1 × 512 × 512 |
| 22 | Enc_Norm_5 | LayerNorm | 1 × 512 × 512 |
| 23 | Enc_FFN_5 | Feed Forward | 1 × 512 × 512 |
| 24 | Enc_Attn_6 | Multi-Head Attention | 1 × 512 × 512 |
| 25 | Enc_Add_6 | Add | 1 × 512 × 512 |
| 26 | Enc_Norm_6 | LayerNorm | 1 × 512 × 512 |
| 27 | Enc_FFN_6 | Feed Forward | 1 × 512 × 512 |
| 28 | Decoder_Embedding | Embedding | 1 × 512 × 512 |
| 29 | Dec_SelfAttn_1 | Multi-Head Attention | 1 × 512 × 512 |
| 30 | Dec_Add_1 | Add | 1 × 512 × 512 |
| 31 | Dec_Norm1_1 | LayerNorm | 1 × 512 × 512 |
| 32 | Dec_CrossAttn_1 | Cross-Attention | 1 × 512 × 512 |
| 33 | Dec_Add2_1 | Add | 1 × 512 × 512 |
| 34 | Dec_Norm2_1 | LayerNorm | 1 × 512 × 512 |
| 35 | Dec_FFN_1 | Feed Forward | 1 × 512 × 512 |
| 36 | Dec_SelfAttn_2 | Multi-Head Attention | 1 × 512 × 512 |
| 37 | Dec_Add_2 | Add | 1 × 512 × 512 |
| 38 | Dec_Norm1_2 | LayerNorm | 1 × 512 × 512 |
| 39 | Dec_CrossAttn_2 | Cross-Attention | 1 × 512 × 512 |
| 40 | Dec_Add2_2 | Add | 1 × 512 × 512 |
| 41 | Dec_Norm2_2 | LayerNorm | 1 × 512 × 512 |
| 42 | Dec_FFN_2 | Feed Forward | 1 × 512 × 512 |
| 43 | Dec_SelfAttn_3 | Multi-Head Attention | 1 × 512 × 512 |
| 44 | Dec_Add_3 | Add | 1 × 512 × 512 |
| 45 | Dec_Norm1_3 | LayerNorm | 1 × 512 × 512 |
| 46 | Dec_CrossAttn_3 | Cross-Attention | 1 × 512 × 512 |
| 47 | Dec_Add2_3 | Add | 1 × 512 × 512 |
| 48 | Dec_Norm2_3 | LayerNorm | 1 × 512 × 512 |
| 49 | Dec_FFN_3 | Feed Forward | 1 × 512 × 512 |
| 50 | Dec_SelfAttn_4 | Multi-Head Attention | 1 × 512 × 512 |
| 51 | Dec_Add_4 | Add | 1 × 512 × 512 |
| 52 | Dec_Norm1_4 | LayerNorm | 1 × 512 × 512 |
| 53 | Dec_CrossAttn_4 | Cross-Attention | 1 × 512 × 512 |
| 54 | Dec_Add2_4 | Add | 1 × 512 × 512 |
| 55 | Dec_Norm2_4 | LayerNorm | 1 × 512 × 512 |
| 56 | Dec_FFN_4 | Feed Forward | 1 × 512 × 512 |
| 57 | Dec_SelfAttn_5 | Multi-Head Attention | 1 × 512 × 512 |
| 58 | Dec_Add_5 | Add | 1 × 512 × 512 |
| 59 | Dec_Norm1_5 | LayerNorm | 1 × 512 × 512 |
| 60 | Dec_CrossAttn_5 | Cross-Attention | 1 × 512 × 512 |
| 61 | Dec_Add2_5 | Add | 1 × 512 × 512 |
| 62 | Dec_Norm2_5 | LayerNorm | 1 × 512 × 512 |
| 63 | Dec_FFN_5 | Feed Forward | 1 × 512 × 512 |
| 64 | Dec_SelfAttn_6 | Multi-Head Attention | 1 × 512 × 512 |
| 65 | Dec_Add_6 | Add | 1 × 512 × 512 |
| 66 | Dec_Norm1_6 | LayerNorm | 1 × 512 × 512 |
| 67 | Dec_CrossAttn_6 | Cross-Attention | 1 × 512 × 512 |
| 68 | Dec_Add2_6 | Add | 1 × 512 × 512 |
| 69 | Dec_Norm2_6 | LayerNorm | 1 × 512 × 512 |
| 70 | Dec_FFN_6 | Feed Forward | 1 × 512 × 512 |
| 71 | LM_Head | Linear | 1 × 512 × 32128 |
| 72 | Output | Output | 1 × 512 × 32128 |
What the verifier says
infoAt 12 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init
deep-attention-default-init
Do this to your own model
Same numbers, on a model in your repo, in one command. No account.
pip install neurarch-trace
neurarch-trace google-t5/t5-small --plan --share