Models / clip
CLIP-ViT-L-14-laion2B-s32B-b82K
Reconstructed from its own config.json
with no weights read. 3.7M downloads on Hugging Face.
Our count against the checkpoint
The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.
Derived from structure
303M
302,835,712 parameters
Delta
-29.2%
multi-tower The config declares 2 sub-models (text_config, vision_config). The published checkpoint carries all of them; the graph below carries the towers the reader reconstructs. A gap here is a statement about what we read, not about the checkpoint.
What it costs to run
Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.
Layers
146
Will it forward-pass
Yes
Priced on
A10G (24GB)
Est. one run
$3.17
| Card | Memory | |
|---|---|---|
| T4 (16GB) | weights + activations | fits |
| A100 (40GB) | weights + activations | fits |
| H100 (80GB) | weights + activations | fits |
Structure
148 nodes. Output shapes are propagated from the input shape, batch dimension excluded.
| Layer | Type | Output shape | |
|---|---|---|---|
| 1 | Input | Input | 1 × 3 × 224 × 224 |
| 2 | PatchEmbedding | Embedding | 1 × 3 × 224 × 224 × 1024 |
| 3 | Patch_Position_Embedding | Learned Pos Embed | 1 × 3 × 224 × 224 × 1024 |
| 4 | Attention_1 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 5 | Add_1 | Add | 1 × 3 × 224 × 224 × 1024 |
| 6 | LayerNorm_1_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 7 | FFN_1 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 8 | Add_1_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 9 | LayerNorm_1_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 10 | Attention_2 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 11 | Add_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 12 | LayerNorm_2_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 13 | FFN_2 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 14 | Add_2_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 15 | LayerNorm_2_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 16 | Attention_3 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 17 | Add_3 | Add | 1 × 3 × 224 × 224 × 1024 |
| 18 | LayerNorm_3_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 19 | FFN_3 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 20 | Add_3_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 21 | LayerNorm_3_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 22 | Attention_4 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 23 | Add_4 | Add | 1 × 3 × 224 × 224 × 1024 |
| 24 | LayerNorm_4_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 25 | FFN_4 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 26 | Add_4_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 27 | LayerNorm_4_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 28 | Attention_5 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 29 | Add_5 | Add | 1 × 3 × 224 × 224 × 1024 |
| 30 | LayerNorm_5_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 31 | FFN_5 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 32 | Add_5_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 33 | LayerNorm_5_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 34 | Attention_6 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 35 | Add_6 | Add | 1 × 3 × 224 × 224 × 1024 |
| 36 | LayerNorm_6_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 37 | FFN_6 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 38 | Add_6_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 39 | LayerNorm_6_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 40 | Attention_7 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 41 | Add_7 | Add | 1 × 3 × 224 × 224 × 1024 |
| 42 | LayerNorm_7_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 43 | FFN_7 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 44 | Add_7_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 45 | LayerNorm_7_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 46 | Attention_8 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 47 | Add_8 | Add | 1 × 3 × 224 × 224 × 1024 |
| 48 | LayerNorm_8_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 49 | FFN_8 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 50 | Add_8_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 51 | LayerNorm_8_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 52 | Attention_9 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 53 | Add_9 | Add | 1 × 3 × 224 × 224 × 1024 |
| 54 | LayerNorm_9_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 55 | FFN_9 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 56 | Add_9_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 57 | LayerNorm_9_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 58 | Attention_10 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 59 | Add_10 | Add | 1 × 3 × 224 × 224 × 1024 |
| 60 | LayerNorm_10_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 61 | FFN_10 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 62 | Add_10_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 63 | LayerNorm_10_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 64 | Attention_11 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 65 | Add_11 | Add | 1 × 3 × 224 × 224 × 1024 |
| 66 | LayerNorm_11_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 67 | FFN_11 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 68 | Add_11_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 69 | LayerNorm_11_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 70 | Attention_12 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 71 | Add_12 | Add | 1 × 3 × 224 × 224 × 1024 |
| 72 | LayerNorm_12_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 73 | FFN_12 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 74 | Add_12_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 75 | LayerNorm_12_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 76 | Attention_13 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 77 | Add_13 | Add | 1 × 3 × 224 × 224 × 1024 |
| 78 | LayerNorm_13_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 79 | FFN_13 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 80 | Add_13_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 81 | LayerNorm_13_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 82 | Attention_14 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 83 | Add_14 | Add | 1 × 3 × 224 × 224 × 1024 |
| 84 | LayerNorm_14_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 85 | FFN_14 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 86 | Add_14_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 87 | LayerNorm_14_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 88 | Attention_15 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 89 | Add_15 | Add | 1 × 3 × 224 × 224 × 1024 |
| 90 | LayerNorm_15_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 91 | FFN_15 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 92 | Add_15_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 93 | LayerNorm_15_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 94 | Attention_16 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 95 | Add_16 | Add | 1 × 3 × 224 × 224 × 1024 |
| 96 | LayerNorm_16_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 97 | FFN_16 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 98 | Add_16_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 99 | LayerNorm_16_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 100 | Attention_17 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 101 | Add_17 | Add | 1 × 3 × 224 × 224 × 1024 |
| 102 | LayerNorm_17_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 103 | FFN_17 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 104 | Add_17_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 105 | LayerNorm_17_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 106 | Attention_18 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 107 | Add_18 | Add | 1 × 3 × 224 × 224 × 1024 |
| 108 | LayerNorm_18_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 109 | FFN_18 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 110 | Add_18_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 111 | LayerNorm_18_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 112 | Attention_19 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 113 | Add_19 | Add | 1 × 3 × 224 × 224 × 1024 |
| 114 | LayerNorm_19_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 115 | FFN_19 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 116 | Add_19_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 117 | LayerNorm_19_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 118 | Attention_20 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 119 | Add_20 | Add | 1 × 3 × 224 × 224 × 1024 |
| 120 | LayerNorm_20_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 121 | FFN_20 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 122 | Add_20_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 123 | LayerNorm_20_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 124 | Attention_21 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 125 | Add_21 | Add | 1 × 3 × 224 × 224 × 1024 |
| 126 | LayerNorm_21_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 127 | FFN_21 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 128 | Add_21_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 129 | LayerNorm_21_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 130 | Attention_22 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 131 | Add_22 | Add | 1 × 3 × 224 × 224 × 1024 |
| 132 | LayerNorm_22_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 133 | FFN_22 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 134 | Add_22_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 135 | LayerNorm_22_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 136 | Attention_23 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 137 | Add_23 | Add | 1 × 3 × 224 × 224 × 1024 |
| 138 | LayerNorm_23_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 139 | FFN_23 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 140 | Add_23_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 141 | LayerNorm_23_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 142 | Attention_24 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 143 | Add_24 | Add | 1 × 3 × 224 × 224 × 1024 |
| 144 | LayerNorm_24_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 145 | FFN_24 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 146 | Add_24_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 147 | LayerNorm_24_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 148 | Output | Output | 1 × 3 × 224 × 224 × 1024 |
What the verifier says
warn"LayerNorm_24_2" (layerNorm) is the last layer before Output. Normalizing the raw logits constrains the output range and breaks standard loss functions. Fix: Move normalization before the final Linear/Conv layer.
bn-at-output
bn-at-output
infoAt 24 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init
deep-attention-default-init
Do this to your own model
Same numbers, on a model in your repo, in one command. No account.
pip install neurarch-trace
neurarch-trace laion/CLIP-ViT-L-14-laion2B-s32B-b82K --plan --share