Comparisons / Qwen3-8B vs Llama-4 Scout
Qwen3-8B vs Llama-4 Scout
Two 2026 open models with different attention budgets.
Llama-4 Scout has 94B more parameters than Qwen3-8B: 249 layers added, 36 removed, 112 changed.
Baseline
Qwen3-8B
- Layers
- 219
- Parameters
- 8.2B
- Input
- 1 × 40960
- Output
- 1 × 40960 × 151936
- Forward-passes
- yes
- Est. train cost
- $20,863
Compared
Llama-4 Scout
- Layers
- 431
- Parameters
- 103B
- Input
- 1 × 10485760
- Output
- 1 × 10486336 × 202048
- Forward-passes
- yes
- Est. train cost
- $1,015,410,581
The deltas
Every number is Llama-4 Scout relative to Qwen3-8B.
Parameters
+94B (13× the size)
Layers
+212
Added
249
Removed
36
Changed
112
Unchanged
73
Which GPUs each one fits
Each side is measured at its own declared input (1 × 40960 against 1 × 10485760). Both columns are right about their own model; the difference between them is not a fact about the designs.
| GPU | Qwen3-8B | Llama-4 Scout |
|---|---|---|
| T4 16GB | no | no |
| A100 40GB | no | no |
| H100 80GB | no | no |
Layer by layer
Aligned in topological order. 73 of 470 rows are the same layer with the same parameters.
Show all 470 rows
| Qwen3-8B | Llama-4 Scout | ||||||
|---|---|---|---|---|---|---|---|
| Layer | Params | Output | Layer | Params | Output | ||
| 1 | added | — | Input Input | 1 × 10485760 | |||
| 2 | changed shape | Input Input | 1 × 40960 | Vision input Input | 3 × 336 × 336 | ||
| 3 | changed vocabSize, embeddingDim, maxSeqLen | Embedding Embedding | 622M | 1 × 40960 × 4096 | Embedding Embedding | 1.0B | 1 × 10485760 × 5120 |
| 4 | added | — | PatchEmbed Patch Embed | 829K | 576 × 1408 | ||
| 5 | added | — | Vision_LN_1 Layer Norm | 2.8K | 576 × 1408 | ||
| 6 | added | — | Vision_Attn_1 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 7 | added | — | Vision_Add_1 Add | 576 × 1408 | |||
| 8 | added | — | Vision_FFN_1 Feed Forward | 16M | 576 × 1408 | ||
| 9 | added | — | Vision_LN_2 Layer Norm | 2.8K | 576 × 1408 | ||
| 10 | added | — | Vision_Attn_2 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 11 | added | — | Vision_Add_2 Add | 576 × 1408 | |||
| 12 | added | — | Vision_FFN_2 Feed Forward | 16M | 576 × 1408 | ||
| 13 | added | — | Vision_LN_3 Layer Norm | 2.8K | 576 × 1408 | ||
| 14 | added | — | Vision_Attn_3 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 15 | added | — | Vision_Add_3 Add | 576 × 1408 | |||
| 16 | added | — | Vision_FFN_3 Feed Forward | 16M | 576 × 1408 | ||
| 17 | added | — | Vision_LN_4 Layer Norm | 2.8K | 576 × 1408 | ||
| 18 | added | — | Vision_Attn_4 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 19 | added | — | Vision_Add_4 Add | 576 × 1408 | |||
| 20 | added | — | Vision_FFN_4 Feed Forward | 16M | 576 × 1408 | ||
| 21 | added | — | Vision_LN_5 Layer Norm | 2.8K | 576 × 1408 | ||
| 22 | added | — | Vision_Attn_5 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 23 | added | — | Vision_Add_5 Add | 576 × 1408 | |||
| 24 | added | — | Vision_FFN_5 Feed Forward | 16M | 576 × 1408 | ||
| 25 | added | — | Vision_LN_6 Layer Norm | 2.8K | 576 × 1408 | ||
| 26 | added | — | Vision_Attn_6 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 27 | added | — | Vision_Add_6 Add | 576 × 1408 | |||
| 28 | added | — | Vision_FFN_6 Feed Forward | 16M | 576 × 1408 | ||
| 29 | added | — | Vision_LN_7 Layer Norm | 2.8K | 576 × 1408 | ||
| 30 | added | — | Vision_Attn_7 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 31 | added | — | Vision_Add_7 Add | 576 × 1408 | |||
| 32 | added | — | Vision_FFN_7 Feed Forward | 16M | 576 × 1408 | ||
| 33 | added | — | Vision_LN_8 Layer Norm | 2.8K | 576 × 1408 | ||
| 34 | added | — | Vision_Attn_8 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 35 | added | — | Vision_Add_8 Add | 576 × 1408 | |||
| 36 | added | — | Vision_FFN_8 Feed Forward | 16M | 576 × 1408 | ||
| 37 | added | — | Vision_LN_9 Layer Norm | 2.8K | 576 × 1408 | ||
| 38 | added | — | Vision_Attn_9 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 39 | added | — | Vision_Add_9 Add | 576 × 1408 | |||
| 40 | added | — | Vision_FFN_9 Feed Forward | 16M | 576 × 1408 | ||
| 41 | added | — | Vision_LN_10 Layer Norm | 2.8K | 576 × 1408 | ||
| 42 | added | — | Vision_Attn_10 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 43 | added | — | Vision_Add_10 Add | 576 × 1408 | |||
| 44 | added | — | Vision_FFN_10 Feed Forward | 16M | 576 × 1408 | ||
| 45 | added | — | Vision_LN_11 Layer Norm | 2.8K | 576 × 1408 | ||
| 46 | added | — | Vision_Attn_11 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 47 | added | — | Vision_Add_11 Add | 576 × 1408 | |||
| 48 | added | — | Vision_FFN_11 Feed Forward | 16M | 576 × 1408 | ||
| 49 | added | — | Vision_LN_12 Layer Norm | 2.8K | 576 × 1408 | ||
| 50 | added | — | Vision_Attn_12 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 51 | added | — | Vision_Add_12 Add | 576 × 1408 | |||
| 52 | added | — | Vision_FFN_12 Feed Forward | 16M | 576 × 1408 | ||
| 53 | added | — | Vision_LN_13 Layer Norm | 2.8K | 576 × 1408 | ||
| 54 | added | — | Vision_Attn_13 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 55 | added | — | Vision_Add_13 Add | 576 × 1408 | |||
| 56 | added | — | Vision_FFN_13 Feed Forward | 16M | 576 × 1408 | ||
| 57 | added | — | Vision_LN_14 Layer Norm | 2.8K | 576 × 1408 | ||
| 58 | added | — | Vision_Attn_14 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 59 | added | — | Vision_Add_14 Add | 576 × 1408 | |||
| 60 | added | — | Vision_FFN_14 Feed Forward | 16M | 576 × 1408 | ||
| 61 | added | — | Vision_LN_15 Layer Norm | 2.8K | 576 × 1408 | ||
| 62 | added | — | Vision_Attn_15 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 63 | added | — | Vision_Add_15 Add | 576 × 1408 | |||
| 64 | added | — | Vision_FFN_15 Feed Forward | 16M | 576 × 1408 | ||
| 65 | added | — | Vision_LN_16 Layer Norm | 2.8K | 576 × 1408 | ||
| 66 | added | — | Vision_Attn_16 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 67 | added | — | Vision_Add_16 Add | 576 × 1408 | |||
| 68 | added | — | Vision_FFN_16 Feed Forward | 16M | 576 × 1408 | ||
| 69 | added | — | Vision_LN_17 Layer Norm | 2.8K | 576 × 1408 | ||
| 70 | added | — | Vision_Attn_17 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 71 | added | — | Vision_Add_17 Add | 576 × 1408 | |||
| 72 | added | — | Vision_FFN_17 Feed Forward | 16M | 576 × 1408 | ||
| 73 | added | — | Vision_LN_18 Layer Norm | 2.8K | 576 × 1408 | ||
| 74 | added | — | Vision_Attn_18 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 75 | added | — | Vision_Add_18 Add | 576 × 1408 | |||
| 76 | added | — | Vision_FFN_18 Feed Forward | 16M | 576 × 1408 | ||
| 77 | added | — | Vision_LN_19 Layer Norm | 2.8K | 576 × 1408 | ||
| 78 | added | — | Vision_Attn_19 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 79 | added | — | Vision_Add_19 Add | 576 × 1408 | |||
| 80 | added | — | Vision_FFN_19 Feed Forward | 16M | 576 × 1408 | ||
| 81 | added | — | Vision_LN_20 Layer Norm | 2.8K | 576 × 1408 | ||
| 82 | added | — | Vision_Attn_20 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 83 | added | — | Vision_Add_20 Add | 576 × 1408 | |||
| 84 | added | — | Vision_FFN_20 Feed Forward | 16M | 576 × 1408 | ||
| 85 | added | — | Vision_LN_21 Layer Norm | 2.8K | 576 × 1408 | ||
| 86 | added | — | Vision_Attn_21 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 87 | added | — | Vision_Add_21 Add | 576 × 1408 | |||
| 88 | added | — | Vision_FFN_21 Feed Forward | 16M | 576 × 1408 | ||
| 89 | added | — | Vision_LN_22 Layer Norm | 2.8K | 576 × 1408 | ||
| 90 | added | — | Vision_Attn_22 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 91 | added | — | Vision_Add_22 Add | 576 × 1408 | |||
| 92 | added | — | Vision_FFN_22 Feed Forward | 16M | 576 × 1408 | ||
| 93 | added | — | Vision_LN_23 Layer Norm | 2.8K | 576 × 1408 | ||
| 94 | added | — | Vision_Attn_23 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 95 | added | — | Vision_Add_23 Add | 576 × 1408 | |||
| 96 | added | — | Vision_FFN_23 Feed Forward | 16M | 576 × 1408 | ||
| 97 | added | — | Vision_LN_24 Layer Norm | 2.8K | 576 × 1408 | ||
| 98 | added | — | Vision_Attn_24 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 99 | added | — | Vision_Add_24 Add | 576 × 1408 | |||
| 100 | added | — | Vision_FFN_24 Feed Forward | 16M | 576 × 1408 | ||
| 101 | added | — | Vision_LN_25 Layer Norm | 2.8K | 576 × 1408 | ||
| 102 | added | — | Vision_Attn_25 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 103 | added | — | Vision_Add_25 Add | 576 × 1408 | |||
| 104 | added | — | Vision_FFN_25 Feed Forward | 16M | 576 × 1408 | ||
| 105 | added | — | Vision_LN_26 Layer Norm | 2.8K | 576 × 1408 | ||
| 106 | added | — | Vision_Attn_26 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 107 | added | — | Vision_Add_26 Add | 576 × 1408 | |||
| 108 | added | — | Vision_FFN_26 Feed Forward | 16M | 576 × 1408 | ||
| 109 | added | — | Vision_LN_27 Layer Norm | 2.8K | 576 × 1408 | ||
| 110 | added | — | Vision_Attn_27 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 111 | added | — | Vision_Add_27 Add | 576 × 1408 | |||
| 112 | added | — | Vision_FFN_27 Feed Forward | 16M | 576 × 1408 | ||
| 113 | added | — | Vision_LN_28 Layer Norm | 2.8K | 576 × 1408 | ||
| 114 | added | — | Vision_Attn_28 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 115 | added | — | Vision_Add_28 Add | 576 × 1408 | |||
| 116 | added | — | Vision_FFN_28 Feed Forward | 16M | 576 × 1408 | ||
| 117 | added | — | Vision_LN_29 Layer Norm | 2.8K | 576 × 1408 | ||
| 118 | added | — | Vision_Attn_29 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 119 | added | — | Vision_Add_29 Add | 576 × 1408 | |||
| 120 | added | — | Vision_FFN_29 Feed Forward | 16M | 576 × 1408 | ||
| 121 | added | — | Vision_LN_30 Layer Norm | 2.8K | 576 × 1408 | ||
| 122 | added | — | Vision_Attn_30 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 123 | added | — | Vision_Add_30 Add | 576 × 1408 | |||
| 124 | added | — | Vision_FFN_30 Feed Forward | 16M | 576 × 1408 | ||
| 125 | added | — | Vision_LN_31 Layer Norm | 2.8K | 576 × 1408 | ||
| 126 | added | — | Vision_Attn_31 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 127 | added | — | Vision_Add_31 Add | 576 × 1408 | |||
| 128 | added | — | Vision_FFN_31 Feed Forward | 16M | 576 × 1408 | ||
| 129 | added | — | Vision_LN_32 Layer Norm | 2.8K | 576 × 1408 | ||
| 130 | added | — | Vision_Attn_32 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 131 | added | — | Vision_Add_32 Add | 576 × 1408 | |||
| 132 | added | — | Vision_FFN_32 Feed Forward | 16M | 576 × 1408 | ||
| 133 | added | — | Vision_LN_33 Layer Norm | 2.8K | 576 × 1408 | ||
| 134 | added | — | Vision_Attn_33 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 135 | added | — | Vision_Add_33 Add | 576 × 1408 | |||
| 136 | added | — | Vision_FFN_33 Feed Forward | 16M | 576 × 1408 | ||
| 137 | added | — | Vision_LN_34 Layer Norm | 2.8K | 576 × 1408 | ||
| 138 | added | — | Vision_Attn_34 Multi Head Attention | 7.9M | 576 × 1408 | ||
| 139 | added | — | Vision_Add_34 Add | 576 × 1408 | |||
| 140 | added | — | Vision_FFN_34 Feed Forward | 16M | 576 × 1408 | ||
| 141 | added | — | Vision projector Projection | 7.2M | 576 × 5120 | ||
| 142 | added | — | Vision tokens Reshape | 1 × 576 × 5120 | |||
| 143 | added | — | Multimodal fusion (concat tokens) Concatenate | 1 × 10486336 × 5120 | |||
| 144 | changed normalizedShape | RMSNorm_1_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_1_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 145 | changed embedDim, numHeads | Attention_1 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_1 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 146 | same | Add_1_attn Add | 1 × 40960 × 4096 | Add_1_attn Add | 1 × 10486336 × 5120 | ||
| 147 | changed normalizedShape | RMSNorm_1_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_1_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 148 | removed | FFN_1 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 149 | added | — | MoE_1 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 150 | same | Add_1_ffn Add | 1 × 40960 × 4096 | Add_1_ffn Add | 1 × 10486336 × 5120 | ||
| 151 | changed normalizedShape | RMSNorm_2_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_2_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 152 | changed embedDim, numHeads | Attention_2 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_2 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 153 | same | Add_2_attn Add | 1 × 40960 × 4096 | Add_2_attn Add | 1 × 10486336 × 5120 | ||
| 154 | changed normalizedShape | RMSNorm_2_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_2_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 155 | removed | FFN_2 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 156 | added | — | MoE_2 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 157 | same | Add_2_ffn Add | 1 × 40960 × 4096 | Add_2_ffn Add | 1 × 10486336 × 5120 | ||
| 158 | changed normalizedShape | RMSNorm_3_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_3_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 159 | changed embedDim, numHeads | Attention_3 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_3 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 160 | same | Add_3_attn Add | 1 × 40960 × 4096 | Add_3_attn Add | 1 × 10486336 × 5120 | ||
| 161 | changed normalizedShape | RMSNorm_3_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_3_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 162 | removed | FFN_3 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 163 | added | — | MoE_3 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 164 | same | Add_3_ffn Add | 1 × 40960 × 4096 | Add_3_ffn Add | 1 × 10486336 × 5120 | ||
| 165 | changed normalizedShape | RMSNorm_4_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_4_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 166 | changed embedDim, numHeads | Attention_4 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_4 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 167 | same | Add_4_attn Add | 1 × 40960 × 4096 | Add_4_attn Add | 1 × 10486336 × 5120 | ||
| 168 | changed normalizedShape | RMSNorm_4_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_4_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 169 | removed | FFN_4 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 170 | added | — | MoE_4 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 171 | same | Add_4_ffn Add | 1 × 40960 × 4096 | Add_4_ffn Add | 1 × 10486336 × 5120 | ||
| 172 | changed normalizedShape | RMSNorm_5_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_5_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 173 | changed embedDim, numHeads | Attention_5 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_5 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 174 | same | Add_5_attn Add | 1 × 40960 × 4096 | Add_5_attn Add | 1 × 10486336 × 5120 | ||
| 175 | changed normalizedShape | RMSNorm_5_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_5_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 176 | removed | FFN_5 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 177 | added | — | MoE_5 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 178 | same | Add_5_ffn Add | 1 × 40960 × 4096 | Add_5_ffn Add | 1 × 10486336 × 5120 | ||
| 179 | changed normalizedShape | RMSNorm_6_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_6_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 180 | changed embedDim, numHeads | Attention_6 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_6 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 181 | same | Add_6_attn Add | 1 × 40960 × 4096 | Add_6_attn Add | 1 × 10486336 × 5120 | ||
| 182 | changed normalizedShape | RMSNorm_6_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_6_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 183 | removed | FFN_6 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 184 | added | — | MoE_6 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 185 | same | Add_6_ffn Add | 1 × 40960 × 4096 | Add_6_ffn Add | 1 × 10486336 × 5120 | ||
| 186 | changed normalizedShape | RMSNorm_7_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_7_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 187 | changed embedDim, numHeads | Attention_7 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_7 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 188 | same | Add_7_attn Add | 1 × 40960 × 4096 | Add_7_attn Add | 1 × 10486336 × 5120 | ||
| 189 | changed normalizedShape | RMSNorm_7_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_7_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 190 | removed | FFN_7 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 191 | added | — | MoE_7 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 192 | same | Add_7_ffn Add | 1 × 40960 × 4096 | Add_7_ffn Add | 1 × 10486336 × 5120 | ||
| 193 | changed normalizedShape | RMSNorm_8_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_8_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 194 | changed embedDim, numHeads | Attention_8 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_8 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 195 | same | Add_8_attn Add | 1 × 40960 × 4096 | Add_8_attn Add | 1 × 10486336 × 5120 | ||
| 196 | changed normalizedShape | RMSNorm_8_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_8_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 197 | removed | FFN_8 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 198 | added | — | MoE_8 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 199 | same | Add_8_ffn Add | 1 × 40960 × 4096 | Add_8_ffn Add | 1 × 10486336 × 5120 | ||
| 200 | changed normalizedShape | RMSNorm_9_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_9_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 201 | changed embedDim, numHeads | Attention_9 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_9 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 202 | same | Add_9_attn Add | 1 × 40960 × 4096 | Add_9_attn Add | 1 × 10486336 × 5120 | ||
| 203 | changed normalizedShape | RMSNorm_9_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_9_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 204 | removed | FFN_9 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 205 | added | — | MoE_9 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 206 | same | Add_9_ffn Add | 1 × 40960 × 4096 | Add_9_ffn Add | 1 × 10486336 × 5120 | ||
| 207 | changed normalizedShape | RMSNorm_10_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_10_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 208 | changed embedDim, numHeads | Attention_10 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_10 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 209 | same | Add_10_attn Add | 1 × 40960 × 4096 | Add_10_attn Add | 1 × 10486336 × 5120 | ||
| 210 | changed normalizedShape | RMSNorm_10_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_10_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 211 | removed | FFN_10 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 212 | added | — | MoE_10 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 213 | same | Add_10_ffn Add | 1 × 40960 × 4096 | Add_10_ffn Add | 1 × 10486336 × 5120 | ||
| 214 | changed normalizedShape | RMSNorm_11_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_11_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 215 | changed embedDim, numHeads | Attention_11 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_11 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 216 | same | Add_11_attn Add | 1 × 40960 × 4096 | Add_11_attn Add | 1 × 10486336 × 5120 | ||
| 217 | changed normalizedShape | RMSNorm_11_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_11_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 218 | removed | FFN_11 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 219 | added | — | MoE_11 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 220 | same | Add_11_ffn Add | 1 × 40960 × 4096 | Add_11_ffn Add | 1 × 10486336 × 5120 | ||
| 221 | changed normalizedShape | RMSNorm_12_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_12_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 222 | changed embedDim, numHeads | Attention_12 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_12 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 223 | same | Add_12_attn Add | 1 × 40960 × 4096 | Add_12_attn Add | 1 × 10486336 × 5120 | ||
| 224 | changed normalizedShape | RMSNorm_12_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_12_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 225 | removed | FFN_12 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 226 | added | — | MoE_12 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 227 | same | Add_12_ffn Add | 1 × 40960 × 4096 | Add_12_ffn Add | 1 × 10486336 × 5120 | ||
| 228 | changed normalizedShape | RMSNorm_13_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_13_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 229 | changed embedDim, numHeads | Attention_13 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_13 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 230 | same | Add_13_attn Add | 1 × 40960 × 4096 | Add_13_attn Add | 1 × 10486336 × 5120 | ||
| 231 | changed normalizedShape | RMSNorm_13_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_13_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 232 | removed | FFN_13 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 233 | added | — | MoE_13 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 234 | same | Add_13_ffn Add | 1 × 40960 × 4096 | Add_13_ffn Add | 1 × 10486336 × 5120 | ||
| 235 | changed normalizedShape | RMSNorm_14_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_14_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 236 | changed embedDim, numHeads | Attention_14 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_14 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 237 | same | Add_14_attn Add | 1 × 40960 × 4096 | Add_14_attn Add | 1 × 10486336 × 5120 | ||
| 238 | changed normalizedShape | RMSNorm_14_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_14_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 239 | removed | FFN_14 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 240 | added | — | MoE_14 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 241 | same | Add_14_ffn Add | 1 × 40960 × 4096 | Add_14_ffn Add | 1 × 10486336 × 5120 | ||
| 242 | changed normalizedShape | RMSNorm_15_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_15_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 243 | changed embedDim, numHeads | Attention_15 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_15 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 244 | same | Add_15_attn Add | 1 × 40960 × 4096 | Add_15_attn Add | 1 × 10486336 × 5120 | ||
| 245 | changed normalizedShape | RMSNorm_15_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_15_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 246 | removed | FFN_15 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 247 | added | — | MoE_15 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 248 | same | Add_15_ffn Add | 1 × 40960 × 4096 | Add_15_ffn Add | 1 × 10486336 × 5120 | ||
| 249 | changed normalizedShape | RMSNorm_16_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_16_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 250 | changed embedDim, numHeads | Attention_16 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_16 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 251 | same | Add_16_attn Add | 1 × 40960 × 4096 | Add_16_attn Add | 1 × 10486336 × 5120 | ||
| 252 | changed normalizedShape | RMSNorm_16_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_16_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 253 | removed | FFN_16 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 254 | added | — | MoE_16 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 255 | same | Add_16_ffn Add | 1 × 40960 × 4096 | Add_16_ffn Add | 1 × 10486336 × 5120 | ||
| 256 | changed normalizedShape | RMSNorm_17_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_17_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 257 | changed embedDim, numHeads | Attention_17 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_17 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 258 | same | Add_17_attn Add | 1 × 40960 × 4096 | Add_17_attn Add | 1 × 10486336 × 5120 | ||
| 259 | changed normalizedShape | RMSNorm_17_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_17_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 260 | removed | FFN_17 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 261 | added | — | MoE_17 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 262 | same | Add_17_ffn Add | 1 × 40960 × 4096 | Add_17_ffn Add | 1 × 10486336 × 5120 | ||
| 263 | changed normalizedShape | RMSNorm_18_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_18_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 264 | changed embedDim, numHeads | Attention_18 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_18 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 265 | same | Add_18_attn Add | 1 × 40960 × 4096 | Add_18_attn Add | 1 × 10486336 × 5120 | ||
| 266 | changed normalizedShape | RMSNorm_18_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_18_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 267 | removed | FFN_18 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 268 | added | — | MoE_18 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 269 | same | Add_18_ffn Add | 1 × 40960 × 4096 | Add_18_ffn Add | 1 × 10486336 × 5120 | ||
| 270 | changed normalizedShape | RMSNorm_19_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_19_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 271 | changed embedDim, numHeads | Attention_19 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_19 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 272 | same | Add_19_attn Add | 1 × 40960 × 4096 | Add_19_attn Add | 1 × 10486336 × 5120 | ||
| 273 | changed normalizedShape | RMSNorm_19_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_19_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 274 | removed | FFN_19 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 275 | added | — | MoE_19 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 276 | same | Add_19_ffn Add | 1 × 40960 × 4096 | Add_19_ffn Add | 1 × 10486336 × 5120 | ||
| 277 | changed normalizedShape | RMSNorm_20_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_20_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 278 | changed embedDim, numHeads | Attention_20 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_20 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 279 | same | Add_20_attn Add | 1 × 40960 × 4096 | Add_20_attn Add | 1 × 10486336 × 5120 | ||
| 280 | changed normalizedShape | RMSNorm_20_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_20_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 281 | removed | FFN_20 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 282 | added | — | MoE_20 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 283 | same | Add_20_ffn Add | 1 × 40960 × 4096 | Add_20_ffn Add | 1 × 10486336 × 5120 | ||
| 284 | changed normalizedShape | RMSNorm_21_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_21_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 285 | changed embedDim, numHeads | Attention_21 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_21 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 286 | same | Add_21_attn Add | 1 × 40960 × 4096 | Add_21_attn Add | 1 × 10486336 × 5120 | ||
| 287 | changed normalizedShape | RMSNorm_21_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_21_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 288 | removed | FFN_21 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 289 | added | — | MoE_21 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 290 | same | Add_21_ffn Add | 1 × 40960 × 4096 | Add_21_ffn Add | 1 × 10486336 × 5120 | ||
| 291 | changed normalizedShape | RMSNorm_22_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_22_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 292 | changed embedDim, numHeads | Attention_22 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_22 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 293 | same | Add_22_attn Add | 1 × 40960 × 4096 | Add_22_attn Add | 1 × 10486336 × 5120 | ||
| 294 | changed normalizedShape | RMSNorm_22_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_22_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 295 | removed | FFN_22 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 296 | added | — | MoE_22 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 297 | same | Add_22_ffn Add | 1 × 40960 × 4096 | Add_22_ffn Add | 1 × 10486336 × 5120 | ||
| 298 | changed normalizedShape | RMSNorm_23_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_23_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 299 | changed embedDim, numHeads | Attention_23 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_23 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 300 | same | Add_23_attn Add | 1 × 40960 × 4096 | Add_23_attn Add | 1 × 10486336 × 5120 | ||
| 301 | changed normalizedShape | RMSNorm_23_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_23_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 302 | removed | FFN_23 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 303 | added | — | MoE_23 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 304 | same | Add_23_ffn Add | 1 × 40960 × 4096 | Add_23_ffn Add | 1 × 10486336 × 5120 | ||
| 305 | changed normalizedShape | RMSNorm_24_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_24_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 306 | changed embedDim, numHeads | Attention_24 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_24 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 307 | same | Add_24_attn Add | 1 × 40960 × 4096 | Add_24_attn Add | 1 × 10486336 × 5120 | ||
| 308 | changed normalizedShape | RMSNorm_24_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_24_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 309 | removed | FFN_24 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 310 | added | — | MoE_24 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 311 | same | Add_24_ffn Add | 1 × 40960 × 4096 | Add_24_ffn Add | 1 × 10486336 × 5120 | ||
| 312 | changed normalizedShape | RMSNorm_25_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_25_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 313 | changed embedDim, numHeads | Attention_25 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_25 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 314 | same | Add_25_attn Add | 1 × 40960 × 4096 | Add_25_attn Add | 1 × 10486336 × 5120 | ||
| 315 | changed normalizedShape | RMSNorm_25_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_25_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 316 | removed | FFN_25 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 317 | added | — | MoE_25 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 318 | same | Add_25_ffn Add | 1 × 40960 × 4096 | Add_25_ffn Add | 1 × 10486336 × 5120 | ||
| 319 | changed normalizedShape | RMSNorm_26_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_26_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 320 | changed embedDim, numHeads | Attention_26 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_26 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 321 | same | Add_26_attn Add | 1 × 40960 × 4096 | Add_26_attn Add | 1 × 10486336 × 5120 | ||
| 322 | changed normalizedShape | RMSNorm_26_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_26_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 323 | removed | FFN_26 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 324 | added | — | MoE_26 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 325 | same | Add_26_ffn Add | 1 × 40960 × 4096 | Add_26_ffn Add | 1 × 10486336 × 5120 | ||
| 326 | changed normalizedShape | RMSNorm_27_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_27_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 327 | changed embedDim, numHeads | Attention_27 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_27 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 328 | same | Add_27_attn Add | 1 × 40960 × 4096 | Add_27_attn Add | 1 × 10486336 × 5120 | ||
| 329 | changed normalizedShape | RMSNorm_27_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_27_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 330 | removed | FFN_27 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 331 | added | — | MoE_27 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 332 | same | Add_27_ffn Add | 1 × 40960 × 4096 | Add_27_ffn Add | 1 × 10486336 × 5120 | ||
| 333 | changed normalizedShape | RMSNorm_28_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_28_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 334 | changed embedDim, numHeads | Attention_28 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_28 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 335 | same | Add_28_attn Add | 1 × 40960 × 4096 | Add_28_attn Add | 1 × 10486336 × 5120 | ||
| 336 | changed normalizedShape | RMSNorm_28_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_28_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 337 | removed | FFN_28 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 338 | added | — | MoE_28 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 339 | same | Add_28_ffn Add | 1 × 40960 × 4096 | Add_28_ffn Add | 1 × 10486336 × 5120 | ||
| 340 | changed normalizedShape | RMSNorm_29_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_29_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 341 | changed embedDim, numHeads | Attention_29 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_29 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 342 | same | Add_29_attn Add | 1 × 40960 × 4096 | Add_29_attn Add | 1 × 10486336 × 5120 | ||
| 343 | changed normalizedShape | RMSNorm_29_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_29_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 344 | removed | FFN_29 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 345 | added | — | MoE_29 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 346 | same | Add_29_ffn Add | 1 × 40960 × 4096 | Add_29_ffn Add | 1 × 10486336 × 5120 | ||
| 347 | changed normalizedShape | RMSNorm_30_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_30_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 348 | changed embedDim, numHeads | Attention_30 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_30 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 349 | same | Add_30_attn Add | 1 × 40960 × 4096 | Add_30_attn Add | 1 × 10486336 × 5120 | ||
| 350 | changed normalizedShape | RMSNorm_30_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_30_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 351 | removed | FFN_30 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 352 | added | — | MoE_30 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 353 | same | Add_30_ffn Add | 1 × 40960 × 4096 | Add_30_ffn Add | 1 × 10486336 × 5120 | ||
| 354 | changed normalizedShape | RMSNorm_31_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_31_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 355 | changed embedDim, numHeads | Attention_31 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_31 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 356 | same | Add_31_attn Add | 1 × 40960 × 4096 | Add_31_attn Add | 1 × 10486336 × 5120 | ||
| 357 | changed normalizedShape | RMSNorm_31_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_31_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 358 | removed | FFN_31 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 359 | added | — | MoE_31 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 360 | same | Add_31_ffn Add | 1 × 40960 × 4096 | Add_31_ffn Add | 1 × 10486336 × 5120 | ||
| 361 | changed normalizedShape | RMSNorm_32_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_32_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 362 | changed embedDim, numHeads | Attention_32 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_32 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 363 | same | Add_32_attn Add | 1 × 40960 × 4096 | Add_32_attn Add | 1 × 10486336 × 5120 | ||
| 364 | changed normalizedShape | RMSNorm_32_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_32_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 365 | removed | FFN_32 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 366 | added | — | MoE_32 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 367 | same | Add_32_ffn Add | 1 × 40960 × 4096 | Add_32_ffn Add | 1 × 10486336 × 5120 | ||
| 368 | changed normalizedShape | RMSNorm_33_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_33_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 369 | changed embedDim, numHeads | Attention_33 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_33 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 370 | same | Add_33_attn Add | 1 × 40960 × 4096 | Add_33_attn Add | 1 × 10486336 × 5120 | ||
| 371 | changed normalizedShape | RMSNorm_33_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_33_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 372 | removed | FFN_33 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 373 | added | — | MoE_33 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 374 | same | Add_33_ffn Add | 1 × 40960 × 4096 | Add_33_ffn Add | 1 × 10486336 × 5120 | ||
| 375 | changed normalizedShape | RMSNorm_34_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_34_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 376 | changed embedDim, numHeads | Attention_34 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_34 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 377 | same | Add_34_attn Add | 1 × 40960 × 4096 | Add_34_attn Add | 1 × 10486336 × 5120 | ||
| 378 | changed normalizedShape | RMSNorm_34_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_34_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 379 | removed | FFN_34 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 380 | added | — | MoE_34 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 381 | same | Add_34_ffn Add | 1 × 40960 × 4096 | Add_34_ffn Add | 1 × 10486336 × 5120 | ||
| 382 | changed normalizedShape | RMSNorm_35_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_35_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 383 | changed embedDim, numHeads | Attention_35 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_35 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 384 | same | Add_35_attn Add | 1 × 40960 × 4096 | Add_35_attn Add | 1 × 10486336 × 5120 | ||
| 385 | changed normalizedShape | RMSNorm_35_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_35_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 386 | removed | FFN_35 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 387 | added | — | MoE_35 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 388 | same | Add_35_ffn Add | 1 × 40960 × 4096 | Add_35_ffn Add | 1 × 10486336 × 5120 | ||
| 389 | changed normalizedShape | RMSNorm_36_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_36_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 390 | changed embedDim, numHeads | Attention_36 Grouped Query Attention | 42M | 1 × 40960 × 4096 | Attention_36 Grouped Query Attention | 63M | 1 × 10486336 × 5120 |
| 391 | same | Add_36_attn Add | 1 × 40960 × 4096 | Add_36_attn Add | 1 × 10486336 × 5120 | ||
| 392 | changed normalizedShape | RMSNorm_36_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | RMSNorm_36_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 393 | removed | FFN_36 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 394 | added | — | MoE_36 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 395 | added | — | Add_36_ffn Add | 1 × 10486336 × 5120 | |||
| 396 | added | — | RMSNorm_37_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 397 | added | — | Attention_37 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 398 | added | — | Add_37_attn Add | 1 × 10486336 × 5120 | |||
| 399 | added | — | RMSNorm_37_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 400 | added | — | MoE_37 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 401 | added | — | Add_37_ffn Add | 1 × 10486336 × 5120 | |||
| 402 | added | — | RMSNorm_38_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 403 | added | — | Attention_38 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 404 | added | — | Add_38_attn Add | 1 × 10486336 × 5120 | |||
| 405 | added | — | RMSNorm_38_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 406 | added | — | MoE_38 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 407 | added | — | Add_38_ffn Add | 1 × 10486336 × 5120 | |||
| 408 | added | — | RMSNorm_39_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 409 | added | — | Attention_39 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 410 | added | — | Add_39_attn Add | 1 × 10486336 × 5120 | |||
| 411 | added | — | RMSNorm_39_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 412 | added | — | MoE_39 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 413 | added | — | Add_39_ffn Add | 1 × 10486336 × 5120 | |||
| 414 | added | — | RMSNorm_40_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 415 | added | — | Attention_40 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 416 | added | — | Add_40_attn Add | 1 × 10486336 × 5120 | |||
| 417 | added | — | RMSNorm_40_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 418 | added | — | MoE_40 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 419 | added | — | Add_40_ffn Add | 1 × 10486336 × 5120 | |||
| 420 | added | — | RMSNorm_41_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 421 | added | — | Attention_41 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 422 | added | — | Add_41_attn Add | 1 × 10486336 × 5120 | |||
| 423 | added | — | RMSNorm_41_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 424 | added | — | MoE_41 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 425 | added | — | Add_41_ffn Add | 1 × 10486336 × 5120 | |||
| 426 | added | — | RMSNorm_42_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 427 | added | — | Attention_42 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 428 | added | — | Add_42_attn Add | 1 × 10486336 × 5120 | |||
| 429 | added | — | RMSNorm_42_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 430 | added | — | MoE_42 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 431 | added | — | Add_42_ffn Add | 1 × 10486336 × 5120 | |||
| 432 | added | — | RMSNorm_43_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 433 | added | — | Attention_43 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 434 | added | — | Add_43_attn Add | 1 × 10486336 × 5120 | |||
| 435 | added | — | RMSNorm_43_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 436 | added | — | MoE_43 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 437 | added | — | Add_43_ffn Add | 1 × 10486336 × 5120 | |||
| 438 | added | — | RMSNorm_44_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 439 | added | — | Attention_44 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 440 | added | — | Add_44_attn Add | 1 × 10486336 × 5120 | |||
| 441 | added | — | RMSNorm_44_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 442 | added | — | MoE_44 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 443 | added | — | Add_44_ffn Add | 1 × 10486336 × 5120 | |||
| 444 | added | — | RMSNorm_45_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 445 | added | — | Attention_45 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 446 | added | — | Add_45_attn Add | 1 × 10486336 × 5120 | |||
| 447 | added | — | RMSNorm_45_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 448 | added | — | MoE_45 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 449 | added | — | Add_45_ffn Add | 1 × 10486336 × 5120 | |||
| 450 | added | — | RMSNorm_46_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 451 | added | — | Attention_46 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 452 | added | — | Add_46_attn Add | 1 × 10486336 × 5120 | |||
| 453 | added | — | RMSNorm_46_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 454 | added | — | MoE_46 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 455 | added | — | Add_46_ffn Add | 1 × 10486336 × 5120 | |||
| 456 | added | — | RMSNorm_47_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 457 | added | — | Attention_47 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 458 | added | — | Add_47_attn Add | 1 × 10486336 × 5120 | |||
| 459 | added | — | RMSNorm_47_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 460 | added | — | MoE_47 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 461 | added | — | Add_47_ffn Add | 1 × 10486336 × 5120 | |||
| 462 | added | — | RMSNorm_48_1 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 463 | added | — | Attention_48 Grouped Query Attention | 63M | 1 × 10486336 × 5120 | ||
| 464 | added | — | Add_48_attn Add | 1 × 10486336 × 5120 | |||
| 465 | added | — | RMSNorm_48_2 Rms Norm | 5.1K | 1 × 10486336 × 5120 | ||
| 466 | added | — | MoE_48 Moe Layer | 2.0B | 1 × 10486336 × 5120 | ||
| 467 | same | Add_36_ffn Add | 1 × 40960 × 4096 | Add_48_ffn Add | 1 × 10486336 × 5120 | ||
| 468 | changed normalizedShape | Final_RMSNorm Rms Norm | 4.1K | 1 × 40960 × 4096 | Final_RMSNorm Rms Norm | 5.1K | 1 × 10486336 × 5120 |
| 469 | changed inFeatures, outFeatures | LM_Head Linear | 622M | 1 × 40960 × 151936 | LM_Head Linear | 1.0B | 1 × 10486336 × 202048 |
| 470 | same | Output Output | 1 × 40960 × 151936 | Output Output | 1 × 10486336 × 202048 | ||
What this is not
- The two are priced at different declared inputs (1 × 40960 against 1 × 10485760), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.
Take it further
Open either graph in the editor, change it, and check it again: Qwen3-8B · Llama-4 Scout
Compare any two models of your own, including anything on Hugging Face: the comparison tool.
Machine-readable: this page as markdown ·
the pair index ·
POST https://www.neurarch.com/api/v1/plan for a graph of your own.