Comparisons / Mixtral MoE Block vs DeepSeek-V3
Mixtral MoE Block vs DeepSeek-V3
Two mixture-of-experts designs that route very differently.
DeepSeek-V3 has 665B more parameters than Mixtral MoE Block: 365 layers added, 2 removed, 4 changed.
Baseline
Mixtral MoE Block
- Layers
- 7
- Parameters
- 1.5B
- Input
- 1 × 4096 × 4096
- Output
- 1 × 4096 × 4096
- Forward-passes
- yes
- Est. train cost
- $63.07
Compared
DeepSeek-V3
- Layers
- 370
- Parameters
- 666B
- Input
- 1 × 163840
- Output
- 1 × 163840 × 129280
- Forward-passes
- yes
- Est. train cost
- $670,780
The deltas
Every number is DeepSeek-V3 relative to Mixtral MoE Block.
Parameters
+665B (459× the size)
Layers
+363
Added
365
Removed
2
Changed
4
Unchanged
3
Which GPUs each one fits
Each side is measured at its own declared input (1 × 4096 × 4096 against 1 × 163840). Both columns are right about their own model; the difference between them is not a fact about the designs.
| GPU | Mixtral MoE Block | DeepSeek-V3 |
|---|---|---|
| T4 16GB | no | no |
| A100 40GB | fits | no |
| H100 80GB | fits | no |
Layer by layer
Aligned in topological order. 3 of 374 rows are the same layer with the same parameters.
Show all 374 rows
| Mixtral MoE Block | DeepSeek-V3 | ||||||
|---|---|---|---|---|---|---|---|
| Layer | Params | Output | Layer | Params | Output | ||
| 1 | changed shape | hidden_states Input | 1 × 4096 × 4096 | Input Input | 1 × 163840 | ||
| 2 | added | — | Embedding Embedding | 927M | 1 × 163840 × 7168 | ||
| 3 | changed dim, maxPositionEmbeddings | rotary_emb Rope | RoPE (decoupled) Rope | 1 × 163840 × 7168 | |||
| 4 | changed normalizedShape | input_norm Rms Norm | 4.1K | 1 × 4096 × 4096 | RMSNorm_1_1 Rms Norm | 7.2K | 1 × 163840 × 7168 |
| 5 | removed | self_attn Grouped Query Attention | 42M | 1 × 4096 × 4096 | — | ||
| 6 | added | — | MLA_1 Mla | 110M | 1 × 163840 × 7168 | ||
| 7 | same | attn_residual Add | 1 × 4096 × 4096 | Add_1_attn Add | 1 × 163840 × 7168 | ||
| 8 | changed normalizedShape | post_attn_norm Rms Norm | 4.1K | 1 × 4096 × 4096 | RMSNorm_1_2 Rms Norm | 7.2K | 1 × 163840 × 7168 |
| 9 | removed | block_sparse_moe Moe Layer | 1.4B | 1 × 4096 × 4096 | — | ||
| 10 | added | — | FFN_1 Swiglu | 396M | 1 × 163840 × 7168 | ||
| 11 | added | — | Add_1_ffn Add | 1 × 163840 × 7168 | |||
| 12 | added | — | RMSNorm_2_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 13 | added | — | MLA_2 Mla | 110M | 1 × 163840 × 7168 | ||
| 14 | added | — | Add_2_attn Add | 1 × 163840 × 7168 | |||
| 15 | added | — | RMSNorm_2_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 16 | added | — | FFN_2 Swiglu | 396M | 1 × 163840 × 7168 | ||
| 17 | added | — | Add_2_ffn Add | 1 × 163840 × 7168 | |||
| 18 | added | — | RMSNorm_3_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 19 | added | — | MLA_3 Mla | 110M | 1 × 163840 × 7168 | ||
| 20 | added | — | Add_3_attn Add | 1 × 163840 × 7168 | |||
| 21 | added | — | RMSNorm_3_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 22 | added | — | FFN_3 Swiglu | 396M | 1 × 163840 × 7168 | ||
| 23 | added | — | Add_3_ffn Add | 1 × 163840 × 7168 | |||
| 24 | added | — | RMSNorm_4_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 25 | added | — | MLA_4 Mla | 110M | 1 × 163840 × 7168 | ||
| 26 | added | — | Add_4_attn Add | 1 × 163840 × 7168 | |||
| 27 | added | — | RMSNorm_4_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 28 | added | — | MoE_4 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 29 | added | — | Add_4_ffn Add | 1 × 163840 × 7168 | |||
| 30 | added | — | RMSNorm_5_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 31 | added | — | MLA_5 Mla | 110M | 1 × 163840 × 7168 | ||
| 32 | added | — | Add_5_attn Add | 1 × 163840 × 7168 | |||
| 33 | added | — | RMSNorm_5_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 34 | added | — | MoE_5 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 35 | added | — | Add_5_ffn Add | 1 × 163840 × 7168 | |||
| 36 | added | — | RMSNorm_6_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 37 | added | — | MLA_6 Mla | 110M | 1 × 163840 × 7168 | ||
| 38 | added | — | Add_6_attn Add | 1 × 163840 × 7168 | |||
| 39 | added | — | RMSNorm_6_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 40 | added | — | MoE_6 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 41 | added | — | Add_6_ffn Add | 1 × 163840 × 7168 | |||
| 42 | added | — | RMSNorm_7_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 43 | added | — | MLA_7 Mla | 110M | 1 × 163840 × 7168 | ||
| 44 | added | — | Add_7_attn Add | 1 × 163840 × 7168 | |||
| 45 | added | — | RMSNorm_7_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 46 | added | — | MoE_7 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 47 | added | — | Add_7_ffn Add | 1 × 163840 × 7168 | |||
| 48 | added | — | RMSNorm_8_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 49 | added | — | MLA_8 Mla | 110M | 1 × 163840 × 7168 | ||
| 50 | added | — | Add_8_attn Add | 1 × 163840 × 7168 | |||
| 51 | added | — | RMSNorm_8_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 52 | added | — | MoE_8 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 53 | added | — | Add_8_ffn Add | 1 × 163840 × 7168 | |||
| 54 | added | — | RMSNorm_9_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 55 | added | — | MLA_9 Mla | 110M | 1 × 163840 × 7168 | ||
| 56 | added | — | Add_9_attn Add | 1 × 163840 × 7168 | |||
| 57 | added | — | RMSNorm_9_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 58 | added | — | MoE_9 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 59 | added | — | Add_9_ffn Add | 1 × 163840 × 7168 | |||
| 60 | added | — | RMSNorm_10_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 61 | added | — | MLA_10 Mla | 110M | 1 × 163840 × 7168 | ||
| 62 | added | — | Add_10_attn Add | 1 × 163840 × 7168 | |||
| 63 | added | — | RMSNorm_10_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 64 | added | — | MoE_10 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 65 | added | — | Add_10_ffn Add | 1 × 163840 × 7168 | |||
| 66 | added | — | RMSNorm_11_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 67 | added | — | MLA_11 Mla | 110M | 1 × 163840 × 7168 | ||
| 68 | added | — | Add_11_attn Add | 1 × 163840 × 7168 | |||
| 69 | added | — | RMSNorm_11_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 70 | added | — | MoE_11 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 71 | added | — | Add_11_ffn Add | 1 × 163840 × 7168 | |||
| 72 | added | — | RMSNorm_12_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 73 | added | — | MLA_12 Mla | 110M | 1 × 163840 × 7168 | ||
| 74 | added | — | Add_12_attn Add | 1 × 163840 × 7168 | |||
| 75 | added | — | RMSNorm_12_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 76 | added | — | MoE_12 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 77 | added | — | Add_12_ffn Add | 1 × 163840 × 7168 | |||
| 78 | added | — | RMSNorm_13_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 79 | added | — | MLA_13 Mla | 110M | 1 × 163840 × 7168 | ||
| 80 | added | — | Add_13_attn Add | 1 × 163840 × 7168 | |||
| 81 | added | — | RMSNorm_13_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 82 | added | — | MoE_13 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 83 | added | — | Add_13_ffn Add | 1 × 163840 × 7168 | |||
| 84 | added | — | RMSNorm_14_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 85 | added | — | MLA_14 Mla | 110M | 1 × 163840 × 7168 | ||
| 86 | added | — | Add_14_attn Add | 1 × 163840 × 7168 | |||
| 87 | added | — | RMSNorm_14_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 88 | added | — | MoE_14 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 89 | added | — | Add_14_ffn Add | 1 × 163840 × 7168 | |||
| 90 | added | — | RMSNorm_15_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 91 | added | — | MLA_15 Mla | 110M | 1 × 163840 × 7168 | ||
| 92 | added | — | Add_15_attn Add | 1 × 163840 × 7168 | |||
| 93 | added | — | RMSNorm_15_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 94 | added | — | MoE_15 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 95 | added | — | Add_15_ffn Add | 1 × 163840 × 7168 | |||
| 96 | added | — | RMSNorm_16_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 97 | added | — | MLA_16 Mla | 110M | 1 × 163840 × 7168 | ||
| 98 | added | — | Add_16_attn Add | 1 × 163840 × 7168 | |||
| 99 | added | — | RMSNorm_16_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 100 | added | — | MoE_16 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 101 | added | — | Add_16_ffn Add | 1 × 163840 × 7168 | |||
| 102 | added | — | RMSNorm_17_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 103 | added | — | MLA_17 Mla | 110M | 1 × 163840 × 7168 | ||
| 104 | added | — | Add_17_attn Add | 1 × 163840 × 7168 | |||
| 105 | added | — | RMSNorm_17_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 106 | added | — | MoE_17 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 107 | added | — | Add_17_ffn Add | 1 × 163840 × 7168 | |||
| 108 | added | — | RMSNorm_18_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 109 | added | — | MLA_18 Mla | 110M | 1 × 163840 × 7168 | ||
| 110 | added | — | Add_18_attn Add | 1 × 163840 × 7168 | |||
| 111 | added | — | RMSNorm_18_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 112 | added | — | MoE_18 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 113 | added | — | Add_18_ffn Add | 1 × 163840 × 7168 | |||
| 114 | added | — | RMSNorm_19_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 115 | added | — | MLA_19 Mla | 110M | 1 × 163840 × 7168 | ||
| 116 | added | — | Add_19_attn Add | 1 × 163840 × 7168 | |||
| 117 | added | — | RMSNorm_19_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 118 | added | — | MoE_19 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 119 | added | — | Add_19_ffn Add | 1 × 163840 × 7168 | |||
| 120 | added | — | RMSNorm_20_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 121 | added | — | MLA_20 Mla | 110M | 1 × 163840 × 7168 | ||
| 122 | added | — | Add_20_attn Add | 1 × 163840 × 7168 | |||
| 123 | added | — | RMSNorm_20_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 124 | added | — | MoE_20 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 125 | added | — | Add_20_ffn Add | 1 × 163840 × 7168 | |||
| 126 | added | — | RMSNorm_21_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 127 | added | — | MLA_21 Mla | 110M | 1 × 163840 × 7168 | ||
| 128 | added | — | Add_21_attn Add | 1 × 163840 × 7168 | |||
| 129 | added | — | RMSNorm_21_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 130 | added | — | MoE_21 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 131 | added | — | Add_21_ffn Add | 1 × 163840 × 7168 | |||
| 132 | added | — | RMSNorm_22_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 133 | added | — | MLA_22 Mla | 110M | 1 × 163840 × 7168 | ||
| 134 | added | — | Add_22_attn Add | 1 × 163840 × 7168 | |||
| 135 | added | — | RMSNorm_22_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 136 | added | — | MoE_22 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 137 | added | — | Add_22_ffn Add | 1 × 163840 × 7168 | |||
| 138 | added | — | RMSNorm_23_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 139 | added | — | MLA_23 Mla | 110M | 1 × 163840 × 7168 | ||
| 140 | added | — | Add_23_attn Add | 1 × 163840 × 7168 | |||
| 141 | added | — | RMSNorm_23_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 142 | added | — | MoE_23 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 143 | added | — | Add_23_ffn Add | 1 × 163840 × 7168 | |||
| 144 | added | — | RMSNorm_24_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 145 | added | — | MLA_24 Mla | 110M | 1 × 163840 × 7168 | ||
| 146 | added | — | Add_24_attn Add | 1 × 163840 × 7168 | |||
| 147 | added | — | RMSNorm_24_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 148 | added | — | MoE_24 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 149 | added | — | Add_24_ffn Add | 1 × 163840 × 7168 | |||
| 150 | added | — | RMSNorm_25_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 151 | added | — | MLA_25 Mla | 110M | 1 × 163840 × 7168 | ||
| 152 | added | — | Add_25_attn Add | 1 × 163840 × 7168 | |||
| 153 | added | — | RMSNorm_25_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 154 | added | — | MoE_25 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 155 | added | — | Add_25_ffn Add | 1 × 163840 × 7168 | |||
| 156 | added | — | RMSNorm_26_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 157 | added | — | MLA_26 Mla | 110M | 1 × 163840 × 7168 | ||
| 158 | added | — | Add_26_attn Add | 1 × 163840 × 7168 | |||
| 159 | added | — | RMSNorm_26_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 160 | added | — | MoE_26 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 161 | added | — | Add_26_ffn Add | 1 × 163840 × 7168 | |||
| 162 | added | — | RMSNorm_27_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 163 | added | — | MLA_27 Mla | 110M | 1 × 163840 × 7168 | ||
| 164 | added | — | Add_27_attn Add | 1 × 163840 × 7168 | |||
| 165 | added | — | RMSNorm_27_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 166 | added | — | MoE_27 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 167 | added | — | Add_27_ffn Add | 1 × 163840 × 7168 | |||
| 168 | added | — | RMSNorm_28_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 169 | added | — | MLA_28 Mla | 110M | 1 × 163840 × 7168 | ||
| 170 | added | — | Add_28_attn Add | 1 × 163840 × 7168 | |||
| 171 | added | — | RMSNorm_28_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 172 | added | — | MoE_28 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 173 | added | — | Add_28_ffn Add | 1 × 163840 × 7168 | |||
| 174 | added | — | RMSNorm_29_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 175 | added | — | MLA_29 Mla | 110M | 1 × 163840 × 7168 | ||
| 176 | added | — | Add_29_attn Add | 1 × 163840 × 7168 | |||
| 177 | added | — | RMSNorm_29_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 178 | added | — | MoE_29 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 179 | added | — | Add_29_ffn Add | 1 × 163840 × 7168 | |||
| 180 | added | — | RMSNorm_30_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 181 | added | — | MLA_30 Mla | 110M | 1 × 163840 × 7168 | ||
| 182 | added | — | Add_30_attn Add | 1 × 163840 × 7168 | |||
| 183 | added | — | RMSNorm_30_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 184 | added | — | MoE_30 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 185 | added | — | Add_30_ffn Add | 1 × 163840 × 7168 | |||
| 186 | added | — | RMSNorm_31_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 187 | added | — | MLA_31 Mla | 110M | 1 × 163840 × 7168 | ||
| 188 | added | — | Add_31_attn Add | 1 × 163840 × 7168 | |||
| 189 | added | — | RMSNorm_31_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 190 | added | — | MoE_31 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 191 | added | — | Add_31_ffn Add | 1 × 163840 × 7168 | |||
| 192 | added | — | RMSNorm_32_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 193 | added | — | MLA_32 Mla | 110M | 1 × 163840 × 7168 | ||
| 194 | added | — | Add_32_attn Add | 1 × 163840 × 7168 | |||
| 195 | added | — | RMSNorm_32_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 196 | added | — | MoE_32 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 197 | added | — | Add_32_ffn Add | 1 × 163840 × 7168 | |||
| 198 | added | — | RMSNorm_33_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 199 | added | — | MLA_33 Mla | 110M | 1 × 163840 × 7168 | ||
| 200 | added | — | Add_33_attn Add | 1 × 163840 × 7168 | |||
| 201 | added | — | RMSNorm_33_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 202 | added | — | MoE_33 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 203 | added | — | Add_33_ffn Add | 1 × 163840 × 7168 | |||
| 204 | added | — | RMSNorm_34_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 205 | added | — | MLA_34 Mla | 110M | 1 × 163840 × 7168 | ||
| 206 | added | — | Add_34_attn Add | 1 × 163840 × 7168 | |||
| 207 | added | — | RMSNorm_34_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 208 | added | — | MoE_34 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 209 | added | — | Add_34_ffn Add | 1 × 163840 × 7168 | |||
| 210 | added | — | RMSNorm_35_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 211 | added | — | MLA_35 Mla | 110M | 1 × 163840 × 7168 | ||
| 212 | added | — | Add_35_attn Add | 1 × 163840 × 7168 | |||
| 213 | added | — | RMSNorm_35_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 214 | added | — | MoE_35 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 215 | added | — | Add_35_ffn Add | 1 × 163840 × 7168 | |||
| 216 | added | — | RMSNorm_36_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 217 | added | — | MLA_36 Mla | 110M | 1 × 163840 × 7168 | ||
| 218 | added | — | Add_36_attn Add | 1 × 163840 × 7168 | |||
| 219 | added | — | RMSNorm_36_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 220 | added | — | MoE_36 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 221 | added | — | Add_36_ffn Add | 1 × 163840 × 7168 | |||
| 222 | added | — | RMSNorm_37_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 223 | added | — | MLA_37 Mla | 110M | 1 × 163840 × 7168 | ||
| 224 | added | — | Add_37_attn Add | 1 × 163840 × 7168 | |||
| 225 | added | — | RMSNorm_37_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 226 | added | — | MoE_37 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 227 | added | — | Add_37_ffn Add | 1 × 163840 × 7168 | |||
| 228 | added | — | RMSNorm_38_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 229 | added | — | MLA_38 Mla | 110M | 1 × 163840 × 7168 | ||
| 230 | added | — | Add_38_attn Add | 1 × 163840 × 7168 | |||
| 231 | added | — | RMSNorm_38_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 232 | added | — | MoE_38 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 233 | added | — | Add_38_ffn Add | 1 × 163840 × 7168 | |||
| 234 | added | — | RMSNorm_39_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 235 | added | — | MLA_39 Mla | 110M | 1 × 163840 × 7168 | ||
| 236 | added | — | Add_39_attn Add | 1 × 163840 × 7168 | |||
| 237 | added | — | RMSNorm_39_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 238 | added | — | MoE_39 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 239 | added | — | Add_39_ffn Add | 1 × 163840 × 7168 | |||
| 240 | added | — | RMSNorm_40_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 241 | added | — | MLA_40 Mla | 110M | 1 × 163840 × 7168 | ||
| 242 | added | — | Add_40_attn Add | 1 × 163840 × 7168 | |||
| 243 | added | — | RMSNorm_40_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 244 | added | — | MoE_40 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 245 | added | — | Add_40_ffn Add | 1 × 163840 × 7168 | |||
| 246 | added | — | RMSNorm_41_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 247 | added | — | MLA_41 Mla | 110M | 1 × 163840 × 7168 | ||
| 248 | added | — | Add_41_attn Add | 1 × 163840 × 7168 | |||
| 249 | added | — | RMSNorm_41_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 250 | added | — | MoE_41 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 251 | added | — | Add_41_ffn Add | 1 × 163840 × 7168 | |||
| 252 | added | — | RMSNorm_42_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 253 | added | — | MLA_42 Mla | 110M | 1 × 163840 × 7168 | ||
| 254 | added | — | Add_42_attn Add | 1 × 163840 × 7168 | |||
| 255 | added | — | RMSNorm_42_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 256 | added | — | MoE_42 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 257 | added | — | Add_42_ffn Add | 1 × 163840 × 7168 | |||
| 258 | added | — | RMSNorm_43_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 259 | added | — | MLA_43 Mla | 110M | 1 × 163840 × 7168 | ||
| 260 | added | — | Add_43_attn Add | 1 × 163840 × 7168 | |||
| 261 | added | — | RMSNorm_43_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 262 | added | — | MoE_43 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 263 | added | — | Add_43_ffn Add | 1 × 163840 × 7168 | |||
| 264 | added | — | RMSNorm_44_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 265 | added | — | MLA_44 Mla | 110M | 1 × 163840 × 7168 | ||
| 266 | added | — | Add_44_attn Add | 1 × 163840 × 7168 | |||
| 267 | added | — | RMSNorm_44_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 268 | added | — | MoE_44 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 269 | added | — | Add_44_ffn Add | 1 × 163840 × 7168 | |||
| 270 | added | — | RMSNorm_45_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 271 | added | — | MLA_45 Mla | 110M | 1 × 163840 × 7168 | ||
| 272 | added | — | Add_45_attn Add | 1 × 163840 × 7168 | |||
| 273 | added | — | RMSNorm_45_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 274 | added | — | MoE_45 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 275 | added | — | Add_45_ffn Add | 1 × 163840 × 7168 | |||
| 276 | added | — | RMSNorm_46_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 277 | added | — | MLA_46 Mla | 110M | 1 × 163840 × 7168 | ||
| 278 | added | — | Add_46_attn Add | 1 × 163840 × 7168 | |||
| 279 | added | — | RMSNorm_46_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 280 | added | — | MoE_46 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 281 | added | — | Add_46_ffn Add | 1 × 163840 × 7168 | |||
| 282 | added | — | RMSNorm_47_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 283 | added | — | MLA_47 Mla | 110M | 1 × 163840 × 7168 | ||
| 284 | added | — | Add_47_attn Add | 1 × 163840 × 7168 | |||
| 285 | added | — | RMSNorm_47_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 286 | added | — | MoE_47 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 287 | added | — | Add_47_ffn Add | 1 × 163840 × 7168 | |||
| 288 | added | — | RMSNorm_48_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 289 | added | — | MLA_48 Mla | 110M | 1 × 163840 × 7168 | ||
| 290 | added | — | Add_48_attn Add | 1 × 163840 × 7168 | |||
| 291 | added | — | RMSNorm_48_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 292 | added | — | MoE_48 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 293 | added | — | Add_48_ffn Add | 1 × 163840 × 7168 | |||
| 294 | added | — | RMSNorm_49_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 295 | added | — | MLA_49 Mla | 110M | 1 × 163840 × 7168 | ||
| 296 | added | — | Add_49_attn Add | 1 × 163840 × 7168 | |||
| 297 | added | — | RMSNorm_49_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 298 | added | — | MoE_49 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 299 | added | — | Add_49_ffn Add | 1 × 163840 × 7168 | |||
| 300 | added | — | RMSNorm_50_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 301 | added | — | MLA_50 Mla | 110M | 1 × 163840 × 7168 | ||
| 302 | added | — | Add_50_attn Add | 1 × 163840 × 7168 | |||
| 303 | added | — | RMSNorm_50_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 304 | added | — | MoE_50 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 305 | added | — | Add_50_ffn Add | 1 × 163840 × 7168 | |||
| 306 | added | — | RMSNorm_51_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 307 | added | — | MLA_51 Mla | 110M | 1 × 163840 × 7168 | ||
| 308 | added | — | Add_51_attn Add | 1 × 163840 × 7168 | |||
| 309 | added | — | RMSNorm_51_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 310 | added | — | MoE_51 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 311 | added | — | Add_51_ffn Add | 1 × 163840 × 7168 | |||
| 312 | added | — | RMSNorm_52_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 313 | added | — | MLA_52 Mla | 110M | 1 × 163840 × 7168 | ||
| 314 | added | — | Add_52_attn Add | 1 × 163840 × 7168 | |||
| 315 | added | — | RMSNorm_52_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 316 | added | — | MoE_52 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 317 | added | — | Add_52_ffn Add | 1 × 163840 × 7168 | |||
| 318 | added | — | RMSNorm_53_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 319 | added | — | MLA_53 Mla | 110M | 1 × 163840 × 7168 | ||
| 320 | added | — | Add_53_attn Add | 1 × 163840 × 7168 | |||
| 321 | added | — | RMSNorm_53_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 322 | added | — | MoE_53 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 323 | added | — | Add_53_ffn Add | 1 × 163840 × 7168 | |||
| 324 | added | — | RMSNorm_54_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 325 | added | — | MLA_54 Mla | 110M | 1 × 163840 × 7168 | ||
| 326 | added | — | Add_54_attn Add | 1 × 163840 × 7168 | |||
| 327 | added | — | RMSNorm_54_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 328 | added | — | MoE_54 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 329 | added | — | Add_54_ffn Add | 1 × 163840 × 7168 | |||
| 330 | added | — | RMSNorm_55_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 331 | added | — | MLA_55 Mla | 110M | 1 × 163840 × 7168 | ||
| 332 | added | — | Add_55_attn Add | 1 × 163840 × 7168 | |||
| 333 | added | — | RMSNorm_55_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 334 | added | — | MoE_55 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 335 | added | — | Add_55_ffn Add | 1 × 163840 × 7168 | |||
| 336 | added | — | RMSNorm_56_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 337 | added | — | MLA_56 Mla | 110M | 1 × 163840 × 7168 | ||
| 338 | added | — | Add_56_attn Add | 1 × 163840 × 7168 | |||
| 339 | added | — | RMSNorm_56_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 340 | added | — | MoE_56 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 341 | added | — | Add_56_ffn Add | 1 × 163840 × 7168 | |||
| 342 | added | — | RMSNorm_57_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 343 | added | — | MLA_57 Mla | 110M | 1 × 163840 × 7168 | ||
| 344 | added | — | Add_57_attn Add | 1 × 163840 × 7168 | |||
| 345 | added | — | RMSNorm_57_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 346 | added | — | MoE_57 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 347 | added | — | Add_57_ffn Add | 1 × 163840 × 7168 | |||
| 348 | added | — | RMSNorm_58_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 349 | added | — | MLA_58 Mla | 110M | 1 × 163840 × 7168 | ||
| 350 | added | — | Add_58_attn Add | 1 × 163840 × 7168 | |||
| 351 | added | — | RMSNorm_58_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 352 | added | — | MoE_58 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 353 | added | — | Add_58_ffn Add | 1 × 163840 × 7168 | |||
| 354 | added | — | RMSNorm_59_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 355 | added | — | MLA_59 Mla | 110M | 1 × 163840 × 7168 | ||
| 356 | added | — | Add_59_attn Add | 1 × 163840 × 7168 | |||
| 357 | added | — | RMSNorm_59_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 358 | added | — | MoE_59 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 359 | added | — | Add_59_ffn Add | 1 × 163840 × 7168 | |||
| 360 | added | — | RMSNorm_60_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 361 | added | — | MLA_60 Mla | 110M | 1 × 163840 × 7168 | ||
| 362 | added | — | Add_60_attn Add | 1 × 163840 × 7168 | |||
| 363 | added | — | RMSNorm_60_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 364 | added | — | MoE_60 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 365 | added | — | Add_60_ffn Add | 1 × 163840 × 7168 | |||
| 366 | added | — | RMSNorm_61_1 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 367 | added | — | MLA_61 Mla | 110M | 1 × 163840 × 7168 | ||
| 368 | added | — | Add_61_attn Add | 1 × 163840 × 7168 | |||
| 369 | added | — | RMSNorm_61_2 Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 370 | added | — | MoE_61 Shared Expert Mo E | 11B | 1 × 163840 × 7168 | ||
| 371 | same | moe_residual Add | 1 × 4096 × 4096 | Add_61_ffn Add | 1 × 163840 × 7168 | ||
| 372 | added | — | Final_RMSNorm Rms Norm | 7.2K | 1 × 163840 × 7168 | ||
| 373 | added | — | LM_Head Linear | 927M | 1 × 163840 × 129280 | ||
| 374 | same | hidden_out Output | 1 × 4096 × 4096 | Output Output | 1 × 163840 × 129280 | ||
What this is not
- The two are priced at different declared inputs (1 × 4096 × 4096 against 1 × 163840), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.
Take it further
Open either graph in the editor, change it, and check it again: Mixtral MoE Block · DeepSeek-V3
Compare any two models of your own, including anything on Hugging Face: the comparison tool.
Machine-readable: this page as markdown ·
the pair index ·
POST https://www.neurarch.com/api/v1/plan for a graph of your own.