Models / nemotron_h

NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4

Reconstructed from its own config.json with no weights read. 726K downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

1496B

1,495,510,032,384 parameters

In the published checkpoint

67.23B

67,228,556,288 scalars · safetensors.total, read 2026-08-24

Delta

+2125%

quantized This checkpoint is stored quantized (modelopt). The tensor count in the file counts stored elements under a packing scheme, not logical parameters, so the two numbers below are not measuring the same thing in either direction.

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.

Layers

532

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$1620896.04

Card	Memory
T4 (16GB)	weights + activations	does not fit
A100 (40GB)	weights + activations	does not fit
H100 (80GB)	weights + activations	does not fit

Structure

534 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Output shape
1	Input	Input	1 × 262144
2	Embedding	Embedding	1 × 262144 × 4096
3	RoPE	RoPE	1 × 262144 × 4096
4	LayerNorm_1_1	LayerNorm	1 × 262144 × 4096
5	Attention_1	Grouped Query Attn	1 × 262144 × 4096
6	Add_1_attn	Add	1 × 262144 × 4096
7	LayerNorm_1_2	LayerNorm	1 × 262144 × 4096
8	MoE_1	Shared-Expert MoE	1 × 262144 × 4096
9	Add_1_ffn	Add	1 × 262144 × 4096
10	LayerNorm_2_1	LayerNorm	1 × 262144 × 4096
11	Attention_2	Grouped Query Attn	1 × 262144 × 4096
12	Add_2_attn	Add	1 × 262144 × 4096
13	LayerNorm_2_2	LayerNorm	1 × 262144 × 4096
14	MoE_2	Shared-Expert MoE	1 × 262144 × 4096
15	Add_2_ffn	Add	1 × 262144 × 4096
16	LayerNorm_3_1	LayerNorm	1 × 262144 × 4096
17	Attention_3	Grouped Query Attn	1 × 262144 × 4096
18	Add_3_attn	Add	1 × 262144 × 4096
19	LayerNorm_3_2	LayerNorm	1 × 262144 × 4096
20	MoE_3	Shared-Expert MoE	1 × 262144 × 4096
21	Add_3_ffn	Add	1 × 262144 × 4096
22	LayerNorm_4_1	LayerNorm	1 × 262144 × 4096
23	Attention_4	Grouped Query Attn	1 × 262144 × 4096
24	Add_4_attn	Add	1 × 262144 × 4096
25	LayerNorm_4_2	LayerNorm	1 × 262144 × 4096
26	MoE_4	Shared-Expert MoE	1 × 262144 × 4096
27	Add_4_ffn	Add	1 × 262144 × 4096
28	LayerNorm_5_1	LayerNorm	1 × 262144 × 4096
29	Attention_5	Grouped Query Attn	1 × 262144 × 4096
30	Add_5_attn	Add	1 × 262144 × 4096
31	LayerNorm_5_2	LayerNorm	1 × 262144 × 4096
32	MoE_5	Shared-Expert MoE	1 × 262144 × 4096
33	Add_5_ffn	Add	1 × 262144 × 4096
34	LayerNorm_6_1	LayerNorm	1 × 262144 × 4096
35	Attention_6	Grouped Query Attn	1 × 262144 × 4096
36	Add_6_attn	Add	1 × 262144 × 4096
37	LayerNorm_6_2	LayerNorm	1 × 262144 × 4096
38	MoE_6	Shared-Expert MoE	1 × 262144 × 4096
39	Add_6_ffn	Add	1 × 262144 × 4096
40	LayerNorm_7_1	LayerNorm	1 × 262144 × 4096
41	Attention_7	Grouped Query Attn	1 × 262144 × 4096
42	Add_7_attn	Add	1 × 262144 × 4096
43	LayerNorm_7_2	LayerNorm	1 × 262144 × 4096
44	MoE_7	Shared-Expert MoE	1 × 262144 × 4096
45	Add_7_ffn	Add	1 × 262144 × 4096
46	LayerNorm_8_1	LayerNorm	1 × 262144 × 4096
47	Attention_8	Grouped Query Attn	1 × 262144 × 4096
48	Add_8_attn	Add	1 × 262144 × 4096
49	LayerNorm_8_2	LayerNorm	1 × 262144 × 4096
50	MoE_8	Shared-Expert MoE	1 × 262144 × 4096
51	Add_8_ffn	Add	1 × 262144 × 4096
52	LayerNorm_9_1	LayerNorm	1 × 262144 × 4096
53	Attention_9	Grouped Query Attn	1 × 262144 × 4096
54	Add_9_attn	Add	1 × 262144 × 4096
55	LayerNorm_9_2	LayerNorm	1 × 262144 × 4096
56	MoE_9	Shared-Expert MoE	1 × 262144 × 4096
57	Add_9_ffn	Add	1 × 262144 × 4096
58	LayerNorm_10_1	LayerNorm	1 × 262144 × 4096
59	Attention_10	Grouped Query Attn	1 × 262144 × 4096
60	Add_10_attn	Add	1 × 262144 × 4096
61	LayerNorm_10_2	LayerNorm	1 × 262144 × 4096
62	MoE_10	Shared-Expert MoE	1 × 262144 × 4096
63	Add_10_ffn	Add	1 × 262144 × 4096
64	LayerNorm_11_1	LayerNorm	1 × 262144 × 4096
65	Attention_11	Grouped Query Attn	1 × 262144 × 4096
66	Add_11_attn	Add	1 × 262144 × 4096
67	LayerNorm_11_2	LayerNorm	1 × 262144 × 4096
68	MoE_11	Shared-Expert MoE	1 × 262144 × 4096
69	Add_11_ffn	Add	1 × 262144 × 4096
70	LayerNorm_12_1	LayerNorm	1 × 262144 × 4096
71	Attention_12	Grouped Query Attn	1 × 262144 × 4096
72	Add_12_attn	Add	1 × 262144 × 4096
73	LayerNorm_12_2	LayerNorm	1 × 262144 × 4096
74	MoE_12	Shared-Expert MoE	1 × 262144 × 4096
75	Add_12_ffn	Add	1 × 262144 × 4096
76	LayerNorm_13_1	LayerNorm	1 × 262144 × 4096
77	Attention_13	Grouped Query Attn	1 × 262144 × 4096
78	Add_13_attn	Add	1 × 262144 × 4096
79	LayerNorm_13_2	LayerNorm	1 × 262144 × 4096
80	MoE_13	Shared-Expert MoE	1 × 262144 × 4096
81	Add_13_ffn	Add	1 × 262144 × 4096
82	LayerNorm_14_1	LayerNorm	1 × 262144 × 4096
83	Attention_14	Grouped Query Attn	1 × 262144 × 4096
84	Add_14_attn	Add	1 × 262144 × 4096
85	LayerNorm_14_2	LayerNorm	1 × 262144 × 4096
86	MoE_14	Shared-Expert MoE	1 × 262144 × 4096
87	Add_14_ffn	Add	1 × 262144 × 4096
88	LayerNorm_15_1	LayerNorm	1 × 262144 × 4096
89	Attention_15	Grouped Query Attn	1 × 262144 × 4096
90	Add_15_attn	Add	1 × 262144 × 4096
91	LayerNorm_15_2	LayerNorm	1 × 262144 × 4096
92	MoE_15	Shared-Expert MoE	1 × 262144 × 4096
93	Add_15_ffn	Add	1 × 262144 × 4096
94	LayerNorm_16_1	LayerNorm	1 × 262144 × 4096
95	Attention_16	Grouped Query Attn	1 × 262144 × 4096
96	Add_16_attn	Add	1 × 262144 × 4096
97	LayerNorm_16_2	LayerNorm	1 × 262144 × 4096
98	MoE_16	Shared-Expert MoE	1 × 262144 × 4096
99	Add_16_ffn	Add	1 × 262144 × 4096
100	LayerNorm_17_1	LayerNorm	1 × 262144 × 4096
101	Attention_17	Grouped Query Attn	1 × 262144 × 4096
102	Add_17_attn	Add	1 × 262144 × 4096
103	LayerNorm_17_2	LayerNorm	1 × 262144 × 4096
104	MoE_17	Shared-Expert MoE	1 × 262144 × 4096
105	Add_17_ffn	Add	1 × 262144 × 4096
106	LayerNorm_18_1	LayerNorm	1 × 262144 × 4096
107	Attention_18	Grouped Query Attn	1 × 262144 × 4096
108	Add_18_attn	Add	1 × 262144 × 4096
109	LayerNorm_18_2	LayerNorm	1 × 262144 × 4096
110	MoE_18	Shared-Expert MoE	1 × 262144 × 4096
111	Add_18_ffn	Add	1 × 262144 × 4096
112	LayerNorm_19_1	LayerNorm	1 × 262144 × 4096
113	Attention_19	Grouped Query Attn	1 × 262144 × 4096
114	Add_19_attn	Add	1 × 262144 × 4096
115	LayerNorm_19_2	LayerNorm	1 × 262144 × 4096
116	MoE_19	Shared-Expert MoE	1 × 262144 × 4096
117	Add_19_ffn	Add	1 × 262144 × 4096
118	LayerNorm_20_1	LayerNorm	1 × 262144 × 4096
119	Attention_20	Grouped Query Attn	1 × 262144 × 4096
120	Add_20_attn	Add	1 × 262144 × 4096
121	LayerNorm_20_2	LayerNorm	1 × 262144 × 4096
122	MoE_20	Shared-Expert MoE	1 × 262144 × 4096
123	Add_20_ffn	Add	1 × 262144 × 4096
124	LayerNorm_21_1	LayerNorm	1 × 262144 × 4096
125	Attention_21	Grouped Query Attn	1 × 262144 × 4096
126	Add_21_attn	Add	1 × 262144 × 4096
127	LayerNorm_21_2	LayerNorm	1 × 262144 × 4096
128	MoE_21	Shared-Expert MoE	1 × 262144 × 4096
129	Add_21_ffn	Add	1 × 262144 × 4096
130	LayerNorm_22_1	LayerNorm	1 × 262144 × 4096
131	Attention_22	Grouped Query Attn	1 × 262144 × 4096
132	Add_22_attn	Add	1 × 262144 × 4096
133	LayerNorm_22_2	LayerNorm	1 × 262144 × 4096
134	MoE_22	Shared-Expert MoE	1 × 262144 × 4096
135	Add_22_ffn	Add	1 × 262144 × 4096
136	LayerNorm_23_1	LayerNorm	1 × 262144 × 4096
137	Attention_23	Grouped Query Attn	1 × 262144 × 4096
138	Add_23_attn	Add	1 × 262144 × 4096
139	LayerNorm_23_2	LayerNorm	1 × 262144 × 4096
140	MoE_23	Shared-Expert MoE	1 × 262144 × 4096
141	Add_23_ffn	Add	1 × 262144 × 4096
142	LayerNorm_24_1	LayerNorm	1 × 262144 × 4096
143	Attention_24	Grouped Query Attn	1 × 262144 × 4096
144	Add_24_attn	Add	1 × 262144 × 4096
145	LayerNorm_24_2	LayerNorm	1 × 262144 × 4096
146	MoE_24	Shared-Expert MoE	1 × 262144 × 4096
147	Add_24_ffn	Add	1 × 262144 × 4096
148	LayerNorm_25_1	LayerNorm	1 × 262144 × 4096
149	Attention_25	Grouped Query Attn	1 × 262144 × 4096
150	Add_25_attn	Add	1 × 262144 × 4096
151	LayerNorm_25_2	LayerNorm	1 × 262144 × 4096
152	MoE_25	Shared-Expert MoE	1 × 262144 × 4096
153	Add_25_ffn	Add	1 × 262144 × 4096
154	LayerNorm_26_1	LayerNorm	1 × 262144 × 4096
155	Attention_26	Grouped Query Attn	1 × 262144 × 4096
156	Add_26_attn	Add	1 × 262144 × 4096
157	LayerNorm_26_2	LayerNorm	1 × 262144 × 4096
158	MoE_26	Shared-Expert MoE	1 × 262144 × 4096
159	Add_26_ffn	Add	1 × 262144 × 4096
160	LayerNorm_27_1	LayerNorm	1 × 262144 × 4096
161	Attention_27	Grouped Query Attn	1 × 262144 × 4096
162	Add_27_attn	Add	1 × 262144 × 4096
163	LayerNorm_27_2	LayerNorm	1 × 262144 × 4096
164	MoE_27	Shared-Expert MoE	1 × 262144 × 4096
165	Add_27_ffn	Add	1 × 262144 × 4096
166	LayerNorm_28_1	LayerNorm	1 × 262144 × 4096
167	Attention_28	Grouped Query Attn	1 × 262144 × 4096
168	Add_28_attn	Add	1 × 262144 × 4096
169	LayerNorm_28_2	LayerNorm	1 × 262144 × 4096
170	MoE_28	Shared-Expert MoE	1 × 262144 × 4096
171	Add_28_ffn	Add	1 × 262144 × 4096
172	LayerNorm_29_1	LayerNorm	1 × 262144 × 4096
173	Attention_29	Grouped Query Attn	1 × 262144 × 4096
174	Add_29_attn	Add	1 × 262144 × 4096
175	LayerNorm_29_2	LayerNorm	1 × 262144 × 4096
176	MoE_29	Shared-Expert MoE	1 × 262144 × 4096
177	Add_29_ffn	Add	1 × 262144 × 4096
178	LayerNorm_30_1	LayerNorm	1 × 262144 × 4096
179	Attention_30	Grouped Query Attn	1 × 262144 × 4096
180	Add_30_attn	Add	1 × 262144 × 4096
181	LayerNorm_30_2	LayerNorm	1 × 262144 × 4096
182	MoE_30	Shared-Expert MoE	1 × 262144 × 4096
183	Add_30_ffn	Add	1 × 262144 × 4096
184	LayerNorm_31_1	LayerNorm	1 × 262144 × 4096
185	Attention_31	Grouped Query Attn	1 × 262144 × 4096
186	Add_31_attn	Add	1 × 262144 × 4096
187	LayerNorm_31_2	LayerNorm	1 × 262144 × 4096
188	MoE_31	Shared-Expert MoE	1 × 262144 × 4096
189	Add_31_ffn	Add	1 × 262144 × 4096
190	LayerNorm_32_1	LayerNorm	1 × 262144 × 4096
191	Attention_32	Grouped Query Attn	1 × 262144 × 4096
192	Add_32_attn	Add	1 × 262144 × 4096
193	LayerNorm_32_2	LayerNorm	1 × 262144 × 4096
194	MoE_32	Shared-Expert MoE	1 × 262144 × 4096
195	Add_32_ffn	Add	1 × 262144 × 4096
196	LayerNorm_33_1	LayerNorm	1 × 262144 × 4096
197	Attention_33	Grouped Query Attn	1 × 262144 × 4096
198	Add_33_attn	Add	1 × 262144 × 4096
199	LayerNorm_33_2	LayerNorm	1 × 262144 × 4096
200	MoE_33	Shared-Expert MoE	1 × 262144 × 4096
201	Add_33_ffn	Add	1 × 262144 × 4096
202	LayerNorm_34_1	LayerNorm	1 × 262144 × 4096
203	Attention_34	Grouped Query Attn	1 × 262144 × 4096
204	Add_34_attn	Add	1 × 262144 × 4096
205	LayerNorm_34_2	LayerNorm	1 × 262144 × 4096
206	MoE_34	Shared-Expert MoE	1 × 262144 × 4096
207	Add_34_ffn	Add	1 × 262144 × 4096
208	LayerNorm_35_1	LayerNorm	1 × 262144 × 4096
209	Attention_35	Grouped Query Attn	1 × 262144 × 4096
210	Add_35_attn	Add	1 × 262144 × 4096
211	LayerNorm_35_2	LayerNorm	1 × 262144 × 4096
212	MoE_35	Shared-Expert MoE	1 × 262144 × 4096
213	Add_35_ffn	Add	1 × 262144 × 4096
214	LayerNorm_36_1	LayerNorm	1 × 262144 × 4096
215	Attention_36	Grouped Query Attn	1 × 262144 × 4096
216	Add_36_attn	Add	1 × 262144 × 4096
217	LayerNorm_36_2	LayerNorm	1 × 262144 × 4096
218	MoE_36	Shared-Expert MoE	1 × 262144 × 4096
219	Add_36_ffn	Add	1 × 262144 × 4096
220	LayerNorm_37_1	LayerNorm	1 × 262144 × 4096
221	Attention_37	Grouped Query Attn	1 × 262144 × 4096
222	Add_37_attn	Add	1 × 262144 × 4096
223	LayerNorm_37_2	LayerNorm	1 × 262144 × 4096
224	MoE_37	Shared-Expert MoE	1 × 262144 × 4096
225	Add_37_ffn	Add	1 × 262144 × 4096
226	LayerNorm_38_1	LayerNorm	1 × 262144 × 4096
227	Attention_38	Grouped Query Attn	1 × 262144 × 4096
228	Add_38_attn	Add	1 × 262144 × 4096
229	LayerNorm_38_2	LayerNorm	1 × 262144 × 4096
230	MoE_38	Shared-Expert MoE	1 × 262144 × 4096
231	Add_38_ffn	Add	1 × 262144 × 4096
232	LayerNorm_39_1	LayerNorm	1 × 262144 × 4096
233	Attention_39	Grouped Query Attn	1 × 262144 × 4096
234	Add_39_attn	Add	1 × 262144 × 4096
235	LayerNorm_39_2	LayerNorm	1 × 262144 × 4096
236	MoE_39	Shared-Expert MoE	1 × 262144 × 4096
237	Add_39_ffn	Add	1 × 262144 × 4096
238	LayerNorm_40_1	LayerNorm	1 × 262144 × 4096
239	Attention_40	Grouped Query Attn	1 × 262144 × 4096
240	Add_40_attn	Add	1 × 262144 × 4096
241	LayerNorm_40_2	LayerNorm	1 × 262144 × 4096
242	MoE_40	Shared-Expert MoE	1 × 262144 × 4096
243	Add_40_ffn	Add	1 × 262144 × 4096
244	LayerNorm_41_1	LayerNorm	1 × 262144 × 4096
245	Attention_41	Grouped Query Attn	1 × 262144 × 4096
246	Add_41_attn	Add	1 × 262144 × 4096
247	LayerNorm_41_2	LayerNorm	1 × 262144 × 4096
248	MoE_41	Shared-Expert MoE	1 × 262144 × 4096
249	Add_41_ffn	Add	1 × 262144 × 4096
250	LayerNorm_42_1	LayerNorm	1 × 262144 × 4096
251	Attention_42	Grouped Query Attn	1 × 262144 × 4096
252	Add_42_attn	Add	1 × 262144 × 4096
253	LayerNorm_42_2	LayerNorm	1 × 262144 × 4096
254	MoE_42	Shared-Expert MoE	1 × 262144 × 4096
255	Add_42_ffn	Add	1 × 262144 × 4096
256	LayerNorm_43_1	LayerNorm	1 × 262144 × 4096
257	Attention_43	Grouped Query Attn	1 × 262144 × 4096
258	Add_43_attn	Add	1 × 262144 × 4096
259	LayerNorm_43_2	LayerNorm	1 × 262144 × 4096
260	MoE_43	Shared-Expert MoE	1 × 262144 × 4096
261	Add_43_ffn	Add	1 × 262144 × 4096
262	LayerNorm_44_1	LayerNorm	1 × 262144 × 4096
263	Attention_44	Grouped Query Attn	1 × 262144 × 4096
264	Add_44_attn	Add	1 × 262144 × 4096
265	LayerNorm_44_2	LayerNorm	1 × 262144 × 4096
266	MoE_44	Shared-Expert MoE	1 × 262144 × 4096
267	Add_44_ffn	Add	1 × 262144 × 4096
268	LayerNorm_45_1	LayerNorm	1 × 262144 × 4096
269	Attention_45	Grouped Query Attn	1 × 262144 × 4096
270	Add_45_attn	Add	1 × 262144 × 4096
271	LayerNorm_45_2	LayerNorm	1 × 262144 × 4096
272	MoE_45	Shared-Expert MoE	1 × 262144 × 4096
273	Add_45_ffn	Add	1 × 262144 × 4096
274	LayerNorm_46_1	LayerNorm	1 × 262144 × 4096
275	Attention_46	Grouped Query Attn	1 × 262144 × 4096
276	Add_46_attn	Add	1 × 262144 × 4096
277	LayerNorm_46_2	LayerNorm	1 × 262144 × 4096
278	MoE_46	Shared-Expert MoE	1 × 262144 × 4096
279	Add_46_ffn	Add	1 × 262144 × 4096
280	LayerNorm_47_1	LayerNorm	1 × 262144 × 4096
281	Attention_47	Grouped Query Attn	1 × 262144 × 4096
282	Add_47_attn	Add	1 × 262144 × 4096
283	LayerNorm_47_2	LayerNorm	1 × 262144 × 4096
284	MoE_47	Shared-Expert MoE	1 × 262144 × 4096
285	Add_47_ffn	Add	1 × 262144 × 4096
286	LayerNorm_48_1	LayerNorm	1 × 262144 × 4096
287	Attention_48	Grouped Query Attn	1 × 262144 × 4096
288	Add_48_attn	Add	1 × 262144 × 4096
289	LayerNorm_48_2	LayerNorm	1 × 262144 × 4096
290	MoE_48	Shared-Expert MoE	1 × 262144 × 4096
291	Add_48_ffn	Add	1 × 262144 × 4096
292	LayerNorm_49_1	LayerNorm	1 × 262144 × 4096
293	Attention_49	Grouped Query Attn	1 × 262144 × 4096
294	Add_49_attn	Add	1 × 262144 × 4096
295	LayerNorm_49_2	LayerNorm	1 × 262144 × 4096
296	MoE_49	Shared-Expert MoE	1 × 262144 × 4096
297	Add_49_ffn	Add	1 × 262144 × 4096
298	LayerNorm_50_1	LayerNorm	1 × 262144 × 4096
299	Attention_50	Grouped Query Attn	1 × 262144 × 4096
300	Add_50_attn	Add	1 × 262144 × 4096
301	LayerNorm_50_2	LayerNorm	1 × 262144 × 4096
302	MoE_50	Shared-Expert MoE	1 × 262144 × 4096
303	Add_50_ffn	Add	1 × 262144 × 4096
304	LayerNorm_51_1	LayerNorm	1 × 262144 × 4096
305	Attention_51	Grouped Query Attn	1 × 262144 × 4096
306	Add_51_attn	Add	1 × 262144 × 4096
307	LayerNorm_51_2	LayerNorm	1 × 262144 × 4096
308	MoE_51	Shared-Expert MoE	1 × 262144 × 4096
309	Add_51_ffn	Add	1 × 262144 × 4096
310	LayerNorm_52_1	LayerNorm	1 × 262144 × 4096
311	Attention_52	Grouped Query Attn	1 × 262144 × 4096
312	Add_52_attn	Add	1 × 262144 × 4096
313	LayerNorm_52_2	LayerNorm	1 × 262144 × 4096
314	MoE_52	Shared-Expert MoE	1 × 262144 × 4096
315	Add_52_ffn	Add	1 × 262144 × 4096
316	LayerNorm_53_1	LayerNorm	1 × 262144 × 4096
317	Attention_53	Grouped Query Attn	1 × 262144 × 4096
318	Add_53_attn	Add	1 × 262144 × 4096
319	LayerNorm_53_2	LayerNorm	1 × 262144 × 4096
320	MoE_53	Shared-Expert MoE	1 × 262144 × 4096
321	Add_53_ffn	Add	1 × 262144 × 4096
322	LayerNorm_54_1	LayerNorm	1 × 262144 × 4096
323	Attention_54	Grouped Query Attn	1 × 262144 × 4096
324	Add_54_attn	Add	1 × 262144 × 4096
325	LayerNorm_54_2	LayerNorm	1 × 262144 × 4096
326	MoE_54	Shared-Expert MoE	1 × 262144 × 4096
327	Add_54_ffn	Add	1 × 262144 × 4096
328	LayerNorm_55_1	LayerNorm	1 × 262144 × 4096
329	Attention_55	Grouped Query Attn	1 × 262144 × 4096
330	Add_55_attn	Add	1 × 262144 × 4096
331	LayerNorm_55_2	LayerNorm	1 × 262144 × 4096
332	MoE_55	Shared-Expert MoE	1 × 262144 × 4096
333	Add_55_ffn	Add	1 × 262144 × 4096
334	LayerNorm_56_1	LayerNorm	1 × 262144 × 4096
335	Attention_56	Grouped Query Attn	1 × 262144 × 4096
336	Add_56_attn	Add	1 × 262144 × 4096
337	LayerNorm_56_2	LayerNorm	1 × 262144 × 4096
338	MoE_56	Shared-Expert MoE	1 × 262144 × 4096
339	Add_56_ffn	Add	1 × 262144 × 4096
340	LayerNorm_57_1	LayerNorm	1 × 262144 × 4096
341	Attention_57	Grouped Query Attn	1 × 262144 × 4096
342	Add_57_attn	Add	1 × 262144 × 4096
343	LayerNorm_57_2	LayerNorm	1 × 262144 × 4096
344	MoE_57	Shared-Expert MoE	1 × 262144 × 4096
345	Add_57_ffn	Add	1 × 262144 × 4096
346	LayerNorm_58_1	LayerNorm	1 × 262144 × 4096
347	Attention_58	Grouped Query Attn	1 × 262144 × 4096
348	Add_58_attn	Add	1 × 262144 × 4096
349	LayerNorm_58_2	LayerNorm	1 × 262144 × 4096
350	MoE_58	Shared-Expert MoE	1 × 262144 × 4096
351	Add_58_ffn	Add	1 × 262144 × 4096
352	LayerNorm_59_1	LayerNorm	1 × 262144 × 4096
353	Attention_59	Grouped Query Attn	1 × 262144 × 4096
354	Add_59_attn	Add	1 × 262144 × 4096
355	LayerNorm_59_2	LayerNorm	1 × 262144 × 4096
356	MoE_59	Shared-Expert MoE	1 × 262144 × 4096
357	Add_59_ffn	Add	1 × 262144 × 4096
358	LayerNorm_60_1	LayerNorm	1 × 262144 × 4096
359	Attention_60	Grouped Query Attn	1 × 262144 × 4096
360	Add_60_attn	Add	1 × 262144 × 4096
361	LayerNorm_60_2	LayerNorm	1 × 262144 × 4096
362	MoE_60	Shared-Expert MoE	1 × 262144 × 4096
363	Add_60_ffn	Add	1 × 262144 × 4096
364	LayerNorm_61_1	LayerNorm	1 × 262144 × 4096
365	Attention_61	Grouped Query Attn	1 × 262144 × 4096
366	Add_61_attn	Add	1 × 262144 × 4096
367	LayerNorm_61_2	LayerNorm	1 × 262144 × 4096
368	MoE_61	Shared-Expert MoE	1 × 262144 × 4096
369	Add_61_ffn	Add	1 × 262144 × 4096
370	LayerNorm_62_1	LayerNorm	1 × 262144 × 4096
371	Attention_62	Grouped Query Attn	1 × 262144 × 4096
372	Add_62_attn	Add	1 × 262144 × 4096
373	LayerNorm_62_2	LayerNorm	1 × 262144 × 4096
374	MoE_62	Shared-Expert MoE	1 × 262144 × 4096
375	Add_62_ffn	Add	1 × 262144 × 4096
376	LayerNorm_63_1	LayerNorm	1 × 262144 × 4096
377	Attention_63	Grouped Query Attn	1 × 262144 × 4096
378	Add_63_attn	Add	1 × 262144 × 4096
379	LayerNorm_63_2	LayerNorm	1 × 262144 × 4096
380	MoE_63	Shared-Expert MoE	1 × 262144 × 4096
381	Add_63_ffn	Add	1 × 262144 × 4096
382	LayerNorm_64_1	LayerNorm	1 × 262144 × 4096
383	Attention_64	Grouped Query Attn	1 × 262144 × 4096
384	Add_64_attn	Add	1 × 262144 × 4096
385	LayerNorm_64_2	LayerNorm	1 × 262144 × 4096
386	MoE_64	Shared-Expert MoE	1 × 262144 × 4096
387	Add_64_ffn	Add	1 × 262144 × 4096
388	LayerNorm_65_1	LayerNorm	1 × 262144 × 4096
389	Attention_65	Grouped Query Attn	1 × 262144 × 4096
390	Add_65_attn	Add	1 × 262144 × 4096
391	LayerNorm_65_2	LayerNorm	1 × 262144 × 4096
392	MoE_65	Shared-Expert MoE	1 × 262144 × 4096
393	Add_65_ffn	Add	1 × 262144 × 4096
394	LayerNorm_66_1	LayerNorm	1 × 262144 × 4096
395	Attention_66	Grouped Query Attn	1 × 262144 × 4096
396	Add_66_attn	Add	1 × 262144 × 4096
397	LayerNorm_66_2	LayerNorm	1 × 262144 × 4096
398	MoE_66	Shared-Expert MoE	1 × 262144 × 4096
399	Add_66_ffn	Add	1 × 262144 × 4096
400	LayerNorm_67_1	LayerNorm	1 × 262144 × 4096
401	Attention_67	Grouped Query Attn	1 × 262144 × 4096
402	Add_67_attn	Add	1 × 262144 × 4096
403	LayerNorm_67_2	LayerNorm	1 × 262144 × 4096
404	MoE_67	Shared-Expert MoE	1 × 262144 × 4096
405	Add_67_ffn	Add	1 × 262144 × 4096
406	LayerNorm_68_1	LayerNorm	1 × 262144 × 4096
407	Attention_68	Grouped Query Attn	1 × 262144 × 4096
408	Add_68_attn	Add	1 × 262144 × 4096
409	LayerNorm_68_2	LayerNorm	1 × 262144 × 4096
410	MoE_68	Shared-Expert MoE	1 × 262144 × 4096
411	Add_68_ffn	Add	1 × 262144 × 4096
412	LayerNorm_69_1	LayerNorm	1 × 262144 × 4096
413	Attention_69	Grouped Query Attn	1 × 262144 × 4096
414	Add_69_attn	Add	1 × 262144 × 4096
415	LayerNorm_69_2	LayerNorm	1 × 262144 × 4096
416	MoE_69	Shared-Expert MoE	1 × 262144 × 4096
417	Add_69_ffn	Add	1 × 262144 × 4096
418	LayerNorm_70_1	LayerNorm	1 × 262144 × 4096
419	Attention_70	Grouped Query Attn	1 × 262144 × 4096
420	Add_70_attn	Add	1 × 262144 × 4096
421	LayerNorm_70_2	LayerNorm	1 × 262144 × 4096
422	MoE_70	Shared-Expert MoE	1 × 262144 × 4096
423	Add_70_ffn	Add	1 × 262144 × 4096
424	LayerNorm_71_1	LayerNorm	1 × 262144 × 4096
425	Attention_71	Grouped Query Attn	1 × 262144 × 4096
426	Add_71_attn	Add	1 × 262144 × 4096
427	LayerNorm_71_2	LayerNorm	1 × 262144 × 4096
428	MoE_71	Shared-Expert MoE	1 × 262144 × 4096
429	Add_71_ffn	Add	1 × 262144 × 4096
430	LayerNorm_72_1	LayerNorm	1 × 262144 × 4096
431	Attention_72	Grouped Query Attn	1 × 262144 × 4096
432	Add_72_attn	Add	1 × 262144 × 4096
433	LayerNorm_72_2	LayerNorm	1 × 262144 × 4096
434	MoE_72	Shared-Expert MoE	1 × 262144 × 4096
435	Add_72_ffn	Add	1 × 262144 × 4096
436	LayerNorm_73_1	LayerNorm	1 × 262144 × 4096
437	Attention_73	Grouped Query Attn	1 × 262144 × 4096
438	Add_73_attn	Add	1 × 262144 × 4096
439	LayerNorm_73_2	LayerNorm	1 × 262144 × 4096
440	MoE_73	Shared-Expert MoE	1 × 262144 × 4096
441	Add_73_ffn	Add	1 × 262144 × 4096
442	LayerNorm_74_1	LayerNorm	1 × 262144 × 4096
443	Attention_74	Grouped Query Attn	1 × 262144 × 4096
444	Add_74_attn	Add	1 × 262144 × 4096
445	LayerNorm_74_2	LayerNorm	1 × 262144 × 4096
446	MoE_74	Shared-Expert MoE	1 × 262144 × 4096
447	Add_74_ffn	Add	1 × 262144 × 4096
448	LayerNorm_75_1	LayerNorm	1 × 262144 × 4096
449	Attention_75	Grouped Query Attn	1 × 262144 × 4096
450	Add_75_attn	Add	1 × 262144 × 4096
451	LayerNorm_75_2	LayerNorm	1 × 262144 × 4096
452	MoE_75	Shared-Expert MoE	1 × 262144 × 4096
453	Add_75_ffn	Add	1 × 262144 × 4096
454	LayerNorm_76_1	LayerNorm	1 × 262144 × 4096
455	Attention_76	Grouped Query Attn	1 × 262144 × 4096
456	Add_76_attn	Add	1 × 262144 × 4096
457	LayerNorm_76_2	LayerNorm	1 × 262144 × 4096
458	MoE_76	Shared-Expert MoE	1 × 262144 × 4096
459	Add_76_ffn	Add	1 × 262144 × 4096
460	LayerNorm_77_1	LayerNorm	1 × 262144 × 4096
461	Attention_77	Grouped Query Attn	1 × 262144 × 4096
462	Add_77_attn	Add	1 × 262144 × 4096
463	LayerNorm_77_2	LayerNorm	1 × 262144 × 4096
464	MoE_77	Shared-Expert MoE	1 × 262144 × 4096
465	Add_77_ffn	Add	1 × 262144 × 4096
466	LayerNorm_78_1	LayerNorm	1 × 262144 × 4096
467	Attention_78	Grouped Query Attn	1 × 262144 × 4096
468	Add_78_attn	Add	1 × 262144 × 4096
469	LayerNorm_78_2	LayerNorm	1 × 262144 × 4096
470	MoE_78	Shared-Expert MoE	1 × 262144 × 4096
471	Add_78_ffn	Add	1 × 262144 × 4096
472	LayerNorm_79_1	LayerNorm	1 × 262144 × 4096
473	Attention_79	Grouped Query Attn	1 × 262144 × 4096
474	Add_79_attn	Add	1 × 262144 × 4096
475	LayerNorm_79_2	LayerNorm	1 × 262144 × 4096
476	MoE_79	Shared-Expert MoE	1 × 262144 × 4096
477	Add_79_ffn	Add	1 × 262144 × 4096
478	LayerNorm_80_1	LayerNorm	1 × 262144 × 4096
479	Attention_80	Grouped Query Attn	1 × 262144 × 4096
480	Add_80_attn	Add	1 × 262144 × 4096
481	LayerNorm_80_2	LayerNorm	1 × 262144 × 4096
482	MoE_80	Shared-Expert MoE	1 × 262144 × 4096
483	Add_80_ffn	Add	1 × 262144 × 4096
484	LayerNorm_81_1	LayerNorm	1 × 262144 × 4096
485	Attention_81	Grouped Query Attn	1 × 262144 × 4096
486	Add_81_attn	Add	1 × 262144 × 4096
487	LayerNorm_81_2	LayerNorm	1 × 262144 × 4096
488	MoE_81	Shared-Expert MoE	1 × 262144 × 4096
489	Add_81_ffn	Add	1 × 262144 × 4096
490	LayerNorm_82_1	LayerNorm	1 × 262144 × 4096
491	Attention_82	Grouped Query Attn	1 × 262144 × 4096
492	Add_82_attn	Add	1 × 262144 × 4096
493	LayerNorm_82_2	LayerNorm	1 × 262144 × 4096
494	MoE_82	Shared-Expert MoE	1 × 262144 × 4096
495	Add_82_ffn	Add	1 × 262144 × 4096
496	LayerNorm_83_1	LayerNorm	1 × 262144 × 4096
497	Attention_83	Grouped Query Attn	1 × 262144 × 4096
498	Add_83_attn	Add	1 × 262144 × 4096
499	LayerNorm_83_2	LayerNorm	1 × 262144 × 4096
500	MoE_83	Shared-Expert MoE	1 × 262144 × 4096
501	Add_83_ffn	Add	1 × 262144 × 4096
502	LayerNorm_84_1	LayerNorm	1 × 262144 × 4096
503	Attention_84	Grouped Query Attn	1 × 262144 × 4096
504	Add_84_attn	Add	1 × 262144 × 4096
505	LayerNorm_84_2	LayerNorm	1 × 262144 × 4096
506	MoE_84	Shared-Expert MoE	1 × 262144 × 4096
507	Add_84_ffn	Add	1 × 262144 × 4096
508	LayerNorm_85_1	LayerNorm	1 × 262144 × 4096
509	Attention_85	Grouped Query Attn	1 × 262144 × 4096
510	Add_85_attn	Add	1 × 262144 × 4096
511	LayerNorm_85_2	LayerNorm	1 × 262144 × 4096
512	MoE_85	Shared-Expert MoE	1 × 262144 × 4096
513	Add_85_ffn	Add	1 × 262144 × 4096
514	LayerNorm_86_1	LayerNorm	1 × 262144 × 4096
515	Attention_86	Grouped Query Attn	1 × 262144 × 4096
516	Add_86_attn	Add	1 × 262144 × 4096
517	LayerNorm_86_2	LayerNorm	1 × 262144 × 4096
518	MoE_86	Shared-Expert MoE	1 × 262144 × 4096
519	Add_86_ffn	Add	1 × 262144 × 4096
520	LayerNorm_87_1	LayerNorm	1 × 262144 × 4096
521	Attention_87	Grouped Query Attn	1 × 262144 × 4096
522	Add_87_attn	Add	1 × 262144 × 4096
523	LayerNorm_87_2	LayerNorm	1 × 262144 × 4096
524	MoE_87	Shared-Expert MoE	1 × 262144 × 4096
525	Add_87_ffn	Add	1 × 262144 × 4096
526	LayerNorm_88_1	LayerNorm	1 × 262144 × 4096
527	Attention_88	Grouped Query Attn	1 × 262144 × 4096
528	Add_88_attn	Add	1 × 262144 × 4096
529	LayerNorm_88_2	LayerNorm	1 × 262144 × 4096
530	MoE_88	Shared-Expert MoE	1 × 262144 × 4096
531	Add_88_ffn	Add	1 × 262144 × 4096
532	Final_LayerNorm	LayerNorm	1 × 262144 × 4096
533	LM_Head	Linear	1 × 262144 × 131072
534	Output	Output	1 × 262144 × 131072

What the verifier says

infoAt 88 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other nemotron_h checkpoints

NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

102B derived · +224% against the checkpoint

NVIDIA-Nemotron-3-Nano-30B-A3B-FP8

102B derived · +224% against the checkpoint

NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4

102B derived · +461% against the checkpoint

NVIDIA-Nemotron-3-Super-120B-A12B-BF16

1496B derived · +1110% against the checkpoint