Models / nemotron-nas

Llama-3_3-Nemotron-Super-49B-v1

Reconstructed from its own config.json with no weights read. 134K downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

29.62B

29,617,427,712 parameters

In the published checkpoint

49.87B

49,867,145,216 scalars · safetensors.total, read 2025-10-15

Delta

-40.6%

custom-code This repository ships its own modeling code (`auto_map`, e.g. `configuration_decilm.py`), so `config.json` names a class in the repo rather than an architecture `transformers` defines. The graph below is what those config keys mean under `transformers` semantics, which is not necessarily what the repo's own file builds. A gap here is a statement about what we read, not about the checkpoint.

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.

Layers

484

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$561534.71

Card	Memory
T4 (16GB)	weights + activations	does not fit
A100 (40GB)	weights + activations	does not fit
H100 (80GB)	weights + activations	does not fit

Structure

486 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Output shape
1	Input	Input	1 × 131072
2	Embedding	Embedding	1 × 131072 × 8192
3	RoPE	RoPE	1 × 131072 × 8192
4	RMSNorm_1_1	RMSNorm	1 × 131072 × 8192
5	Attention_1	Grouped Query Attn	1 × 131072 × 8192
6	Add_1_attn	Add	1 × 131072 × 8192
7	RMSNorm_1_2	RMSNorm	1 × 131072 × 8192
8	FFN_1	SwiGLU	1 × 131072 × 8192
9	Add_1_ffn	Add	1 × 131072 × 8192
10	RMSNorm_2_1	RMSNorm	1 × 131072 × 8192
11	Attention_2	Grouped Query Attn	1 × 131072 × 8192
12	Add_2_attn	Add	1 × 131072 × 8192
13	RMSNorm_2_2	RMSNorm	1 × 131072 × 8192
14	FFN_2	SwiGLU	1 × 131072 × 8192
15	Add_2_ffn	Add	1 × 131072 × 8192
16	RMSNorm_3_1	RMSNorm	1 × 131072 × 8192
17	Attention_3	Grouped Query Attn	1 × 131072 × 8192
18	Add_3_attn	Add	1 × 131072 × 8192
19	RMSNorm_3_2	RMSNorm	1 × 131072 × 8192
20	FFN_3	SwiGLU	1 × 131072 × 8192
21	Add_3_ffn	Add	1 × 131072 × 8192
22	RMSNorm_4_1	RMSNorm	1 × 131072 × 8192
23	Attention_4	Grouped Query Attn	1 × 131072 × 8192
24	Add_4_attn	Add	1 × 131072 × 8192
25	RMSNorm_4_2	RMSNorm	1 × 131072 × 8192
26	FFN_4	SwiGLU	1 × 131072 × 8192
27	Add_4_ffn	Add	1 × 131072 × 8192
28	RMSNorm_5_1	RMSNorm	1 × 131072 × 8192
29	Attention_5	Grouped Query Attn	1 × 131072 × 8192
30	Add_5_attn	Add	1 × 131072 × 8192
31	RMSNorm_5_2	RMSNorm	1 × 131072 × 8192
32	FFN_5	SwiGLU	1 × 131072 × 8192
33	Add_5_ffn	Add	1 × 131072 × 8192
34	RMSNorm_6_1	RMSNorm	1 × 131072 × 8192
35	Attention_6	Grouped Query Attn	1 × 131072 × 8192
36	Add_6_attn	Add	1 × 131072 × 8192
37	RMSNorm_6_2	RMSNorm	1 × 131072 × 8192
38	FFN_6	SwiGLU	1 × 131072 × 8192
39	Add_6_ffn	Add	1 × 131072 × 8192
40	RMSNorm_7_1	RMSNorm	1 × 131072 × 8192
41	Attention_7	Grouped Query Attn	1 × 131072 × 8192
42	Add_7_attn	Add	1 × 131072 × 8192
43	RMSNorm_7_2	RMSNorm	1 × 131072 × 8192
44	FFN_7	SwiGLU	1 × 131072 × 8192
45	Add_7_ffn	Add	1 × 131072 × 8192
46	RMSNorm_8_1	RMSNorm	1 × 131072 × 8192
47	Attention_8	Grouped Query Attn	1 × 131072 × 8192
48	Add_8_attn	Add	1 × 131072 × 8192
49	RMSNorm_8_2	RMSNorm	1 × 131072 × 8192
50	FFN_8	SwiGLU	1 × 131072 × 8192
51	Add_8_ffn	Add	1 × 131072 × 8192
52	RMSNorm_9_1	RMSNorm	1 × 131072 × 8192
53	Attention_9	Grouped Query Attn	1 × 131072 × 8192
54	Add_9_attn	Add	1 × 131072 × 8192
55	RMSNorm_9_2	RMSNorm	1 × 131072 × 8192
56	FFN_9	SwiGLU	1 × 131072 × 8192
57	Add_9_ffn	Add	1 × 131072 × 8192
58	RMSNorm_10_1	RMSNorm	1 × 131072 × 8192
59	Attention_10	Grouped Query Attn	1 × 131072 × 8192
60	Add_10_attn	Add	1 × 131072 × 8192
61	RMSNorm_10_2	RMSNorm	1 × 131072 × 8192
62	FFN_10	SwiGLU	1 × 131072 × 8192
63	Add_10_ffn	Add	1 × 131072 × 8192
64	RMSNorm_11_1	RMSNorm	1 × 131072 × 8192
65	Attention_11	Grouped Query Attn	1 × 131072 × 8192
66	Add_11_attn	Add	1 × 131072 × 8192
67	RMSNorm_11_2	RMSNorm	1 × 131072 × 8192
68	FFN_11	SwiGLU	1 × 131072 × 8192
69	Add_11_ffn	Add	1 × 131072 × 8192
70	RMSNorm_12_1	RMSNorm	1 × 131072 × 8192
71	Attention_12	Grouped Query Attn	1 × 131072 × 8192
72	Add_12_attn	Add	1 × 131072 × 8192
73	RMSNorm_12_2	RMSNorm	1 × 131072 × 8192
74	FFN_12	SwiGLU	1 × 131072 × 8192
75	Add_12_ffn	Add	1 × 131072 × 8192
76	RMSNorm_13_1	RMSNorm	1 × 131072 × 8192
77	Attention_13	Grouped Query Attn	1 × 131072 × 8192
78	Add_13_attn	Add	1 × 131072 × 8192
79	RMSNorm_13_2	RMSNorm	1 × 131072 × 8192
80	FFN_13	SwiGLU	1 × 131072 × 8192
81	Add_13_ffn	Add	1 × 131072 × 8192
82	RMSNorm_14_1	RMSNorm	1 × 131072 × 8192
83	Attention_14	Grouped Query Attn	1 × 131072 × 8192
84	Add_14_attn	Add	1 × 131072 × 8192
85	RMSNorm_14_2	RMSNorm	1 × 131072 × 8192
86	FFN_14	SwiGLU	1 × 131072 × 8192
87	Add_14_ffn	Add	1 × 131072 × 8192
88	RMSNorm_15_1	RMSNorm	1 × 131072 × 8192
89	Attention_15	Grouped Query Attn	1 × 131072 × 8192
90	Add_15_attn	Add	1 × 131072 × 8192
91	RMSNorm_15_2	RMSNorm	1 × 131072 × 8192
92	FFN_15	SwiGLU	1 × 131072 × 8192
93	Add_15_ffn	Add	1 × 131072 × 8192
94	RMSNorm_16_1	RMSNorm	1 × 131072 × 8192
95	Attention_16	Grouped Query Attn	1 × 131072 × 8192
96	Add_16_attn	Add	1 × 131072 × 8192
97	RMSNorm_16_2	RMSNorm	1 × 131072 × 8192
98	FFN_16	SwiGLU	1 × 131072 × 8192
99	Add_16_ffn	Add	1 × 131072 × 8192
100	RMSNorm_17_1	RMSNorm	1 × 131072 × 8192
101	Attention_17	Grouped Query Attn	1 × 131072 × 8192
102	Add_17_attn	Add	1 × 131072 × 8192
103	RMSNorm_17_2	RMSNorm	1 × 131072 × 8192
104	FFN_17	SwiGLU	1 × 131072 × 8192
105	Add_17_ffn	Add	1 × 131072 × 8192
106	RMSNorm_18_1	RMSNorm	1 × 131072 × 8192
107	Attention_18	Grouped Query Attn	1 × 131072 × 8192
108	Add_18_attn	Add	1 × 131072 × 8192
109	RMSNorm_18_2	RMSNorm	1 × 131072 × 8192
110	FFN_18	SwiGLU	1 × 131072 × 8192
111	Add_18_ffn	Add	1 × 131072 × 8192
112	RMSNorm_19_1	RMSNorm	1 × 131072 × 8192
113	Attention_19	Grouped Query Attn	1 × 131072 × 8192
114	Add_19_attn	Add	1 × 131072 × 8192
115	RMSNorm_19_2	RMSNorm	1 × 131072 × 8192
116	FFN_19	SwiGLU	1 × 131072 × 8192
117	Add_19_ffn	Add	1 × 131072 × 8192
118	RMSNorm_20_1	RMSNorm	1 × 131072 × 8192
119	Attention_20	Grouped Query Attn	1 × 131072 × 8192
120	Add_20_attn	Add	1 × 131072 × 8192
121	RMSNorm_20_2	RMSNorm	1 × 131072 × 8192
122	FFN_20	SwiGLU	1 × 131072 × 8192
123	Add_20_ffn	Add	1 × 131072 × 8192
124	RMSNorm_21_1	RMSNorm	1 × 131072 × 8192
125	Attention_21	Grouped Query Attn	1 × 131072 × 8192
126	Add_21_attn	Add	1 × 131072 × 8192
127	RMSNorm_21_2	RMSNorm	1 × 131072 × 8192
128	FFN_21	SwiGLU	1 × 131072 × 8192
129	Add_21_ffn	Add	1 × 131072 × 8192
130	RMSNorm_22_1	RMSNorm	1 × 131072 × 8192
131	Attention_22	Grouped Query Attn	1 × 131072 × 8192
132	Add_22_attn	Add	1 × 131072 × 8192
133	RMSNorm_22_2	RMSNorm	1 × 131072 × 8192
134	FFN_22	SwiGLU	1 × 131072 × 8192
135	Add_22_ffn	Add	1 × 131072 × 8192
136	RMSNorm_23_1	RMSNorm	1 × 131072 × 8192
137	Attention_23	Grouped Query Attn	1 × 131072 × 8192
138	Add_23_attn	Add	1 × 131072 × 8192
139	RMSNorm_23_2	RMSNorm	1 × 131072 × 8192
140	FFN_23	SwiGLU	1 × 131072 × 8192
141	Add_23_ffn	Add	1 × 131072 × 8192
142	RMSNorm_24_1	RMSNorm	1 × 131072 × 8192
143	Attention_24	Grouped Query Attn	1 × 131072 × 8192
144	Add_24_attn	Add	1 × 131072 × 8192
145	RMSNorm_24_2	RMSNorm	1 × 131072 × 8192
146	FFN_24	SwiGLU	1 × 131072 × 8192
147	Add_24_ffn	Add	1 × 131072 × 8192
148	RMSNorm_25_1	RMSNorm	1 × 131072 × 8192
149	Attention_25	Grouped Query Attn	1 × 131072 × 8192
150	Add_25_attn	Add	1 × 131072 × 8192
151	RMSNorm_25_2	RMSNorm	1 × 131072 × 8192
152	FFN_25	SwiGLU	1 × 131072 × 8192
153	Add_25_ffn	Add	1 × 131072 × 8192
154	RMSNorm_26_1	RMSNorm	1 × 131072 × 8192
155	Attention_26	Grouped Query Attn	1 × 131072 × 8192
156	Add_26_attn	Add	1 × 131072 × 8192
157	RMSNorm_26_2	RMSNorm	1 × 131072 × 8192
158	FFN_26	SwiGLU	1 × 131072 × 8192
159	Add_26_ffn	Add	1 × 131072 × 8192
160	RMSNorm_27_1	RMSNorm	1 × 131072 × 8192
161	Attention_27	Grouped Query Attn	1 × 131072 × 8192
162	Add_27_attn	Add	1 × 131072 × 8192
163	RMSNorm_27_2	RMSNorm	1 × 131072 × 8192
164	FFN_27	SwiGLU	1 × 131072 × 8192
165	Add_27_ffn	Add	1 × 131072 × 8192
166	RMSNorm_28_1	RMSNorm	1 × 131072 × 8192
167	Attention_28	Grouped Query Attn	1 × 131072 × 8192
168	Add_28_attn	Add	1 × 131072 × 8192
169	RMSNorm_28_2	RMSNorm	1 × 131072 × 8192
170	FFN_28	SwiGLU	1 × 131072 × 8192
171	Add_28_ffn	Add	1 × 131072 × 8192
172	RMSNorm_29_1	RMSNorm	1 × 131072 × 8192
173	Attention_29	Grouped Query Attn	1 × 131072 × 8192
174	Add_29_attn	Add	1 × 131072 × 8192
175	RMSNorm_29_2	RMSNorm	1 × 131072 × 8192
176	FFN_29	SwiGLU	1 × 131072 × 8192
177	Add_29_ffn	Add	1 × 131072 × 8192
178	RMSNorm_30_1	RMSNorm	1 × 131072 × 8192
179	Attention_30	Grouped Query Attn	1 × 131072 × 8192
180	Add_30_attn	Add	1 × 131072 × 8192
181	RMSNorm_30_2	RMSNorm	1 × 131072 × 8192
182	FFN_30	SwiGLU	1 × 131072 × 8192
183	Add_30_ffn	Add	1 × 131072 × 8192
184	RMSNorm_31_1	RMSNorm	1 × 131072 × 8192
185	Attention_31	Grouped Query Attn	1 × 131072 × 8192
186	Add_31_attn	Add	1 × 131072 × 8192
187	RMSNorm_31_2	RMSNorm	1 × 131072 × 8192
188	FFN_31	SwiGLU	1 × 131072 × 8192
189	Add_31_ffn	Add	1 × 131072 × 8192
190	RMSNorm_32_1	RMSNorm	1 × 131072 × 8192
191	Attention_32	Grouped Query Attn	1 × 131072 × 8192
192	Add_32_attn	Add	1 × 131072 × 8192
193	RMSNorm_32_2	RMSNorm	1 × 131072 × 8192
194	FFN_32	SwiGLU	1 × 131072 × 8192
195	Add_32_ffn	Add	1 × 131072 × 8192
196	RMSNorm_33_1	RMSNorm	1 × 131072 × 8192
197	Attention_33	Grouped Query Attn	1 × 131072 × 8192
198	Add_33_attn	Add	1 × 131072 × 8192
199	RMSNorm_33_2	RMSNorm	1 × 131072 × 8192
200	FFN_33	SwiGLU	1 × 131072 × 8192
201	Add_33_ffn	Add	1 × 131072 × 8192
202	RMSNorm_34_1	RMSNorm	1 × 131072 × 8192
203	Attention_34	Grouped Query Attn	1 × 131072 × 8192
204	Add_34_attn	Add	1 × 131072 × 8192
205	RMSNorm_34_2	RMSNorm	1 × 131072 × 8192
206	FFN_34	SwiGLU	1 × 131072 × 8192
207	Add_34_ffn	Add	1 × 131072 × 8192
208	RMSNorm_35_1	RMSNorm	1 × 131072 × 8192
209	Attention_35	Grouped Query Attn	1 × 131072 × 8192
210	Add_35_attn	Add	1 × 131072 × 8192
211	RMSNorm_35_2	RMSNorm	1 × 131072 × 8192
212	FFN_35	SwiGLU	1 × 131072 × 8192
213	Add_35_ffn	Add	1 × 131072 × 8192
214	RMSNorm_36_1	RMSNorm	1 × 131072 × 8192
215	Attention_36	Grouped Query Attn	1 × 131072 × 8192
216	Add_36_attn	Add	1 × 131072 × 8192
217	RMSNorm_36_2	RMSNorm	1 × 131072 × 8192
218	FFN_36	SwiGLU	1 × 131072 × 8192
219	Add_36_ffn	Add	1 × 131072 × 8192
220	RMSNorm_37_1	RMSNorm	1 × 131072 × 8192
221	Attention_37	Grouped Query Attn	1 × 131072 × 8192
222	Add_37_attn	Add	1 × 131072 × 8192
223	RMSNorm_37_2	RMSNorm	1 × 131072 × 8192
224	FFN_37	SwiGLU	1 × 131072 × 8192
225	Add_37_ffn	Add	1 × 131072 × 8192
226	RMSNorm_38_1	RMSNorm	1 × 131072 × 8192
227	Attention_38	Grouped Query Attn	1 × 131072 × 8192
228	Add_38_attn	Add	1 × 131072 × 8192
229	RMSNorm_38_2	RMSNorm	1 × 131072 × 8192
230	FFN_38	SwiGLU	1 × 131072 × 8192
231	Add_38_ffn	Add	1 × 131072 × 8192
232	RMSNorm_39_1	RMSNorm	1 × 131072 × 8192
233	Attention_39	Grouped Query Attn	1 × 131072 × 8192
234	Add_39_attn	Add	1 × 131072 × 8192
235	RMSNorm_39_2	RMSNorm	1 × 131072 × 8192
236	FFN_39	SwiGLU	1 × 131072 × 8192
237	Add_39_ffn	Add	1 × 131072 × 8192
238	RMSNorm_40_1	RMSNorm	1 × 131072 × 8192
239	Attention_40	Grouped Query Attn	1 × 131072 × 8192
240	Add_40_attn	Add	1 × 131072 × 8192
241	RMSNorm_40_2	RMSNorm	1 × 131072 × 8192
242	FFN_40	SwiGLU	1 × 131072 × 8192
243	Add_40_ffn	Add	1 × 131072 × 8192
244	RMSNorm_41_1	RMSNorm	1 × 131072 × 8192
245	Attention_41	Grouped Query Attn	1 × 131072 × 8192
246	Add_41_attn	Add	1 × 131072 × 8192
247	RMSNorm_41_2	RMSNorm	1 × 131072 × 8192
248	FFN_41	SwiGLU	1 × 131072 × 8192
249	Add_41_ffn	Add	1 × 131072 × 8192
250	RMSNorm_42_1	RMSNorm	1 × 131072 × 8192
251	Attention_42	Grouped Query Attn	1 × 131072 × 8192
252	Add_42_attn	Add	1 × 131072 × 8192
253	RMSNorm_42_2	RMSNorm	1 × 131072 × 8192
254	FFN_42	SwiGLU	1 × 131072 × 8192
255	Add_42_ffn	Add	1 × 131072 × 8192
256	RMSNorm_43_1	RMSNorm	1 × 131072 × 8192
257	Attention_43	Grouped Query Attn	1 × 131072 × 8192
258	Add_43_attn	Add	1 × 131072 × 8192
259	RMSNorm_43_2	RMSNorm	1 × 131072 × 8192
260	FFN_43	SwiGLU	1 × 131072 × 8192
261	Add_43_ffn	Add	1 × 131072 × 8192
262	RMSNorm_44_1	RMSNorm	1 × 131072 × 8192
263	Attention_44	Grouped Query Attn	1 × 131072 × 8192
264	Add_44_attn	Add	1 × 131072 × 8192
265	RMSNorm_44_2	RMSNorm	1 × 131072 × 8192
266	FFN_44	SwiGLU	1 × 131072 × 8192
267	Add_44_ffn	Add	1 × 131072 × 8192
268	RMSNorm_45_1	RMSNorm	1 × 131072 × 8192
269	Attention_45	Grouped Query Attn	1 × 131072 × 8192
270	Add_45_attn	Add	1 × 131072 × 8192
271	RMSNorm_45_2	RMSNorm	1 × 131072 × 8192
272	FFN_45	SwiGLU	1 × 131072 × 8192
273	Add_45_ffn	Add	1 × 131072 × 8192
274	RMSNorm_46_1	RMSNorm	1 × 131072 × 8192
275	Attention_46	Grouped Query Attn	1 × 131072 × 8192
276	Add_46_attn	Add	1 × 131072 × 8192
277	RMSNorm_46_2	RMSNorm	1 × 131072 × 8192
278	FFN_46	SwiGLU	1 × 131072 × 8192
279	Add_46_ffn	Add	1 × 131072 × 8192
280	RMSNorm_47_1	RMSNorm	1 × 131072 × 8192
281	Attention_47	Grouped Query Attn	1 × 131072 × 8192
282	Add_47_attn	Add	1 × 131072 × 8192
283	RMSNorm_47_2	RMSNorm	1 × 131072 × 8192
284	FFN_47	SwiGLU	1 × 131072 × 8192
285	Add_47_ffn	Add	1 × 131072 × 8192
286	RMSNorm_48_1	RMSNorm	1 × 131072 × 8192
287	Attention_48	Grouped Query Attn	1 × 131072 × 8192
288	Add_48_attn	Add	1 × 131072 × 8192
289	RMSNorm_48_2	RMSNorm	1 × 131072 × 8192
290	FFN_48	SwiGLU	1 × 131072 × 8192
291	Add_48_ffn	Add	1 × 131072 × 8192
292	RMSNorm_49_1	RMSNorm	1 × 131072 × 8192
293	Attention_49	Grouped Query Attn	1 × 131072 × 8192
294	Add_49_attn	Add	1 × 131072 × 8192
295	RMSNorm_49_2	RMSNorm	1 × 131072 × 8192
296	FFN_49	SwiGLU	1 × 131072 × 8192
297	Add_49_ffn	Add	1 × 131072 × 8192
298	RMSNorm_50_1	RMSNorm	1 × 131072 × 8192
299	Attention_50	Grouped Query Attn	1 × 131072 × 8192
300	Add_50_attn	Add	1 × 131072 × 8192
301	RMSNorm_50_2	RMSNorm	1 × 131072 × 8192
302	FFN_50	SwiGLU	1 × 131072 × 8192
303	Add_50_ffn	Add	1 × 131072 × 8192
304	RMSNorm_51_1	RMSNorm	1 × 131072 × 8192
305	Attention_51	Grouped Query Attn	1 × 131072 × 8192
306	Add_51_attn	Add	1 × 131072 × 8192
307	RMSNorm_51_2	RMSNorm	1 × 131072 × 8192
308	FFN_51	SwiGLU	1 × 131072 × 8192
309	Add_51_ffn	Add	1 × 131072 × 8192
310	RMSNorm_52_1	RMSNorm	1 × 131072 × 8192
311	Attention_52	Grouped Query Attn	1 × 131072 × 8192
312	Add_52_attn	Add	1 × 131072 × 8192
313	RMSNorm_52_2	RMSNorm	1 × 131072 × 8192
314	FFN_52	SwiGLU	1 × 131072 × 8192
315	Add_52_ffn	Add	1 × 131072 × 8192
316	RMSNorm_53_1	RMSNorm	1 × 131072 × 8192
317	Attention_53	Grouped Query Attn	1 × 131072 × 8192
318	Add_53_attn	Add	1 × 131072 × 8192
319	RMSNorm_53_2	RMSNorm	1 × 131072 × 8192
320	FFN_53	SwiGLU	1 × 131072 × 8192
321	Add_53_ffn	Add	1 × 131072 × 8192
322	RMSNorm_54_1	RMSNorm	1 × 131072 × 8192
323	Attention_54	Grouped Query Attn	1 × 131072 × 8192
324	Add_54_attn	Add	1 × 131072 × 8192
325	RMSNorm_54_2	RMSNorm	1 × 131072 × 8192
326	FFN_54	SwiGLU	1 × 131072 × 8192
327	Add_54_ffn	Add	1 × 131072 × 8192
328	RMSNorm_55_1	RMSNorm	1 × 131072 × 8192
329	Attention_55	Grouped Query Attn	1 × 131072 × 8192
330	Add_55_attn	Add	1 × 131072 × 8192
331	RMSNorm_55_2	RMSNorm	1 × 131072 × 8192
332	FFN_55	SwiGLU	1 × 131072 × 8192
333	Add_55_ffn	Add	1 × 131072 × 8192
334	RMSNorm_56_1	RMSNorm	1 × 131072 × 8192
335	Attention_56	Grouped Query Attn	1 × 131072 × 8192
336	Add_56_attn	Add	1 × 131072 × 8192
337	RMSNorm_56_2	RMSNorm	1 × 131072 × 8192
338	FFN_56	SwiGLU	1 × 131072 × 8192
339	Add_56_ffn	Add	1 × 131072 × 8192
340	RMSNorm_57_1	RMSNorm	1 × 131072 × 8192
341	Attention_57	Grouped Query Attn	1 × 131072 × 8192
342	Add_57_attn	Add	1 × 131072 × 8192
343	RMSNorm_57_2	RMSNorm	1 × 131072 × 8192
344	FFN_57	SwiGLU	1 × 131072 × 8192
345	Add_57_ffn	Add	1 × 131072 × 8192
346	RMSNorm_58_1	RMSNorm	1 × 131072 × 8192
347	Attention_58	Grouped Query Attn	1 × 131072 × 8192
348	Add_58_attn	Add	1 × 131072 × 8192
349	RMSNorm_58_2	RMSNorm	1 × 131072 × 8192
350	FFN_58	SwiGLU	1 × 131072 × 8192
351	Add_58_ffn	Add	1 × 131072 × 8192
352	RMSNorm_59_1	RMSNorm	1 × 131072 × 8192
353	Attention_59	Grouped Query Attn	1 × 131072 × 8192
354	Add_59_attn	Add	1 × 131072 × 8192
355	RMSNorm_59_2	RMSNorm	1 × 131072 × 8192
356	FFN_59	SwiGLU	1 × 131072 × 8192
357	Add_59_ffn	Add	1 × 131072 × 8192
358	RMSNorm_60_1	RMSNorm	1 × 131072 × 8192
359	Attention_60	Grouped Query Attn	1 × 131072 × 8192
360	Add_60_attn	Add	1 × 131072 × 8192
361	RMSNorm_60_2	RMSNorm	1 × 131072 × 8192
362	FFN_60	SwiGLU	1 × 131072 × 8192
363	Add_60_ffn	Add	1 × 131072 × 8192
364	RMSNorm_61_1	RMSNorm	1 × 131072 × 8192
365	Attention_61	Grouped Query Attn	1 × 131072 × 8192
366	Add_61_attn	Add	1 × 131072 × 8192
367	RMSNorm_61_2	RMSNorm	1 × 131072 × 8192
368	FFN_61	SwiGLU	1 × 131072 × 8192
369	Add_61_ffn	Add	1 × 131072 × 8192
370	RMSNorm_62_1	RMSNorm	1 × 131072 × 8192
371	Attention_62	Grouped Query Attn	1 × 131072 × 8192
372	Add_62_attn	Add	1 × 131072 × 8192
373	RMSNorm_62_2	RMSNorm	1 × 131072 × 8192
374	FFN_62	SwiGLU	1 × 131072 × 8192
375	Add_62_ffn	Add	1 × 131072 × 8192
376	RMSNorm_63_1	RMSNorm	1 × 131072 × 8192
377	Attention_63	Grouped Query Attn	1 × 131072 × 8192
378	Add_63_attn	Add	1 × 131072 × 8192
379	RMSNorm_63_2	RMSNorm	1 × 131072 × 8192
380	FFN_63	SwiGLU	1 × 131072 × 8192
381	Add_63_ffn	Add	1 × 131072 × 8192
382	RMSNorm_64_1	RMSNorm	1 × 131072 × 8192
383	Attention_64	Grouped Query Attn	1 × 131072 × 8192
384	Add_64_attn	Add	1 × 131072 × 8192
385	RMSNorm_64_2	RMSNorm	1 × 131072 × 8192
386	FFN_64	SwiGLU	1 × 131072 × 8192
387	Add_64_ffn	Add	1 × 131072 × 8192
388	RMSNorm_65_1	RMSNorm	1 × 131072 × 8192
389	Attention_65	Grouped Query Attn	1 × 131072 × 8192
390	Add_65_attn	Add	1 × 131072 × 8192
391	RMSNorm_65_2	RMSNorm	1 × 131072 × 8192
392	FFN_65	SwiGLU	1 × 131072 × 8192
393	Add_65_ffn	Add	1 × 131072 × 8192
394	RMSNorm_66_1	RMSNorm	1 × 131072 × 8192
395	Attention_66	Grouped Query Attn	1 × 131072 × 8192
396	Add_66_attn	Add	1 × 131072 × 8192
397	RMSNorm_66_2	RMSNorm	1 × 131072 × 8192
398	FFN_66	SwiGLU	1 × 131072 × 8192
399	Add_66_ffn	Add	1 × 131072 × 8192
400	RMSNorm_67_1	RMSNorm	1 × 131072 × 8192
401	Attention_67	Grouped Query Attn	1 × 131072 × 8192
402	Add_67_attn	Add	1 × 131072 × 8192
403	RMSNorm_67_2	RMSNorm	1 × 131072 × 8192
404	FFN_67	SwiGLU	1 × 131072 × 8192
405	Add_67_ffn	Add	1 × 131072 × 8192
406	RMSNorm_68_1	RMSNorm	1 × 131072 × 8192
407	Attention_68	Grouped Query Attn	1 × 131072 × 8192
408	Add_68_attn	Add	1 × 131072 × 8192
409	RMSNorm_68_2	RMSNorm	1 × 131072 × 8192
410	FFN_68	SwiGLU	1 × 131072 × 8192
411	Add_68_ffn	Add	1 × 131072 × 8192
412	RMSNorm_69_1	RMSNorm	1 × 131072 × 8192
413	Attention_69	Grouped Query Attn	1 × 131072 × 8192
414	Add_69_attn	Add	1 × 131072 × 8192
415	RMSNorm_69_2	RMSNorm	1 × 131072 × 8192
416	FFN_69	SwiGLU	1 × 131072 × 8192
417	Add_69_ffn	Add	1 × 131072 × 8192
418	RMSNorm_70_1	RMSNorm	1 × 131072 × 8192
419	Attention_70	Grouped Query Attn	1 × 131072 × 8192
420	Add_70_attn	Add	1 × 131072 × 8192
421	RMSNorm_70_2	RMSNorm	1 × 131072 × 8192
422	FFN_70	SwiGLU	1 × 131072 × 8192
423	Add_70_ffn	Add	1 × 131072 × 8192
424	RMSNorm_71_1	RMSNorm	1 × 131072 × 8192
425	Attention_71	Grouped Query Attn	1 × 131072 × 8192
426	Add_71_attn	Add	1 × 131072 × 8192
427	RMSNorm_71_2	RMSNorm	1 × 131072 × 8192
428	FFN_71	SwiGLU	1 × 131072 × 8192
429	Add_71_ffn	Add	1 × 131072 × 8192
430	RMSNorm_72_1	RMSNorm	1 × 131072 × 8192
431	Attention_72	Grouped Query Attn	1 × 131072 × 8192
432	Add_72_attn	Add	1 × 131072 × 8192
433	RMSNorm_72_2	RMSNorm	1 × 131072 × 8192
434	FFN_72	SwiGLU	1 × 131072 × 8192
435	Add_72_ffn	Add	1 × 131072 × 8192
436	RMSNorm_73_1	RMSNorm	1 × 131072 × 8192
437	Attention_73	Grouped Query Attn	1 × 131072 × 8192
438	Add_73_attn	Add	1 × 131072 × 8192
439	RMSNorm_73_2	RMSNorm	1 × 131072 × 8192
440	FFN_73	SwiGLU	1 × 131072 × 8192
441	Add_73_ffn	Add	1 × 131072 × 8192
442	RMSNorm_74_1	RMSNorm	1 × 131072 × 8192
443	Attention_74	Grouped Query Attn	1 × 131072 × 8192
444	Add_74_attn	Add	1 × 131072 × 8192
445	RMSNorm_74_2	RMSNorm	1 × 131072 × 8192
446	FFN_74	SwiGLU	1 × 131072 × 8192
447	Add_74_ffn	Add	1 × 131072 × 8192
448	RMSNorm_75_1	RMSNorm	1 × 131072 × 8192
449	Attention_75	Grouped Query Attn	1 × 131072 × 8192
450	Add_75_attn	Add	1 × 131072 × 8192
451	RMSNorm_75_2	RMSNorm	1 × 131072 × 8192
452	FFN_75	SwiGLU	1 × 131072 × 8192
453	Add_75_ffn	Add	1 × 131072 × 8192
454	RMSNorm_76_1	RMSNorm	1 × 131072 × 8192
455	Attention_76	Grouped Query Attn	1 × 131072 × 8192
456	Add_76_attn	Add	1 × 131072 × 8192
457	RMSNorm_76_2	RMSNorm	1 × 131072 × 8192
458	FFN_76	SwiGLU	1 × 131072 × 8192
459	Add_76_ffn	Add	1 × 131072 × 8192
460	RMSNorm_77_1	RMSNorm	1 × 131072 × 8192
461	Attention_77	Grouped Query Attn	1 × 131072 × 8192
462	Add_77_attn	Add	1 × 131072 × 8192
463	RMSNorm_77_2	RMSNorm	1 × 131072 × 8192
464	FFN_77	SwiGLU	1 × 131072 × 8192
465	Add_77_ffn	Add	1 × 131072 × 8192
466	RMSNorm_78_1	RMSNorm	1 × 131072 × 8192
467	Attention_78	Grouped Query Attn	1 × 131072 × 8192
468	Add_78_attn	Add	1 × 131072 × 8192
469	RMSNorm_78_2	RMSNorm	1 × 131072 × 8192
470	FFN_78	SwiGLU	1 × 131072 × 8192
471	Add_78_ffn	Add	1 × 131072 × 8192
472	RMSNorm_79_1	RMSNorm	1 × 131072 × 8192
473	Attention_79	Grouped Query Attn	1 × 131072 × 8192
474	Add_79_attn	Add	1 × 131072 × 8192
475	RMSNorm_79_2	RMSNorm	1 × 131072 × 8192
476	FFN_79	SwiGLU	1 × 131072 × 8192
477	Add_79_ffn	Add	1 × 131072 × 8192
478	RMSNorm_80_1	RMSNorm	1 × 131072 × 8192
479	Attention_80	Grouped Query Attn	1 × 131072 × 8192
480	Add_80_attn	Add	1 × 131072 × 8192
481	RMSNorm_80_2	RMSNorm	1 × 131072 × 8192
482	FFN_80	SwiGLU	1 × 131072 × 8192
483	Add_80_ffn	Add	1 × 131072 × 8192
484	Final_RMSNorm	RMSNorm	1 × 131072 × 8192
485	LM_Head	Linear	1 × 131072 × 128256
486	Output	Output	1 × 131072 × 128256

What the verifier says

infoLLaMA uses intermediateSize ≈ ⌊(8/3 × D) / 256⌋ × 256. Current: 3072 (0.38× embedDim). Expected: ~21760. Fix: Set intermediateSize to 21760 for embedDim=8192.
swiglu-dim-convention

warnLinear "LM_Head" is 8192 × 128256 = 1051M parameters (~3.9 GB float32). A single dense layer this large usually means a feature map was flattened without pooling first; embedding / vocab-projection heads are the expected exception. Fix: Add a Global Average Pool or more downsampling before the Linear, or factorize it (low-rank / bottleneck projection).
huge-linear-params

info80 attention layers at embedDim 8192 cache full per-head K/V: about 2560 KB per token at fp16, which dominates memory at long context. Grouped-query attention (e.g. 8:1) would cut this ~8×; multi-head latent attention (MLA) shrinks it ~10× or more. This is the move production LLMs make; it does not change the parameter count. Fix: Switch attention to groupedQueryAttention (set numKVHeads below numHeads, e.g. numHeads/4) or mla (a low-rank cached latent).
full-mha-serving-cost

infoAt 80 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

warnAcross 80 attention layers this design caches 2560 KB per token, so a single 8,192-token sequence needs ~21.5 GB of KV cache before weights or activations. That exceeds the 4 GB budget this rule assumes for serving headroom. Fix: Cut KV width: raise the GQA ratio (fewer numKVHeads), switch to MLA, reduce depth or embedDim, or accept a shorter serving context.
kv-cache-context-budget

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace nvidia/Llama-3_3-Nemotron-Super-49B-v1 --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other nemotron-nas checkpoints

Llama-3_3-Nemotron-Super-49B-v1_5-FP8

29.62B derived · -40.6% against the checkpoint