Qwen-72B

Reconstructed from its own config.json with no weights read. 3.7M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

121B

120,605,807,104 parameters

In the published checkpoint

72.29B

72,287,920,128 scalars · safetensors.total, read 2026-09-06

Delta

+66.8%

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.

Layers

484

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$172725.48

Card	Memory
T4 (16GB)	weights + activations	does not fit
A100 (40GB)	weights + activations	does not fit
H100 (80GB)	weights + activations	does not fit

Structure

486 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Output shape
1	Input	Input	1 × 32768
2	Embedding	Embedding	1 × 32768 × 8192
3	RoPE	RoPE	1 × 32768 × 8192
4	LayerNorm_1_1	LayerNorm	1 × 32768 × 8192
5	Attention_1	Grouped Query Attn	1 × 32768 × 8192
6	Add_1_attn	Add	1 × 32768 × 8192
7	LayerNorm_1_2	LayerNorm	1 × 32768 × 8192
8	FFN_1	SwiGLU	1 × 32768 × 8192
9	Add_1_ffn	Add	1 × 32768 × 8192
10	LayerNorm_2_1	LayerNorm	1 × 32768 × 8192
11	Attention_2	Grouped Query Attn	1 × 32768 × 8192
12	Add_2_attn	Add	1 × 32768 × 8192
13	LayerNorm_2_2	LayerNorm	1 × 32768 × 8192
14	FFN_2	SwiGLU	1 × 32768 × 8192
15	Add_2_ffn	Add	1 × 32768 × 8192
16	LayerNorm_3_1	LayerNorm	1 × 32768 × 8192
17	Attention_3	Grouped Query Attn	1 × 32768 × 8192
18	Add_3_attn	Add	1 × 32768 × 8192
19	LayerNorm_3_2	LayerNorm	1 × 32768 × 8192
20	FFN_3	SwiGLU	1 × 32768 × 8192
21	Add_3_ffn	Add	1 × 32768 × 8192
22	LayerNorm_4_1	LayerNorm	1 × 32768 × 8192
23	Attention_4	Grouped Query Attn	1 × 32768 × 8192
24	Add_4_attn	Add	1 × 32768 × 8192
25	LayerNorm_4_2	LayerNorm	1 × 32768 × 8192
26	FFN_4	SwiGLU	1 × 32768 × 8192
27	Add_4_ffn	Add	1 × 32768 × 8192
28	LayerNorm_5_1	LayerNorm	1 × 32768 × 8192
29	Attention_5	Grouped Query Attn	1 × 32768 × 8192
30	Add_5_attn	Add	1 × 32768 × 8192
31	LayerNorm_5_2	LayerNorm	1 × 32768 × 8192
32	FFN_5	SwiGLU	1 × 32768 × 8192
33	Add_5_ffn	Add	1 × 32768 × 8192
34	LayerNorm_6_1	LayerNorm	1 × 32768 × 8192
35	Attention_6	Grouped Query Attn	1 × 32768 × 8192
36	Add_6_attn	Add	1 × 32768 × 8192
37	LayerNorm_6_2	LayerNorm	1 × 32768 × 8192
38	FFN_6	SwiGLU	1 × 32768 × 8192
39	Add_6_ffn	Add	1 × 32768 × 8192
40	LayerNorm_7_1	LayerNorm	1 × 32768 × 8192
41	Attention_7	Grouped Query Attn	1 × 32768 × 8192
42	Add_7_attn	Add	1 × 32768 × 8192
43	LayerNorm_7_2	LayerNorm	1 × 32768 × 8192
44	FFN_7	SwiGLU	1 × 32768 × 8192
45	Add_7_ffn	Add	1 × 32768 × 8192
46	LayerNorm_8_1	LayerNorm	1 × 32768 × 8192
47	Attention_8	Grouped Query Attn	1 × 32768 × 8192
48	Add_8_attn	Add	1 × 32768 × 8192
49	LayerNorm_8_2	LayerNorm	1 × 32768 × 8192
50	FFN_8	SwiGLU	1 × 32768 × 8192
51	Add_8_ffn	Add	1 × 32768 × 8192
52	LayerNorm_9_1	LayerNorm	1 × 32768 × 8192
53	Attention_9	Grouped Query Attn	1 × 32768 × 8192
54	Add_9_attn	Add	1 × 32768 × 8192
55	LayerNorm_9_2	LayerNorm	1 × 32768 × 8192
56	FFN_9	SwiGLU	1 × 32768 × 8192
57	Add_9_ffn	Add	1 × 32768 × 8192
58	LayerNorm_10_1	LayerNorm	1 × 32768 × 8192
59	Attention_10	Grouped Query Attn	1 × 32768 × 8192
60	Add_10_attn	Add	1 × 32768 × 8192
61	LayerNorm_10_2	LayerNorm	1 × 32768 × 8192
62	FFN_10	SwiGLU	1 × 32768 × 8192
63	Add_10_ffn	Add	1 × 32768 × 8192
64	LayerNorm_11_1	LayerNorm	1 × 32768 × 8192
65	Attention_11	Grouped Query Attn	1 × 32768 × 8192
66	Add_11_attn	Add	1 × 32768 × 8192
67	LayerNorm_11_2	LayerNorm	1 × 32768 × 8192
68	FFN_11	SwiGLU	1 × 32768 × 8192
69	Add_11_ffn	Add	1 × 32768 × 8192
70	LayerNorm_12_1	LayerNorm	1 × 32768 × 8192
71	Attention_12	Grouped Query Attn	1 × 32768 × 8192
72	Add_12_attn	Add	1 × 32768 × 8192
73	LayerNorm_12_2	LayerNorm	1 × 32768 × 8192
74	FFN_12	SwiGLU	1 × 32768 × 8192
75	Add_12_ffn	Add	1 × 32768 × 8192
76	LayerNorm_13_1	LayerNorm	1 × 32768 × 8192
77	Attention_13	Grouped Query Attn	1 × 32768 × 8192
78	Add_13_attn	Add	1 × 32768 × 8192
79	LayerNorm_13_2	LayerNorm	1 × 32768 × 8192
80	FFN_13	SwiGLU	1 × 32768 × 8192
81	Add_13_ffn	Add	1 × 32768 × 8192
82	LayerNorm_14_1	LayerNorm	1 × 32768 × 8192
83	Attention_14	Grouped Query Attn	1 × 32768 × 8192
84	Add_14_attn	Add	1 × 32768 × 8192
85	LayerNorm_14_2	LayerNorm	1 × 32768 × 8192
86	FFN_14	SwiGLU	1 × 32768 × 8192
87	Add_14_ffn	Add	1 × 32768 × 8192
88	LayerNorm_15_1	LayerNorm	1 × 32768 × 8192
89	Attention_15	Grouped Query Attn	1 × 32768 × 8192
90	Add_15_attn	Add	1 × 32768 × 8192
91	LayerNorm_15_2	LayerNorm	1 × 32768 × 8192
92	FFN_15	SwiGLU	1 × 32768 × 8192
93	Add_15_ffn	Add	1 × 32768 × 8192
94	LayerNorm_16_1	LayerNorm	1 × 32768 × 8192
95	Attention_16	Grouped Query Attn	1 × 32768 × 8192
96	Add_16_attn	Add	1 × 32768 × 8192
97	LayerNorm_16_2	LayerNorm	1 × 32768 × 8192
98	FFN_16	SwiGLU	1 × 32768 × 8192
99	Add_16_ffn	Add	1 × 32768 × 8192
100	LayerNorm_17_1	LayerNorm	1 × 32768 × 8192
101	Attention_17	Grouped Query Attn	1 × 32768 × 8192
102	Add_17_attn	Add	1 × 32768 × 8192
103	LayerNorm_17_2	LayerNorm	1 × 32768 × 8192
104	FFN_17	SwiGLU	1 × 32768 × 8192
105	Add_17_ffn	Add	1 × 32768 × 8192
106	LayerNorm_18_1	LayerNorm	1 × 32768 × 8192
107	Attention_18	Grouped Query Attn	1 × 32768 × 8192
108	Add_18_attn	Add	1 × 32768 × 8192
109	LayerNorm_18_2	LayerNorm	1 × 32768 × 8192
110	FFN_18	SwiGLU	1 × 32768 × 8192
111	Add_18_ffn	Add	1 × 32768 × 8192
112	LayerNorm_19_1	LayerNorm	1 × 32768 × 8192
113	Attention_19	Grouped Query Attn	1 × 32768 × 8192
114	Add_19_attn	Add	1 × 32768 × 8192
115	LayerNorm_19_2	LayerNorm	1 × 32768 × 8192
116	FFN_19	SwiGLU	1 × 32768 × 8192
117	Add_19_ffn	Add	1 × 32768 × 8192
118	LayerNorm_20_1	LayerNorm	1 × 32768 × 8192
119	Attention_20	Grouped Query Attn	1 × 32768 × 8192
120	Add_20_attn	Add	1 × 32768 × 8192
121	LayerNorm_20_2	LayerNorm	1 × 32768 × 8192
122	FFN_20	SwiGLU	1 × 32768 × 8192
123	Add_20_ffn	Add	1 × 32768 × 8192
124	LayerNorm_21_1	LayerNorm	1 × 32768 × 8192
125	Attention_21	Grouped Query Attn	1 × 32768 × 8192
126	Add_21_attn	Add	1 × 32768 × 8192
127	LayerNorm_21_2	LayerNorm	1 × 32768 × 8192
128	FFN_21	SwiGLU	1 × 32768 × 8192
129	Add_21_ffn	Add	1 × 32768 × 8192
130	LayerNorm_22_1	LayerNorm	1 × 32768 × 8192
131	Attention_22	Grouped Query Attn	1 × 32768 × 8192
132	Add_22_attn	Add	1 × 32768 × 8192
133	LayerNorm_22_2	LayerNorm	1 × 32768 × 8192
134	FFN_22	SwiGLU	1 × 32768 × 8192
135	Add_22_ffn	Add	1 × 32768 × 8192
136	LayerNorm_23_1	LayerNorm	1 × 32768 × 8192
137	Attention_23	Grouped Query Attn	1 × 32768 × 8192
138	Add_23_attn	Add	1 × 32768 × 8192
139	LayerNorm_23_2	LayerNorm	1 × 32768 × 8192
140	FFN_23	SwiGLU	1 × 32768 × 8192
141	Add_23_ffn	Add	1 × 32768 × 8192
142	LayerNorm_24_1	LayerNorm	1 × 32768 × 8192
143	Attention_24	Grouped Query Attn	1 × 32768 × 8192
144	Add_24_attn	Add	1 × 32768 × 8192
145	LayerNorm_24_2	LayerNorm	1 × 32768 × 8192
146	FFN_24	SwiGLU	1 × 32768 × 8192
147	Add_24_ffn	Add	1 × 32768 × 8192
148	LayerNorm_25_1	LayerNorm	1 × 32768 × 8192
149	Attention_25	Grouped Query Attn	1 × 32768 × 8192
150	Add_25_attn	Add	1 × 32768 × 8192
151	LayerNorm_25_2	LayerNorm	1 × 32768 × 8192
152	FFN_25	SwiGLU	1 × 32768 × 8192
153	Add_25_ffn	Add	1 × 32768 × 8192
154	LayerNorm_26_1	LayerNorm	1 × 32768 × 8192
155	Attention_26	Grouped Query Attn	1 × 32768 × 8192
156	Add_26_attn	Add	1 × 32768 × 8192
157	LayerNorm_26_2	LayerNorm	1 × 32768 × 8192
158	FFN_26	SwiGLU	1 × 32768 × 8192
159	Add_26_ffn	Add	1 × 32768 × 8192
160	LayerNorm_27_1	LayerNorm	1 × 32768 × 8192
161	Attention_27	Grouped Query Attn	1 × 32768 × 8192
162	Add_27_attn	Add	1 × 32768 × 8192
163	LayerNorm_27_2	LayerNorm	1 × 32768 × 8192
164	FFN_27	SwiGLU	1 × 32768 × 8192
165	Add_27_ffn	Add	1 × 32768 × 8192
166	LayerNorm_28_1	LayerNorm	1 × 32768 × 8192
167	Attention_28	Grouped Query Attn	1 × 32768 × 8192
168	Add_28_attn	Add	1 × 32768 × 8192
169	LayerNorm_28_2	LayerNorm	1 × 32768 × 8192
170	FFN_28	SwiGLU	1 × 32768 × 8192
171	Add_28_ffn	Add	1 × 32768 × 8192
172	LayerNorm_29_1	LayerNorm	1 × 32768 × 8192
173	Attention_29	Grouped Query Attn	1 × 32768 × 8192
174	Add_29_attn	Add	1 × 32768 × 8192
175	LayerNorm_29_2	LayerNorm	1 × 32768 × 8192
176	FFN_29	SwiGLU	1 × 32768 × 8192
177	Add_29_ffn	Add	1 × 32768 × 8192
178	LayerNorm_30_1	LayerNorm	1 × 32768 × 8192
179	Attention_30	Grouped Query Attn	1 × 32768 × 8192
180	Add_30_attn	Add	1 × 32768 × 8192
181	LayerNorm_30_2	LayerNorm	1 × 32768 × 8192
182	FFN_30	SwiGLU	1 × 32768 × 8192
183	Add_30_ffn	Add	1 × 32768 × 8192
184	LayerNorm_31_1	LayerNorm	1 × 32768 × 8192
185	Attention_31	Grouped Query Attn	1 × 32768 × 8192
186	Add_31_attn	Add	1 × 32768 × 8192
187	LayerNorm_31_2	LayerNorm	1 × 32768 × 8192
188	FFN_31	SwiGLU	1 × 32768 × 8192
189	Add_31_ffn	Add	1 × 32768 × 8192
190	LayerNorm_32_1	LayerNorm	1 × 32768 × 8192
191	Attention_32	Grouped Query Attn	1 × 32768 × 8192
192	Add_32_attn	Add	1 × 32768 × 8192
193	LayerNorm_32_2	LayerNorm	1 × 32768 × 8192
194	FFN_32	SwiGLU	1 × 32768 × 8192
195	Add_32_ffn	Add	1 × 32768 × 8192
196	LayerNorm_33_1	LayerNorm	1 × 32768 × 8192
197	Attention_33	Grouped Query Attn	1 × 32768 × 8192
198	Add_33_attn	Add	1 × 32768 × 8192
199	LayerNorm_33_2	LayerNorm	1 × 32768 × 8192
200	FFN_33	SwiGLU	1 × 32768 × 8192
201	Add_33_ffn	Add	1 × 32768 × 8192
202	LayerNorm_34_1	LayerNorm	1 × 32768 × 8192
203	Attention_34	Grouped Query Attn	1 × 32768 × 8192
204	Add_34_attn	Add	1 × 32768 × 8192
205	LayerNorm_34_2	LayerNorm	1 × 32768 × 8192
206	FFN_34	SwiGLU	1 × 32768 × 8192
207	Add_34_ffn	Add	1 × 32768 × 8192
208	LayerNorm_35_1	LayerNorm	1 × 32768 × 8192
209	Attention_35	Grouped Query Attn	1 × 32768 × 8192
210	Add_35_attn	Add	1 × 32768 × 8192
211	LayerNorm_35_2	LayerNorm	1 × 32768 × 8192
212	FFN_35	SwiGLU	1 × 32768 × 8192
213	Add_35_ffn	Add	1 × 32768 × 8192
214	LayerNorm_36_1	LayerNorm	1 × 32768 × 8192
215	Attention_36	Grouped Query Attn	1 × 32768 × 8192
216	Add_36_attn	Add	1 × 32768 × 8192
217	LayerNorm_36_2	LayerNorm	1 × 32768 × 8192
218	FFN_36	SwiGLU	1 × 32768 × 8192
219	Add_36_ffn	Add	1 × 32768 × 8192
220	LayerNorm_37_1	LayerNorm	1 × 32768 × 8192
221	Attention_37	Grouped Query Attn	1 × 32768 × 8192
222	Add_37_attn	Add	1 × 32768 × 8192
223	LayerNorm_37_2	LayerNorm	1 × 32768 × 8192
224	FFN_37	SwiGLU	1 × 32768 × 8192
225	Add_37_ffn	Add	1 × 32768 × 8192
226	LayerNorm_38_1	LayerNorm	1 × 32768 × 8192
227	Attention_38	Grouped Query Attn	1 × 32768 × 8192
228	Add_38_attn	Add	1 × 32768 × 8192
229	LayerNorm_38_2	LayerNorm	1 × 32768 × 8192
230	FFN_38	SwiGLU	1 × 32768 × 8192
231	Add_38_ffn	Add	1 × 32768 × 8192
232	LayerNorm_39_1	LayerNorm	1 × 32768 × 8192
233	Attention_39	Grouped Query Attn	1 × 32768 × 8192
234	Add_39_attn	Add	1 × 32768 × 8192
235	LayerNorm_39_2	LayerNorm	1 × 32768 × 8192
236	FFN_39	SwiGLU	1 × 32768 × 8192
237	Add_39_ffn	Add	1 × 32768 × 8192
238	LayerNorm_40_1	LayerNorm	1 × 32768 × 8192
239	Attention_40	Grouped Query Attn	1 × 32768 × 8192
240	Add_40_attn	Add	1 × 32768 × 8192
241	LayerNorm_40_2	LayerNorm	1 × 32768 × 8192
242	FFN_40	SwiGLU	1 × 32768 × 8192
243	Add_40_ffn	Add	1 × 32768 × 8192
244	LayerNorm_41_1	LayerNorm	1 × 32768 × 8192
245	Attention_41	Grouped Query Attn	1 × 32768 × 8192
246	Add_41_attn	Add	1 × 32768 × 8192
247	LayerNorm_41_2	LayerNorm	1 × 32768 × 8192
248	FFN_41	SwiGLU	1 × 32768 × 8192
249	Add_41_ffn	Add	1 × 32768 × 8192
250	LayerNorm_42_1	LayerNorm	1 × 32768 × 8192
251	Attention_42	Grouped Query Attn	1 × 32768 × 8192
252	Add_42_attn	Add	1 × 32768 × 8192
253	LayerNorm_42_2	LayerNorm	1 × 32768 × 8192
254	FFN_42	SwiGLU	1 × 32768 × 8192
255	Add_42_ffn	Add	1 × 32768 × 8192
256	LayerNorm_43_1	LayerNorm	1 × 32768 × 8192
257	Attention_43	Grouped Query Attn	1 × 32768 × 8192
258	Add_43_attn	Add	1 × 32768 × 8192
259	LayerNorm_43_2	LayerNorm	1 × 32768 × 8192
260	FFN_43	SwiGLU	1 × 32768 × 8192
261	Add_43_ffn	Add	1 × 32768 × 8192
262	LayerNorm_44_1	LayerNorm	1 × 32768 × 8192
263	Attention_44	Grouped Query Attn	1 × 32768 × 8192
264	Add_44_attn	Add	1 × 32768 × 8192
265	LayerNorm_44_2	LayerNorm	1 × 32768 × 8192
266	FFN_44	SwiGLU	1 × 32768 × 8192
267	Add_44_ffn	Add	1 × 32768 × 8192
268	LayerNorm_45_1	LayerNorm	1 × 32768 × 8192
269	Attention_45	Grouped Query Attn	1 × 32768 × 8192
270	Add_45_attn	Add	1 × 32768 × 8192
271	LayerNorm_45_2	LayerNorm	1 × 32768 × 8192
272	FFN_45	SwiGLU	1 × 32768 × 8192
273	Add_45_ffn	Add	1 × 32768 × 8192
274	LayerNorm_46_1	LayerNorm	1 × 32768 × 8192
275	Attention_46	Grouped Query Attn	1 × 32768 × 8192
276	Add_46_attn	Add	1 × 32768 × 8192
277	LayerNorm_46_2	LayerNorm	1 × 32768 × 8192
278	FFN_46	SwiGLU	1 × 32768 × 8192
279	Add_46_ffn	Add	1 × 32768 × 8192
280	LayerNorm_47_1	LayerNorm	1 × 32768 × 8192
281	Attention_47	Grouped Query Attn	1 × 32768 × 8192
282	Add_47_attn	Add	1 × 32768 × 8192
283	LayerNorm_47_2	LayerNorm	1 × 32768 × 8192
284	FFN_47	SwiGLU	1 × 32768 × 8192
285	Add_47_ffn	Add	1 × 32768 × 8192
286	LayerNorm_48_1	LayerNorm	1 × 32768 × 8192
287	Attention_48	Grouped Query Attn	1 × 32768 × 8192
288	Add_48_attn	Add	1 × 32768 × 8192
289	LayerNorm_48_2	LayerNorm	1 × 32768 × 8192
290	FFN_48	SwiGLU	1 × 32768 × 8192
291	Add_48_ffn	Add	1 × 32768 × 8192
292	LayerNorm_49_1	LayerNorm	1 × 32768 × 8192
293	Attention_49	Grouped Query Attn	1 × 32768 × 8192
294	Add_49_attn	Add	1 × 32768 × 8192
295	LayerNorm_49_2	LayerNorm	1 × 32768 × 8192
296	FFN_49	SwiGLU	1 × 32768 × 8192
297	Add_49_ffn	Add	1 × 32768 × 8192
298	LayerNorm_50_1	LayerNorm	1 × 32768 × 8192
299	Attention_50	Grouped Query Attn	1 × 32768 × 8192
300	Add_50_attn	Add	1 × 32768 × 8192
301	LayerNorm_50_2	LayerNorm	1 × 32768 × 8192
302	FFN_50	SwiGLU	1 × 32768 × 8192
303	Add_50_ffn	Add	1 × 32768 × 8192
304	LayerNorm_51_1	LayerNorm	1 × 32768 × 8192
305	Attention_51	Grouped Query Attn	1 × 32768 × 8192
306	Add_51_attn	Add	1 × 32768 × 8192
307	LayerNorm_51_2	LayerNorm	1 × 32768 × 8192
308	FFN_51	SwiGLU	1 × 32768 × 8192
309	Add_51_ffn	Add	1 × 32768 × 8192
310	LayerNorm_52_1	LayerNorm	1 × 32768 × 8192
311	Attention_52	Grouped Query Attn	1 × 32768 × 8192
312	Add_52_attn	Add	1 × 32768 × 8192
313	LayerNorm_52_2	LayerNorm	1 × 32768 × 8192
314	FFN_52	SwiGLU	1 × 32768 × 8192
315	Add_52_ffn	Add	1 × 32768 × 8192
316	LayerNorm_53_1	LayerNorm	1 × 32768 × 8192
317	Attention_53	Grouped Query Attn	1 × 32768 × 8192
318	Add_53_attn	Add	1 × 32768 × 8192
319	LayerNorm_53_2	LayerNorm	1 × 32768 × 8192
320	FFN_53	SwiGLU	1 × 32768 × 8192
321	Add_53_ffn	Add	1 × 32768 × 8192
322	LayerNorm_54_1	LayerNorm	1 × 32768 × 8192
323	Attention_54	Grouped Query Attn	1 × 32768 × 8192
324	Add_54_attn	Add	1 × 32768 × 8192
325	LayerNorm_54_2	LayerNorm	1 × 32768 × 8192
326	FFN_54	SwiGLU	1 × 32768 × 8192
327	Add_54_ffn	Add	1 × 32768 × 8192
328	LayerNorm_55_1	LayerNorm	1 × 32768 × 8192
329	Attention_55	Grouped Query Attn	1 × 32768 × 8192
330	Add_55_attn	Add	1 × 32768 × 8192
331	LayerNorm_55_2	LayerNorm	1 × 32768 × 8192
332	FFN_55	SwiGLU	1 × 32768 × 8192
333	Add_55_ffn	Add	1 × 32768 × 8192
334	LayerNorm_56_1	LayerNorm	1 × 32768 × 8192
335	Attention_56	Grouped Query Attn	1 × 32768 × 8192
336	Add_56_attn	Add	1 × 32768 × 8192
337	LayerNorm_56_2	LayerNorm	1 × 32768 × 8192
338	FFN_56	SwiGLU	1 × 32768 × 8192
339	Add_56_ffn	Add	1 × 32768 × 8192
340	LayerNorm_57_1	LayerNorm	1 × 32768 × 8192
341	Attention_57	Grouped Query Attn	1 × 32768 × 8192
342	Add_57_attn	Add	1 × 32768 × 8192
343	LayerNorm_57_2	LayerNorm	1 × 32768 × 8192
344	FFN_57	SwiGLU	1 × 32768 × 8192
345	Add_57_ffn	Add	1 × 32768 × 8192
346	LayerNorm_58_1	LayerNorm	1 × 32768 × 8192
347	Attention_58	Grouped Query Attn	1 × 32768 × 8192
348	Add_58_attn	Add	1 × 32768 × 8192
349	LayerNorm_58_2	LayerNorm	1 × 32768 × 8192
350	FFN_58	SwiGLU	1 × 32768 × 8192
351	Add_58_ffn	Add	1 × 32768 × 8192
352	LayerNorm_59_1	LayerNorm	1 × 32768 × 8192
353	Attention_59	Grouped Query Attn	1 × 32768 × 8192
354	Add_59_attn	Add	1 × 32768 × 8192
355	LayerNorm_59_2	LayerNorm	1 × 32768 × 8192
356	FFN_59	SwiGLU	1 × 32768 × 8192
357	Add_59_ffn	Add	1 × 32768 × 8192
358	LayerNorm_60_1	LayerNorm	1 × 32768 × 8192
359	Attention_60	Grouped Query Attn	1 × 32768 × 8192
360	Add_60_attn	Add	1 × 32768 × 8192
361	LayerNorm_60_2	LayerNorm	1 × 32768 × 8192
362	FFN_60	SwiGLU	1 × 32768 × 8192
363	Add_60_ffn	Add	1 × 32768 × 8192
364	LayerNorm_61_1	LayerNorm	1 × 32768 × 8192
365	Attention_61	Grouped Query Attn	1 × 32768 × 8192
366	Add_61_attn	Add	1 × 32768 × 8192
367	LayerNorm_61_2	LayerNorm	1 × 32768 × 8192
368	FFN_61	SwiGLU	1 × 32768 × 8192
369	Add_61_ffn	Add	1 × 32768 × 8192
370	LayerNorm_62_1	LayerNorm	1 × 32768 × 8192
371	Attention_62	Grouped Query Attn	1 × 32768 × 8192
372	Add_62_attn	Add	1 × 32768 × 8192
373	LayerNorm_62_2	LayerNorm	1 × 32768 × 8192
374	FFN_62	SwiGLU	1 × 32768 × 8192
375	Add_62_ffn	Add	1 × 32768 × 8192
376	LayerNorm_63_1	LayerNorm	1 × 32768 × 8192
377	Attention_63	Grouped Query Attn	1 × 32768 × 8192
378	Add_63_attn	Add	1 × 32768 × 8192
379	LayerNorm_63_2	LayerNorm	1 × 32768 × 8192
380	FFN_63	SwiGLU	1 × 32768 × 8192
381	Add_63_ffn	Add	1 × 32768 × 8192
382	LayerNorm_64_1	LayerNorm	1 × 32768 × 8192
383	Attention_64	Grouped Query Attn	1 × 32768 × 8192
384	Add_64_attn	Add	1 × 32768 × 8192
385	LayerNorm_64_2	LayerNorm	1 × 32768 × 8192
386	FFN_64	SwiGLU	1 × 32768 × 8192
387	Add_64_ffn	Add	1 × 32768 × 8192
388	LayerNorm_65_1	LayerNorm	1 × 32768 × 8192
389	Attention_65	Grouped Query Attn	1 × 32768 × 8192
390	Add_65_attn	Add	1 × 32768 × 8192
391	LayerNorm_65_2	LayerNorm	1 × 32768 × 8192
392	FFN_65	SwiGLU	1 × 32768 × 8192
393	Add_65_ffn	Add	1 × 32768 × 8192
394	LayerNorm_66_1	LayerNorm	1 × 32768 × 8192
395	Attention_66	Grouped Query Attn	1 × 32768 × 8192
396	Add_66_attn	Add	1 × 32768 × 8192
397	LayerNorm_66_2	LayerNorm	1 × 32768 × 8192
398	FFN_66	SwiGLU	1 × 32768 × 8192
399	Add_66_ffn	Add	1 × 32768 × 8192
400	LayerNorm_67_1	LayerNorm	1 × 32768 × 8192
401	Attention_67	Grouped Query Attn	1 × 32768 × 8192
402	Add_67_attn	Add	1 × 32768 × 8192
403	LayerNorm_67_2	LayerNorm	1 × 32768 × 8192
404	FFN_67	SwiGLU	1 × 32768 × 8192
405	Add_67_ffn	Add	1 × 32768 × 8192
406	LayerNorm_68_1	LayerNorm	1 × 32768 × 8192
407	Attention_68	Grouped Query Attn	1 × 32768 × 8192
408	Add_68_attn	Add	1 × 32768 × 8192
409	LayerNorm_68_2	LayerNorm	1 × 32768 × 8192
410	FFN_68	SwiGLU	1 × 32768 × 8192
411	Add_68_ffn	Add	1 × 32768 × 8192
412	LayerNorm_69_1	LayerNorm	1 × 32768 × 8192
413	Attention_69	Grouped Query Attn	1 × 32768 × 8192
414	Add_69_attn	Add	1 × 32768 × 8192
415	LayerNorm_69_2	LayerNorm	1 × 32768 × 8192
416	FFN_69	SwiGLU	1 × 32768 × 8192
417	Add_69_ffn	Add	1 × 32768 × 8192
418	LayerNorm_70_1	LayerNorm	1 × 32768 × 8192
419	Attention_70	Grouped Query Attn	1 × 32768 × 8192
420	Add_70_attn	Add	1 × 32768 × 8192
421	LayerNorm_70_2	LayerNorm	1 × 32768 × 8192
422	FFN_70	SwiGLU	1 × 32768 × 8192
423	Add_70_ffn	Add	1 × 32768 × 8192
424	LayerNorm_71_1	LayerNorm	1 × 32768 × 8192
425	Attention_71	Grouped Query Attn	1 × 32768 × 8192
426	Add_71_attn	Add	1 × 32768 × 8192
427	LayerNorm_71_2	LayerNorm	1 × 32768 × 8192
428	FFN_71	SwiGLU	1 × 32768 × 8192
429	Add_71_ffn	Add	1 × 32768 × 8192
430	LayerNorm_72_1	LayerNorm	1 × 32768 × 8192
431	Attention_72	Grouped Query Attn	1 × 32768 × 8192
432	Add_72_attn	Add	1 × 32768 × 8192
433	LayerNorm_72_2	LayerNorm	1 × 32768 × 8192
434	FFN_72	SwiGLU	1 × 32768 × 8192
435	Add_72_ffn	Add	1 × 32768 × 8192
436	LayerNorm_73_1	LayerNorm	1 × 32768 × 8192
437	Attention_73	Grouped Query Attn	1 × 32768 × 8192
438	Add_73_attn	Add	1 × 32768 × 8192
439	LayerNorm_73_2	LayerNorm	1 × 32768 × 8192
440	FFN_73	SwiGLU	1 × 32768 × 8192
441	Add_73_ffn	Add	1 × 32768 × 8192
442	LayerNorm_74_1	LayerNorm	1 × 32768 × 8192
443	Attention_74	Grouped Query Attn	1 × 32768 × 8192
444	Add_74_attn	Add	1 × 32768 × 8192
445	LayerNorm_74_2	LayerNorm	1 × 32768 × 8192
446	FFN_74	SwiGLU	1 × 32768 × 8192
447	Add_74_ffn	Add	1 × 32768 × 8192
448	LayerNorm_75_1	LayerNorm	1 × 32768 × 8192
449	Attention_75	Grouped Query Attn	1 × 32768 × 8192
450	Add_75_attn	Add	1 × 32768 × 8192
451	LayerNorm_75_2	LayerNorm	1 × 32768 × 8192
452	FFN_75	SwiGLU	1 × 32768 × 8192
453	Add_75_ffn	Add	1 × 32768 × 8192
454	LayerNorm_76_1	LayerNorm	1 × 32768 × 8192
455	Attention_76	Grouped Query Attn	1 × 32768 × 8192
456	Add_76_attn	Add	1 × 32768 × 8192
457	LayerNorm_76_2	LayerNorm	1 × 32768 × 8192
458	FFN_76	SwiGLU	1 × 32768 × 8192
459	Add_76_ffn	Add	1 × 32768 × 8192
460	LayerNorm_77_1	LayerNorm	1 × 32768 × 8192
461	Attention_77	Grouped Query Attn	1 × 32768 × 8192
462	Add_77_attn	Add	1 × 32768 × 8192
463	LayerNorm_77_2	LayerNorm	1 × 32768 × 8192
464	FFN_77	SwiGLU	1 × 32768 × 8192
465	Add_77_ffn	Add	1 × 32768 × 8192
466	LayerNorm_78_1	LayerNorm	1 × 32768 × 8192
467	Attention_78	Grouped Query Attn	1 × 32768 × 8192
468	Add_78_attn	Add	1 × 32768 × 8192
469	LayerNorm_78_2	LayerNorm	1 × 32768 × 8192
470	FFN_78	SwiGLU	1 × 32768 × 8192
471	Add_78_ffn	Add	1 × 32768 × 8192
472	LayerNorm_79_1	LayerNorm	1 × 32768 × 8192
473	Attention_79	Grouped Query Attn	1 × 32768 × 8192
474	Add_79_attn	Add	1 × 32768 × 8192
475	LayerNorm_79_2	LayerNorm	1 × 32768 × 8192
476	FFN_79	SwiGLU	1 × 32768 × 8192
477	Add_79_ffn	Add	1 × 32768 × 8192
478	LayerNorm_80_1	LayerNorm	1 × 32768 × 8192
479	Attention_80	Grouped Query Attn	1 × 32768 × 8192
480	Add_80_attn	Add	1 × 32768 × 8192
481	LayerNorm_80_2	LayerNorm	1 × 32768 × 8192
482	FFN_80	SwiGLU	1 × 32768 × 8192
483	Add_80_ffn	Add	1 × 32768 × 8192
484	Final_LayerNorm	LayerNorm	1 × 32768 × 8192
485	LM_Head	Linear	1 × 32768 × 152064
486	Output	Output	1 × 32768 × 152064

What the verifier says

infoLLaMA uses intermediateSize ≈ ⌊(8/3 × D) / 256⌋ × 256. Current: 49152 (6.00× embedDim). Expected: ~21760. Fix: Set intermediateSize to 21760 for embedDim=8192.
swiglu-dim-convention

warnLinear "LM_Head" is 8192 × 152064 = 1246M parameters (~4.6 GB float32). A single dense layer this large usually means a feature map was flattened without pooling first; embedding / vocab-projection heads are the expected exception. Fix: Add a Global Average Pool or more downsampling before the Linear, or factorize it (low-rank / bottleneck projection).
huge-linear-params

info80 attention layers at embedDim 8192 cache full per-head K/V: about 2560 KB per token at fp16, which dominates memory at long context. Grouped-query attention (e.g. 8:1) would cut this ~8×; multi-head latent attention (MLA) shrinks it ~10× or more. This is the move production LLMs make; it does not change the parameter count. Fix: Switch attention to groupedQueryAttention (set numKVHeads below numHeads, e.g. numHeads/4) or mla (a low-rank cached latent).
full-mha-serving-cost

infoAt 80 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

warnAcross 80 attention layers this design caches 2560 KB per token, so a single 8,192-token sequence needs ~21.5 GB of KV cache before weights or activations. That exceeds the 4 GB budget this rule assumes for serving headroom. Fix: Cut KV width: raise the GQA ratio (fewer numKVHeads), switch to MLA, reduce depth or embedDim, or accept a shorter serving context.
kv-cache-context-budget

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace Qwen/Qwen-72B --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan