dolphin-2.9.1-yi-1.5-34b

Reconstructed from its own config.json with no weights read. 4.8M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

34.39B

34,388,981,248 parameters

In the published checkpoint

34.39B

34,388,917,248 scalars · safetensors.total, read 2026-09-06

Delta

+0.00%

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.

Layers

364

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$11483.95

Card	Memory
T4 (16GB)	weights + activations	does not fit
A100 (40GB)	weights + activations	does not fit
H100 (80GB)	weights + activations	does not fit

Structure

366 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Output shape
1	Input	Input	1 × 8192
2	Embedding	Embedding	1 × 8192 × 7168
3	RoPE	RoPE	1 × 8192 × 7168
4	RMSNorm_1_1	RMSNorm	1 × 8192 × 7168
5	Attention_1	Grouped Query Attn	1 × 8192 × 7168
6	Add_1_attn	Add	1 × 8192 × 7168
7	RMSNorm_1_2	RMSNorm	1 × 8192 × 7168
8	FFN_1	SwiGLU	1 × 8192 × 7168
9	Add_1_ffn	Add	1 × 8192 × 7168
10	RMSNorm_2_1	RMSNorm	1 × 8192 × 7168
11	Attention_2	Grouped Query Attn	1 × 8192 × 7168
12	Add_2_attn	Add	1 × 8192 × 7168
13	RMSNorm_2_2	RMSNorm	1 × 8192 × 7168
14	FFN_2	SwiGLU	1 × 8192 × 7168
15	Add_2_ffn	Add	1 × 8192 × 7168
16	RMSNorm_3_1	RMSNorm	1 × 8192 × 7168
17	Attention_3	Grouped Query Attn	1 × 8192 × 7168
18	Add_3_attn	Add	1 × 8192 × 7168
19	RMSNorm_3_2	RMSNorm	1 × 8192 × 7168
20	FFN_3	SwiGLU	1 × 8192 × 7168
21	Add_3_ffn	Add	1 × 8192 × 7168
22	RMSNorm_4_1	RMSNorm	1 × 8192 × 7168
23	Attention_4	Grouped Query Attn	1 × 8192 × 7168
24	Add_4_attn	Add	1 × 8192 × 7168
25	RMSNorm_4_2	RMSNorm	1 × 8192 × 7168
26	FFN_4	SwiGLU	1 × 8192 × 7168
27	Add_4_ffn	Add	1 × 8192 × 7168
28	RMSNorm_5_1	RMSNorm	1 × 8192 × 7168
29	Attention_5	Grouped Query Attn	1 × 8192 × 7168
30	Add_5_attn	Add	1 × 8192 × 7168
31	RMSNorm_5_2	RMSNorm	1 × 8192 × 7168
32	FFN_5	SwiGLU	1 × 8192 × 7168
33	Add_5_ffn	Add	1 × 8192 × 7168
34	RMSNorm_6_1	RMSNorm	1 × 8192 × 7168
35	Attention_6	Grouped Query Attn	1 × 8192 × 7168
36	Add_6_attn	Add	1 × 8192 × 7168
37	RMSNorm_6_2	RMSNorm	1 × 8192 × 7168
38	FFN_6	SwiGLU	1 × 8192 × 7168
39	Add_6_ffn	Add	1 × 8192 × 7168
40	RMSNorm_7_1	RMSNorm	1 × 8192 × 7168
41	Attention_7	Grouped Query Attn	1 × 8192 × 7168
42	Add_7_attn	Add	1 × 8192 × 7168
43	RMSNorm_7_2	RMSNorm	1 × 8192 × 7168
44	FFN_7	SwiGLU	1 × 8192 × 7168
45	Add_7_ffn	Add	1 × 8192 × 7168
46	RMSNorm_8_1	RMSNorm	1 × 8192 × 7168
47	Attention_8	Grouped Query Attn	1 × 8192 × 7168
48	Add_8_attn	Add	1 × 8192 × 7168
49	RMSNorm_8_2	RMSNorm	1 × 8192 × 7168
50	FFN_8	SwiGLU	1 × 8192 × 7168
51	Add_8_ffn	Add	1 × 8192 × 7168
52	RMSNorm_9_1	RMSNorm	1 × 8192 × 7168
53	Attention_9	Grouped Query Attn	1 × 8192 × 7168
54	Add_9_attn	Add	1 × 8192 × 7168
55	RMSNorm_9_2	RMSNorm	1 × 8192 × 7168
56	FFN_9	SwiGLU	1 × 8192 × 7168
57	Add_9_ffn	Add	1 × 8192 × 7168
58	RMSNorm_10_1	RMSNorm	1 × 8192 × 7168
59	Attention_10	Grouped Query Attn	1 × 8192 × 7168
60	Add_10_attn	Add	1 × 8192 × 7168
61	RMSNorm_10_2	RMSNorm	1 × 8192 × 7168
62	FFN_10	SwiGLU	1 × 8192 × 7168
63	Add_10_ffn	Add	1 × 8192 × 7168
64	RMSNorm_11_1	RMSNorm	1 × 8192 × 7168
65	Attention_11	Grouped Query Attn	1 × 8192 × 7168
66	Add_11_attn	Add	1 × 8192 × 7168
67	RMSNorm_11_2	RMSNorm	1 × 8192 × 7168
68	FFN_11	SwiGLU	1 × 8192 × 7168
69	Add_11_ffn	Add	1 × 8192 × 7168
70	RMSNorm_12_1	RMSNorm	1 × 8192 × 7168
71	Attention_12	Grouped Query Attn	1 × 8192 × 7168
72	Add_12_attn	Add	1 × 8192 × 7168
73	RMSNorm_12_2	RMSNorm	1 × 8192 × 7168
74	FFN_12	SwiGLU	1 × 8192 × 7168
75	Add_12_ffn	Add	1 × 8192 × 7168
76	RMSNorm_13_1	RMSNorm	1 × 8192 × 7168
77	Attention_13	Grouped Query Attn	1 × 8192 × 7168
78	Add_13_attn	Add	1 × 8192 × 7168
79	RMSNorm_13_2	RMSNorm	1 × 8192 × 7168
80	FFN_13	SwiGLU	1 × 8192 × 7168
81	Add_13_ffn	Add	1 × 8192 × 7168
82	RMSNorm_14_1	RMSNorm	1 × 8192 × 7168
83	Attention_14	Grouped Query Attn	1 × 8192 × 7168
84	Add_14_attn	Add	1 × 8192 × 7168
85	RMSNorm_14_2	RMSNorm	1 × 8192 × 7168
86	FFN_14	SwiGLU	1 × 8192 × 7168
87	Add_14_ffn	Add	1 × 8192 × 7168
88	RMSNorm_15_1	RMSNorm	1 × 8192 × 7168
89	Attention_15	Grouped Query Attn	1 × 8192 × 7168
90	Add_15_attn	Add	1 × 8192 × 7168
91	RMSNorm_15_2	RMSNorm	1 × 8192 × 7168
92	FFN_15	SwiGLU	1 × 8192 × 7168
93	Add_15_ffn	Add	1 × 8192 × 7168
94	RMSNorm_16_1	RMSNorm	1 × 8192 × 7168
95	Attention_16	Grouped Query Attn	1 × 8192 × 7168
96	Add_16_attn	Add	1 × 8192 × 7168
97	RMSNorm_16_2	RMSNorm	1 × 8192 × 7168
98	FFN_16	SwiGLU	1 × 8192 × 7168
99	Add_16_ffn	Add	1 × 8192 × 7168
100	RMSNorm_17_1	RMSNorm	1 × 8192 × 7168
101	Attention_17	Grouped Query Attn	1 × 8192 × 7168
102	Add_17_attn	Add	1 × 8192 × 7168
103	RMSNorm_17_2	RMSNorm	1 × 8192 × 7168
104	FFN_17	SwiGLU	1 × 8192 × 7168
105	Add_17_ffn	Add	1 × 8192 × 7168
106	RMSNorm_18_1	RMSNorm	1 × 8192 × 7168
107	Attention_18	Grouped Query Attn	1 × 8192 × 7168
108	Add_18_attn	Add	1 × 8192 × 7168
109	RMSNorm_18_2	RMSNorm	1 × 8192 × 7168
110	FFN_18	SwiGLU	1 × 8192 × 7168
111	Add_18_ffn	Add	1 × 8192 × 7168
112	RMSNorm_19_1	RMSNorm	1 × 8192 × 7168
113	Attention_19	Grouped Query Attn	1 × 8192 × 7168
114	Add_19_attn	Add	1 × 8192 × 7168
115	RMSNorm_19_2	RMSNorm	1 × 8192 × 7168
116	FFN_19	SwiGLU	1 × 8192 × 7168
117	Add_19_ffn	Add	1 × 8192 × 7168
118	RMSNorm_20_1	RMSNorm	1 × 8192 × 7168
119	Attention_20	Grouped Query Attn	1 × 8192 × 7168
120	Add_20_attn	Add	1 × 8192 × 7168
121	RMSNorm_20_2	RMSNorm	1 × 8192 × 7168
122	FFN_20	SwiGLU	1 × 8192 × 7168
123	Add_20_ffn	Add	1 × 8192 × 7168
124	RMSNorm_21_1	RMSNorm	1 × 8192 × 7168
125	Attention_21	Grouped Query Attn	1 × 8192 × 7168
126	Add_21_attn	Add	1 × 8192 × 7168
127	RMSNorm_21_2	RMSNorm	1 × 8192 × 7168
128	FFN_21	SwiGLU	1 × 8192 × 7168
129	Add_21_ffn	Add	1 × 8192 × 7168
130	RMSNorm_22_1	RMSNorm	1 × 8192 × 7168
131	Attention_22	Grouped Query Attn	1 × 8192 × 7168
132	Add_22_attn	Add	1 × 8192 × 7168
133	RMSNorm_22_2	RMSNorm	1 × 8192 × 7168
134	FFN_22	SwiGLU	1 × 8192 × 7168
135	Add_22_ffn	Add	1 × 8192 × 7168
136	RMSNorm_23_1	RMSNorm	1 × 8192 × 7168
137	Attention_23	Grouped Query Attn	1 × 8192 × 7168
138	Add_23_attn	Add	1 × 8192 × 7168
139	RMSNorm_23_2	RMSNorm	1 × 8192 × 7168
140	FFN_23	SwiGLU	1 × 8192 × 7168
141	Add_23_ffn	Add	1 × 8192 × 7168
142	RMSNorm_24_1	RMSNorm	1 × 8192 × 7168
143	Attention_24	Grouped Query Attn	1 × 8192 × 7168
144	Add_24_attn	Add	1 × 8192 × 7168
145	RMSNorm_24_2	RMSNorm	1 × 8192 × 7168
146	FFN_24	SwiGLU	1 × 8192 × 7168
147	Add_24_ffn	Add	1 × 8192 × 7168
148	RMSNorm_25_1	RMSNorm	1 × 8192 × 7168
149	Attention_25	Grouped Query Attn	1 × 8192 × 7168
150	Add_25_attn	Add	1 × 8192 × 7168
151	RMSNorm_25_2	RMSNorm	1 × 8192 × 7168
152	FFN_25	SwiGLU	1 × 8192 × 7168
153	Add_25_ffn	Add	1 × 8192 × 7168
154	RMSNorm_26_1	RMSNorm	1 × 8192 × 7168
155	Attention_26	Grouped Query Attn	1 × 8192 × 7168
156	Add_26_attn	Add	1 × 8192 × 7168
157	RMSNorm_26_2	RMSNorm	1 × 8192 × 7168
158	FFN_26	SwiGLU	1 × 8192 × 7168
159	Add_26_ffn	Add	1 × 8192 × 7168
160	RMSNorm_27_1	RMSNorm	1 × 8192 × 7168
161	Attention_27	Grouped Query Attn	1 × 8192 × 7168
162	Add_27_attn	Add	1 × 8192 × 7168
163	RMSNorm_27_2	RMSNorm	1 × 8192 × 7168
164	FFN_27	SwiGLU	1 × 8192 × 7168
165	Add_27_ffn	Add	1 × 8192 × 7168
166	RMSNorm_28_1	RMSNorm	1 × 8192 × 7168
167	Attention_28	Grouped Query Attn	1 × 8192 × 7168
168	Add_28_attn	Add	1 × 8192 × 7168
169	RMSNorm_28_2	RMSNorm	1 × 8192 × 7168
170	FFN_28	SwiGLU	1 × 8192 × 7168
171	Add_28_ffn	Add	1 × 8192 × 7168
172	RMSNorm_29_1	RMSNorm	1 × 8192 × 7168
173	Attention_29	Grouped Query Attn	1 × 8192 × 7168
174	Add_29_attn	Add	1 × 8192 × 7168
175	RMSNorm_29_2	RMSNorm	1 × 8192 × 7168
176	FFN_29	SwiGLU	1 × 8192 × 7168
177	Add_29_ffn	Add	1 × 8192 × 7168
178	RMSNorm_30_1	RMSNorm	1 × 8192 × 7168
179	Attention_30	Grouped Query Attn	1 × 8192 × 7168
180	Add_30_attn	Add	1 × 8192 × 7168
181	RMSNorm_30_2	RMSNorm	1 × 8192 × 7168
182	FFN_30	SwiGLU	1 × 8192 × 7168
183	Add_30_ffn	Add	1 × 8192 × 7168
184	RMSNorm_31_1	RMSNorm	1 × 8192 × 7168
185	Attention_31	Grouped Query Attn	1 × 8192 × 7168
186	Add_31_attn	Add	1 × 8192 × 7168
187	RMSNorm_31_2	RMSNorm	1 × 8192 × 7168
188	FFN_31	SwiGLU	1 × 8192 × 7168
189	Add_31_ffn	Add	1 × 8192 × 7168
190	RMSNorm_32_1	RMSNorm	1 × 8192 × 7168
191	Attention_32	Grouped Query Attn	1 × 8192 × 7168
192	Add_32_attn	Add	1 × 8192 × 7168
193	RMSNorm_32_2	RMSNorm	1 × 8192 × 7168
194	FFN_32	SwiGLU	1 × 8192 × 7168
195	Add_32_ffn	Add	1 × 8192 × 7168
196	RMSNorm_33_1	RMSNorm	1 × 8192 × 7168
197	Attention_33	Grouped Query Attn	1 × 8192 × 7168
198	Add_33_attn	Add	1 × 8192 × 7168
199	RMSNorm_33_2	RMSNorm	1 × 8192 × 7168
200	FFN_33	SwiGLU	1 × 8192 × 7168
201	Add_33_ffn	Add	1 × 8192 × 7168
202	RMSNorm_34_1	RMSNorm	1 × 8192 × 7168
203	Attention_34	Grouped Query Attn	1 × 8192 × 7168
204	Add_34_attn	Add	1 × 8192 × 7168
205	RMSNorm_34_2	RMSNorm	1 × 8192 × 7168
206	FFN_34	SwiGLU	1 × 8192 × 7168
207	Add_34_ffn	Add	1 × 8192 × 7168
208	RMSNorm_35_1	RMSNorm	1 × 8192 × 7168
209	Attention_35	Grouped Query Attn	1 × 8192 × 7168
210	Add_35_attn	Add	1 × 8192 × 7168
211	RMSNorm_35_2	RMSNorm	1 × 8192 × 7168
212	FFN_35	SwiGLU	1 × 8192 × 7168
213	Add_35_ffn	Add	1 × 8192 × 7168
214	RMSNorm_36_1	RMSNorm	1 × 8192 × 7168
215	Attention_36	Grouped Query Attn	1 × 8192 × 7168
216	Add_36_attn	Add	1 × 8192 × 7168
217	RMSNorm_36_2	RMSNorm	1 × 8192 × 7168
218	FFN_36	SwiGLU	1 × 8192 × 7168
219	Add_36_ffn	Add	1 × 8192 × 7168
220	RMSNorm_37_1	RMSNorm	1 × 8192 × 7168
221	Attention_37	Grouped Query Attn	1 × 8192 × 7168
222	Add_37_attn	Add	1 × 8192 × 7168
223	RMSNorm_37_2	RMSNorm	1 × 8192 × 7168
224	FFN_37	SwiGLU	1 × 8192 × 7168
225	Add_37_ffn	Add	1 × 8192 × 7168
226	RMSNorm_38_1	RMSNorm	1 × 8192 × 7168
227	Attention_38	Grouped Query Attn	1 × 8192 × 7168
228	Add_38_attn	Add	1 × 8192 × 7168
229	RMSNorm_38_2	RMSNorm	1 × 8192 × 7168
230	FFN_38	SwiGLU	1 × 8192 × 7168
231	Add_38_ffn	Add	1 × 8192 × 7168
232	RMSNorm_39_1	RMSNorm	1 × 8192 × 7168
233	Attention_39	Grouped Query Attn	1 × 8192 × 7168
234	Add_39_attn	Add	1 × 8192 × 7168
235	RMSNorm_39_2	RMSNorm	1 × 8192 × 7168
236	FFN_39	SwiGLU	1 × 8192 × 7168
237	Add_39_ffn	Add	1 × 8192 × 7168
238	RMSNorm_40_1	RMSNorm	1 × 8192 × 7168
239	Attention_40	Grouped Query Attn	1 × 8192 × 7168
240	Add_40_attn	Add	1 × 8192 × 7168
241	RMSNorm_40_2	RMSNorm	1 × 8192 × 7168
242	FFN_40	SwiGLU	1 × 8192 × 7168
243	Add_40_ffn	Add	1 × 8192 × 7168
244	RMSNorm_41_1	RMSNorm	1 × 8192 × 7168
245	Attention_41	Grouped Query Attn	1 × 8192 × 7168
246	Add_41_attn	Add	1 × 8192 × 7168
247	RMSNorm_41_2	RMSNorm	1 × 8192 × 7168
248	FFN_41	SwiGLU	1 × 8192 × 7168
249	Add_41_ffn	Add	1 × 8192 × 7168
250	RMSNorm_42_1	RMSNorm	1 × 8192 × 7168
251	Attention_42	Grouped Query Attn	1 × 8192 × 7168
252	Add_42_attn	Add	1 × 8192 × 7168
253	RMSNorm_42_2	RMSNorm	1 × 8192 × 7168
254	FFN_42	SwiGLU	1 × 8192 × 7168
255	Add_42_ffn	Add	1 × 8192 × 7168
256	RMSNorm_43_1	RMSNorm	1 × 8192 × 7168
257	Attention_43	Grouped Query Attn	1 × 8192 × 7168
258	Add_43_attn	Add	1 × 8192 × 7168
259	RMSNorm_43_2	RMSNorm	1 × 8192 × 7168
260	FFN_43	SwiGLU	1 × 8192 × 7168
261	Add_43_ffn	Add	1 × 8192 × 7168
262	RMSNorm_44_1	RMSNorm	1 × 8192 × 7168
263	Attention_44	Grouped Query Attn	1 × 8192 × 7168
264	Add_44_attn	Add	1 × 8192 × 7168
265	RMSNorm_44_2	RMSNorm	1 × 8192 × 7168
266	FFN_44	SwiGLU	1 × 8192 × 7168
267	Add_44_ffn	Add	1 × 8192 × 7168
268	RMSNorm_45_1	RMSNorm	1 × 8192 × 7168
269	Attention_45	Grouped Query Attn	1 × 8192 × 7168
270	Add_45_attn	Add	1 × 8192 × 7168
271	RMSNorm_45_2	RMSNorm	1 × 8192 × 7168
272	FFN_45	SwiGLU	1 × 8192 × 7168
273	Add_45_ffn	Add	1 × 8192 × 7168
274	RMSNorm_46_1	RMSNorm	1 × 8192 × 7168
275	Attention_46	Grouped Query Attn	1 × 8192 × 7168
276	Add_46_attn	Add	1 × 8192 × 7168
277	RMSNorm_46_2	RMSNorm	1 × 8192 × 7168
278	FFN_46	SwiGLU	1 × 8192 × 7168
279	Add_46_ffn	Add	1 × 8192 × 7168
280	RMSNorm_47_1	RMSNorm	1 × 8192 × 7168
281	Attention_47	Grouped Query Attn	1 × 8192 × 7168
282	Add_47_attn	Add	1 × 8192 × 7168
283	RMSNorm_47_2	RMSNorm	1 × 8192 × 7168
284	FFN_47	SwiGLU	1 × 8192 × 7168
285	Add_47_ffn	Add	1 × 8192 × 7168
286	RMSNorm_48_1	RMSNorm	1 × 8192 × 7168
287	Attention_48	Grouped Query Attn	1 × 8192 × 7168
288	Add_48_attn	Add	1 × 8192 × 7168
289	RMSNorm_48_2	RMSNorm	1 × 8192 × 7168
290	FFN_48	SwiGLU	1 × 8192 × 7168
291	Add_48_ffn	Add	1 × 8192 × 7168
292	RMSNorm_49_1	RMSNorm	1 × 8192 × 7168
293	Attention_49	Grouped Query Attn	1 × 8192 × 7168
294	Add_49_attn	Add	1 × 8192 × 7168
295	RMSNorm_49_2	RMSNorm	1 × 8192 × 7168
296	FFN_49	SwiGLU	1 × 8192 × 7168
297	Add_49_ffn	Add	1 × 8192 × 7168
298	RMSNorm_50_1	RMSNorm	1 × 8192 × 7168
299	Attention_50	Grouped Query Attn	1 × 8192 × 7168
300	Add_50_attn	Add	1 × 8192 × 7168
301	RMSNorm_50_2	RMSNorm	1 × 8192 × 7168
302	FFN_50	SwiGLU	1 × 8192 × 7168
303	Add_50_ffn	Add	1 × 8192 × 7168
304	RMSNorm_51_1	RMSNorm	1 × 8192 × 7168
305	Attention_51	Grouped Query Attn	1 × 8192 × 7168
306	Add_51_attn	Add	1 × 8192 × 7168
307	RMSNorm_51_2	RMSNorm	1 × 8192 × 7168
308	FFN_51	SwiGLU	1 × 8192 × 7168
309	Add_51_ffn	Add	1 × 8192 × 7168
310	RMSNorm_52_1	RMSNorm	1 × 8192 × 7168
311	Attention_52	Grouped Query Attn	1 × 8192 × 7168
312	Add_52_attn	Add	1 × 8192 × 7168
313	RMSNorm_52_2	RMSNorm	1 × 8192 × 7168
314	FFN_52	SwiGLU	1 × 8192 × 7168
315	Add_52_ffn	Add	1 × 8192 × 7168
316	RMSNorm_53_1	RMSNorm	1 × 8192 × 7168
317	Attention_53	Grouped Query Attn	1 × 8192 × 7168
318	Add_53_attn	Add	1 × 8192 × 7168
319	RMSNorm_53_2	RMSNorm	1 × 8192 × 7168
320	FFN_53	SwiGLU	1 × 8192 × 7168
321	Add_53_ffn	Add	1 × 8192 × 7168
322	RMSNorm_54_1	RMSNorm	1 × 8192 × 7168
323	Attention_54	Grouped Query Attn	1 × 8192 × 7168
324	Add_54_attn	Add	1 × 8192 × 7168
325	RMSNorm_54_2	RMSNorm	1 × 8192 × 7168
326	FFN_54	SwiGLU	1 × 8192 × 7168
327	Add_54_ffn	Add	1 × 8192 × 7168
328	RMSNorm_55_1	RMSNorm	1 × 8192 × 7168
329	Attention_55	Grouped Query Attn	1 × 8192 × 7168
330	Add_55_attn	Add	1 × 8192 × 7168
331	RMSNorm_55_2	RMSNorm	1 × 8192 × 7168
332	FFN_55	SwiGLU	1 × 8192 × 7168
333	Add_55_ffn	Add	1 × 8192 × 7168
334	RMSNorm_56_1	RMSNorm	1 × 8192 × 7168
335	Attention_56	Grouped Query Attn	1 × 8192 × 7168
336	Add_56_attn	Add	1 × 8192 × 7168
337	RMSNorm_56_2	RMSNorm	1 × 8192 × 7168
338	FFN_56	SwiGLU	1 × 8192 × 7168
339	Add_56_ffn	Add	1 × 8192 × 7168
340	RMSNorm_57_1	RMSNorm	1 × 8192 × 7168
341	Attention_57	Grouped Query Attn	1 × 8192 × 7168
342	Add_57_attn	Add	1 × 8192 × 7168
343	RMSNorm_57_2	RMSNorm	1 × 8192 × 7168
344	FFN_57	SwiGLU	1 × 8192 × 7168
345	Add_57_ffn	Add	1 × 8192 × 7168
346	RMSNorm_58_1	RMSNorm	1 × 8192 × 7168
347	Attention_58	Grouped Query Attn	1 × 8192 × 7168
348	Add_58_attn	Add	1 × 8192 × 7168
349	RMSNorm_58_2	RMSNorm	1 × 8192 × 7168
350	FFN_58	SwiGLU	1 × 8192 × 7168
351	Add_58_ffn	Add	1 × 8192 × 7168
352	RMSNorm_59_1	RMSNorm	1 × 8192 × 7168
353	Attention_59	Grouped Query Attn	1 × 8192 × 7168
354	Add_59_attn	Add	1 × 8192 × 7168
355	RMSNorm_59_2	RMSNorm	1 × 8192 × 7168
356	FFN_59	SwiGLU	1 × 8192 × 7168
357	Add_59_ffn	Add	1 × 8192 × 7168
358	RMSNorm_60_1	RMSNorm	1 × 8192 × 7168
359	Attention_60	Grouped Query Attn	1 × 8192 × 7168
360	Add_60_attn	Add	1 × 8192 × 7168
361	RMSNorm_60_2	RMSNorm	1 × 8192 × 7168
362	FFN_60	SwiGLU	1 × 8192 × 7168
363	Add_60_ffn	Add	1 × 8192 × 7168
364	Final_RMSNorm	RMSNorm	1 × 8192 × 7168
365	LM_Head	Linear	1 × 8192 × 64000
366	Output	Output	1 × 8192 × 64000

What the verifier says

infoAt 60 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace dphn/dolphin-2.9.1-yi-1.5-34b --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan