Models / wav2vec2-bert

w2v-bert-2.0

Reconstructed from its own config.json with no weights read. 2.3M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

739M

739,278,848 parameters

In the published checkpoint

580M

580,493,120 scalars · safetensors.total, read 2026-09-06

Delta

+27.4%

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.

Layers

268

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$12.73

Card	Memory
T4 (16GB)	weights + activations	fits
A100 (40GB)	weights + activations	fits
H100 (80GB)	weights + activations	fits

Structure

270 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Output shape
1	Input	Input	1 × 512
2	Encoder_Embedding	Embedding	1 × 512 × 1024
3	Encoder_Position_Embedding	Learned Pos Embed	1 × 512 × 1024
4	Enc_Attn_1	Multi-Head Attention	1 × 512 × 1024
5	Enc_Add_1	Add	1 × 512 × 1024
6	Enc_Norm_1	LayerNorm	1 × 512 × 1024
7	Enc_FFN_1	Feed Forward	1 × 512 × 1024
8	Enc_Attn_2	Multi-Head Attention	1 × 512 × 1024
9	Enc_Add_2	Add	1 × 512 × 1024
10	Enc_Norm_2	LayerNorm	1 × 512 × 1024
11	Enc_FFN_2	Feed Forward	1 × 512 × 1024
12	Enc_Attn_3	Multi-Head Attention	1 × 512 × 1024
13	Enc_Add_3	Add	1 × 512 × 1024
14	Enc_Norm_3	LayerNorm	1 × 512 × 1024
15	Enc_FFN_3	Feed Forward	1 × 512 × 1024
16	Enc_Attn_4	Multi-Head Attention	1 × 512 × 1024
17	Enc_Add_4	Add	1 × 512 × 1024
18	Enc_Norm_4	LayerNorm	1 × 512 × 1024
19	Enc_FFN_4	Feed Forward	1 × 512 × 1024
20	Enc_Attn_5	Multi-Head Attention	1 × 512 × 1024
21	Enc_Add_5	Add	1 × 512 × 1024
22	Enc_Norm_5	LayerNorm	1 × 512 × 1024
23	Enc_FFN_5	Feed Forward	1 × 512 × 1024
24	Enc_Attn_6	Multi-Head Attention	1 × 512 × 1024
25	Enc_Add_6	Add	1 × 512 × 1024
26	Enc_Norm_6	LayerNorm	1 × 512 × 1024
27	Enc_FFN_6	Feed Forward	1 × 512 × 1024
28	Enc_Attn_7	Multi-Head Attention	1 × 512 × 1024
29	Enc_Add_7	Add	1 × 512 × 1024
30	Enc_Norm_7	LayerNorm	1 × 512 × 1024
31	Enc_FFN_7	Feed Forward	1 × 512 × 1024
32	Enc_Attn_8	Multi-Head Attention	1 × 512 × 1024
33	Enc_Add_8	Add	1 × 512 × 1024
34	Enc_Norm_8	LayerNorm	1 × 512 × 1024
35	Enc_FFN_8	Feed Forward	1 × 512 × 1024
36	Enc_Attn_9	Multi-Head Attention	1 × 512 × 1024
37	Enc_Add_9	Add	1 × 512 × 1024
38	Enc_Norm_9	LayerNorm	1 × 512 × 1024
39	Enc_FFN_9	Feed Forward	1 × 512 × 1024
40	Enc_Attn_10	Multi-Head Attention	1 × 512 × 1024
41	Enc_Add_10	Add	1 × 512 × 1024
42	Enc_Norm_10	LayerNorm	1 × 512 × 1024
43	Enc_FFN_10	Feed Forward	1 × 512 × 1024
44	Enc_Attn_11	Multi-Head Attention	1 × 512 × 1024
45	Enc_Add_11	Add	1 × 512 × 1024
46	Enc_Norm_11	LayerNorm	1 × 512 × 1024
47	Enc_FFN_11	Feed Forward	1 × 512 × 1024
48	Enc_Attn_12	Multi-Head Attention	1 × 512 × 1024
49	Enc_Add_12	Add	1 × 512 × 1024
50	Enc_Norm_12	LayerNorm	1 × 512 × 1024
51	Enc_FFN_12	Feed Forward	1 × 512 × 1024
52	Enc_Attn_13	Multi-Head Attention	1 × 512 × 1024
53	Enc_Add_13	Add	1 × 512 × 1024
54	Enc_Norm_13	LayerNorm	1 × 512 × 1024
55	Enc_FFN_13	Feed Forward	1 × 512 × 1024
56	Enc_Attn_14	Multi-Head Attention	1 × 512 × 1024
57	Enc_Add_14	Add	1 × 512 × 1024
58	Enc_Norm_14	LayerNorm	1 × 512 × 1024
59	Enc_FFN_14	Feed Forward	1 × 512 × 1024
60	Enc_Attn_15	Multi-Head Attention	1 × 512 × 1024
61	Enc_Add_15	Add	1 × 512 × 1024
62	Enc_Norm_15	LayerNorm	1 × 512 × 1024
63	Enc_FFN_15	Feed Forward	1 × 512 × 1024
64	Enc_Attn_16	Multi-Head Attention	1 × 512 × 1024
65	Enc_Add_16	Add	1 × 512 × 1024
66	Enc_Norm_16	LayerNorm	1 × 512 × 1024
67	Enc_FFN_16	Feed Forward	1 × 512 × 1024
68	Enc_Attn_17	Multi-Head Attention	1 × 512 × 1024
69	Enc_Add_17	Add	1 × 512 × 1024
70	Enc_Norm_17	LayerNorm	1 × 512 × 1024
71	Enc_FFN_17	Feed Forward	1 × 512 × 1024
72	Enc_Attn_18	Multi-Head Attention	1 × 512 × 1024
73	Enc_Add_18	Add	1 × 512 × 1024
74	Enc_Norm_18	LayerNorm	1 × 512 × 1024
75	Enc_FFN_18	Feed Forward	1 × 512 × 1024
76	Enc_Attn_19	Multi-Head Attention	1 × 512 × 1024
77	Enc_Add_19	Add	1 × 512 × 1024
78	Enc_Norm_19	LayerNorm	1 × 512 × 1024
79	Enc_FFN_19	Feed Forward	1 × 512 × 1024
80	Enc_Attn_20	Multi-Head Attention	1 × 512 × 1024
81	Enc_Add_20	Add	1 × 512 × 1024
82	Enc_Norm_20	LayerNorm	1 × 512 × 1024
83	Enc_FFN_20	Feed Forward	1 × 512 × 1024
84	Enc_Attn_21	Multi-Head Attention	1 × 512 × 1024
85	Enc_Add_21	Add	1 × 512 × 1024
86	Enc_Norm_21	LayerNorm	1 × 512 × 1024
87	Enc_FFN_21	Feed Forward	1 × 512 × 1024
88	Enc_Attn_22	Multi-Head Attention	1 × 512 × 1024
89	Enc_Add_22	Add	1 × 512 × 1024
90	Enc_Norm_22	LayerNorm	1 × 512 × 1024
91	Enc_FFN_22	Feed Forward	1 × 512 × 1024
92	Enc_Attn_23	Multi-Head Attention	1 × 512 × 1024
93	Enc_Add_23	Add	1 × 512 × 1024
94	Enc_Norm_23	LayerNorm	1 × 512 × 1024
95	Enc_FFN_23	Feed Forward	1 × 512 × 1024
96	Enc_Attn_24	Multi-Head Attention	1 × 512 × 1024
97	Enc_Add_24	Add	1 × 512 × 1024
98	Enc_Norm_24	LayerNorm	1 × 512 × 1024
99	Enc_FFN_24	Feed Forward	1 × 512 × 1024
100	Decoder_Embedding	Embedding	1 × 512 × 1024
101	Dec_SelfAttn_1	Multi-Head Attention	1 × 512 × 1024
102	Dec_Add_1	Add	1 × 512 × 1024
103	Dec_Norm1_1	LayerNorm	1 × 512 × 1024
104	Dec_CrossAttn_1	Cross-Attention	1 × 512 × 1024
105	Dec_Add2_1	Add	1 × 512 × 1024
106	Dec_Norm2_1	LayerNorm	1 × 512 × 1024
107	Dec_FFN_1	Feed Forward	1 × 512 × 1024
108	Dec_SelfAttn_2	Multi-Head Attention	1 × 512 × 1024
109	Dec_Add_2	Add	1 × 512 × 1024
110	Dec_Norm1_2	LayerNorm	1 × 512 × 1024
111	Dec_CrossAttn_2	Cross-Attention	1 × 512 × 1024
112	Dec_Add2_2	Add	1 × 512 × 1024
113	Dec_Norm2_2	LayerNorm	1 × 512 × 1024
114	Dec_FFN_2	Feed Forward	1 × 512 × 1024
115	Dec_SelfAttn_3	Multi-Head Attention	1 × 512 × 1024
116	Dec_Add_3	Add	1 × 512 × 1024
117	Dec_Norm1_3	LayerNorm	1 × 512 × 1024
118	Dec_CrossAttn_3	Cross-Attention	1 × 512 × 1024
119	Dec_Add2_3	Add	1 × 512 × 1024
120	Dec_Norm2_3	LayerNorm	1 × 512 × 1024
121	Dec_FFN_3	Feed Forward	1 × 512 × 1024
122	Dec_SelfAttn_4	Multi-Head Attention	1 × 512 × 1024
123	Dec_Add_4	Add	1 × 512 × 1024
124	Dec_Norm1_4	LayerNorm	1 × 512 × 1024
125	Dec_CrossAttn_4	Cross-Attention	1 × 512 × 1024
126	Dec_Add2_4	Add	1 × 512 × 1024
127	Dec_Norm2_4	LayerNorm	1 × 512 × 1024
128	Dec_FFN_4	Feed Forward	1 × 512 × 1024
129	Dec_SelfAttn_5	Multi-Head Attention	1 × 512 × 1024
130	Dec_Add_5	Add	1 × 512 × 1024
131	Dec_Norm1_5	LayerNorm	1 × 512 × 1024
132	Dec_CrossAttn_5	Cross-Attention	1 × 512 × 1024
133	Dec_Add2_5	Add	1 × 512 × 1024
134	Dec_Norm2_5	LayerNorm	1 × 512 × 1024
135	Dec_FFN_5	Feed Forward	1 × 512 × 1024
136	Dec_SelfAttn_6	Multi-Head Attention	1 × 512 × 1024
137	Dec_Add_6	Add	1 × 512 × 1024
138	Dec_Norm1_6	LayerNorm	1 × 512 × 1024
139	Dec_CrossAttn_6	Cross-Attention	1 × 512 × 1024
140	Dec_Add2_6	Add	1 × 512 × 1024
141	Dec_Norm2_6	LayerNorm	1 × 512 × 1024
142	Dec_FFN_6	Feed Forward	1 × 512 × 1024
143	Dec_SelfAttn_7	Multi-Head Attention	1 × 512 × 1024
144	Dec_Add_7	Add	1 × 512 × 1024
145	Dec_Norm1_7	LayerNorm	1 × 512 × 1024
146	Dec_CrossAttn_7	Cross-Attention	1 × 512 × 1024
147	Dec_Add2_7	Add	1 × 512 × 1024
148	Dec_Norm2_7	LayerNorm	1 × 512 × 1024
149	Dec_FFN_7	Feed Forward	1 × 512 × 1024
150	Dec_SelfAttn_8	Multi-Head Attention	1 × 512 × 1024
151	Dec_Add_8	Add	1 × 512 × 1024
152	Dec_Norm1_8	LayerNorm	1 × 512 × 1024
153	Dec_CrossAttn_8	Cross-Attention	1 × 512 × 1024
154	Dec_Add2_8	Add	1 × 512 × 1024
155	Dec_Norm2_8	LayerNorm	1 × 512 × 1024
156	Dec_FFN_8	Feed Forward	1 × 512 × 1024
157	Dec_SelfAttn_9	Multi-Head Attention	1 × 512 × 1024
158	Dec_Add_9	Add	1 × 512 × 1024
159	Dec_Norm1_9	LayerNorm	1 × 512 × 1024
160	Dec_CrossAttn_9	Cross-Attention	1 × 512 × 1024
161	Dec_Add2_9	Add	1 × 512 × 1024
162	Dec_Norm2_9	LayerNorm	1 × 512 × 1024
163	Dec_FFN_9	Feed Forward	1 × 512 × 1024
164	Dec_SelfAttn_10	Multi-Head Attention	1 × 512 × 1024
165	Dec_Add_10	Add	1 × 512 × 1024
166	Dec_Norm1_10	LayerNorm	1 × 512 × 1024
167	Dec_CrossAttn_10	Cross-Attention	1 × 512 × 1024
168	Dec_Add2_10	Add	1 × 512 × 1024
169	Dec_Norm2_10	LayerNorm	1 × 512 × 1024
170	Dec_FFN_10	Feed Forward	1 × 512 × 1024
171	Dec_SelfAttn_11	Multi-Head Attention	1 × 512 × 1024
172	Dec_Add_11	Add	1 × 512 × 1024
173	Dec_Norm1_11	LayerNorm	1 × 512 × 1024
174	Dec_CrossAttn_11	Cross-Attention	1 × 512 × 1024
175	Dec_Add2_11	Add	1 × 512 × 1024
176	Dec_Norm2_11	LayerNorm	1 × 512 × 1024
177	Dec_FFN_11	Feed Forward	1 × 512 × 1024
178	Dec_SelfAttn_12	Multi-Head Attention	1 × 512 × 1024
179	Dec_Add_12	Add	1 × 512 × 1024
180	Dec_Norm1_12	LayerNorm	1 × 512 × 1024
181	Dec_CrossAttn_12	Cross-Attention	1 × 512 × 1024
182	Dec_Add2_12	Add	1 × 512 × 1024
183	Dec_Norm2_12	LayerNorm	1 × 512 × 1024
184	Dec_FFN_12	Feed Forward	1 × 512 × 1024
185	Dec_SelfAttn_13	Multi-Head Attention	1 × 512 × 1024
186	Dec_Add_13	Add	1 × 512 × 1024
187	Dec_Norm1_13	LayerNorm	1 × 512 × 1024
188	Dec_CrossAttn_13	Cross-Attention	1 × 512 × 1024
189	Dec_Add2_13	Add	1 × 512 × 1024
190	Dec_Norm2_13	LayerNorm	1 × 512 × 1024
191	Dec_FFN_13	Feed Forward	1 × 512 × 1024
192	Dec_SelfAttn_14	Multi-Head Attention	1 × 512 × 1024
193	Dec_Add_14	Add	1 × 512 × 1024
194	Dec_Norm1_14	LayerNorm	1 × 512 × 1024
195	Dec_CrossAttn_14	Cross-Attention	1 × 512 × 1024
196	Dec_Add2_14	Add	1 × 512 × 1024
197	Dec_Norm2_14	LayerNorm	1 × 512 × 1024
198	Dec_FFN_14	Feed Forward	1 × 512 × 1024
199	Dec_SelfAttn_15	Multi-Head Attention	1 × 512 × 1024
200	Dec_Add_15	Add	1 × 512 × 1024
201	Dec_Norm1_15	LayerNorm	1 × 512 × 1024
202	Dec_CrossAttn_15	Cross-Attention	1 × 512 × 1024
203	Dec_Add2_15	Add	1 × 512 × 1024
204	Dec_Norm2_15	LayerNorm	1 × 512 × 1024
205	Dec_FFN_15	Feed Forward	1 × 512 × 1024
206	Dec_SelfAttn_16	Multi-Head Attention	1 × 512 × 1024
207	Dec_Add_16	Add	1 × 512 × 1024
208	Dec_Norm1_16	LayerNorm	1 × 512 × 1024
209	Dec_CrossAttn_16	Cross-Attention	1 × 512 × 1024
210	Dec_Add2_16	Add	1 × 512 × 1024
211	Dec_Norm2_16	LayerNorm	1 × 512 × 1024
212	Dec_FFN_16	Feed Forward	1 × 512 × 1024
213	Dec_SelfAttn_17	Multi-Head Attention	1 × 512 × 1024
214	Dec_Add_17	Add	1 × 512 × 1024
215	Dec_Norm1_17	LayerNorm	1 × 512 × 1024
216	Dec_CrossAttn_17	Cross-Attention	1 × 512 × 1024
217	Dec_Add2_17	Add	1 × 512 × 1024
218	Dec_Norm2_17	LayerNorm	1 × 512 × 1024
219	Dec_FFN_17	Feed Forward	1 × 512 × 1024
220	Dec_SelfAttn_18	Multi-Head Attention	1 × 512 × 1024
221	Dec_Add_18	Add	1 × 512 × 1024
222	Dec_Norm1_18	LayerNorm	1 × 512 × 1024
223	Dec_CrossAttn_18	Cross-Attention	1 × 512 × 1024
224	Dec_Add2_18	Add	1 × 512 × 1024
225	Dec_Norm2_18	LayerNorm	1 × 512 × 1024
226	Dec_FFN_18	Feed Forward	1 × 512 × 1024
227	Dec_SelfAttn_19	Multi-Head Attention	1 × 512 × 1024
228	Dec_Add_19	Add	1 × 512 × 1024
229	Dec_Norm1_19	LayerNorm	1 × 512 × 1024
230	Dec_CrossAttn_19	Cross-Attention	1 × 512 × 1024
231	Dec_Add2_19	Add	1 × 512 × 1024
232	Dec_Norm2_19	LayerNorm	1 × 512 × 1024
233	Dec_FFN_19	Feed Forward	1 × 512 × 1024
234	Dec_SelfAttn_20	Multi-Head Attention	1 × 512 × 1024
235	Dec_Add_20	Add	1 × 512 × 1024
236	Dec_Norm1_20	LayerNorm	1 × 512 × 1024
237	Dec_CrossAttn_20	Cross-Attention	1 × 512 × 1024
238	Dec_Add2_20	Add	1 × 512 × 1024
239	Dec_Norm2_20	LayerNorm	1 × 512 × 1024
240	Dec_FFN_20	Feed Forward	1 × 512 × 1024
241	Dec_SelfAttn_21	Multi-Head Attention	1 × 512 × 1024
242	Dec_Add_21	Add	1 × 512 × 1024
243	Dec_Norm1_21	LayerNorm	1 × 512 × 1024
244	Dec_CrossAttn_21	Cross-Attention	1 × 512 × 1024
245	Dec_Add2_21	Add	1 × 512 × 1024
246	Dec_Norm2_21	LayerNorm	1 × 512 × 1024
247	Dec_FFN_21	Feed Forward	1 × 512 × 1024
248	Dec_SelfAttn_22	Multi-Head Attention	1 × 512 × 1024
249	Dec_Add_22	Add	1 × 512 × 1024
250	Dec_Norm1_22	LayerNorm	1 × 512 × 1024
251	Dec_CrossAttn_22	Cross-Attention	1 × 512 × 1024
252	Dec_Add2_22	Add	1 × 512 × 1024
253	Dec_Norm2_22	LayerNorm	1 × 512 × 1024
254	Dec_FFN_22	Feed Forward	1 × 512 × 1024
255	Dec_SelfAttn_23	Multi-Head Attention	1 × 512 × 1024
256	Dec_Add_23	Add	1 × 512 × 1024
257	Dec_Norm1_23	LayerNorm	1 × 512 × 1024
258	Dec_CrossAttn_23	Cross-Attention	1 × 512 × 1024
259	Dec_Add2_23	Add	1 × 512 × 1024
260	Dec_Norm2_23	LayerNorm	1 × 512 × 1024
261	Dec_FFN_23	Feed Forward	1 × 512 × 1024
262	Dec_SelfAttn_24	Multi-Head Attention	1 × 512 × 1024
263	Dec_Add_24	Add	1 × 512 × 1024
264	Dec_Norm1_24	LayerNorm	1 × 512 × 1024
265	Dec_CrossAttn_24	Cross-Attention	1 × 512 × 1024
266	Dec_Add2_24	Add	1 × 512 × 1024
267	Dec_Norm2_24	LayerNorm	1 × 512 × 1024
268	Dec_FFN_24	Feed Forward	1 × 512 × 1024
269	LM_Head	Linear	1 × 512 × 32128
270	Output	Output	1 × 512 × 32128

What the verifier says

infoAt 48 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace facebook/w2v-bert-2.0 --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan