N Neurarch Architectures Models Checks Data Docs Open the app

Models / qwen3

Qwen3-0.6B

Reconstructed from its own config.json with no weights read. 21.9M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure
596M
596,041,728 parameters
In the published checkpoint
752M
751,632,384 scalars · safetensors.total, read 2026-09-06
Delta
-20.7%

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.

Layers
170
Will it forward-pass
Yes
Priced on
A10G (24GB)
Est. one run
$4276.23
CardMemory
T4 (16GB)weights + activationsfits
A100 (40GB)weights + activationsfits
H100 (80GB)weights + activationsfits

Structure

172 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

LayerTypeOutput shape
1InputInput1 × 40960
2EmbeddingEmbedding1 × 40960 × 1024
3RoPERoPE1 × 40960 × 1024
4RMSNorm_1_1RMSNorm1 × 40960 × 1024
5Attention_1Grouped Query Attn1 × 40960 × 1024
6Add_1_attnAdd1 × 40960 × 1024
7RMSNorm_1_2RMSNorm1 × 40960 × 1024
8FFN_1SwiGLU1 × 40960 × 1024
9Add_1_ffnAdd1 × 40960 × 1024
10RMSNorm_2_1RMSNorm1 × 40960 × 1024
11Attention_2Grouped Query Attn1 × 40960 × 1024
12Add_2_attnAdd1 × 40960 × 1024
13RMSNorm_2_2RMSNorm1 × 40960 × 1024
14FFN_2SwiGLU1 × 40960 × 1024
15Add_2_ffnAdd1 × 40960 × 1024
16RMSNorm_3_1RMSNorm1 × 40960 × 1024
17Attention_3Grouped Query Attn1 × 40960 × 1024
18Add_3_attnAdd1 × 40960 × 1024
19RMSNorm_3_2RMSNorm1 × 40960 × 1024
20FFN_3SwiGLU1 × 40960 × 1024
21Add_3_ffnAdd1 × 40960 × 1024
22RMSNorm_4_1RMSNorm1 × 40960 × 1024
23Attention_4Grouped Query Attn1 × 40960 × 1024
24Add_4_attnAdd1 × 40960 × 1024
25RMSNorm_4_2RMSNorm1 × 40960 × 1024
26FFN_4SwiGLU1 × 40960 × 1024
27Add_4_ffnAdd1 × 40960 × 1024
28RMSNorm_5_1RMSNorm1 × 40960 × 1024
29Attention_5Grouped Query Attn1 × 40960 × 1024
30Add_5_attnAdd1 × 40960 × 1024
31RMSNorm_5_2RMSNorm1 × 40960 × 1024
32FFN_5SwiGLU1 × 40960 × 1024
33Add_5_ffnAdd1 × 40960 × 1024
34RMSNorm_6_1RMSNorm1 × 40960 × 1024
35Attention_6Grouped Query Attn1 × 40960 × 1024
36Add_6_attnAdd1 × 40960 × 1024
37RMSNorm_6_2RMSNorm1 × 40960 × 1024
38FFN_6SwiGLU1 × 40960 × 1024
39Add_6_ffnAdd1 × 40960 × 1024
40RMSNorm_7_1RMSNorm1 × 40960 × 1024
41Attention_7Grouped Query Attn1 × 40960 × 1024
42Add_7_attnAdd1 × 40960 × 1024
43RMSNorm_7_2RMSNorm1 × 40960 × 1024
44FFN_7SwiGLU1 × 40960 × 1024
45Add_7_ffnAdd1 × 40960 × 1024
46RMSNorm_8_1RMSNorm1 × 40960 × 1024
47Attention_8Grouped Query Attn1 × 40960 × 1024
48Add_8_attnAdd1 × 40960 × 1024
49RMSNorm_8_2RMSNorm1 × 40960 × 1024
50FFN_8SwiGLU1 × 40960 × 1024
51Add_8_ffnAdd1 × 40960 × 1024
52RMSNorm_9_1RMSNorm1 × 40960 × 1024
53Attention_9Grouped Query Attn1 × 40960 × 1024
54Add_9_attnAdd1 × 40960 × 1024
55RMSNorm_9_2RMSNorm1 × 40960 × 1024
56FFN_9SwiGLU1 × 40960 × 1024
57Add_9_ffnAdd1 × 40960 × 1024
58RMSNorm_10_1RMSNorm1 × 40960 × 1024
59Attention_10Grouped Query Attn1 × 40960 × 1024
60Add_10_attnAdd1 × 40960 × 1024
61RMSNorm_10_2RMSNorm1 × 40960 × 1024
62FFN_10SwiGLU1 × 40960 × 1024
63Add_10_ffnAdd1 × 40960 × 1024
64RMSNorm_11_1RMSNorm1 × 40960 × 1024
65Attention_11Grouped Query Attn1 × 40960 × 1024
66Add_11_attnAdd1 × 40960 × 1024
67RMSNorm_11_2RMSNorm1 × 40960 × 1024
68FFN_11SwiGLU1 × 40960 × 1024
69Add_11_ffnAdd1 × 40960 × 1024
70RMSNorm_12_1RMSNorm1 × 40960 × 1024
71Attention_12Grouped Query Attn1 × 40960 × 1024
72Add_12_attnAdd1 × 40960 × 1024
73RMSNorm_12_2RMSNorm1 × 40960 × 1024
74FFN_12SwiGLU1 × 40960 × 1024
75Add_12_ffnAdd1 × 40960 × 1024
76RMSNorm_13_1RMSNorm1 × 40960 × 1024
77Attention_13Grouped Query Attn1 × 40960 × 1024
78Add_13_attnAdd1 × 40960 × 1024
79RMSNorm_13_2RMSNorm1 × 40960 × 1024
80FFN_13SwiGLU1 × 40960 × 1024
81Add_13_ffnAdd1 × 40960 × 1024
82RMSNorm_14_1RMSNorm1 × 40960 × 1024
83Attention_14Grouped Query Attn1 × 40960 × 1024
84Add_14_attnAdd1 × 40960 × 1024
85RMSNorm_14_2RMSNorm1 × 40960 × 1024
86FFN_14SwiGLU1 × 40960 × 1024
87Add_14_ffnAdd1 × 40960 × 1024
88RMSNorm_15_1RMSNorm1 × 40960 × 1024
89Attention_15Grouped Query Attn1 × 40960 × 1024
90Add_15_attnAdd1 × 40960 × 1024
91RMSNorm_15_2RMSNorm1 × 40960 × 1024
92FFN_15SwiGLU1 × 40960 × 1024
93Add_15_ffnAdd1 × 40960 × 1024
94RMSNorm_16_1RMSNorm1 × 40960 × 1024
95Attention_16Grouped Query Attn1 × 40960 × 1024
96Add_16_attnAdd1 × 40960 × 1024
97RMSNorm_16_2RMSNorm1 × 40960 × 1024
98FFN_16SwiGLU1 × 40960 × 1024
99Add_16_ffnAdd1 × 40960 × 1024
100RMSNorm_17_1RMSNorm1 × 40960 × 1024
101Attention_17Grouped Query Attn1 × 40960 × 1024
102Add_17_attnAdd1 × 40960 × 1024
103RMSNorm_17_2RMSNorm1 × 40960 × 1024
104FFN_17SwiGLU1 × 40960 × 1024
105Add_17_ffnAdd1 × 40960 × 1024
106RMSNorm_18_1RMSNorm1 × 40960 × 1024
107Attention_18Grouped Query Attn1 × 40960 × 1024
108Add_18_attnAdd1 × 40960 × 1024
109RMSNorm_18_2RMSNorm1 × 40960 × 1024
110FFN_18SwiGLU1 × 40960 × 1024
111Add_18_ffnAdd1 × 40960 × 1024
112RMSNorm_19_1RMSNorm1 × 40960 × 1024
113Attention_19Grouped Query Attn1 × 40960 × 1024
114Add_19_attnAdd1 × 40960 × 1024
115RMSNorm_19_2RMSNorm1 × 40960 × 1024
116FFN_19SwiGLU1 × 40960 × 1024
117Add_19_ffnAdd1 × 40960 × 1024
118RMSNorm_20_1RMSNorm1 × 40960 × 1024
119Attention_20Grouped Query Attn1 × 40960 × 1024
120Add_20_attnAdd1 × 40960 × 1024
121RMSNorm_20_2RMSNorm1 × 40960 × 1024
122FFN_20SwiGLU1 × 40960 × 1024
123Add_20_ffnAdd1 × 40960 × 1024
124RMSNorm_21_1RMSNorm1 × 40960 × 1024
125Attention_21Grouped Query Attn1 × 40960 × 1024
126Add_21_attnAdd1 × 40960 × 1024
127RMSNorm_21_2RMSNorm1 × 40960 × 1024
128FFN_21SwiGLU1 × 40960 × 1024
129Add_21_ffnAdd1 × 40960 × 1024
130RMSNorm_22_1RMSNorm1 × 40960 × 1024
131Attention_22Grouped Query Attn1 × 40960 × 1024
132Add_22_attnAdd1 × 40960 × 1024
133RMSNorm_22_2RMSNorm1 × 40960 × 1024
134FFN_22SwiGLU1 × 40960 × 1024
135Add_22_ffnAdd1 × 40960 × 1024
136RMSNorm_23_1RMSNorm1 × 40960 × 1024
137Attention_23Grouped Query Attn1 × 40960 × 1024
138Add_23_attnAdd1 × 40960 × 1024
139RMSNorm_23_2RMSNorm1 × 40960 × 1024
140FFN_23SwiGLU1 × 40960 × 1024
141Add_23_ffnAdd1 × 40960 × 1024
142RMSNorm_24_1RMSNorm1 × 40960 × 1024
143Attention_24Grouped Query Attn1 × 40960 × 1024
144Add_24_attnAdd1 × 40960 × 1024
145RMSNorm_24_2RMSNorm1 × 40960 × 1024
146FFN_24SwiGLU1 × 40960 × 1024
147Add_24_ffnAdd1 × 40960 × 1024
148RMSNorm_25_1RMSNorm1 × 40960 × 1024
149Attention_25Grouped Query Attn1 × 40960 × 1024
150Add_25_attnAdd1 × 40960 × 1024
151RMSNorm_25_2RMSNorm1 × 40960 × 1024
152FFN_25SwiGLU1 × 40960 × 1024
153Add_25_ffnAdd1 × 40960 × 1024
154RMSNorm_26_1RMSNorm1 × 40960 × 1024
155Attention_26Grouped Query Attn1 × 40960 × 1024
156Add_26_attnAdd1 × 40960 × 1024
157RMSNorm_26_2RMSNorm1 × 40960 × 1024
158FFN_26SwiGLU1 × 40960 × 1024
159Add_26_ffnAdd1 × 40960 × 1024
160RMSNorm_27_1RMSNorm1 × 40960 × 1024
161Attention_27Grouped Query Attn1 × 40960 × 1024
162Add_27_attnAdd1 × 40960 × 1024
163RMSNorm_27_2RMSNorm1 × 40960 × 1024
164FFN_27SwiGLU1 × 40960 × 1024
165Add_27_ffnAdd1 × 40960 × 1024
166RMSNorm_28_1RMSNorm1 × 40960 × 1024
167Attention_28Grouped Query Attn1 × 40960 × 1024
168Add_28_attnAdd1 × 40960 × 1024
169RMSNorm_28_2RMSNorm1 × 40960 × 1024
170FFN_28SwiGLU1 × 40960 × 1024
171Add_28_ffnAdd1 × 40960 × 1024
172OutputOutput1 × 40960 × 1024

What the verifier says

infoAt 28 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace Qwen/Qwen3-0.6B --plan --share
Paste a model instead Markdown twin Index JSON POST /api/v1/plan