Comparisons / ViT-B/16 vs Swin-Tiny

ViT-B/16 vs Swin-Tiny

A flat vision transformer against a hierarchical one.

Swin-Tiny has 20M more parameters than ViT-B/16: 74 layers added, 2 removed, 4 changed.

Baseline

ViT-B/16

Layers: 11
Parameters: 8.4M
Input: 3 × 224 × 224
Output: 196 × 1000
Forward-passes: yes
Est. train cost: $0.105

Compared

Swin-Tiny

Layers: 83
Parameters: 28M
Input: 3 × 224 × 224
Output: 1000
Forward-passes: yes
Est. train cost: $0.288

The deltas

Every number is Swin-Tiny relative to ViT-B/16.

Parameters

+20M (+234%)

Layers

+72

Added

Removed

Changed

Unchanged

Which GPUs each one fits

Memory for the graph at the input shape both declare. A highlighted row is a card one of them fits and the other does not, which is the difference that decides a purchase.

GPU	ViT-B/16	Swin-Tiny
T4 16GB	fits	fits
A100 40GB	fits	fits
H100 80GB	fits	fits

Layer by layer

Aligned in topological order. 7 of 87 rows are the same layer with the same parameters.

Show all 87 rows

		ViT-B/16			Swin-Tiny
		Layer	Params	Output	Layer	Params	Output
1	same	image Input		3 × 224 × 224	image Input		3 × 224 × 224
2	changed patchSize, embedDim	patch_embed Patch Embed	591K	196 × 768	patch_embed_4x4 Patch Embed	4.7K	3136 × 96
3	removed	pos_embed Positional Encoding		196 × 768	—
4	removed	dropout Dropout		196 × 768	—
5	added	—			s1b1_norm1 Layer Norm	192	3136 × 96
6	added	—			s1b1_window_attn Multi Head Attention	37K	3136 × 96
7	added	—			s1b1_res1 Add		3136 × 96
8	added	—			s1b1_norm2 Layer Norm	192	3136 × 96
9	added	—			s1b1_mlp Feed Forward	74K	3136 × 96
10	added	—			s1b1_res2 Add		3136 × 96
11	added	—			s1b2_norm1 Layer Norm	192	3136 × 96
12	added	—			s1b2_shifted_window_attn Multi Head Attention	37K	3136 × 96
13	added	—			s1b2_res1 Add		3136 × 96
14	added	—			s1b2_norm2 Layer Norm	192	3136 × 96
15	added	—			s1b2_mlp Feed Forward	74K	3136 × 96
16	added	—			s1b2_res2 Add		3136 × 96
17	added	—			merge_patches_n15 Reshape		784 × 384
18	added	—			patch_merging_2 Linear	74K	784 × 192
19	added	—			s2b1_norm1 Layer Norm	384	784 × 192
20	added	—			s2b1_window_attn Multi Head Attention	148K	784 × 192
21	added	—			s2b1_res1 Add		784 × 192
22	added	—			s2b1_norm2 Layer Norm	384	784 × 192
23	added	—			s2b1_mlp Feed Forward	296K	784 × 192
24	added	—			s2b1_res2 Add		784 × 192
25	added	—			s2b2_norm1 Layer Norm	384	784 × 192
26	added	—			s2b2_shifted_window_attn Multi Head Attention	148K	784 × 192
27	added	—			s2b2_res1 Add		784 × 192
28	added	—			s2b2_norm2 Layer Norm	384	784 × 192
29	added	—			s2b2_mlp Feed Forward	296K	784 × 192
30	added	—			s2b2_res2 Add		784 × 192
31	added	—			merge_patches_n28 Reshape		196 × 768
32	added	—			patch_merging_3 Linear	295K	196 × 384
33	added	—			s3b1_norm1 Layer Norm	768	196 × 384
34	added	—			s3b1_window_attn Multi Head Attention	591K	196 × 384
35	added	—			s3b1_res1 Add		196 × 384
36	added	—			s3b1_norm2 Layer Norm	768	196 × 384
37	added	—			s3b1_mlp Feed Forward	1.2M	196 × 384
38	added	—			s3b1_res2 Add		196 × 384
39	added	—			s3b2_norm1 Layer Norm	768	196 × 384
40	added	—			s3b2_shifted_window_attn Multi Head Attention	591K	196 × 384
41	added	—			s3b2_res1 Add		196 × 384
42	added	—			s3b2_norm2 Layer Norm	768	196 × 384
43	added	—			s3b2_mlp Feed Forward	1.2M	196 × 384
44	added	—			s3b2_res2 Add		196 × 384
45	added	—			s3b3_norm1 Layer Norm	768	196 × 384
46	added	—			s3b3_window_attn Multi Head Attention	591K	196 × 384
47	added	—			s3b3_res1 Add		196 × 384
48	added	—			s3b3_norm2 Layer Norm	768	196 × 384
49	added	—			s3b3_mlp Feed Forward	1.2M	196 × 384
50	added	—			s3b3_res2 Add		196 × 384
51	added	—			s3b4_norm1 Layer Norm	768	196 × 384
52	added	—			s3b4_shifted_window_attn Multi Head Attention	591K	196 × 384
53	added	—			s3b4_res1 Add		196 × 384
54	added	—			s3b4_norm2 Layer Norm	768	196 × 384
55	added	—			s3b4_mlp Feed Forward	1.2M	196 × 384
56	added	—			s3b4_res2 Add		196 × 384
57	added	—			s3b5_norm1 Layer Norm	768	196 × 384
58	added	—			s3b5_window_attn Multi Head Attention	591K	196 × 384
59	added	—			s3b5_res1 Add		196 × 384
60	added	—			s3b5_norm2 Layer Norm	768	196 × 384
61	added	—			s3b5_mlp Feed Forward	1.2M	196 × 384
62	added	—			s3b5_res2 Add		196 × 384
63	added	—			s3b6_norm1 Layer Norm	768	196 × 384
64	added	—			s3b6_shifted_window_attn Multi Head Attention	591K	196 × 384
65	added	—			s3b6_res1 Add		196 × 384
66	added	—			s3b6_norm2 Layer Norm	768	196 × 384
67	added	—			s3b6_mlp Feed Forward	1.2M	196 × 384
68	added	—			s3b6_res2 Add		196 × 384
69	added	—			merge_patches_n65 Reshape		49 × 1536
70	added	—			patch_merging_4 Linear	1.2M	49 × 768
71	added	—			s4b1_norm1 Layer Norm	1.5K	49 × 768
72	added	—			s4b1_window_attn Multi Head Attention	2.4M	49 × 768
73	added	—			s4b1_res1 Add		49 × 768
74	added	—			s4b1_norm2 Layer Norm	1.5K	49 × 768
75	added	—			s4b1_mlp Feed Forward	4.7M	49 × 768
76	added	—			s4b1_res2 Add		49 × 768
77	same	norm_1 Layer Norm	1.5K	196 × 768	s4b2_norm1 Layer Norm	1.5K	49 × 768
78	changed numHeads	attn Multi Head Attention	2.4M	196 × 768	s4b2_shifted_window_attn Multi Head Attention	2.4M	49 × 768
79	same	residual_1 Add		196 × 768	s4b2_res1 Add		49 × 768
80	same	norm_2 Layer Norm	1.5K	196 × 768	s4b2_norm2 Layer Norm	1.5K	49 × 768
81	changed hiddenDim, embedDim	mlp Feed Forward	4.7M	196 × 768	s4b2_mlp Feed Forward	4.7M	49 × 768
82	same	residual_2 Add		196 × 768	s4b2_res2 Add		49 × 768
83	same	norm_final Layer Norm	1.5K	196 × 768	final_norm Layer Norm	1.5K	49 × 768
84	added	—			to_channels Permute		768 × 49
85	added	—			avgpool Global Avg Pool1d		768
86	changed inFeatures	head Linear		196 × 1000	classifier Linear	769K	1000
87	same	class_logits Output		196 × 1000	class_logits Output		1000

What this is not

Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

Take it further

Open either graph in the editor, change it, and check it again: ViT-B/16 · Swin-Tiny

Compare any two models of your own, including anything on Hugging Face: the comparison tool.

Machine-readable: this page as markdown · the pair index · POST https://www.neurarch.com/api/v1/plan for a graph of your own.

Related comparisons

ResNet Block vs ViT-B/16

Convolution against attention for images.

BERT Base vs ViT-B/16

The same transformer applied to text and to images.

ViT-B/16 vs DiT-XL/2

A vision transformer against a diffusion transformer.