N Neurarch Architectures Models Checks Data Docs Open the app

Comparisons / LLaVA-1.5 vs CLIP ViT-B/32

LLaVA-1.5 vs CLIP ViT-B/32

A vision-language model against the encoder it is built on.

CLIP ViT-B/32 has 6.9B fewer parameters than LLaVA-1.5: 7 layers added, 196 removed, 32 changed.

Baseline

LLaVA-1.5

Layers
226
Parameters
7.1B
Input
3 × 336 × 336
Output
1 × 2624 × 32000
Forward-passes
yes
Est. train cost
$690.06
Compared

CLIP ViT-B/32

Layers
37
Parameters
151M
Input
3 × 224 × 224
Output
512
Forward-passes
yes
Est. train cost
$0.313

The deltas

Every number is CLIP ViT-B/32 relative to LLaVA-1.5.

Parameters
-6.9B (-97.9%)
Layers
-189
Added
7
Removed
196
Changed
32
Unchanged
1

Which GPUs each one fits

Each side is measured at its own declared input (3 × 336 × 336 against 3 × 224 × 224). Both columns are right about their own model; the difference between them is not a fact about the designs.

GPULLaVA-1.5CLIP ViT-B/32
T4 16GBnofits
A100 40GBnofits
H100 80GBnofits

Layer by layer

Aligned in topological order. 1 of 236 rows are the same layer with the same parameters.

Show all 236 rows
LLaVA-1.5CLIP ViT-B/32
LayerParamsOutputLayerParamsOutput
1changed
shape
image
Input
3 × 336 × 336image
Input
3 × 224 × 224
2changed
shape
text_tokens
Input
1 × 2048tokens
Input
77
3changed
imgSize, patchSize, embedDim
clip_patch_14
Patch Embed
603K576 × 1024patch_embed
Patch Embed
2.4M49 × 768
4changed
numEmbeddings, embeddingDim
token_embed
Embedding
131M1 × 2048 × 4096tok_embed
Embedding
25M77 × 512
5addedvis_pos
Positional Encoding
49 × 768
6changed
maxLen, embedDim
vis_pos
Positional Encoding
576 × 1024txt_pos
Positional Encoding
77 × 512
7changed
embedDim, numHeads, ffDim
clip_block_1
Transformer Block
13M576 × 1024vis_block_1
Transformer Block
7.1M49 × 768
8changed
embedDim, numHeads, ffDim
clip_block_2
Transformer Block
13M576 × 1024txt_block_1
Transformer Block
3.2M77 × 512
9changed
embedDim, numHeads, ffDim
clip_block_3
Transformer Block
13M576 × 1024vis_block_2
Transformer Block
7.1M49 × 768
10changed
embedDim, numHeads, ffDim
clip_block_4
Transformer Block
13M576 × 1024txt_block_2
Transformer Block
3.2M77 × 512
11changed
embedDim, numHeads, ffDim
clip_block_5
Transformer Block
13M576 × 1024vis_block_3
Transformer Block
7.1M49 × 768
12changed
embedDim, numHeads, ffDim
clip_block_6
Transformer Block
13M576 × 1024txt_block_3
Transformer Block
3.2M77 × 512
13changed
embedDim, numHeads, ffDim
clip_block_7
Transformer Block
13M576 × 1024vis_block_4
Transformer Block
7.1M49 × 768
14changed
embedDim, numHeads, ffDim
clip_block_8
Transformer Block
13M576 × 1024txt_block_4
Transformer Block
3.2M77 × 512
15changed
embedDim, numHeads, ffDim
clip_block_9
Transformer Block
13M576 × 1024vis_block_5
Transformer Block
7.1M49 × 768
16changed
embedDim, numHeads, ffDim
clip_block_10
Transformer Block
13M576 × 1024txt_block_5
Transformer Block
3.2M77 × 512
17changed
embedDim, numHeads, ffDim
clip_block_11
Transformer Block
13M576 × 1024vis_block_6
Transformer Block
7.1M49 × 768
18changed
embedDim, numHeads, ffDim
clip_block_12
Transformer Block
13M576 × 1024txt_block_6
Transformer Block
3.2M77 × 512
19changed
embedDim, numHeads, ffDim
clip_block_13
Transformer Block
13M576 × 1024vis_block_7
Transformer Block
7.1M49 × 768
20changed
embedDim, numHeads, ffDim
clip_block_14
Transformer Block
13M576 × 1024txt_block_7
Transformer Block
3.2M77 × 512
21changed
embedDim, numHeads, ffDim
clip_block_15
Transformer Block
13M576 × 1024vis_block_8
Transformer Block
7.1M49 × 768
22changed
embedDim, numHeads, ffDim
clip_block_16
Transformer Block
13M576 × 1024txt_block_8
Transformer Block
3.2M77 × 512
23changed
embedDim, numHeads, ffDim
clip_block_17
Transformer Block
13M576 × 1024vis_block_9
Transformer Block
7.1M49 × 768
24changed
embedDim, numHeads, ffDim
clip_block_18
Transformer Block
13M576 × 1024txt_block_9
Transformer Block
3.2M77 × 512
25changed
embedDim, numHeads, ffDim
clip_block_19
Transformer Block
13M576 × 1024vis_block_10
Transformer Block
7.1M49 × 768
26changed
embedDim, numHeads, ffDim
clip_block_20
Transformer Block
13M576 × 1024txt_block_10
Transformer Block
3.2M77 × 512
27changed
embedDim, numHeads, ffDim
clip_block_21
Transformer Block
13M576 × 1024vis_block_11
Transformer Block
7.1M49 × 768
28changed
embedDim, numHeads, ffDim
clip_block_22
Transformer Block
13M576 × 1024txt_block_11
Transformer Block
3.2M77 × 512
29changed
embedDim, numHeads, ffDim
clip_block_23
Transformer Block
13M576 × 1024vis_block_12
Transformer Block
7.1M49 × 768
30changed
embedDim, numHeads, ffDim
clip_block_24
Transformer Block
13M576 × 1024txt_block_12
Transformer Block
3.2M77 × 512
31removedprojector_1
Linear
4.2M576 × 4096
32removedprojector_gelu
Gelu
576 × 4096
33removedprojector_2
Linear
17M576 × 4096
34removedvision_tokens
Reshape
1 × 576 × 4096
35removedimage+text_tokens
Concatenate
1 × 2624 × 4096
36removedllm_attn_norm_1
Rms Norm
4.1K1 × 2624 × 4096
37removedllm_attn_1
Grouped Query Attention
67M1 × 2624 × 4096
38removedllm_res1_1
Add
1 × 2624 × 4096
39removedllm_ffn_norm_1
Rms Norm
4.1K1 × 2624 × 4096
40removedllm_ffn_1
Swiglu
135M1 × 2624 × 4096
41removedllm_res2_1
Add
1 × 2624 × 4096
42removedllm_attn_norm_2
Rms Norm
4.1K1 × 2624 × 4096
43removedllm_attn_2
Grouped Query Attention
67M1 × 2624 × 4096
44removedllm_res1_2
Add
1 × 2624 × 4096
45removedllm_ffn_norm_2
Rms Norm
4.1K1 × 2624 × 4096
46removedllm_ffn_2
Swiglu
135M1 × 2624 × 4096
47removedllm_res2_2
Add
1 × 2624 × 4096
48removedllm_attn_norm_3
Rms Norm
4.1K1 × 2624 × 4096
49removedllm_attn_3
Grouped Query Attention
67M1 × 2624 × 4096
50removedllm_res1_3
Add
1 × 2624 × 4096
51removedllm_ffn_norm_3
Rms Norm
4.1K1 × 2624 × 4096
52removedllm_ffn_3
Swiglu
135M1 × 2624 × 4096
53removedllm_res2_3
Add
1 × 2624 × 4096
54removedllm_attn_norm_4
Rms Norm
4.1K1 × 2624 × 4096
55removedllm_attn_4
Grouped Query Attention
67M1 × 2624 × 4096
56removedllm_res1_4
Add
1 × 2624 × 4096
57removedllm_ffn_norm_4
Rms Norm
4.1K1 × 2624 × 4096
58removedllm_ffn_4
Swiglu
135M1 × 2624 × 4096
59removedllm_res2_4
Add
1 × 2624 × 4096
60removedllm_attn_norm_5
Rms Norm
4.1K1 × 2624 × 4096
61removedllm_attn_5
Grouped Query Attention
67M1 × 2624 × 4096
62removedllm_res1_5
Add
1 × 2624 × 4096
63removedllm_ffn_norm_5
Rms Norm
4.1K1 × 2624 × 4096
64removedllm_ffn_5
Swiglu
135M1 × 2624 × 4096
65removedllm_res2_5
Add
1 × 2624 × 4096
66removedllm_attn_norm_6
Rms Norm
4.1K1 × 2624 × 4096
67removedllm_attn_6
Grouped Query Attention
67M1 × 2624 × 4096
68removedllm_res1_6
Add
1 × 2624 × 4096
69removedllm_ffn_norm_6
Rms Norm
4.1K1 × 2624 × 4096
70removedllm_ffn_6
Swiglu
135M1 × 2624 × 4096
71removedllm_res2_6
Add
1 × 2624 × 4096
72removedllm_attn_norm_7
Rms Norm
4.1K1 × 2624 × 4096
73removedllm_attn_7
Grouped Query Attention
67M1 × 2624 × 4096
74removedllm_res1_7
Add
1 × 2624 × 4096
75removedllm_ffn_norm_7
Rms Norm
4.1K1 × 2624 × 4096
76removedllm_ffn_7
Swiglu
135M1 × 2624 × 4096
77removedllm_res2_7
Add
1 × 2624 × 4096
78removedllm_attn_norm_8
Rms Norm
4.1K1 × 2624 × 4096
79removedllm_attn_8
Grouped Query Attention
67M1 × 2624 × 4096
80removedllm_res1_8
Add
1 × 2624 × 4096
81removedllm_ffn_norm_8
Rms Norm
4.1K1 × 2624 × 4096
82removedllm_ffn_8
Swiglu
135M1 × 2624 × 4096
83removedllm_res2_8
Add
1 × 2624 × 4096
84removedllm_attn_norm_9
Rms Norm
4.1K1 × 2624 × 4096
85removedllm_attn_9
Grouped Query Attention
67M1 × 2624 × 4096
86removedllm_res1_9
Add
1 × 2624 × 4096
87removedllm_ffn_norm_9
Rms Norm
4.1K1 × 2624 × 4096
88removedllm_ffn_9
Swiglu
135M1 × 2624 × 4096
89removedllm_res2_9
Add
1 × 2624 × 4096
90removedllm_attn_norm_10
Rms Norm
4.1K1 × 2624 × 4096
91removedllm_attn_10
Grouped Query Attention
67M1 × 2624 × 4096
92removedllm_res1_10
Add
1 × 2624 × 4096
93removedllm_ffn_norm_10
Rms Norm
4.1K1 × 2624 × 4096
94removedllm_ffn_10
Swiglu
135M1 × 2624 × 4096
95removedllm_res2_10
Add
1 × 2624 × 4096
96removedllm_attn_norm_11
Rms Norm
4.1K1 × 2624 × 4096
97removedllm_attn_11
Grouped Query Attention
67M1 × 2624 × 4096
98removedllm_res1_11
Add
1 × 2624 × 4096
99removedllm_ffn_norm_11
Rms Norm
4.1K1 × 2624 × 4096
100removedllm_ffn_11
Swiglu
135M1 × 2624 × 4096
101removedllm_res2_11
Add
1 × 2624 × 4096
102removedllm_attn_norm_12
Rms Norm
4.1K1 × 2624 × 4096
103removedllm_attn_12
Grouped Query Attention
67M1 × 2624 × 4096
104removedllm_res1_12
Add
1 × 2624 × 4096
105removedllm_ffn_norm_12
Rms Norm
4.1K1 × 2624 × 4096
106removedllm_ffn_12
Swiglu
135M1 × 2624 × 4096
107removedllm_res2_12
Add
1 × 2624 × 4096
108removedllm_attn_norm_13
Rms Norm
4.1K1 × 2624 × 4096
109removedllm_attn_13
Grouped Query Attention
67M1 × 2624 × 4096
110removedllm_res1_13
Add
1 × 2624 × 4096
111removedllm_ffn_norm_13
Rms Norm
4.1K1 × 2624 × 4096
112removedllm_ffn_13
Swiglu
135M1 × 2624 × 4096
113removedllm_res2_13
Add
1 × 2624 × 4096
114removedllm_attn_norm_14
Rms Norm
4.1K1 × 2624 × 4096
115removedllm_attn_14
Grouped Query Attention
67M1 × 2624 × 4096
116removedllm_res1_14
Add
1 × 2624 × 4096
117removedllm_ffn_norm_14
Rms Norm
4.1K1 × 2624 × 4096
118removedllm_ffn_14
Swiglu
135M1 × 2624 × 4096
119removedllm_res2_14
Add
1 × 2624 × 4096
120removedllm_attn_norm_15
Rms Norm
4.1K1 × 2624 × 4096
121removedllm_attn_15
Grouped Query Attention
67M1 × 2624 × 4096
122removedllm_res1_15
Add
1 × 2624 × 4096
123removedllm_ffn_norm_15
Rms Norm
4.1K1 × 2624 × 4096
124removedllm_ffn_15
Swiglu
135M1 × 2624 × 4096
125removedllm_res2_15
Add
1 × 2624 × 4096
126removedllm_attn_norm_16
Rms Norm
4.1K1 × 2624 × 4096
127removedllm_attn_16
Grouped Query Attention
67M1 × 2624 × 4096
128removedllm_res1_16
Add
1 × 2624 × 4096
129removedllm_ffn_norm_16
Rms Norm
4.1K1 × 2624 × 4096
130removedllm_ffn_16
Swiglu
135M1 × 2624 × 4096
131removedllm_res2_16
Add
1 × 2624 × 4096
132removedllm_attn_norm_17
Rms Norm
4.1K1 × 2624 × 4096
133removedllm_attn_17
Grouped Query Attention
67M1 × 2624 × 4096
134removedllm_res1_17
Add
1 × 2624 × 4096
135removedllm_ffn_norm_17
Rms Norm
4.1K1 × 2624 × 4096
136removedllm_ffn_17
Swiglu
135M1 × 2624 × 4096
137removedllm_res2_17
Add
1 × 2624 × 4096
138removedllm_attn_norm_18
Rms Norm
4.1K1 × 2624 × 4096
139removedllm_attn_18
Grouped Query Attention
67M1 × 2624 × 4096
140removedllm_res1_18
Add
1 × 2624 × 4096
141removedllm_ffn_norm_18
Rms Norm
4.1K1 × 2624 × 4096
142removedllm_ffn_18
Swiglu
135M1 × 2624 × 4096
143removedllm_res2_18
Add
1 × 2624 × 4096
144removedllm_attn_norm_19
Rms Norm
4.1K1 × 2624 × 4096
145removedllm_attn_19
Grouped Query Attention
67M1 × 2624 × 4096
146removedllm_res1_19
Add
1 × 2624 × 4096
147removedllm_ffn_norm_19
Rms Norm
4.1K1 × 2624 × 4096
148removedllm_ffn_19
Swiglu
135M1 × 2624 × 4096
149removedllm_res2_19
Add
1 × 2624 × 4096
150removedllm_attn_norm_20
Rms Norm
4.1K1 × 2624 × 4096
151removedllm_attn_20
Grouped Query Attention
67M1 × 2624 × 4096
152removedllm_res1_20
Add
1 × 2624 × 4096
153removedllm_ffn_norm_20
Rms Norm
4.1K1 × 2624 × 4096
154removedllm_ffn_20
Swiglu
135M1 × 2624 × 4096
155removedllm_res2_20
Add
1 × 2624 × 4096
156removedllm_attn_norm_21
Rms Norm
4.1K1 × 2624 × 4096
157removedllm_attn_21
Grouped Query Attention
67M1 × 2624 × 4096
158removedllm_res1_21
Add
1 × 2624 × 4096
159removedllm_ffn_norm_21
Rms Norm
4.1K1 × 2624 × 4096
160removedllm_ffn_21
Swiglu
135M1 × 2624 × 4096
161removedllm_res2_21
Add
1 × 2624 × 4096
162removedllm_attn_norm_22
Rms Norm
4.1K1 × 2624 × 4096
163removedllm_attn_22
Grouped Query Attention
67M1 × 2624 × 4096
164removedllm_res1_22
Add
1 × 2624 × 4096
165removedllm_ffn_norm_22
Rms Norm
4.1K1 × 2624 × 4096
166removedllm_ffn_22
Swiglu
135M1 × 2624 × 4096
167removedllm_res2_22
Add
1 × 2624 × 4096
168removedllm_attn_norm_23
Rms Norm
4.1K1 × 2624 × 4096
169removedllm_attn_23
Grouped Query Attention
67M1 × 2624 × 4096
170removedllm_res1_23
Add
1 × 2624 × 4096
171removedllm_ffn_norm_23
Rms Norm
4.1K1 × 2624 × 4096
172removedllm_ffn_23
Swiglu
135M1 × 2624 × 4096
173removedllm_res2_23
Add
1 × 2624 × 4096
174removedllm_attn_norm_24
Rms Norm
4.1K1 × 2624 × 4096
175removedllm_attn_24
Grouped Query Attention
67M1 × 2624 × 4096
176removedllm_res1_24
Add
1 × 2624 × 4096
177removedllm_ffn_norm_24
Rms Norm
4.1K1 × 2624 × 4096
178removedllm_ffn_24
Swiglu
135M1 × 2624 × 4096
179removedllm_res2_24
Add
1 × 2624 × 4096
180removedllm_attn_norm_25
Rms Norm
4.1K1 × 2624 × 4096
181removedllm_attn_25
Grouped Query Attention
67M1 × 2624 × 4096
182removedllm_res1_25
Add
1 × 2624 × 4096
183removedllm_ffn_norm_25
Rms Norm
4.1K1 × 2624 × 4096
184removedllm_ffn_25
Swiglu
135M1 × 2624 × 4096
185removedllm_res2_25
Add
1 × 2624 × 4096
186removedllm_attn_norm_26
Rms Norm
4.1K1 × 2624 × 4096
187removedllm_attn_26
Grouped Query Attention
67M1 × 2624 × 4096
188removedllm_res1_26
Add
1 × 2624 × 4096
189removedllm_ffn_norm_26
Rms Norm
4.1K1 × 2624 × 4096
190removedllm_ffn_26
Swiglu
135M1 × 2624 × 4096
191removedllm_res2_26
Add
1 × 2624 × 4096
192removedllm_attn_norm_27
Rms Norm
4.1K1 × 2624 × 4096
193removedllm_attn_27
Grouped Query Attention
67M1 × 2624 × 4096
194removedllm_res1_27
Add
1 × 2624 × 4096
195removedllm_ffn_norm_27
Rms Norm
4.1K1 × 2624 × 4096
196removedllm_ffn_27
Swiglu
135M1 × 2624 × 4096
197removedllm_res2_27
Add
1 × 2624 × 4096
198removedllm_attn_norm_28
Rms Norm
4.1K1 × 2624 × 4096
199removedllm_attn_28
Grouped Query Attention
67M1 × 2624 × 4096
200removedllm_res1_28
Add
1 × 2624 × 4096
201removedllm_ffn_norm_28
Rms Norm
4.1K1 × 2624 × 4096
202removedllm_ffn_28
Swiglu
135M1 × 2624 × 4096
203removedllm_res2_28
Add
1 × 2624 × 4096
204removedllm_attn_norm_29
Rms Norm
4.1K1 × 2624 × 4096
205removedllm_attn_29
Grouped Query Attention
67M1 × 2624 × 4096
206removedllm_res1_29
Add
1 × 2624 × 4096
207removedllm_ffn_norm_29
Rms Norm
4.1K1 × 2624 × 4096
208removedllm_ffn_29
Swiglu
135M1 × 2624 × 4096
209removedllm_res2_29
Add
1 × 2624 × 4096
210removedllm_attn_norm_30
Rms Norm
4.1K1 × 2624 × 4096
211removedllm_attn_30
Grouped Query Attention
67M1 × 2624 × 4096
212removedllm_res1_30
Add
1 × 2624 × 4096
213removedllm_ffn_norm_30
Rms Norm
4.1K1 × 2624 × 4096
214removedllm_ffn_30
Swiglu
135M1 × 2624 × 4096
215removedllm_res2_30
Add
1 × 2624 × 4096
216removedllm_attn_norm_31
Rms Norm
4.1K1 × 2624 × 4096
217removedllm_attn_31
Grouped Query Attention
67M1 × 2624 × 4096
218removedllm_res1_31
Add
1 × 2624 × 4096
219removedllm_ffn_norm_31
Rms Norm
4.1K1 × 2624 × 4096
220removedllm_ffn_31
Swiglu
135M1 × 2624 × 4096
221removedllm_res2_31
Add
1 × 2624 × 4096
222removedllm_attn_norm_32
Rms Norm
4.1K1 × 2624 × 4096
223removedllm_attn_32
Grouped Query Attention
67M1 × 2624 × 4096
224removedllm_res1_32
Add
1 × 2624 × 4096
225changed
type, normalizedShape
llm_ffn_norm_32
Rms Norm
4.1K1 × 2624 × 4096vis_norm
Layer Norm
1.5K49 × 768
226removedllm_ffn_32
Swiglu
135M1 × 2624 × 4096
227removedllm_res2_32
Add
1 × 2624 × 4096
228changed
type, normalizedShape
final_norm
Rms Norm
4.1K1 × 2624 × 4096txt_norm
Layer Norm
1.0K77 × 512
229addedto_channels_vis
Permute
768 × 49
230addedto_channels_txt
Permute
512 × 77
231addedvis_pool
Global Avg Pool1d
768
232addedtxt_pool
Global Avg Pool1d
512
233addedimg_proj
Linear
394K512
234changed
inFeatures, outFeatures, bias
lm_head
Linear
131M1 × 2624 × 32000txt_proj
Linear
263K512
235addedsimilarity
Matmul
512
236samelogits
Output
1 × 2624 × 32000logits
Output
512

What this is not

Take it further

Open either graph in the editor, change it, and check it again: LLaVA-1.5 · CLIP ViT-B/32

Compare any two models of your own, including anything on Hugging Face: the comparison tool.

Machine-readable: this page as markdown · the pair index · POST https://www.neurarch.com/api/v1/plan for a graph of your own.