Comparisons / ResNet Block vs ViT-B/16

ResNet Block vs ViT-B/16

Convolution against attention for images.

ViT-B/16 has 8.4M more parameters than ResNet Block: 8 layers added, 4 removed, 3 changed.

Baseline

ResNet Block

Layers: 7
Parameters: 74K
Input: 64 × 32 × 32
Output: 64 × 32 × 32
Forward-passes: yes
Est. train cost: $0.045

Compared

ViT-B/16

Layers: 11
Parameters: 8.4M
Input: 3 × 224 × 224
Output: 196 × 1000
Forward-passes: yes
Est. train cost: $0.105

The deltas

Every number is ViT-B/16 relative to ResNet Block.

Parameters

+8.4M (114× the size)

Layers

Added

Removed

Changed

Unchanged

Which GPUs each one fits

Each side is measured at its own declared input (64 × 32 × 32 against 3 × 224 × 224). Both columns are right about their own model; the difference between them is not a fact about the designs.

GPU	ResNet Block	ViT-B/16
T4 16GB	fits	fits
A100 40GB	fits	fits
H100 80GB	fits	fits

Layer by layer

Aligned in topological order. 2 of 17 rows are the same layer with the same parameters.

Hide all 17 rows

		ResNet Block			ViT-B/16
		Layer	Params	Output	Layer	Params	Output
1	changed shape	Input Input		64 × 32 × 32	image Input		3 × 224 × 224
2	removed	Conv2D_1 Conv2d	640	64 × 32 × 32	—
3	added	—			patch_embed Patch Embed	591K	196 × 768
4	added	—			pos_embed Positional Encoding		196 × 768
5	added	—			dropout Dropout		196 × 768
6	changed type, normalizedShape	BatchNorm_1 Batch Norm		64 × 32 × 32	norm_1 Layer Norm	1.5K	196 × 768
7	removed	ReLU_1 Relu		64 × 32 × 32	—
8	removed	Conv2D_2 Conv2d	640	64 × 32 × 32	—
9	added	—			attn Multi Head Attention	2.4M	196 × 768
10	added	—			residual_1 Add		196 × 768
11	changed type, normalizedShape	BatchNorm_2 Batch Norm		64 × 32 × 32	norm_2 Layer Norm	1.5K	196 × 768
12	added	—			mlp Feed Forward	4.7M	196 × 768
13	same	Add Add		64 × 32 × 32	residual_2 Add		196 × 768
14	removed	ReLU_2 Relu		64 × 32 × 32	—
15	added	—			norm_final Layer Norm	1.5K	196 × 768
16	added	—			head Linear		196 × 1000
17	same	Output Output		64 × 32 × 32	class_logits Output		196 × 1000

What this is not

The two are priced at different declared inputs (64 × 32 × 32 against 3 × 224 × 224), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

Take it further

Open either graph in the editor, change it, and check it again: ResNet Block · ViT-B/16

Compare any two models of your own, including anything on Hugging Face: the comparison tool.

Machine-readable: this page as markdown · the pair index · POST https://www.neurarch.com/api/v1/plan for a graph of your own.

Related comparisons

ResNet Block vs Simple CNN

What the residual connection costs and what it buys.

ViT-B/16 vs Swin-Tiny

A flat vision transformer against a hierarchical one.

BERT Base vs ViT-B/16

The same transformer applied to text and to images.

ViT-B/16 vs DiT-XL/2

A vision transformer against a diffusion transformer.