Comparisons / ResNet Block vs Simple CNN
ResNet Block vs Simple CNN
What the residual connection costs and what it buys.
Simple CNN has 731K more parameters than ResNet Block: 4 layers added, 4 removed, 2 changed.
ResNet Block
- Layers
- 7
- Parameters
- 74K
- Input
- 64 × 32 × 32
- Output
- 64 × 32 × 32
- Forward-passes
- yes
- Est. train cost
- $0.045
Simple CNN
- Layers
- 7
- Parameters
- 805K
- Input
- 1 × 28 × 28
- Output
- 10
- Forward-passes
- yes
- Est. train cost
- $0.042
The deltas
Every number is Simple CNN relative to ResNet Block.
Which GPUs each one fits
Each side is measured at its own declared input (64 × 32 × 32 against 1 × 28 × 28). Both columns are right about their own model; the difference between them is not a fact about the designs.
| GPU | ResNet Block | Simple CNN |
|---|---|---|
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |
Layer by layer
Aligned in topological order. 3 of 13 rows are the same layer with the same parameters.
Hide all 13 rows
| ResNet Block | Simple CNN | ||||||
|---|---|---|---|---|---|---|---|
| Layer | Params | Output | Layer | Params | Output | ||
| 1 | changed shape | Input Input | 64 × 32 × 32 | Input Input | 1 × 28 × 28 | ||
| 2 | changed outChannels | Conv2D_1 Conv2d | 640 | 64 × 32 × 32 | Conv2D_1 Conv2d | 320 | 32 × 28 × 28 |
| 3 | removed | BatchNorm_1 Batch Norm | 64 × 32 × 32 | — | |||
| 4 | same | ReLU_1 Relu | 64 × 32 × 32 | ReLU_1 Relu | 32 × 28 × 28 | ||
| 5 | removed | Conv2D_2 Conv2d | 640 | 64 × 32 × 32 | — | ||
| 6 | removed | BatchNorm_2 Batch Norm | 64 × 32 × 32 | — | |||
| 7 | removed | Add Add | 64 × 32 × 32 | — | |||
| 8 | added | — | MaxPool2D_1 Maxpool2d | 32 × 14 × 14 | |||
| 9 | added | — | Flatten Flatten | 6272 | |||
| 10 | added | — | Linear_1 Linear | 128 | |||
| 11 | same | ReLU_2 Relu | 64 × 32 × 32 | ReLU_2 Relu | 128 | ||
| 12 | added | — | Linear_2 Linear | 10 | |||
| 13 | same | Output Output | 64 × 32 × 32 | Output Output | 10 | ||
What this is not
- The two are priced at different declared inputs (64 × 32 × 32 against 1 × 28 × 28), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.
Take it further
Open either graph in the editor, change it, and check it again: ResNet Block · Simple CNN
Compare any two models of your own, including anything on Hugging Face: the comparison tool.
Machine-readable: this page as markdown ·
the pair index ·
POST https://www.neurarch.com/api/v1/plan for a graph of your own.