Comparisons / DLRM vs Wide & Deep
DLRM vs Wide & Deep
Two production ranking models with different interaction layers.
Wide & Deep has 61M fewer parameters than DLRM: 3 layers added, 4 removed, 6 changed.
Wide & Deep
- Layers
- 10
- Parameters
- 3.7M
- Input
- 10000
- Output
- 1
- Forward-passes
- yes
- Est. train cost
- $0.043
The deltas
Every number is Wide & Deep relative to DLRM.
Which GPUs each one fits
Each side is measured at its own declared input (13 against 10000). Both columns are right about their own model; the difference between them is not a fact about the designs.
| GPU | DLRM | Wide & Deep |
|---|---|---|
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |
Layer by layer
Aligned in topological order. 4 of 17 rows are the same layer with the same parameters.
Hide all 17 rows
| DLRM | Wide & Deep | ||||||
|---|---|---|---|---|---|---|---|
| Layer | Params | Output | Layer | Params | Output | ||
| 1 | changed shape | Dense Features Input | 13 | Wide Input (cross feats) Input | 10000 | ||
| 2 | changed shape | Sparse Features Input | 26 | Deep Input (sparse cat) Input | 50 | ||
| 3 | removed | Bottom MLP 1 Linear | 896 | 64 | — | ||
| 4 | removed | EmbeddingBag Embedding Bag | 64M | 26 × 64 | — | ||
| 5 | removed | ReLU Relu | 64 | — | |||
| 6 | changed inFeatures, outFeatures | Bottom MLP 2 Linear | 4.2K | 64 | Wide Linear Linear | 10K | 1 |
| 7 | removed | Feature Interaction Feature Interaction | 415 | — | |||
| 8 | added | — | Embeddings Embedding | 3.2M | 50 × 32 | ||
| 9 | added | — | Flatten Flatten | 1600 | |||
| 10 | changed inFeatures, outFeatures | Top MLP 1 Linear | 213K | 512 | Deep FC 1 Linear | 410K | 256 |
| 11 | same | ReLU Relu | 512 | ReLU 1 Relu | 256 | ||
| 12 | changed inFeatures, outFeatures | Top MLP 2 Linear | 131K | 256 | Deep FC 2 Linear | 33K | 128 |
| 13 | same | ReLU Relu | 256 | ReLU 2 Relu | 128 | ||
| 14 | changed inFeatures | CTR Head Linear | 257 | 1 | Deep Out Linear | 129 | 1 |
| 15 | added | — | Wide + Deep Add | 1 | |||
| 16 | same | Sigmoid Sigmoid | 1 | Sigmoid CTR Sigmoid | 1 | ||
| 17 | same | P(click) Output | 1 | P(click) Output | 1 | ||
What this is not
- The two are priced at different declared inputs (13 against 10000), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.
Take it further
Open either graph in the editor, change it, and check it again: DLRM · Wide & Deep
Compare any two models of your own, including anything on Hugging Face: the comparison tool.
Machine-readable: this page as markdown ·
the pair index ·
POST https://www.neurarch.com/api/v1/plan for a graph of your own.