Models / distilbert
distilbert-base-uncased
Reconstructed from its own config.json
with no weights read. 7.1M downloads on Hugging Face.
Our count against the checkpoint
The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.
What it costs to run
Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.
| Card | Memory | |
|---|---|---|
| T4 (16GB) | weights + activations | fits |
| A100 (40GB) | weights + activations | fits |
| H100 (80GB) | weights + activations | fits |
Structure
28 nodes. Output shapes are propagated from the input shape, batch dimension excluded.
| Layer | Type | Output shape | |
|---|---|---|---|
| 1 | Input | Input | 1 × 512 |
| 2 | Embedding | Embedding | 1 × 512 × 768 |
| 3 | Positional_Embedding | Learned Pos Embed | 1 × 512 × 768 |
| 4 | Attention_1 | Multi-Head Attention | 1 × 512 × 768 |
| 5 | Add_1 | Add | 1 × 512 × 768 |
| 6 | LayerNorm_1_1 | LayerNorm | 1 × 512 × 768 |
| 7 | FFN_1 | Feed Forward | 1 × 512 × 768 |
| 8 | Attention_2 | Multi-Head Attention | 1 × 512 × 768 |
| 9 | Add_2 | Add | 1 × 512 × 768 |
| 10 | LayerNorm_2_1 | LayerNorm | 1 × 512 × 768 |
| 11 | FFN_2 | Feed Forward | 1 × 512 × 768 |
| 12 | Attention_3 | Multi-Head Attention | 1 × 512 × 768 |
| 13 | Add_3 | Add | 1 × 512 × 768 |
| 14 | LayerNorm_3_1 | LayerNorm | 1 × 512 × 768 |
| 15 | FFN_3 | Feed Forward | 1 × 512 × 768 |
| 16 | Attention_4 | Multi-Head Attention | 1 × 512 × 768 |
| 17 | Add_4 | Add | 1 × 512 × 768 |
| 18 | LayerNorm_4_1 | LayerNorm | 1 × 512 × 768 |
| 19 | FFN_4 | Feed Forward | 1 × 512 × 768 |
| 20 | Attention_5 | Multi-Head Attention | 1 × 512 × 768 |
| 21 | Add_5 | Add | 1 × 512 × 768 |
| 22 | LayerNorm_5_1 | LayerNorm | 1 × 512 × 768 |
| 23 | FFN_5 | Feed Forward | 1 × 512 × 768 |
| 24 | Attention_6 | Multi-Head Attention | 1 × 512 × 768 |
| 25 | Add_6 | Add | 1 × 512 × 768 |
| 26 | LayerNorm_6_1 | LayerNorm | 1 × 512 × 768 |
| 27 | FFN_6 | Feed Forward | 1 × 512 × 768 |
| 28 | Output | Output | 1 × 512 × 768 |
What the verifier says
No finding on the reconstructed graph. See the checks.
Do this to your own model
Same numbers, on a model in your repo, in one command. No account.
pip install neurarch-trace
neurarch-trace distilbert/distilbert-base-uncased --plan --share