Models / llama
llama2-embedding-1b-8k
Reconstructed from its own config.json
with no weights read. 216K downloads on Hugging Face.
Our count against the checkpoint
The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.
custom-code This repository ships its own modeling code (`auto_map`, e.g. `modeling.py`), so `config.json` names a class in the repo rather than an architecture `transformers` defines. The graph below is what those config keys mean under `transformers` semantics, which is not necessarily what the repo's own file builds. A gap here is a statement about what we read, not about the checkpoint.
What it costs to run
Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.
| Card | Memory | |
|---|---|---|
| T4 (16GB) | weights + activations | does not fit |
| A100 (40GB) | weights + activations | fits |
| H100 (80GB) | weights + activations | fits |
Structure
30 nodes. Output shapes are propagated from the input shape, batch dimension excluded.
| Layer | Type | Output shape | |
|---|---|---|---|
| 1 | Input | Input | 1 × 32768 |
| 2 | Embedding | Embedding | 1 × 32768 × 4096 |
| 3 | RoPE | RoPE | 1 × 32768 × 4096 |
| 4 | RMSNorm_1_1 | RMSNorm | 1 × 32768 × 4096 |
| 5 | Attention_1 | Grouped Query Attn | 1 × 32768 × 4096 |
| 6 | Add_1_attn | Add | 1 × 32768 × 4096 |
| 7 | RMSNorm_1_2 | RMSNorm | 1 × 32768 × 4096 |
| 8 | FFN_1 | SwiGLU | 1 × 32768 × 4096 |
| 9 | Add_1_ffn | Add | 1 × 32768 × 4096 |
| 10 | RMSNorm_2_1 | RMSNorm | 1 × 32768 × 4096 |
| 11 | Attention_2 | Grouped Query Attn | 1 × 32768 × 4096 |
| 12 | Add_2_attn | Add | 1 × 32768 × 4096 |
| 13 | RMSNorm_2_2 | RMSNorm | 1 × 32768 × 4096 |
| 14 | FFN_2 | SwiGLU | 1 × 32768 × 4096 |
| 15 | Add_2_ffn | Add | 1 × 32768 × 4096 |
| 16 | RMSNorm_3_1 | RMSNorm | 1 × 32768 × 4096 |
| 17 | Attention_3 | Grouped Query Attn | 1 × 32768 × 4096 |
| 18 | Add_3_attn | Add | 1 × 32768 × 4096 |
| 19 | RMSNorm_3_2 | RMSNorm | 1 × 32768 × 4096 |
| 20 | FFN_3 | SwiGLU | 1 × 32768 × 4096 |
| 21 | Add_3_ffn | Add | 1 × 32768 × 4096 |
| 22 | RMSNorm_4_1 | RMSNorm | 1 × 32768 × 4096 |
| 23 | Attention_4 | Grouped Query Attn | 1 × 32768 × 4096 |
| 24 | Add_4_attn | Add | 1 × 32768 × 4096 |
| 25 | RMSNorm_4_2 | RMSNorm | 1 × 32768 × 4096 |
| 26 | FFN_4 | SwiGLU | 1 × 32768 × 4096 |
| 27 | Add_4_ffn | Add | 1 × 32768 × 4096 |
| 28 | Final_RMSNorm | RMSNorm | 1 × 32768 × 4096 |
| 29 | LM_Head | Linear | 1 × 32768 × 32000 |
| 30 | Output | Output | 1 × 32768 × 32000 |
What the verifier says
No finding on the reconstructed graph. See the checks.
Do this to your own model
Same numbers, on a model in your repo, in one command. No account.
pip install neurarch-trace
neurarch-trace mesolitica/llama2-embedding-1b-8k --plan --share