Architectures / NLP/LLM

🐋 DeepSeek-V3

671B MoE LLM — Multi-head Latent Attention (MLA) compresses the KV cache; fine-grained MoE with shared + routed experts (DeepSeek 2024)

From DeepSeek-AI (2024). DeepSeek-V3 Technical Report. This page is the graph, not the PDF: open it, edit it, verify it, export it.

Every number on this page is computed from the graph by the same functions the app runs, not written by hand.

When to pick it

Structure

372 layers. Output shapes are propagated from the input shape, batch dimension excluded.

What the verifier says

The same 43 single-graph checks that run on every edit in the app, on this graph.

The PyTorch it exports

Generated from the graph above. First 46 lines; the app exports the whole file, plus the training loop, the data contract and a deploy bundle.

# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class SharedExpertMoE(nn.Module):
    """Shared-expert MoE: top-k routed experts plus always-on shared experts."""
    def __init__(self, embed_dim, num_experts, num_shared, expert_dim, top_k):
        super().__init__()
        def expert():
            return nn.ModuleDict({
                'gate_proj': nn.Linear(embed_dim, expert_dim, bias=False),
                'up_proj':   nn.Linear(embed_dim, expert_dim, bias=False),
                'down_proj': nn.Linear(expert_dim, embed_dim, bias=False),
            })
        self.router  = nn.Linear(embed_dim, num_experts, bias=False)
        self.experts = nn.ModuleList([expert() for _ in range(num_experts)])
        self.shared  = nn.ModuleList([expert() for _ in range(num_shared)])
        self.top_k   = top_k

    @staticmethod
    def _ffn(e, x):
        return e['down_proj'](F.silu(e['gate_proj'](x)) * e['up_proj'](x))

    def forward(self, x):
        scores = self.router(x).softmax(dim=-1)
        top_w, top_i = scores.topk(self.top_k, dim=-1)
        out = x.new_zeros(x.shape) + sum(self._ffn(s, x) for s in self.shared)
        flat_x, flat_o = x.reshape(-1, x.size(-1)), out.reshape(-1, x.size(-1))
        flat_i, flat_w = top_i.reshape(-1, self.top_k), top_w.reshape(-1, self.top_k)
        for e_idx in flat_i.unique():
            hit  = flat_i == e_idx
            rows = hit.any(dim=-1)
            w    = (flat_w * hit).sum(dim=-1)[rows].unsqueeze(-1)
            flat_o[rows] += w * self._ffn(self.experts[int(e_idx)], flat_x[rows])
        return flat_o.view_as(x)

class DeepSeek_V3(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(129280, 7168)

For agents

Also in NLP/LLM