Architectures / NLP/LLM

🔁 T5 Small

T5 encoder-decoder — bidirectional encoder + masked decoder with cross-attention (512D, 8 heads)

Layers

Parameters

56.73M

Input

1 × 512

Output

1 × 128 × 32128

Verifier

1 advisory

Every number on this page is computed from the graph by the same functions the app runs, not written by hand.

Open T5 Small on the canvas Free, no account needed

When to pick it

Pick for seq2seq tasks (summarization, translation, QA) where you need both bidirectional understanding and generation in one model.

Structure

23 layers. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Parameters	Output shape
1	encoder_ids	Input	shape=[1, 512]	1 × 512
2	shared_embed	Embedding		1 × 512 × 512
3	enc_norm	RMSNorm	normalizedShape=512	1 × 512 × 512
4	enc_self_attn	Multi-Head Attention	embedDim=512, numHeads=8	1 × 512 × 512
5	enc_residual	Add		1 × 512 × 512
6	enc_ffn_norm	RMSNorm	normalizedShape=512	1 × 512 × 512
7	enc_ffn	Feed Forward	embedDim=512, ffDim=2048	1 × 512 × 512
8	enc_ffn_residual	Add		1 × 512 × 512
9	enc_out_norm	LayerNorm	normalizedShape=512	1 × 512 × 512
10	decoder_ids	Input	shape=[1, 128]	1 × 128
11	dec_embed	Embedding		1 × 128 × 512
12	dec_sa_norm	RMSNorm	normalizedShape=512	1 × 128 × 512
13	dec_self_attn	Causal Attention	embedDim=512, numHeads=8	1 × 128 × 512
14	dec_sa_residual	Add		1 × 128 × 512
15	dec_ca_norm	RMSNorm	normalizedShape=512	1 × 128 × 512
16	cross_attn	Multi-Head Attention	embedDim=512, numHeads=8	1 × 128 × 512
17	dec_ca_residual	Add		1 × 128 × 512
18	dec_ffn_norm	RMSNorm	normalizedShape=512	1 × 128 × 512
19	dec_ffn	Feed Forward	embedDim=512, ffDim=2048	1 × 128 × 512
20	dec_ffn_residual	Add		1 × 128 × 512
21	dec_out_norm	LayerNorm	normalizedShape=512	1 × 128 × 512
22	lm_head	Linear	outFeatures=32128	1 × 128 × 32128
23	logits	Output		1 × 128 × 32128

What the verifier says

The same 41 structural checks that run on every edit in the app, on this graph.

warn3 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer. (enc_self_attn)
attention-no-pe

The PyTorch it exports

Generated from the graph above. First 46 lines; the app exports the whole file, plus the training loop, the data contract and a deploy bundle.

# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class T5Small(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(32128, 512)
        self.rmsNorm_1 = nn.RMSNorm(512)
        self.multiHeadAttention_1 = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
        self.rmsNorm_2 = nn.RMSNorm(512)
        self.feedForward_1 = nn.Sequential(
            nn.Linear(512, 2048),
            nn.ReLU(),
            nn.Linear(2048, 512)
        )
        self.layerNorm_1 = nn.LayerNorm(512)
        self.embedding_2 = nn.Embedding(32128, 512)
        self.rmsNorm_3 = nn.RMSNorm(512)
        self.causalAttention_1 = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
        self.rmsNorm_4 = nn.RMSNorm(512)
        self.multiHeadAttention_2 = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
        self.rmsNorm_5 = nn.RMSNorm(512)
        self.feedForward_2 = nn.Sequential(
            nn.Linear(512, 2048),
            nn.ReLU(),
            nn.Linear(2048, 512)
        )
        self.layerNorm_2 = nn.LayerNorm(512)
        self.linear_1 = nn.Linear(512, 32128)

    def forward(self, src, tgt=None):
        # encoder_ids shape: [1,512]
        # decoder_ids shape: [1,128]
        embedding_ng_enc = self.embedding_1(src)
        rms_norm_rm_enc = self.rmsNorm_1(embedding_ng_enc)
        multi_head_attention_ha_enc = self.multiHeadAttention_1(rms_norm_rm_enc, rms_norm_rm_enc, rms_norm_rm_enc)[0]
        add_dd_enc = multi_head_attention_ha_enc
        rms_norm_m_enc2 = self.rmsNorm_2(add_dd_enc)
        feed_forward_rd_enc = self.feedForward_1(rms_norm_m_enc2)

For agents

This architecture is machine-readable end to end. An agent can list the set, fetch this graph, edit it, and have the edit verified before any GPU time is spent.

Markdown twin Index JSON This graph as JSON MCP server POST /api/v1/check

Also in NLP/LLM

🤖 Transformer Block

Transformer encoder block

8 layers · 7.09M

📖 BERT Base

BERT-Base encoder — bidirectional MHA

11 layers · 31.12M

🧠 GPT-2

GPT-2 Small — causal transformer block

12 layers · 84.33M

🦙 LLaMA-3 Block

LLaMA-3 decoder block — GQA

10 layers · 702.55M