Architectures / Multimodal

🔗 CLIP ViT-B/32

Dual-encoder contrastive model — a ViT image tower and a Transformer text tower projected into a shared embedding space (OpenAI 2021)

Every number on this page is computed from the graph by the same functions the app runs, not written by hand.

When to pick it

Structure

38 layers. Output shapes are propagated from the input shape, batch dimension excluded.

What the verifier says

The same 41 structural checks that run on every edit in the app, on this graph.

The PyTorch it exports

Generated from the graph above. First 46 lines; the app exports the whole file, plus the training loop, the data contract and a deploy bundle.

# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)
#
# WARNING: 3 layer(s) below are not yet supported by the PyTorch
# exporter and pass their input through UNCHANGED in forward():
#   - vis_pool (globalAvgPool1d)
#   - txt_pool (globalAvgPool1d)
#   - similarity (matmul)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class CLIPViT_B32(nn.Module):
    def __init__(self):
        super().__init__()

        self.patchEmbed_1 = nn.Conv2d(3, 768, kernel_size=32, stride=32)  # Patch embedding (ViT-style)
        self.transformerBlock_1 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_2 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_3 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_4 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_5 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_6 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_7 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_8 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_9 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_10 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_11 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.transformerBlock_12 = nn.TransformerEncoderLayer(d_model=768, nhead=12, dim_feedforward=3072, batch_first=True)
        self.layerNorm_1 = nn.LayerNorm(768)
        self.linear_1 = nn.Linear(768, 512)
        self.embedding_1 = nn.Embedding(49408, 512)
        self.transformerBlock_13 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_14 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_15 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_16 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_17 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_18 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_19 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_20 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_21 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_22 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
        self.transformerBlock_23 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)

For agents

Also in Multimodal