🪟 Swin-Tiny

Hierarchical vision transformer — shifted-window attention builds a feature pyramid for dense prediction (Liu 2021)

From Liu et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV 2021. This page is the graph, not the PDF: open it, edit it, verify it, export it.

Layers

Parameters

28.26M

Input

3 × 224 × 224

Output

1000

Verifier

1 advisory

Every number on this page is computed from the graph by the same functions the app runs, not written by hand.

Open Swin-Tiny on the canvas Free, no account needed

When to pick it

Pick when you need a ViT that produces multi-scale features (detection, segmentation) rather than a single-scale ViT. Windowed attention keeps compute linear in image size.

Structure

85 layers. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Parameters	Output shape
1	image	Input	shape=[3, 224, 224]	3 × 224 × 224
2	patch_embed_4x4	Patch Embed	embedDim=96, patchSize=4	3136 × 96
3	s1b1_norm1	LayerNorm	normalizedShape=96	3136 × 96
4	s1b1_window_attn	Multi-Head Attention	embedDim=96, numHeads=3	3136 × 96
5	s1b1_res1	Add		3136 × 96
6	s1b1_norm2	LayerNorm	normalizedShape=96	3136 × 96
7	s1b1_mlp	Feed Forward	embedDim=96, ffDim=384	3136 × 96
8	s1b1_res2	Add		3136 × 96
9	s1b2_norm1	LayerNorm	normalizedShape=96	3136 × 96
10	s1b2_shifted_window_attn	Multi-Head Attention	embedDim=96, numHeads=3	3136 × 96
11	s1b2_res1	Add		3136 × 96
12	s1b2_norm2	LayerNorm	normalizedShape=96	3136 × 96
13	s1b2_mlp	Feed Forward	embedDim=96, ffDim=384	3136 × 96
14	s1b2_res2	Add		3136 × 96
15	merge_patches_n15	Reshape	shape=[784, 384]	784 × 384
16	patch_merging_2	Linear	outFeatures=192, inFeatures=384	784 × 192
17	s2b1_norm1	LayerNorm	normalizedShape=192	784 × 192
18	s2b1_window_attn	Multi-Head Attention	embedDim=192, numHeads=6	784 × 192
19	s2b1_res1	Add		784 × 192
20	s2b1_norm2	LayerNorm	normalizedShape=192	784 × 192
21	s2b1_mlp	Feed Forward	embedDim=192, ffDim=768	784 × 192
22	s2b1_res2	Add		784 × 192
23	s2b2_norm1	LayerNorm	normalizedShape=192	784 × 192
24	s2b2_shifted_window_attn	Multi-Head Attention	embedDim=192, numHeads=6	784 × 192
25	s2b2_res1	Add		784 × 192
26	s2b2_norm2	LayerNorm	normalizedShape=192	784 × 192
27	s2b2_mlp	Feed Forward	embedDim=192, ffDim=768	784 × 192
28	s2b2_res2	Add		784 × 192
29	merge_patches_n28	Reshape	shape=[196, 768]	196 × 768
30	patch_merging_3	Linear	outFeatures=384, inFeatures=768	196 × 384
31	s3b1_norm1	LayerNorm	normalizedShape=384	196 × 384
32	s3b1_window_attn	Multi-Head Attention	embedDim=384, numHeads=12	196 × 384
33	s3b1_res1	Add		196 × 384
34	s3b1_norm2	LayerNorm	normalizedShape=384	196 × 384
35	s3b1_mlp	Feed Forward	embedDim=384, ffDim=1536	196 × 384
36	s3b1_res2	Add		196 × 384
37	s3b2_norm1	LayerNorm	normalizedShape=384	196 × 384
38	s3b2_shifted_window_attn	Multi-Head Attention	embedDim=384, numHeads=12	196 × 384
39	s3b2_res1	Add		196 × 384
40	s3b2_norm2	LayerNorm	normalizedShape=384	196 × 384
41	s3b2_mlp	Feed Forward	embedDim=384, ffDim=1536	196 × 384
42	s3b2_res2	Add		196 × 384
43	s3b3_norm1	LayerNorm	normalizedShape=384	196 × 384
44	s3b3_window_attn	Multi-Head Attention	embedDim=384, numHeads=12	196 × 384
45	s3b3_res1	Add		196 × 384
46	s3b3_norm2	LayerNorm	normalizedShape=384	196 × 384
47	s3b3_mlp	Feed Forward	embedDim=384, ffDim=1536	196 × 384
48	s3b3_res2	Add		196 × 384
49	s3b4_norm1	LayerNorm	normalizedShape=384	196 × 384
50	s3b4_shifted_window_attn	Multi-Head Attention	embedDim=384, numHeads=12	196 × 384
51	s3b4_res1	Add		196 × 384
52	s3b4_norm2	LayerNorm	normalizedShape=384	196 × 384
53	s3b4_mlp	Feed Forward	embedDim=384, ffDim=1536	196 × 384
54	s3b4_res2	Add		196 × 384
55	s3b5_norm1	LayerNorm	normalizedShape=384	196 × 384
56	s3b5_window_attn	Multi-Head Attention	embedDim=384, numHeads=12	196 × 384
57	s3b5_res1	Add		196 × 384
58	s3b5_norm2	LayerNorm	normalizedShape=384	196 × 384
59	s3b5_mlp	Feed Forward	embedDim=384, ffDim=1536	196 × 384
60	s3b5_res2	Add		196 × 384
61	s3b6_norm1	LayerNorm	normalizedShape=384	196 × 384
62	s3b6_shifted_window_attn	Multi-Head Attention	embedDim=384, numHeads=12	196 × 384
63	s3b6_res1	Add		196 × 384
64	s3b6_norm2	LayerNorm	normalizedShape=384	196 × 384
65	s3b6_mlp	Feed Forward	embedDim=384, ffDim=1536	196 × 384
66	s3b6_res2	Add		196 × 384
67	merge_patches_n65	Reshape	shape=[49, 1536]	49 × 1536
68	patch_merging_4	Linear	outFeatures=768, inFeatures=1536	49 × 768
69	s4b1_norm1	LayerNorm	normalizedShape=768	49 × 768
70	s4b1_window_attn	Multi-Head Attention	embedDim=768, numHeads=24	49 × 768
71	s4b1_res1	Add		49 × 768
72	s4b1_norm2	LayerNorm	normalizedShape=768	49 × 768
73	s4b1_mlp	Feed Forward	embedDim=768, ffDim=3072	49 × 768
74	s4b1_res2	Add		49 × 768
75	s4b2_norm1	LayerNorm	normalizedShape=768	49 × 768
76	s4b2_shifted_window_attn	Multi-Head Attention	embedDim=768, numHeads=24	49 × 768
77	s4b2_res1	Add		49 × 768
78	s4b2_norm2	LayerNorm	normalizedShape=768	49 × 768
79	s4b2_mlp	Feed Forward	embedDim=768, ffDim=3072	49 × 768
80	s4b2_res2	Add		49 × 768
81	final_norm	LayerNorm	normalizedShape=768	49 × 768
82	to_channels	Permute		768 × 49
83	avgpool	GlobalAvgPool1D		768
84	classifier	Linear	outFeatures=1000, inFeatures=768	1000
85	class_logits	Output		1000

What the verifier says

The same 43 single-graph checks that run on every edit in the app, on this graph.

warn12 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer. (s1b1_window_attn)
attention-no-pe

infoAt 12 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers)) (s1b1_window_attn)
deep-attention-default-init

The PyTorch it exports

Generated from the graph above. First 46 lines; the app exports the whole file, plus the training loop, the data contract and a deploy bundle.

# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class SwinTransformerTiny(nn.Module):
    def __init__(self):
        super().__init__()

        self.patchEmbed_1 = nn.Conv2d(3, 96, kernel_size=4, stride=4)  # Patch embedding (ViT-style)
        self.layerNorm_1 = nn.LayerNorm(96)
        self.multiHeadAttention_1 = nn.MultiheadAttention(embed_dim=96, num_heads=3, batch_first=True)
        self.layerNorm_2 = nn.LayerNorm(96)
        self.feedForward_1 = nn.Sequential(
            nn.Linear(96, 384),
            nn.ReLU(),
            nn.Linear(384, 96)
        )
        self.layerNorm_3 = nn.LayerNorm(96)
        self.multiHeadAttention_2 = nn.MultiheadAttention(embed_dim=96, num_heads=3, batch_first=True)
        self.layerNorm_4 = nn.LayerNorm(96)
        self.feedForward_2 = nn.Sequential(
            nn.Linear(96, 384),
            nn.ReLU(),
            nn.Linear(384, 96)
        )
        self.linear_1 = nn.Linear(384, 192)
        self.layerNorm_5 = nn.LayerNorm(192)
        self.multiHeadAttention_3 = nn.MultiheadAttention(embed_dim=192, num_heads=6, batch_first=True)
        self.layerNorm_6 = nn.LayerNorm(192)
        self.feedForward_3 = nn.Sequential(
            nn.Linear(192, 768),
            nn.ReLU(),
            nn.Linear(768, 192)
        )
        self.layerNorm_7 = nn.LayerNorm(192)
        self.multiHeadAttention_4 = nn.MultiheadAttention(embed_dim=192, num_heads=6, batch_first=True)
        self.layerNorm_8 = nn.LayerNorm(192)
        self.feedForward_4 = nn.Sequential(
            nn.Linear(192, 768),
            nn.ReLU(),
            nn.Linear(768, 192)

For agents

This architecture is machine-readable end to end. An agent can list the set, fetch this graph, edit it, and have the edit verified before any GPU time is spent.

Markdown twin Index JSON This graph as JSON MCP server POST /api/v1/check

Also in Computer Vision

🖼️ Simple CNN

Simple Convolutional Neural Network for image classification

9 layers · 804.6K

🔗 ResNet Block

ResNet residual block with skip connections

9 layers · 74.0K

🩻 U-Net

Encoder-decoder with skip connections — Ronneberger et al

24 layers · 720.7K

👁️ ViT-B/16

Vision Transformer — patch embedding stem + 1 encoder block

13 layers · 8.45M