Architectures / NLP/LLM

🔄 Simple RNN

Simple Recurrent Neural Network for sequence processing

From Elman (1990). Finding Structure in Time. Cognitive Science. This page is the graph, not the PDF: open it, edit it, verify it, export it.

Every number on this page is computed from the graph by the same functions the app runs, not written by hand.

When to pick it

Structure

4 layers. Output shapes are propagated from the input shape, batch dimension excluded.

What the verifier says

The same 43 single-graph checks that run on every edit in the app, on this graph.

The PyTorch it exports

Generated from the graph above.

# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class SimpleRNN(nn.Module):
    def __init__(self):
        super().__init__()

        self.lstm_1 = nn.LSTM(300, 256, num_layers=2, batch_first=True)
        self.linear_1 = nn.Linear(256, 10)

    def forward(self, x):
        # Input shape: [128,300]
        lstm_lstm_1 = self.lstm_1(x)[0]
        linear_near_1 = self.linear_1(lstm_lstm_1)
        # Output
        return linear_near_1


if __name__ == '__main__':
    model = SimpleRNN()
    model.eval()

    x = torch.randn(1, 128, 300)  # (batch, channels, length)
    with torch.no_grad():
        output = model(x)

    print(f'Input  shape : {tuple(x.shape)}')
    print(f'Output shape : {tuple(output.shape)}')
    total = sum(p.numel() for p in model.parameters())
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print(f'Parameters   : {total:,} total, {trainable:,} trainable')

For agents

Also in NLP/LLM