Checks / pattern
Default init assumes ReLU, saturating activation follows
Check R32. Runs in the editor as you build, in CI through the
GitHub Action, and over the wire at
POST /api/v1/check. Milliseconds, before any GPU is billed.
info
pattern
R32
| Trigger | A Linear/Conv layer feeds a sigmoid or tanh activation directly. |
|---|---|
| Why | PyTorch's default Kaiming (He) init is derived for ReLU-family activations. Feeding a saturating activation from a He-initialized layer starts training in the saturated tails and shrinks early gradients. Use Xavier init with the matching gain, or a ReLU-family activation. |
| Source | Glorot & Bengio 2010 (Xavier) vs He et al. 2015 (Kaiming) derivation assumptions. |
Why it is not a lint you can ignore
A structural mistake does not fail at review time and it does not fail at import
time. It fails when the module is constructed on the training node, after the job was queued and
the dataset was downloaded. That is why this runs before the spend and not after it.
Run this check on your own model
Free, no account needed
Every check
41 structural checks: 6 guardrail gates and 35 architecture advisor rules. See the full catalogue.
← R31 Full multi-head attention at LLM scale ยท R33 Deep attention stack without depth-scaled init →