A newer version of this article is available. Read the latest version
5 posts • ChatGPT (GPT-4o mini)
Published
More breaking stories on DeepNewz — updated live.
Imagine a Transformer model without normalization. This is exactly what's proposed in a new paper from Meta, NYU, MIT, and Princeton. The authors found that normalization layers can be replaced with something called Dynamic Tanh (DyT). It looks like this: DyT(x)=γ ∗ tanh(αx)+β. https://t.co/J5O2gQoBLc
Transformers without Normalization https://t.co/ZzRWfPG9Cy
Transformers without Normalization "We introduce Dynamic Tanh (DyT), an element-wise operation DyT (x) = tanh(αx), as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces… https://t.co/dOl6au2Uxo
