Recent research highlights the emergence of credible alternatives to vanilla Transformers in protein language models. Convolutional neural networks (CNNs) are shown to perform on par with transformers in building such models. DeProt, a Transformer-based protein language model, integrates protein sequences and structures effectively.
DeProt (Disentangled Protein sequence-structure model), a Transformer-based protein language model designed to incorporate protein sequences. DeProt can quantize protein structures to mitigate overfitting and is adeptly engineered to amalgamate sequence and structure tokens.
DeProt: A protein language model with quantizied structure and disentangled attention https://t.co/Mp4dximqwo https://t.co/XMBU2vLTR2
DeProt: A protein language model with quantizied structure and disentangled attention https://t.co/ovTq1a8XLx #biorxiv_bioinfo
Research Highlight by Lin Tang: Transformers dominate the architecture of protein language models, but are they the only choice? This study demonstrates that convolutional neural networks (CNNs) perform on par with transformers for building such models. https://t.co/KMlHMu5USI
I wrote an in depth research piece on how for the first time since their invention, vanilla Transformers have credible alternatives in linear RNNs, SSMs and sparse attention They're competitive in accuracy w higher throughput and/or longer context https://t.co/ZVScve33sC https://t.co/y40ebHiN0Y