Transformer News

Google DeepMind Introduces TransNAR Hybrid Architecture with Improvements for Algorithmic Reasoning
Authors
6
17 days
AI
Tech

Is attention all you need? (transformers SOTA in 2027)

Nov 16, 7:02 PMJan 31, 11:30 PM

61.5%chance

12021853

OptionVotes

YES

4037

3229

Will superposition in transformers be mostly solved by 2026?

Mar 16, 7:31 AMDec 31, 11:59 PM

63.63%chance

7915992

OptionVotes

YES

3021

1249

By EOY 2025, will the model with the lowest perplexity on Common Crawl will not be based on transformers?

Jan 5, 4:19 AMDec 31, 11:59 PM

25.2%chance

313460

OptionVotes

YES

1243

877

Will a big transformer LM compose these facts without chain of thought by 2026?

Jan 19, 5:08 AMJan 1, 11:59 PM

85.72%chance

6132248

OptionVotes

YES

3180

1513

Will Transformer based architectures still be SOTA for language modelling by 2026?

Oct 27, 3:05 AMJan 2, 7:59 AM

73.77%chance

493219

OptionVotes

YES

2015

1196

In 2025, will I teach people that Projective Geometric Algebra (PGA)'s "dual" is similar to the Fourier transform?

Sep 20, 7:52 AMSep 21, 10:59 PM

48%chance

191284

OptionVotes

YES

1530

452

Are Mixture of Expert (MoE) transformer models generally more human interpretable than dense transformers?

Sep 15, 11:23 AMDec 31, 6:29 PM

52.11%chance

131161

OptionVotes

YES

994

991

If OpenAI makes a transformer sized advancement in the next 5 years, will they publish an accompanying paper?

Nov 29, 12:12 AMDec 6, 7:59 AM

43.88%chance

18788

OptionVotes

YES

1091

915

6) An alternative to the transformer architecture will see meaningful adoption.

Dec 25, 1:35 PMDec 31, 10:59 PM

62.96%chance

533275

OptionVotes

YES

1052

852

[Metaculus] Will an International AI regulatory agency for oversight of transformative AI be established before 2030?

Oct 16, 1:24 PMJan 2, 8:00 PM

67.77%chance

10360

OptionVotes

YES

1057

963

LLMs by EOY 2025: Will Retentive Learning Surpass Transformers? (Subsidised 400 M$)

Sep 15, 7:21 PMDec 31, 10:59 PM

14.29%chance

343409

OptionVotes

YES

1948

826

Will the transformer architecture be replaced in SOTA LLMs by 2028?

Dec 26, 8:37 PMJan 1, 10:59 PM

63.45%chance

12241

OptionVotes

YES

1003

988

Latest stories

Google DeepMind Introduces TransNAR Hybrid Architecture with Improvements for Algorithmic Reasoning
Authors
6
17 days
AI
Tech
Hybrid State Space Models Samba 3.8B, Mamba-2-Hybrid Outperform Transformers
Authors
10
17 days
AI
Tech
NVIDIA's 8B-Parameter Hybrid Models Outperform Transformers, Achieving 2.65 Points Improvement
Authors
5
17 days
AI
Tech
New Research Shows 10x-50x More Parameter-Efficient Fine-Tuning of Large Language Models
Authors
4
26 days
AI
Tech
DeProt Emerges as Transformer Alternative in Protein Language Models
Authors
4
2 months
Bio
Science
Google's 2024 Breakthrough: 'Infini-attention' for LLMs
Authors
10
3 months
AI
Tech
Eagle and Finch Models Introduce Advanced Architectural Design with Multi-Headed Matrix-Valued States for 7.5b and 3.1b Models
Authors
6
3 months
AI
Tech
AI21 Labs Launches Jamba, Outperforming Similar Models with Improved Performance and Efficiency on Long Context Tasks
Authors
15
3 months
AI
Tech
Nature Methods Publishes Multi-Modality Cell Segmentation Challenge Results with Transformer-based Model and AI Applications
Authors
4
3 months
AI
Bio
Tech
VideoMamba: Revolutionizing Video Understanding with State Space Model
Authors
4
3 months
AI
Tech
BurstAttention: DMC Improves Speed and Performance with KV Cache for LLaMa 2
Authors
6
4 months
AI
Tech
AtP* and Based Models Revolutionize AI with 24x Higher Throughput
Authors
5
4 months
AI
Tech
Meta's 'Beyond A*': Transformers Achieve 94% Optimal Plans with 27% Less Search
Authors
5
4 months
AI
Tech
Efficient 'MoE-Mamba' Outperforms Mamba and Transformer Models
Authors
6
6 months
AI
Tech
Mamba AI Model in 300 Lines Rivals Transformers; Gains Benchmarking Support, Learning Resources
Authors
5
6 months
AI
Tech
Razoralign Unveils LncLocFormer, ENTRAIN, and Escort for Enhanced RNA-seq Analysis
Authors
5
6 months
Bio
Science
Scientists Merge Human Brain Tissue with Circuits for More Powerful Computer, AI's Memory Mechanism Similar to Brain, Handles 228 Trillion Operations/Sec
Authors
6
6 months
AI
Tech
Mistral MoE Outperforms GPT 3.5 in Efficient Model Evaluations, Mixtral-8x7B Surpasses Llama-2-70B
Authors
6
7 months
AI
Tech
Mistral AI Releases Mixtral-8x7B Model, Outperforms Existing Language Models with Performance Metrics and Demos, Sparks GPT-4 Class Model Discussions
Authors
18
7 months
AI
Tech
Stanford, Microsoft, Google Publish MegaBlocks for Efficient Sparse Training with MoE
Authors
6
7 months
AI
Tech
Mamba, a Sequence Model with Fast Inference, 5× Higher Throughput, and Linear Scaling, Challenges Transformers
Authors
6
7 months
AI
Tech
New Model Mamba Outperforms in Performance, Language Modeling, and Fast Inference
Authors
4
7 months
AI
Tech
MIT Develops Soft Robot Inspired by Sea Turtles That Can Burrow Through Sand; Japanese Tech Company Rolls Out Transformer-like Factory Robots
Authors
4
8 months
China
AI
World

Transformer News

Top stories

Prediction markets for Transformer

Prediction markets for Transformer

Prediction markets for Transformer

Is attention all you need? (transformers SOTA in 2027)

Will superposition in transformers be mostly solved by 2026?

By EOY 2025, will the model with the lowest perplexity on Common Crawl will not be based on transformers?

Will a big transformer LM compose these facts without chain of thought by 2026?

Will Transformer based architectures still be SOTA for language modelling by 2026?

In 2025, will I teach people that Projective Geometric Algebra (PGA)'s "dual" is similar to the Fourier transform?

Are Mixture of Expert (MoE) transformer models generally more human interpretable than dense transformers?

If OpenAI makes a transformer sized advancement in the next 5 years, will they publish an accompanying paper?

6) An alternative to the transformer architecture will see meaningful adoption.

[Metaculus] Will an International AI regulatory agency for oversight of transformative AI be established before 2030?

LLMs by EOY 2025: Will Retentive Learning Surpass Transformers? (Subsidised 400 M$)

Will the transformer architecture be replaced in SOTA LLMs by 2028?

Latest stories

Top stories

Prediction markets for Transformer

Prediction markets for Transformer

Prediction markets for Transformer

Is attention all you need? (transformers SOTA in 2027)

Will superposition in transformers be mostly solved by 2026?

By EOY 2025, will the model with the lowest perplexity on Common Crawl will not be based on transformers?

Will a big transformer LM compose these facts without chain of thought by 2026?

Will Transformer based architectures still be SOTA for language modelling by 2026?

In 2025, will I teach people that Projective Geometric Algebra (PGA)'s "dual" is similar to the Fourier transform?

Are Mixture of Expert (MoE) transformer models generally more human interpretable than dense transformers?