Selective State Space Model

Comparison of Mamba, RNN, and Transformer Architectures

by
Comparison of Mamba, RNN, and Transformer Architectures

The Transformer architecture has become a major component of the success of large language models (LLMs). To further improve LLMs, new architectures that may outperform the Transformer architecture are being developed. One such approach is Mamba, a state space model. The paper “Mamba: Linear-Time Sequence Modeling with Selective State Spaces” introduces Mamba, which we have … Read more

New Architecture Surpasses Transformer? CMU and Princeton Launch with 5x Inference Speed Boost and Performance Optimization

by
New Architecture Surpasses Transformer? CMU and Princeton Launch with 5x Inference Speed Boost and Performance Optimization

Big Data Digest Authorized Reprint from Leading Technology Author丨Congerry Transformer Challenged! In June 2017, eight Google researchers published a groundbreaking paper titled “Attention is All You Need”. It is called groundbreaking because this paper proposed a new neural network architecture – the Transformer, which opened a new era of generative artificial intelligence and large models. … Read more