Source: Algorithm Advancement This article is approximately 4100 words and is recommended to be read in 8 minutes. LongLLaVA performs excellently in long-context multimodal understanding. The authors of this article come from The Chinese University of Hong Kong, Shenzhen, and the Shenzhen Big Data Research Institute. The first authors are PhD student Wang Xidong and … Read more
Follow our public account to discover the beauty of CV technology This article is reprinted from Machine Heart. The key to the tremendous success of the Transformer in deep learning is the attention mechanism. The attention mechanism allows Transformer-based models to focus on parts of the input sequence that are relevant, achieving better contextual understanding. … Read more
MLNLP community is a well-known machine learning and natural language processing community in China and abroad, covering NLP master’s and doctoral students, university teachers, and researchers from enterprises. The Community’s Vision is to promote communication and progress between the academic and industrial circles of natural language processing and machine learning at home and abroad, especially … Read more
Follow the public account to discover the beauty of CV technology This article is reprinted from Machine Heart, edited by Panda W. Transformers are powerful but not perfect, especially when dealing with long sequences. State Space Models (SSMs) perform quite well on long sequences. Researchers proposed last year that SSMs could replace Transformers, as seen … Read more
The Transformer architecture has become a major component of the success of large language models (LLMs). To further improve LLMs, new architectures that may outperform the Transformer architecture are being developed. One such approach is Mamba, a state space model. The paper “Mamba: Linear-Time Sequence Modeling with Selective State Spaces” introduces Mamba, which we have … Read more
Big Data Digest Authorized Reprint from Leading Technology Author丨Congerry Transformer Challenged! In June 2017, eight Google researchers published a groundbreaking paper titled “Attention is All You Need”. It is called groundbreaking because this paper proposed a new neural network architecture – the Transformer, which opened a new era of generative artificial intelligence and large models. … Read more
Feng Se from Aofeisi Quantum Bit | Public Account QbitAI Exciting news! The first project to truly scale the popular Mamba architecture to a sufficiently large size has arrived. 52 billion parameters, still using the Mamba+Transformer hybrid architecture. Its name is Jamba. By taking the strengths of both architectures, it achieves both model quality and … Read more
New Intelligence Report Editor: Editorial Department [New Intelligence Guide] The production-grade Mamba model with 52B parameters is here! This powerful variant, Jamba, has just broken the world record, capable of directly competing with Transformers, featuring a 256K ultra-long context window and a threefold throughput increase, with weights available for free download. The Mamba architecture, which … Read more
Machine Heart reports Editor: Rome Rome Video understanding faces immense challenges due to significant spatiotemporal redundancy and complex spatiotemporal dependencies. Overcoming these two issues is extremely difficult, and CNNs, Transformers, and Uniformers struggle to meet these demands. Mamba presents a promising approach; let’s explore how this article creates video understanding with VideoMamba. The core goal … Read more