Transformer

The Battle of Three Neural Network Structures: CNN, Transformer, and MLP

by
The Battle of Three Neural Network Structures: CNN, Transformer, and MLP

Click belowCard, follow the “CVer” public account AI/CV heavy content, delivered first-hand Author丨happy Reprinted from丨Extreme City Platform Introduction University of Science and Technology of China & MSRA analyzed the characteristics of three major neural network architectures, comparing CNN, Transformer, and MLP by constructing a unified architecture called SPACH, concluding that multi-stage models are always superior … Read more

New Architecture Introduced for Spiking Neural Networks

by
New Architecture Introduced for Spiking Neural Networks

Neuromorphic computing is a brain-like computing paradigm, generally referring to running Spiking Neural Networks (SNN) on neuromorphic chips. Essentially, neuromorphic computing is a design paradigm driven by algorithms. With its low-power advantages, neuromorphic computing is also considered a “potential substitute” for traditional AI. Understanding neuromorphic computing should be approached from a system level, rather than … Read more

What Are Artificial Neural Networks?

by
What Are Artificial Neural Networks?

*This article is from the 22nd issue of “Banyue Tan” in 2024 The 2024 Nobel Prize in Physics has unexpectedly honored the achievement of “fundamental discoveries and inventions to promote the use of artificial neural networks for machine learning.” What exactly are artificial neural networks? Can their potential really be compared to fundamental physical sciences? … Read more

Understanding the Nine Layers of Attention Mechanism

by
Understanding the Nine Layers of Attention Mechanism

This article is written by: Electric Light Phantom Alchemist Graduate topic Top 1, Shanghai Jiao Tong University Computer Science first place, first prize in high school physics competition, meme master, national award in computer science from Shanghai Jiao Tong University, currently a PhD student at CUHK https://zhuanlan.zhihu.com/p/362366192 Attention has become a hot topic in the … Read more

Next-Generation Attention Mechanism: Lightning Attention-2

by
Next-Generation Attention Mechanism: Lightning Attention-2

Click above toComputer Vision Alliance get more insights For academic sharing only, does not represent the position of this public account. Contact for deletion in case of infringement. Reprinted from: Machine Heart Recommended notes from 985 AI PhD Zhou Zhihua’s “Machine Learning” handwritten notes are officially open-source! Includes PDF download link, 2500 stars on GitHub! … Read more

Attention Mechanism Bug: Softmax as the Culprit Affecting All Transformers

by
Attention Mechanism Bug: Softmax as the Culprit Affecting All Transformers

“The stone from other hills can serve to polish jade.” Only by standing on the shoulders of giants can we see further and go farther. On the path of scientific research, we need to leverage favorable conditions to move forward faster. Therefore, we have specially collected and organized some practical code links, datasets, software, programming … Read more

Nine Layers of Understanding Attention Mechanism

by
Nine Layers of Understanding Attention Mechanism

↑ ClickBlue Text Follow the Extreme City Platform Author丨Electric Light Phantom Alchemy@Zhihu (Authorized) Source丨https://zhuanlan.zhihu.com/p/362366192 Editor丨Extreme City Platform Extreme City Guide Attention has become popular in the entire AI field, whether in machine vision or natural language processing, it is inseparable from Attention, transformer, or BERT. The author of this article follows the EM nine-layer tower … Read more

Attention Mechanism Bug: Softmax as the Culprit Affecting All Transformers

by
Attention Mechanism Bug: Softmax as the Culprit Affecting All Transformers

Machine Heart reports Machine Heart Editorial Team “Big model developers, you are wrong.” “I discovered a bug in the attention formula that no one has found for eight years. All Transformer models, including GPT and LLaMA, are affected.” Yesterday, a statistician named Evan Miller stirred up a storm in the AI field with his statement. … Read more

New Ideas on Attention Mechanism: Frequency Domain + Attention, Precision Exceeds SOTA 22.6%

by
New Ideas on Attention Mechanism: Frequency Domain + Attention, Precision Exceeds SOTA 22.6%

The Combination of Frequency Domain and Attention Mechanism is an innovative network design approach that utilizes frequency domain analysis to enhance the feature extraction process and further optimizes the efficiency of feature utilization through attention mechanisms. This strategy helps the model capture and utilize key frequency components in signals, which not only improves the model’s … Read more

Comprehensive Overview of Attention Mechanisms

by
Comprehensive Overview of Attention Mechanisms

1. Understanding the Principle of Attention Mechanism The Attention mechanism, in simple terms, refers to the output y at a certain moment and its attention on various parts of the input x. Here, attention represents weights, indicating the contribution of each part of the input x to the output y at that moment. Based on … Read more