An Overview of 11 Mainstream Attention Mechanisms in 2024

Attention mechanisms have become the foundational architecture for model design; nowadays, it’s almost embarrassing to release a model without any Attention.

Since the release of the attention mechanism, the academic community has been continuously modifying Attention in various innovative ways. The modified Attention can enhance the model’s expressive capability, improve cross-modal abilities and interpretability, as well as optimize model size and efficiency.

Most importantly, many attention modules are plug-and-play. We can utilize the attention modules developed by academic leaders in our own models, making experiments and writing papers much more efficient.

Recently, there have been numerous innovative studies on 11 mainstream attention mechanisms, including scaled dot-product attention, multi-head attention, cross-attention, spatial attention, and channel attention. Today, I will share 112 innovative studies on 11 mainstream attention mechanisms, updated as of September 2024. The latest innovative ideas are very suitable for use in your experiments!

112 innovative studies on 11 mainstream attention mechanisms, with papers and codes organized for easy download. Feel free to scan the code to receive them.

• 5.Sep.2024—LMLT: Low-to-high Multi-Level Vision Transformer for Image Super-Resolution

• 4.Sep.2024—MobileUNETR: A Lightweight End-To-End Hybrid Vision Transformer For Efficient Medical Image Segmentation

• 4.Sep.2024—More is More Addition Bias in Large Language Models

• 4.Sep.2024—LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via Hybrid Architecture

……

• 4.Sep.2024—Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening

• 30.Aug.2024—From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs

• 25.Jun.2024—Temporal-Channel Modeling in Multi-head Self-Attention for Synthetic Speech Detection

• 14.May.2024—Improving Transformers with Dynamically Composable Multi-Head Attention

……

• 25.Aug.2024—Vision-Language and Large Language Model Performance in Gastroenterology: GPT, Claude, Llama, Phi, Mistral, Gemma, and Quantized Models

• 21.Aug.2024—Unlocking Adversarial Suffix Optimization Without Affirmative Phrases: Efficient Black-box Jailbreaking via LLM as Optimizer

• 16.Aug.2024—Fine-tuning LLMs for Autonomous Spacecraft Control: A Case Study Using Kerbal Space Program

• 15.Aug.2024—FuseChat Knowledge Fusion of Chat Models

……