Transformer

Reinventing RNNs for the Transformer Era: RWKV Model

by
Reinventing RNNs for the Transformer Era: RWKV Model

Machine Heart Report Machine Heart Editorial Department Transformer models have revolutionized almost all natural language processing (NLP) tasks, but their memory and computational complexity grows quadratically with sequence length. In contrast, Recurrent Neural Networks (RNNs) grow linearly in memory and computational requirements, but due to limitations in parallelization and scalability, it is difficult to achieve … Read more

Innovative Application of Nested Transformer Model in Protein-Nucleic Acid Binding Site Prediction

by
Innovative Application of Nested Transformer Model in Protein-Nucleic Acid Binding Site Prediction

———————————— Exclusive arrangement, unauthorized use will be pursued In life science research, the interaction between proteins and nucleic acids plays a crucial role. Accurately predicting nucleic acid binding sites on protein surfaces is significant for understanding life regulatory mechanisms and developing new drugs. Recently, a research team from the University of Hong Kong and ShanghaiTech … Read more

Discussion on Absolute, Relative, and Rotational Position Encoding in Transformers

by
Discussion on Absolute, Relative, and Rotational Position Encoding in Transformers

Click the card below to follow the “AI Frontier Express” public account Various important resources delivered promptly Reprinted from Zhihu: Yao Yuan Link: https://zhuanlan.zhihu.com/p/17311602488 1. Introduction The attention mechanism in Transformer [1] can effectively model the correlations between tokens, achieving significant performance improvements in many tasks. However, the attention mechanism itself does not have the … Read more

Transformers as Support Vector Machines

by
Transformers as Support Vector Machines

Machine Heart reports Editors: Danjiang, Xiaozhou SVM is all you need; Support Vector Machines are never out of date. The Transformer is a new theoretical model of Support Vector Machines (SVM) that has sparked discussion in academia. Last weekend, a paper from the University of Pennsylvania and the University of California, Riverside, sought to explore … Read more

A Comprehensive Guide to Building Transformers

by
A Comprehensive Guide to Building Transformers

This article aims to introduce the Transformer model. Originally developed for machine translation, this model has since been widely applied in various fields such as computer recognition and multimodal tasks. The Transformer model introduces self-attention mechanisms and positional encoding, and its architecture mainly consists of an input part, an output part, and encoders and decoders. … Read more

Understanding the Transformer Model: A Visual Guide

by
Understanding the Transformer Model: A Visual Guide

Introduction In recent years, deep learning has made tremendous progress in the field of Natural Language Processing (NLP), and the Transformer model is undoubtedly one of the best. Since the Google research team proposed the Transformer model in their paper “Attention is All You Need” in 2017, it has become the cornerstone for many NLP … Read more

Overview of Transformer Compression

by
Overview of Transformer Compression

Large models based on the Transformer architecture are playing an increasingly important role in artificial intelligence, especially in the fields of natural language processing (NLP) and computer vision (CV). Model compression methods reduce their memory and computational costs, which is a necessary step for implementing Transformer models on practical devices. Given the unique architecture of … Read more

WTPose Framework: Enhancing Pose Estimation with Waterfall Module Based on Transformer

by
WTPose Framework: Enhancing Pose Estimation with Waterfall Module Based on Transformer

Click the card below to follow the 「Intelligent Book Boy」 public account Click to join👉「Intelligent Book Boy」 group chat Want to learn more: Cutting-edge AI visual perception full-stack knowledge👉「Classification, Detection, Segmentation, Key Points, Lane Line Detection, 3D Vision (Segmentation, Detection), Multi-modal, Object Tracking, NerF」 Industry Technical Solutions👉「AI Security, AI Healthcare, AI Autonomous Driving」 AI Model … Read more

Overview of Transformer Privacy Inference Technology

by
Overview of Transformer Privacy Inference Technology

Overview of Transformer Privacy Inference Technology 1. Introduction The Transformer model, especially large language models (LLMs) based on Transformers, has made significant progress in the field of artificial intelligence in recent years. The emergence of applications such as ChatGPT and Bing has made the capabilities of these models known and utilized by the public. These … Read more

Illustrated Transformer: Principles of Attention Calculation

by
Illustrated Transformer: Principles of Attention Calculation

This is the fourth translation in the Illustrated Transformer series. The series is authored by Ketan Doshi and published on Medium. During the translation process, I modified some illustrations and optimized and supplemented some descriptions based on the code provided in Li Mu’s “Hands-On Deep Learning with Pytorch”. The original article link can be found … Read more