Attention Mechanism in Deep Learning

Attention Mechanism in Deep Learning

by

Introduction

Alexander J. Smola, the head of machine learning at Amazon Web Services, presented on the attention mechanism in deep learning at the ICML2019 conference, detailing the evolution from the earliest Nadaraya-Watson Estimator (NWE) to the latest Multiple Attention Heads.
Authors | Alex Smola, Aston Zhang
Translator | Xiaowen
Attention Mechanism in Deep Learning
The report is divided into six parts:
1. Watson Nadaraya Estimator (NWE): One of the earliest tools for Nonparametric Regression, specifically the first Kernel Regression Technique proposed independently by Nadaraya and Watson in 1964.
2. Pooling
  • Single-target – From pooling to attention pooling
  • Hierarchical – Hierarchical attention network
3. Iterative Pooling
  • Question answering / memory networks
4. Iterative Pooling and Generation
  • Neural Machine Translation
5. Multiple Attention Heads
  • Transformers / BERT
  • Lightweight, structured, sparse
6. Resources
Download Complete PPT
Please follow the Special Knowledge WeChat account (click the blue button above to follow)
  • Reply “ADL” to get the download link for the complete《Attention in Deep Learning》~
1. WNE
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
2. Pooling
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
3. Iterative Pooling
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
4. Iterative Pooling and Generation
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Due to space limitations, only part of the content is included. Please download the PPT for the complete content.
5. Multiple Attention Heads
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Due to space limitations, only part of the content is included. Please download the PPT for the complete content.
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
6. Resources
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning
Attention Mechanism in Deep Learning

-END-

Special Knowledge

Special Knowledge, a professional and reliable distribution of artificial intelligence knowledge, enables cognitive collaboration faster and better! Welcome to log in to www.zhuanzhi.ai, register and log in to obtain more AI knowledge materials!

Attention Mechanism in Deep Learning

Welcome to scan the WeChat code to join the Special Knowledge AI Knowledge Group to get the latest AI professional resources and tutorial videos and consult with experts!

Attention Mechanism in Deep Learning

Please add Special Knowledge Assistant WeChat (scan the QR code below to add), join the Special Knowledge AI Theme Group, consult technical business cooperation~

Attention Mechanism in Deep Learning

Special Knowledge’s “Deep Learning: From Algorithms to Practice” course has been completed! Over 560 students are learning, enroll now for a limited-time discount! The top-selling course on Netease Cloud Classroom in the AI category!

Attention Mechanism in Deep Learning

Click “Read Original” to learn about the enrollment forSpecial Knowledge’s “Deep Learning: From Algorithms to Practice” course

Leave a Comment