The attention mechanism can be likened to how humans read a book.
When you read, you don’t treat all content equally; you may pay more attention to certain keywords or sentences because they are more important for understanding the overall meaning.
Image: Highlighting key content in a book with background colors and comments.
The role of the attention mechanism in machine learning is similar.
Model’s Focal Lens
In machine learning, especially when processing language or images, the model needs to identify the most critical parts from a vast amount of information to understand and make decisions. The attention mechanism acts like the model’s focal lens, helping it to “focus” on the most important information while ignoring less significant parts.
Image: The painting “Madonna and Child” created by the Florentine artist Domenico Ghirlandaio in the 15th century, currently housed in the Palazzo Vecchio in Florence. If you don’t look back at this painting, do you remember the donkey and cow on the right side?
Translation Around Keywords -> Paraphrase
For example, in the context of translating a sentence, the model does not translate word for word but first identifies the keywords in the sentence and then translates around these keywords to ensure accuracy and fluency.
Taking it a step further, let ChatGPT translate twice: first, a direct translation, and then a paraphrase based on the first result. The overall translation quality will improve significantly. https://weibo.com/1727858283/NlsDSpPaa
Effect of translating only once:
Like Zuckerberg, Gates also dropped out of Harvard to establish a historically significant tech company. Like Zuckerberg, he is a prodigy in his field. Like Zuckerberg, he has created fans, made enemies, and raised antitrust concerns along the way.
Effect of translating twice:
Similar to Zuckerberg, Gates also left Harvard to create a tech company of significant historical importance; they were both miracle children in their respective fields; and during their strong push forward, they gained fans, created enemies, and raised antitrust concerns.
Image: The different feelings of direct and paraphrased translations.
The prompt used for the two translations is as follows:
You are a professional translator proficient in Simplified Chinese, having participated in the translations of the Chinese editions of The New York Times and The Economist, thus having a deep understanding of news and current affairs translation. I hope you can help me translate the following English news paragraph into Chinese, in a style similar to the aforementioned magazines.
Rules:
Ensure accurate transmission of news facts and context during translation.
Retain specific English terms or names, adding spaces before and after, e.g., “中 UN 文”.
Split into two translations, printing each result:
Translate directly based on the news content, without omitting any information.
Based on the first direct translation result, paraphrase while ensuring the original meaning is preserved and the content is more colloquially understandable, in line with Chinese expression habits.
This message only needs to reply OK; I will send you the complete content in the next message, and upon receipt, please print both translation results according to the above rules.
Implementing Attention with Weights
On a technical level, the attention mechanism is usually implemented by calculating a series of weights. These weights determine how much the model should “pay attention” to.
Any adjustment of items on the balance in the image will affect its balance; weights are similar, as changes will impact the final outcome.
In neural networks, these weights are multiplied by the input data to highlight important information and reduce the influence of unimportant information.
For example, suppose we have a sentence: “The weather is nice, suitable for going out.” If the task is to extract key information, the attention mechanism may assign higher weights to “The weather is nice” and “suitable for going out” because these two pieces of information are relatively important.
Calculating Weights
The calculation of weights usually relies on the training process of the model.
During model training, the model attempts to learn what type of information is important and how important that information should be.
This learning process is completed by continuously comparing predicted results with actual results and adjusting the weights to reduce the difference between the two.
Image: Aligning between reality and illusion by adjusting buttons (weights).
The specific steps involved include:
1. Initialization:
The model’s weights are randomly initialized at the start of training. This means that initially, the model does not know which information is more important.
2. Forward Propagation:
During the model’s training process, each time input data is provided, the attention mechanism generates an “attention distribution” based on the current weights. This distribution reflects the model’s current “guess” of the importance of each part of the data.
3. Loss Calculation:
The model’s output is compared with the actual data, and by calculating the loss (e.g., cross-entropy loss), we can determine how accurate the model’s predictions are compared to the actual situation.
4. Backpropagation and Weight Update:
After loss calculation, the model adjusts the weights through the backpropagation algorithm to reduce the loss.
This process involves calculating the gradient of the loss with respect to each weight and adjusting the weights based on these gradients, which is the famous gradient descent method.
For weights in the attention mechanism, if adjusting a particular weight reduces the loss, then in subsequent training, that weight will increase, indicating that the model has learned that this piece of information is relatively important.
5. Iterative Process:
This process is repeated across many different datasets, allowing the model to gradually learn which information should be given higher weights under different inputs through continuous iterative training.
Through this training process, the model can dynamically decide how much weight different inputs should be assigned in various contexts. This is an automated learning process that does not require manual specification of weights.
How to Determine Which Answer is Suitable When There are Multiple Answers?
In natural language processing, there is indeed a problem where there may be multiple different but correct answers to the same question.
Therefore, when evaluating the model’s output, we cannot rely solely on a single correct answer. To address this issue, researchers have developed various methods to calculate loss, allowing the model to consider answer diversity during training.
Here are some methods to handle this diversity:
1. Use More Flexible Loss Functions:
For example, when using cross-entropy loss, the model’s output is a probability distribution, meaning it can assign high probabilities to multiple possible answers. Thus, even if the reference answer (ground truth) does not exactly match the model’s prediction, as long as the predicted distribution is similar to the true distribution, the model’s loss can still be low.
2. Use Multiple Reference Answers:
During training and evaluation, multiple correct answers can be provided.
The model’s output will be compared with all of these answers, and as long as it matches one of them, it can be considered correct. This method is commonly used in evaluating translation quality, known as the BLEU (Bilingual Evaluation Understudy) score, which considers the degree of match between the output and a set of reference translations.
3. Use Fuzzy Matching and Semantic Similarity:
Some metrics consider not only whether the answers match on the surface but also the semantic similarity. For example, using word embedding vectors (such as Word2Vec or BERT) to assess the semantic distance between words.
4. Reinforcement Learning:
In some cases, the model’s output can be evaluated through interaction with the environment rather than directly comparing with a fixed reference answer. This method allows the model to learn useful behaviors in environments without exact answers.
5. Manual Evaluation:
In certain tasks, automatic evaluation may not be accurate or feasible. In such cases, manual evaluation can supplement the evaluation of the model’s output. Although this method is costlier, it can provide an intuitive and accurate measure of model performance.
6. Contrastive Learning:
The model can be trained to distinguish between correct and incorrect answers rather than just outputting a fixed answer. This method can help the model focus more on the quality of the answers rather than their exact form.
Through these methods, the model can consider answer diversity while reducing the discrepancy between predictions and actual situations.
Conclusion
In summary, the attention mechanism is a technology that enables machines to automatically discover and focus on the most useful information in data, making models more efficient and precise when handling complex tasks.