© Author | Wang Kunze Affiliation | The University of Sydney Research Direction | NLP The comparison between Batch Norm and Layer Norm has become a cliché in the field of algorithms. The question of why BERT uses layer norm instead of batch norm has been asked to death, and a casual search on Zhihu … Read more
©PaperWeekly Original · Author｜Su Jianlin Unit｜Zhuiyi Technology Research Direction｜NLP, Neural Networks ▲ The Years of “Chickens and Rabbits in the Same Cage” “Profit and loss problems”, “age problems”, “planting trees problems”, “cows eating grass problems”, “profit problems”… Have you ever been tormented by various types of math word problems during elementary school? No worries, machine … Read more
New Intelligence Report Editor: QJP [New Intelligence Guide] As NLP models become increasingly powerful and are deployed in real-world scenarios, understanding the predictions made by these models becomes more crucial. Recently, Google released a new language interpretability tool (LIT), which is a new approach to explain and analyze NLP models, making their results less of … Read more
Author丨Zhou Peng Affiliation丨Tencent Research Direction丨Natural Language Processing, Knowledge Graph Background In the past two years, unsupervised pre-trained language representation models such as Google’s BERT have achieved remarkable results in various NLP tasks. These models are pre-trained on large-scale open-domain corpora to obtain general language representations and then fine-tuned on specific downstream tasks to absorb domain-specific … Read more
This article introduces several papers that improve the pretraining process of BERT, including Pre-Training with Whole Word Masking for Chinese BERT, ERNIE: Enhanced Representation through Knowledge Integration, and ERNIE 2.0: A Continual Pre-training Framework for Language Understanding. Note: These papers all implement different improvements to the masking of BERT’s pretraining phase, but do not modify … Read more
©PaperWeekly Original · Author｜Xiong Zhiwei School｜Tsinghua University Research Direction｜Natural Language Processing Since BERT was proposed in 2018, it has gained significant success and attention. Based on this, various related models have been proposed in academia to improve BERT. This article attempts to summarize and organize these models. MT-DNN MT-DNN (Multi-Task DNN) was proposed by Microsoft … Read more
Every time I start a new machine learning project, the first thing that gives me a headache is not model selection, but the dataset. Downloading datasets, unzipping, cleaning, formatting—a series of steps makes me feel like I’m facing a “programmer’s physical labor” challenge. And once the dataset is too large to load into memory all … Read more
In Chapter 2, we saw what is needed to fine-tune and evaluate a Transformer. Now let’s take a look at how they work under the hood. In this chapter, we will explore the main components of the Transformer model and how to implement them using PyTorch. We will also provide guidance on how to do … Read more
Follow our public account “ML_NLP“ Set as “Starred“, heavy content delivered first time! Reprinted from | PaperWeekly ©PaperWeekly Original · Author｜Li Luoqiu School｜Zhejiang University Master’s Student Research Direction｜Natural Language Processing, Knowledge Graph Continuing from the previous article, I will record my understanding of the HuggingFace open-source Transformers project code. This article is based on the … Read more
Machine Heart reports Editor: Du Wei The Hugging Face NLP course is now live, and all courses are completely free. Those in the NLP field should be very familiar with the renowned Hugging Face, a startup focused on solving various NLP problems that has brought many beneficial technical achievements to the community. Last year, the … Read more