General Processing Flow of Natural Language Processing (NLP)

Follow the WeChat public account “ML_NLP“

Set as “Starred“, delivering heavy content at the first time!

Source | Zhihu

Address | https://zhuanlan.zhihu.com/p/79041829

Author | mantch

Editor | Machine Learning Algorithms and Natural Language Processing WeChat public account

This article is for academic sharing only. If there is any infringement, please contact the backend for deletion.

1. What is NLP

Natural Language Processing (NLP) is a subfield of Artificial Intelligence (AI). NLP is a discipline that studies language issues in human-to-human and human-to-computer interactions. To build and improve language models, NLP establishes computational frameworks, proposes corresponding methods to continuously improve the design of various practical systems, and explores evaluation methods for these practical systems.

2. Main Research Directions of NLP

Information Extraction: Extracting important information from given text, such as time, place, person, event, reason, result, numbers, dates, currency, proper nouns, etc. Simply put, it is about understanding who did what to whom, when, for what reason, and with what result.
Text Generation: Machines express and write using natural language like humans. Depending on the input, text generation technology mainly includes data-to-text generation and text-to-text generation. Data-to-text generation refers to converting data containing key-value pairs into natural language text; text-to-text generation transforms and processes input text to produce new text.
Question Answering Systems: Given a question expressed in natural language, the question-answering system provides an accurate answer. This requires a certain degree of semantic analysis of the natural language query, including entity linking and relationship recognition, forming logical expressions, and then searching for possible candidate answers in the knowledge base, finding the best answer through a ranking mechanism.
Dialogue Systems: The system chats, answers, and completes a task with users through a series of dialogues. This involves understanding user intentions, general chat engines, question-answer engines, dialogue management, and other technologies. Additionally, to reflect contextual relevance, it must have multi-turn dialogue capabilities.
Text Mining: This includes text clustering, classification, sentiment analysis, and the visualization and interactive expression of mined information and knowledge. Currently, mainstream technologies are based on statistical machine learning.
Speech Recognition and Generation: Speech recognition is converting spoken symbols input into written language representation. Speech generation, also known as text-to-speech or speech synthesis, refers to automatically converting written text into corresponding speech representation.
Information Filtering: Automatically identifying and filtering document information that meets specific conditions through computer systems. This usually refers to the automatic identification and filtering of harmful information on the internet, mainly used for information security and protection, network content management, etc.
Public Opinion Analysis: This refers to collecting and processing massive information and automating the analysis of online public opinion to achieve timely responses to online public sentiment.
Information Retrieval: Indexing large-scale documents. It can simply assign different weights to the vocabulary in the documents to establish an index or create a deeper index. During queries, it analyzes the input query expression, such as a search term or sentence, then searches for matching candidate documents in the index, ranks them according to a ranking mechanism, and finally outputs the document with the highest ranking score.
Machine Translation: Automatically translating input source language text into another language. Machine translation has evolved from rule-based methods in the early days to statistical methods twenty years ago, and now to neural network-based (encoder-decoder) methods, gradually forming a rigorous methodological system.

3. Development of NLP

Before 1950: Turing Test – Before 1950, Alan Turing proposed the Turing Test: if a person cannot determine whether they are communicating with a human or a machine, it indicates that the machine possesses intelligence.
1950-1970: Mainstream: Rule-Based Formal Language Theory

Noam Chomsky studied natural language using the axiomatic method from mathematics, defining formal languages as sequences of symbols using algebra and set theory. He attempted to describe infinite language phenomena with a finite set of rules, discovering the universal language mechanisms of humans and establishing what is called universal grammar.
1970-Present: Mainstream: Statistical Approaches – Google, Microsoft, IBM. In the 1970s, Frederick Jelinek and his team at IBM’s Watson Lab improved speech recognition rates from 70% to 90%. In 1988, Peter Brown of IBM proposed a statistical machine translation method. In 2005, Google Translate surpassed rule-based Sys Tran.
After 2010: Comeback: Machine Learning

AlphaGo successively defeated Lee Sedol and Ke Jie, sparking a wave of interest in artificial intelligence. Deep learning and artificial neural networks became buzzwords. Fields: speech recognition, image recognition, machine translation, autonomous driving, smart homes.

4. General Steps of NLP Tasks

If the image below is unclear, you can view the mind map on Baidu, click the link

General Processing Flow of Natural Language Processing (NLP)

5. My Introductory Readings on NLP

Mathematics is Beautiful, Second Edition.pdf – Lanzou Cloudwww.lanzous.com

6. NLP or CV, Which to Choose?

NLP: Natural Language Processing, data is text.

CV: Computer Vision, data is images.

The two belong to different fields. When faced with this question, I also hesitated for a long time and thought a lot, and thus came to the conclusion: both use deep learning to solve problems in the real world. Without CV, NLP cannot survive; without NLP, CV cannot survive. The two are like siblings, the entire “family” cannot be divided, yet individuals exist with differences!

NLP/CV belong to two different research fields, both are excellent fields, and one can choose based on personal interests. Artificial intelligence is a multidisciplinary field that requires not only one-sided abilities but also multifaceted capabilities. For everyone, there is a focus, after all, human energy is limited. As long as you continue to delve into your area of expertise, I believe everyone will achieve success!

Here are some reference materials for everyone to read and make suitable choices:

2018 AI Technology Breakthroughs: NLP Crossed a Watershed, CV Research Effects Remarkablewww.toutiao.com