General Processing Workflow of Natural Language Processing (NLP)

Follow the public account “ML_NLP“

Set as “Starred“, heavy content delivered immediately!

Source | Zhihu

Address | https://zhuanlan.zhihu.com/p/79041829

Author | mantch

Editor | Machine Learning Algorithms and Natural Language Processing Public Account

This article is for academic sharing only. If there is any infringement, please contact us to delete it.

1. What is NLP

Natural Language Processing (NLP) is a subfield of Artificial Intelligence (AI). NLP is a discipline that studies language issues in human-to-human interactions as well as human-to-computer interactions. To build and improve language models, NLP establishes computational frameworks and proposes corresponding methods to continuously refine the design of various practical systems and explore the evaluation methods for these practical systems.

2. Main Research Directions of NLP

Information Extraction: Extracting important information from given text, such as time, place, person, event, cause, result, numbers, dates, currency, proper nouns, etc. In simple terms, it aims to understand who did what to whom, when, for what reason, and what the result was.
Text Generation: Machines express and write using natural language like humans. Depending on the input, text generation techniques mainly include data-to-text generation and text-to-text generation. Data-to-text generation refers to converting data containing key-value pairs into natural language text; text-to-text generation transforms and processes input text to produce new text.
Question Answering System: Given a question expressed in natural language, the question answering system provides an accurate answer. It requires some level of semantic analysis of natural language queries, including entity linking and relationship recognition, to form logical expressions, then searches for possible candidate answers in a knowledge base and identifies the best answer through a ranking mechanism.
Dialogue System: The system chats with users, answers questions, and completes tasks through a series of dialogues. It involves user intent understanding, general chat engines, question answering engines, dialogue management, and other technologies. Additionally, to reflect contextual relevance, it must possess multi-turn dialogue capabilities.
Text Mining: Includes text clustering, classification, sentiment analysis, and the visualization and interactive expression of mined information and knowledge. The mainstream technologies are currently based on statistical machine learning.
Speech Recognition and Generation: Speech recognition converts spoken symbols input into written language representation. Speech generation, also known as text-to-speech conversion, refers to automatically converting written text into corresponding speech representation.
Information Filtering: Automatically identifying and filtering document information that meets specific criteria through computer systems. It usually refers to the automatic identification and filtering of harmful information on the internet, mainly used for information security and protection, network content management, etc.
Public Opinion Analysis: Refers to collecting and processing massive amounts of information to automate the analysis of online public opinion to achieve timely responses to online public sentiment.
Information Retrieval: Indexing large-scale documents. It can simply assign different weights to words in documents to establish an index or create deeper indexes. During querying, the input query expression, such as a search term or sentence, is analyzed, then matching candidate documents are searched in the index, sorted based on a ranking mechanism, and finally, the document with the highest score is output.
Machine Translation: Automatically translating input source language text into another language text. Machine translation has evolved from rule-based methods to statistical methods developed two decades ago, and now to neural network-based (encoder-decoder) methods, gradually forming a rigorous methodological system.

3. Development of NLP

Before 1950: Turing Test Before 1950, Alan Turing proposed the Turing Test: if a human cannot determine whether they are communicating with a human or a machine, it indicates that the machine possesses intelligence.
1950-1970: Mainstream: Rule-Based Formal Language Theory

Noam Chomsky studied natural language using the axiomatic method in mathematics, defining formal languages as sequences of symbols using algebra and set theory. He attempted to describe infinite language phenomena with a finite set of rules and discovered universal language mechanisms, establishing what is known as universal grammar.
1970-Present: Mainstream: Statistical-Based Google, Microsoft, IBM; in the 1970s, Frederick Jelinek and his team at IBM’s Watson Laboratory improved speech recognition rates from 70% to 90%. In 1988, Peter Brown from IBM proposed a statistical-based machine translation method. In 2005, Google’s machine translation defeated rule-based Sys Tran.
After 2010: Comeback: Machine Learning

AlphaGo defeated Lee Sedol and Ke Jie, sparking a boom in artificial intelligence. Deep learning and artificial neural networks became buzzwords. Fields: speech recognition, image recognition, machine translation, autonomous driving, smart home.

4. General Steps of NLP Tasks

If the image below is unclear, you can check the mind map on Baidu, click the link

General Processing Workflow of Natural Language Processing (NLP)

5. My Introductory Reading on NLP

Mathematics is Beautiful Second Edition.pdf – Lanzou Cloudwww.lanzous.com

6. NLP or CV, Which One to Choose?

NLP: Natural Language Processing, data is text.

CV: Computer Vision, data is images.

Both belong to different fields. When faced with this question, I hesitated for a long time and thought a lot, thus concluding: both use deep learning to solve real-world problems; without CV, NLP cannot survive; without NLP, CV cannot survive. They are like siblings; the entire “family” cannot be separated, but each individual exists with differences!

NLP and CV are two different research fields, both excellent fields. You can make a suitable choice based on your preferences. Artificial intelligence is a multidisciplinary field that requires not only one-sided abilities but also multifaceted skills. Everyone has their own focus, as human energy is limited. As long as you continue to delve deeply into your area of expertise, I believe everyone will achieve success!

Here are some reference materials for everyone to read and make a suitable choice:

A Comprehensive Overview of AI Breakthroughs in 2018: NLP Crossed a Watershed, CV Research Results are Impressivewww.toutiao.com