Where Do Knowledge Graphs Come From: The Current Status and Future of Entity Relationship Extraction

Machine Heart Reprint

Authors:Han Xu, Gao Tianyu, Liu Zhiyuan

This article is written by Professor Liu Zhiyuan from Tsinghua University and his students Han Xu and Gao Tianyu, introducing the topic of knowledge graphs. The article reviews the development of the knowledge graph field and summarizes recent research progress. It has been authorized for reprint by Machine Heart.

In recent years, the wave of artificial intelligence driven by deep learning has swept the world, rapidly improving under the dual support of massive data resources brought by the Internet and computational power governed by Moore’s Law. Deep learning has profoundly influenced various directions in natural language processing, greatly advancing its development.

As of 2019, many limitations of deep learning are gradually being recognized. For natural language processing, achieving fine-grained semantic understanding cannot be resolved solely by data annotation and computational investment. Without prior knowledge support, “No one can beat China’s table tennis” and “No one can beat China’s soccer” have no significant semantic difference from a computer’s perspective, while in reality, the meanings of “beat” in the two sentences are exactly opposite. Therefore, integrating knowledge to guide natural language processing is the necessary path to achieve fine and deep language understanding. However, where does this knowledge come from? This involves a key research question in artificial intelligence—knowledge acquisition.

Knowledge Graphs

Existing large knowledge graphs, such as Wikidata, Yago, and DBpedia, contain a vast amount of world knowledge stored in a structured format. As shown in the figure below, each node represents an entity in the real world, and the edges denote the relationships between entities. For instance, knowledge related to American author Mark Twain is recorded in a structured form.

Currently, this structured knowledge has been widely applied in natural language processing applications such as search engines and question-answering systems. However, compared to the rapidly growing amount of knowledge in the real world, the coverage of knowledge graphs still falls short. Due to the vast scale of knowledge and the high cost of manual annotation, it is nearly impossible to add new knowledge solely through human annotation. To enrich knowledge graphs with more comprehensive world knowledge in a timely and accurate manner, researchers are exploring efficient methods for automatic knowledge acquisition, namely entity relationship extraction technology.

Specifically, given a sentence and the entities within it, the entity relationship extraction model needs to infer the relationship between the entities based on the semantic information of the sentence. For example, given the sentence: “Tsinghua University is located near Beijing” and the entities “Tsinghua University” and “Beijing”, the model can infer the relationship “is located in” and ultimately extract the knowledge triplet (Tsinghua University, is located in, Beijing).

Entity relationship extraction is a classic task that has seen ongoing research for over 20 years, with feature engineering, kernel methods, and graph models being widely applied, achieving some phase results. With the advent of the deep learning era, neural network models have brought new breakthroughs to entity relationship extraction.

Neural Network Relationship Extraction Models

There are many types of neural networks designed for natural language text sequences, such as Recurrent Neural Networks (RNN, LSTM), Convolutional Neural Networks (CNN), and Transformers, which can all be adapted for relationship extraction. Initially, work [1, 2] proposed using CNNs to encode the semantic information of sentences for relationship classification, significantly improving performance compared to non-neural network methods; work [3, 4] applied RNNs and LSTMs for relationship extraction; additionally, work [5] proposed using recursive neural networks to model the syntactic parse tree of sentences, attempting to consider both lexical and syntactic features while extracting semantic characteristics, a concept that has been further explored by many subsequent works. Here, we present a table summarizing the performance of various typical neural networks on the benchmark dataset SemEval-2010 Task-8 [6].

From the above table, it can be seen that these neural network models have achieved excellent experimental results, and there are no significant performance differences among them. Does this mean the relationship extraction problem has been solved? In fact, it is not. The task setting of SemEval-2010 Task-8 was to label a large number of training and test samples for predefined relationship categories, with samples being relatively simple short sentences, and the distribution of samples for each relationship being quite uniform. However, in practical applications, many challenges often arise:

• Data Scale Issue: Accurately annotating sentence-level data manually is extremely expensive and requires a lot of time and manpower. In practical scenarios, facing thousands of relationships, tens of millions of entity pairs, and hundreds of millions of sentences, relying on manual annotation to train data is almost an impossible task.

• Learning Ability Issue: In real situations, the frequency of relationships between entities and entity pairs often follows a long-tail distribution, with a large number of relationships or entity pairs having few samples. The performance of neural network models relies on a large amount of annotated data, leading to the “ten to one” problem. How to enhance the learning ability of deep models to achieve “one to many” is a problem that relationship extraction needs to solve.

• Complex Context Issue: Existing models mainly extract relationships between entities from individual sentences, requiring that the sentences must contain both entities. In reality, many relationships between entities are often expressed across multiple sentences in a document, or even across multiple documents. How to extract relationships in more complex contexts is also a challenge for relationship extraction.

• Open Relationship Issue: Current task settings generally assume a predefined closed set of relationships, transforming the task into a relationship classification problem. In this case, new types of relationships between entities contained in the text cannot be effectively captured. How to utilize deep learning models to automatically discover new types of relationships between entities and achieve open relationship extraction remains an “open” problem.

Thus, there is a significant gap between the ideal setting of SemEval-2010 Task-8 and the practical scenarios; relying solely on neural networks to extract semantic features from individual sentences is insufficient to meet the various complex demands and challenges of relationship extraction. We urgently need to explore newer frameworks for relationship extraction, acquire larger-scale training data, enhance learning capabilities, effectively understand complex document-level contextual information, and easily extend to open relationship extraction.

We believe that these four aspects constitute the main directions for further exploration in entity relationship extraction. Next, we will introduce the current status and challenges of these four aspects, along with some of our thoughts and efforts.

Larger Scale Training Data

Neural network relationship extraction requires a large amount of training data; however, manually annotating this training data is very time-consuming and expensive. To automatically acquire more training data, work [16] proposed the idea of Distant Supervision, aligning pure text with existing knowledge graphs to automatically annotate large-scale training data.

The idea of distant supervision is not complicated. Specifically, if two entities are labeled as a certain relationship in the knowledge graph, we consider all sentences containing both entities as expressing this relationship. Taking (Tsinghua University, is located in, Beijing) as an example, we would treat all sentences containing both “Tsinghua University” and “Beijing” as training samples for the relationship “is located in”.

The heuristic annotation rules of distant supervision are a double-edged sword; they are an effective strategy for automatically annotating training data, but their strong assumptions inevitably lead to incorrect annotations. For example, for the knowledge graph triplet (Tsinghua University, president, Qiu Yong), the sentence “Qiu Yong serves as the president of Tsinghua University” reflects the relationship “president” between “Tsinghua University” and “Qiu Yong”; however, the sentences “Qiu Yong entered the Department of Chemistry and Chemical Engineering at Tsinghua University” and “Qiu Yong serves as a member of the Tsinghua University Party Committee” do not express the “president” relationship but would be incorrectly annotated as training instances for the “president” relationship by the heuristic rules of distant supervision.

Although the distant supervision idea is very simple and has many issues, it has opened a new era for collecting training data. Inspired by this idea, many scholars are actively considering how to minimize the interference of noisy annotations in distant supervision data. Since 2015, neural relationship extraction models based on distant supervision and noise reduction mechanisms have made significant progress. Work [17] introduced a multi-instance learning method that uses all instances containing the same entity pair to jointly predict the relationship between entities. Our research group, led by Lin Yankai and others, proposed a sentence-level attention mechanism that assigns different weights to different instances to mitigate the impact of noisy instances. Work [20] introduced adversarial training to enhance the model’s resistance to noisy data. Work [21] constructed a reinforcement learning mechanism to filter out noisy data and used the remaining data to train the model.

In summary, existing noise reduction methods based on distant supervision can balance the robustness and effectiveness of relationship extraction, and they also have strong operability and practicality. However, using existing knowledge graphs to align text for acquiring data to train relationship extraction models, and then using these models to extract knowledge to add to knowledge graphs, presents a chicken-and-egg scenario. Imperfect knowledge graph alignments will also yield imperfect training data, and for long-tail knowledge, it remains challenging to obtain training instances through this distant supervision mechanism. How to propose more effective mechanisms to efficiently acquire high-quality, high-coverage, and balanced training data remains a question worth deeper consideration.

More Efficient Learning Ability

Even if high-quality training data can be automatically acquired through distant supervision, due to the long-tail distribution characteristics of relationships and entity pairs in real-world scenarios, the majority of available samples for relationships and entity pairs remain limited. Additionally, for specialized relationships in fields such as healthcare and finance, the available samples are also quite limited due to data scale issues. As a typical data-hungry technology, neural network models are greatly affected when the training samples are too few. Therefore, researchers are looking to explore effective methods to enhance model learning ability to better utilize limited training samples and achieve satisfactory extraction performance.

In fact, humans can quickly learn knowledge from a small number of samples and possess the ability to generalize. To explore the ability of deep learning and machine learning to generalize, the task of Few-shot Learning has been proposed. By designing a few-shot learning mechanism, models can leverage generalized knowledge learned from past data and combine it with a small number of training samples of new types of data to achieve rapid transfer learning, demonstrating a certain level of generalization ability.

In the past, few-shot learning research primarily focused on computer vision; there has been little exploration in the field of natural language processing. Our research group, led by Han Xu and others, was the first to introduce few-shot learning into relationship extraction, constructing the FewRel few-shot relation extraction dataset, hoping to drive research in few-shot learning specifically for natural language processing and especially for relationship extraction tasks. As shown in the figure below, the few-shot learning problem for relationship extraction only provides a very small number of samples (e.g., 3-5) for each relationship, requiring improvement in relationship classification performance on test samples.

The FewRel paper initially attempted several representative few-shot learning methods, including metric learning, meta-learning, and parameter prediction, with evaluations indicating that even the best-performing Prototypical Networks model still significantly lags behind human performance in few-shot relationship extraction.

To better address the few-shot learning problem in distant supervised relationship extraction, our research group, led by Gao Tianyu and others, proposed a hybrid attention mechanism-based prototypical network that considers both instance-level and feature-level attention mechanisms, reducing the impact of noisy annotations while better focusing on useful features in the sentence to achieve efficient few-shot learning. Work [23] proposed a multi-level matching and aggregation structure to fully learn the latent associations between training samples, attempting to extract potential information from the limited samples. Work [24] adopted the pre-trained language model BERT to tackle the few-shot learning problem in relationship extraction, achieving performance that exceeds human relationship classification levels on the FewRel dataset.

During the exploration of few-shot learning for relationship extraction, our research group further identified two long-neglected aspects [25]: to apply few-shot learning models in production environments, they should possess the ability to transfer knowledge from resource-rich domains to resource-poor domains, and they should also be able to detect whether a sentence truly expresses a predefined relationship or does not express any relationship at all. To address this, they proposed FewRel 2.0, which builds upon the original FewRel dataset and adds two major challenges: domain adaptation and “none-of-the-above” detection.

For the domain adaptation challenge, FewRel 2.0 collected and annotated a large amount of data from the medical field, requiring relationship extraction models to perform few-shot learning on these new domain data after training on the original corpus. For the “none-of-the-above” detection, FewRel 2.0 added a “none-of-the-above” option to the original N-way K-shot setting (given N new types, each type given K training samples), significantly increasing the difficulty of classification and detection.

Preliminary experiments revealed that previously effective models (including BERT-based models) experienced significant performance drops on both of these challenge tasks. Although Gao Tianyu and others attempted some possible solutions in the FewRel 2.0 paper: for domain adaptation, they tried classic adversarial learning methods, which provided some performance improvement; for “none-of-the-above” detection, they proposed the BERT-PAIR model based on the BERT next sentence prediction task, achieving some success in the “none-of-the-above” challenge. However, these two challenges still require more innovative exploration.

In summary, exploring few-shot learning for relationship extraction to equip relationship extraction models with stronger and more efficient learning capabilities is still a very emerging research direction, particularly for the few-shot learning problem in relationship extraction, which has unique characteristics and challenges compared to few-shot learning problems in other fields. Whether it is improving existing few-shot learning techniques to suit NLP and relationship extraction or proposing entirely new few-shot learning models for relationship extraction, both will maximize the utilization of limited annotated data and promote the practical application of relationship extraction technology.

More Complex Textual Contexts

Current relationship extraction work mainly focuses on sentence-level relationship extraction, i.e., extracting relationships based on information within sentences. Various neural network models excel at encoding sentence-level semantic information and can achieve the best results on many public evaluation datasets. However, in practical scenarios, many relationships between entities are expressed across multiple sentences. As shown in the figure below, multiple entities are mentioned in the text, demonstrating complex interrelations. Statistics from manually annotated data sampled from Wikipedia indicate that at least 40% of entity relationship facts can only be jointly obtained from multiple sentences. To achieve cross-sentence relationship extraction between multiple entities, models need to read and reason across multiple sentences in a document, which clearly exceeds the capabilities of sentence-level relationship extraction methods. Therefore, document-level relationship extraction is imperative.

Research on document-level relationship extraction requires large-scale manually annotated datasets for training and evaluation. Currently, there are very few document-level relationship extraction datasets. Works [26, 27] constructed two datasets based on distant supervision, but since they were not manually annotated, their evaluation results are not very reliable. BC5CDR [28] is a manually annotated document-level relationship extraction dataset consisting of 1,500 PubMed documents in a biomedical-specific domain, only considering the “chemically induced disease” relationship, which may not be suitable for exploring general methods for document-level relationship extraction. Work [29] proposed extracting entity relationship facts from documents using reading comprehension techniques to answer questions, with these questions derived from “entity-relationship” pairs. Since this work’s dataset is tailored for this method, it may not be suitable for exploring general methods for document-level relationship extraction. These datasets either have very few manually annotated relationships and entities, or they contain noise from distant supervision, or they serve specific domains or methods, each having its own limitations.

To promote research in document-level relationship extraction, our research group, led by Yao Yuan and others [30], proposed the DocRED dataset, which is a large-scale manually annotated document-level relationship extraction dataset constructed based on the main text of Wikipedia and the WikiData knowledge graph. It contains 5,053 Wikipedia documents, 132,375 entities, and 53,554 entity relationship facts, making it the largest manually annotated document-level relationship extraction dataset available. As shown in the figure below, the document-level relationship extraction task requires models to possess strong pattern recognition, logical reasoning, coreference resolution, and common sense reasoning abilities, all of which require more long-term research exploration.

More Open Relationship Types

Current relationship extraction work generally assumes a predefined closed set of relationships, transforming the task into a relationship classification problem. However, in real-world open-domain relationship extraction scenarios, texts contain a large number of open entity relationships, with a wide variety of relationship types that continue to grow, far exceeding the number of relationships defined by humans. In this case, traditional relationship classification models cannot effectively capture new types of relationships between entities contained in the text. How to utilize deep learning models to automatically discover new types of relationships between entities and achieve open relationship extraction remains an “open” problem.

To achieve open relationship extraction in open domains, the Open Relation Extraction (OpenRE) task has been proposed, which aims to extract any relationship facts between entities from open texts. Open relationship extraction involves three aspects of “openness”: first is the openness of the relationship types extracted; unlike traditional relationship extraction, it aims to extract all known and unknown relationships; second is the openness of the test corpus, such as news, healthcare, etc., where different texts have different characteristics, requiring exploration of cross-domain robust algorithms; third is the openness of the training corpus; to obtain the best possible open relationship extraction model, it is necessary to fully utilize existing annotated data of various types, including precisely annotated and distantly supervised annotated data, and the relationship definitions and distributions of different training datasets vary, necessitating the simultaneous utilization of multiple source datasets.

In the pre-deep learning era, researchers also explored the Open Information Extraction (OpenIE) task. Open relationship extraction can be seen as a special case of OpenIE. At that time, OpenIE mainly achieved its goals through unsupervised statistical learning methods, such as the Snowball algorithm. Although these algorithms demonstrate good robustness across different datasets, their precision is often low, and they remain far from practical application.

Recently, work by our research group, led by Wu Ruidong and others [31], proposed a supervised open relationship extraction framework that can freely switch between supervised and weakly supervised modes through a “Relation Siamese Network” (RSN), allowing simultaneous learning of the semantic similarity of different relationship facts using both supervised data of predefined relationships and unsupervised data of new relationships in open texts. Specifically, the RSN employs a Siamese network structure to learn deep semantic features of relationship samples from labeled data of predefined relationships and their semantic similarities, which can be used to calculate the semantic similarity of open relationship texts. Moreover, the RSN also utilizes conditional entropy minimization and virtual adversarial training as semi-supervised learning methods to further leverage unlabeled open relationship text data, enhancing the stability and generalization ability of open relationship extraction. Based on the semantic similarity computed by the RSN, the model can cluster text relationships in open domains, thereby inferring new types of relationships.

Work by our group led by Gao Tianyu and others [32] took a different approach; for specific new relationships in open domains, only a small number of precise instances are needed as seeds, allowing the use of a pre-trained relationship Siamese network for neural snowballing to extract more instances of that new relationship from large volumes of unlabeled text, iteratively training a relationship extraction model suitable for the new relationship.

In summary, while open-domain relationship extraction achieved some success in the pre-deep learning era, how to effectively combine it with the advantages of neural network models in the deep learning era to significantly expand the generalization ability of neural network relationship extraction models is a topic worthy of further exploration.

Conclusion

To expand knowledge graphs in a timely manner, automatically acquiring new world knowledge from massive data has become an essential path. Knowledge acquisition technologies represented by entity relationship extraction have achieved some results, especially in recent years, deep learning models have greatly advanced the development of relationship extraction. However, compared to the complex challenges of relationship extraction in practical scenarios, existing technologies still have significant limitations. We urgently need to address challenges in training data acquisition, few-shot learning capabilities, complex textual contexts, and open relationship modeling, establishing effective and robust relationship extraction systems, which is also the direction that the entity relationship extraction task needs to continue to strive for.

Our research group has been working on the entity relationship extraction task since 2016, with students such as Lin Yankai, Han Xu, Yao Yuan, Zeng Wenyuan, Zhang Zhengyan, Zhu Hao, Yu Pengfei, Yu Zhijing, Gao Tianyu, Wang Xiaozhi, and Wu Ruidong conducting research in various areas. Last year, thanks to the efforts of Han Xu and Gao Tianyu, we released the OpenNRE toolkit [33], which has been continuously improved over the past two years, covering a rich array of scenarios including supervised relationship extraction, distant supervised relationship extraction, few-shot learning relationship extraction, and document-level relationship extraction. Additionally, we have invested significant research funding to annotate datasets such as FewRel (1.0 and 2.0) and DocRED, aiming to promote research in related directions.

This article summarizes our understanding of the current status, challenges, and future directions of entity relationship extraction, as well as our efforts in these areas, hoping to spark interest and provide some assistance. We look forward to more scholars and students joining this field of research. Of course, this article does not address an important challenge, namely the complex structured knowledge acquisition represented by event extraction; we will discuss this in detail in the future.

Due to my personal limitations, there may inevitably be biases and errors, so please feel free to point them out in the comments, and we will strive to improve. It should be noted that we did not intend to write this article as a rigorous academic paper, so we did not cover every direction comprehensively; if there are important omissions, please criticize and correct.

Advertisement Time

Many of the works our research group has conducted in entity relationship extraction (such as FewRel, DocRED, etc.) have been completed in collaboration with the Tencent WeChat Pattern Recognition Center team.The WeChat Pattern Recognition Center is one of the centers under WeChat AI, primarily focusing on research and products related to natural language processing.In terms of research, their work covers multiple directions including dialogue systems, knowledge learning, cognitive reasoning, and machine translation. This year, they published over 20 papers at conferences such as ACL, EMNLP, and AAAI, and achieved excellent results in various competitions, demonstrating substantial academic achievements.In terms of products, their developed Xiao Wei dialogue system and WeChat dialogue open platform have performed well in scenarios such as speakers and automatic customer service for public accounts, but the manpower invested is much less than that of the Amazon Alexa team, reflecting WeChat’s “small” team achieving big things.The academic and product strengths of the WeChat Pattern Recognition Center team have also brought a unique experience to our collaboration; on one hand, both sides maintain a high enthusiasm for researching world-leading technologies, working together on explorations that require time and effort but have far-reaching impacts; on the other hand, real products provide different perspectives and application scenarios for our research, truly achieving a strong alliance and complementary advantages, making it a team worth collaborating with.

Our cooperation with Tencent WeChat is based on the “Tsinghua-Tencent Joint Laboratory”.We have collaborated with the Tencent University Cooperation Center for many years, participating in various projects including the Tsinghua-Tencent Joint Laboratory (a university-level platform for collaboration with various departments at Tsinghua), the Rhinoceros Bird Special Fund (a frontier exploration research project for various teachers), and the Rhinoceros Bird Elite Talent Training Program (for students, a joint mentorship program by Tencent and Tsinghua to cultivate research-oriented talents).We thank the Tencent University Cooperation Center for building a collaborative and win-win platform for industry-university-research cooperation, and everyone can follow the Tencent University Cooperation WeChat account Tencent_UR for the latest information.

Here, I would like to particularly introduce our contact person at the WeChat Pattern Recognition Center, Patrick Li (for reasons that cannot be disclosed, I can only use his English name, not to show off), and Lin Yankai.Patrick is my junior, and we have known each other for over a decade. He is a 2005 undergraduate and 2009 doctoral student from Tsinghua University.He joined our research group during his undergraduate studies, and I collaborated with him to publish my first EMNLP 2009 paper. Later, he came to our research group for his PhD, working under Professors Sun Maosong and Liu Yang on machine translation research, achieving many influential results.He is the project leader for our collaboration and currently leads the research and application work on NLP foundational technologies at the WeChat Pattern Recognition Center, well-respected in both technology and character, with students who have collaborated with him having deep impressions.Currently, he mainly focuses on automatic question answering, information extraction, and machine translation.Perhaps inspired by Patrick, our student Lin Yankai (http://nlp.csai.tsinghua.edu.cn/~lyk/) chose to join their team after graduating with his PhD this year, and he is also collaborating with us on research related to knowledge graphs and automatic question answering.

If any readers are looking for internships or jobs in natural language processing and knowledge graphs, please feel free to contact Patrick Li ([email protected]), and let us work together to conduct interesting and meaningful research.

Patrick Li:http://www.lpeng.net

Author Biography

Han Xu, a third-year PhD student in the Department of Computer Science and Technology at Tsinghua University, mainly researches natural language processing, knowledge graphs, and information extraction. He has published multiple papers at internationally renowned conferences in the field of artificial intelligence, including AAAI, ACL, EMNLP, COLING, and NAACL, and is one of the developers of open-source projects such as OpenKE and OpenNRE. Homepage:

https://thucsthanxu13.github.io/thucsthanxu13.github.io

Gao Tianyu, a senior undergraduate student in the Department of Computer Science at Tsinghua University, mainly researches natural language processing, knowledge graphs, and relationship extraction. He has published multiple papers at internationally renowned conferences in the field of artificial intelligence, including AAAI and EMNLP, and is one of the main developers of open-source projects such as OpenNRE. Homepage:

gaotianyu.xyz

Liu Zhiyuan, associate professor and doctoral supervisor in the Department of Computer Science at Tsinghua University. His main research areas include representation learning, knowledge graphs, and social computing. Homepage:

nlp.csai.tsinghua.edu.cn

Original link:https://zhuanlan.zhihu.com/p/91762831

References

[1] ChunYang Liu, WenBo Sun, WenHan Chao, Wanxiang Che. Convolution Neural Network for Relation Extraction. The 9th International Conference on Advanced Data Mining and Applications (ADMA 2013).

[2] Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao. Relation Classification via Convolutional Deep Neural Network. The 25th International Conference on Computational Linguistics (COLING 2014).

[3] Dongxu Zhang, Dong Wang. Relation Classification via Recurrent Neural Network. arXiv preprint arXiv:1508.01006 (2015).

[4] Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, Bo Xu. Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. The 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016).

[5] Richard Socher, Brody Huval, Christopher D. Manning, Andrew Y. Ng. Semantic Compositionality through Recursive Matrix-Vector Spaces. The 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2012).

[6] Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, Stan Szpakowicz. SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals. The 5th International Workshop on Semantic Evaluation (SEMEVAL 2010).

[7] Thien Huu Nguyen, Ralph Grishman. Relation Extraction: Perspective from Convolutional Neural Networks. The 1st Workshop on Vector Space Modeling for Natural Language Processing (LatentVar 2015).

[8] Cícero dos Santos, Bing Xiang, Bowen Zhou. Classifying Relations by Ranking with Convolutional Neural Networks. The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2015).

[9] Shu Zhang, Dequan Zheng, Xinchen Hu, Ming Yang. Bidirectional Long Short-Term Memory Networks for Relation Classification. The 29th Pacific Asia Conference on Language, Information and Computation (PACLIC 2015).

[10] Minguang Xiao, Cong Liu. Semantic Relation Classification via Hierarchical Recurrent Neural Network with Attention. The 26th International Conference on Computational Linguistics (COLING 2016).

[11] Kun Xu, Yansong Feng, Songfang Huang, Dongyan Zhao. Semantic Relation Classification via Convolutional Neural Networks with Simple Negative Sampling. The 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015).

[12] Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, Zhi Jin. Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths. The 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015).

[13] Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, Houfeng Wang. A Dependency-Based Neural Network for Relation Classification. The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2015).

[14] Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, Zhi Jin. Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentation. The 26th International Conference on Computational Linguistics (COLING 2016).

[15] Rui Cai, Xiaodong Zhang, Houfeng Wang. Bidirectional Recurrent Convolutional Neural Network for Relation Classification. The 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016).

[16] Mike Mintz, Steven Bills, Rion Snow, Daniel Jurafsky. Distant Supervision for Relation Extraction without Labeled Data. The 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing (ACL-IJNLP 2009).

[17] Daojian Zeng, Kang Liu, Yubo Chen, Jun Zhao. Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks. The 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015).

[18] Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, Maosong Sun. Neural Relation Extraction with Selective Attention over Instances. The 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016).

[19] Yi Wu, David Bamman, Stuart Russell. Adversarial Training for Relation Extraction. The 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017).

[20] Jun Feng, Minlie Huang, Li Zhao, Yang Yang, Xiaoyan Zhu. Reinforcement Learning for Relation Classification from Noisy Data. The 32th AAAI Conference on Artificial Intelligence (AAAI 2018).

[21] Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, Maosong Sun. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation. The 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018).

[22] Tianyu Gao, Xu Han, Zhiyuan Liu, Maosong Sun. Hybrid Attention-based Prototypical Networks for Noisy Few-Shot Relation Classification. The 33th AAAI Conference on Artificial Intelligence (AAAI 2019).

[23] Zhi-Xiu Ye, Zhen-Hua Ling. Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification. The 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).

[24] Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, Tom Kwiatkowski. Matching the Blanks: Distributional Similarity for Relation Learning. The 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).

[25] Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou. FewRel 2.0: Towards More Challenging Few-Shot Relation Classification. 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019).

[26] Chris Quirk, Hoifung Poon. Distant Supervision for Relation Extraction beyond the Sentence Boundary. The 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017).

[27] Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, Wen-tau Yih. Cross-Sentence N-ary Relation Extraction with Graph LSTMs. Transactions of the Association for Computational Linguistics (TACL 2017).

[28] Chih-Hsuan Wei, Yifan Peng, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Jiao Li, Thomas C. Wiegers, Zhiyong Lu. Overview of the BioCreative V Chemical Disease Relation (CDR) Task. The 5th BioCreative Challenge Evaluation Workshop (BioC 2015).

[29] Omer Levy, Minjoon Seo, Eunsol Choi, Luke Zettlemoyer. Zero-Shot Relation Extraction via Reading Comprehension. The 21st Conference on Computational Natural Language Learning (CoNLL 2017).

[30] Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, Maosong Sun. DocRED: A Large-Scale Document-Level Relation Extraction Dataset. The 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).

[31] Ruidong Wu, Yuan Yao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun. Open Relation Extraction: Relational Knowledge Transfer from Supervised Data to Unsupervised Data. 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019).

[32] Tianyu Gao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun. Neural Snowball for Few-Shot Relation Learning. The 34th AAAI Conference on Artificial Intelligence (AAAI 2020).

[33] Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, Zhiyuan Liu, Maosong Sun. OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction. The Conference on Empirical Methods in Natural Language Processing (EMNLP 2019).

Machine Heart “SOTA Models”:22Large Fields, 127 Tasks, Covering All Machine Learning SOTA Research.