How to Write a Qualified NLP Paper

Reprinted with permission from Liu Zhiyuan’s Zhihu column

Author: Liu Zhiyuan

https://www.zhihu.com/people/zibuyu9/posts

A few days ago, I just finished the ACL 2019 submission season, providing revision suggestions for many students’ papers. Many of these papers, especially those of beginners, have similar issues. Thinking about repeating these words to more new students in the future, I decided to summarize these suggestions so that I can save some breath later and perhaps help more students. Thus, this short article was born.

This article is titled “Qualified” papers rather than beautiful or brilliant papers. One reason is that I know my English level, especially my vocabulary, is limited, and I have never written a paper that I consider brilliant or beautiful, so I am not qualified to provide advice in that regard. Another reason is that, as will be discussed below, the key objective of academic papers is not to be beautifully worded but to be clear and accurate; I have accumulated quite a bit of experience in this area. With this experience, I believe that achieving “clarity” is not difficult, while “elegance” depends on one’s ability.

In fact, Liu Yang from the same group has given a very comprehensive and brilliant report on writing academic papers in NLP [1], which I strongly recommend all NLP students to read carefully, as it will save you from many detours in research. This article can be seen as a footnote or supplement to that report.

NLP is a field that emphasizes practice and application, where innovative results can be new algorithms, tasks, applications, data, discoveries, etc., with the aim of achieving something “new”. Its impact depends on its role in promoting the development of the field. As shown in the figure below, academic research is a systematic project that includes multiple links, all working together to pursue “innovation”: the problem must be challenging, the model must be innovative, the implementation must be accurate, and the experiments must be in-depth.

In this systematic project, the role of a paper is to clearly and accurately describe the innovative points, technical ideas, algorithm details, and validation results to peers in the academic community. Understanding this is crucial for properly approaching paper writing: a work that is unremarkable is unlikely to become a star through writing; a work full of innovation may fail to convey its important value to reviewers due to poor writing, delaying its publication.

Papers for NLP academic conferences (including journals) have formed a relatively fixed structure. The vast majority of papers consist of the following six main parts: Abstract, Introduction, Related Work, Method, Experiment, and Conclusion. A few papers may vary slightly based on the form of innovative results, such as papers proposing new datasets, which may adjust the Method section to focus on Dataset annotation and analysis, but this does not affect the overall composition of the paper. Each part serves a different purpose:

• Abstract: A brief introduction of 100-200 words about the research task and challenges, solution ideas and methods, experimental effects, and conclusions.

• Introduction: A more detailed introduction of about one page that describes the research task, existing methods, main challenges, solution ideas, specific methods, and experimental results.

• Related Work: An introduction of about 0.5-1 page that discusses related work on the research task, explaining the similarities and differences between this work and existing work.

• Method: An introduction of 2-3 pages that details the model and methods proposed in this paper.

• Experiment: An introduction of 2-3 pages that describes the experimental setup, datasets, experimental results, analysis, and discussion that validate the effectiveness of the proposed methods.

• Conclusion: A simple summary of the main work of this paper and an outlook on future research directions.

At first glance, this structure may seem rigid, but it actually highlights the true significance of academic papers, which is not to surprise readers with form but to focus their attention on the research results presented in the paper.

As mentioned earlier, the role of a paper is to clearly and accurately describe the innovative points, technical ideas, algorithm details, and validation results to peers in the academic community. Due to the peer review system in academia, the thread and goal throughout the paper must demonstrate the innovative value of this work, with each part serving that goal. To achieve this, authors must pay special attention to the following points:

1. Learn to Empathize.Always view the paper from the perspective of the reviewer or reader, thinking about how to express more clearly. This is the most easily overlooked issue for beginners: as the firsthand experiencers of the research results, authors know all the details. If not careful, writing may include new concepts that are used without clear definitions, and many descriptions and analyses lack logical connections. For the author, these omissions do not affect their understanding of the text; however, for readers who are not familiar with this work, it is undoubtedly a nightmare, as they do not have the background information that exists in the author’s mind. Therefore, while writing, one must always be alert to whether the reader can understand a sentence and whether the necessary background knowledge has been introduced earlier.

2. Be Logically Rigorous.Rigor is the foundation of academic papers, from citation formats and formula symbols to the crafting of sentences. While it may not be as extreme as the Bluebook for American law journals, strive for stylistic consistency and rigorous writing. Citation, formulas, spelling, etc., are relatively easy to learn, but beginners need to pay special attention to maintaining logical rigor at all levels, from chapters, paragraphs, to sentences, aiming for no sentence to be without reason and no sentence to lack a response:

• At the chapter level, if the Introduction mentions several challenges faced by existing methods, it should correspond to the several innovative ideas proposed in this paper, which should link to specific algorithms in the Method section and to several experimental validations in the Experiment section.

• At the paragraph and sentence level, attention should be paid to coherence between paragraphs, whether they are parallel, progressive, contrasting, or summarizing relationships, which need to be planned appropriately, with corresponding sentences or adverbs connecting them. Within a paragraph, there should be a central idea sentence and supporting sentences working together.

In addition to overall structural suggestions, each section has its own conventions. Below, I will provide some writing suggestions for each section, using one of our recently published ACL 2018 papers [2] as an example.

The Abstract can be seen as a summary of the Introduction, so we will first introduce how to write the Introduction and then discuss how to write the Abstract. The Introduction is a comprehensive introduction to the entire work and is crucial in determining whether a paper will be accepted. Generally, the Introduction is structured as follows: start by introducing the research task and its significance; then briefly introduce existing methods related to this task; next, explain the key challenges faced by existing methods; in response to these challenges, what innovative ideas and specific methods does this paper propose; finally, introduce experimental results that prove the effectiveness of the proposed methods. Each part serves its purpose while maintaining a strict internal logic. Each part also has its own rules, which will be introduced below:

1. Research Task.Introduce the research task of this paper and its importance and significance in the research field. If it is a widely recognized important task, detailed discussion of its value/significance may not be necessary; if it is a newly proposed research task, more space should be devoted to arguing its value. The first paragraph of the paper [2] focuses on explaining the reading comprehension research task.

2. Existing Methods.Progressing from the research task, introduce the representative existing methods for this task. In the second paragraph of paper [2], the discussion begins with DS-QA. It is important to note that the existing method introduced should be the best and most representative one that this paper intends to improve. The so-called standing on the shoulders of giants means that a paper worthy of publication must find the highest giant.

3. Challenges Faced.The existing methods must still have certain shortcomings or challenges that necessitate further research and improvement. Therefore, it is necessary to summarize the challenges faced by existing methods. This is a key part of the Introduction, serving as a bridge. Beginners should pay special attention to this part, as it involves evaluating existing work, which must be precise and objective. It should be noted that when a paper is submitted to an NLP international conference, acceptance is determined through peer review, and reviewers are often small peers, with a high probability of being authors of existing work. Therefore, this part must be objective and fair, ensuring that even the authors of those works can be convinced.

In the third and fourth paragraphs of paper [2], the challenges of noisy labeling faced by DS-QA are introduced, and examples are provided for intuitive presentation. In response to this challenge, some related works have been mentioned, and their respective shortcomings and challenges should also be explained to lay the groundwork for the innovative ideas of this paper.

4. Innovative Ideas.When there are shortcomings and challenges in existing methods, new innovative ideas and methods are needed. This part needs to closely correspond with the previous “Challenges” section, clearly showing the readers that these innovative ideas and methods can indeed solve or alleviate these challenges.

The fifth paragraph of paper [2] introduces the innovative ideas and methods. Generally, the sections on “Challenges” and “Innovative Ideas” will include illustrations to visually present the challenges and innovative ideas that this paper aims to address. For example, the ugly diagram in paper [2] intuitively shows that the innovative methods include two modules: Selector and Reader, and their functions. You can also look at our other papers [3]; most of them will provide illustrations in the Introduction.

6. Experimental Conclusions.In addition to visually illustrating the innovative work in the “Innovative Ideas” section, it is also necessary to validate the effectiveness of the methods through reasonable experimental verification. Generally, it should conclude that “our method achieves significant and consistent improvement compared to other baselines,” thereby validating the innovativeness of this work.

Some papers will also summarize the main contributions at the end, generally stating, “In summary, the key contributions are x-fold: (1)…(2)…(3)…”. The advantage of doing this is that it helps reviewers summarize the innovative points of this paper in their review comments, saving a lot of work.

However, it is important to note that these innovative points should be concise and clear, not merely a repetition of previous content, and should not overclaim. If stating “for the first time” or “discovered,” it is generally necessary to prepend “to the best of our knowledge.” Additionally, the last paragraph of the paper will often outline the structure of the following sections, but I feel that this may not be necessary for an 8-page paper.

For the Abstract, it can be viewed as a brief introduction to the Introduction. The simplest approach is to condense each part into 1-2 sentences to form the Abstract. Below is the Abstract content of paper [2], which shows the correspondence with the Introduction.

This section should detail the specific details of the innovative methods proposed in this paper. Due to the involvement of very intricate details, a “general-specific” structure should be adopted for the introduction.

The “general” part at the beginning should introduce the symbol definitions of the task in this paper, as well as the framework composition of the proposed methods, either by steps or by modules, allowing readers to have a panoramic understanding of the methods proposed. The “general” part of the Methodology in paper [2] first introduces some symbols and then describes the main functions of the two modules: Selector and Reader.

Then, in the “specific” part, it is necessary to correspond to the framework in the “general” part and introduce the details of each key module/step. For example, the “specific” part of the Methodology includes 3.1 Paragraph Selector, 3.2 Paragraph Reader, and 3.3 Learning and Prediction. Readers have already gained a panoramic understanding of the methods in the “general” part, making it easier to understand the specific details of each module.

In each “specific” part, a “general-specific” structure can be further adopted. For example, after providing an overall introduction in section 3.1, the introduction can be further divided into Paragraph Encoding and Question Encoding. To clearly reflect the “general-specific” structure, the titles of each “specific” part can be named and bolded.

Beginners should pay special attention to: (1) The introduction of innovative ideas and methods in the Introduction should not be simply repeated in the Method; otherwise, it will annoy reviewers who read the entire paper thoroughly. The writing should be coherent, with progression, and it is advisable to use phrases like “as mentioned in Section 1” to make connections. (2) The Method section often contains a large number of formulas, and it is necessary to ensure that the style of formulas and the use of symbols are consistent throughout; any new symbols must be explicitly explained.

This section should detail the specific details related to the experiments. Generally, it begins by introducing the experimental data, evaluation criteria, and comparison methods.

Taking paper [2] as an example, the experimental section first introduces the experimental data and evaluation criteria (4.1 Datasets and Evaluation Metrics), the representative methods for comparison (4.2 Baselines), and the parameter settings for the experimental methods (4.3 Experimental Settings).

After introducing the basic information of the experiments, two types of experiments are generally conducted:

(1) Main Experiments. The purpose is to demonstrate the effectiveness of the proposed methods compared to existing methods. Generally, it is necessary to select widely recognized datasets or experimental validation methods used in existing work to enhance the credibility of the experiments. For academic papers, it is not necessary to compare with the best methods for the task; it is sufficient to prove that the innovative methods proposed in this paper are more effective than not using them. In other words, the experiments should control other variables as much as possible, focusing only on the challenges addressed in this paper. Of course, if the innovative ideas lead to the best results for the task, it would be more attractive, but it is not always necessary to demand that.

Generally, experimental results are presented in tables and figures, followed by observations and analyses in the text. For example, the main experimental section of paper [2] first discusses the effects of different Selectors and Readers on experimental results (4.4 Effect of Different Paragraph Selectors, 4.5 Effect of Different Paragraph Readers), and then introduces the main experimental results and observations (4.6 Overall Results). The best results are typically highlighted in bold, and most should be from the methods proposed in this paper. To make the conclusions clearer, they can be listed as (1), (2), (3), with the first point generally being the main conclusion that the proposed method significantly outperforms existing methods.

Main Experimental Results

Main Experimental Analysis

(2) Auxiliary Experiments. The purpose is to demonstrate the advantages and features of the innovative methods proposed in this paper. For example, the effects of different hyperparameters on the proposed methods (Hyper-Parameter Effect), the contributions of different modules to the effectiveness of the proposed methods (Ablation Test), the impact of different data partitions on the proposed methods (common in Few-shot Learning-related works), the main error types of the proposed methods (Error Analysis), and typical examples where the proposed methods can improve results (Case Study), etc. These experiments need to be designed specifically based on the characteristics of the innovative work presented in the paper, serving to highlight the innovative value of this paper.

For example, the auxiliary experiments in paper [2] include 4.7 Paragraph Selector Performance Analysis, 4.8 Performance with Different Numbers of Paragraphs, 4.9 Potential Improvement, and 4.10 Case Study, showcasing the characteristics of the proposed methods from various perspectives.

The Experiment section should be rich in visuals, emphasizing the advantages and characteristics of the proposed methods through multiple tables and figures, with attention paid to the uniformity of the style of the figures and tables. Beginners should pay special attention to ensuring that each table’s content can be understood solely based on the explanatory text below the table, without requiring readers to search the main text for related explanations. Many experienced reviewers, after reading the Introduction, will directly jump to the Experiment tables to look for comparative results.

1

This section mainly introduces the related work of the tasks and methods of this paper, aiming to highlight the innovative value of this work through a review of existing works. The review of existing works should not be a simple introduction to each work but should focus on summarizing, categorizing, and analyzing, either chronologically or by technical routes, as shown in paper [2], which introduces the timeline.

In introducing related works, it is important to align with the challenges that the innovative ideas of this paper aim to address. The review should not be purely descriptive but should include commentary, always keeping in mind the connection with this work. At the end of the Related Work section, it should conclude with the new ideas of this work compared to existing works and the challenges it addresses.

Beginners should pay special attention to the fact that the Introduction and Related Work sections require careful oversight from mentors or experienced scholars. First, important related works should not be overlooked; this requires the authors to keep track of relevant work in the field. Second, similar to the Introduction, evaluations of existing works must be precise and objective.

Related Work is generally placed after the Introduction or before the Conclusion, depending on the characteristics of the paper. For works closely related to existing works with subtle innovations, it is generally recommended to place it after the Introduction, allowing readers to fully understand the relationship between this work and existing works before moving on to the Method section. For some framework-based innovative works, if the main contribution is a combination of existing methods, it is generally recommended to place Related Work after the Method and Experiment sections. There are no strict rules; it is entirely based on the convenience of writing.

At the end of the paper, there will be a summary and outlook, generally using one paragraph to summarize and emphasize the innovative ideas and experimental results of this paper, followed by a discussion of suggested future research directions and open questions. This part is relatively fixed. It is worth noting that during the final stages of preparing the paper, if it is found that there are things that should have been done but have not yet been completed, one can write about future work in this paper. At the very least, this shows reviewers that you have also considered this issue, gaining some sympathy.

To write a qualified NLP paper, the first issue is attitude; only by valuing the attitude can one persistently revise and seek various ways to improve the paper (e.g., seeking help from seniors or foreign teachers, using tools like Grammarly, etc.). Secondly, it is a matter of taking action; only by writing can one continuously revise and improve. Lastly, it is about experience; writing brilliantly may require talent, but writing adequately only requires persistence and continuous reflection and revision based on feedback from reviewers and others. In short, persistence is victory.

In fact, I believe that writing papers is a training of thinking patterns. Perhaps in the future, you may not engage in academic research, but the ability to refine the innovative value of work and clearly convey complex information developed through writing will be of significant help in future work, whether in communication or presentation of results. Therefore, I hope everyone can value this rare training opportunity on the research path. Keep it up!

Writing papers has many details and techniques that need attention, and many fields even have thick guides dedicated to writing skills. This short article cannot cover everything; it only introduces some of the suggestions I frequently provide to students preparing papers, hoping to be useful to everyone. I will update any new suggestions whenever I think of them. Feedback and questions are also welcome for mutual progress.

Related Links:

1. Liu Yang. Methods and Techniques for Writing Academic Papers in Machine Translation:

http://nlp.csai.tsinghua.edu.cn/~ly/talks/cwmt14_tut.pdf

2. Yankai Lin, Haozhe Ji, Zhiyuan Liu, Maosong Sun. Denoising Distantly Supervised Open-Domain Question Answering. ACL 2018:

http://nlp.csai.tsinghua.edu.cn/~lzy/publications/acl2018_qa.pdf/

3. Personal Homepage:

http://nlp.csai.tsinghua.edu.cn/~lzy/publication.html

Original link:

https://zhuanlan.zhihu.com/p/58752815