Editor丨Wang Duoyu
Typesetting丨Shui Chengwen
With the rapid development of artificial intelligence (AI) technology, especially the advances in large language models (LLM), generative AI is becoming increasingly widespread in academic writing.
Recently, researchers from the University of Waterloo and other institutions published a preprint paper titled: Generative AI in Academic Writing: A Comparison of DeepSeek, Qwen, ChatGPT, Gemini, Llama, Mistral, and Gemma, which aims to evaluate the performance of several mainstream large language models (including DeepSeek v3, Owen 2.5 Max, ChatGPT, Gemini, Llama, Mistral, and Gemma) in academic writing, particularly their ability to generate high-quality academic content.
This paper provides a detailed assessment and comparison of the application of generative AI in academic writing, offering important references for future research.
The authors of the paper used 40 academic articles on the topics of “digital twins” and “healthcare” to generate text using generative AI tools and evaluated the generated text in the following ways:
1. Plagiarism detection: Using the iThenticate tool to check the plagiarism rate of the text.
2. AI detection: Using StealthWriter.ai and Quillbot.com to detect whether the text was generated by AI.
3. Word count comparison: Comparing the word counts of texts generated by different large language models.
4. Semantic similarity: Evaluating the semantic similarity of the generated text to the original using ChatGPT, DeepSeek v3, and Owen 2.5 Max tools.
5. Readability assessment: Assessing the readability of the text using Hemingway Editor, Grammarly, and WebFX tools.
The results showed:
In terms of cost-effectiveness, DeepSeek v3 offers the best value for money.
In terms of text generation capability, Owen 2.5 Max and DeepSeek v3 generated the most words, with more detailed content, while Mistral 7B and Deepseek-coder-v2 16B produced more concise texts.
In terms of plagiarism rate, the text generated by ChatGPT 4o mini had the highest plagiarism rate (57%), while Llama 3.1 8B had the lowest (9%).
In terms of AI detection, almost all large models’ generated texts could be identified as AI-generated by AI detection tools, varying in the extent of AI traces.
In terms of readability, all large models’ generated texts performed poorly, especially with generally low scores from the Hemingway Editor.
In terms of semantic similarity, all large models’ generated texts maintained high semantic similarity to the original text, particularly with Owen 2.5 Max and DeepSeek v3 showing the most consistent performance.
The research indicates that Owen 2.5 Max and DeepSeek v3 perform well in academic writing tasks, especially in generating detailed content. However, different models have their advantages in different scenarios, such as Llama 3.1 8B performing better in plagiarism rate and readability. Future research could further explore how to optimize these models to enhance the quality and readability of generated texts and reduce traces of AI generation.
The authors further pointed out that future improvements in the performance of generative AI in academic writing could include expanding datasets, using larger datasets to evaluate the generalization ability of models; exploring how to more effectively combine AI tools with human users to improve the quality of academic writing through human-computer collaboration; optimizing the model generation process to reduce traces of AI generation, making texts more natural. Additionally, further research is needed on the ethical and legal issues of AI-generated content (e.g., copyright, citation rules, etc.).
Paper link:
https://www.researchgate.net/publication/388681921
Click to view, share your taste