0 Introduction
C language programming is widely offered as an introductory course in the field of information technology across various universities. Due to differences in student abilities and enthusiasm, ensuring that students effectively master C language poses a significant teaching challenge. With the emergence of generative AI, these tools can utilize conversational models to provide quick answers to questions and rapidly generate code, sparking various opinions and concerns regarding changes in the programming industry and the potential for AI to replace programmers.
1 Overview
Currently, over 90% of domestic universities offer C language programming-related courses, incorporating them as a core part of the training programs for science and engineering majors. Most universities schedule these courses during the first year of enrollment, making the target demographic quite complex. Due to varying prior experiences, students exhibit differences in programming fundamentals, motivation, and knowledge absorption capabilities. To ensure the quality of learning for all types of students, the teaching model for C language programming typically consists of classroom instruction and practical sessions: first, the classroom teaches foundational knowledge, such as concepts and norms related to identifiers; the practical sessions require students to consolidate knowledge through exercises, during which the problems raised demand high levels of logical thinking, patience, and attention to detail from the students.
There are many knowledge points in C language programming, and students often adopt auxiliary strategies to aid their learning. A survey on auxiliary strategies among 100 students currently learning C language programming (hereinafter referred to as C programming) yielded results as shown in Figure 1.
From Figure 1, it can be seen that nearly half of the students chose the “online information query” strategy, relying on search engines like Baidu, Microsoft Bing, or communities like Blog Park and Juejin; “offline consultation” is less popular due to its lower convenience; “others” include less frequently used methods such as social media; while “generative AI” has only recently emerged, it has already become the choice of over one-third of the students with the advent of related tools.
Generative AI typically refers to chatbots built on Generative Pre-trained Transformer (GPT) technology, which are applications of Artificial Intelligence Generated Content (AIGC). These tools can recognize and generate content using AI models, enabling user interaction. In China, Alibaba’s “Tongyi Qianwen”, Baidu’s “Wenxin Yiyan”, SenseTime’s “Shangliang”, iFLYTEK’s “Xinghuo”, and 360’s “360 Zhinao” are all generative AI tools that have entered beta testing.
With its vast knowledge base and capabilities in context understanding and memory, generative AI can perform various complex tasks, such as software programming, answering subject-related questions, and even poetry creation[1]. Additionally, generative AI has demonstrated its capabilities in the field of computer science, assisting in writing articles, speeches, and even generating code from natural language[2]. Furthermore, it can help optimize code or provide suggestions for system development. These capabilities enable researchers to conduct scientific writing more smoothly, thus shortening and accelerating the innovation process[3].
Currently, research on generative AI in academia is still in its early stages, focusing mainly on three aspects: first, analyzing performance capabilities; second, exploring algorithm principles from a technical perspective; and third, examining its applications in digital resources, teaching practices, and information management[4].
2 Investigating Generative AI Capabilities Based on Teaching Practices
This section conducts experiments on generative AI and discusses with programming-related personnel, ultimately deriving conclusions about its capabilities in programming (code generation) and communication (problem-solving)[5].
After generative AI generates programming code, its accuracy is first evaluated using the “Aibiancheng OJ” (Online Judge System, OJ), followed by an assessment of code standards. To ensure rigor and richness, the questions selected are from the final exam of the C programming course at Beihang University (hereinafter referred to as BUAA), consisting of 10 problems with a smooth difficulty curve, and the scores of over 1300 students conform to a normal distribution.
This primarily compares the responses of generative AI and search engines (specifically the widely used Microsoft Bing) to questions related to C language, and analyzes the communication process between programming experiment personnel and generative AI.
To ensure the universality of conclusions, the programming experiment selects well-known tools such as Tongyi Qianwen, Wenxin Yiyan, and Xinghuo, while the communication experiment uses only Tongyi Qianwen for comparison.
The difficulty levels of the problems range from 1 (easiest) to 10 (hardest), with levels 1-3 being easy, 4-7 being medium, and 8-10 being difficult. Specific information about the 10 programming problems is detailed in Table 1.
Four experiment personnel completed the code generation and evaluation for 10 programming problems within 2 hours using three generative AI tools. Here, the highest scores for each problem and the corresponding lowest communication frequencies are summarized in Table 3.
1) Accuracy Analysis.
The results show that generative AI performs excellently on easy problems, but its scores drop sharply as difficulty increases, sometimes even scoring zero; for a maximum score of 450, the ranking among over 1300 reference students was 570, placing it in the top 50%. This indicates that, based solely on accuracy, generative AI is comparable to a beginner-level C language programmer, thus providing significant assistance to students with lower or average scores.
2) Standards Analysis.
Programming proficiency also involves code standards, with standard assessments based on four full-score codes generated by generative AI. Ten professional programmers (doctoral and master’s students from the Software School at BUAA) were selected as judges to evaluate based on naming standards, indentation standards, and symbol standards; average scores are shown in Table 4.
The code standards of generative AI did not vary with the difficulty level of the problems, consistently maintaining a high level. The low score in naming standards was due to the frequent use of ambiguous variable names like i and j; the low score in symbol standards resulted from mixing single and double quotes in character and string content. Overall, its code standards received recognition from professionals, indicating that generative AI is well-suited for improving code quality.
The limitations of generative AI in programming mainly stem from the differences in code accuracy across various difficulty levels.
(1) Easy problems (ABD, concept questions) are simple and can quickly yield high-score code after brief descriptions. Compared to textbooks and the internet, generative AI’s response speed is faster and the code more precise.
(2) Medium problems (CEFG, reasoning questions) mostly involve single knowledge points but require time for consideration and repeated guidance for generative AI to understand. At this point, due to varying user levels, there is a significant difference in code accuracy.
(3) Difficult problems (HI, comprehensive design questions) involve multiple knowledge points and application scenarios, making them difficult for most people to understand, and thus even harder for generative AI to comprehend. Miscommunication can easily lead to incorrect results, making it nearly impossible to answer based on generative AI.
(4) Innovative problems (J, image and text combination questions) typically contain a large number of images and formulas. Due to generative AI’s current proficiency primarily in text-based scenarios (some versions can understand images, but this is not discussed here for generality), it cannot communicate effectively and therefore cannot provide answers.
The communication experiment ultimately yielded 10 responses from Tongyi Qianwen and the search engine, inviting 10 C programming beginners to rate them based on speed (time taken to get an answer), accuracy (whether results matched), and clarity (whether results were organized) on three aspects, with results shown in Table 5.
The high ratings for the search engine demonstrate its excellence as a commonly used auxiliary strategy, while Tongyi Qianwen surpassed the search engine in all three aspects, further enhancing communication efficiency. The specific manifestations of efficiency improvement can be summarized as follows.
1) Reduced User Filtering.
For example, in the response to communication question 2, “C language calculates the Euclidean distance between two points”, the search engine provided several pages of answers, all containing keywords like “C language” and “Euclidean”; generative AI directly elaborated, sequentially providing step analysis, specific code, and precautions. Further study revealed that the search engine gave some inaccurate matching results due to its keyword-based retrieval mechanism; whereas generative AI can accurately understand needs by analyzing text and provide answers from various angles in a concise, comprehensible manner[6]. It can be concluded that as long as the question is clearly expressed, users can obtain accurate answers without the need for filtering.
2) Clearer Results.
Thanks to the vast training data of generative AI, it can reference excellent articles to provide organized content. The second example is the answer generated by generative AI for communication question 4, “C language recursion principles”. It provided a well-structured answer: first explaining the concept of recursion, then giving a code example of a recursive function and explaining its meaning, and finally offering usage suggestions. This organized content can be considered above average compared to internet articles. In contrast to the mixed quality of search engine results, the high-quality answers from generative AI further demonstrate its strong auxiliary capabilities.
Through the previous analysis, it can be validated that generative AI improves communication efficiency, but upon deeper analysis, its limitations gradually become apparent, specifically categorized into three aspects: correctness limitations, information quantity limitations, and user limitations, which will be confirmed with additional tests.
1) Correctness Limitations.
Generative AI does not possess true understanding capabilities; its dialogue model logic operates as if it remembers vast amounts of data, allowing it to provide theoretically accurate answers based on descriptions. This flaw can sometimes lead generative AI to produce misleading incorrect content. For instance, when asked to recommend articles on “path optimization algorithms”, the results it provided are shown in Figure 2.
The list in Figure 2 is quite clear, containing article titles, links, and summaries. However, when attempting to access the link of the first article, the browser redirected to another article as shown in Figure 3.
Even attempts to guide the AI to correct errors may fail. For example, in the programming experiment, the E problem, due to its many details and difficulty in description, resulted in all initial code generations failing to pass the samples, and all attempts to correct the errors also ended in failure, leading to a maximum score of 10 for that problem.
2) Information Quantity Limitations.
As previously mentioned, generative AI dialogues are based on data, so its knowledge scope is directly limited by the range of training data. For example, the popular ChatGPT’s publicly available version during the writing of this article had training data sourced only up to September 2021. Therefore, users of the public version cannot obtain effective answers when inquiring about content after that date, and similar issues exist for various domestic generative AI tools. Although some generative AI can avoid this limitation through real-time data updates, these versions are generally not accessible to the public, making this limitation temporarily unavoidable for the general public.
3) User Limitations.
To effectively avoid the aforementioned limitations and efficiently use generative AI, certain requirements are placed on the user. For instance, in the programming experiment, the highest scores for various problems were almost achieved by the same individual, and even when different participants achieved the same score, their communication frequencies varied. This is due to differences in user capabilities.
Specifically, generative AI requires users to possess professional knowledge and communication skills. When users are experienced and articulate, they can clearly describe problems, aiding AI in accurate understanding. For incorrect answers, they can also guide corrections more quickly, thus increasing the likelihood of obtaining accurate answers. For example, in the response to communication question 5, “What issues should be noted when using linked lists in C language?” the generative AI initially provided a rudimentary list of precautions, scoring poorly compared to the search engine. However, by refining the question to, “What issues may arise when using linked lists in C language, what are the reasons for these issues, and how should they be avoided during use? Please categorize these issues and provide explanations and solutions for each reason,” a high-quality answer was obtained as shown in Figure 5. It can be seen that while long-form questions are low-quality for search engines, they are high-quality for generative AI. The improvements in the analysis perspective and solution descriptions are evident; although lacking code, simply adding “provide corresponding code examples” at the end would resolve that.
Due to the existence of this limitation requiring users to possess a certain level of capability, less skilled users should pay attention to verifying answers to avoid repeated mistakes; however, as users continuously improve their abilities, they can mitigate this limitation to some extent.
3 Traditional Teaching Models Cannot Be Replaced
From the previous analysis, it can be concluded that the capabilities of generative AI stem from AI models trained on data. They can simulate human dialogue and exhibit relatively complete logic[7], but they do not possess true understanding; they may sometimes provide seemingly logically sound but actually incorrect answers. In addition to the three communication limitations mentioned above, generative AI also has programming limitations: it may produce incorrect code due to vague expressions and may even struggle to communicate effectively on problems that are difficult to describe in text. Therefore, the viewpoint of “generative AI replacing traditional programming education” cannot be upheld.
In practice, C language programming teachers aim to provide guidance tailored to students of varying levels, often using a systematic approach and knowledge maps to help students understand: starting from a point and continuously extending until knowledge is fully conveyed. A comparison between a portion of the “C Language Knowledge Map” obtained from CSDN and the “Knowledge Map” generated by generative AI is illustrated in Figure 6. The responses from generative AI demonstrate that systematic summaries like knowledge maps are currently beyond its capabilities: first, because the map system is difficult to describe in text; second, because knowledge systems must be continuously refined with professional knowledge, requiring a high level of expertise from users. It can be concluded that generative AI cannot genuinely help students fully understand the programming world; students still require teachers’ assistance.
3.2 The Irreplaceable Role of Teachers as Guides
Combining the metaphor of foreign researcher Wollowski[8], current generative AI resembles a less capable upperclassman; it can remember most knowledge but lacks clarity in understanding, confidently providing potentially incorrect answers. In contrast, traditional education involves teachers who are responsible for teaching and nurturing students, strictly controlling teaching content, and continuously improving based on feedback to ensure accuracy before imparting knowledge.Unlike generative AI, which may return false information[9], the rigor of teachers is something these “upperclassmen” lack.
Moreover, due to the powerful answering capabilities of generative AI, the phenomenon of students completely relying on it to complete assignments has become widespread globally, prompting institutions like New York University and the University of Tübingen to ban its use. This phenomenon arises from human laziness—”it is easy to transition from frugality to extravagance, but difficult to revert from extravagance to frugality”—as people are reluctant to revert to less efficient methods (learning summaries) once efficient means (intelligent generation) become available. This mindset is perilous; it not only dampens students’ enthusiasm for knowledge exploration but also leads to catastrophic stagnation in education. Conversely, while shouldering the heavy responsibility of education, teachers can also clearly recognize the significance of practical learning[10] and guide students’ moral qualities through communication, ensuring they resist temptation and truly understand the meaning of learning.
In summary, the role of teachers as guides is undoubtedly irreplaceable, reflecting the fundamental position of traditional education.
Currently, although generative AI has limitations, it indeed possesses numerous capabilities to improve C language programming education, serving as an important tool for both students and teachers.
In theory, generative AI has mastered all knowledge of the C language and can communicate clearly about basic content, providing the following assistance.
1) Rapid Learning of Programming.
Generative AI can provide clear introductions to various knowledge points, aiding beginners in memorization. Referencing its responses to communication questions 2 and 4, these high-quality answers containing concepts, examples, and precautions help students quickly grasp unfamiliar knowledge points and review old ones. The explanations of code logic help students develop programming thinking, and the high standards of the code also assist students in forming good programming habits after comprehension.
2) Intelligent Assistance in the Learning Process.
Thanks to the many excellent articles and programming problems within generative AI, it can quickly formulate suitable learning plans and reference exercise lists based on students’ situations, facilitating an efficient learning journey from simple to complex, as shown in Figure 7.
In summary, generative AI can be seen as a comprehensive improvement over traditional online query strategies, serving as an efficient tool to safeguard the learning process and significantly enhance learning efficiency.
1) Teaching Phase.
Generative AI can assist teachers in finding relevant articles, videos, and other materials for courses, and can explain concepts or network vocabulary in areas where teachers may lack expertise, thereby accelerating lesson preparation and optimizing lecture content, overall improving teaching quality. Additionally, its rapid responses to basic questions can significantly shorten Q&A time, helping teachers better plan lessons and explain core content.
2) Practical Phase.
For specific after-class practical sessions, generative AI includes problems from major programming platforms (e.g., LeetCode, CodeForce), allowing teachers to quickly obtain references through keywords, thereby rapidly generating quality problems. Preliminary estimates suggest that with the assistance of generative AI, the average time to create a routine practice problem can be reduced from half an hour to 10 minutes. As long as teachers keep pace with the times, teaching efficiency will continue to improve with the aid of this reliable assistant.
4 Conclusion
Based on the result analysis, it can be concluded that generative AI has numerous limitations; the viewpoint that it will disrupt the programming industry is overly exaggerated and, at least for now, unfounded. However, it does possess excellent programming and auxiliary capabilities, which teachers and students can leverage to enhance their work and learning efficiency.
For programming educators and learners, generative AI presents an opportunity; it can serve as a reliable assistant and efficient tool to promote reforms in programming education. Generative AI can be applied at various stages of teaching—before, during, and after classes—driving education to become more intelligent and efficient, while also further demonstrating the advantages of traditional education and the irreplaceability of human involvement. This new model of “AI + Programming Education” is undoubtedly the future of programming teaching and can also serve as a pioneer for the practical application of generative AI.
References:
[1] ZDNET. ChatGPT can write code. Now researchers say it’s good at fixing bugs, too[EB/OL]. [2023-06-11]. https://www.zdnet.com/article/chatgpt-can-write-code-now-researchers-say-its-good-at-fixing-bugs-too/.
[2] Cornell University. Evaluating the code quality of AI-Assisted code generation tools: An empirical study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT[EB/OL]. [2023-04-21]. https://arxiv.org/abs/2304.10778.
[3] Van D E A M, Bollen J, Zuidema W, et al. ChatGPT five priorities for research[J]. Nature, 2023, 614(7947): 224-226.
[4] Qian Li, Liu Yi, Zhang Zhixiong, et al. Technical foundation analysis of ChatGPT[J]. Data Analysis and Knowledge Discovery, 2023, 7(3): 6-15.
[5] Zhang Yihua, Zhao Dong, An Yantao, et al. Application of flipped classroom based on Rain Classroom in professional course teaching[J]. Higher Education Journal, 2019(12): 105-107.
[6] Rahman M M, Watanobe Y. ChatGPT for education and research: Opportunities, threats, and strategies[J]. Applied Sciences, 2023, 13(9): 5783.
[7] Stokel W C, Noorden R V. The promise and peril of generative AI[J]. Nature, 2023, 614(7947): 214-216.
[8] Wollowski M. Using ChatGPT to produce code for a typical college-level assignment[J]. AI Magazine, 2023(44): 129-130.
[9] Wang Tian’en. The characteristics, educational significance, and problem responses of ChatGPT[J]. Ideological Theory Education, 2023(4): 19-25.
[10] Cornell University. Exploring the use of ChatGPT as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges[EB/OL]. [2023-04-16]. https://arxiv.org/abs/2304.11214.
（End）
More Exciting:
Principal Interview｜Rooted in Border Minority Areas, Focusing on Teacher Education to Cultivate High-Quality Application-Oriented Talents——Interview with Principal Chen Benhui of Lijiang Normal University
Yan Ten│Review and Prospects of Computer System Capability Cultivation
Discussion on the Concept of “Student-Centered” Teaching and Its Implementation Path
Principal Interview｜Promoting Interdisciplinary Integration to Cultivate Innovative Talents in the New Era——Interview with Professor Ni Mingxuan, Founding Principal of Hong Kong University of Science and Technology (Guangzhou)
New Year Message from the Seventh Editorial Committee
Guidelines for Ideological Education in Computer Science Courses
Academician Chen Guoliang｜Cultural Construction of Virtual Teaching and Research Office for Computer Course Ideological Education
Professor Chen Daoxu of Nanda｜Change and Constancy: The Dialectic of Learning Processes
Yan Ten│Reflections and Suggestions on the