Application Of Multimodal Artificial Intelligence In Nursing Education

Nursing education plays a key role in preparing future nurses to meet complex challenges in healthcare services. However, traditional nursing education faces numerous challenges due to limitations in clinical environments, time constraints, and limited resources. In recent years, artificial intelligence technology, especially multimodal artificial intelligence, has been transforming the teaching and learning processes in various educational environments^[1-3]. In nursing education, multimodal artificial intelligence has shown great potential in creating interactive simulation environments, providing personalized learning experiences, achieving knowledge integration and collaborative learning, as well as in cross-language and cross-cultural education. However, multimodal artificial intelligence also faces many challenges in data security and privacy protection, technical training and usage costs, instructional design, and curriculum development. Therefore, this paper explores the application of multimodal artificial intelligence in nursing education and analyzes its future development trends.

1. Concept of Multimodal Artificial Intelligence

Multimodal artificial intelligence is an advanced artificial intelligence system that integrates multiple perceptual modalities and processing capabilities^[4]. Traditional artificial intelligence systems are usually limited to processing specific types or single perceptual sources of data, such as text data or image data. In contrast, multimodal artificial intelligence systems can simultaneously process multiple data modalities from different perceptual sources, including text, speech, images, videos, etc., achieving more intelligent data processing and decision-making through comprehensive analysis and cross-validation of these data^[5].

The advantage of multimodal artificial intelligence systems lies in their ability to comprehensively consider multiple sources of information, allowing for a more thorough understanding and processing of data. By integrating different perceptual modalities, the system can more accurately recognize, understand, and analyze things, achieving more detailed and in-depth data mining and knowledge discovery. For example, in the medical field, multimodal artificial intelligence systems can simultaneously analyze patients’ clinical records, imaging data, and physiological signals, helping doctors provide more accurate diagnoses and treatment plans^[6-8].

2. Application of Multimodal Artificial Intelligence in Nursing Education

Multimodal artificial intelligence has enormous application potential in nursing education. By combining visual, auditory, and verbal modalities, it can provide nursing students with a more comprehensive and in-depth learning experience, thereby improving their learning outcomes and skill levels.

Multimodal artificial intelligence systems can integrate virtual reality or augmented reality technologies to create interactive simulation environments. Students can engage in clinical skills simulation training in these environments, such as simulated injections and bandaging^[9]. This interactive learning model allows students to practice repeatedly in a safe virtual environment, enhancing their practical experience and improving learning outcomes^[10]. The system records students’ operational data in real-time and provides immediate feedback based on the accuracy and efficiency of their actions, helping students continuously optimize their skills. Wang Ying et al.^[11] applied virtual simulation technology combined with Rain Classroom in stoma care practical teaching, showing that this combination enhances students’ autonomy in learning, stimulates interest, cultivates clinical thinking, and effectively improves the quality of stoma care teaching. Li Longti et al.^[12] applied multimodal virtual reality technology in orthopedic nursing clinical practice teaching, demonstrating that multimodal virtual technology helps improve nursing students’ theoretical knowledge and clinical operational skills. Although multimodal artificial intelligence combined with virtual and augmented technologies is beneficial for strengthening students’ operational skills, the established fixed operational processes differ from actual clinical practices, and virtual patients cannot completely replace real patients; attention should be paid to strengthening students’ humanistic care concepts and communication skills with patients.

Multimodal artificial intelligence systems, by integrating video, audio, and sensor data, can achieve more refined scenario reproduction and emotion recognition. By simulating complex patient care situations, the system can capture and analyze patients’ emotional changes, physical responses, and verbal expressions, helping caregivers better understand patients’ needs and emotional states^[13]. In this multimodal simulation environment, the emotions of virtual patients can change in real-time with the caregivers’ actions and communications. By analyzing audio and video data, the system can recognize patients’ vocal emotions, expression changes, and body language, simulating patient interactions in real clinical situations^[14]. Muzammel et al.^[15] used deep neural networks for multimodal clinical depression recognition, with results showing that the best-performing architecture successfully detected depression using less than 8 seconds of voice clips; Aslam et al.^[16] introduced a new framework for “attention-based multimodal sentiment analysis and emotion recognition,” with experimental results confirming that the framework achieved accuracy rates of 85% and 93% in sentiment analysis and emotion classification, respectively. In recent years, multimodal sentiment analysis and emotion recognition have gained increasing attention due to their wide practical applications, but the current development of multimodal artificial intelligence technologies for emotion recognition is still immature. Future efforts should focus on developing accurate feature extraction technologies for emotions in nursing contexts and their application in real clinical settings.

Multimodal artificial intelligence systems can provide personalized learning resource recommendations based on students’ learning habits, cognitive styles, and learning needs to enhance learning effectiveness and interest. By comprehensively analyzing students’ performance data in multimodal learning environments, the system can gain insights into students’ learning preferences and behaviors, tailoring targeted learning resources for them^[17]. Additionally, the system can recommend learning content suitable for students’ levels and interests based on their performance in different subjects or skills, helping them master knowledge and skills more efficiently. Chen Mengyue et al.^[18] applied a personalized learning model based on vocational education cloud platforms in the teaching of “Adult Nursing,” showing that this personalized learning model helps improve students’ knowledge mastery, teaching satisfaction, and autonomous learning abilities. Artificial intelligence injects new momentum into the development of nursing education by meeting personalized learning support needs, and future explorations should focus on key technologies, main pathways, and effectiveness evaluations for personalized learning in nursing education.

Knowledge integration and collaborative learning are important concepts in nursing education, and multimodal artificial intelligence provides new possibilities for achieving the organic combination of theoretical knowledge and practical skills^[19]. By integrating and jointly utilizing teaching resources from multiple modalities such as text, images, and videos, multimodal artificial intelligence can offer nursing students a richer, more vivid, and three-dimensional learning experience, promoting their comprehensive mastery of nursing knowledge and skills. In the process of knowledge integration, multimodal artificial intelligence can combine theoretical knowledge with clinical practice, breaking the disconnect between theory and practice in traditional teaching, helping students better understand the relationship between abstract concepts and practical operations. Zhang Shan et al.^[20] used Python crawling technology and natural language processing technologies to construct a “four-dimensional” knowledge graph of “course objectives + basic knowledge layer + problem system layer + teaching resource layer” in “Internal Medicine Nursing,” aiming to help students master the basic theoretical knowledge of the course and enhance their ability to solve problems using knowledge. To achieve knowledge integration and collaborative learning in nursing education, educators need to improve their capabilities in artificial intelligence and closely integrate with clinical practice to maintain the timeliness and practicality of teaching content.

In the field of cross-language and cross-cultural education, multimodal artificial intelligence demonstrates enormous potential and value. For nursing students from different cultural and linguistic backgrounds, the system can utilize advanced technologies such as speech recognition, natural language processing, and machine translation to provide cross-language learning resources, facilitating their smoother integration and learning of nursing knowledge^[21]. At the same time, the system can offer opportunities for cross-cultural communication and collaboration, promoting understanding and respect between different cultures and cultivating students’ cross-cultural communication skills and social adaptability. Research by Long et al.^[22] indicates that students interacting with AI teaching models exhibit greater confidence and inclusivity in cross-cultural communication. Research by Zhong Wenxi et al.^[23] shows that generative artificial intelligence can provide diverse language expressions and cultural habits, enhancing students’ cultural awareness and cultural intelligence, improving their cultural inclusivity, and enhancing their communication skills in multicultural environments. Future multimodal artificial intelligence should focus on using diverse and inclusive datasets to ensure coverage of different cultures and contexts, improving understanding and accuracy of various cultural backgrounds and contexts, and ensuring that the outputs of intelligent models meet the requirements of cultural sensitivity.

3. Challenges Faced by Multimodal Artificial Intelligence in Nursing Education

Despite the existing advantages and future potential of multimodal artificial intelligence, its application in nursing education is still in the exploratory and developmental stages, facing some challenges and limitations.

Multimodal artificial intelligence integrates data from various modalities, including visual, auditory, and tactile, and its development and maintenance involve highly complex technologies^[24]. The complexity of technology not only requires educators, students, and technical personnel in nursing education to possess certain knowledge of artificial intelligence but also necessitates reliable technical support and resources. Therefore, it is essential to establish multidisciplinary teams, including educators, technical experts, and medical professionals, to promote cross-disciplinary cooperation. Through teamwork, they can collaboratively solve technical issues, design course content, and evaluate teaching effectiveness.

Multimodal artificial intelligence requires access to and processing of large amounts of personal information and medical data, including privacy data of students and patients. Ensuring data security and privacy protection, as well as adhering to ethical norms, is a significant challenge^[25-27]. Therefore, it is crucial to implement strict data encryption and access control policies to ensure that all sensitive data is adequately protected. Data processing should be conducted within the framework allowed by laws and regulations, and all personnel involved in data management should receive training on data security and privacy protection.

There are also challenges regarding technical training and usage costs for multimodal artificial intelligence in nursing education. Teachers and students need to undergo relevant technical training to familiarize themselves with the application methods and operational processes of multimodal artificial intelligence technologies. However, the investment costs for training faculty and facilities must be considered. Additionally, using multimodal artificial intelligence technologies requires supporting devices and software, such as virtual reality equipment and intelligent simulators. Moreover, with the continuous updates and upgrades of technology, funding will also be needed to update devices and software to ensure teaching quality and sustainability. To overcome these challenges, educational institutions can seek external cooperation and funding, such as collaborating with companies or institutions to co-build laboratories or provide equipment support. Furthermore, adopting cloud services and other technological means can help reduce costs and improve efficiency.

4. Future Trends of Integrating Multimodal Artificial Intelligence into Nursing Education

The application of multimodal artificial intelligence in nursing education can provide nursing students with a more comprehensive and in-depth learning experience, enhancing their clinical operational skills, decision-making abilities, and nursing communication techniques. With the development of technology and the growth of demand, the future trends of integrating multimodal artificial intelligence into nursing education mainly include the following aspects.

There is an urgent need to construct curricula integrating multimodal artificial intelligence into nursing education. The University of Pennsylvania School of Nursing has established a course on “Artificial Intelligence + Nursing,” covering principles of artificial intelligence technology, health data analysis, virtual reality simulation, etc.^[23]. Similarly, Duke University School of Nursing launched a course on “Integrating Technology into Nursing Education,” aimed at exploring how artificial intelligence technology can improve nursing workflows and enhance communication skills^[28]. Currently, although some domestic universities are attempting to offer related courses, there is a general lack of systematic and comprehensive approaches. Therefore, in the future, it is necessary to form interdisciplinary educational teams, including experts in education, nursing, and artificial intelligence, to accelerate the construction of relevant curricula in China and promote the rapid transformation of nursing education towards intelligence.

Integrating multimodal artificial intelligence into nursing education requires the establishment of industry standards and evaluation systems. Students may utilize artificial intelligence technologies to cheat, producing high-quality written works that evade detection by plagiarism software, undermining many traditional assessment methods in higher education that rely on students preparing and submitting text-based responses^[29]. Therefore, it is necessary to develop practical assessment standards and a multidimensional evaluation system that combines student feedback, learning outcomes, and teaching quality to comprehensively and effectively assess the application effects of multimodal artificial intelligence in education.

Building a high-level, well-qualified nursing faculty team is key to cultivating intelligent nursing talent. Nursing faculty teams need to enhance their capabilities in artificial intelligence and big data analysis to meet modern educational needs. Universities can hold regular seminars or workshops to expose faculty to the latest developments and applications of artificial intelligence, emphasizing the connection between theory and practice during training and learning, allowing faculty to apply what they learn to real teaching scenarios^[30]. At the same time, opportunities for further professional development in artificial intelligence should be provided for faculty, such as short-term research projects, advanced courses, or certification programs, enabling them to gain deeper understanding and skills.

5. Conclusion

The integration of various perceptual modalities and processing capabilities through multimodal artificial intelligence technology brings opportunities and innovations to nursing education. Multimodal artificial intelligence has the potential to create interactive simulation environments, achieve refined scenario reproduction and emotion recognition, and provide personalized learning experiences. However, the current application of multimodal artificial intelligence technology in nursing education still faces challenges, such as technical training and usage costs, data security and privacy protection, etc. In the future, the integration of multimodal artificial intelligence into nursing education should focus on the construction of “multimodal artificial intelligence + nursing” curricula, the establishment of industry standards and evaluation systems, and the development of high-level, well-qualified teams to accelerate the intelligent transformation of nursing education in China.

References:

[1] NG F Y C, THIRUNAVUKARASU A J, CHENG H R, et al. Artificial intelligence education: an evidence-based medicine approach for consumers, translators, and developers[J]. Cell Rep Med, 2023, 4(10): 101230. DOI: 10.1016/j.xcrm.2023.101230.[2] SOENKSEN L R, MA Y, ZENG C, et al. Integrated multimodal artificial intelligence framework for healthcare applications[J]. NPJ Digit Med, 2022, 5(1): 149. DOI: 10.1038/s41746-022-00689-4.[3] NARAYANAN S, RAMAKRISHNAN R, DURAIRAJ E, et al. Artificial intelligence revolutionizing the field of medical education[J]. Cureus, 2023, 15(11): e49604. DOI: 10.7759/cureus.49604.[4] Wu Anxiong, Zhao Jialing, Huang Shaowei, et al. Construction of multimodal artificial intelligence data analysis experimental service platform[J]. Laboratory Research and Exploration, 2023, 42(4): 188-193. DOI: 10.19927/j.cnki.syyt.2023.04.038.[5] Li Zhaoyu. Multimodal artificial intelligence is stepping into a new stage of scenario application[N]. Science and Technology Daily, 2022-10-10(006). DOI: 10.28502/n.cnki.nkjrb.2022.005715.[6] ZHOU T X, RUAN S, CANU S. A review: deep learning for medical image segmentation using multi-modality fusion[J]. Array, 2019, 3/4: 100004. DOI: 10.1016/j.array.2019.100004.[7] Yu Miaoru, Zhang Defu, Zeng Wei, et al. Application of artificial intelligence in cardiac multimodal imaging[J]. Clinical Cardiovascular Disease Journal, 2023, 39(12): 922-929. DOI: 10.13201/j.issn.1001-1439.2023.12.005.[8] LIU H, ZHUANG Y Z, SONG E M, et al. A modality-collaborative convolution and transformer hybrid network for unpaired multi-modal medical image segmentation with limited annotations[J]. Med Phys, 2023, 50(9): 5460-5478. DOI: 10.1002/mp.16338.[9] KIEGALDIE D, SHAW L. Virtual reality simulation for nursing education: effectiveness and feasibility[J]. BMC Nurs, 2023, 22(1): 488. DOI: 10.1186/s12912-023-01639-5.[10] LIAW S Y, TAN J Z, LIM S, et al. Artificial intelligence in virtual reality simulation for interprofessional communication training: mixed method study[J]. Nurse Educ Today, 2023, 122: 105718. DOI: 10.1016/j.nedt.2023.105718.[11] Wang Ying, Tian Jianli, Jiao Yahui, et al. Application of virtual simulation technology combined with Rain Classroom in stoma care practical teaching[J]. Journal of Nursing, 2024, 39(2): 83-86.[12] Li Longti, Zhou Qinghuan, Chen Xin, et al. Application of multimodal virtual reality technology in orthopedic nursing clinical practice teaching[J]. Health Vocational Education, 2023, 41(24): 92-95. DOI: 10.20037/j.issn.1671-1246.2023.24.28.[13] Zhao Xiaoming, Yang Yijiao, Zhang Shiqing. Research progress on multimodal emotion recognition based on deep learning[J]. Computer Science and Exploration, 2022, 16(7): 1479-1503. DOI: 10.3778/j.issn.1673-9418.2112081.[14] Liu Jiming, Zhang Peixiang, Liu Ying, et al. Overview of multimodal emotion analysis technology[J]. Computer Science and Exploration, 2021, 15(7): 1165-1182. DOI: 10.3778/j.issn.1673-9418.2012075.[15] MUZAMMEL M, SALAM H, OTHMANI A. End-to-end multimodal clinical depression recognition using deep neural networks: a comparative analysis[J]. Comput Methods Programs Biomed, 2021, 211: 106433. DOI: 10.1016/j.cmpb.2021.106433.[16] ASLAM A, SARGANO A B, HABIB Z. Attention-based multimodal sentiment analysis and emotion recognition using deep neural networks[J]. Appl Soft Comput, 2023, 144: 110494. DOI: 10.1016/j.asoc.2023.110494.[17] CASTONGUAY A, FARTHING P, DAVIES S, et al. Revolutionizing nursing education through AI integration: a reflection on the disruptive impact of ChatGPT[J]. Nurse Educ Today, 2023, 129: 105916. DOI: 10.1016/j.nedt.2023.105916.[18] Chen Mengyue, Jiang Shi, Liu Chunna, et al. Practical research on personalized learning model based on vocational education cloud platform in “Adult Nursing” teaching[J]. Continuing Medical Education, 2019, 33(7): 2-4. DOI: 10.3969/j.issn.1004-6763.2019.07.002.[19] Jia Junzhi, Cui Xiyan, Zhang Guixiang. The impact of artificial intelligence technology on knowledge organization: a perspective of knowledge graph[J]. Library Forum, 2024, 44(2): 1-8.[20] Zhang Shan, Gao Li, Wang Yanling, et al. Construction of a “four-dimensional” intelligent service system for “Internal Medicine Nursing” based on knowledge graphs[J/OL]. Journal of Nursing Continuing Education: 1-9[2024-05-07].https://read.cnki.net/web/Journal/Article/FSJX20240321003.html.

[21] Xu Yuemei, Hu Ling, Zhao Jiayi, et al. Research progress and implications of language models and multilingual intelligence[J]. Computer Applications, 2023, 43(S2): 1-8.[22] LONG J J, LIN J X. An empirical study on cultivating college students’ cross-cultural communicative competence based on the artificial-intelligence English-teaching mode[J]. Front Psychol, 2022, 13: 976310. DOI: 10.3389/fpsyg.2022.976310.[23] Zhong Wenxi, Mao Xianjun, Cheng Jing, et al. Situational analysis of using generative artificial intelligence to cultivate nursing talents’ cross-professional communication skills[J]. Chinese Nursing Education, 2024, 21(3): 282-288.[24] Zhao Chaoyang, Zhu Guibo, Wang Jinqiao. Insights from ChatGPT on language large models and new development ideas for multimodal large models[J]. Data Analysis and Knowledge Discovery, 2023, 7(3): 26-35. DOI: 10.11925/infotech.2096-3467.2023.0216.[25] GRUNHUT J, WYATT A T, MARQUES O. Educating future physicians in artificial intelligence (AI): an integrative review and proposed changes[J]. J Med Educ Curric Dev, 2021, 8: 23821205211036836. DOI: 10.1177/23821205211036836.[26] WEIDENER L, FISCHER M. Proposing a principle-based approach for teaching AI ethics in medical education[J]. JMIR Med Educ, 2024, 10: e55368. DOI: 10.2196/55368.[27] BOSCARDIN C K, GIN B, GOLDE P B, et al. ChatGPT and generative artificial intelligence for medical education: potential impact and opportunity[J]. Acad Med, 2024, 99(1): 22-27. DOI: 10.1097/ACM.0000000000005439.[28] Duke University School of Nursing. Academic Programs[EB/OL]. [2023-7-10].https://nursing.duke.edu/academic-programs#program-select.[29] GRAHAM F. Daily briefing: will ChatGPT kill the essay assignment?[J]. Nature, 2022-12-12. DOI: 10.1038/d41586-022-04437-2.Epub ahead of print.[30] Shang Lili. Reflections on graduate medical education under the new medical background[J]. Journal of Medical Graduates, 2018, 31(10): 1078-1081. DOI: 10.16571/j.cnki.1008-8199.2018.10.017.

(Editor: Qiu Yongqiang)