Many clinical tasks require understanding of specialized data, such as medical imaging and genomic data, which are often not included in general large multimodal models. Based on the Gemini multimodal model, we have developed a new series of Med-Gemini models, which inherit the core capabilities of Gemini and are optimized for medical use through fine-tuning with 2D and 3D radiology, histopathology, ophthalmology, dermatology, and genomic data. Med-Gemini-2D has set a new standard for AI-driven chest X-ray (CXR) report generation based on expert evaluation, outperforming previous best results on two independent datasets with absolute advantages of 1% and 12%, where 57% and 96% of AI reports in normal cases, and 43% and 65% of AI reports in abnormal cases, were rated as “comparable to or better than the original radiologist’s report.” We showcase the first report generation for 3D computed tomography (CT) body data based on large multimodal models, using Med-Gemini-3D, where 53% of AI reports were deemed clinically acceptable, though further research is needed to achieve the quality of expert radiologist reports. Beyond report generation, Med-Gemini-2D surpassed previous best performance in chest X-ray visual question answering (VQA) and performed well in chest X-ray classification and radiology VQA, exceeding existing techniques or benchmarks in 17 out of 20 tasks. In histopathology, ophthalmology, and dermatology image classification, Med-Gemini-2D outperformed the baseline in 18 out of 20 tasks, coming close to the performance of task-specific models. Outside imaging, Med-Gemini-Polygenic exceeded standard linear polygenic risk score methods in disease risk prediction and was able to generalize to gene-related diseases that were never trained on. Despite needing further development and evaluation in critical medical areas, our results highlight the potential of Med-Gemini across a wide range of medical tasks.
https://www.zhuanzhi.ai/paper/bc65b01ad1b8e0ddec8f98eac621daa7
Convenient Access to Specialized Knowledge
Easy Download, please followZhuanzhi public account (click the above blue Zhuanzhi to follow)
Click “Read Original“, to learn about usingZhuanzhi, and access over 100,000 AI-themed knowledge resources
