Clinical Verification of AI Models for Radiology Report Annotation Using a Physician-Annotated MIMIC-CXR
9th International Symposium on Innovative Approaches in Smart Technologies, ISAS 2025, Gaziantep, Türkiye, 27 - 28 Haziran 2025, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/isas66241.2025.11101953
- Basıldığı Şehir: Gaziantep
- Basıldığı Ülke: Türkiye
- Anahtar Kelimeler: Clinical Validation, Large Language Models, Radiology Reports
- Ankara Yıldırım Beyazıt Üniversitesi Adresli: Evet
Özet
Accurate evaluation of AI-generated radiology reports is crucial for clinical applications. This study investigates the use of large language models (LLMs) - specifically Google Gemini and OpenAI GPT-4o - for this task. We compare their performance against CheXpert in annotating both raw radiology reports and reports generated by image-to-report systems (MedRAX, RGRG), using a physician-validated subset of MIMIC-CXR as ground truth. Our evaluation demonstrates Gemini's strong performance in this context. This work provides a clinical validation of LLMs against expert annotations, highlighting their potential for medical text processing and the challenges in accurately identifying rare findings.