先進大型語言模型對複雜膀胱病灶評估能力之比較：準確性與推理能力之盲測評估

陳劭威¹、石永頎¹、吳政陽²、黃士維^2,3、蔡宗佑^1,3,4

¹亞東紀念醫院 外科部 泌尿科；²國立臺灣大學醫學院附設醫院雲林分院 泌尿部；³國立臺灣大學醫學院附設醫院；⁴元智大學 電機工程所

A Head-to-Head “Stress Test" of State-of-the-Art LLMs on Challenging Cystoscopic Lesions: A Blinded Evaluation of Accuracy and Reasoning

Shao-Wei Chen¹, Yung-Chi Shih¹, Cheng-Yang Wu², Shi-Wei Huang^1,3, Chung-You Tsai ^1,3,4,

¹ Division of Urology, Department of Surgery, Far Eastern Memorial Hospital, New Taipei, Taiwan;

² Department of Urology, National Taiwan University Hospital, Yunlin Branch, Yunlin, Taiwan;

³ Department of Urology, National Taiwan University Hospital, Taipei, Taiwan;

⁴ Department of Electrical Engineering, Yuan Ze University, Taoyuan, Taiwan

Purpose: Multimodal Large Language Models (MM-LLMs) represent a paradigm shift in artificial intelligence by integrating visual perception with sophisticated contextual reasoning. This study rigorously evaluated the performance of leading MM-LLMs in cystoscopic image analysis. We leveraged clinician-curated "stress-test" datasets—comprising rare and diverse lesions—to assess diagnostic accuracy, reasoning quality, and clinical utility.

Methods: Four prominent MM-LLMs (OpenAI-o3, ChatGPT-4o, Gemini-2.5-Pro, and MedGemma-27B) were benchmarked under blinded, randomized conditions. Performance was evaluated across two tasks: free-text image interpretation (n=401) and seven-class classification of tumor-like lesions (n=113). Responses were independently rated by three experts using a five-point Likert scale. Diagnostic metrics, including accuracy, sensitivity, specificity, Youden’s J index, and MCC, were calculated for key clinical endpoints: lesion detection, biopsy indication, and malignancy prediction.

Results: In free-text interpretation, OpenAI-o3 demonstrated superior performance, achieving high expert satisfaction for anatomical descriptions (84.5%) and finding-based observations (76.0%). However, its proficiency declined in complex reasoning (52.5%) and definitive diagnosis (48.2%). For lesion classification, OpenAI-o3 emerged as the top performer with a balanced diagnostic profile, achieving accuracies of 88.3% for detection, 72.3% for biopsy triage, and 63.4% for malignancy. Conversely, ChatGPT-4o and Gemini-2.5-Pro exhibited high sensitivity at the expense of specificity, while MedGemma-27B’s performance was suboptimal. Notably, text-based in-context learning yielded only marginal gains in specificity without improving overall accuracy.

Conclusions: Our findings highlight the substantial potential of MM-LLMs as assistive tools in cystoscopy, particularly in generating interpretable rationales and aiding biopsy triage. Nevertheless, their ability to navigate complex differential diagnoses remains limited. Further refinement and domain-specific optimization are essential before these models can be reliably integrated into routine clinical workflows.

Keywords: multimodal large language model, artificial intelligence, cystoscopy, diagnostic reasoning, finding description, biopsy indication, bladder tumor.