Performance of Multigenerational ChatGPT Models in Taiwan’s Urology Board Exam: Accuracy and Domain Analysis

多代 ChatGPT 模型在台灣泌尿科專科考試中的表現：準確率及領域分析

Yung-Hao Liu ¹, Ho Li ^{1, 2}, Hui-Kung Ting ¹, Yu-Cing Jhuo ¹, Chin-Li Chen ¹, Chien-Chang Kao ¹, Ming-Hsin Yang ¹, Chih-Wei Tsao ¹, En Meng ¹, Sheng‐Tang Wu ¹, and Pei-Jhang Chiang ^{1, 3}

¹Division of Urology, Department of Surgery, Tri‐Service General Hospital, National Defense Medical University

²Division of Urology, Department of Surgery, Gangshan Branch of Zuoying Armed Forces General Hospital, Kaohsiung

³In-Service Master Program in Artificial Intelligence in Medicine, College of Medicine, Taipei Medical University

劉永浩¹, 黎赫^1,2, 丁慧恭¹, 卓育慶¹, 陳進利¹, 高建璋¹, 楊明昕¹, 曹智惟¹, 蒙恩¹, 吳勝堂¹, 江佩璋^1,3

¹國防醫學大學三軍總醫院泌尿外科

²高雄左營空軍總醫院岡山分院泌尿外科

³臺北醫學大學醫學院醫療人工智慧在職專班

Abstract

Purpose: Large language models demonstrate increasing utility in healthcare; however, their capacity to process medical content beyond training datasets remains unclear. This study evaluated the performance of four ChatGPT models on entirely novel medical content using 120 questions from the 2024 Taiwan Urology Board Examination, developed after all models’ October 2023 training cutoff.

Materials and Methods: Model performance was assessed using accuracy and processing time. Comparisons were conducted across four ChatGPT models and 12 urological subspecialties, with additional analyses according to question complexity and surgical anatomy requiring spatial reasoning.

Results: The chain-of-thought model o1-preview achieved 66.7% accuracy, significantly outperforming GPT-4o at 55.8% (p = 0.012), though requiring longer processing time (19.20 versus 14.94 s). Performance varied significantly across 12 urological subspecialties (p < 0.001). Unlike other models that showed decreased performance with increasing complexity, o1-preview uniquely improved from 65.0% on low-complexity questions to 68.3% on high-complexity questions. Most notably, for high-complexity surgical anatomy questions requiring spatial reasoning, o1-preview achieved 80.0% accuracy, whereas GPT-4o and GPT-4o mini scored 0.0% each (p = 0.024).

Conclusion: These findings establish that chain-of-thought architectures can successfully generalize novel medical content while excelling at complex spatial reasoning tasks, representing a fundamental advancement beyond pattern matching. Results provide evidence-based guidance for the strategic deployment of AI architectures in specialized medical education and clinical support, emphasizing their role as sophisticated tools to augment clinical expertise.