以 RAG 技術評估多 AI 模型於台灣泌尿科專考之效能分析  

陳鐸文、何東儒¹

嘉義長庚紀念醫院 外科部 泌尿科¹

Multi-AI Models Performance Evaluation by using Retrieval-Augmented Generation (RAG) for the Taiwanese Urology Board Examination

Duo-Wun Chen, Dong-Ru Ho  ¹

Chiayi Chang Gung Memorial Hospital, Dept. Of Surgery, Division Of Urology, Chia-Yi, Taiwan 

Purpose:

Large language models (LLMs) are increasingly applied in medical education and reasoning, yet performance in subspecialties like urology remains limited. Retrieval-Augmented Generation (RAG) aims to improve accuracy and reduce hallucination by grounding responses in factual sources. This study compares four LLMs—GPT-4o mini, GPT-4o, Claude 3.5 Sonnet, and Gemini 2.5—on Taiwanese urology board questions before and after RAG integration.

Materials and Methods:

Question Source: 150 single-choice questions (A–E) authored by board-certified urologists, covering diagnosis, surgery, oncology, and pediatrics for residents with ≥4 years of training.

Retrieval Corpus: Campbell-Walsh-Wein Urology (13th ed.) served as the RAG knowledge base, simulating textbook-grounded reasoning.

Models: GPT-4o, GPT-4o mini, Claude 3.5, and Gemini 2.5 were tested under Baseline (no retrieval) and RAG (Faiss retriever, top ≥ 0.75, fallback to baseline).

Metrics: Standardized prompts and parameters were used. Accuracy was compared using McNemar's test (p < 0.05 significant).

Results: Each model response characteristics are addressed below and shown in table. 1.

The result in different model with baseline and RAG mode was showed in table.2.

Distinct behaviors noted. GPT-4o-mini, though smaller, showed strong structure and precision under token limits. GPT-4o offered deeper reasoning but sometimes exceeded scope, slightly lowering factuality. Claude 3.5 produced fluent yet hedged responses (“the most likely answer may be…”), reflecting cautious alignment. Gemini 2.5 was excluded due to API truncation errors. Baseline accuracy: GPT-4o mini 36.7%, GPT-4o 58.0%, Claude 3.5 33.3%. RAG showed modest gains without significance: +6.0%, +2.0%, +2.7% respectively.

Table.1

Table.2

Conclusions:

RAG yielded small but consistent accuracy gains across models, with GPT-4o-mini showing the most notable improvement (+6.0%). Despite modest statistical significance, the integration of a textbook-grounded knowledge base (Campbell-Walsh-Walsh Urology) effectively reduces hallucinations and provides traceable, evidence-based reasoning. This study establishes the cross-performance comparison for AI models on the Taiwanese Urology Board Examination. It demonstrates the potential of RAG-enhanced AI as a brand-new educational tool for resident training within the TUA community, offering a framework for developing reliable, localized clinical decision support systems grounded in urological evidence..