#0954
Vulnerabilities to stealth prompt injection attacks in medical recommendation systems using large-scale language models: Focusing on urology
J. Suh1
1Asan Medical Center, Urology, Seoul, Korea (Republic of)
Introduction:
This study aims to evaluate the susceptibility of clinical recommendation systems using large language models (LLMs) to stealth prompt injection attacks in simulated urologic situations of dialogues, specifically assessing whether controlled prompt injections could covertly influence model-generated treatment recommendations.
Material and methods:
A controlled, paired-design simulation study was conducted using Google's gemma-2-2b-it LLM. Dialogue samples were derived from urologic scenarios in the MedQA-USMLE dataset, explicitly excluding pediatric cases. Each simulated conversation consisted of eight alternating conversational turns between the user and LLM, covering urologic symptoms, preliminary diagnoses, treatment recommendations, and follow-up queries. Stealth prompt injections, promoting complementary therapies such as "red ginseng," were covertly inserted at critical conversational points (turns 4, 6, and 8), employing obfuscated Python scripts (PyArmor and Cython) to simulate realistic third-party attacks. Outcomes measured included recommendation strength, response time, coherence scores, medical term density, readability (Flesch Reading Ease), and inter-turn correlation.
Results:
Prompt injections significantly increased the frequency and intensity of red ginseng recommendations by turn 6 (Injection: 78% vs. Control: 0%, p<0.001), with persistent effects at turn 8 (Injection: 63% vs. Control: 0%, p<0.001). Response times were slightly prolonged in the Injection group at later dialogue turns, although differences did not reach statistical significance. By turn 8, medical term density was significantly higher in injected dialogues (p=0.032), and coherence scores were significantly reduced compared to controls (Injection median: 0.40 vs. Control median: 0.42, p<0.001). Injection dialogues also demonstrated lower inter-turn correlations, indicating subtle disruptions in conversational consistency.