Purpose To develop and evaluate a large language model (LLM)-based learning tool, featuring virtual patients (VPs) and virtual assessors (VAs), and to assess its impact on medical students’ perceptions of history-taking education compared to conventional learning methods.
Methods A tool using the GPT-4 API was developed to provide seven clinical VP scenarios and a VA that delivered both immediate, reflective dialogue and comprehensive written feedback. First- and second-year medical students participated in a 6-day study. Pre- and post-participation surveys using a 5-point Likert scale assessed perceptions of the LLM tool versus conventional methods across usability, self-efficacy, and feedback quality domains.
Results Twenty-one students completed the study. The LLM-based tool demonstrated statistically significant improvements over conventional methods in all assessed domains. Students reported greater comfort during practice (mean 4.57 vs. 2.95, p=0.0002). Significant gains were seen in six of eight self-efficacy measures, including confidence in handling unfamiliar cases (4.00 vs. 2.90, p=0.0002). All nine feedback quality dimensions improved significantly, with feedback perceived as more specific (4.43 vs. 3.24, p=0.0005) and personalized (4.19 vs. 3.19, p=0.0001).
Conclusion An LLM-based learning tool featuring VPs and VAs can significantly enhance medical students’ perceived learning experience in history-taking education. It offers a scalable, accessible, and cost-effective complementary training method. Future research should validate these subjective improvements with objective performance metrics.
Citations
Citations to this article as recorded by
AI-Assisted Training for Teleconsultation Competencies in Undergraduate Medical Education: A Narrative Review Wojciech Michał Glinkowski, Barbara Jacennik, Aldona Katarzyna Jankowska, Tomasz Cedro, Szymon Wilk, Rafał Doniec Applied Sciences.2026; 16(10): 4858. CrossRef
Purpose Information gathering ability had been evaluated mainly via checklists in clinical performance examinations (CPX). But, it is not proved yet if students write the information correctly in postencounter note (PN), although they asked questions or performed physical examinations (PE) about the information when they interacted with standardized patients in CPX. This study addressed the necessity of introducing PN to evaluate the ability in CPX.
Methods After patient encounters, students were instructed to write the findings of history taking and physical examination that they considered as important information in approaching the patient’s problems in PN. PNs were scored using answer keys selected from checklist items, which were considered to be recorded in PN by CPX experts.
Results PNs of six CPX cases from 54 students were analyzed. Correlation coefficients between the key-checklist scores and PN scores of six cases were moderate to high (0.52 to 0.79). However, students frequently neglected some cardinal features of chief complains, pertinent findings of past/social history and PE, and pertinent negative findings of associated symptoms in PNs, which were checked as ‘done’ in the keys of checklists.
Conclusion It is necessary to introduce PN in CPX to evaluate the students’ ability of synthesis and integration of patient information.
Citations
Citations to this article as recorded by
Deliberate practice of diagnostic clinical reasoning reveals low performance and improvement of diagnostic justification in pre-clerkship students Justine Staal, Jason Waechter, Jon Allen, Chel Hee Lee, Laura Zwaan BMC Medical Education.2023;[Epub] CrossRef
Restoring force of medical school Eun Kyung Chung Korean Journal of Medical Education.2014; 26(2): 81. CrossRef
PURPOSE Evaluation of clinical skills and attitude including development of dynamic patient-doctor relationship is important in board examination (BE). Korean Neurological Association (KNA) has introduced clinical performance examination (CPX) utilizing standardized patients (SP) to BE in 2007. In this study, the authors describe the 3-year experience of CPX in BE through 2009.
METHODS To implement CPX session in BE, KNA developed CPX workshop for BE attendees and members of grading committee.
CPX sessions in BE consisted of two model scenarios mimicking neurological patients in clinical practice. The total score and itemized scores of CPX sessions were compared with other areas of BE, and scores from each year were also compared.
RESULTS Scores from CPX sessions were significantly correlated with BE step II. Among the itemized scores of CPX sessions, clinical items including history taking and physical examination were significantly correlated with scores from other areas of BE. However, scores from global assessment from SP were strongly associated with patient-doctor relationship, history taking, and patient education.
CONCLUSION Our experiences suggest that CPX utilizing SP is a useful tool to assess the clinical skills in BE. In order to produce clinically well qualified neurologists, more efforts should be made to develop cases and to improve assessment tools for CPX.
Citations
Citations to this article as recorded by
Clinical Performance and Communication Skills of ChatGPT Versus Physicians in Emergency Medicine: Simulated Patient Study ChulHyoung Park, Min Ho An, Gyubeom Hwang, Rae Woong Park, Juho An JMIR Medical Informatics.2025; 13: e68409. CrossRef
Future Development for the Board Licensing Examination of Internal Medicine Jin Seok Kim, Jong-Won Ha The Korean Journal of Medicine.2017; 92(2): 109. CrossRef
Comparison of Standardized Patient and Faculty Agreement in Evaluating Nursing Students’ Assessment and Communication Skills Young Ju Kim Journal of Korean Academy of Fundamentals of Nursing.2017; 24(3): 189. CrossRef
Impact of Clinical Performance Examination on Incoming Interns' Clinical Competency in Differential Diagnosis of Headache Seong-Min Park, Yun-Mi Song, Bo-Kyoung Kim, Hyoeun Kim Korean Journal of Family Medicine.2014; 35(2): 56. CrossRef
Medical Education: Where are We Going? Eunbae B. Yang Korean Journal of Medical Education.2011; 23(2): 77. CrossRef
Correlation of CPX Scores with the Scores on Written Multiple-Choice Examinations on the Certifying Examination for Family Medicine in 2009 to 2011 Jung Jin Cho, Ji Yong Kim, Hoon Ki Park, In Hong Hwang Korean Journal of Medical Education.2011; 23(4): 315. CrossRef
Predictive Value of the Korean Academy of Family Medicine In-Training Examination for Certifying Examination Jung-Jin Cho, Ji-Yong Kim Korean Journal of Family Medicine.2011; 32(6): 352. CrossRef
PURPOSE The purpose of this research was to identify the effectiveness of a standardized patient (SP) instructor. The study was performed to compare the effects of two types of instruction on breast examination skills by an SP instructor and a professor.
METHODS Fifty-four medical students in the third year and 4 SPs participated in the study. In a pretest, each student was judged by the SP individually in performing a breast examination and completed a questionnaire about perception.
Next, students were randomly assigned to 2 experimental groups (small-group instruction by an SP instructor or professor). A posttest of performance and perception was administered using the same procedures as in the pretest. To identify the effectiveness of the SP instructor, differences in student scores between the groups were analyzed by t-test.
RESULTS In the posttest of student performance on the breast examination, there was no significant difference between the groups (p=0.270). With regard to student perception, scores on self-confidence in the breast examination were significantly different (p=0.026) between the groups, of which students who received instruction from the professor showed a higher level. Additionally, there was a significant effect on students' perceived reliability of instruction by the SP instructor (p=0.011), and students who were instructed by the SP demonstrated higher reliability compared with professor-taught students.
CONCLUSION Instruction by an SP instructor has the same effectiveness as instruction by a professor. SP instructors have very powerful effects on student performance and perception.
Citations
Citations to this article as recorded by
The perspective of ‘a good doctor’ on university students YoungJoon Ahn Medical Biological Science and Engineering.2021; 4(1): 42. CrossRef
Effect of feedback from standardized patients on medical students’ performance and perceptions of the neurological examination Joo Hyun Park, Ji Young Son, Sun Kim, Win May Medical Teacher.2011; 33(12): 1005. CrossRef
How to Make a ‘Good Doctor’ Young-Mee Lee Korean Journal of Medical Education.2010; 22(4): 247. CrossRef
PURPOSE The optimal type of patient simulation for different levels of learners has not been extensively studied. The purpose of the study was to compare preclerkship medical student responses and course achievement according to different types of patient simulations in an introductory advanced life support (IALS) course.
METHODS A full-day, simulation-based IALS course was developed for preclerkship medical students who attended a four-week introduction to a clinical medicine program. One hundred eighteen students were trained in three days. Onsite interactive simulation with verbal debriefing (interactive type) was applied on the first day, and full-mission, realistic simulation with video-assisted debriefing (realistic type) was applied on the second and third days.
At the end of course, students evaluated the course and their simulation experiences and completed a written post-test.
RESULTS Student responses to the course and patient simulations were very positive. Students who experienced the realistic type of patient simulations more highly rated in realistic experiences, such as patient care, than the interactive type group (3.83+/-0.88 vs. 3.41+/-0.84, p=0.018). Values for team communication training were more highly rated by students in the interactive type group than the realistic type (4.69+/-0.52 vs. 4.39+/-0.86, p=0.022).
There was no significant difference in post-test scores between the two groups (realistic, 67.63+/-10.80; interactive, 66.73+/-9.93, p=0.654).
CONCLUSION Both types of patient simulation provide valuable learning experiences to preclerkship medical students, with their own advantages in an IALS course.
Onsite interactive simulation with verbal debriefing may be more cost-effective tool for preclerkship medical students.
Citations
Citations to this article as recorded by
Video-based feedback as a method for training rural healthcare workers to manage medical emergencies: a pilot study Zainab Oseni, Hla Hla Than, Edyta Kolakowska, Lauren Chalmers, Borimas Hanboonkunupakarn, Rose McGready BMC Medical Education.2017;[Epub] CrossRef
The effects of simulation-based resuscitation training on nurses' self-efficacy and satisfaction Young Sook Roh, Woo Sook Lee, Hyun Soo Chung, Young Mi Park Nurse Education Today.2013; 33(2): 123. CrossRef
It Is Time to Consider Cultural Differences in Debriefing Hyun Soo Chung, Peter Dieckmann, Saul Barry Issenberg Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare.2013; 8(3): 166. CrossRef
An Effective Method of Teaching Advanced Cardiac Life Support (ACLS) Skills in Simulation-Based Training Hyo Bin Yoo, Jae Hyun Park, Jin Kyung Ko Korean Journal of Medical Education.2012; 24(1): 7. CrossRef
A Systematic Review of Training That Use an Integrated Patient Simulator Hyo Bin Yoo, Jae Hyun Park, Jin Kyung Ko, Tai Young Yoon Korean Journal of Medical Education.2010; 22(4): 257. CrossRef
PURPOSE A high degree of agreement between standardized patients (SP) check-list recordings and those of faculty will be necessary if SPs are to eventually replace faculties in the OSCE evaluaton process. This study was conducted to know to what degree SPs' checklist recordings agree with those of faculties during an OSCE. METHODS: One hundred and twenty one fourth-year medical students of Hanyang University College of Medicine took an OSCE. In each of two study stations, a student saw an SP for four minutes and the SP recorded the same checklists as a faculty examiner did, for the following fifty seconds. RESULTS: For the 'bad news delivery' station, SP evaluations were more lenient compared to those of faculties (56 vs 45, p< 0.01), but in the case of 'chest pain', there was no significant difference.
Pearson correlation coefficients for the 'bad news delivery' station and for the 'chest pain' case were 0.60 and 0.65, respectively. The mean percentages of agreement for the 'bad news delivery' and the 'chest pain' checklists were 71% and 82%, respectively. The mean kappa statistics for the 'bad news delivery' and the 'chest pain' check-lists were 0.19 and 0.49, respectively. CONCLUSION: The ratings by SPs were found to be consistent with those of faculties only in moderate degree. The exactness of scoring criteria, and the optimal SP training are to be the premise for the replacement of faculties by SPs during OSCE checklist recordings.
Citations
Citations to this article as recorded by
A study on evaluator factors affecting physician-patient interaction scores in clinical performance examinations: a single medical school experience Young Soon Park, Kyung Hee Chun, Kyeong Soo Lee, Young Hwan Lee Yeungnam University Journal of Medicine.2021; 38(2): 118. CrossRef
Holistic rubric vs. analytic rubric for measuring clinical performance levels in medical students So Jung Yune, Sang Yeoup Lee, Sun Ju Im, Bee Sung Kam, Sun Yong Baek BMC Medical Education.2018;[Epub] CrossRef
Comparison of Standardized Patient and Faculty Agreement in Evaluating Nursing Students’ Assessment and Communication Skills Young Ju Kim Journal of Korean Academy of Fundamentals of Nursing.2017; 24(3): 189. CrossRef
Verbal communication of students with high patient–physician interaction scores in a clinical performance examination assessed by standardized patients HyeRin Roh, Kyung Hye Park, Song Yi Park Korean Journal of Medical Education.2017; 29(4): 241. CrossRef
The Effects of Preclinical Clinical Performance Examination on Nursing Students' Confidence in Nursing Skills and Critical Thinking Competence Jeong Sook Park, Mi Jung Choi, Soon Yang Jang The Journal of Korean Academic Society of Nursing Education.2015; 21(1): 75. CrossRef
Development and Evaluation of Simulation-Based Training for Obstetrical Nursing Using Human Patient Simulators MIOK KIM, MINHO SHIN CIN: Computers, Informatics, Nursing.2013; 31(2): 76. CrossRef
Correlations of Clinical Assessment Tools with Written Examinations Eun Jeong Choi, Sung Sunwoo Korean Journal of Medical Education.2009; 21(1): 43. CrossRef
Validity and Reliability of a Clinical Performance Examination using Standardized Patients Ja Yun Choi, Keum Seong Jang, Soon Hee Choi, Mi Soon Hong Journal of Korean Academy of Nursing.2008; 38(1): 83. CrossRef
Effects of Case Type and Standardized Patient Gender on Student Performance in a Clinical Performance Examination Jonghoon Kim, Kiyoung Lee, Dongmi Yoo, Eunbae Yang Korean Journal of Medical Education.2007; 19(1): 23. CrossRef
First
Prev
