Purpose The purpose of this study was to apply alternative standard setting methods for the Korean Medical Licensing Examination (KMLE), a criterion-referenced written examination, and to compare them to the conventional cut score used on the KMLE.
Methods The process and results of criterion-referenced standard settings (i.e., the modified-Angoff and bookmark methods) were evaluated. The ratio of passing and failing examinees determined using these alternative standard setting methods was compared to the results of the conventional criteria. Additionally, the external, internal and procedural evaluation of these methods were reviewed.
Results The modified-Angoff method yielded the highest cut score, followed sequentially by the conventional method and the bookmark method. The classification agreement between the modified-Angoff and bookmark methods was 0.720 measured by Cohen’s κ coefficient. The intra-panelist classification consistency of modified-Angoff method was higher than bookmark method. However, the inter-panelist classification consistency was vice versa. The standard setting panelists’ survey results showed that the procedures of both methods were satisfactory, but panelists had more confidence in the results of the modified-Angoff method.
Conclusion The modified-Angoff method showed results that were more similar to those of the conventional method. Both new methods showed very high concordance with the conventional method, as well as with each other. The modified-Angoff method was considered feasible for adoption on the KMLE. The standard setting panelists responded positively to the modified-Angoff method in terms of its practical applicability, despite certain advantages of the bookmark method.
Citations
Citations to this article as recorded by
Refining competency benchmarks: a scoping review of Angoff standard-setting in dental education Galvin Sim Siang Lin, Abdul Rauf Badrul Hisham, Muhammad Nazmi Abdul Majid, Chan Choong Foong, Ting Khee Ho, Lara T. Friedlander BMC Oral Health.2026;[Epub] CrossRef
Angoff methods in standard setting in health professional education: a systematic review and meta-analysis Kannan Sridharan, Gowri Sivaramakrishnan BMC Medical Education.2025;[Epub] CrossRef
Unveiling fairness in scoring: a thorough method for precise cutoff score calculation in education assessment Alireza Akbari, Mohsen Shahrokhi Quality Assurance in Education.2024; 32(3): 493. CrossRef
Analysis on Validity and Academic Competency of Mock Test for Korean Medicine National Licensing Examination Using Item Response Theory Han Chae, Eunbyul Cho, SeonKyoung Kim, DaHye Choi, Seul Lee Keimyung Medical Journal.2023; 42(1): 7. CrossRef
Standard-Setting for Continuing Education Assessment of Select New Competencies Dixon Thomas, Amal Abdalla, Kimberly McKeirnan, Sherief Khalifa Journal of Continuing Education in the Health Professions.2022; 42(1): e96. CrossRef
Evaluating the cutoff score of the advanced practice nurse certification examination in Korea Younhee Kang Nurse Education in Practice.2022; 63: 103407. CrossRef
Possibility of using the yes/no Angoff method as a substitute for the percent Angoff method for estimating the cutoff score of the Korean Medical Licensing Examination: a simulation study Janghee Park Journal of Educational Evaluation for Health Professions.2022; 19: 23. CrossRef
Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park Journal of Educational Evaluation for Health Professions.2022; 19: 33. CrossRef
Comparison of the validity of bookmark and Angoff standard setting methods in medical performance tests Majid Yousefi Afrashteh BMC Medical Education.2021;[Epub] CrossRef
Multiple‐Choice versus Open‐Ended Questions in Advanced Clinical Neuroanatomy: Using a National Neuroanatomy Assessment to Investigate Variability in Performance Using Different Question Types Wassim H. Merzougui, Matthew A. Myers, Samuel Hall, Ahmad Elmansouri, Rob Parker, Alistair D. Robson, Octavia Kurn, Rachel Parrott, Kate Geoghegan, Charlotte H. Harrison, Deepika Anbu, Oliver Dean, Scott Border Anatomical Sciences Education.2021; 14(3): 296. CrossRef
Setting a standard for low reading proficiency: A comparison of the bookmark procedure and constrained mixture Rasch model Tabea Feseker, Timo Gnambs, Cordula Artelt, Karl Bang Christensen PLOS ONE.2021; 16(11): e0257871. CrossRef
Item Analysis of Multiple-choice Questions (MCQs): Assessment Tool For Quality Assurance Measures Amani H. Elgadal, Abdalbasit A. Mariod Sudan Journal of Medical Sciences.2021; 16(3): 334. CrossRef
Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination Mi Kyoung Yim, Sujin Shin Journal of Educational Evaluation for Health Professions.2020; 17: 14. CrossRef
Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination Janghee Park, Mi Kyoung Yim, Na Jin Kim, Duck Sun Ahn, Young-Min Kim Journal of Educational Evaluation for Health Professions.2020; 17: 28. CrossRef
The Use of Rasch and Item Mapping in Determining Cut Score of Comprehensive Pre internship Exam Sh Habibzadeh, A Delavar, NA Farrokhi, A Minaei, M Jalili Research in Medical Education.2019; 11(3): 59. CrossRef
Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee Janghee Park, Duck-Sun Ahn, Mi Kyoung Yim, Jaehyoung Lee Journal of Educational Evaluation for Health Professions.2018; 15: 32. CrossRef
Purpose This study analyzed the perceptions of medical students and faculty regarding disclosure of test items on the Korean medical licensing examination.
Methods I conducted a survey of medical students from medical colleges and professional medical schools nationwide. Responses were analyzed from 718 participants as well as 69 faculty members who participated in creating the medical licensing examination item sets. Data were analyzed using descriptive statistics and the chi-square test.
Results It is important to maintain test quality and to keep the test items unavailable to the public. There are also concerns among students that disclosure of test items would prompt increasing difficulty of test items (48.3%). Further, few students found it desirable to disclose test items regardless of any considerations (28.5%). The professors, who had experience in designing the test items, also expressed their opposition to test item disclosure (60.9%).
Conclusion It is desirable not to disclose the test items of the Korean medical licensing examination to the public on the condition that students are provided with a sufficient amount of information regarding the examination. This is so that the exam can appropriately identify candidates with the required qualifications.
Citations
Citations to this article as recorded by
The legality and appropriateness of keeping Korean Medical Licensing Examination items confidential: a comparative analysis and review of court rulings Jae Sun Kim, Dae Un Hong, Ju Yoen Lee Journal of Educational Evaluation for Health Professions.2024; 21: 28. CrossRef
Analysis on Validity and Academic Competency of Mock Test for Korean Medicine National Licensing Examination Using Item Response Theory Han Chae, Eunbyul Cho, SeonKyoung Kim, DaHye Choi, Seul Lee Keimyung Medical Journal.2023; 42(1): 7. CrossRef
Possibility of independent use of the yes/no Angoff and Hofstee methods for the standard setting of the Korean Medical Licensing Examination written test: a descriptive study Do-Hwan Kim, Ye Ji Kang, Hoon-Ki Park Journal of Educational Evaluation for Health Professions.2022; 19: 33. CrossRef
Comparison of National License Tests between Optometrist and Other Medical Occupational Categories Sung-Soo Kang, Sang-Hyun Kim, Hyun-Suk Shim, Duk-Yong Sung, Jun-Beom Shim Journal of Korean Ophthalmic Optics Society.2017; 22(1): 1. CrossRef
Hopes for the new challenges in <i>Korean Journal of Medical Education</i> Young Hwan Lee Korean Journal of Medical Education.2015; 27(3): 159. CrossRef
PURPOSE To evaluate the clinical performance through the Korean Medical Licensing Examination clinical skills assessment (KMLE CSA) this survey was done.
METHODS A survey of 130 interns and residents (46 applicants and 84 non-applicants for the KMLE CSA) at a university hospital in Seoul was conducted in January and February 2012. The data were gathered using a structured and self-administered questionnaire. For the items that assessed the clinical performance of these subjects, we selected 15 items that are mostly frequently used by Delphi's technique, and difficult procedural skills based on the results of medical students' performance. We also used subcomponents of the clinical problems test of the KMLE CSA.
RESULTS The total score on the KMLE CSA improved by 1.33 points (a perfect score is 10), 1.49 points for procedural skills, and 0.84 points for clinical problems by multiple regression analysis. The variables that influenced clinical skills were sex (females had 0.86 more points than males), experience in military or public services (1.04 points higher than persons without experience), and type of school (graduates of medical school had 1.41 more points than graduates of professional graduate school).
CONCLUSION Implementation of the KMLE CSA improved the clinical performance of medical graduates.
Citations
Citations to this article as recorded by
Developing a best practice framework for clinical competency education in the traditional East-Asian medicine curriculum Sang Yun Han, Seung-Hee Lee, Han Chae BMC Medical Education.2022;[Epub] CrossRef
Study on Employee Performance Evaluation Based on Adaptive Feature Selection Fuzzy Algorithm Yuan Qian, Jiemin Yin, R. Mo Mobile Information Systems.2022; 2022: 1. CrossRef
A one-day surgical-skill training course for medical students’ improved surgical skills and increased interest in surgery as a career Ho Seok Seo, Yong Hwa Eom, Min Ki Kim, Young-Min Kim, Byung Joo Song, Kyo Young Song BMC Medical Education.2017;[Epub] CrossRef
Does medical students' clinical performance affect the actual performance during medical internship? ER Han, EK Chung Singapore Medical Journal.2016; 57(02): 87. CrossRef
Reforms of the Korean Medical Licensing Examination regarding item development and performance evaluation Mi Kyoung Yim Journal of Educational Evaluation for Health Professions.2015; 12: 6. CrossRef
Educational intervention as an effective step for reducing blood culture contamination: a prospective cohort study W.B. Park, S.J. Myung, M.-d. Oh, J. Lee, N.-J. Kim, E.-C. Kim, J.S. Park Journal of Hospital Infection.2015; 91(2): 111. CrossRef
Student Centered Medical Education Sun Kim Korean Journal of Medical Education.2012; 24(4): 279. CrossRef
PURPOSE In order to apply the item response theory to test results, the assumptions made about unidimensionality and item goodness-of-fitness should be tested before analysis.
To know if the Korean Medical Licensing Examination results fulfill these assumptions, appropriate tests were done on these two features.
METHODS Three results(from three subjects) of the Korean Medical Licensing Examination, done in January 2004, were converted into 1, 0 data. The Holland and Rosenbaum Method(HRMH) and DETECT were used to test unidimensionality.
The Winsteps was used to test goodness-of-fitness according to the Rasch model.
RESULTS Unidimensionality was rejected by HRMH but accepted by DETECT. With the Rasch model, 2 out of 550 items were inadequate. Each 137, 73 and 455 examinees out of 3,881 were inadequate in three subjects of 126, 400 and 24 items.
CONCLUSION The above results suggest that DETECT is desirable for testing unidimensionality of high-stakes tests with more than 100 items. According to the Rasch model, the Korean Medical Licensing Examination fulfills the goodness-of-fitness to be analyzed according to the item response theory.
Citations
Citations to this article as recorded by
Analysis on Validity and Academic Competency of Mock Test for Korean Medicine National Licensing Examination Using Item Response Theory Han Chae, Eunbyul Cho, SeonKyoung Kim, DaHye Choi, Seul Lee Keimyung Medical Journal.2023; 42(1): 7. CrossRef
Item difficulty index, discrimination index, and reliability of the 26 health professions licensing examinations in 2022, Korea: a psychometric study Yoon Hee Kim, Bo Hyun Kim, Joonki Kim, Bokyoung Jung, Sangyoung Bae Journal of Educational Evaluation for Health Professions.2023; 20: 31. CrossRef
Relationship of self-determined motivation with time-related academic behavior in Korean primary school students: A person-centered approach Sanghyun Park Heliyon.2022; 8(10): e11191. CrossRef
A Study on the Reliability and Validity of the Korean Health Literacy Instrument for Late School-Aged Children Sook-Kyoung Park, Eun-Gyeong Kim International Journal of Environmental Research and Public Health.2021; 18(19): 10304. CrossRef
Self-Determination Studies for Elementary Students with and Without Disabilities in Korea Eunbyul Cho, Hyojeong Seo Advances in Neurodevelopmental Disorders.2019; 3(2): 161. CrossRef
Study of Adolescents' Academic Personality Types, Learning Behavioral Types and Self-determinative Learning Motivations Hun Kang, Sang-Hun Han, Ju-Hyeong Ku Journal of the Korea Academia-Industrial cooperation Society.2014; 15(8): 4919. CrossRef
The Relationship among Self-efficacy, Self-determination and Academic achievement of Middle/High School Student Chung-Ja Kim Journal of the Korea Academia-Industrial cooperation Society.2013; 14(3): 1148. CrossRef
Can computerized tests be introduced to the Korean Medical Licensing Examination? Sun Huh Journal of the Korean Medical Association.2012; 55(2): 124. CrossRef
The Relationships between Mother's Psychological Control and Self-Directed Learning Ability in Elementary School Students: Focusing on the Mediating Effects of Self-Determined Motivation Heesun Lee, Yongae Kwon Journal of the Korean Home Economics Association.2012; 50(8): 125. CrossRef
PURPOSE We analyzed the correlation between the year-end student academic scores (grade score), the trial examination scores, and the Korean Medical Licensing Examination (KMLE) score by correlation analysis and multiple regression method. METHODS: Our subjects included 217 graduates between the years 2002 and 2003. Yearly, trial examinations, composed according to the principles of KMLE setting, are performed four times. For correlation and regression analysis, all scores were transformed to the standard score according to the standard score norm. RESULTS: In the regression analysis among each grade scores, the lower grade score always affected the higher grade score. Also all academic grade scores correlated significantly with the trial examination scores and KMLE score in bivariate correlation analysis (P< 0.05). But only grade score during the senior years (5th and 6th) affected the trial examination and KMLE score in multiple regression. According to the multiple regressions by each grade score and trial examination score, the 6th grade score and trial exanimation score except the 1st trial examination score have a significant effect on the KMLE score. And we can also establish the regression formula such as [KMLE score]=11.377+0.752*[mean of trail examination] with significant high power of explanation (R (2) =0.709, P< 0.001). CONCLUSION: Our results show that previous scores always affect the latter scores; and the last score in time sequence is a result of the accumulation of previous education and learning. From this point of view, the trial examination is a useful tool for not only the final assessment of medical achievements but also for preparing for the KMLE.
Citations
Citations to this article as recorded by
Analysis on Validity and Academic Competency of Mock Test for Korean Medicine National Licensing Examination Using Item Response Theory Han Chae, Eunbyul Cho, SeonKyoung Kim, DaHye Choi, Seul Lee Keimyung Medical Journal.2023; 42(1): 7. CrossRef
A Study on the Variables of Success in the National Dental Hygienist Licence Examination at D College Sun-Hee Lee Journal of the Korea Academia-Industrial cooperation Society.2012; 13(8): 3486. CrossRef
Relationship between the Content of the Medical Knowledge Written Examination and Clinical Skill Score in Medical Students Jee-Young Hwang, Hwee-Soo Jeong Korean Journal of Medical Education.2011; 23(4): 305. CrossRef
The Relationship between Senior Year Examinations at a Medical School and the Korean Medical Licensing Examination Ki Hoon Jung, Ho Keun Jung, Kwan Lee Korean Journal of Medical Education.2009; 21(1): 17. CrossRef
PURPOSE This study analyzed the correlation between grade point average (GPA) of medical school and the score of Korean Medical Licensing Examination (KMLE). METHODS: This study based on the results of 67th KMLE applicants who graduated from a college of medicine in 2003. We also gathered data of these applicants from the college of medicine: gender, age, type of entrance, GPA of basic medicine, clinical medicine, clinical clerkships and final test scores. We analyzed whether there was discrimination between achievement of KMLE passed and that of KMLE failed, which of variables affected the results of KMLE. RESULTS: 173 applicants passed KMLE among 189. There were significant correlations between basic medicine, clinical medicine, final test score and the score of KMLE (respective p-value; < 0.0001). There were also significant differences of GPA between KMLE passed applicants and failed. Final test scores were the most correlated with those of KMLE. If the GPA of 2nd grade was below 2.5 and the GPA of 3rd year grade was below 2.3, they was a high-risk group for failing KMLE (sensitivity 100%, specificity 90%). CONCLUSION: There were significant correlations between the GPA of medical school and the score of KMLE, and significant differences between KMLE passed applicants and failed. A high-risk group of failing KMLE was the students that the GPA of 2nd grade was below 2.5 and the GPA of 3rd grade was below 2.3.
Citations
Citations to this article as recorded by
Analysis on Validity and Academic Competency of Mock Test for Korean Medicine National Licensing Examination Using Item Response Theory Han Chae, Eunbyul Cho, SeonKyoung Kim, DaHye Choi, Seul Lee Keimyung Medical Journal.2023; 42(1): 7. CrossRef
A Comparative Study of Predictive Factors for Passing the National Physical Therapy Examination using Logistic Regression Analysis and Decision Tree Analysis So Hyun Kim, Sung Hyoun Cho Physical Therapy Rehabilitation Science.2022; 11(3): 285. CrossRef
Correlation between academic achievements and admission criteria at the School of Dentistry, Chonnam National University Han-Joo Jung, Eun-Ju Lee, Min-Seok Kim Oral Biology Research.2019; 43(4): 289. CrossRef
A Six-Year Study of Relationship between Academic Performance in Dental Hygiene School and Performance on the Korean Dental Hygiene Licensing Examination at Yonsei University So-Jung Mun, Hie-Jin Noh, Hyun-Sun Jeon, Ji-Eun Heo, Won-Gyun Chung Journal of dental hygiene science.2014; 14(3): 332. CrossRef
Relationship between the Content of the Medical Knowledge Written Examination and Clinical Skill Score in Medical Students Jee-Young Hwang, Hwee-Soo Jeong Korean Journal of Medical Education.2011; 23(4): 305. CrossRef
The Relationship between Senior Year Examinations at a Medical School and the Korean Medical Licensing Examination Ki Hoon Jung, Ho Keun Jung, Kwan Lee Korean Journal of Medical Education.2009; 21(1): 17. CrossRef
In order to identify students who needs more intensive guide, we analysed the association between the results of Korean medical licensing examination(KMLE) and two different types of scholastic achievement; 4th grade score and graduation examination score. The scholastic achievement between passed and failed the KMLE at Konkuk University College of Medicine from 1995 to 1998 was investigated.
First, we grouped the students into four groups according to the year they graduated. Second, we regrouped the group into two subgroups according to whether passed or failed in the KMLE. In the 4th grade score, the passed KMLE is higher than the failed from 0.49 to 0.71 and which is statistically significant. In the graduation examination score, the passed KMLE is higher than the failed from 4.9 to 9.9 and which is also statistically significant.
We analyzed the degree of agreements between the results of the KMLE and the two different types of scholastic achievement. In 4th grade score, the highest degree of agreement shows 2.2 and its kappa index is 0.49. In graduation examination score, the highest degree of agreement shows 71 and its kappa index is 0.64.
In conclusion, students who have lower than 2.2 in 4th grade score or lower than 71 in graduation examination score are tend to be failed the KMLE. So they need more intensive guide for the KMLE.
Citations
Citations to this article as recorded by
The Relationship between Senior Year Examinations at a Medical School and the Korean Medical Licensing Examination Ki Hoon Jung, Ho Keun Jung, Kwan Lee Korean Journal of Medical Education.2009; 21(1): 17. CrossRef
First
Prev
