Purpose This study investigated the relationship between the item response time (iRT) and classic item analysis indicators obtained from computer-based test (CBT) results and deduce students’ problem-solving behavior using the relationship.
Methods We retrospectively analyzed the results of the Comprehensive Basic Medical Sciences Examination conducted for 5 years by a CBT system in Dankook University College of Medicine. iRT is defined as the time spent to answer the question. The discrimination index and the difficulty level were used to analyze the items using classical test theory (CTT). The relationship of iRT and the CTT were investigated using a correlation analysis. An analysis of variance was performed to identify the difference between iRT and difficulty level. A regression analysis was conducted to examine the effect of the difficulty index and discrimination index on iRT.
Results iRT increases with increasing difficulty index, and iRT tends to decrease with increasing discrimination index. The students’ effort is increased when they solve difficult items but reduced when they are confronted with items with a high discrimination. The students’ test effort represented by iRT was properly maintained when the items have a ‘desirable’ difficulty and a ‘good’ discrimination.
Conclusion The results of our study show that an adequate degree of item difficulty and discrimination is required to increase students’ motivation. It might be inferred that with the combination of CTT and iRT, we can gain insights about the quality of the examination and test behaviors of the students, which can provide us with more powerful tools to improve them.
Citations
Citations to this article as recorded by
Conditional Dependencies Between Response Time and Item Discrimination: An Item-Level Meta-Analysis Joshua B. Gilbert, William S. Young, Zachary Himmelsbach, Esther Ulitzsch, Benjamin W. Domingue Educational and Psychological Measurement.2026; 86(5): 1039. CrossRef
Measuring coach attitudes toward toxic coaching practice in youth sport: Initial validation of the ATCPM-6 Julie McCleery, Longxi Li, Irina Tereschenko International Journal of Sports Science & Coaching.2026;[Epub] CrossRef
A preliminary analysis of the psychometric properties of the German O*NET interest profiler short form in rehabilitation education Steffen Wild, Michélle Möhring Frontiers in Rehabilitation Sciences.2026;[Epub] CrossRef
Comparison of the response time-based effort-moderated IRT model and three-parameter logistic model according to computerized adaptive test performances: a simulation study Yusuf Kemal Arslan, Afra Alkan, Atilla Halil Elhan Communications in Statistics - Simulation and Computation.2025; 54(1): 44. CrossRef
Toward sustainable farming: Assessing and validating green skills for agricultural professionals in China Bowei Hu, Khahan Na-Nan, Yotsaphat Kittichotsatsawat Environmental Challenges.2025; 18: 101067. CrossRef
Knowledge, attitude and practice towards multiple myeloma among medical staff in Enshi Region Luping Zou, Jinhua Li, Hang Xiang, Jun Tan, Yan Zeng Scientific Reports.2025;[Epub] CrossRef
The Competent Computational Thinking Test (cCTt): A Valid, Reliable and Gender-Fair Test for Longitudinal CT Studies in Grades 3–6 Laila El-Hamamsy, María Zapata-Cáceres, Estefanía Martín-Barroso, Francesco Mondada, Jessica Dehler Zufferey, Barbara Bruno, Marcos Román-González Technology, Knowledge and Learning.2025; 30(3): 1607. CrossRef
Comparison of a generative large language model to pharmacy student performance on therapeutics examinations Christopher J. Edwards, Bernadette Cornelison, Brian L. Erstad Currents in Pharmacy Teaching and Learning.2025; 17(9): 102394. CrossRef
Designing Biochemical Visual Literacy Assessments: Insights from Classroom Testing and Student Interviews Kristen Procko, Josh T. Beckham, Roderico Acevedo, Swati Agrawal, Shane Austin, Charmita Burch, Shelly Engelman, Kristin M. Fox, Lauren A. Genova, Pamela S. Mertz, Rachel M. Mitton-Fry, Didem Vardar-Ulu Journal of Chemical Education.2025; 102(12): 5045. CrossRef
The impact of repeated item development training on the prediction of medical faculty members’ item difficulty index Hye Yoon Lee, So Jung Yune, Sang Yeoup Lee, Sunju Im, Bee Sung Kam BMC Medical Education.2024;[Epub] CrossRef
Identification of parameters for electronic distance examinations Robin Richter, Andrea Tipold, Elisabeth Schaper Frontiers in Veterinary Science.2024;[Epub] CrossRef
Validation of the Perceived Islamophobia Scale (PIS) among Muslims living in the United States Khulud Almutairi, Salman Shaheen Ahmad, Merranda Marie McLaughlin, Karina Gattamorta, Amy Weisman de Mamani Social Sciences & Humanities Open.2024; 10: 101054. CrossRef
Analysis of Nutrition Knowledge After One Year of Intervention in a National Extracurricular Athletics Program: A Cross-Sectional Study with Pair-Matched Controls of Polish Adolescents Dominika Skolmowska, Dominika Głąbska, Dominika Guzek, Jakub Grzegorz Adamczyk, Hanna Nałęcz, Blanka Mellová, Katarzyna Żywczyk, Krystyna Gutkowska Nutrients.2024; 17(1): 64. CrossRef
Differences in Multiple-Choice Questions of Opposite Stem Orientations Based on a Novel Item Quality Measure Samuel Olusegun Adeosun American Journal of Pharmaceutical Education.2023; 87(2): ajpe8934. CrossRef
Examination of response time effort in TIMSS 2019: Comparison of Singapore and Türkiye Esin YILMAZ KOĞAR, Sümeyra SOYSAL International Journal of Assessment Tools in Education.2023; 10(Special Is): 174. CrossRef
The development and validation of a questionnaire to assess relative energy deficiency in sport (RED-S) knowledge Namratha N. Pai, Rachel C. Brown, Katherine E. Black Journal of Science and Medicine in Sport.2022; 25(10): 794. CrossRef
Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners' CT test (BCTt) and the competent CT test (cCTt) Laila El-Hamamsy, María Zapata-Cáceres, Pedro Marcelino, Barbara Bruno, Jessica Dehler Zufferey, Estefanía Martín-Barroso, Marcos Román-González Frontiers in Psychology.2022;[Epub] CrossRef
Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students Youngsup Christopher Lee, Oh Young Kwon, Ho Jin Hwang, Seok Hoon Ko Korean Journal of Medical Education.2020; 32(3): 213. CrossRef
Effect of Smart Device Ability on the Smart Device-Based Testing National Board Examination for Optometry Students Eun Joo Kim, Koon-Ja Lee, Jung Un Jang The Korean Journal of Vision Science.2019; 21(4): 631. CrossRef
PURPOSE In 2002, extended-matching type (R-type) items were introduced to the Korean Medical Licensing Examination. To evaluate the usability of R-type items, the results of the Korean Medical Licensing Examination in 2002 and 2003 were analyzed based on item types and knowledge levels. METHODS: Item parameters, such as difficulty and discrimination indexes, were calculated using the classical test theory.
The item parameters were compared across three item types and three knowledge levels. RESULTS: The values of R-type item parameters were higher than those of A- or K-type items. There was no significant difference in item parameters according to knowledge level, including recall, interpretation, and problem solving. The reliability of R-type items exceeded 0.99. With the R-type, an increasing number in correct answers was associated with a decreasing difficulty index. CONCLUSION: The introduction of R-type items is favorable from the perspective of item parameters.
However, an increase in the number of correct answers in pick 'n'-type questions results in the items being more difficult to solve.
Citations
Citations to this article as recorded by
Reforms of the Korean Medical Licensing Examination regarding item development and performance evaluation Mi Kyoung Yim Journal of Educational Evaluation for Health Professions.2015; 12: 6. CrossRef
After item analysis of examinations in College of Medicine, the correlation among characteristics were examined for the better understanding of their meaning. The 78 subjected examinations in College of Medicine, Hallym University, Korea from March 1999 to October 2000 were analyzed.
Discrimination indexes (D) by the method of extreme group were positively correlated with item-total correlation(ITC) with mean correlation coefficient r=0.8506 ranged between 0.6430 and 0.9520. Number of items of each examination was positively correlated with Cronbach coefficient alpha reliability index(r=0.7920) whereas negatively correlated with standard deviation(r=-0.5691), odd-even split reliability index(r=-0.8767) and mean ITC(r=-0.4079).
Thereafter, the standard deviation was positively correlated with odd-even split reliability index (r=0.5072) and mean ITC(r=0.6166). There was negative correlation between Cronbach coefficient alpha reliability index and odd-even split reliability index(r=-0.7385). Above results suggested that the number of items in each examination was most powerful factor affecting to other item analysis characteristics. The appropriate number of items should be considered for better result of item analysis characteristics. Odd-even split reliability index is not appropriate for the estimation of the reliability among item, since it decreased according to the increase of number of items. Positive and high correlation between D and ITC means that both methods are appropriate to interpretate the discriminating power of the items.
Item analysis is the evaluating process of items used for tests. Item difficulty, discrimination, and distractor analysis are the main components of the analysis.
Discrimination index(D) by the method of extreme groups had been used for the item discrimination, but it had been known to have some disadvantages compared to item-total correlation(ITC).
This study was conducted to evaluated the feasibility and the advantages of the ITC. Medical specialist qualifying examination carried out in Jan. 1999 was selected for the study material and the items of tests for the 4 major disciplines(internal medicine, general surgery, pediatrics, and obstetrics & gynecology) were analysed.
The numbers of the items and examinee are 120 items/428 persons, 140/219, 140/229, and 140/226 (in the order of IM, GS, Ped, OB & Gyn) respectively.
The average discrimination index(D) of all items is 0.170 and the standard deviation is 0.120. For the ITC, average is 0.210 and standard deviation is 0.117. There is positive correlation between D and ITC(r=0.677).
The variation of the ITC is 0.880, which is wider than that of discrimination index(D), 0.713. Especially on the items with item's p-value greater than 0.9(n=140), the variations are 0.542 and 0.273 respectively. The difference is much distinct.
These results imply that ITC can be used as the index of the item discrimination, and has some advantages compared to discrimination index(D). The advantages are the significance of the number itself and rather independence from the item difficulty.
Citations
Citations to this article as recorded by
Cross-cultural adaptation and validation of the Korean version of the Post-Intensive Care Syndrome Knowledge Test (K-PICS-KT): a methodological study Jiyoon Kang, Hyosung Cha Journal of Korean Biological Nursing Science.2026; 28(3): 413. CrossRef
Turkish Validity and Reliability Study of ‘The Children’s Trust in General Nurses Scale’ Gülsüm Gülcenbay, Türkan Turan Journal of Basic and Clinical Health Sciences.2025; 9(2): 309. CrossRef
Direct measurement of learning outcomes in higher education: A proposal of nine standardized scales for continuous improvement in engineering programs Mónica Hernández-Campos, Jorge Esteban Prado-Calderón, Antonio Gonzalez-Torres, Francisco José García-Peñalvo Evaluation and Program Planning.2025; 112: 102638. CrossRef
Development of the Knowledge Scale of the Life-Sustaining Treatment for Clinical Nurses Sojung Park, Mihyun Park, Suyoun Hong Korean Journal of Adult Nursing.2022; 34(5): 488. CrossRef
Preliminary validation of the Dental Clinical Learning Environment Instrument in a Brazilian dental school Nicole Krois, Anastasia Kossioni, Patrick B. Barlow, Mateus Bertolini Fernandes dos Santos, Eduarda Carrera Malhão, Leonardo Marchini European Journal of Dental Education.2021; 25(1): 5. CrossRef
Methodology for developing and evaluating diagnostic tools in Ayurveda – A review Mukesh Edavalath, Benil P. Bharathan Journal of Ayurveda and Integrative Medicine.2021; 12(2): 389. CrossRef
Psychometric Properties of the Schizophrenia Oral Health Profile: Preliminary Results Frédéric Denis, Ines Rouached, Francesca Siu-Paredes, Alexis Delpierre, Gilles Amador, Wissam El-Hage, Nathalie Rude International Journal of Environmental Research and Public Health.2021; 18(17): 9090. CrossRef
Dimensional Structure and Preliminary Results of the External Constructs of the Schizophrenia Coping Oral Health Profile and Index (SCOOHPI) Francesca Siu-Paredes, Nathalie Rude, Ines Rouached, Corinne Rat, Rachid Mahalli, Wissam El-Hage, Katherine Rozas, Frédéric Denis International Journal of Environmental Research and Public Health.2021; 18(23): 12413. CrossRef
Cross-cultural adaptation and measurement properties of the Patient-Rated Tennis Elbow Evaluation for the Persian language Erfan Shafiee, Maryam Farzad, Joy Macdermid, Amirreza Smaeel Beygi, Atefeh Vafaei, Amirreza Farhoud Hand Therapy.2020; 25(2): 56. CrossRef
Steps towards validation of the Dental Education Clinical Learning Instrument (DECLEI) in American dental schools (DECLEI‐USA) Nicole R. Krois, Anastassia E. Kossioni, Patrick B. Barlow, Maryam Tabrizi, Leonardo Marchini Journal of Dental Education.2020; 84(8): 895. CrossRef
GAME Takahiro Miura, Masaki Matsuo, Ken-ichiro Yabu, Atsushi Katagiri, Masatsugu Sakajiri, Junji Onishi, Takeshi Kurata, Tohru Ifukube Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.2020; 4(4): 1. CrossRef
Psychometric properties of the chinese version of autism spectrum quotient‐children's version: A sex‐specific analysis Fan Sun, Meixia Dai, Lizi Lin, Xiang Sun, Aja Louise Murray, Bonnie Auyeung, Jin Jing Autism Research.2019; 12(2): 303. CrossRef
Applicability of Item Response Theory to the Korean Nurses' Licensing Examination Geum-Hee Jeong, Mi Kyoung Yim Journal of Educational Evaluation for Health Professions.2005; 2(1): 23. CrossRef
First
Prev
