Purpose This study investigated the relationship between the item response time (iRT) and classic item analysis indicators obtained from computer-based test (CBT) results and deduce students’ problem-solving behavior using the relationship.
Methods We retrospectively analyzed the results of the Comprehensive Basic Medical Sciences Examination conducted for 5 years by a CBT system in Dankook University College of Medicine. iRT is defined as the time spent to answer the question. The discrimination index and the difficulty level were used to analyze the items using classical test theory (CTT). The relationship of iRT and the CTT were investigated using a correlation analysis. An analysis of variance was performed to identify the difference between iRT and difficulty level. A regression analysis was conducted to examine the effect of the difficulty index and discrimination index on iRT.
Results iRT increases with increasing difficulty index, and iRT tends to decrease with increasing discrimination index. The students’ effort is increased when they solve difficult items but reduced when they are confronted with items with a high discrimination. The students’ test effort represented by iRT was properly maintained when the items have a ‘desirable’ difficulty and a ‘good’ discrimination.
Conclusion The results of our study show that an adequate degree of item difficulty and discrimination is required to increase students’ motivation. It might be inferred that with the combination of CTT and iRT, we can gain insights about the quality of the examination and test behaviors of the students, which can provide us with more powerful tools to improve them.
Citations
Citations to this article as recorded by
Conditional Dependencies Between Response Time and Item Discrimination: An Item-Level Meta-Analysis Joshua B. Gilbert, William S. Young, Zachary Himmelsbach, Esther Ulitzsch, Benjamin W. Domingue Educational and Psychological Measurement.2026;[Epub] CrossRef
Comparison of the response time-based effort-moderated IRT model and three-parameter logistic model according to computerized adaptive test performances: a simulation study Yusuf Kemal Arslan, Afra Alkan, Atilla Halil Elhan Communications in Statistics - Simulation and Computation.2025; 54(1): 44. CrossRef
Toward sustainable farming: Assessing and validating green skills for agricultural professionals in China Bowei Hu, Khahan Na-Nan, Yotsaphat Kittichotsatsawat Environmental Challenges.2025; 18: 101067. CrossRef
Knowledge, attitude and practice towards multiple myeloma among medical staff in Enshi Region Luping Zou, Jinhua Li, Hang Xiang, Jun Tan, Yan Zeng Scientific Reports.2025;[Epub] CrossRef
The Competent Computational Thinking Test (cCTt): A Valid, Reliable and Gender-Fair Test for Longitudinal CT Studies in Grades 3–6 Laila El-Hamamsy, María Zapata-Cáceres, Estefanía Martín-Barroso, Francesco Mondada, Jessica Dehler Zufferey, Barbara Bruno, Marcos Román-González Technology, Knowledge and Learning.2025; 30(3): 1607. CrossRef
Comparison of a generative large language model to pharmacy student performance on therapeutics examinations Christopher J. Edwards, Bernadette Cornelison, Brian L. Erstad Currents in Pharmacy Teaching and Learning.2025; 17(9): 102394. CrossRef
Designing Biochemical Visual Literacy Assessments: Insights from Classroom Testing and Student Interviews Kristen Procko, Josh T. Beckham, Roderico Acevedo, Swati Agrawal, Shane Austin, Charmita Burch, Shelly Engelman, Kristin M. Fox, Lauren A. Genova, Pamela S. Mertz, Rachel M. Mitton-Fry, Didem Vardar-Ulu Journal of Chemical Education.2025; 102(12): 5045. CrossRef
The impact of repeated item development training on the prediction of medical faculty members’ item difficulty index Hye Yoon Lee, So Jung Yune, Sang Yeoup Lee, Sunju Im, Bee Sung Kam BMC Medical Education.2024;[Epub] CrossRef
Identification of parameters for electronic distance examinations Robin Richter, Andrea Tipold, Elisabeth Schaper Frontiers in Veterinary Science.2024;[Epub] CrossRef
Validation of the Perceived Islamophobia Scale (PIS) among Muslims living in the United States Khulud Almutairi, Salman Shaheen Ahmad, Merranda Marie McLaughlin, Karina Gattamorta, Amy Weisman de Mamani Social Sciences & Humanities Open.2024; 10: 101054. CrossRef
Analysis of Nutrition Knowledge After One Year of Intervention in a National Extracurricular Athletics Program: A Cross-Sectional Study with Pair-Matched Controls of Polish Adolescents Dominika Skolmowska, Dominika Głąbska, Dominika Guzek, Jakub Grzegorz Adamczyk, Hanna Nałęcz, Blanka Mellová, Katarzyna Żywczyk, Krystyna Gutkowska Nutrients.2024; 17(1): 64. CrossRef
Differences in Multiple-Choice Questions of Opposite Stem Orientations Based on a Novel Item Quality Measure Samuel Olusegun Adeosun American Journal of Pharmaceutical Education.2023; 87(2): ajpe8934. CrossRef
Examination of response time effort in TIMSS 2019: Comparison of Singapore and Türkiye Esin YILMAZ KOĞAR, Sümeyra SOYSAL International Journal of Assessment Tools in Education.2023; 10(Special Is): 174. CrossRef
The development and validation of a questionnaire to assess relative energy deficiency in sport (RED-S) knowledge Namratha N. Pai, Rachel C. Brown, Katherine E. Black Journal of Science and Medicine in Sport.2022; 25(10): 794. CrossRef
Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners' CT test (BCTt) and the competent CT test (cCTt) Laila El-Hamamsy, María Zapata-Cáceres, Pedro Marcelino, Barbara Bruno, Jessica Dehler Zufferey, Estefanía Martín-Barroso, Marcos Román-González Frontiers in Psychology.2022;[Epub] CrossRef
Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students Youngsup Christopher Lee, Oh Young Kwon, Ho Jin Hwang, Seok Hoon Ko Korean Journal of Medical Education.2020; 32(3): 213. CrossRef
Effect of Smart Device Ability on the Smart Device-Based Testing National Board Examination for Optometry Students Eun Joo Kim, Koon-Ja Lee, Jung Un Jang The Korean Journal of Vision Science.2019; 21(4): 631. CrossRef
Purpose The purpose of this study was to evaluate the suitability (convenience, objectiveness, and satisfaction) of ubiquitous-based testing (UBT) as a medical education evaluation tool.
Methods UBT was administered using a smart pad in our medical school in May 2012. A questionnaire was given twice. The pre-UBT questionnaire examined possession of a tablet computer, skillfulness of smart devices, the convenience of UBT, and the usefulness of a medical educational assessment tool. The post-UBT questionnaire evaluated the satisfaction, convenience, and preference of UBT and the usefulness of a medical educational assessment tool, as in the pre-UBT test. The survey was measured on a 4-point scale: 1 is “strongly disagree” and 4 is “strongly agree.”
Results One hundred three students (male, 55.3%) participated in the UBT. The mean age was 29.2±2.4 years. In the pre-UBT questionnaire analysis, students responded affirmatively to the items about the skillfulness of smart devices, clinical skill assessment, and achievement of educational objectives. The responses to the items on the convenience and satisfaction with the UBT were positive in the post-UBT. The factors that affected the post-UBT questionnaire were as follows: knowledge assessment (p=0.041) and achievement of educational objectives (p=0.015) were significant, based on gender, and satisfaction with the UBT (p=0.002) was significant, based on possession of a tablet computer. The relationship between the ranks of this UBT and the average ranks of the three previous semesters was statistically significant (p<0.001).
Conclusion Convenience, objectiveness, knowledge assessment, and composition and completion were useful items in the UBT.
Citations
Citations to this article as recorded by
Ubiquitous‐Based Testing in Dental Education: Insights From the Unified Theory of Acceptance and Use of Technology Jae‐Hoon Kim, Hyoung Seok Shin Journal of Dental Education.2026; 90(6): 801. CrossRef
Survey of dental students’ perception of ubiquitous-based test (UBT) Hyoung Seok Shin, Jae-Hoon Kim The Journal of The Korean Dental Association.2024; 62(5): 270. CrossRef
Perceptions of the application of computer-based testing in the National Examination for Dental Technicians Sun-Kyoung Lee Journal of Korean Acedemy of Dental Technology.2024; 46(4): 182. CrossRef
Development and application of a mobile-based multimedia nursing competency evaluation system for nursing students: A mixed-method randomized controlled study Soyoung Jang, Eunyoung E. Suh Nurse Education in Practice.2022; 64: 103458. CrossRef
Development, Application, and Effectiveness of a Smart Device-based Nursing Competency Evaluation Test Soyoung Jang, Eunyoung E. Suh CIN: Computers, Informatics, Nursing.2021; 39(11): 634. CrossRef
Evaluation of Student Satisfaction with Ubiquitous-Based Tests in Women’s Health Nursing Course Mi-Young An, Yun-Mi Kim Healthcare.2021; 9(12): 1664. CrossRef
Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students Youngsup Christopher Lee, Oh Young Kwon, Ho Jin Hwang, Seok Hoon Ko Korean Journal of Medical Education.2020; 32(3): 213. CrossRef
Analysis of the Satisfaction with Computer Based Test Program and Test Environment in Medical School Soon Gu Kim, Aehwa Lee, Ilseon Hwang Korean Medical Education Review.2020; 22(3): 198. CrossRef
Effect of Smart Device Ability on the Smart Device-Based Testing National Board Examination for Optometry Students Eun Joo Kim, Koon-Ja Lee, Jung Un Jang The Korean Journal of Vision Science.2019; 21(4): 631. CrossRef
Smart device-based testing for medical students in Korea: satisfaction, convenience, and advantages Eun Young Lim, Mi Kyoung Yim, Sun Huh Journal of Educational Evaluation for Health Professions.2017; 14: 7. CrossRef
Evaluation in medical education Tai Young Yoon Korean Journal of Medical Education.2015; 27(1): 1. CrossRef
PURPOSE The purpose of this study was to establish an e-learning system to support learning in medical education and identify solutions for improving the system.
METHODS A learning management system (LMS) and computer-based test (CBT) system were established to support e-learning for medical students. A survey of 219 first- and second-grade medical students was administered. The questionnaire included 9 forced choice questions about the usability of system and 2 open-ended questions about necessary improvements to the system.
RESULTS The LMS consisted of a class management, class evaluation, and class attendance system. CBT consisted of a test management, item bank, and authoring tool system. The results of the survey showed a high level of satisfaction in all system usability items except for stability. Further, the advantages of the e-learning system were ensuring information accessibility, providing constant feedback, and designing an intuitive interface. Necessary improvements to the system were stability, user control, readability, and diverse device usage.
CONCLUSION Based on the findings, suggestions for developing an e-learning system to improve usability by medical students and support learning effectively are recommended.
Citations
Citations to this article as recorded by
Digital Innovation in Medical Education: The Process and Challenges of Digital Transformation Youngjon Kim Korean Medical Education Review.2025; 27(1): 6. CrossRef
Survey of dental students’ perception of ubiquitous-based test (UBT) Hyoung Seok Shin, Jae-Hoon Kim The Journal of The Korean Dental Association.2024; 62(5): 270. CrossRef
Perceptions of the application of computer-based testing in the National Examination for Dental Technicians Sun-Kyoung Lee Journal of Korean Acedemy of Dental Technology.2024; 46(4): 182. CrossRef
Study on the Academic Competency Assessment of Herbology Test using Rasch Model Han Chae, Soo Jin Lee, Chang-ho Han, Young Il Cho, Hyungwoo Kim Journal of Korean Medicine.2022; 43(2): 27. CrossRef
Evaluation of Student Satisfaction with Ubiquitous-Based Tests in Women’s Health Nursing Course Mi-Young An, Yun-Mi Kim Healthcare.2021; 9(12): 1664. CrossRef
Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students Youngsup Christopher Lee, Oh Young Kwon, Ho Jin Hwang, Seok Hoon Ko Korean Journal of Medical Education.2020; 32(3): 213. CrossRef
Analysis of the Satisfaction with Computer Based Test Program and Test Environment in Medical School Soon Gu Kim, Aehwa Lee, Ilseon Hwang Korean Medical Education Review.2020; 22(3): 198. CrossRef
An Examination and Evaluation of Question Preparation for Computer-based Testing in Dental Education Hiroya Gotouda, Hitoshi Nishimura, Hiroko Matsumoto, Kaori Ishii, Makiko Kageyama, Takahiro Yazaki, Naotaka Kamiya, Junichi Yamane, Masanori Saito, Suguru Kimoto, Hiroyasu Endo, Masanobu Wakami, Ryo Tamamura, Hiroyuki Okada, Takehiko Shimizu, Koichi Hirat International Journal of Oral-Medical Sciences.2017; 15(2): 46. CrossRef
Computer‐Based Testing and Construction of an Item Bank Database for Medical Education in Korea Sun Huh Korean Medical Education Review.2014; 16(1): 11. CrossRef
Development of an Outcome-Based Medical Curriculum: A Case Report from The Catholic University of Korea School of Medicine Sun Kim, Joo Hyun Park, Nam Jin Yoo, Soo Jung Lee Korean Medical Education Review.2013; 15(1): 39. CrossRef
Survival Games at Medical Schools Young-Mee Lee Korean Journal of Medical Education.2012; 24(3): 185. CrossRef
PURPOSE The purpose of this study is to understand student recognition of CBT as well as its strengths and weaknesses, and to explore the improvement methodologies for the effective development and implementation of CBT. METHODS: A questionnaire survey was conducted twice (before and after implementation of CBT) with a total of 17 multiple-choice and 2 essay-type questions. The multiple-choice questions were analyzed by frequency analysis and the essay-type questions were coded by content analysis. RESULTS: The results are as follows. First, the overall satisfaction with CBT was shown to be high. Second, students listed the merits of CBT as follows: simple correction of answers (before: 89.4%; after: 80.8%), presentation of realistic materials (before: 72.9%; after: 84.7%), prompt feedback on grades (before: 60.3%; after: 71.1%), shortened exam time and effective time scheduling (before: 86.5%; after: 66.4%), accurate estimation of abilities (before: 70.2%; after: 36.6%), and assistance in academic improvement (before: 70.9%; after: 22.1%). Drawbacks of CBT were: inconvenience of review (before: 70.9%; after: 22.1%), inconvenient screen organizations (before: 0%; after: 48.1%), possibility of cheating (before: 73.9%; after: 31.8%), and equality issue of test (before: 47.3%; after: 17.3%). CONCLUSION: Assessment paradigms are currently shifting from summative evaluation to formative evaluation, from one-off assessment to continuous assessment, and from output assessment to process assessment. Therefore, CBT must be expanded to move from result-oriented summative evaluation to formative evaluation continuously monitoring the student learning process.
Citations
Citations to this article as recorded by
Development and Evaluation of Computerized Adaptive Testing for General English Proficiency Test Hyeon Park, Woojae Han, Mikyung Park, Dong gi Seo The Korean Association of General Education.2026; 20(2): 191. CrossRef
Dental technicians’ satisfaction with a computer-based mock national examination Sun-Kyoung Lee Journal of Korean Acedemy of Dental Technology.2025; 47(2): 156. CrossRef
Survey of dental students’ perception of ubiquitous-based test (UBT) Hyoung Seok Shin, Jae-Hoon Kim The Journal of The Korean Dental Association.2024; 62(5): 270. CrossRef
Perceptions of the application of computer-based testing in the National Examination for Dental Technicians Sun-Kyoung Lee Journal of Korean Acedemy of Dental Technology.2024; 46(4): 182. CrossRef
Novel innovative computer-based test (Inno-CBT) item types for national licensing examinations for health care professionals Kwang-Hoon Chun, Hye Kyung Jin, Jeong-Hyun Yoon, Myeong Gyu Kim, Kyung Hee Choi, Eunyoung Kim, Hyunah Kim, Jin-Ki Kim, Gyudong Kim, Kyungim Kim, Ju-Yeun Lee, Eun Kyoung Chung, Young Sook Lee, Sandy Jeong Rhie BMC Medical Education.2023;[Epub] CrossRef
Speech-Language Pathologists’ Perceptions and Satisfaction towards Computer-Based National Examination Eun Kyoung Lee, Ji-Wan Ha, Hyun Jung Lee, Seong Hee Choi Communication Sciences & Disorders.2023; 28(4): 884. CrossRef
Study on the Academic Competency Assessment of Herbology Test using Rasch Model Han Chae, Soo Jin Lee, Chang-ho Han, Young Il Cho, Hyungwoo Kim Journal of Korean Medicine.2022; 43(2): 27. CrossRef
Accuracy and Efficiency of Web-based Assessment Platform (LIVECAT) for Computerized Adaptive Testing Do-Gyeong Kim, Dong-Gi Seo The Journal of Korean Institute of Information Technology.2020; 18(4): 77. CrossRef
Analysis of the Satisfaction with Computer Based Test Program and Test Environment in Medical School Soon Gu Kim, Aehwa Lee, Ilseon Hwang Korean Medical Education Review.2020; 22(3): 198. CrossRef
Computer‐Based Testing and Construction of an Item Bank Database for Medical Education in Korea Sun Huh Korean Medical Education Review.2014; 16(1): 11. CrossRef
Design and Development of a Medical Education System Using Information Technology: A Case Report from the Pusan National University School of Medicine Sun Ju Im, Sang Yeoup Lee, Sun Yong Baek, Jae Seok Woo, Beesung Kam Korean Medical Education Review.2014; 16(1): 16. CrossRef
Can computerized tests be introduced to the Korean Medical Licensing Examination? Sun Huh Journal of the Korean Medical Association.2012; 55(2): 124. CrossRef
Experiences with Establishing and Implementing Learning Management System and Computer-Based Test System in Medical College Joo Hyun Park, Ji Young Son, Sun Kim Korean Journal of Medical Education.2012; 24(3): 213. CrossRef
The Development Study of a Medical Ethics Education Program by Using Simple ISD Model Young Jon Kim, Hyoo Hyun Yoo, Chan Uhng Joo Korean Journal of Medical Education.2011; 23(3): 185. CrossRef
First
Prev
