Purpose This study investigated the relationship between the item response time (iRT) and classic item analysis indicators obtained from computer-based test (CBT) results and deduce students’ problem-solving behavior using the relationship.
Methods We retrospectively analyzed the results of the Comprehensive Basic Medical Sciences Examination conducted for 5 years by a CBT system in Dankook University College of Medicine. iRT is defined as the time spent to answer the question. The discrimination index and the difficulty level were used to analyze the items using classical test theory (CTT). The relationship of iRT and the CTT were investigated using a correlation analysis. An analysis of variance was performed to identify the difference between iRT and difficulty level. A regression analysis was conducted to examine the effect of the difficulty index and discrimination index on iRT.
Results iRT increases with increasing difficulty index, and iRT tends to decrease with increasing discrimination index. The students’ effort is increased when they solve difficult items but reduced when they are confronted with items with a high discrimination. The students’ test effort represented by iRT was properly maintained when the items have a ‘desirable’ difficulty and a ‘good’ discrimination.
Conclusion The results of our study show that an adequate degree of item difficulty and discrimination is required to increase students’ motivation. It might be inferred that with the combination of CTT and iRT, we can gain insights about the quality of the examination and test behaviors of the students, which can provide us with more powerful tools to improve them.
Citations
Citations to this article as recorded by
Conditional Dependencies Between Response Time and Item Discrimination: An Item-Level Meta-Analysis Joshua B. Gilbert, William S. Young, Zachary Himmelsbach, Esther Ulitzsch, Benjamin W. Domingue Educational and Psychological Measurement.2026;[Epub] CrossRef
Comparison of the response time-based effort-moderated IRT model and three-parameter logistic model according to computerized adaptive test performances: a simulation study Yusuf Kemal Arslan, Afra Alkan, Atilla Halil Elhan Communications in Statistics - Simulation and Computation.2025; 54(1): 44. CrossRef
Toward sustainable farming: Assessing and validating green skills for agricultural professionals in China Bowei Hu, Khahan Na-Nan, Yotsaphat Kittichotsatsawat Environmental Challenges.2025; 18: 101067. CrossRef
Knowledge, attitude and practice towards multiple myeloma among medical staff in Enshi Region Luping Zou, Jinhua Li, Hang Xiang, Jun Tan, Yan Zeng Scientific Reports.2025;[Epub] CrossRef
The Competent Computational Thinking Test (cCTt): A Valid, Reliable and Gender-Fair Test for Longitudinal CT Studies in Grades 3–6 Laila El-Hamamsy, María Zapata-Cáceres, Estefanía Martín-Barroso, Francesco Mondada, Jessica Dehler Zufferey, Barbara Bruno, Marcos Román-González Technology, Knowledge and Learning.2025; 30(3): 1607. CrossRef
Comparison of a generative large language model to pharmacy student performance on therapeutics examinations Christopher J. Edwards, Bernadette Cornelison, Brian L. Erstad Currents in Pharmacy Teaching and Learning.2025; 17(9): 102394. CrossRef
Designing Biochemical Visual Literacy Assessments: Insights from Classroom Testing and Student Interviews Kristen Procko, Josh T. Beckham, Roderico Acevedo, Swati Agrawal, Shane Austin, Charmita Burch, Shelly Engelman, Kristin M. Fox, Lauren A. Genova, Pamela S. Mertz, Rachel M. Mitton-Fry, Didem Vardar-Ulu Journal of Chemical Education.2025; 102(12): 5045. CrossRef
The impact of repeated item development training on the prediction of medical faculty members’ item difficulty index Hye Yoon Lee, So Jung Yune, Sang Yeoup Lee, Sunju Im, Bee Sung Kam BMC Medical Education.2024;[Epub] CrossRef
Identification of parameters for electronic distance examinations Robin Richter, Andrea Tipold, Elisabeth Schaper Frontiers in Veterinary Science.2024;[Epub] CrossRef
Validation of the Perceived Islamophobia Scale (PIS) among Muslims living in the United States Khulud Almutairi, Salman Shaheen Ahmad, Merranda Marie McLaughlin, Karina Gattamorta, Amy Weisman de Mamani Social Sciences & Humanities Open.2024; 10: 101054. CrossRef
Analysis of Nutrition Knowledge After One Year of Intervention in a National Extracurricular Athletics Program: A Cross-Sectional Study with Pair-Matched Controls of Polish Adolescents Dominika Skolmowska, Dominika Głąbska, Dominika Guzek, Jakub Grzegorz Adamczyk, Hanna Nałęcz, Blanka Mellová, Katarzyna Żywczyk, Krystyna Gutkowska Nutrients.2024; 17(1): 64. CrossRef
Differences in Multiple-Choice Questions of Opposite Stem Orientations Based on a Novel Item Quality Measure Samuel Olusegun Adeosun American Journal of Pharmaceutical Education.2023; 87(2): ajpe8934. CrossRef
Examination of response time effort in TIMSS 2019: Comparison of Singapore and Türkiye Esin YILMAZ KOĞAR, Sümeyra SOYSAL International Journal of Assessment Tools in Education.2023; 10(Special Is): 174. CrossRef
The development and validation of a questionnaire to assess relative energy deficiency in sport (RED-S) knowledge Namratha N. Pai, Rachel C. Brown, Katherine E. Black Journal of Science and Medicine in Sport.2022; 25(10): 794. CrossRef
Comparing the psychometric properties of two primary school Computational Thinking (CT) assessments for grades 3 and 4: The Beginners' CT test (BCTt) and the competent CT test (cCTt) Laila El-Hamamsy, María Zapata-Cáceres, Pedro Marcelino, Barbara Bruno, Jessica Dehler Zufferey, Estefanía Martín-Barroso, Marcos Román-González Frontiers in Psychology.2022;[Epub] CrossRef
Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students Youngsup Christopher Lee, Oh Young Kwon, Ho Jin Hwang, Seok Hoon Ko Korean Journal of Medical Education.2020; 32(3): 213. CrossRef
Effect of Smart Device Ability on the Smart Device-Based Testing National Board Examination for Optometry Students Eun Joo Kim, Koon-Ja Lee, Jung Un Jang The Korean Journal of Vision Science.2019; 21(4): 631. CrossRef
PURPOSE Self-assessment is important for learning. But, the reliability of self-assessment has long been questioned. We investigated this problem in an objective structured clinical examination (OSCE) setting. METHODS: Forty third-year nursing students who participated in the OSCE were given the same checklist that was used for assessment by the examiners after they finished the OSCE. Then, the scores that were calculated by the examiners and examinees were compared, measuring the average, correlation, difference in score between examinee and examiner, difference in score according to the examinee's level, and Z-score. RESULTS: Scores that were made by the examinees were significantly higher than those of the examiners (average 72.34 vs. 64.03), between which there was a loose positive correlation (r=0.34). The correlation coefficient increased (r=0.62) when a difference in score (examinee-examiner) was compared with the examiner score.
When examinees were divided into 3 groups according to their scores, the difference tended to become less as the scores of examinee rose. When the Z-score was compared with the level of the examinee, examinees who had higher scores evaluated themselves lower than the examiners, and examinees with lower scores assessed themselves higher than the examiners. CONCLUSION: According to comparison of averages or correlation analysis, self-assessment by the examinee appeared to be less reliable. But when data were analyzed using difference in score, examinee levels, and Z-score, a certain tendency developed. This pattern may be due to the heterogeneity of the examinee group. Selection of appropriate analysis methods might be important to estabilish the reliability of self-assessment skills.
Citations
Citations to this article as recorded by
A Study on Multiple Sources Evaluation for Practical Education in the Core Basic Nursing Skills in Nursing Students Hyo-Suk Song, So Hee Lim The Journal of Korean Academic Society of Nursing Education.2016; 22(1): 5. CrossRef
Comparison of Teacher's and Students' Evaluations on OSCE Performance among College Nursing Students Atsumi Iikura, Eunyoung Suh Perspectives in Nursing Science.2014; 11(2): 153. CrossRef
Unawareness of One's Own Task Incompetence: Replication of Kruger and Dunning (1999) in the Academic Domain 박상희 한국심리학회지: 사회및성격.2013; 27(3): 59. CrossRef
The Change of perception according to the fidelity of simulation in Objective Structured Clinical Examination for Procedural Skill of 4th Medical Students Hee-Jeong Son, Jin-Uk Kim, Yu-Ri Yi, Byeong-Moon Hwang Journal of the Korea Academia-Industrial cooperation Society.2012; 13(3): 1178. CrossRef
Analysis of First Clinical Skills Examination in the Korean Medical Licensing Examination: Focus on Examinees' Experience in a Medical School Kyung Ae Jun, Sang Yop Shin Korean Journal of Medical Education.2011; 23(3): 203. CrossRef
How to Succeed in Carrying Out the First Trial of the Clinical Skills Examination in the Korean Medical Licensing Examination Sun Huh Korean Journal of Medical Education.2009; 21(3): 211. CrossRef
PURPOSE To know the current status and to improve the internal medicine clinical clerkship in one medical school, the author performed a mid-term survey of the students.
METHODS Questions were given to 48 students at the midpoint of their internal medicine clinical clerkship. Questions were about understanding the of purpose of the rotation, ability to do physical examination and procedures, bedside teaching experience and conferences, and support during clinical clerkship. Questions used 5-point Likert scale. 48 of 48 students responded. Average points for each question were calculated, and similar free answers were categorized together. RESULTS: Students began their clinical clerkship with high expectations, but soon fell into a 'busy but know-nothing' state. The purpose of the clinical clerkship was not clearly outlined. Students learned physical examination skills and procedures not directly from instructors but by watching over-the-shoulder. Bedside teaching was, in fact, 'table side' teaching. Instructors did not provide immediate feedback at the bedside on how the student gathered information from patients, Students attended conferences targeted towards residents, and they could not understand the ensuing discussions. Students had no sense of belonging to the ward team. This result was reported during a medical grand round with simple short-term strategies, which could be performed easily, and with longer-term strategies requiring more resources. CONCLUSION: The author did a mid-term survey of a clinical clerkship and reported the result with suggested strategies for improvement. The result was distributed to the faculty at other departments as well. Although there was resistance to the suggestions from the internal medicine department, this report was used to initiate clinical clerkship improvement to the clerkship in other departments. The author discusses this result along with a literature review. This paper can be a good resource for medical schools wanting to evaluate and improve their clinical clerkship.
Citations
Citations to this article as recorded by
Assessment of Medical Students in Clinical Clerkships Sang Yeoup Lee, Sun Ju Im, So Jung Yune, Sunyong Baek, Jae Seok Woo Korean Medical Education Review.2013; 15(3): 120. CrossRef
The Present Conditions of Clinical Clerkship Management in Korea Sang Hyun Kim, Eun Bae Yang, Duck Sun Ahn, Woo Tack Jeon, Chuhl Joo Lyu Korean Journal of Medical Education.1970; 21(4): 373. CrossRef
PURPOSE The evaluation of a lecture by students is one of the good way to improve a lecture. In one medical school, we started one-year-integrated-lecture curriculum for sophomore. And we evaluated lectures using formal lecture evaluation sheet after completion of each integrated lectures. The results were fed back to lecturers. But lecturers were indifferent to the results. So we performed this study to evaluate the current lecture evaluation method. METHODS: Answering patterns of evaluation sheets were reviewed. 50 students entitled to integrated lectures were given structured questions. 44 of them recovered. And randomly selected 10 students were interviewed. RESULTS: Analysis of evaluation sheets showed that some students checked the items just for fun. More than half of the students answered that they did not check the questions seriously. Reasons why the evaluation of lectures were not going well were; students were not trained to evaluate something; evaluation sheet was not designed to evaluate individual lecturer; listed items were too many in number and inappropriate; some students were suspicious about the impact of evaluation of lectures. In addition to this, it was revealed that students thought the timing of evaluation of lecture was important, too. CONCLUSION: Indifference of lecturers to the results is one form of resistance to the evaluation of lectures by students. To make evaluation of lectures effective, numbers and contents of evaluation items must be selected carefully, students need to be trained, and adequate evaluation timing should be determined with frequent feed backs.
Citations
Citations to this article as recorded by
Instructor factors associated with medical students’ lecture evaluation: a longitudinal analysis Jeongbae Rhie, Yoo Mi Chae, Seok-gun Park, Jae-hyun Kim, Hong Ja Kim Korean Journal of Medical Education.2023; 35(2): 165. CrossRef
Course quality management based on monitoring by students at a medical school Sanghee Yeo Korean Journal of Medical Education.2018; 30(2): 141. CrossRef
Factors that influence student ratings of instruction Su Jin Chae, Yun Hoon Choung, Yoon Sok Chung Korean Journal of Medical Education.2015; 27(1): 19. CrossRef
Analysis on Teaching Evaluation of Clinical Clerkship at Korean Medical Schools Eun-Jung Im, Bong-Hyun Chang Korean Journal of Medical Education.2011; 23(2): 119. CrossRef
Perception of Medical Students on Real-Time Class Evaluation Using Mobile Phone Survey System A-Ra Cho, Soo-Jung Lee, Sang-Heon Yoon, Nam-Jin Yoo, Sun Kim, Joo-Hyun Park Korean Journal of Medical Education.2010; 22(1): 15. CrossRef
A Trend Study of Student' Consistent Responses to Course Evaluation Su-Jin Chae, Ki-Young Lim Korean Journal of Medical Education.2009; 21(3): 307. CrossRef
Experience and Consideration on Online Course Evaluation by Medical Students So Dug Lim, Jongmin Lee, Hyung Seok ParK, Jae-Ran Yu, Kyung Yung Lee, In Sook Sohn, Ran Lee Korean Journal of Medical Education.2008; 20(4): 367. CrossRef
The Korean medical establishment has traditionally shown little concern for formal training in medical ethics; only recently have courses in medical ethics been included in the curricula of Korea's medical schools. The importance and effectiveness of such courses is still a matter of some debate. This study attempted to measure the effects of a course in medical ethics on senior medical students. For this study, a discussion based course was designed, which made considerable use of video-taped ethical situations.
Students were requested to answer a series of ethics related questions both before and after taking the course. Prior to taking the course, students indicated that treatment refusal, abortion, sex pre-selection, and euthanasia are the most important ethical issues in contemporary medical practice in Korea. However, after taking the course, students modified this list slightly by replacing the issue of euthanasia with that of equal access to health care resources. No significant changes were observed in students' value systems. The students' response to the course was favorable.
Citations
Citations to this article as recorded by
Medical Ethics Education: Its Objectives and Curriculum Ilhak Lee Korean Medical Education Review.2014; 16(3): 147. CrossRef
Validation of a Korean version of the Moral Sensitivity Questionnaire Sung-Suk Han, Juhu Kim, Yong-Soon Kim, Sunghee Ahn Nursing Ethics.2010; 17(1): 99. CrossRef
Student Satisfaction and Self-Assessment after Small Group Discussion in a Medical Ethics Education Program Hee-Kyung Joh, Jwa-Seop Shin Korean Journal of Medical Education.2009; 21(3): 243. CrossRef
Assessing the Effectiveness of Medical Ethics Education Sung Soo Kim, Byung Kyu Park, Chulhun L. Chang, Hae Kyu Kim, Shin Young Kang, Seong Wan Baik Korean Journal of Medical Education.2008; 20(1): 73. CrossRef
First
Prev
