Evaluation of usefulness of smart device-based testing: a survey study of Korean medical students
Article information
Abstract
Purpose
This study aims to understand the characteristics of smart device-based testing (SBT) by comparing the typical characteristics of students’ satisfaction with SBT, its usefulness, advantages, and disadvantages when compared with existing testing methods.
Methods
A total of 250 students from the first to third year were selected as the final targets of the study and the questionnaire was developed by faculty members who participated in the survey from the start of the SBT. The total number of questions is 12, and the questionnaire used a 4-point scale. The data obtained were analyzed using the IBM SPSS ver. 23.0 (IBM Corp., Armonk, USA).
Results
Answers to the “satisfaction with SBT” were generally negative, while answers to the “usefulness of SBT” were generally positive. There was no difference in satisfaction across gender and smart device ownership, whereas there were significant differences across years. With reference to the usefulness of SBT, students responded positively, while about the overall configuration and completeness of SBT, students responded negatively. Students also seemed to show a greater preference toward the pencil-paper test.
Conclusion
On the other hand, students generally thought that SBT helped to assess medical knowledge better and was a more objective method of knowledge assessment than a pencil-paper test. We believe that students preferred the traditional paper-pencil test due to their unfamiliarity with SBT. We believe that an appropriate and careful remedy for drawbacks of the SBT will have a significant impact in the accumulation of actual clinical knowledge and in the improvement of practical skills for medical students.
Introduction
The development of the computer along with the advances in information and communication technologies has brought about changes in many areas related to education. It is also changing the field of medical education and computer-based tests (CBT), and many institutions are already replacing the paper-pencil test as a method of assessment. With the recent development of smart devices and their high penetration, smart device-based testing (SBT) is gaining prominence as a useful and accurate evaluation method in various areas of education [1,2]. Compared to the paper-pencil test, SBT has many more advantages such as cost reduction and improvement of learning efficiency [3,4], as well as space utilization and mobility, which are cited as shortcomings in CBT. Hence, expectations from SBTs are growing. This transition is expected to become a necessity in assessment tools because the audio-visual materials that are in use in traditional medical education today are more effective through platforms such as e-learning; these platforms utilize the strengths of technology and networks in conjunction with classroom training [5-8].
In the field of medical education, SBTs are expected to have the following advantages: First, printing and administrative costs spent in preparation of the test can be saved. Second, it can save manpower, time, and expenses required to assess, compute results, and facilitate analysis and feedback [9,10]. Third, it can apply various types of questions through multimedia data. In practice, medical judgment on a patient’s treatment or progress is very important when using dynamic imaging information such as computed tomography and ultrasonography. If medical education and evaluation are carried out using such multimedia information, it is thought to have a good effect on actual clinical knowledge accumulation and improvement of practical functions.
In order for SBT to be recognized as a useful and efficient assessment tool in the field of medical education and to establish its status, various kinds of studies are required, and the actual characteristics of SBT must be identified. Studies on SBT as an assessment tool are gradually increasing and domestic and international studies have been published on smart device ownership, proficiency, preference, and so forth, or its relation to existing pencil-paper tests [10-13]. However, previous studies in Korea limited either the length of use or the number of people involved. And almost studies were conducted on a single year.
Therefore, we planned this study, based on 3 years of SBT operational experience, for medical students from the first to third year at medical schools or medical graduate schools to understand the characteristics of SBT as a means of assessment, by comparing the students’ satisfaction with SBT, its usefulness, advantages, and disadvantages with existing testing methods.
Methods
1. Overview of SBT
Medical undergraduates and graduates of Kyung Hee University have been taking examinations via SBT since February 2016. These examinations range from basic courses such as anatomy and pathology to clinical courses such as pulmonology all across medical education curriculum. The previous test questions of paper-pencil included images of one or two cuts, on the other hand, test questions of SBT include pictures, videos, and various multimedia images such as partial or full images of computed tomography and ultrasonography. Test questions are answered in either a multiple-choice or descriptive manner. After professors in the medical education committee provide the questions, the SBT staff electronically transform these questions for smart devices and conduct a simulation test to ensure the questions are presented properly. The smart devices used are 25.654-cm (10.1-inch) Android-based pads developed by NSdevil (Daejeon, Korea). On the day of an exam, 130 pre-charged smart devices are placed in an examination room installed with four wireless hubs for smooth transmission. Once a student enters the authentication number provided on the exam day along with their class number and name, the student can begin the examination.
2. Objectives and composition of the questionnaire
The survey was conducted in December 2019 by selecting 330 medical students from the first to third year who had already been taking examinations via both paper-pencil test and the SBT. The aim of the study was fully explained prior to the survey. All surveys were conducted after taking SBT. Of the students who agreed to the survey, 250 students who responded sincerely were selected as the final target. The questionnaire was developed by faculty members who had participated in the survey from the beginning of the SBT, and a few of the questionnaires from that time were used to compare the results of an existing study by SBT [11]. The questions were as follows: (1) general characteristics, such as gender, age, smart device ownership, SBT experience, and so forth; (2) satisfaction with SBT with respect to convenience, configuration, completeness, and preference over existing test methods; (3) usefulness of SBT in terms of objectivity and fairness; and (4) advantages and disadvantages of SBT and comparison with conventional testing methods. The total number of questions is 12, except general characteristics. The questions that dealt with the advantages and disadvantages of SBT had multiple-choice responses. The rest of the questions were answered in the 4-point scale (“strongly agree,” “agree,” “disagree,” and “strongly disagree”) manner. Other opinion can be made through open response question. For the reliability measurement of the questionnaire, the Cronbach α value was analyzed as 0.885.
3. Statistical analysis
The data investigated were analyzed using the IBM SPSS ver. 23.0 (IBM Corp., Armonk, USA). The validity of the survey was given a Cronbach α reliability coefficient. The factor analysis that affected the survey results was used to verify statistical significance using the Pearson chi-square test, with a significance level less than the p-value (0.05). This study was previously approved by the Institutional Review Board for Human Research at Kyung Hee University. The informed consent was obtained from all participants included in this study. (IRB approval no., KHSIRB-19-330[NA]).
4. General characteristics of the subject
The response rate of survey was 98%. The average age of the subjects was 24.2±2.7 years, comprising of 155 men (61.5%) and 95 women (37.7%). A total of 164 students (65.1%) owned smart devices. With respect to the SBT experience, “more than 6 times” made up 75%, while “2 to 5 times” comprised 23.4% of the participants. One student had never experienced SBT before and one who had experienced SBT only once (Table 1).
Profile of the Students Who Responded to the Survey
Results
1. Satisfaction with SBT
With respect to the convenience of a smart device, 12.3% answered, “strongly agree” and 41.3% responded with “agree.” When asked whether they were satisfied with the overall composition and completeness of the smart device test, 38.9% answered “disagree” and 17.9% reported, “strongly disagree.” When asked whether they preferred SBT over traditional paper testing methods, 38.9% answered “disagree” and 25.4% responded, “strongly disagree.” When asked about applying the SBT to other subjects, 42.5% answered “disagree” and 15.9% reported, “strongly disagree.” When asked about taking the same test using computers instead of smart devices, 40.5% answered “disagree” and 22.6% responded, “strongly disagree.” Others suggested that SBT could be used to refer to images and photographic materials on smart devices (Table 2). The score for SBT was 4.8±2.4 on a scale of 10.
Responses to Survey about the Satisfaction and Usefulness of Students Who Took SBT
2. The usefulness of SBT
When asked whether SBT is more effective in assessing medical knowledge in comparison to pencil-paper tests, 13.9% answered, “strongly agree” and 46.8% responded “agree.” Similarly, when asked whether SBT can be evaluated more objectively, 13.9% answered, “strongly agree,” and 40.1% responded, “agree.” When asked whether SBT was a fairer testing method, 11.1% answered, “strongly agree,” 40.1% “agree,” 34.5% “disagree,” and 14.3% “strongly disagree” (Table 2).
3. Advantages, disadvantages and other features of SBT
The following were the advantages of SBT, which is indicated by the following data: 168 (69.7%) answered that “it is easy to recognize letters or picture” and 131 (54.4%) answered “possibility of multi-media implementation, such as video and sound.” A total of 86 (35.7%) answered “test process is more convenient than a paper test,” 41 (17.0%) answered “no possibility of cheating,” 35 (14.5%) answered “reduced test time,” and 19 (7.9%) answered “increased test concentration.” Another answer included “There was no mistake in marking the OMR.”
The following disadvantages were indicated by the data: A total of 161 (64.7%) answered “unable to write on exam paper,” 144 (57.8%) answered “system error during test,” 144 (57.8%) answered “difficult to enter letters and numbers,” 105 (42.2%) answered “dissatisfaction with the order of the test questions,” 65 (26.1%) answered “slow pad drive,” 25 (10.0%) answered “lack of pad mounting functions such as bookmarks,” 14 (5.6%) answered “smaller screen than a paper test,” 11 (4.4%) answered “difficult to recognize letters or pictures,” and 3 (1.2%) answered “possibly of cheating such as hacking.” Other answers included “it’s hard to get back to the test questions” and “can’t see the whole problem at one glance.”
Other opinions on SBT were “it is recommended to provide a Bluetooth keyboard rather than a smart pen for a descriptive problem” and “pads don’t perform well,” and pointed out poor operations in the preparation of exams.
4. Analysis according to gender, year, and smart device ownership
Satisfaction and usefulness of the SBT by gender did not convey significant difference statistically. However, convenience, satisfaction, and preference of SBT by different years displayed a significant difference as first and second year students mainly gave negative responses, while third year students mainly gave positive responses. Third year students answered with more positive responses, especially with regard to the usefulness of SBT (Table 3). In the analysis by smart device ownership, with respect to the overall satisfaction with SBT, smart device owners mainly responded negatively, while non-owners mainly responded positively, which displayed a significant difference according to smart device ownership. There were no significant statistical differences in other areas according to smart device ownership (Table 4).
Differences in Responses to Survey about Recognition of SBT by Year
Differences in Responses to Survey about Recognition of SBT according to Smart Device Possession
Discussion
Although SBT is claimed to be a useful objective assessment tool, this alone does not warrant a complete replacement of the pencil-paper test. SBT will only be able to settle into a complete medical education assessment tool when we understand exactly how SBT is accepted by actual examinees, identify various features in the stages of exam preparation and completion, and remedy any related drawbacks. To that extent, this study was conducted in order to understand the current state of SBT in medical education and help overcome any shortcomings.
As mentioned above, for questions related to the convenience of smart devices, students generally gave a positive response, while for questions related to the overall configuration and completeness of SBT, students mainly gave a negative response. Students also seemed to prefer the traditional pencil-paper test. On the other hand, students generally thought SBT helped assess medical knowledge better and that it is a more objective method of knowledge evaluation than a pencil-paper test. This led us to believe that students just preferred the traditional paper-pencil test due to their unfamiliarity with SBT in the context of medical school examinations.
With respect to potential SBT application in national medical license examination, the concept of adding smart devices in pencil-paper tests for providing media and photographic materials has been suggested several times. This use of smart devices is believed to be the greatest benefit of SBT in that it incorporates dynamic and visual forms of additional information with the existing written information. This was also consistent with the fact that students generally reported a convenience in the recognition of letters and pictures and the incorporation of multimedia images as one of the benefits of using SBT.
The perception of SBT according to gender proved to be of no significant difference. Lim et al. [13] proved that SBT was preferred over the pencil-paper test and CBT, and there were no significant differences in the analysis of satisfaction and preference according to gender, age, and so forth. In addition, the study by Lazarus et al. [14] proved that there was no difference in the perception of the tablet technology in the field of medical education with respect to racial or socioeconomic differences. Therefore, we concluded that SBT could be a fair assessment method.
Meanwhile, the study by Lim et al. [13] proved that SBT was the most preferred, but in our study, the students preferred the pencil-paper test the most, followed by SBT and CBT. Our results, as mentioned earlier, may be due to preference for traditional methods or poor operations in the preparation of exams.
There was a difference in the perception of SBT in the context of convenience, satisfaction, and preference of SBT, as third-year students gave positive reviews on the convenience, while first and second year students gave negative responses. Such difference was believed to exist because third-year students generally learn via practical-based education in a clinical environment using various video materials, while first and second-year students learn by textbook-based education in a classroom setting. In addition, it was thought that third-year students gave relatively positive responses due to the accumulated experiences with SBT.
The differences in overall satisfaction of SBT according to smart device ownership was also prevalent, as students who owned smart devices generally displayed negative responses, while students who did not own smart devices mainly displayed positive responses. We can explain this by the fact that the use and performance of personal smart devices differ from those used solely for educational purposes. Therefore, students who own smart devices find low-performance school smart devices rather outdated.
Meanwhile, students found writing, and entering letters and numbers on smart devices challenging during exams, and claimed it was difficult to view the test at one glance like during a paper test. This was probably suggested as a disadvantage in the context of the peculiarity and stressful intensity of medical school exams. Other potential problems are related to pad mounting functions, such as system errors and bookmarking, and include difficulty entering in long descripting answers, which are all technical or operational shortcomings that need to be addressed.
In summing up the analysis, it is believed that the biggest advantage of SBT seems to be its practical use in providing various real-life media materials in examinations. If the unfamiliarity of this new method is carefully relieved and shortcomings effectively supplemented, SBT can have a significant impact on the accumulation of clinical knowledge and in enhancing practical skills.
In the previous study, it was found that there was no significant relationship between SBT exam scores and ownership of smart devices, and that SBT exam scores were neither affected by gender nor smart device proficiency. In addition, the association between accumulative rank in the pencil-paper test and SBT scores was found to be significant [11], and a study by Kim and Hwang [12], proved that the difficulty of pencil-paper test and the multimedia test using tablets was not significantly different [12]. Therefore, it is believed that SBT will not have any special effect on the results even if it replaces the existing pencil-paper test.
The limitation of this study is that a small number of assessments and sample size does not allow it to be a generalized study. More accurate results can be obtained if data from various schools, different years, and diverse subjects are collected in the near future. Moreover, it seemed that the students under study tend to focus more on the functional problems of the device or on the regulation of the study itself, rather than on the perception of the SBT method.
Smart devices for learning and/or testing are widely being utilized in the field of medical education in many foreign countries [15]. Consistent with the study by Baumgart et al. [16] that demonstrates improved exam performance using tablet learning, the development of such smart devices will improve the efficiency of medical education. Accordingly, this study analyzed the general perception of SBT with respect to its pros and cons. Hence, we believe that appropriate and careful remedy for drawbacks of the SBT, without a doubt, will have a significant impact in the accumulation of actual clinical knowledge and improvement of practical skills for medical students.
Acknowledgements
None.
Notes
Funding
No funding was obtained for this study.
Conflicts of interest
No potential conflict of interest relevant to this article was reported.
Author contributions
Conception or design of the work: OYK; drafting the article: YCL; data collection, data analysis and interpretation: YCL, OYK, HJH, SHK; and critical revision of the article and final approval of the version to be published: OYK.