Authentic assessment is central to medical licensure examinations to ensure that examinees’ clinical competencies are assessed. The adoption of computer-based testing using multimedia such as images, sounds, and video clips allows the presentation of clinical findings in a more authentic and undigested format than when they are presented in text items [1]. Multimedia-based presentation of clinical findings in assessment items, henceforth multimedia items, enhances the authenticity of the assessment, which leads to more valid assessment of examinees’ clinical competencies [2]. Multimedia items have been implemented in some medical licensure exams such as the United States Medical Licensing Examination and studies have shown that assessment using multimedia items is feasible and can measure some constructs different from what text items can measure [2,3].

In South Korea, multimedia items are to be introduced in the national medical licensure exam by the year 2020. With computer-based testing not yet introduced in the medical licensing exam in South Korea, the use of tablets is under consideration. Using tablets for assessment is expected to be efficient because they are ubiquitous and thus require less financial resources and space [4]. Therefore, ubiquitous testing, in which assessment is delivered by ubiquitous computing technologies such as smartphones and tablets, is practical in nationwide tests such as healthcare personnel licensure exams.

The most salient change likely with the use of ubiquitous testing in assessment in medical education is the adoption of multimedia items, which is expected to influence how students learn because assessment drives and even enhances learning [5,6]. In particular, it is speculated that teaching and learning in clinical contexts will be more emphasized with the adoption of multimedia items, as examinees will need to be able to interpret clinical findings in an authentic format similar to that are encountered in clinical practice [7]. Yet, relatively little known about student experience with this new type of assessment that offers empirical evidence to support this assumption. Therefore, this study investigated students’ experience with ubiquitous testing using tablets and its impact on their perceptions of and engagement in learning. Our research questions were (1) Do the item characteristics differ across text items and multimedia items using ubiquitous testing? (2) What are students’ perceptions of the benefits and challenges of ubiquitous testing? (3) Do students’ perceptions of and engagement in learning change after they experience ubiquitous testing?

It is expected that the present study help enhance our understanding of the validity of ubiquitous testing. To adopt an assessment tool, we need to evaluate its validity. Yet, there is a paucity of evidence to support the validity of multimedia items when they are delivered in the form of ubiquitous testing. One of the standards in establishing the validity of an educational assessment is to collect evidence on its impact on examinees and on teaching and learning [8]. Therefore, this study aimed to gain more insight into the feasibility of ubiquitous testing by exploring its impact on medical education.

A cohort of all Year 3 students (n=48) in the 4-year medical program at Dongguk University Medical School (DUMS) in South Korea participated in this study. The curriculum at DUMS consists of mostly lecture-driven preclinical courses for the first 2 years, followed by 2 years of clinical clerkships. The participants were attending core clinical clerkships in one of the two academic medical centers affiliated with the university when this study was conducted. Both undergraduate-entry (n=18) and graduate-entry (n=30) students were admitted in the study cohort.

Participants were given 10 content-matched items in either multimedia or text versions in multiple-choice question format. These items were in the domain of clinical knowledge for general practitioners and were developed by one of the authors. The 10 items consisted of two items on clinical anatomy on laparoscopic operation (organ, artery), four items on physical examination findings (ear, knee, brain, and muscle), two items on real time sonography findings (heart, fetus), one item on chest auscultation finding (heart and lung), and one item on patient encounter (abdominal pain). Fig. 1 shows sample item on auscultation in text and multimedia formats.

The participants were randomly assigned to one of two groups according to their placement of clinical rotations at the time of the study. The control group (the text group) took the test in the conventional text format and the experimental group (the multimedia group) was given multimedia items. Nine video clips and one sound clip of chest auscultation were incorporated in the multimedia items. The multimedia items were delivered using 10.1 inch tablets with Android operating system offered by the school’s medical education committee (Fig. 2). The ubiquitous testing system was developed by a vendor, NSDevil (Seoul, Korea) and was equipped with features for examinees to see the elapse of time, navigate between items, mark on skipped items, and take notes. The multimedia group used headphones to listen to audio clips of auscultation findings and video clips of interviews or physical examinations, whereas for the text group these findings were described in texts using standard terminology.

The participants took the test in their placement of clinical rotations in September 2014. Students at DUMS had taken tests in the pencil-and-paper format, and thus had not encountered any multimedia items prior to participation in this study. This was a pilot test and students’ test scores did not count towards their course grades. Institutional Review Board (IRB) approval was not requested for the present study, because it fell under the general exemption from our IRB for educational outcomes data.

Item analyses (test score, item difficulty, item discrimination, and response time) were performed and Student t-test was used to compare characteristics between text items and multimedia ones. For baseline comparison, participants’ cumulative grade point averages through Years 1 and 2 were compared across the two groups using Student t-test. Additionally, distributions of participant backgrounds (i.e., gender, entry-level, and ages) were compared across the two groups using chi-square analysis. IBM-SPSS version 20 (IBM Corp., Armonk, USA) was used and the significance level was 0.05 for the statistical analysis.

Additionally, we conducted focus group interviews of the participants in the multimedia group to investigate their experiences of ubiquitous testing. Focus group interview is a qualitative research method for collecting data on the phenomenon being studied by analyzing conversations among study participants [9]. We used the qualitative research method as it is known to be useful in eliciting student perspective [10]. Although all 24 participants in the multimedia group were placed in the focus groups to ensure diversity of opinions elicited from the study, they were divided into three groups to give groups small enough to ensure input from all participants.

The interviews were conducted in a semi-structured format with 12 questions: three related to participant perceptions of item characteristics of multimedia items, five to students’ emotional responses to ubiquitous testing, which is known to be linked to student learning and performance (e.g., enjoyment, anxiety, and boredom) [11], and four to changes in participant perceptions of and engagement in learning after experiencing ubiquitous testing. The interview questions are presented in Appendix 1.

The interviews were conducted 3 weeks after the participants took the test to investigate the impact of their experience with ubiquitous testing on their learning behaviors. Each interview session took 40 to 45 minutes. Although one of the authors participated in the interview sessions, he did not play an active role during the interview sessions and let one of the participants in the group facilitate the discussions during the interview and the author observed the sessions to minimize the possibility of him making an impact on the discussions. All the interviews were audio recorded and were transcribed by the authors. The transcripts were analyzed and emerging themes were identified from the data using the thematic analysis method.

Our study found that the mean test score of the multimedia group was lower than that of the text group, although there were no significant differences in item difficulty or discrimination between the text and multimedia items. These findings are similar to those found from studies of computer-based testing [2,3]. Furthermore, we found that the item difficulty was generally higher in the text items, but it was higher in the multimedia format in three items, which pertained to the sonographic structure of heart, muscle examination, and patient encounter. These findings are consistent with those from previous studies that multimedia items are more difficult than text items when text descriptions using textbook terminologies are replaced by multimedia presentations, such as in auscultation findings, and that multimedia items are easier when multimedia presentations provide richer information on the conditions of the patient than in text descriptions, such as in a video clip portraying hip flexion [2,3].

The total response time was longer in the multimedia group, which is consistent with findings from previous studies [2,3]. We speculated that the students needed to devote more time and effort to interpreting information from multimedia presentations due to their unfamiliarity with such presentations. This experience may have motivated the students to reflect upon their learning behaviors and prompted them to think about how they should change their learning behaviors to adapt to this new type of assessment.

Students generally showed positive responses to ubiquitous testing. They quickly became comfortable with this new technology, some even found it interesting to take a test involving clinical videos, and some felt their clinical competence could be assessed better in this testing format. These findings are consistent with those of Roh et al. [4] who found that students showed positive reactions to their experience with ubiquitous testing. However, our study revealed student perceptions that traditional teaching and learning methods have not adequately prepared them for this type of assessment. This highlights the need for medical schools to support student learning by providing more multimedia resources.

Our study confirms our speculation that ubiquitous testing can positively affect student learning by reinforcing the importance of being able to understand and apply knowledge in clinical contexts, which drives students to engage more actively in learning in clinical settings. Possibly, the method of assessment rather than the delivery medium influenced the students’ perceptions and attitudes on learning. Whether it is computer-based testing or ubiquitous testing, multimedia presentations in test items make assessment more authentic, which stimulates student awareness of the importance of being able to understand and apply knowledge in clinical contexts. This would help move students up the ladder in their learning and performance from “knowing” to “knowing how” in the Miller’s pyramid [12].

Three study limitations can be discerned. First, this study was performed with a small number of students in one medical school who had had no experience with computer-based testing and little clinical exposure during preclinical years. The extent to which students are experienced in computer-based testing and exposed to clinical settings from early on in their medical studies may differ across medical schools. Therefore, readers should take such differences into consideration when generalizing our findings to their own contexts. Second, participants in our study generally felt comfortable with ubiquitous testing because they were already familiar with ubiquitous technologies. Hence, our findings may not be generalized to students who are new to mobile technology or in locations with limited access to Wifi or mobile network for displaying multimedia contents. Third, the item analyses were conducted with a small sample of students using a small number of items. We did not attempt to ensure the statistical power in our analysis of the quantitative data as it was a pilot study conducted for the subsequent qualitative study of the students’ experiences with ubiquitous testing. Future research is warranted through larger scale studies to ensure statistical power and enhance the generalizability of our findings.

Although our study indicates students’ positive responses to ubiquitous testing and its potential to make a positive impact on their perceptions of learning, our study did not investigate whether such an impact actually leads to better clinical performance, which it was beyond the scope of our study. Therefore, future research is recommended that investigates whether the changes in student perceptions of and engagement in learning driven by the adoption of ubiquitous testing leads to better learning outcomes.