Abstract
-
Purpose:
Ubiquitous testing has the potential to affect medical education by enhancing the authenticity of the assessment using multimedia items. This study explored medical students’ experience with ubiquitous testing and its impact on student learning.
-
Methods:
A cohort (n=48) of third-year students at a medical school in South Korea participated in this study. The students were divided into two groups and were given different versions of 10 content-matched items: one in text version (the text group) and the other in multimedia version (the multimedia group). Multimedia items were delivered using tablets. Item response analyses were performed to compare item characteristics between the two versions. Additionally, focus group interviews were held to investigate the students’ experiences of ubiquitous testing.
-
Results:
The mean test score was significantly higher in the text group. Item difficulty and discrimination did not differ between text and multimedia items. The participants generally showed positive responses on ubiquitous testing. Still, they felt that the lectures that they had taken in preclinical years did not prepare them enough for this type of assessment and clinical encounters during clerkships were more helpful. To be better prepared, the participants felt that they needed to engage more actively in learning in clinical clerkships and have more access to multimedia learning resources.
-
Conclusion:
Ubiquitous testing can positively affect student learning by reinforcing the importance of being able to understand and apply knowledge in clinical contexts, which drives students to engage more actively in learning in clinical settings.
-
Key Words: Educational measurement, Multimedia, Ubiquitous testing, Assessment
Introduction
Authentic assessment is central to medical licensure examinations to ensure that examinees’ clinical competencies are assessed. The adoption of computer-based testing using multimedia such as images, sounds, and video clips allows the presentation of clinical findings in a more authentic and undigested format than when they are presented in text items [
1]. Multimedia-based presentation of clinical findings in assessment items, henceforth multimedia items, enhances the authenticity of the assessment, which leads to more valid assessment of examinees’ clinical competencies [
2]. Multimedia items have been implemented in some medical licensure exams such as the United States Medical Licensing Examination and studies have shown that assessment using multimedia items is feasible and can measure some constructs different from what text items can measure [
2,
3].
In South Korea, multimedia items are to be introduced in the national medical licensure exam by the year 2020. With computer-based testing not yet introduced in the medical licensing exam in South Korea, the use of tablets is under consideration. Using tablets for assessment is expected to be efficient because they are ubiquitous and thus require less financial resources and space [
4]. Therefore, ubiquitous testing, in which assessment is delivered by ubiquitous computing technologies such as smartphones and tablets, is practical in nationwide tests such as healthcare personnel licensure exams.
The most salient change likely with the use of ubiquitous testing in assessment in medical education is the adoption of multimedia items, which is expected to influence how students learn because assessment drives and even enhances learning [
5,
6]. In particular, it is speculated that teaching and learning in clinical contexts will be more emphasized with the adoption of multimedia items, as examinees will need to be able to interpret clinical findings in an authentic format similar to that are encountered in clinical practice [
7]. Yet, relatively little known about student experience with this new type of assessment that offers empirical evidence to support this assumption. Therefore, this study investigated students’ experience with ubiquitous testing using tablets and its impact on their perceptions of and engagement in learning. Our research questions were (1) Do the item characteristics differ across text items and multimedia items using ubiquitous testing? (2) What are students’ perceptions of the benefits and challenges of ubiquitous testing? (3) Do students’ perceptions of and engagement in learning change after they experience ubiquitous testing?
It is expected that the present study help enhance our understanding of the validity of ubiquitous testing. To adopt an assessment tool, we need to evaluate its validity. Yet, there is a paucity of evidence to support the validity of multimedia items when they are delivered in the form of ubiquitous testing. One of the standards in establishing the validity of an educational assessment is to collect evidence on its impact on examinees and on teaching and learning [
8]. Therefore, this study aimed to gain more insight into the feasibility of ubiquitous testing by exploring its impact on medical education.
Subjects and methods
A cohort of all Year 3 students (n=48) in the 4-year medical program at Dongguk University Medical School (DUMS) in South Korea participated in this study. The curriculum at DUMS consists of mostly lecture-driven preclinical courses for the first 2 years, followed by 2 years of clinical clerkships. The participants were attending core clinical clerkships in one of the two academic medical centers affiliated with the university when this study was conducted. Both undergraduate-entry (n=18) and graduate-entry (n=30) students were admitted in the study cohort.
Participants were given 10 content-matched items in either multimedia or text versions in multiple-choice question format. These items were in the domain of clinical knowledge for general practitioners and were developed by one of the authors. The 10 items consisted of two items on clinical anatomy on laparoscopic operation (organ, artery), four items on physical examination findings (ear, knee, brain, and muscle), two items on real time sonography findings (heart, fetus), one item on chest auscultation finding (heart and lung), and one item on patient encounter (abdominal pain).
Fig. 1 shows sample item on auscultation in text and multimedia formats.
The participants were randomly assigned to one of two groups according to their placement of clinical rotations at the time of the study. The control group (the text group) took the test in the conventional text format and the experimental group (the multimedia group) was given multimedia items. Nine video clips and one sound clip of chest auscultation were incorporated in the multimedia items. The multimedia items were delivered using 10.1 inch tablets with Android operating system offered by the school’s medical education committee (
Fig. 2). The ubiquitous testing system was developed by a vendor, NSDevil (Seoul, Korea) and was equipped with features for examinees to see the elapse of time, navigate between items, mark on skipped items, and take notes. The multimedia group used headphones to listen to audio clips of auscultation findings and video clips of interviews or physical examinations, whereas for the text group these findings were described in texts using standard terminology.
The participants took the test in their placement of clinical rotations in September 2014. Students at DUMS had taken tests in the pencil-and-paper format, and thus had not encountered any multimedia items prior to participation in this study. This was a pilot test and students’ test scores did not count towards their course grades. Institutional Review Board (IRB) approval was not requested for the present study, because it fell under the general exemption from our IRB for educational outcomes data.
Item analyses (test score, item difficulty, item discrimination, and response time) were performed and Student t-test was used to compare characteristics between text items and multimedia ones. For baseline comparison, participants’ cumulative grade point averages through Years 1 and 2 were compared across the two groups using Student t-test. Additionally, distributions of participant backgrounds (i.e., gender, entry-level, and ages) were compared across the two groups using chi-square analysis. IBM-SPSS version 20 (IBM Corp., Armonk, USA) was used and the significance level was 0.05 for the statistical analysis.
Additionally, we conducted focus group interviews of the participants in the multimedia group to investigate their experiences of ubiquitous testing. Focus group interview is a qualitative research method for collecting data on the phenomenon being studied by analyzing conversations among study participants [
9]. We used the qualitative research method as it is known to be useful in eliciting student perspective [
10]. Although all 24 participants in the multimedia group were placed in the focus groups to ensure diversity of opinions elicited from the study, they were divided into three groups to give groups small enough to ensure input from all participants.
The interviews were conducted in a semi-structured format with 12 questions: three related to participant perceptions of item characteristics of multimedia items, five to students’ emotional responses to ubiquitous testing, which is known to be linked to student learning and performance (e.g., enjoyment, anxiety, and boredom) [
11], and four to changes in participant perceptions of and engagement in learning after experiencing ubiquitous testing. The interview questions are presented in
Appendix 1.
The interviews were conducted 3 weeks after the participants took the test to investigate the impact of their experience with ubiquitous testing on their learning behaviors. Each interview session took 40 to 45 minutes. Although one of the authors participated in the interview sessions, he did not play an active role during the interview sessions and let one of the participants in the group facilitate the discussions during the interview and the author observed the sessions to minimize the possibility of him making an impact on the discussions. All the interviews were audio recorded and were transcribed by the authors. The transcripts were analyzed and emerging themes were identified from the data using the thematic analysis method.
Results
1. Study participants
Table 1 shows the study participants’ demographics and the results of comparisons of baseline performance across the two groups. There was no statistically significant difference in participants’ academic performance or distributions of participant demographics between the two groups.
2. Item analyses
Table 2 compares item responses across the two groups. The mean test score of the multimedia group was significantly lower than that of the text group. The mean item difficulty of the text items was higher than that of multimedia items and the mean item discrimination was comparable between the two groups. Still, there were no statistically significant differences in item difficulty and discrimination. The mean total response time for the 10 items was longer in the multimedia group than in the text group (14.98 minutes vs. 5.79 minutes).
3. Benefits and challenges of ubiquitous testing using tablets
Participants generally showed some positive responses to ubiquitous testing. Most mentioned they were a little anxious about this new type of assessment in the beginning, but they quickly became comfortable because they were used to mobiles devices. The participants felt using tablets was more convenient than pencil-andpaper tests as the display quality is much better in tablets than in papers and they did not have to mark answers on the optical mark recognition sheet, which sometimes they could make errors. Additionally, some participants pointed out the feature in the tablet system that showed the time elapsed was helpful.
Moreover, participants pointed out they would likely make less mistakes in ubiquitous testing than in traditional tests. One student stated, “I have more chances to make mistakes in taking a test when a lot of items are presented. In this type of (ubiquitous) testing, only one item is shown on each page, so that helps me concentrate on the question one by one, and that helps me make fewer mistakes.” Other students also agreed with the statement that they could pay better attention to each item on the test in ubiquitous testing and felt that would make them less likely to make mistakes. Additionally, some even found it interesting to view video clips in the items because they felt like they were watching what was really happing in the clinical settings. Yet, some participants mentioned that they felt more time pressure in ubiquitous testing because they needed time to view video clips in the items.
Most of the participants pointed out that ubiquitous testing will be beneficial in enhancing their clinical competencies. One student stated, “Physical exams is fundamental in clinical competencies, and we can’t learn physical exams merely by seeing some images. So, if we could learn how physical exams are performed by watching some clinical videos and if we were tested that way, we could learn better about physical exams and that could enhance our clinical competencies. That way we could be assessed on our clinical competencies more effectively.”
4. Impact of ubiquitous testing on student perceptions of and engagement in learning
The participants reflected on how the teaching and learning in preclinical years prepared them for ubiquitous testing. Most of the participants felt that what they had learned from lectures in the preclinical years was insufficient for them to perform well in ubiquitous testing. One student noted “I lost confidence (while taking the ubiquitous testing) feeling that I knew only superficially. There was a video clip of a physical exam in one of the items. I knew what that exam was, but I did not know its procedures well enough, so that was a difficult question for me.” Another student added, “Most of the images presented in the tests that I had taken (before the ubiquitous testing) were familiar ones because they were similar to those in textbooks. But, those in multimedia items were not something that I had seen in textbooks or somethings that I saw during clinical rotations, so these were difficult to me.”
Most of the participants mentioned that what they learned from clinical encounters during their clerkships helped them more than lectures to prepare for the ubiquitous testing. One student mentioned “I think lectures that I took in Year 2 courses did not help much. What I learned during clerkships when the preceptors had us do auscultations, explained us about clinical images to interpret them, and by observing surgery in the operation room was more helpful.”
The participants also highlighted the need for more multimedia learning resources to help them prepare for ubiquitous testing. One stated that “We viewed some clinical videos during lectures when we were Year 2. I did not pay a lot of attention to those videos then and skimmed them over, because they would not be in the exams. But, now with ubiquitous testing, such video clips will be very helpful in preparing for this kind of test.” Yet, most of the participants pointed out the lack of available multimedia learning resources. As one student mentioned, “I could do well on the test about auscultations and anatomical structures that I saw from actual cases that I encountered during clinical clerkships, but I don’t think I can do well on others. I cannot come across every case that I need to know during clinical clerkships, so it would be more helpful if there are more multimedia learning resources available for us.”
The participants reflected on their engagement in clinical rotations during the past 3 weeks after they experienced ubiquitous testing. Some participants mentioned that such an experience did not influence their learning behaviors because they knew multimedia items would be introduced after they graduate from medical school. Still, some participants mentioned that their experience with ubiquitous testing made them think about their learning behaviors in clinical clerkships. One participant noted that “This (ubiquitous testing) would have been a much more difficult test if I had taken it in Year 1 or 2, but I could answer some of the questions from what I learned during clinical rotations. So, I was thinking that I will need to change my attitude toward learning during clinical clerkships if I take this kind of test.”
In addition, some participants reported that they engaged in learning in clinical clerkships more actively after they experienced ubiquitous testing. One student mentioned that “I attended clinical rotations in otorhinolaryngology last week and there were a lot of physical exams done there. Those were something that could be in multimedia items, so I tried to learn more about the principles behind those exams. So, I asked the preceptors questions more often about why the physical exam is performed and how it is done and I tried to observe the anatomical structures more closely when I was attending surgery in the operation room. Now, I try to learn things more precisely and do not want to skip things that are not clear to me from what I see during clinical rotations.”
Discussion
Our study found that the mean test score of the multimedia group was lower than that of the text group, although there were no significant differences in item difficulty or discrimination between the text and multimedia items. These findings are similar to those found from studies of computer-based testing [
2,
3]. Furthermore, we found that the item difficulty was generally higher in the text items, but it was higher in the multimedia format in three items, which pertained to the sonographic structure of heart, muscle examination, and patient encounter. These findings are consistent with those from previous studies that multimedia items are more difficult than text items when text descriptions using textbook terminologies are replaced by multimedia presentations, such as in auscultation findings, and that multimedia items are easier when multimedia presentations provide richer information on the conditions of the patient than in text descriptions, such as in a video clip portraying hip flexion [
2,
3].
The total response time was longer in the multimedia group, which is consistent with findings from previous studies [
2,
3]. We speculated that the students needed to devote more time and effort to interpreting information from multimedia presentations due to their unfamiliarity with such presentations. This experience may have motivated the students to reflect upon their learning behaviors and prompted them to think about how they should change their learning behaviors to adapt to this new type of assessment.
Students generally showed positive responses to ubiquitous testing. They quickly became comfortable with this new technology, some even found it interesting to take a test involving clinical videos, and some felt their clinical competence could be assessed better in this testing format. These findings are consistent with those of Roh et al. [
4] who found that students showed positive reactions to their experience with ubiquitous testing. However, our study revealed student perceptions that traditional teaching and learning methods have not adequately prepared them for this type of assessment. This highlights the need for medical schools to support student learning by providing more multimedia resources.
Our study confirms our speculation that ubiquitous testing can positively affect student learning by reinforcing the importance of being able to understand and apply knowledge in clinical contexts, which drives students to engage more actively in learning in clinical settings. Possibly, the method of assessment rather than the delivery medium influenced the students’ perceptions and attitudes on learning. Whether it is computer-based testing or ubiquitous testing, multimedia presentations in test items make assessment more authentic, which stimulates student awareness of the importance of being able to understand and apply knowledge in clinical contexts. This would help move students up the ladder in their learning and performance from “knowing” to “knowing how” in the Miller’s pyramid [
12].
Three study limitations can be discerned. First, this study was performed with a small number of students in one medical school who had had no experience with computer-based testing and little clinical exposure during preclinical years. The extent to which students are experienced in computer-based testing and exposed to clinical settings from early on in their medical studies may differ across medical schools. Therefore, readers should take such differences into consideration when generalizing our findings to their own contexts. Second, participants in our study generally felt comfortable with ubiquitous testing because they were already familiar with ubiquitous technologies. Hence, our findings may not be generalized to students who are new to mobile technology or in locations with limited access to Wifi or mobile network for displaying multimedia contents. Third, the item analyses were conducted with a small sample of students using a small number of items. We did not attempt to ensure the statistical power in our analysis of the quantitative data as it was a pilot study conducted for the subsequent qualitative study of the students’ experiences with ubiquitous testing. Future research is warranted through larger scale studies to ensure statistical power and enhance the generalizability of our findings.
Although our study indicates students’ positive responses to ubiquitous testing and its potential to make a positive impact on their perceptions of learning, our study did not investigate whether such an impact actually leads to better clinical performance, which it was beyond the scope of our study. Therefore, future research is recommended that investigates whether the changes in student perceptions of and engagement in learning driven by the adoption of ubiquitous testing leads to better learning outcomes.
ACKNOWLEDGMENTS
None.
ACKNOWLEDGMENTS
None.
Funding
None.
Conflicts of interest
None.
Fig. 1. A Sample Item on Auscultation in Text (Top) and Multimedia (Bottom) Formats
Fig. 2. A Picture of a Student Taking the Ubiquitous Testing Using Tablet
Table 1.Cross-Tabulation of Participant Demographics and Academic Performance
Table 1.
|
Variable |
Text group (n=24) |
Multimedia group (n=24) |
p-value |
|
Entry-level |
|
|
|
|
Undergraduate |
9 |
9 |
1.00 |
|
Graduate |
15 |
15 |
|
|
Sex |
|
|
|
|
Male |
13 |
14 |
0.77 |
|
Female |
11 |
10 |
|
|
Age |
26.54±3.24 |
26.21±3.11 |
0.72 |
|
GPA |
3.18±0.59 |
3.10±0.48 |
0.61 |
Table 2.Mean Scores, Item Difficulty, and Item Discrimination for Text and Multimedia Items
Table 2.
|
Item |
Text group (n=24) |
Multimedia group (n=24) |
p-value |
|
Test score (10 items) |
6.25±1.54 |
5.08±1.56 |
0.012 |
|
Item difficulty |
0.63±0.20 |
0.51±0.25 |
0.261 |
|
Item discrimination |
0.33±0.18 |
0.33±0.25 |
0.984 |
References
- 1. Peterson MW, Gordon J, Elliott S, Kreiter C. Computer-based testing: initial report of extensive use in a medical school curriculum. Teach Learn Med. 2004;16:51-59.
- 2. Shen L, Li F, Wattleworth R, Filipetto F. The promise and challenge of including multimedia items in medical licensure examinations: some insights from an empirical trial. Acad Med. 2010;85(10 Suppl):S56-S59.
- 3. Holtzman KZ, Swanson DB, Ouyang W, Hussie K, Allbee K. Use of multimedia on the step 1 and step 2 clinical knowledge components of USMLE: a controlled trial of the impact on item characteristics. Acad Med. 2009.
- 4. Roh H, Lee JT, Rhee BD. Ubiquitous-based testing in medical education. Med Teach. 2015;37:302-303.
- 5. Wood T. Assessment not only drives learning, it may also help learning. Med Educ. 2009;43:5-6.
- 6. Larsen DP, Butler AC, Roediger HL 3rd. Test-enhanced learning in medical education. Med Educ. 2008;42:959-966.
- 7. Huh S. Can computerized tests be introduced to the Korean medical licensing examination? J Korean Med Assoc. 2012;55:124-130.
- 8. Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37:830-837.
- 9. Stalmeijer RE, Mcnaughton N, Van Mook WN. Using focus groups in medical education research: AMEE Guide No. 91. Med Teach. 2014;36:923-939.
- 10. Barbour RS. Making sense of focus groups. Med Educ. 2005;39:742-750.
- 11. Pekrun R. The control-value theory of achievement emotions: assumptions, corollaries, and implications for educational research and practice. Educ Psychol Rev. 2006;18:315-341.
- 12. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63-S67.
Appendices
- Appendix 1.
Focus Group Interview Questions (Only with Multimedia Group Participants)
1. Do you think multimedia items were easier or more difficult than text items?
2. Which do you think can better assess your clinical competence—text or multimedia items?
3. What type of multimedia items were the most difficult for you?
4. In which format do you think you can do better in the test—text or multimedia items?
5. Did you become more or less confident about the test after you took ubiquitous testing?
6. Do you think you can make more mistakes when you take ubiquitous testing than in traditional tests?
7. Did you get more nervous when you took the ubiquitous testing?
8. Did you feel more interested when you took the ubiquitous testing?
9. How did you feel about taking the test using tablets? What were the advantages and challenges?
10. Do you feel the courses that you had taken in the medical school prepared you well for ubiquitous testing? If not, what were those
courses, and what courses helped you the most?
11. Do you think you would prepare for ubiquitous test the same way as you did with traditional tests? What would you do differently?
12. Do you think the experience of ubiquitous testing has affected your learning behavior in clinical clerkships?
