Abstract
-
Purpose
We aimed to explore medical students’ online learning patterns and needs by analyzing data obtained from an e-learning portal of Korean medical schools.
-
Methods
Data were obtained from learning resources and registered users of the e-learning portal by the consortium of 36 Korean medical schools, e-MedEdu (www.mededu.or.kr) over a period of 10 years. Data analytics were performed of its contents and usage patterns using descriptive statistics.
-
Results
The website currently has over 1,600 resources, which have almost tripled over the past decade, and 28,000 registered users. Two hundred and twenty medical faculty have contributed the resources; a majority of them were clinical cases and video clips, which accounted for 30% and 27% of all resources, respectively. The website has received increasing hits over the past decade; annual website hits increased from 80,000 in 2009 to over 300,000 in 2018. The number of hits on resources varied across resource types and subjects; 90% of all website hits were on online videos, and 28% of them originated from mobile devices. Among the online videos, those on procedural skills received more hits than those on patient encounters and video lectures.
-
Conclusion
Our findings demonstrate the increasing use of e-learning in medical education in Korea over the past decade. Our study also shows a wide disparity in the frequency of use in learning resources across resource types and subjects, which have implications for improvements in the design and development of learning resources to better meet medical students’ curricular needs and their learning styles.
-
Key Words: e-learning, Online learning, Medical education
Introduction
e-Learning has been a key feature of learning technologies in medical education and is expected to maintain its position of prominence [
1,
2]. e-Learning encompasses many technologies and activities that constitute it, where the learning process is supported or mediated using technologies [
3]. A meta-analysis of research in medical education indicates e-learning is associated with positive outcomes across a wide variety of learners, learning contexts, clinical topics, and learning outcomes [
4].
Still, studies have shown several significant barriers hinder the adoption and implementation of e-learning by medical schools [
5]. Universities and colleges in South Korea have actively adopted e-learning since the early 2000s in parallel with rapid, nationwide developments in information and communication technologies (ICT). This was due, in large part, to a government initiative to become a global leader in ICT and to provide lifelong education. In 2006, about half of all colleges in Korea were offering e-learning, but relatively few of the contents of these courses were developed in medical schools [
6]. This tardy adoption of e-learning by Korean medical schools was largely due to lack of resources for it [
6]. To resolve this issue, Korean medical schools established a national consortium with the aim of offering quality e-learning resources developed by medical faculty to medical students. Currently, 36 of 40 Korean medical schools are participating in this consortium, which has grown from 25 schools since its inception in 2007.
Despite the wide adoption of e-learning in Korean medial schools initiated by the consortium, little is known about the current status of e-learning in Korean medical education. This paper reports the use of e-learning based upon the data from e-MedEdu accumulated over a decade to investigate medical students’ online learning patterns and needs. We sought to answer the following questions: (1) How many users have registered and how many faculties have contributed contents to e-MedEdu? (2) What types of resources are available at e-MedEdu and in what subjects? (3) How often these resources are accessed and which ones are the most frequently visited?
Methods
1. Development and organization of learning resources
The consortium offers an e-learning portal (
www.mededu.or.kr), which is a repository of online resources contributed by faculty members from medical schools participating in the consortium. These resources are available free of charge to members of participating schools. An executive committee, whose members are representatives of member schools, runs the consortium. This committee meets regularly, reviews the status of content developments and sets goals for future activities. Member schools in the consortium have collaborated to develop various learning resources, which include clinical cases, clinical videos, clinical images, and quizzes, which are peer-reviewed before being made available on the website.
The learning resources at e-Mededu are intended to be used in various teaching and learning contexts. Most importantly, students can access resources at their convenience to meet their learning needs to supplement conventional instructions and for self-study outside of class. Thus, the availability of these resources aims to facilitate students’ self-directed learning. Faculty members can also utilize these resources in various instructional settings, as student-centered learning and active learning are of increasing importance in medical education [
7,
8]. For example, faculty members can ask students to pre-study e-learning resources before a class, or use these resources as teaching materials during lectures or for out-of-class activities in flipped classrooms [
9].
Users can find resources in e-MedEdu using a search engine or by browsing them by the categories offered at the website. In addition to the website, mobile applications were launched in 2011 that enable users to use e-MedEdu in mobile settings in response to the growing demand for mobile learning [
10]. Users can also browse through the resources by major learning outcomes in basic medical education. This organization of learning resources help students find resources in subjects of their interest with more ease. This also helps faculty members identify needs for content development by mapping them with learning outcomes.
Data analysis was performed on data collected from e-MedEdu using the data analytics method. Data analytics for education involves the analysis of large datasets, typically concerning the use of e-learning technologies, to identify patterns in educational practice or performance in single or aggregate datasets to deal with educational issues and problems [
11]. Data on registered users, contents, and website hits at e-MedEdu were collected from September 2009 (when the website was launched) to September 2018. Data on cumulative hits on contents were collected over a 7-year period from October 2011, when we started collecting log data for each resource from the database, until October 2018. Data analytics were performed of its contents and usage patterns. The data was obtained by running queries on the database regarding users, contents, and number of visits on them. The data were also stratified by resource types and subject categories. Descriptive statistics was performed for data analysis.
Institutional review board (IRB) approval was not requested for the present study, because it complied with the general exemption issued by our IRB, as the data utilized already existed and its acquisition did not involve any direct contact with any human subject. All data were collected anonymously.
Results
1. Resources and user statistics for e-MedEdu
Table 1 shows the number of resources and registered users at e-MedEdu. Currently, e-MedEdu has over 1,600 resources, of which 87 (5.3%) were on basic sciences, 7 (0.4%) were on medical humanities; others (94.3%) were on clinical sciences. When classified by content type, the majority of these resources were clinical cases and online videos, followed by clinical images, quizzes, and audio clips. Other resources included online modules and animations. There were currently 27,747 registered users; approximately 25,000 were students, 2,000 were faculty members, and the others were primarily administrative staff.
Fig. 1 shows screenshots of sample resources at eMedEdu. A repository of clinical cases were available in e-MedEdu to supplement students’ clinical experiences as increasing emphasis is being placed on learning and assessment in authentic contexts in an outcome-based education framework [
12]. The cases were presented in texts accompanied by multimedia, which included clinical images (e.g., chest X-rays and computed tomography), sound clips, and video clips. These cases were originally categorized into 17 themes based upon organ systems. With the learning outcomes for basic medical education based upon clinical presentations developed by the Association of Korean Medical Colleges, the clinical cases were also mapped with the clinical presentations since 2014 so that users could browse through cases by clinical presentations as well as by organ-systems.
The online videos available in e-MedEdu consisted of: (1) video clips that demonstrated procedural skills, (2) video clips on patient encounters, and (3) video lectures on various topics, which included basic and clinical sciences, and medical humanities. Online videos included all subjects covered in the clinical skills test in the Korean medical licensing exam. Other resources at e-MedEdu included clinical images (e.g., X-rays, magnetic resonance imaging, and pathological slides), audio clips of heart and lung sounds, online modules on various topics (basic science, clinical science, and medical humanities), and quizzes.
The contents of e-MedEdu have grown consistently over the past decade from 811 in 2009 to 1,636 in 2018.
Fig. 2 shows changes in the number of contents over the past decade by resource types. As seen in
Fig. 2, content development has been active on online videos and clinical cases, whereas it has been slow on image resources.
The number of faculty members that have contributed to e-MedEdu has also grown from 80 in 2009 to 220 in 2018. The faculty members who contributed contents to e-MedEdu were from 29 medical schools. The number of resources contributed by a faculty member ranged from 1 to 334 (median=1); approximately half of them (n=112) contributed more than one resource.
2. Usage statistics on resources
Fig. 3 illustrates the growth of the website visits over the past decade. The annual website hits have increased consistently from 84,361 in 2009 to 508,857 in 2018.
Table 2 shows the number of accumulative hits by resource types over the past 7 years. The contents that users visited mostly frequently were online videos, which accounted for 90% of all website hits. This was followed by clinical cases (5.8%) and clinical images (2.7%). Of all hits on online videos over the past 7 years, 72% (n=636,027) were from the Web and 28% (n= 243,499) were from mobile devices.
Table 3 presents the number of hits on clinical cases categorized by organ systems and clinical presentations. In terms of organ-systems, obstetrics & gynecology, gastroenterology, and oncology received most hits. e-MedEdu contained clinical cases for 80 of the 101 clinical presentations identified as learning outcomes for basic medical education in South Korea. Of these clinical presentations, users most frequently looked up cases of dyspnea, fever (of unknown cause), and cough. An average number of accumulated hits on clinical cases for the duration of 7 years were 243.41 per case. The number of hits on clinical cases differed across subjects; they ranged from 98.05 to 360.22 per case across different organ-based subjects and from 60.00 to 411.40 across different clinical presentations.
Table 4 illustrates usage statistics on online videos. Of all video clips in e-MedEdu available, 65% (n=285) were on procedural skills, 19% (n=80) were on patient encounters, and 16% (n=68) were video lectures on various topics, including basic and clinical sciences and medical humanities. Among the online videos, those on procedural skills received the greatest hits for the duration of 7 years, where an average number of hits per video clip was 4,163.26. This was followed by video clips on patient encounters (1,043.47 hits per clip). The number of hits on video clips differed across subjects; they ranged from 2,505.00 to 6,717.86 across different subjects on procedural skills, and the number of hits on videos on patient encounter ranged from 681.89 to 1,594.50.
Discussion
Our study demonstrates e-learning is being increasingly used for medical education in South Korea. We have seen that e-learning has increasingly become mainstream in the teaching and learning in medical education over the past decade. Our experience shows e-learning in this context is feasible, given collaboration among medical schools to develop strategic plans for e-learning and implement them. We believe the Korean e-learning consortium has demonstrated a successful model for implementing e-learning in medical education at the national level.
One of reasons for the increasing use of e-learning found in the present study was the consistent efforts made by medical educators to develop learning resources that meet the curricular needs in basic medical education. Due to the broad range of subjects that must be covered in medical education, it is a considerable challenge to develop resources on all subjects. In terms of the on-going development of e-MedEdu, we have matched existing contents with learning outcomes for basic medical education. In doing so, we were able to recognize that there were some clinical presentation topics in which clinical cases were not available. As a result of this process, we have built up a repository of contents that covers most of the basic learning outcomes in terms of clinical presentations and clinical skills. Having over 30 faculty members from various disciplines in the executive committee of the consortium and a large volume of medical teachers who have contributed contents to e-MedEdu, we were able to draw on these members for support and identify experts in subject areas on a nationwide basis. Expanding collaboration with faculties with expertise in other diverse disciplines will further develop our ability to promote advances in e-learning.
This study also shows the popularity of learning resources offered by e-MedEdu differs substantially across resources types. Our study illustrates the current popularity of online videos as learning resources among medical students. This finding concurs with that of Khogali et al. [
13] that students valued animation and video demonstrations highest among e-learning resources. It is likely that the prevalent use of online videos continues in the foreseeable future as they play an important role in clinical education. It is also likely that the creation and use of educational videos becomes more widespread in medical education with the increasing popularity of social media. This trend may emulate the learning styles of today’s medical students, who are millennial learners living and learning with technologies [
14].
Moreover, the effectiveness of using technology to enhance the learning of clinical skills using video clips appears to have motivated students to engage in e-learning. Our previous study showed medical student perceptions of clinical videos available at e-MedEdu that those resources were helpful in supplementing conventional face-to-face instruction on clinical skills [
15]. This finding is also consistent with those of previous studies that showed student satisfaction with clinical videos for the learning of clinical skills [
16,
17]. This rise in popularity of clinical videos is also in large part due to the implementation of clinical skills test in the Korean licensing exam since 2009, which illustrates the medical student needs for e-learning that meet their curricular needs.
Despite the importance of clinical cases in medical education, our findings show they are not being widely used at e-MedEdu. This finding indicates our clinical cases need to be better integrated with medical school curriculum, which warrants more active use of these resources as teaching materials on the part of the faculty. In the previous study, we found the low use of learning resources at e-MedEdu by faculty members for their teaching and the main reasons were lack of awareness of these resources and of pedagogical understanding of their utilities [
15]. The consortium may need to enhance its efforts to identify best practices in the use of clinical cases by faculty members and to disseminate them so that they make active and effective use of learning resources in their teaching.
Furthermore, our finding of the low use of clinical cases by medical students may suggest they are not suited for the learning styles of today’s students as well as online videos are. This indicates we need to make clinical cases more interactive and engaging for the students. This warrants future research on the design of clinical cases. There are various formats for presenting clinical cases, from a simple text-based format to promoting interaction with the virtual patient in a computer-based simulation environment [
18]. It is also plausible to present patient encounters using virtual and augmented reality, which are likely to become a prevalent technology in the near future [
19] and are increasingly garnering the attention of medical educators [
20]. Still, research is scant on such learning environments. Future research is warranted on more effective design of clinical cases to investigate whether they are used more widely by medical students and how to design them to better promote students’ clinical reasoning skills.
Our study also found differences in the popularity of learning resources across subjects. Among online videos, clips on procedural skills were more frequently viewed by users than those on patient encounters and video lectures. Moreover, there were also differences in the frequency of use of online videos on procedural skills across content domains. Such differences might reflect the learning needs for medical students to supplement conventional instructions on clinical skills particularly on those not readily available for direct observation or those that demonstrate somewhat complex procedural skills. There were also differences in the frequency of use in clinical cases, and there might be several reasons for that. Some cases likely fit more closely with the core curriculum and some might be beyond the scope of basic medical education.
Our study has implications for future directions of e-learning in Korean medical education. First, there is a need to build up more multimedia resources in response to the trend of incorporation of multimedia items for student assessment to enhance its authenticity [
21,
22]. Our efforts to build up resources have been active on clinical cases and online videos, yet it has been slow in other multimedia resources such as images and sound clips. Second, our findings show our resource development has focused on the core curriculum pertinent to learning outcomes for basic medical education. There is a growing emphasis on “student-selected components” (SSCs) in medical school curricular to promote student-centered learning by offering subjects of more choice and depth of study than the core curriculum [
23]. Therefore, there is a need to expand our resources to support such curricular components. Furthermore, e-learning has the potential to offer an effective learning environment for SSCs by offering diverse learning resources and the flexibility for the learning of such components. The inter-institutional collaboration model of the e-learning consortium offers valuable experience in such collaborative efforts to provide learning resources that cover a broad range of subjects for SSCs.
This study has some limitations. First, this study was limited to the format of e-leaning which offer online learning resources through an e-learning portal. As there are various formats of e-learning, such as online courses, and various forms of learning resources are available in medical education, cautions need to be taken in generalizing our findings to other e-learning contexts. Second, although this study found the use of learning resources differed across resource types and subjects, it does not offer direct evidence as to what caused such differences. This study calls for future research to explore what causes such differences to better understand medical students’ use of learning resources in their learning. Such a study will help us advance of our knowledge of more effective use of e-learning that meets the learning needs of today’s students and help improve medical education by supplementing face-to-face instructions. Third, our data was not sufficient to explore students’ activities at our website associated with their academic performance. Future study of learner analytics regarding differences in the use of resources at e-MedEdu between high-achieving and low-achieving students is recommended to advance our understanding of effective use of e-learning.
ACKNOWLEDGMENTS
None.
ACKNOWLEDGMENTS
None.
Funding
None.
Conflicts of interest
No potential conflict of interest relevant to this article was reported.
Author contributions
KJK and GK had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Concept and design: all authors; acquisition, analysis, and interpretation of data: all authors; drafting of the manuscript: KJK; and critical revision of the manuscript for important intellectual content: all authors.
Fig. 1. Screenshots of Sample Resources at e-MedEdu
Fig. 2. Changes in the Number of Learning Resources in e-MedEdu by Resource Types
Fig. 3. Annual Website Hits at e-MedEdu
Table 1.e-MedEdu Content and User Statistics
Table 1.
|
Variable |
No. of contents or users |
|
Content types |
|
|
Clinical cases |
497 |
|
Video clips |
437 |
|
Clinical images |
335 |
|
Quizzes |
286 |
|
Audio clips |
46 |
|
Others |
35 |
|
Total |
1,636 |
|
User types |
|
|
Faculty |
1,983 |
|
Students |
25,571 |
|
Others |
193 |
|
Total |
27,747 |
Table 2.Number of Accumulate Hits on Contents by Resource Types over the Past 7 Years
Table 2.
|
Resource type |
Accumulate hits (%) |
|
Online videos |
879,670 (89.6) |
|
Clinical cases |
56,832 (5.79) |
|
Clinical images |
27,026 (2.75) |
|
Quizzes |
15,581 (1.59) |
|
Others (e.g., online modules) |
2,673 (0.27) |
|
Total |
981,782 (100) |
Table 3.Usage Statistics of Clinical Cases at e-MedEdu
Table 3.
|
No. |
Objects by organ systems |
No. of cumulative hits |
No. of cases |
No. of hits per case |
No. |
Subjects by clinical presentations |
No. of cumulative hits |
No. of cases |
No. of hits per case |
|
1 |
Obstetrics and gynecology |
10,786 |
110 |
98.05 |
1 |
Dyspnea |
9,849 |
32 |
307.78 |
|
2 |
Gastroenterology |
10,549 |
31 |
340.29 |
2 |
Fever, unknown cause |
8,742 |
30 |
291.40 |
|
3 |
Oncology |
10,188 |
44 |
231.55 |
3 |
Cough |
6,569 |
18 |
364.94 |
|
4 |
Pulmonology, allergy |
9,726 |
27 |
360.22 |
4 |
Acute abdominal pain |
5,914 |
31 |
190.77 |
|
5 |
Infectious diseases |
9,015 |
39 |
231.15 |
5 |
Weakness, paralysis |
5,082 |
19 |
267.47 |
|
6 |
Neurology |
5,708 |
18 |
317.11 |
6 |
Chest pain |
4,905 |
15 |
327.00 |
|
7 |
Endocrinology |
5,473 |
24 |
228.04 |
7 |
Headache |
3,270 |
13 |
251.54 |
|
8 |
Cardiovascular |
5,160 |
16 |
322.50 |
8 |
Abdominal mass |
3,190 |
20 |
159.50 |
|
9 |
Pediatrics |
4,633 |
22 |
210.59 |
9 |
Loss of consciousness, coma |
2,796 |
10 |
279.60 |
|
10 |
Musculoskeletal |
4,623 |
20 |
231.15 |
10 |
Chronic abdominal pain, dyspepsia, heartburn |
2,455 |
12 |
204.58 |
|
11 |
Nephrology, urinary system |
3,978 |
16 |
248.63 |
11 |
Nausea, vomiting |
2,170 |
9 |
241.11 |
|
12 |
Hematology |
3,229 |
14 |
230.64 |
12 |
Vaginal bleeding |
2,164 |
19 |
113.89 |
|
13 |
Emergency medicine |
2,070 |
10 |
207.00 |
13 |
Dizziness, vertigo |
2,061 |
8 |
257.63 |
|
14 |
Otorhinolaryngology |
1,528 |
6 |
254.67 |
14 |
Joint pain, joint swelling |
2,057 |
5 |
411.40 |
|
15 |
Psychiatry |
849 |
3 |
283.00 |
15 |
Dysphagia |
1,960 |
3 |
653.33 |
|
16 |
Dermatology |
247 |
1 |
247.00 |
… |
… |
|
|
|
|
17 |
Ophthalmology |
194 |
1 |
194.00 |
80 |
Contraception |
63 |
1 |
63.00 |
|
Total |
105,638 |
434 |
243.41 |
|
Total |
105,638 |
434 |
243.41 |
Table 4.Usage Statistics of Clinical Cases at e-MedEdu
Table 4.
|
No. |
Videos on procedural skills
|
Videos on patient encounter
|
|
Category |
No. of cumulative hits |
No. of video clips |
No. of hits per clip |
Category |
No. of cumulative hits |
No. of video clips |
No. of hits per clip |
|
1 |
Physical examination |
323,098 |
89 |
3,630.31 |
General symptoms |
26,400 |
24 |
1,100.00 |
|
2 |
Vascular procedures |
201,051 |
37 |
5,433.81 |
Respiratory and cardiovascular symptoms |
17,961 |
13 |
1,381.62 |
|
3 |
Surgical procedures |
172,486 |
39 |
4,422.72 |
Psychological, neurological, and musculoskeletal symptoms |
14,884 |
21 |
708.76 |
|
4 |
Emergent procedures |
143,277 |
35 |
4,093.63 |
Gastrointestinal symptoms |
12,990 |
14 |
927.86 |
|
5 |
Miscellaneous procedures |
119,882 |
34 |
3,525.94 |
Difficult communication |
10,408 |
7 |
1,486.86 |
|
6 |
Neurologic examination |
103,721 |
28 |
3,704.32 |
Women health |
6,378 |
4 |
1,594.50 |
|
7 |
Obstetric and gynecologic procedures |
68,475 |
13 |
5,267.31 |
Others |
6,137 |
9 |
681.89 |
|
8 |
Urologic procedures |
47,025 |
7 |
6,717.86 |
Genitourinary symptoms |
4,151 |
4 |
1,037.75 |
|
9 |
Mental status examination |
7,515 |
3 |
2,505.00 |
- |
- |
- |
- |
|
Total |
1,186,530 |
285 |
4,163.26 |
Total |
99,309 |
96 |
1,034.47 |
References
- 1. Harden RM. Ten key features of the future medical school-not an impossible dream. Med Teach. 2018;40(10):1010-1015.
- 2. Ellaway R, Masters K. AMEE guide 32: e-Learning in medical education part 1: learning, teaching and assessment. Med Teach. 2008;30(5):455-473.
- 3. Ellaway R. E-learning: is the revolution over? Med Teach. 2011;33(4):297-302.
- 4. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Internet-based learning in the health professions: a meta-analysis. JAMA. 2008;300(10):1181-1196.
- 5. O’Doherty D, Dromey M, Lougheed J, Hannigan A, Last J, McGrath D. Barriers and solutions to online learning in medical education: an integrative review. BMC Med Educ. 2018;18(1):130.
- 6. Kim S, Shin J. Report on the present state of e-learning in medical schools. Paper presented at: Proceedings of the 19th meeting of the Korean Society for Medical Education; 2006; Seoul, Korea.
- 7. Prober CG, Khan S. Medical education reimagined: a call to action. Acad Med. 2013;88(10):1407-1410.
- 8. Prober CG, Heath C. Lecture halls without lectures: a proposal for medical education. N Engl J Med. 2012;366(18):1657-1659.
- 9. Bouwmeester RA, de Kleijn RA, ten Cate OT, van Rijen HV, Westerveld HE. How do medical students prepare for flipped classrooms? Med Sci Educ. 2016;26(1):53-60.
- 10. Masters K, Ellaway RH, Topps D, Archibald D, Hogue RJ. Mobile technologies in medical education: AMEE guide no. 105. Med Teach. 2016;38(6):537-549.
- 11. Ellaway RH, Pusic MV, Galbraith RM, Cameron T. Developing the role of big data and analytics in health professional education. Med Teach. 2014;36(3):216-222.
- 12. Harden RM. Developments in outcome-based education. Med Teach. 2002;24(2):117-120.
- 13. Khogali SE, Davies DA, Donnan PT, et al. Integration of e-learning resources into a medical school curriculum. Med Teach. 2011;33(4):311-318.
- 14. Roberts DH, Newman LR, Schwartzstein RM. Twelve tips for facilitating Millennials’ learning. Med Teach. 2012;34(4):274-278.
- 15. Kim KJ, Kang Y, Kim G. The gap between medical faculty’s perceptions and use of e-learning resources. Med Educ Online. 2017;22(1):1338504.
- 16. Gormley GJ, Collins K, Boohan M, Bickle IC, Stevenson M. Is there a place for e-learning in clinical skills?: a survey of undergraduate medical students’ experiences and attitudes. Med Teach. 2009;31(1):e6-e12.
- 17. Jang HW, Kim KJ. Use of online clinical videos for clinical skills training for medical students: benefits and challenges. BMC Med Educ. 2014;14:56.
- 18. Bateman J, Allen M, Samani D, Kidd J, Davies D. Virtual patient design: exploring what works and why: a grounded theory study. Med Educ. 2013;47(6):595-606.
- 19. Becker SA, Brown M, Dahlstrom E, et al. NMC Horizon report: 2018 higher education edition. Louisville, USA: EDUCAUSE; 2018.
- 20. Craig E, Georgieva M. VR and AR: driving a revolution in medical education & patient care. Louisville, USA: EDUCAUSE; 2017.
- 21. Shen L, Li F, Wattleworth R, Filipetto F. The promise and challenge of including multimedia items in medical licensure examinations: some insights from an empirical trial. Acad Med. 2010;85(10 Suppl):S56-S59.
- 22. Holtzman KZ, Swanson DB, Ouyang W, Hussie K, Allbee K. Use of multimedia on the step 1 and step 2 clinical knowledge components of USMLE: a controlled trial of the impact on item characteristics. Acad Med. 2009;84(10 Suppl):S90-S93.
- 23. Riley SC. Student selected components (SSCs): AMEE guide no 46. Med Teach. 2009;31(10):885-894.
