Abstract
-
Purpose
Evaluating the performance of medical school graduates after graduation is important. However, reliable and comprehensive tools to evaluate the performance of medical professionals after graduation are lacking. The purpose of this study was to evaluate the validity and reliability of performance indicators for post-graduation competencies of medical school graduates.
-
Methods
Nineteen performance indicators were validated to evaluate competencies considering the talent image of a medical school, which are medical professionalism, clinical skills, communication, collaboration, and leadership. The reliability of the assessment tool was tested using Cronbach’s alpha, construct validity was evaluated through factor analysis, and content validity was evaluated using a Delphi expert panel.
-
Results
The overall reliability of the performance indicators was high, with a Cronbach’s alpha of 0.9110. Factor analysis revealed five core factors accounting for 70% of the total variance. These factors were classified as “collaboration,” “clinical professionalism,” “patient-centered care,” “professionalism,” and “systematic treatment and self-development.” Content validity was confirmed by the Delphi panel, and all items achieved a content validity ratio of 1, indicating strong content validity.
-
Conclusion
The developed performance indicators are reliable and valid tools for assessing the competencies of medical school graduates. These indicators can be used to evaluate the quality of medical education and to improve the curriculum. It is also important to establish a system to periodically assess competencies after graduation.
-
Key Words: Competency‑based education, Delphi technique, Reliability, Professional competence, Validity
Introduction
Consequent to changes in the medical environment and increases in social demands, interest in the competencies that medical school graduates should possess has also increased [
1,
2]. Traditionally, medical education has focused on acquiring medical knowledge and clinical skills; however, recently, various competencies like patient-centered treatment ability, ethical judgment, communication ability, collaboration ability, and continuous self-directed learning ability have been emphasized [
3,
4]. Accordingly, it is necessary to develop an objective and reliable evaluation index that can systematically evaluate the performance of medical school graduates and verify its reliability and validity [
5].
Healthcare service quality is significantly influenced by physicians’ professionalism and performance. Physicians’ competencies encompass not only the accumulation of knowledge but also clinical decision-making, communication with patients, collaboration with colleagues, and continuous self-improvement [
6,
7]. Thus, it is important to understand that medical education does not end with earning a degree; it extends to understanding how well graduates perform in real-world clinical settings. Furthermore, it is essential to evaluate whether graduates continue to fulfill their professional roles responsibly by applying their medical knowledge and clinical skills after graduation [
4,
8].
Currently, most medical schools use a variety of assessment tools, such as written exams, clinical practice evaluations, standardized patient assessments, and objective structured clinical examinations, to evaluate students’ knowledge and skills. However, comprehensive tools that assess the actual clinical performance post-graduation remain limited, and existing evaluations may not fully reflect graduates’ real-world competence. Additionally, each medical school strives to nurture its ideal type of physician as defined by its unique educational philosophy, mission, and desired competencies.
However, without a systematic post-graduation evaluation aligned with these institutional goals, it is difficult to determine whether the intended competencies have been achieved. Medical education should extend beyond academic instruction to cultivate professionals capable of fulfilling their roles in clinical settings, and robust assessment tools are essential for evaluating such competencies [
2].
Inje University College of Medicine (IUCM) adopted an integrated evaluation model combining Stufflebeam’s Context, Input, Process, Product (CIPP) model and Kirkpatrick’s four-level evaluation model [
9].
In this model, the “Product” part of the CIPP model was applied to Kirkpatrick’s four-stage model, and each stage of the evaluation was designed to set evaluation criteria and develop evaluation indicators to collect and evaluate data, considering the desired competencies of IUCM. Additionally, the IUCM defined desirable long-term competencies of “a good doctor” based on its desired competencies and collected data on students’ and graduates’ performance to build a longitudinal cohort [
10].
Although many medical schools have proposed common graduate competencies, the assessment tools tailored to each school’s unique mission are lacking. This study is meaningful because it presents validated assessment indicators developed by considering the specific educational goals of medical schools.
Therefore, this study shares the results of the validation and reliability analysis of assessment indicators developed to evaluate the post-graduate performance of medical school graduates based on desired institutional competencies. Ultimately, we expect that this result will contribute to curriculum improvement and enhancement of medical education quality by ensuring that graduates are equipped to function effectively within their educational and healthcare systems.
Methods
1. Post-graduation performance evaluation items
The IUCM applied the “Product” stage of Stufflebeam’s CIPP model in conjunction with Kirkpatrick’s four-stage evaluation model (Reaction, Learning, Behavior, Results). Particularly, the “Results” level aimed to comprehensively understand the actual clinical performance capabilities of graduates, and for this purpose, detailed the evaluation criteria based on the desired competencies of IUCM: medical expert, communicator, collaborator, leader, scholar, and professional. To derive the 19 evaluation items, we first mapped the six desired competencies of IUCM “( a good doctor”: medical expert, communicator, collaborator, leader, scholar, and professional) onto the Behavior and Results levels of Kirkpatrick’s model within the framework of the CIPP “Product” stage. Each competency was then operationalized into specific and measurable behaviors through committee meeting. Based on this mapping, draft items were developed to reflect IUCM’s talent image and educational philosophy. Overlapping or ambiguous items were revised through iterative consensus among committee members. As a result, 19 items were finalized, representing an integration of IUCM’s competency framework with outcome-centered evaluation principles [
9] (
Fig. 1).
1) Study participants and data collection
This study uses data from a graduate cohort established using the integrated evaluation model of the IUCM, specifically focusing on responses to postgraduate performance evaluation items.
2) Ethical considerations
The data utilized in this study were evaluation items of the post-graduation performance ability assessment index collected from a cohort of medical school graduates. The Institutional Review Board (IRB) of Busan Paik Hospital (IRB number: 2021-02) approved this study to use the collected cohort data.
3) Data analysis
Descriptive statistics were used to summarize continuous variables as means and standard deviations (SDs), and categorical variables as frequencies and percentages. Cronbach’s alpha was calculated to assess the internal consistency and reliability of the evaluation tool [
11].
Both construct and content validity were examined. Construct validity was evaluated through factor analysis using principal component analysis with varimax rotation. Eigenvalues and factor loadings were extracted to determine the factor structure.
The content validity was assessed using the Delphi method. The panel for the Delphi study consisted of five experts in medical school, education, medical education, and medical education evaluation who had extensive experience in medical education or who had experience in medical education, medical education evaluation, and postgraduate education, and were working in related fields. The Delphi survey was conducted in two rounds. Invitations were sent via email, and those who consented participated in both rounds of the survey. The content validity ratio (CVR) was calculated for each item based on the final round of results to determine content validity.
Given the number of experts (N=5), a CVR greater than 0.99 was considered acceptable, following the criteria by Lawshe [
12]. The CVR was calculated as follows [
11]:
CVR=(ne–N/2)/(N/2)
where ne is the number of panelists indicating the item is essential.
Results
1. General characteristics of the participants
Table 1 summarizes the general characteristics of those included in the graduate cohort and who responded to the items of the performance evaluation indicator after graduation. A total of 208 graduates participated in this study. Among them, 175 (84.13%) were male, and 33 (15.87%) were female. There were 55 participants (26.44%) holding academic positions, and 153 (73.56%) engaged in private or hospital-based clinical practice. The mean period after graduation was 25.38 years (SD=8.24 years), with a median of 26.00 years and a range of 5 to 39 years.
2. Reliability analysis
The overall Cronbach’s alpha coefficient for the reliability analysis of the post-graduation performance evaluation index was 0.9110 (
Table 1), indicating high internal consistency. The subscale reliability coefficients ranged from 0.8645–0.8831. The item with the highest individual reliability was “I play the role of an educator who educates students.” However, when this item was excluded, the Cronbach’s alpha coefficient increased, suggesting that its removal improved the internal consistency. Cronbach’s alpha was 0.8925. The item with the highest reliability in the revised set was “I understand research ethics and apply them in practice,” followed by “I search for literature with a high level of evidence and conduct self-directed learning,” and “I treat patients within the current medical system or insurance system.” In the professionalism domain, the highest reliability was observed for the item “emergency patient treatment.”
1) Construct validity
To verify whether the number of variables and cases in this study were suitable for factor analysis, a Kaiser-Meyer-Olkin test was conducted, and the result was 0.889. Prior to factor analysis, Bartlett’s sphericity test was conducted to verify the validity of the factor analysis. To determine the appropriateness of factor analysis, Bartlett’s sphericity test was conducted, and the result χ
2=2,247.19 (p<0.01), confirming that factor analysis was appropriate. Factor analysis was conducted to evaluate the construct validity of the performance ability evaluation items after graduation. Principal component analysis and varimax rotation, which is an orthogonal rotation, were performed to extract as many factors as possible. When factor analysis was conducted on the 19 evaluation items, five factors were extracted based on eigenvalues greater than 1.0. The number of factors was also determined to be five according to the Scree test (
Table 2). The factor loadings were as follows: (1) factor 1: 0.3708–0.8548; (2) factor 2: 0.7460–0.8222; (3) factor 3: 0.4458–0.9048; (4) factor 4: 0.4893– 0.8417; and (5) factor 5: 0.5973–0.6825.
None of the items had loadings below 0.3. The explanatory power of each factor was as follows: (1) factor 1: 19.5%; (2) factor 2: 17.3%; (3) factor 3: 13.3%; (4) factor 4: 12.1%; and (5) factor 5: 7.8%. The five factors explained 70.0% of the total variance (
Table 3). The five factors were interpreted as follows:
a. Factor 1 (collaboration)
This included six items, such as “I collaborate effectively with physician colleagues in clinical settings” and “I engage in interprofessional collaboration with physicians and other healthcare professionals to deliver patient-centered care.”
b. Factor 2 (medical expertise)
This included four items related to core clinical competencies, such as “diagnosis and planning, clinical skills, emergency care,” and “efforts to identify adverse events and minimize risks.”
c. Factor 3 (patient-centered care)
This consisted of three items, namely “I explain and educate patients about their diagnosis and treatment process,” “I explain the diagnosis and treatment process to patients’ families or caregivers,” and “I complete medical records in a timely and accurate manner.”
d. Factor 4 (professional development)
This included three items, such as “I search for literature with a high level of evidence and conduct self-directed learning” and “I make continuous efforts to stay up to date with current medical knowledge after board certification.”
e. Factor 5 (systematic practice and self-reflection) This included two items, namely “I treat patients within the current medical system or insurance system,” and “I reflect on areas for improvement in my clinical practice.”
2) Content validity
To evaluate the content validity of the post-graduation performance assessment items, a Delphi panel was formed with a group of experts to provide advice. The Delphi panel consisted of five experts in medical schools, education, medical education, and medical education evaluation who had extensive experience in medical education or who had experience in medical education, medical education evaluation, and postgraduate education, and were working in related fields. The panel consisted of two females (40%) and three males (60%), with an average of 21 years of experience (SD=3.24 years).
In the first round, the average score for each item was 4.73 (SD=0.28), with 4 points or more for each evaluation item. In the second round, the average score increased to 4.96 (SD=0.18). The CVR for each item in the second round was 1.0, exceeding the critical value of 0.99 for five experts, as suggested by Lawshe [
12], and confirming excellent content validity (
Table 4).
Discussion
This study aimed to validate the reliability of the items for evaluating postgraduate performance ability, considering the desired competencies of medical school personnel. Evaluation items for postgraduate performance ability are an important index that can objectively and systematically assess the professional competence of graduates. These indicators can provide fundamental data for evaluating the quality of medical education and determining whether graduates fulfill their institutional missions and desired competencies.
The Cronbach’s alpha coefficient for the overall evaluation items was 0.9110, indicating excellent internal consistency. Each subscale showed a reliability coefficient above 0.86, suggesting that the items consistently measured the intended competencies. Interestingly, the reliability increased slightly when the item “I play a role as an educator who educates students” was excluded. This may reflect the diverse career paths of graduates, including clinical practitioners, who may not be actively involved in formal medical education. As many graduates contribute to education by mentoring residents or junior colleagues, this finding offers useful insights for refining future versions of this tool.
Construct validity was supported by a factor analysis, which identified five factors that explained 70% of the total variance. These factors aligned well with the key domains of graduate performance, such as collaboration, professionalism, patient-centered care, and system-based practice. Content validity was also demonstrated as all items received high ratings in the Delphi study, and the CVR for each item exceeded the critical threshold, confirming that the indicators were appropriate and meaningful. Furthermore, the five factors derived in this study show substantial alignment with the “Diagram of Korean doctors’ roles” announced by the Korean Medical Association (KMA) in 2014 [
13]. The KMA framework defines five domains—personal, practice/education/research, social, communication/cooperation/social responsibility, and professionalism—and our factors correspond to these domains as follows. Our factor of medical expertise corresponds closely to the practice/education/research domain, while collaboration and patient-centered care reflect the domain of communication, cooperation, and social responsibility. The factor of professional development overlaps with professionalism, and system-based practice/self-reflection incorporates elements of both the social and personal domains. This concordance suggests that the validated indicators in this study not only represent institution-specific competencies but also resonate with nationally defined physician roles, thereby enhancing their credibility and applicability in the Korean medical education context.
In addition to confirming reliability and validity, these indicators are practical tools that can be applied in multiple contexts. First, they can be used as performance metrics in graduate tracking systems, allowing medical schools to monitor whether graduates maintain expected competencies over time. Second, they provide a framework for curriculum evaluation and quality assurance, helping institutions assess their educational programs. Third, these indicators can serve as common benchmarks across medical schools, providing a reference point for other institutions developing or implementing similar tools. Finally, they can inform national discussions on physician competency standards, connecting institutional missions to broader professional roles. By clarifying these applications, this study can contribute not only to the validation of items but also to establishing a systematic foundation for postgraduate competency-based assessment.
The study participants included a diverse group of graduates: academic faculty, private practitioners, and hospital-employed physicians. Given differences in job roles and workplace environments, certain items may vary in relevance. For example, the lower reliability of the teaching item may reflect the fact that not all graduates were involved in formal education. Therefore, it may be necessary to revise or differentiate the assessment items according to the professional context in future applications of the tool.
This study was limited by the fact that it validated the assessment tool using data from a single medical school. Therefore, the generalizability of these findings to other medical schools may be limited. Additionally, since a self-report survey was used, the responses may have been influenced by social desirability bias, which could have compromised the accuracy of the self-assessed performance. Another limitation is that the validation was conducted with a single annual cohort. Although this provides an initial confirmation of reliability, further studies involving multi-year cohorts are necessary to evaluate the longitudinal stability of the indicators and to ensure that they remain robust across different graduation years.
In future studies, it will be necessary to supplement the assessment validity by utilizing a multi-institutional cohort or mixed methodology, including actual behavioral observations. Since performance ability after graduation may change over time, it is necessary to establish a systematic framework that can continuously evaluate this ability even after a certain period has passed. While common roles were included in the instrument, they were not directly operationalized as Entrustable Professional Activities (EPAs). Therefore, future research is needed to map roles to EPAs and validate the delegation-based framework to enhance its workplace validity. This study is significant because it validates the evaluation indicators developed considering the desired outcomes of a medical school. These findings suggest that this approach can be effectively used to evaluate whether graduates have developed the competencies needed to serve as responsible, competent, and reflective physicians in the healthcare system.
Acknowledgements
None.
Funding
This research was supported by the 2023 Academic Research Project Grant from the College of Medicine, Inje University.
Conflicts of interest
No potential conflict of interest relevant to this article was reported.
Author contributions
Minkyung Oh conceptualized the study, collected the data, wrote the draft of the manuscript, and approved the final version.
Fig. 1.The Integrated Program Evaluation Model Adapted from the CIPP Model and Kirkpatrick’s Four Level, Inje University
Table 1.General Characteristics of the Participants
Table 1.
|
Characteristic |
Value |
|
Gender |
|
|
Male |
175 (84.13) |
|
Female |
33 (15.87) |
|
Position |
|
|
Academic |
55 (26.44) |
|
Private or hospital-based clinical practice |
153 (73.56) |
|
Period after graduation |
25.38±8.24 (26.00, 5.00–39.00) |
Table 2.Reliability for Indicators
Table 2.
|
No. |
Indicators |
Cronbach’s α if item deleted |
Cronbach’s α |
Cronbach’s α if item deleted |
Cronbach’s α |
|
1 |
I collaborate effectively with physician colleagues in clinical settings. |
0.8697 |
0.8831 |
0.8810 |
0.8925 |
|
2 |
I engage in interprofessional collaboration with physicians and other healthcare professionals to deliver patient-centered care. |
0.8748 |
|
0.8862 |
|
|
3 |
I complete medical records in a timely and accurate manner. |
0.8765 |
|
0.8865 |
|
|
4 |
I communicate well with my colleague doctors in my job. |
0.8674 |
|
0.8778 |
|
|
5 |
I communicate well with other professionals I work with. |
0.8702 |
|
0.8801 |
|
|
6 |
I explain and educate patients about the diagnosis and treatment process. |
0.8773 |
|
0.8864 |
|
|
7 |
I explain the diagnosis and treatment process to patients’ families or caregivers. |
0.8757 |
|
0.8851 |
|
|
8 |
I play a role as an educator who educates students. |
0.8925 |
|
|
|
|
9 |
I have a good understanding of medical ethics and am able to resolve ethical dilemmas when treating patients. |
0.8735 |
|
0.8837 |
|
|
10 |
I understand research ethics and apply them in practice. |
0.8784 |
|
0.8922 |
|
|
11 |
I treat patients within the current medical system or insurance system. |
0.8794 |
|
0.8891 |
|
|
12 |
I reflect on areas for improvement in my clinical practice. |
0.8749 |
|
0.8841 |
|
|
13 |
I search for literature with a high level of evidence and conduct self-directed learning. |
0.8769 |
|
0.8893 |
|
|
14 |
I make continuous efforts to stay up to date with current medical knowledge after board certification. |
0.8752 |
|
0.8866 |
|
|
15 |
I am good at resolving conflicts that arise within the team I work on. |
0.8733 |
|
0.8835 |
|
|
I believe that the following competencies are sufficient when providing medical care. |
|
0.8645 |
|
0.8645 |
|
16 |
Diagnostic planning and practice |
0.8156 |
|
0.8156 |
|
|
17 |
Clinical skills |
0.8361 |
|
0.8361 |
|
|
18 |
Emergency care |
0.8549 |
|
0.8549 |
|
|
19 |
Efforts to identify adverse events and minimize risks |
0.8042 |
|
0.8042 |
|
|
Total |
|
0.9110 |
|
0.9161 |
Table 3.Results of Factor Analysis for Indicators
Table 3.
|
Factors |
No. |
Indicators |
Factor 1 |
Factor 2 |
Factor 3 |
Factor 4 |
Factor 5 |
|
Collaboration |
1 |
I collaborate effectively with physician colleagues in clinical settings. |
0.8548 |
0.1860 |
0.1084 |
0.1091 |
0.0388 |
|
2 |
I engage in interprofessional collaboration with physicians and other healthcare professionals to deliver patient-centered care. |
0.8428 |
0.1669 |
0.0550 |
–0.0227 |
0.0908 |
|
5 |
I communicate well with other professionals I work with. |
0.7767 |
0.1294 |
0.2595 |
0.1126 |
0.1356 |
|
4 |
I communicate well with my colleague doctors in my job. |
0.7741 |
0.2201 |
0.2335 |
0.1970 |
0.1064 |
|
15 |
I am good at resolving conflicts that arise within the team I work on. |
0.4164 |
0.2659 |
0.3751 |
0.2523 |
0.1761 |
|
9 |
I have a good understanding of medical ethics and am able to resolve ethical dilemmas when treating patients. |
0.3708 |
0.3393 |
0.3462 |
0.1856 |
0.2781 |
|
Medical expert |
17 |
Clinical skills |
0.1549 |
0.8222 |
0.0348 |
0.1209 |
0.1229 |
|
19 |
Efforts to identify adverse events and minimize risks |
0.2145 |
0.7648 |
0.2605 |
0.2355 |
0.1399 |
|
16 |
Diagnostic planning and practice |
0.1105 |
0.7468 |
0.2879 |
0.2240 |
0.2079 |
|
18 |
Emergency care |
0.2576 |
0.7460 |
0.1299 |
0.1603 |
–0.0035 |
|
Patient-centered medical services |
6 |
I explain and educate patients about the diagnosis and treatment process. |
0.1607 |
0.1581 |
0.9048 |
0.0636 |
0.0862 |
|
7 |
I explain the diagnosis and treatment process to patients’ families or caregivers. |
0.1724 |
0.1820 |
0.8858 |
0.1289 |
0.0520 |
|
3 |
I complete medical records in a timely and accurate manner. |
0.3805 |
0.1099 |
0.4458 |
0.1905 |
0.2175 |
|
Professionalism |
13 |
I search for literature with a high level of evidence and conduct self-directed learning. |
0.0548 |
0.2924 |
0.0644 |
0.8417 |
0.2257 |
|
14 |
I make continuous efforts to stay up to date with current medical knowledge after board certification. |
0.1314 |
0.1521 |
0.2086 |
0.8405 |
0.1350 |
|
10 |
I understand research ethics and apply them in practice. |
0.2286 |
0.3760 |
0.2040 |
0.4893 |
–0.2953 |
|
Systematic treatment/self-development |
11 |
I treat patients within the current medical system or insurance system. |
0.2440 |
0.2544 |
0.1382 |
0.1336 |
0.6825 |
|
12 |
I reflect on areas for improvement in my clinical practice. |
0.3898 |
0.2089 |
0.1691 |
0.3226 |
0.5973 |
|
Eigen value |
|
|
7.8199 |
1.8398 |
1.5229 |
1.0841 |
1.0359 |
|
Variance (%) |
|
|
19.5 |
17.3 |
13.3 |
12.1 |
7.8 |
|
Cumulated total variance (%) |
|
|
19.5 |
36.8 |
50.1 |
62.2 |
70.0 |
Table 4.Validity of the Indicators
Table 4.
|
No. |
Indicators |
1st round |
2nd round |
Nea)
|
CVR |
|
1 |
I collaborate effectively with physician colleagues in clinical settings. |
5.00±0.00 |
5.00±0.00 |
5 |
1 |
|
2 |
I engage in interprofessional collaboration with physicians and other healthcare professionals to deliver patient-centered care. |
4.80±0.45 |
5.00±0.00 |
5 |
1 |
|
3 |
I complete medical records in a timely and accurate manner. |
4.80±0.45 |
5.00±0.00 |
5 |
1 |
|
4 |
I communicate well with my colleague doctors in my job. |
4.80±0.45 |
5.00±0.00 |
5 |
1 |
|
5 |
I communicate well with other professionals I work with. |
4.80±0.45 |
5.00±0.00 |
5 |
1 |
|
6 |
I explain and educate patients about the diagnosis and treatment process. |
5.00±0.00 |
5.00±0.00 |
5 |
1 |
|
7 |
I explain the diagnosis and treatment process to patients’ families or caregivers |
5.00±0.00 |
5.00±0.00 |
5 |
1 |
|
8 |
I play a role as an educator who educates students. |
4.80±0.45 |
5.00±0.00 |
5 |
1 |
|
9 |
I understand research ethics and apply them in practice. |
4.80±0.45 |
5.00±0.00 |
5 |
1 |
|
10 |
I search for literature with a high level of evidence and conduct self-directed learning. |
4.80±0.45 |
4.80±0.45 |
5 |
1 |
|
11 |
I treat patients within the current medical system or insurance system. |
4.20±0.84 |
4.60±0.55 |
5 |
1 |
|
12 |
I reflect on my shortcomings during my clinical practice. |
4.80±0.45 |
5.00±0.00 |
4 |
1 |
|
13 |
I do self-directed learning by seeking out literature with a high level of evidence. |
4.80±0.45 |
5.00±0.00 |
5 |
1 |
|
14 |
I make continuous efforts to stay up to date with current medical knowledge after board certification. |
4.80±0.45 |
5.00±0.00 |
5 |
1 |
|
15 |
I am good at resolving conflicts that arise within the team I work on. |
4.60±0.55 |
5.00±0.00 |
5 |
1 |
|
I believe that the following competencies are sufficient when providing medical care. |
|
|
|
|
|
16 |
Diagnostic planning and practice |
4.60±0.89 |
5.00±0.00 |
5 |
1 |
|
17 |
Clinical skills |
4.40±0.89 |
5.00±0.00 |
5 |
1 |
|
18 |
Emergency care |
4.60±0.89 |
5.00±0.00 |
5 |
1 |
|
19 |
Efforts to identify adverse events and minimize risks |
4.60±0.89 |
5.00±0.00 |
5 |
1 |
References
- 1. Harden RM. AMEE Guide No. 21: curriculum mapping: a tool for transparent and authentic teaching and learning. Med Teach. 2001;23(2):123-137.
- 2. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638-645.
- 3. Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach. 2007;29(7):648-654.
- 4. Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287(2):226-235.
- 5. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676-682.
- 6. Norcini JJ, McKinley DW. Assessment methods in medical education. Teach Teach Educ. 2007;23:239-250.
- 7. Frenk J, Chen L, Bhutta ZA, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010;376(9756):1923-1958.
- 8. Association of American Medical Colleges; Liaison Committee on Medical Education. Functions and structure of a medical school: standards for accreditation of medical education programs leading to the M.D. degree. Washington DC, USA: Liaison Committee on Medical Education; 2003
- 9. Ju H, Oh M, Lee JT, Yoon BY. Adapting an integrated program evaluation for promoting competency‐based medical education. Korean Med Educ Rev. 2021;23(1):56-67.
- 10. Oh M, Ju H, Yoon BY, Lee JT. Establishment of cohorts to evaluate the performance of students and graduates at a medical school. Korean Med Educ Rev. 2022;24(3):250-260.
- 11. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334.
- 12. Lawshe CH. A quantitative approach to content validity. Pers Psychol. 1975;28(4):563-575.
- 13. Ahn D; Working Group for Project on the Future Global Role of the Doctor in Health Care. Korean doctor’s role. J Korean Med Assoc. 2014;57:3-7. https://doi.org/doi.org/10.5124/jkma.2014.57.1.3
