Summative program evaluation of a Student-Led Seminar Series in the subject of physiology: an outcome-based study
Article information
Abstract
Purpose
Active involvement of students in their learning process is a recommended andragogical approach to increase student engagement. Many new teaching-learning strategies based on active learning have been implemented, but their efficacies to achieve the proposed benefits of attaining knowledge, skills, and attitude have not been evaluated, especially in the field of medical education. We substituted passive learning in the conventional tutorial classes with an active-learning strategy of Student-Led Seminar Series (SLSS) in the subject of physiology over 4 months and performed program evaluation for the SLSS.
Methods
Sixty-four first-year medical undergraduate students volunteered to participate, who were divided into groups to present seminars on the allocated topics under the guidance of a mentor. At the end of 4 months, program evaluation was done using Kirkpatrick’s model of evaluation—levels 1 and 2, which correspond to reaction and learning, respectively.
Results
Statistically significant improvement was observed in students’ satisfaction, and the self-perceived increase was observed in knowledge, skills, and attitude.
Conclusion
Program evaluation of SLSS not only established the significant impact of SLSS as an andragogical approach but also helped us in the improvisation of the program for the next cycle.
Introduction
Methods of teaching other than conventional methods of teaching have shown promising results with an increased level of intellectual performance shown by the students in recent times. Various studies have argued that traditional teaching methods lead to a lack of critical thinking and problem-solving skills which could be enhanced by teaching methods having active involvement of students [1].
The benefits of active learning outweigh those of being a passive recipient of learning instructions; therefore, educators must strive to incorporate activelearning strategies within the class to ascertain a grasping environment [2].
Consequently, many student-centric activities (programs) implemented so far have followed the evidence-based approach for active learning [3]. Very few studies have been conducted to evaluate the effectiveness of active learning strategies for improved outcomes [4,5]. Often, the evaluation of these learning strategies is based on student feedback which corresponds to level 1 of Kirkpatrick’s model of evaluation, while there are limited studies that evaluate change in learning, behavior, or results (corresponding to levels 2, 3, and 4 of the Kirkpatrick’s model, respectively) [6].
Further, it is also known that evaluating a technique only based on post-session feedback leaves little scope for quality improvement. Hence, evaluating the impact and effectiveness of any novel approach in the teaching-learning methodology is essential to identify its strengths and weaknesses, so that the necessary improvements can be made [7].
The present study was focused on examining the Student-Led Seminar Series (SLSS) as an educational intervention which inherently, by design, warrants active involvement of the targeted students. The basic premise for initiating such activity at Padmashree Dr D Y Patil Medical College was the dismal trend in students’ attendance in the conventional tutorial classes held during the previous academic year, which employed a passive teaching approach. Therefore, it was decided to substitute passive teaching tutorials with SLSS to promote the active-learning andragogical approach.
Additionally, it was proposed that the effectiveness of SLSS should be assessed by program evaluation using Kirkpatrick’s model of evaluation. Program evaluation in educational settings determines whether the program fulfills its proposed goals and objectives. Thus, the aim of our study was to determine the level of student satisfaction with SLSS and students’ perceptions of the role of SLSS in the acquisition of relevant knowledge, generic skills, and attitudes. Specific evaluation questions that were addressed during program evaluation are: (1) Do students feel satisfied with the SLSS? and (2) Does SLSS help in the self-perceived acquisition of knowledge, skills, and attitude?
Methods
1. Ethics statement
Ethical approval was sought from the Institutional Ethics Committee of Padmashree Dr D Y Patil Medical College (PDYPMC/Ethics/May-06/2016). All the class students were provided with a participant information sheet and informed consent form.
2. Study design and participants
This educational research was conducted at the Department of Physiology, Padmashree Dr. D Y Patil Medical College, Navi Mumbai, India over a period of 8 months (4 months for program administration, 4 months for program evaluation using Kirkpatrick’s model). Out of a batch of 150 first-year medical undergraduates (MBBS [Bachelor of Medicine, Bachelor of Surgery] students), 64 volunteers participated in the study. Other students (non-participants) only attended the seminar series that was presented by the participating students.
3. Educational intervention: Student-Led Seminar Series
This study for evaluating the efficiency of SLSS was conducted along with the progression of the curriculum when a few topics were already covered. An announcement was made in the class regarding SLSS; the topics were announced, and students were encouraged to participate voluntarily. Using audio-visual aids for the presentation was at the discretion of the presenting student. One faculty was designated as a mentor for the participating students.
The whole batch (n=150) was divided into three batches of 50 students each. The seminars were scheduled on Tuesdays, Thursdays, and Saturdays, with one batch presenting each day. Topics from nine main systems were considered for SLSS. Each system was divided into sub-topics and each sub-topic was presented by a group of 7–8 students; in the case of more than eight volunteer students, two groups were formed. All groups presented their seminar after discussion with the designated mentor and each topic was further subdivided within the group. Other students of the batch who did not volunteer for SLSS attended the seminar presentation with all other faculty members. A tabular representation of the implementation of SLSS is given in Table 1.
Tabular Representation of the Implementation of SLSS
4. Program evaluation
The SLSS program was evaluated using levels 1 and 2 of Kirkpatrick’s model of evaluation which is a four-step evaluation model for evaluating and improving the quality and efficiency of the implemented educational programs [8]. Various methods are employed to evaluate the four levels viz: level 1 (reaction)—such as using a questionnaire survey immediately after attendance; level 2 (learning)—such as evaluation of learners’ achievement by written tests or reports; level 3 (behavior)—evaluation of behavior change, such as through interview with students; level 4 (results)—using an index of four grades for evaluating the degree of improvement in learners and workplace performance after taking a training course [6].
In our study, level 1 was evaluated using the administration of a 5-point Likert questionnaire to evaluate the students’ satisfaction with the SLSS [9] (Table 2). Level 2 was evaluated using a 5-point Likert questionnaire regarding the self-perceived acquisition of knowledge, skills, and attitude. Both the questionnaires had responses ranging from “strongly disagree” marked as 1 to “strongly agree” marked as 5 [10] (Table 3).
The authors decided to consider the outcome as favorable for both the questionnaires if at least 75% of the students respond positively to the elements in the questionnaire i.e., Likert score ≥4.
5. Statistical analysis
Frequency distribution of responses to different questions related to satisfaction with the SLSS, and self-perceived increase in knowledge, generic skills, and attitude were obtained. A null hypothesis of 50% correct response (p=0.5) to each question was considered and tested using the exact binomial test. The p-values corresponding to each question indicated rejection of the null hypothesis, showing that the proportion of correctness of responses was significantly higher. All analyses were performed using IBM SPSS ver. 20.0 (IBM Corp, Armonk, USA) and the statistical significance was tested at a 5% level.
Results
1. Students’ responses to different questions related to satisfaction with SLSS
Descriptive statistics for responses of participating students to questionnaire 1 are depicted in Table 4. The students’ responses related to satisfaction with SLSS showed that the mean score for question 1 was found to be <4 (3.89±0.87), and for the remaining questions (question no. 2 to 9), the mean scores were found to be >4. This suggested a good level of agreement among students towards the questions.
Descriptive Statistics for Responses of Participating Students to Different Questions Related to Student Satisfaction with Seminar (Questionnaire 1)
2. Student’s response to a self-perceived increase in knowledge, generic skills, and attitude
Descriptive statistics for responses of participating students to questionnaire 2 are depicted in Table 5. In the case of a self-perceived increase in knowledge, generic skills, and attitude, the overall mean score across questions was 4.30±0.10, suggesting that the students agreed with the benefits of SLSS. The questions were categorized according to knowledge, skills, and attitude. The mean score for knowledge-related questions was 4.28±0.1, for skills questions was 4.25±0.07 and for attitude was 4.36±0.06. Overall, there was good agreement among students regarding the benefits of SLSS.
Descriptive Statistics for Responses of Participating Students to Different Questions Related to Self-Perceived Increase in Knowledge, Generic Skills, and Attitude (Questionnaire 2)
Discussion
Herein, this study we had implemented SLSS to increase the students’ engagement in their studies. The participating students used different ideas other than the conventional PowerPoint presentation to present their seminars. One group of students used flashcards to denote various components of the cardiac conducting system while explaining each component. Another group depicted the cardiac cycle in a very explanatory manner by using the idea of a human formation to show the atria and the ventricles. Two students held a red-colored cloth to depict the blood flow and two students represented valves while clapping of hands depicted the valvular function. To explain the mechanism of breathing, one group of students made a model using a rubber sheet to depict the diaphragm movement, besides other materials like thermocol and cardboard. Many students employed various other novel audio-visual mediums like Prezi presentation software and animations for making presentations. The results of the program evaluation conducted at the end of the seminar series were reflective of the success of the SLSS.
Our findings of level 1 of Kirkpatrick’s model showed that the students found SLSS useful in improving the knowledge and problem-solving skills of the students. These seminars were found to be enjoyable, provided better understanding of the subject to the students and students were more attentive towards this mode of study. Similar to our findings, a study conducted at the Department of Community Medicine, Jawaharlal Institute of Postgraduate Medical Education and Research, Puducherry assessed the effect of implementing the “student-centered learning” methodology by the reaction level of Kirkpatrick’s training evaluation model using a rating scale [9]. The majority of students found the sessions on innovative teaching-learning and assessment techniques enjoyable, useful, and informative. Also, students actively participated in curriculum planning, execution, and evaluation. Likewise, another study evaluated the effectiveness of modified seminars as a teaching-learning method in Pharmacology through feedback for traditional seminars, so that new interventions in the form of modified seminars can be introduced and compared. Participants of this study opined that interaction was the most important component of a seminar which was also in line with the findings of our study [2].
A few studies have evaluated student-led seminars emphasizing the active-learning component. In our study also, students agreed to the fact that their active participation in their own learning, process helped them to develop better information processing and analytical skills. Students confirmed in their replies to the questionnaire that they became a self-directed learner and that this form of learning improved their decision-making skills. A similar study was conducted to investigate the effect of active learning among first-semester general biology students by implementing a flipped active-learning model in the course. It was observed that flipped active-learning pedagogy was a more effective teaching method as compared to traditional didactic methods for teaching general biology concepts [11].
Our findings of level 2 of Kirkpatrcik’s model showed that students who participated in SLSS confirmed that this mode of learning helped them to voice their ideas effectively to the group and increased their acceptance towards ideas/views presented within the group. While there are very few similar studies which evaluated the SLSS beyond level 1 of Kirkpatrick’s model, we found an interesting study in concordance to our study which evaluated a pedagogical intervention, the student-led tutorial (SLT), using a mixed-methods approach for the data collection. This was implemented within a first-year undergraduate university module based on the concept of tutor-less tutorials and presents a platform for student learning which was designed to enhance active learning. The findings demonstrated that SLTs can potentially facilitate active learning and aids comprehension and understanding. Students particularly value the social aspect of the SLTs, which enables extended peer-to-peer interaction. The results suggested that students develop a sense of responsibility and ownership of their learning [12].
Similar to our findings of level 2 of Kirkpatrick’s model, a study conducted for assessing the perception of the medical students about seminars in the curriculum compared to conventional teaching-learning activity showed seminars to be more effective in a deep understanding of the subject along with improving self-confidence and communication skills among the students [13].
The present study had few limitations. One limitation was that no control group was considered to compare the satisfaction of students who attended SLSS. Since, we assessed the satisfaction of students who attended the seminar; the control group who did not attend the seminar would have given no relevant information for comparison as assessment was based on satisfaction/reaction of the student after attending SLSS, so we did not take any control group. Similarly, while assessing the self-perceived increase in knowledge after attending SLSS, a control group that did not attend the seminar would not have added any relevant information or comparison. Had we followed these students up to their second year and documented their attendance, then this could have been a level 3 evaluation as per Kirkpatrick’s model, which registers a change in behavior.
In conclusion, it is important to conduct a program evaluation of the new teaching-learning strategy so that feedback can assist the stakeholders to modify and accommodate the necessary changes. The program evaluation of SLSS established the impact of the implementation of active learning strategies in increasing the engagement of students in their learning process. It also resulted in the incorporation of the lessons learned to improve the quality of the teaching-learning strategy in the next cycle. It was decided that SLSS should start at the beginning of the next academic year and should continue throughout the year, unlike for a short period as in our study. A sensitization and orientation programme for the students can be conducted at the beginning of the academic year to increase the participation of students. It is worthwhile to plan the evaluation of level 3 by following up with the same set of students in their next academic year.
Acknowledgements
None.
Notes
Funding
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Conflicts of interest
No potential conflict of interest relevant to this article was reported.
Author contributions
Conceptualization: VV, PC, SM; data curation: VV; methodology, formal analysis, validation: VV, PC; project administration: VV, PC, SM; funding acquisition: not applicable; and writing–review & editing: VV, PC, SM.