Effectiveness of Tutor Observation on Faculty Development in Problem-Based Learning | BMC medical training


At NTUCM, APP was first integrated into the medical curriculum in 1993. APP course themes emphasize “humanity and society” for second-year medical students (M2) , “anatomy and physiology” for M3 students, and “pathology and pharmacology”. » for M4 students. In each PBL class, there are 8 to 11 students and a tutor to lead the course, which lasts 2 hours per week. Discussion topics are made up of several clinical scenarios with a predefined schedule provided by the course administrator at the start of each semester. For example, discussion scenarios for M2 students include professional standards, medical ethics, health insurance, laws and regulations, stigma, vulnerable people, etc. PBL cases include clear objectives, and paragraphs describing actual clinical scenarios with issues and references. Before each session, the student studies the relevant literature on his own according to the assignment and formulates his own knowledge necessary for problem solving. Students can share preparatory work before the PBL course, present it onsite with slides, or write it directly on the whiteboard. Some groups have class leaders during their PBL sessions, some don’t. Each student must express their own point of view on the learning objectives. Students would discuss their observations, ask questions, brainstorm, and come up with solutions to problems to develop teamwork and problem-solving skills. [10]. During the course, the facilitator guides the seven steps of APP [11]. Tutors promote the right path for PBL discussion by helping students define and analyze the problem by asking open-ended questions. Students are encouraged to formulate learning objectives, collect additional information, and synthesize and test newly acquired information. [12]. Over the years, student learning strategies for APP have shifted from offline research to online surveys. Between the students, the preparatory discussion takes place on social networks rather than in the form of face-to-face conversations as in previous years. The majority of PBL classes remain on campus courses which involve tutors.

study design

Recruitment of tutors

In the NTUCM, qualified PBL tutors are attending physicians from the National Taiwan University Hospital (NTUH) health system who volunteer to participate in the tutoring. PBL tutors from different specialist backgrounds join a coordinated FD program and learn to become facilitators. The classic FD course at NTUCM consists of workshops or seminars aimed at teaching and learning strategies for novice PBL tutors. Starting in 2018, NTUCM integrated tutor observation of PBL classes into the Novice Tutor FD.

Description of the DF course

The FD course for PBL tutors consisted of an FD workshop and tutor shadowing activities. The FD workshop consisted of a one-day program with two panels. In the first panel, there were several sessions based on lectures on educational theory, advances in medical education, and the professional development of educators. The second panel consisted of group discussions divided according to the class taught by the tutors. Topics included instructional and content training, APP facilitation skills, and assessment. During the tutor observation activities, each of the novice tutors joined 2 PBL discussions, which were randomly selected from the medical PBL courses. Novice tutors joined PBL classes only as observers. Prior to the observation activity, the observed tutor would present the steps of their group discussion to the observer. The observing tutor silently engaged in the observation exercise exploring the student-centred learning process and the role of facilitator tutors. At the end of the observation activity, the observer and the observed had the opportunity to reflect and give their opinion. They could exchange the practice of running their tutoring groups. Afterwards, the tutors were encouraged to share their impressions of the observation activity by e-mail or on the tutor forums.

Study procedure

This study used a before and after study design after a scale development process to explore the effectiveness of FD courses incorporating tutor observation activities [13]. This research consisted of three phases: Phase 1 was the development of TS scale items, Phase 2 was a validity review, and Phase 3 was an efficacy study.

Phase 1: literature review and item development

The process of developing the TS scale followed Hinkin’s recommendations [14]. A review of the literature on the PBLs and the pre-existing scales collected was carried out. Items were generated and modified based on extensive survey data.

Phase 2: Validity review

Validity of content by professional commentary

The TS Scale feedback questionnaire was submitted to experts in the field of medical education to ensure content validity. The experts checked the declarations and the correspondence between the expression and the design of the articles. Observation scale items for experts were rated on a 4-point Likert scale ranging from 1 to 4, with 4 being very appropriate, 3 being appropriate, 2 being inappropriate, and 1 being very inappropriate. The recommendations of the experts were therefore adopted during the revision of the questionnaire. Content validity indices (CVI) are reported at the item level (I-CVI) or at the scale level (S-CVI) [15, 16]. An I-CVI of 0.78 or more for three or more experts is considered to indicate acceptable content validity. An S-CVI of 0.80 or greater is considered reasonable.

Internal consistency

A group of novice PBL tutors were asked to complete the 32-item TS scale based on self-rated confidence before the FD activity. The reliability of the TS scale on internal consistency was tested by Cronbach’s alpha. A Cronbach’s alpha value greater than 0.8 was considered acceptable [17].

EFA construction validity

Regarding the validity analysis based on the theoretical construct, an exploratory factor analysis (EFA) was performed. A new training of the TS scale was validated and adapted according to the results of the EFA.

Phase 3: efficacy study

Qualified novice PBL tutors who participated in the training and completed the validated TS scale (24 items) with self-rated confidence before (pre-test) and after (post-test) the FD course were included for the to analyse.

statistical analyzes

A descriptive analysis of the characteristics of the participants was carried out. Cronbach’s alpha was calculated for the internal consistency of the scales in our sample. We obtained the Kaiser-Meyer-Olkin (KMO) index and performed Bartlett’s sphericity test to explore sampling and data adequacy. EFA using the maximum likelihood method and varimax rotation following recommended standards was performed [18]. Items with loading values ​​greater than 0.5 were retained. Changes in self-rated confidence before and after tutor observation with a paired t-test were compared. A p a value less than 0.05 was considered statistically significant. All statistical analyzes were performed using SPSS (Statistical Package for the Social Sciences) 20.0.

Sara H. Byrd