The design , delivery and evaluation of an essential teaching skills course for preceptors in family medicine

Objectives: This paper outlines the design, delivery and evaluation of a hybrid face-to-face/online Essential Teaching Skills for Preceptors in Family Medicine course. Methods: Eighty-six preceptors attended one of ten fourhour Essential Teaching Skills for Preceptors course workshops. Data were collected through post-workshop evaluation surveys, post-module online evaluation surveys, postworkshop focus groups, and a final online reflective exercise. Focus groups were audiotaped, transcribed and analyzed by grouping common codes together to form themes. Evaluation surveys were analyzed using descriptive statistics and response frequencies. Results: A total of 79% (68/86) workshop participants completed the post-workshop evaluation survey. However, only 36% (31/86) of workshop participants completed the online modules and online evaluation surveys. Preceptors' responses from the focus groups and open-ended questions on the workshop evaluation survey emerged into seven themes: Sharing, Content, Support, Learning, Back to Basics, Course Facilitators, and Improvements. Conclusions: Faculty appreciated that the course development team addressed their expressed desire to become better teachers and offered ‘in-house’ faculty development. Low participation in the online modules indicated that preceptors preferred the face-to-face workshop. It is our expectation that other family medicine departments, as well as faculties of medicine, can benefit from our experiences designing and teaching the Essential Teaching Skills course as well as from using or adapting the ready-made workshop participant guide.


Introduction
The Department of Family Medicine (DFM) at the University of Ottawa (U of O) offers medical training at both the undergraduate (UG) and postgraduate (PG) levels.In recent years, the DFM has experienced rapid expansion and recruited increased numbers of family medicine educators (preceptors) to accommodate this reality.This in turn has produced a need for effective faculty development (FD) opportunities for both new and experienced preceptors within the faculty.In addition, decentralization of teaching from strictly urban academic teaching centers to include urban community and rural-based teaching environments have diversified the faculty profile.Concomitantly, medical education in Canada has seen a marked increase in the diversity of its PG learners with the acceptance of International Medical Graduates (IMG) into residency programs.These learners bring a wealth of knowledge and experience from their previous training and practice environments; however, their presence highlights cultural and clinical challenges requiring increasingly skilled faculty to support them to reach competency.
Expansion, increasing demands on human resources, diversity in learner populations, and new curriculum, evaluation and accreditation demands by the College of Family Physicians of Canada (CFPC) point to the need for faculty development offerings at the DFM to ensure program and teaching quality, and curriculum consistency Correspondence: Colla J. MacDonald, Faculty of Education, University of Ottawa, Ottawa, Canada, Email: cjmacdon@uottawa.caamong teaching sites.This means in addition to finding FD opportunities from the Faculty of Medicine, The College of Family Physicians of Canada (CFPC), and the broader hospital continuing medical education (CME), the DFM is responsible to provide FD opportunities to support their preceptors in being effective teachers.In 2010, the DFM at the U of O hired a Director of Faculty Development (DFD) to help elucidate a clear department-wide FD vision, cultivate a culture of on-going medical education and provide strong leadership.
The new DFD began her mandate with informal meetings with the DFM Program Directors (PG, UG Education and Evaluation), her predecessor in Faculty Development and her counterpart at the Faculty of Medicine.Needs analysis visits were organized at the seven main academic teaching units to poll teaching staff on "how best to support faculty in their role as teachers."The two needs most frequently voiced by preceptors at these informal meetings were "I want to be a better teacher", and "we want more convenient 'in-house' faculty development."In response, the FD team embarked on the creation of "Essential Teaching Skills for Preceptors" (ETS) to address preceptors' teaching needs.
The purpose of this paper is twofold.First, to outline the six-month team approach taken to design and deliver this College of Family Physicians of Canada (CFPC) accredited hybrid (face-to-face and online components) course (available in French and English).Second, to present the findings from a comprehensive evaluation of the ETS course, which incorporates mixed methods, in order to capture the complexities of the program. 1

Course development team
A focus group, facilitated by the DFD, an expert in curriculum design, program evaluation and eLearning, was held in May 2011.The purpose of the focus group was to identify what preceptors and residents felt should be included in an ETS course.Equipped with this information, the DFD invited a very experienced DFM clinician preceptor to become a member of the course development team as the subject matter expert (SME) and the initial instructor for the ETS course.
The course development team also included a Ph.D. student with an extensive background in technology and instructional design for face-to-face and technology enabled courses, and a Ph.D. healthcare researcher with a background in curriculum design, program evaluation and eLearning.Weekly team meetings were organized over a two month design process.Additional team members were recruited at crucial points along the project timeline (DFM Physician Lead for Faculty Development, a web-designer, both the DFM Manager and Coordinator of Special Projects and Faculty Support).

Course development
During focus groups, preceptors reported they preferred face-to-face workshops.Four hours was deemed long enough to cover a significant amount of material and for making traveling to the face-to-face workshop worthwhile, while still allowing participants to return to their clinical settings for half a day.The broad course content themes were chosen based on the prior needs assessment, information from the focus group and SME recommendations.These themes included an orientation to the new DFM postgraduate curriculum conceptual framework, based on the CFPC's Triple C competency-based curriculum.In addition, the workshop was geared to review the fundamentals around (1) direct observation, (2) providing feedback, and (3) principles of evaluation of a medical student or resident -see Appendix A for an outline of the ETS course.The face-to-face workshop was complemented by three (thirty-minute) online modules providing additional information on: Being an Effective Preceptor, Learning Plans, and Learners in Difficulty.The purpose of the online modules was to provide faculty with further opportunities to solidify their knowledge acquisition and provide access to online resources for ongoing support and self-directed learning.Development of the website containing the online modules was outsourced to a private company so learner activity could be tracked to meet the criteria for CFPC credit.A final reflective exercise relating to the course as a whole was also included to meet CFPC accreditation requirements.Preceptors were required to complete the three online modules within 30 days following the face-toface workshop.

Pilot ETS
In an attempt to promote buy-in and participation, the FD team began an active promotional campaign positioning ETS as a relevant department-wide faculty development resource.A train-the-trainer approach was adopted with the idea that preceptors representing each of the DFM teaching sites would first attend the pilot ETS course, then participate in a train-the-trainer workshop to support them teaching an ETS course, and finally facilitate the same ETS course at their own site.The pilot ETS workshop was offered on October 6, 2011.
Time was scheduled immediately following the workshop for the design team to debrief.The completed evaluation forms were reviewed and following a team discussion, minor revisions were made to the course.ETS was then translated into French to accommodate the needs of the Francophone community.

Train-the-trainer model
The 'train-the-trainer' model has proven successful as a strategy for disseminating FD across multiple sites. 2,3The purpose of the train-the-trainer workshop was to provide support to future course facilitators and ensure they felt prepared and comfortable with the related material and technology.Training other preceptors to teach the ETS course allowed the DFM to meet the preceptors' request to offer convenient in house FD.Offering multiple in house sessions radically increased participation in the course and contributed to meeting the requirements of the CFPC FD accreditation standards.
The train-the-trainer workshop was offered on October 27, 2011 and all 12 preceptors who attended the ETS pilot returned to participate.The train-the-trainer facilitator was also the facilitator from the pilot.She used the information from the initial course debrief and her own reflections to create a list of 'hints and tips' for future ETS facilitators, and identified key reference articles to provide background information to assist with teaching the course.A USB memory stick was prepared for each of the ETS champions with the course reference material, the workshop Power-Point presentation, and participant student guide.To eliminate the necessity for Internet access while delivering the workshop, permission was obtained to embed the presentation videos directly into the PowerPoint slide deck.Finally, a series of archived video excerpts from the initial pilot course offering were included in the facilitator package, so that new instructors could review how to present any of the material with which they felt unfamiliar.

Participants
The Ottawa Hospital Ethics Review Board reviewed the study and determined it to be exempt.Twelve preceptors representing seven teaching units and the community sites registered and attended the face-to-face pilot workshop.Following the pilot with 12 participants representing all of the DFM teaching sites, nine additional workshops (with a total of 86 participants) were delivered over a five-month period in eight different teaching settings.One of the workshops was conducted in French as shown in Table 1.Nine of the twelve preceptors who participated in the pilot and train-the-trainer sessions facilitated one or more of the additional nine workshops.

Study design
A mixed methods approach was used to evaluate the ETS training program for the reason of combining the strengths of both qualitative and quantitative studies. 4In this mixed methods approach, quantitative and qualitative data were collected concurrently to obtain a full understanding of the program.6][7] The mixed methods approach was used as the ETS training modules were established from a complex real-world practice; and complex problems require complex methods.In this study mixed methods were useful to triangulate the data from the evaluation surveys with the data from the focus groups.In addition, the focus groups enabled us to obtain rich data and delve deeper into the issues that emerged from the survey questions.

Data collection
A comprehensive evaluation of ETS included the collection of both quantitative and qualitative data.Data were collected in the form of post-workshop evaluation surveys distributed in person to a total of 86 preceptors who participated in the ten (four-hour) workshops; post-online module evaluation surveys, two focus groups held with a total of 16 preceptors following one of the four-hour workshops, and a final online reflective exercise.Focus groups were audiotaped, transcribed and analyzed by grouping common codes together to form themes. Surveys were analyzed using descriptive statistics and response frequencies.

Procedure
Following each of the ten (four hour) workshops, preceptors were invited to complete a workshop evaluation.The evaluation consisted of several demographic questions, a series of questions related to the curricular domains and CanMEDS-FM of the College of Family Physicians of Canada Red Book, 8 ten items related to course content, delivery, service and outcomes on a rating scale questions, 9 and two open ended questions.The evaluation survey was developed by the course development team and revised after the ETS pilot based on how the questions were interpreted by the participants.These two initial processes contributed to the content and response process evidence of modern test validity. 10The final version was reduced to eight Likert-type items (on a four point scale ranging from strongly disagree to strongly agree) and three open ended questions: (1) How well did the teaching session enhance your learning needs?
(2) What was the most valuable aspect of the teaching session?(3) How could the teaching session be improved and/or what should be added?Following the workshop preceptors were invited to complete three online modules.Preceptors were asked to complete a two-minute evaluation survey following each module identifying their perceptions of the content and learning outcomes.In addition to providing feedback, the evaluations monitored that the online program had been completed in order to satisfy CFPC requirements for professional development (PD) credits.
Finally preceptors were invited to complete a final reflective exercise.Furthermore, two focus groups were held following one of the workshops to obtain in-depth information regarding preceptors' experiences with the content, delivery, service and outcomes of the workshop that would supplement the findings from the evaluation surveys.Two focus groups each with eight community-based physicians were held simultaneously immediately following one of the workshops.Each focus group lasted approximately fifty minutes (see Appendix B for interview protocol).

Final reflection
Participants were asked to complete a final reflection of their experience in the ETS course three months following the completion of the fourth and final online module.The final reflection provided additional information and also was necessary to meet the requirements of the CFPC for PD credits.

Qualitative analysis
Each focus group was audiotaped then transcribed verbatim.The researchers reviewed the transcripts and performed open coding of the text by hand.After a preliminary list of codes was developed, the transcripts were coded a second time.The findings from the open-ended workshop evaluations were then added to the focus group findings.The coding process consisted of grouping the common codes together to form themes.The coding was reviewed several times to ensure that no new themes emerged from the data.

Quantitative analysis
Workshop evaluations and online module evaluations were transferred into Microsoft Excel worksheets.Data were analyzed using descriptive statistics and response frequencies.These results were then combined with the themes derived from the qualitative analyses.

Results
A total of 79% (68/86) workshop participants completed the post-workshop survey.However, only 36% (31/86) of workshop participants completed the online modules and online surveys, and fewer still --19% (16/86) --completed the final post-program reflection activity.Data from the post-workshop surveys, post-module online surveys, final reflective exercises and focus groups have been combined or integrated through triangulation.
Feedback from the workshops was overwhelmingly positive.Responses on the scale questions ranged from 96% who felt the length of the workshop was appropriate to 100% for most of the other items (see Table 2).

Table 2. Workshop responses
Although only a small percentage of the workshop participants completed the online module evaluations, those who did really enjoyed all three of the modules (see Appendix C).Participants enjoyed reading the examples of how to build a learning plan and learned the importance of stating and sharing the learning objectives with learners and having a plan to reach these objectives.They also felt the supplemental resources, including relevant web sites, were very good.However, they did find downloading the Academic Support Plan (which is an online DFM teaching resource) to be slow and some of the links not to be active.Providing consistent content (slides and teaching manual) and consistent training to the facilitators appeared to affect the quality of the program.As shown in Findings from the focus groups and open-ended workshop evaluations indicated that preceptors were appreciative for having the opportunity to attend the ETS workshop.When asked about their experience with the workshop, preceptors' responses emerged into the following seven themes: Sharing, Content, Support, Learning, Back to Basics, Course Facilitators, and Improvements.

Sharing
Preceptors from both focus groups reported they benefited from sharing thoughts and ideas with other preceptors.One preceptor stated: "I thought the most useful part was the interaction with the other participants." One preceptor talked about the benefits of having the time to share ideas with "I feel it is one collective pool of mind that you never have access to when you are all by your lonesome".

Another preceptor stated,
"For community doctors … This is our only chance to really bounce ideas off one another.Find out if we are doing it right." The most common comment appearing in the open-ended questions from the workshop evaluations was that preceptors enjoyed the discussions with their colleagues and sharing of personal experiences.Moreover, in the postworkshop survey, 90% of the preceptors reported the facilitators actively encouraged questions and participant interaction during the workshops.

Content
Preceptors reported they found the content in the ETS workshop relevant.One preceptor summarized his impression of the content.

"This comes directly from what the preceptors have asked for. It
[ETS] is definitely ear to the ground, tire to the pavement type stuff."

Another preceptor shared.
"There is a lot of information.It is well organized and the discussion is always useful." Preceptors said they valued the DFM's effort in offering a 'common course' for all preceptors in an effort to standardize the learning experience for FM residents.
"I like the effort the department is taking to really formalize the education feedback and evaluation for residents.By standardizing how things are done you will improve the quality overall." For the most part, preceptors stated that the ETS content was a review or confirmation of what this experienced group already knew.

"The actual content of the workshop were things I had heard. In the future it would be helpful to have more concrete examples … exactly what words I can use that won't offend people when giving feedback and how to actually label behaviors."
This theme was also very prevalent in the online module evaluations.Many commented that the content was a good review.The preceptors' comments regarding the course content corroborated the data in the workshop evaluations as 62% felt the balance of theory and clinical practice was appropriate and overwhelmingly, 86% felt the knowledge gained during the session would be useful in their clinical practices.However, in the open-ended responses, many preceptors reported they wanted more role-playing and practice providing feedback.

Support
The community preceptors indicated they sometimes feel isolated and are not always confident they are providing residents with an optimal learning experience.A preceptor shared that hearing other preceptors talk about their problems made him feel supported: "It made me feel much more supported in my role as a teacher which is a role that I'm somewhat insecure about.Sharing my ideas I realized that we are all fairly similar in ideas and abilities." Preceptors reported they benefitted from the supportive resources that were made available during the ETS workshop.
"I enjoyed the fact that we got some practical resources.It wasn't just the discussion." One preceptor shared that he enjoyed being exposed to the online resource PracticalDoc. 11Another preceptor communicated, "… being oriented to the DFM website and what things are available on it.…There are a lot of resources there." Furthermore, from the workshop evaluations, over 71% of the preceptors strongly agreed that relevant resources and references were provided.

Learning
Preceptors appreciated the DFM offering them practical ideas that were grounded in theories of teaching and pedagogy.One preceptor commented: "The group had a few hundred years of teaching experience.So just the exchange of experiences alone is learning ..

. now there is actually a reason behind why we do this. It is because it is shown in research that this is what works."
Several preceptors said they learned something new about evaluation during the workshop.One preceptor stated,

"I struggled with how to use it [the evaluation form]. It is clearer to me after this presentation."
Another preceptor stated the structure for providing feedback to residents presented was very helpful: "A framework about how to approach it.It's a quick and dirty approach and broadly applicable.That was very useful." Similarly, several preceptors commented on the coaching metaphor: "I also liked the cognitive framing for feedback.It is not to confirm that you are doing something well but to show you how to get to that next step.It creates a less threatening way to give someone constructive feedback." For one preceptor, one of the best lessons learned from the workshop was that it was okay to give a resident a '2' [out of 4: barely objectives] on their evaluation.
"I think it de-emphasized the worry of giving a '2' on an evaluation.I think that is one of the big fears everybody has.I think that was very helpful." When asked if they learned anything new, participants in one of the focus groups responded: "The resources were useful.Most of the material was not new but the resources were useful." A second response was, "RIME [Reporter, Interpreter, Manager, Educator] was definitely very useful to me." A third preceptor stated, "For me it was the level of functioning at various times throughout residency." Participants were also very satisfied with the learning content of the online modules.

Back to basics
When designing ETS, the subject matter expert was concerned about the content being too basic for her fellow experienced colleagues.The design team struggled to provide basic information for beginning preceptors and content that would challenge experienced preceptors.Ultimately, the decision to develop a workshop on the 'basics' was made in order to ensure consistency in teaching across learning environments.
As the ETS course design team suspected, one preceptor stated she felt the content in the four-hour workshop was basic.

"I've been teaching for twenty years. I thought I would get more than the basics of how to give feedback. … I wish the content was at a deeper level."
A second experienced preceptor saw the situation differently: "I also have attended a lot of these [workshops].I think the objective was to get 'the basics'.There are a lot of people who have not attended any [professional development] so at least everyone is now at this level . . .I just think part of the problem is we should have had this [ETS] twenty years ago."

Course facilitators
Preceptors were in agreement that course facilitators were knowledgeable and set the right atmosphere for learning.When asked if the facilitators were approachable and knowledgeable, one preceptor responded: "I really liked the atmosphere and I think they made it very comfortable.I found it a very good environment to learn." A second preceptor stated, "What was really good was the fact that they used anecdotes.I like the fact they went off on tangents a couple times because that was just as important as anything else." Many expressed in the open-ended evaluation questions that the facilitators were excellent, enthusiastic and knowledgeable presenters.One hundred percent of the respondents either agreed or strongly agreed with the statement that the presenter was engaging.

Improvements
When asked how the course could be improved, one preceptor suggested, "You could make it [ETS] beefier." Another recommended sending a reminder email the day before with the room number.
"I lose track of these workshops." One preceptor suggested using an iPad to deliver the course participant guide as opposed to a paper manual.

Discussion
This paper outlines the design, delivery and evaluation of a hybrid face-to-face/online Essential Teaching Skills for Preceptors in Family Medicine course.Lessons learned highlight the importance of curriculum design to support effective curriculum delivery and quality outcomes.As evidenced by the evaluation findings, the success of the course can be attributed to first conducting a needs analysis and then listening to and addressing preceptors' needs and concerns when making course design decisions, and finally building the right development team, as supported by Norman et al. 12 Specifically, preceptors appreciated the FD team addressing their expressed desire to become better teachers and offering 'in-house' faculty development.The fact that 86% of the DFM PG preceptors took the ETS course in a nine-month period is indicative that offering 'in house' training is an effective strategy to address a consistent FD concern 'getting bums in seats'.Another sign of success is interest by the Faculty of Medicine to adapt ETS for other Departments within the Faculty of Medicine.
Steinert 13 indicated in her report to the Association of Faculties of Medicine of Canada (AFMC) for the Future of Medical Education in Canada PG Project, faculty development needs to include learning from experience, from peers, mentors, and communities of practice.ETS exemplifies these forms of FD.5][16][17] A team approach utilizing team members' complementary knowledge and skills also contributed to the quality of ETS by making development relevant and effective. 18,19he evaluations from all ten ETS workshops were very positive and praised the overall organization of the course.Preceptors reported that having dedicated time to discuss teaching issues, challenges, and strategies with their colleagues as the most beneficial aspect of the course.In terms of becoming trainers themselves, several participants expressed some anxiety over teaching the course to their peers (mostly related to technical concerns with video which was reduced by embedding the videos into the slides).Many expressed a desire for a follow-up ETS 2 workshop.Despite the basic level of the ETS course, most preceptors felt it was beneficial to all preceptors regardless of their experience or skill level the richness of the experience came from the discussions between participants who shared their own stories, challenges and successes.
The online follow-up modules and reflection exercise were not well received by preceptors.The DFM hired a software company to design a learning management system for the online modules.There were technical problems with the system leading to a lack of automatic reminders prompting participants to complete their online modules and the reflective exercise.Some users who went directly online to complete their modules experienced difficulty trying to login to the system.
Technical difficulties were mitigated by email and phone support provided by the DFM Coordinator who sent out reminders to participants via email and also sent preceptors Microsoft Word copies of the reflective exercise to ensure participants met the requirements for PD credits.Technical issues and complications included preceptors not receiving the reminders to participate in the online modules and final reflection.In addition, some participants had difficulty logging on to the site containing the online modules.Although the online follow-up modules were developed in an attempt to provide preceptors with an innovative, convenient curriculum option, the technology turned into a barrier, reducing participation and data collection.Preceptors reported that they preferred the faceto-face workshop.Consequently, lack of participation and frustration with technology in the online modules in ETS 1 resulted in the decision not to include online modules when the DFM developed the ETS 2 course.Preceptors from the community and rural sites reported they appreciated the effort made to offer an ETS session especially for them.The Francophone community appreciated the efforts made to address their language needs.Preceptors also reported it was important to offer a standard course across teaching sites to add consistency to the delivery of the PG family medicine curriculum.They stated the facilitators were knowledgeable and approachable.In all ten offerings of the course, preceptors said they learned new ideas from their peers, appreciated the organization of the course and felt the course was relevant and useful.Preceptors consistently reported they found the resources in the course useful especially Internet-based resources such as the DFM website, and in particular, the FD webpage.Many said they would refer to the resources in the future and would share the resources with their residents and peers.
One of the most important benefits of ETS was that it provided a venue for preceptors to come together and share ideas, challenges and solutions surrounding the teaching of medical learners.The networking and team building that occurred as a result of ETS cannot be understated.ETS has paved the way for future FD offerings.One such initiative that emerged as a result of the findings from the ETS evaluation is 'Teaching Tips at your Fingertips', where preceptors describe their tried and true teaching tips in a YouTube video.The video links are emailed and tweeted to their colleagues in order to share best practices.
Since the design, delivery and evaluation of ETS 1, the DFM has developed ETS 2 on a topic identified by the preceptors in the evaluation of ETS 1, specifically 'ETS2: Learner in Difficulty'.Many of the lessons learned from ETS 1 were implemented into the experience designing, delivering and evaluation ETS 2. A team approach to design and a train-the-trainer approach to disseminate ETS 2 were again implemented.The online modules that were expensive to develop and not well used in ETS 1 were not developed in ETS 2. Instead, a participant guide in an electronic format and eBook for the iPad were used to facilitate the delivery of ETS 2.

Table 2 ,
the responses for all workshops were very positive.