The aim of this review was to assess the current evidence regarding the efficacy of teaching skills programs for junior medical officers.  We aimed to compare and contrast these results with findings from previous literature reviews, the last of which were published in 2009, in order to provide up to date conclusions and recommendations regarding this topic

Medline, Embase, ERIC, CINAHL, and PsychINFO were searched from January 2008 to January 2015 by JD and HN. Grey literature, including that published by relevant medical bodies, was also utilised. Key words used were resident, registrar, teach, teacher, and education. Relevant papers were all those that had a focus of improving teaching skills of JMOs. Search results were screened by two authors JD and HN. The title and abstract of all results were screened. Full papers were then retrieved for further review if relevant. The references in these articles were also reviewed for further appropriate papers.

A similar search for systematic reviews on the topic without the timeframe limitation was performed to ensure that no studies were missed. Individual studies examined by the systematic reviews on the topic were also included in the analysis.

The following inclusion and exclusion criteria were developed for systematic reviews and individual studies.

Studies were included if participants were JMOs (i.e. not specialists/consultants). If they facilitated a structured teaching programme, detailing the teaching method and content. Included evaluation up to at least Level 2 of Kirkpatrick’s model. Non full text studies, not in English or those that included duplicated results were excluded.

A data extraction table was created in order to gather the required data for the review. Included studies were reviewed by JD and HN independently, and resulting data was synthesised. Study objective, context, method (study design, teaching intervention details, evaluation method), results (determined by Kirkpatrick’s levels of learning),²⁴study quality and study outcome were retrieved.

Following independent data collection tables were merged into one and any discrepancies were resolved by consensus. In cases where consensus was not reached, a third reviewer (AS) determined the outcome.

 Kirkpatrick’s model of evaluation²⁴ (Table 1) was used to categorise the outcomes reported by each of the included studies. The model outlines four progressively sophisticated levels of outcomes. Kirkpatrick’s model has existed for many years, is well known, logical, and also provides a convenient way to present outcome information.  It has been modified a number of times over the years and for the purpose of our review we have chosen to use the Best Evidence Medical Education (BEME) Collaboration²⁵ version subsequently adapted by Hill et al. 2009¹⁹ to facilitate comparison of the literature.

Overall risk of bias was assessed for each included study. Two authors JD and HN independently evaluated bias using the critical appraisal worksheets for systematic reviews and therapy questions provided by the Centre for Evidence Based Medicine.²⁶

Of the 12 individual studies included, all but one was based in North America. One was based in New Zealand,²⁷two in Canada^28,29 and the rest in the USA (Table 2). A wide variety of specialties were included. Nine papers focused on one specialty.^28-36 Five of which were Psychiatry.^28,30-32,36 Studies included a range of clinical experience levels from postgraduate years 1 to 5. All studies stated the number of participants except one. The total number of included participants over all studies was 1,659, the minimum was 11, maximum 479 and the median was 78.

All studies outlined an aim. Most JMO teaching skills programs were established due to the large amount of resident teaching documented in literature or via internal audits and a lack of available teaching skills opportunities for JMOs. The specificity of study aims varied. Many included secondary objectives such as assessing resident self-reported efficacy.

There were no randomised controlled trials (RCTs). Two were non randomised controlled trials^27,34 and the rest were uncontrolled trials. Only six studies included a pre intervention outcome measure.^{27, 29, 31, 32, 35, 37}

There was a variety of follow up times. Six studies stopped follow up immediately after collecting data post workshop. Five studies included follow up to or greater than one year.^27,34,36-38 The longest follow up was four years, the entire duration of the residency program.³³

Most program types detailed by the included studies were workshops. The shortest teaching program was 1.5 hours²⁹ and the longest was 1.5 days.²⁷ Seven included a single workshop as their intervention, whereas four delivered a series of workshops over time.

A variety of instructional methods were utilised. Most studies delivered content by mixed methods including role-play, brainstorming, reflection, and small-group discussion. Daniels-Brady et al.³⁰ used direct supervision whereby the program director would meet the student weekly in order to advise them. Pernar et al.³⁴ delivered content via emails on a weekly basis over the course of a year.

A variety of teaching content was delivered. Common topics included adult learning principles, feedback, reflection, curriculum orientation, and evaluation. Some authors utilised panels (including various stakeholders such as doctors and program directors) to design course content, while others utilised tools described in the literature. For example, Ostapchuck et al.³⁷ utilised the previously validated^39,40 Bringing Education and Service Together (BEST) curriculum (with some modifications). Daniels-Brady et al.³⁰ utilised one on one, face to face individualised approach. The program director gave advice on teaching, curriculum of educational resources such as journal articles and observed residents giving feedback. Pernar et al.³⁴ delivered brief statements (29 in total) via email of what constituted good teaching.

All studies included a survey as part of their evaluation.  Only two studies included alternative evaluation methods. For example, Ostapchuck et al.³⁷ included focus groups of medical students, and Ricciotti et al.³³ included a faculty assessed Observed Structured Teaching Evaluation (OSTE) (both videotaped and in person). All programs except one⁴¹ (who, instead, surveyed the students taught by them) sought the opinion of the JMOs. Faculty were also utilised to provide assessment of JMOs.

Individual studies published after 2008 showed that a range of Kirkpatrick levels were evaluated (Table 2). Ten studies reported level 1 outcomes, 11 reported level 2a outcomes and 10 level 2b, seven studies reported level 3 outcomes and four studies reported level 4a outcomes. One study reported level 4b outcomes. All studies reported positive findings. Five studies demonstrated positive outcomes in follow up more than or equal to 1 year.^{27,34, 36-38}

Studies published prior to 2008 demonstrated that Kirkpatrick outcome levels were distributed similarly. Sixteen studies reported level 1 outcomes, 18 reported level 2a outcomes and 18 level 2b, 21 studies reported level 3 outcomes and two studies reported level 4a, outcomes. One study reported level 4b outcomes and five studies reported findings that were not positive (Table 3).         

Individual studies published after 2008 showed substantial threats to their internal validity. Of the uncontrolled trials only five included pre and post intervention outcome measures and none of the non-randomised controlled trials did so. Some studies had small samples sizes. Six studies had more than 10% loss to follow up which was not consistently explained. Only two papers used validated (or variations of) surveys and objective outcome measures.^27,33 There was response bias present in some studies as participation was voluntary and thus likely to attract those who were keen to teach.

Five literature reviews of varying quality were included in the analysis. The study by Wamsley et al. 2004¹⁶ was the earliest detected review in this area. The review only searched Medline for studies between 1975 and 2003. Fourteen papers were included in the analysis. There was no bias risk assessment performed, and there was some heterogeneity in the results found.

Dewey and colleagues¹⁸ conducted a systematic review of trials with an interest in gathering information on how to develop teaching skills in psychiatry residents. They searched a number of databases and 13 studies were identified and analysed. Methodological quality and study characteristics were quantified, but they did not specify the period by which they searched.

Hill and colleagues¹⁹ performed a review to determine the characteristics of effective teaching skills programs. They searched a number of databases from 1971 and 2008, and performed a bias risk assessment of the 29 included studies.

Lacasse et al.²¹ conducted a systematic review of resident teaching with a focus on family medicine. They identified eight studies from 1950 to 2008 (they only searched two databases and have missed studies) that all included family medicine residents. They did not perform a bias risk assessment of included studies.

Post et al.²³ reported on 24 studies from 1975 to 2008 that looked at programs that were designed to improve resident teaching skills. Although they provided a descriptive review of these studies, they did not include an individualised risk to bias of the studies included in their review. They did quantify aspects of study design.

Studies published prior to 2008 of which were included in the five literature reviews demonstrated many threats to validity. That said the quality of these was generally more robust than that of the studies published after 2008. There were 12 RCTs. There were nine non-randomised controlled trials and 14 uncontrolled trials. The methodological quality of the RCTs varied. Randomisation methods, concealment of randomisation, group differences at baseline and blinding of raters were poorly reported and adjusted for, although this is difficult to do with RCTs of this topic. Only three studies had over 10% loss to follow up.^42-44 Only four studies did not utilise objective outcome measures.^44-47 Of the non-randomised controlled trials all included pre and post intervention outcome measures. Three used validated, reliable and objective outcome measures.^48-50 Seven of the uncontrolled trials included complete pre and post intervention outcome measures.^51-56 Six used validated and reliable outcome measures.^10,41,54-57 Seven studies utilised objective outcome measures.^{52, 54, 55, 57-59}

There were five systematic reviews identified through the search with a combined total of 39 studies that were analysed. Hill et al.¹⁹ analysed 29 of these studies. The six studies^{39,45,47,51,52,60} that they did not include have added further information although do not change the overall findings of their review. We therefore feel that the review by Hill and colleagues represent well the literature prior to 2008.

The individual analysis of all 39 studies demonstrates clearly that there is convincing evidence that supports the efficacy of JMO teaching skills courses in improving Kirkpatrick outcomes up to level 3. Six studies had a low risk of bias. The most well-constructed studies were those by Morrison et al.^39,40 who also generally reported positive outcomes up to and including level 3. One of the six studies⁵³ found results that may not be sustainable, which indicates the ongoing need for longer detailed follow up in this area. There were 12 randomised controlled trials of varying quality. When these studies where analysed as a group they reveal similar findings as discussed above.

Twelve studies published after 2008 were identified that reported outcomes regarding teaching skills programs for JMOs.  This group consists of studies with heterogeneous designs, programs, and outcome measures. All studies reveal useful insights, although it is difficult to quantify the effects of programs on teaching skills of JMOs. There is significant evidence that teaching skills programs have subjective positive effects on all Kirkpatrick outcome levels. Unfortunately, what is not present is strong evidence for objective positive effects for higher levels of outcomes (Kirkpatrick levels 2b, 3 or 4) and their effects over time. In this sense these studies have not been able to add more evidence to this important topic.

Literature reviews to date have aimed to find the aspects of a JMO teaching skills program that make it effective. To date this has proven difficult due to the heterogeneity of programs reported. The evidence so far suggests that looking for a common “formula” may be a flawed approach, but that the incorporation of individualised teaching skills initiatives into the JMO curriculum is likely a more effective approach. Detailing this process is beyond the scope of this article but the following are some suggestions.

Important components of curriculum design include a situational analysis, statements of intent (aims, objectives, outcomes), content, implementation and organisational strategies, assessment, and monitoring and evaluation all need to be considered (not necessarily in any particular order). One helpful guide is Susan Toohey’s (1999) “course design process”⁶¹ that suggests a sequence of events for the design process. Adult learning principles should be considered at all points during the design process.⁶² Finally, the Kirkpatrick’s model of evaluation is a good way to conceptualise, design and report evaluation. Evaluation methods, ideally, need to make use of valid and reliable outcome measures such as the Personal Teacher Identity Scale, OSTE, Clinical Teaching Assessment Form, Stanford Faculty Development Program Form-26, Clinical Teaching Effectiveness Instrument or the Resident Leadership Scale to establish the effect of programs on teaching skills of JMOs and their impact on those who they teach.

There were some limitations to our review. Included studies were largely heterogeneous in their methods and chosen interventions. All studies but one were conducted in North America.  A large number were conducted at single institutions and only involved one discipline, which makes generalizability of results difficult.

Unfortunately, we have demonstrated that studies on this subject have substantial methodological flaws. We do however appreciate the difficulties involved with designing the ideal trial. From a methodological perspective future trials should ideally be prospective randomised controlled trials. These trials should be designed to avoid issues with internal validity. This includes pre and post intervention assessment, valid randomisation methods with concealment, blinding of assessors if applicable and the use of validated, reliable and objective outcome measures. Also, these trials should use a standardised process of reporting outcomes (we suggest the Kirkpatrick model), so that their results can be easily interpreted and compared with other studies. They should focus on determining the effects of programs on high Kirkpatrick’s levels including 2b, 3, 4a, 4b and whether the effects are long lasting.

The authors declare that they have no conflict of interest.