Electronic databases ERIC, Ovid Medline and PubMed were searched for full-text articles describing the delivery of CAM teaching in UGME (see Table 1). Additional papers were found through a hand search of the reference lists of articles identified through the online database search.

There was no limit on the publication date. Only full-text articles written in English were included. Articles were limited to those that looked exclusively at UGME, with medical students only. Articles exploring medical students’ or faculty members’ attitudes regarding CAM were excluded. Articles that did not clearly describe the characteristics of CAM teaching within the UGME curricula (for example, topics taught in CAM programs, duration or frequency of teaching programs, methods used to assess student learning, etc.) were also excluded, as were general articles about CAM in medical education, and proposals for CAM curricula without implementation and opportunity for subsequent evaluation.

Complementary and alternative medicine, CAM, complementary medicine or alternative medicine, homeopath*, Chinese medicine, undergraduate medical, medical school, medical education, medical course, medical cirricul*, teach, university.

Data extraction was performed using a predetermined list, and included:

·            Article details: first author and publication year

·            Participant information: student cohort, institution, or country/city of institution

·            Information gathering approach: questionnaire, survey, telephone interview

·            Educational intervention: presence or absence of CAM integration, topics taught in CAM programs

·            Duration or frequency: time spent delivering teaching within the program

·            Instructors background: qualifications or title of educators

·            Outcome assessment: methods used to assess program aims, primarily students’ qualitative and quantitative course evaluation

·            Educational outcome: assessed using Kirkpatrick’s Hierarchy of Educational Outcomes, a well-recognised tool for the evaluation of the effectiveness of medical, educational outcomes.^5-7 The bottom level assesses learners' satisfaction with, or reaction to, the Intervention; the second level assesses modification of students’ attitudes and perceptions and/or the knowledge and skills learned; the third level assesses changes in health professionals' behaviour or an institution's practice, and; at the top of the hierarchy, changes in patient health care outcomes. In this review, Kirkpatrick’s Hierarchy was used to assess educational interventions as they relate to specific CAM programs taught within UGME curricula.

Ten studies directly addressed the inclusion of CAM in UGME curricula (see Table 2). Seven studies^9-15 utilised written questionnaires or surveys to collect program information, whilst two^{16, 17} used a telephone interview, and one ¹⁸ used a combination of both. Sampson and Colleagues¹³ and Brokaw and Colleagues¹⁰ assessed CAM education in U.S. medical schools. Sampson and Colleagues reported 45% (56/125) of medical schools offered some form of CAM education. Brokaw and Colleagues reported of 53 U.S medical schools surveyed, 75.3% (40/53) taught an elective CAM course, and 30.1% (16/53) taught a required course, with several schools (7/53) offering both. In neighbouring regions of North America, Ruedy and Colleagues¹⁷ found of the 16 Canadian medical schools evaluated, 81% (13/16) reported the inclusion of CAM in their medical programs, and the remaining 19% (3/16) were planning to include CAM teaching in the future.  Rampes and Colleagues¹² survey of British medical schools reported only 3 of 24 schools (12.5%) offered CAM teaching, and none provided practical training. Smith¹⁴ found CAM was included in the curricula of all 18 responding UK medical schools (58% response rate), with 33.3% (6/18) indicating it was taught formally within the UGME curriculum.

In Europe, Brinkhaus and Colleagues ⁹ surveyed 487 department directors at medical schools in Austria, Germany and Switzerland. Of these, 34% (162/487) indicated CAM had been integrated into their medical schools’ curricula. There was noted inter-country variability with lower CAM integration in Switzerland (20%) compared to Austria (28%) and Germany (27%). Varga and Colleagues¹⁵ surveyed 265 medical faculties in European medical schools, reporting that 40% of responding universities offer some form of CAM teaching.  In Japan, Tsuruoka and Colleagues¹⁸ reported CAM teaching was included in 16 of 80 (20%) schools, with 19 different teaching and learning approaches described. Kim and Colleagues (2012) reported CAM was taught at 85.4% (35/41) of participating Korean medical schools.¹¹ Chitindingu and Colleagues¹⁶ survey of seven South African schools reported one school was teaching both Traditional Medicine (T.M.) and CAM, five were teaching either T.M. or CAM, and one was teaching neither. T.M. relates to therapeutic practices that incorporate plant, animal and mineral-based medicines, spiritual therapies, manual techniques and exercises, often practiced in developing countries.¹⁹ Whilst TM is regarded as distinct from CAM, TM practices may be included within CAM.²⁰

Data on what elements of CAM are taught in UGME was inconsistently reported. In general, of studies that looked at whether CAM was taught in UGME, acupuncture was the most frequently taught modality^11,12,17,18 followed by homeopathy^10-12,17 and manipulation or chiropractic therapies.^10,12 In studies evaluating how implemented CAM programs are taught, the most commonly taught or demonstrated CAM disciplines were acupuncture^21-25 manipulation/chiropractic ^{24,26, 27} and massage.^21-23 Less frequently taught therapies included biofeedback^21,26 nutrition^21,28 analysis of medicinal plants²⁴ homeopathy^26,27 hypnosis²⁶ and osteopathy.²⁷ Several of the identified programs taught about the evidence base of CAM alongside the risks and benefits or offered an opportunity for scientifically evaluating CAM efficacy.^{21,24,27, 29-32}

Sixteen articles^21-30,32-37 described specific CAM teaching, learning, and assessment approaches in UGME curricula. A detailed breakdown of the different approaches used is described in Table 3. The structure of teaching modules varied greatly between programs. In a unique approach, Da Silva and Colleagues²² reported a split teaching program, where students attended CAM classes in their third year, followed by clinical placements in their fifth year. Meanwhile, other programs included integrated CAM teaching across three ²⁸ four ^{24, 34} five³⁰ and six years²⁵ of their UGME programs. Other programs adopted a block approach, where teaching was delivered over a period of weeks to months.^{27-29, 31, 35, 36}

Teaching time also varied greatly across institutions. Duration of time ranged from a relatively brief 5 hours²¹, to the longest of 90 hours.²⁵ Several of the programs with integrated teaching across several years did not report on the total teaching time, including Frenkel and Colleagues³⁴ and Perlman and Stagnaro.²⁴ The teaching modalities used in the delivery of CAM education varied widely. Only one program²⁴ used a solely didactic approach. Others used a didactic approach in conjunction with some form of interactive teachings such as tutorials^{24, 30} hands-on practice²³ discussion-based learning ^31,35,36 workshop³⁴ case and team-based learning²⁸ or student-led presentations.³⁵ In a commonly utilised approach, several programs included a clinical placement in combination with formal lectures or tutorials.^22,25-28,31 These placements varied in length from one day²⁶ to four-afternoon sessions of unspecified duration²² to 30 hours.²⁵ Bailey and Colleagues²¹ reported a unique approach involving a seminar followed by an Integrative Medicine fair where over 30 providers interacted with students through a series of student-selected workshops that introduced fields such as nutrition, massage, acupuncture, yoga and biofeedback. Similarly, the program described by Lehmann and Colleagues³⁵ offered a unique one-day excursion to the European Library for Homeopathy (Kothen, Germany).

Generally, the instructors delivering CAM programs were either faculty members of the medical school or clinicians^29,30,34-36 external CAM practitioners only^26,31 or both.^{21,24, 25,27,32} Several programs also included visiting experts and scholars from other universities.^24,28  In some instances, faculty members were noted to have undergone advanced CAM training.^24,32 Several programs involved qualified community or student CAM providers and practitioners, including those with an element of clinical placement.^{21,23,26,27,31} Post course student evaluations were infrequently used to assess student experiences and overall course satisfaction.^26,31 Objective assessment of knowledge and learning was not performed. Where knowledge assessment was completed, it often entailed subjective self-reporting using a Likert scale or other quantitative measure.^{22,23,27-29,36} Few programs used appropriately coded and thematically interpreted qualitative measures.^34,35 Several programs did not have formal evaluative or assessment processes in place^24,25,30,37 or removed the evaluative process following a period of time.²¹

This review assessed the educational and learning outcomes of the aforementioned studies using Kirkpatrick’s Hierarchy (Table 2).^5-7 There was a clear trend toward Level 1 outcomes, where student reaction and satisfaction were measured in all programs, bar one³⁴ using a post-course evaluation. Likewise, a majority of programs achieved Level 2 outcomes by considering the achievement of learning outcomes by students who completed self-evaluations.^{21,22,27-29,32,34-36} Only Frenkel and Colleagues³⁴ achieved Level 3 outcomes, where students were asked if and how learned knowledge from the CAM program would change their behaviour. It should be noted behavioural changes were self-reported and not directly observed or analysed. Level 4 outcomes were not achieved by any of the UGME programs included in this review.

The duration of teaching is one of the greatest areas of inconsistency amongst the included programs. Whilst a number had integrated CAM teaching across one or several years of their curriculum, the design of others was relatively brief, with few - or unspecified - dedicated teaching hours.^21,32,37 Meanwhile, several ‘integrated’ programs regarded the practices of mindfulness and self-care for students as an element of their holistic CAM curriculum.^23,30 Other medical schools considered mentioning CAM in areas of relevance - such as drug interactions and clinical oncology - constituted sufficient CAM teaching.¹⁴ This could lead to a potentially inflated and inaccurate measure of the duration of CAM teaching, particularly within these integrated approaches.

In analysing the approach to teaching, it is clear experiential learning is favoured compared to a solely didactic approach. Nearly all programs favoured delivery methods that would enhance student engagement. These included tutorials, workshops and case discussions, alongside clinical placements with CAM practitioners. Whilst well received by students, most immersive methods only achieved Kirkpatrick’s Level 1 and/or 2 outcomes, without an assessment of translatability to clinical practice and impacts on patient outcomes, which would require Level 3 and 4 outcomes, respectively. Various programs included clinical placements with CAM practitioners, the longest being 30 hours.²⁵ However, in the absence of a student evaluation or objective assessment tool, the benefits of clinical placement cannot be commented on.

Various programs focussed on specific CAM disciplines, the most common of which was acupuncture.^21-25 This reflects the identified difficulties in establishing a discrete knowledge base, as CAM disciplines require constant updating as new evidence and novel therapies emerge and gain traction within the general population.³ In what proved to be a popular approach^{21,24,27,29-32} analysis of the evidence base surrounding CAM is one of the more common methods of teaching. In equipping students with the skill set needed to critically appraise evidence, the plethora of CAM therapies need not be delved into individually. Rather, students can draw their own informed conclusions without the influence of potentially biased and unsubstantiated claims. Despite this, not all programs included teaching on evidence-based medicine (EBM), rather focusing on specific CAM disciplines. In designing a sustainable CAM program, curriculum developers should consider the need for constant review and critical appraisal as new evidence emerges. The EBM teaching model presents a potential solution to a rather overwhelming and resource intensive area of education, where learned knowledge and skills can be adapted for differing CAM modalities, and beyond.

Likewise, the qualifications of CAM educators for such programs must be consistent and their repeated involvement sustainable. Whilst many programs utilised community-based CAM practitioners^25,26,31,35 difficulties in repeatedly sourcing appropriately qualified instructors were identified.²¹ Where faculty staff were involved in teaching, several programs were sustainable for longer.^21,22,30 Whilst both CAM practitioners and faculty staff could both introduce an element of bias, this could be overcome with clear learning outcomes and an evidence-based teaching approach. In view of sustainability, it may be more appropriate to deliver teaching through faculty staff, some of who receive further training in EBM. Curriculum developers should consider and account for the additional cost of incorporating CAM programs within UGME.

Few of the programs performed formalised pre and/or post-course student assessment or evaluation as a measure of the change in knowledge. This reaffirmed findings from Stratton and Colleagues³ which reported the same observation in the programs receiving CAM education grants from the NCCAM. Instead, general qualitative statements were regarded as a measure of overall student satisfaction and increased knowledge.^21,27,34 Student reactions and knowledge (Kirkpatrick’s Level 1, 2a, 2b) were the most commonly achieved outcomes, with student knowledge measured subjectively through student questionnaires.^{22,27-29,31,32} Whilst Frenkel and Colleagues³⁴ was successful in determining the willingness of learners to apply new knowledge and skills (Kirkpatrick’s’ Level 3 outcome), it was the only study to do so. Course designers within medical schools must create educational programs that aim to directly impact patient care, rather than increasing knowledge without a foreseeable change to clinical practice. This is an undoubtedly challenging task, particularly since many widely taught CAM therapies have not yet been proven efficacious by scientific standards. For medical educators to educate appropriately, an evidence base for positive patient outcomes must first be established.

Articles may have been omitted due to the adopted search strategy, inclusion criteria, and limit of English language articles only. Grey literature was not performed. Several programs were incompletely reported, potentially influencing the outcomes reported in this review.  Many medical school faculty and Deans who were approached did not participate, leading to potential selection bias. Studies incorporating face-to-face interviewing may also be subject to observation bias. A small sample size, apparent in several studies, could also reduce the power and, therefore, reliability of results. Post-evaluations using subjective qualitative Likert-based assessment do not provide an objective measure of program success, particularly where students’ results cannot be compared to pre-program standards. As only the first three levels of Kirkpatrick’s Hierarchy were reached, the patient implications of CAM in UGME could not be determined. No studies reported on the change in patient outcomes or healthcare delivery (Kirkpatrick’s Level 4). Longitudinal prospective studies would provide curriculum developers with insight to the real-world effects of CAM education, where patient outcomes can be correlated to teaching interventions. Several studies did not report outcomes of interest, such as teaching duration, disciplines and teaching staff.^{22, 27, 28, 31, 32, 37}

The authors declare that they have no conflict of interest.