To describe the published literature on EBM curricula for physicians in training and barriers during curriculum implementation.
We performed a systematic search and review of the medical literature on PubMed, Embase, ERIC, Scopus and Web of Science from the earliest available date until September 4, 2019.
We screened 9,042 references and included 29 full-text studies and 14 meeting abstracts. Eighteen studies had moderate validity, and 6 had high validity. The EBM curricular structure proved highly variable in between studies. The majority of the EBM curricula was longitudinal with different lengths. Only five studies reported using Kern's six-step approach for curriculum development. Twenty-one articles reported on EBM skills and knowledge, and only 5/29 full-text articles used a validated assessment tool. Time was the main barrier to EBM curriculum implementation. All the included studies and abstracts, independent of the EBM curriculum structure or evaluation method used, found an improvement in the residents' attitudes and/or EBM skills and knowledge.
The current body of literature available to guide educators in EBM curriculum development is enough to constitute a strong scaffold for developing any EBM curriculum. Given the amount of time and resources needed to develop and implement an EBM curriculum, it is very important to follow the curriculum development steps and use validated assessment tools.
Evidence-based medicine (EBM) has been defined as the "conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients".
Understanding the concepts of EBM and mastering the abilities to retrieve, critically appraise, and apply medical literature to patient care are essential skills for clinicians.
Despite the fact that EBM training is now standard in physician training programs, there is still no consensus on how best to ensure that these skills are taught effectively for life-long learning. The most successful teaching methods for EBM are still unknown, and there is little evidence about which teaching methods lead to improved patient outcomes.
The objectives of this scoping review are 1. Describe the structure of the EBM curricula and its impact on the residents' attitudes, behaviors, skills and knowledge, and 2. Describe the barriers that were identified during the EBM curriculum implementation process.
The methodology for this scoping review was based on the framework outlined by Arksey and O'Malley.
For the purpose of this scoping review, the word "resident" was used interchangeably with "physician in training", as the term "resident" is mainly used in North America and not widely recognized. This review was guided by the question, 'What is the structure of the existent EBM curricula and its impact on the residents' attitudes, behaviors, skills and knowledge and what are the barriers identified during the EBM curriculum implementation process? Given the complexity of the research question, we decided that a scoping review would be the ideal study design to answer our question.
The following databases were searched: PubMed, Embase, ERIC, Scopus and Web of Science from the earliest available date for each database until April 25, 2019, and rechecked on September 4, 2019. Gray literature was also searched. No limits or barriers were applied, but only articles written in English were included. We screened the reference lists of retrieved studies for relevant publications.
PubMed search strategy is described: step #1 "Evidence-Based Practice" [Mesh:NoExp] OR "Evidence-Based Medicine" [Mesh] OR "evidence-based health care" OR "evidence-based healthcare" OR "evidence-based medicine" OR "evidence-based practice" OR "evidence-based care" OR ebm OR ebp OR (critical* AND apprais*) OR ebhc; step #2 "Education, Medical, Graduate" [Mesh] OR gme OR (graduate AND medical AND education) OR postgrad* OR "post-grad" OR "post-graduate" OR "post-graduates" OR residency OR residencies OR resident* OR trainee*; step #3 "Teaching" [Mesh] OR teach OR teaching OR learn* OR curriculum OR curricula OR workshop* OR instruct* OR educat* OR train* OR "journal club" OR "journal clubs" OR lectur* OR module* OR course*; step #4: #1 AND #2 AND #3.
The Appendix contains the search strategies for each database. To identify articles relating to teaching EBM in residencies in curricular format and barriers to teach EBM during the curriculum implementation, the following search terms and their word variants were used: resident, evidence-based, education, teaching, lecture, curriculum, barriers. Exclusion criteria were: student, practitioner, private. The word 'curriculum' was not an inclusion criterion word because not all the articles that described an EBM curriculum had the word 'curriculum' included in the title or abstract. Because the assessment is part of the curriculum development process, studies that reported on an EBM curriculum but didn't have an assessment were excluded. Studies reporting on a single EBM education session or single journal club were excluded as they were thought not to qualify for the curriculum description if they did not address all of the EBM steps.
We aimed to identify all randomized, nonrandomized, and before and after studies that reported on: 1. teaching EBM in a curriculum format to residents and on: 2. barriers during curriculum implementation. To be included, studies had to provide details about the format of their EBM curriculum and report on the evaluation of their curricula, as evaluation is the last step in the curriculum development process. We used the following criteria to screen studies for inclusion in our review: 1. Study design: original studies and abstracts presented at national conferences, quantitative, qualitative, and mixed methods studies were included if they reported on EBM curriculum with or without barriers to teaching EBM to residents; abstracts describing an EBM curriculum and its evaluation were included to avoid the risk of reporting bias if they had been excluded. Systematic reviews on the EBM curriculum and barriers were not included. Their references were manually verified against our search, and the articles not initially included were uploaded for screening; 2. Population: residents of any specialty, working under the supervision of a medical specialist. Medical students and attending physicians were not included as their learning environment, patient care exposure, and motivation are significantly different than for the physicians in training; 3. Outcomes: pertaining to curriculum evaluation, the following categories based on Kirkpatrick's training evaluation model
A detailed excel spreadsheet with all the data points that were planned to be collected was developed through collaboration between 2 of the authors (AH and BH). The form was calibrated through concomitant (both authors) analysis and data extraction from various studies. Data charting was done independently and in duplicate. Potentially relevant article abstracts were independently reviewed by two authors (AH and BH), who then met and agreed on article selection. For each potentially eligible study, the full paper was read and reviewed by the same two authors (AH and BH) independently. They both assessed whether the study fulfilled the inclusion criteria. Studies about which both reviewers had doubt or disagreement about inclusion were thoroughly discussed during three consensus meetings, and a common decision was reached.
Two authors (TD) and (BH) performed the data extractions from the full-text articles included, which were then fully reviewed by another author (AH) and any disagreements on data extractions were resolved by consensus between AH and BH. Two authors (BH) and (EN) performed the data extraction from the abstracts included. A prespecified data extraction form was used to extract and collect information from the included studies on the author, year of publication, type of study design, country, residency type, number of participants, study period, specific details of the EBM curriculum structure, length, teachers, content, outcome measures, evaluation methods, results, and barriers. Because we expected to find no homogenous studies in this area, no pooling of data was attempted.
To assess the risk of bias of individual studies, two authors (BH and TD) independently rated the quality of the studies at the study level. All disagreements were reviewed and resolved by a third author, AH.
We assessed the quality of the pre-post studies using the National Heart, Lung and Blood Institute Quality Assessment Tool for Before-After (Pre-Post) Studies with no Control Group (
Nine-thousand-forty-two references were imported for screening, and 214 duplicates were removed. Eighteen extra articles were considered for inclusion from a manual review of the references, and after cross-checking with the articles already included in the initial screening, 8 of them were duplicates and were removed. Four extra articles were suggested by experts in the field and were included for screening. A total of 9064 articles were included after automatically removing the duplicates. Further duplicates were removed manually, and 8842 articles were screened against the title and abstract, and 8742 articles were found irrelevant and were excluded. One hundred articles were assessed for full-text eligibility, and 57 articles were excluded because they didn't meet the eligibility criteria. Only one out of the 18 studies manually added after searching the references was added to this review. One of the 4 studies suggested by the experts fulfilled all inclusion criteria. The final review included 43 articles (29 full-text articles and 14 abstracts) describing the structure of the EBM curriculum. Thirteen of these articles reported on barriers to EBM curriculum implementation.
| Pre-Post study Design21 | |
|---|---|
| 1. Was the study question or objective clearly stated? | |
| 2. Were eligibility/selection criteria for the study population prespecified and clearly described? | |
| 3. Were the participants in the study representative of those who would be eligible for the test/service/intervention in the general or clinical population of interest? | |
| 4. Were all eligible participants that met the prespecified entry criteria enrolled? | |
| 5. Was the sample size sufficiently large to provide confidence in the findings? | |
| 6. Was the test/service/intervention clearly described and delivered consistently across the study population? | |
| 7. Were the outcome measures prespecified, clearly defined, valid, reliable, and assessed consistently across all study participants? | |
| 8. Were the people assessing the outcomes blinded to the participants' exposures/interventions? | |
| 9. Was the loss to follow-up after baseline 20% or less? Were those lost to follow-up accounted for in the analysis? | |
| 10. Did the statistical methods examine changes in outcome measures from before to after the intervention? Were statistical tests done that provided p values for the pre-to-post changes? | |
| 11. Were outcome measures of interest taken multiple times before the intervention and multiple times after the intervention (i.e., did they use an interrupted time-series design)? | |
| 12. If the intervention was conducted at a group level (e.g., a whole hospital, a community, etc.) did the statistical analysis take into account the use of individual-level data to determine effects at the group level? | |
| Prospective cohort study design23 | |
| Selection | |
| 1) Representativeness of the exposed cohort | |
| 2) Selection of the non-exposed cohort | |
| 3) Ascertainment of exposure | |
| 4) Demonstration that outcome of interest was not present at the start of the study | |
| Comparability | |
| 1) Comparability of cohorts on the basis of the design or analysis controlled for confounders | |
| Outcome | |
| 1) Assessment of outcome | |
| 2) Was follow-up long enough for outcomes to occur | |
| 3) Adequacy of follow-up of cohorts | |
| Randomized controlled trial study design22 | |
| 1) Random sequence generation | |
| 2) Allocation concealment | |
| 3) Selective reporting | |
| 4) Other sources of bias | |
| 5) Blinding (participants and personnel) | |
| 6) Blinding (outcome assessment) | |
| 7) Incomplete outcome | |
Of the 43 studies included in the final systematic review, 29 studies were reported as full text as follows: 18 described the results of quantitative pre-post study design, 5 described the results of quantitative surveys, 4 described prospective cohort studies, two randomized controlled trials, and 14 meeting abstracts.
The risk of bias was assessed only for the studies included as full text. The abstracts were not assessed for their quality, as it was appreciated that they contained very limited information about the methodology and possible bias. Also, the full-text surveys were not assessed for risk of bias. The overall quality of the 24 studies evaluated was determined to be moderate or low risk of bias (Appendix 1 and Appendix 2).
| Author (year) | Allan et al. (2008) | Aneese et al. (2019) | Bentley et al. (2018) | Burneo et al. (2006) | Chitkara et al. (2016) | Friedman et al. (2010) | ||
|---|---|---|---|---|---|---|---|---|
| Study design | Pre-post study | Pre-post study | Pre-post study | Survey | Pre-post study | Pre-post study | ||
| Country | Canada | USA | USA | Canada | USA | USA | ||
| Residency type | Family medicine | Internal Medicine | Emergency Medicine | Neurology | Pediatrics | Emergency Medicine | ||
| Number of Participants, n | 181 | 60 | 53 | 12 | 17 | 24 seniors (PGY-3 and PGY-4) | ||
| Study period | 3 years | 1 year | 1 year | single point in time survey | 3 years | 3 months | ||
| EBM Curriculum Structure | Workshop, Family Medicine Desktop, brief evidence- based assessment of the research [BEAR], journal club; teaching on industry–physician interaction, presentation skills, and a quarterly review of the literature | 1)Monthly EBM workshop; 2)monthly Journal Club; 3)resident led morning report; 4)teaching rounds | 8 self-paced modules | 24 clinical topics chosen at beginning of the year; prior to each session, resident chooses real encounter on the topic, develops question, searches, selects best article and distributes to all trainees and faculty. CAT is designed and published to website. | 1) first year residents: foundational course as small groups and independent study 2) 2nd year: application, CAT presentation 3) 3rd year: integration (act as mentors to 1st and 2nd) | 1 hour EBM online tutorial; 3 dedicated formal literature searches for active management questions; brief EBM teaching round (reviewed the clinical question, method, literature search and the relevant evidence was used to answer the management question) | ||
| Length/frequency | 1) Workshop: 4 half-days, each consisting of a one-hour lecture followed by a two-hour, small- group session. 2) Each BEAR is presented in approximately 15 minutes; 3- 4/resident; 3) Journal club - 2.5 hours | 1) 90 minutes X 4/year; 2)60 minutes X 12/year; 3)1 hour X 1/week; 4)1 hour X 3/week | Monthly | Biweekly (every other week), 90 minutes (total of 24 per year) | Longitudinal, year long, frequency not specified | 1 hour tutorial; 3 x 8 hour shifts | ||
| Teachers/ facilitators | Faculty physician, faculty librarian, second-year resident | Faculty physician, faculty librarian, second and third year resident | Not reported | Faculty physicians, residents of all levels | Faculty, librarian, third year residents as mentors | Not reported | ||
| Clinically integrated (lectures versus point of care) | Yes, partially | Yes, partially | No | No | Yes, partially | Yes | ||
| Curriculum content | Knowledge component: 1) Understand the rationale and benefits of EBM 2) Provide a strong foundation in the basic principles of EBM, including: Recognizing and formulating clinical questions; Finding and accessing information, interpreting information, applying information 3) Provide and identify online resources, educational tools, and web-links. Skill component: 1) Learn to identify problems/questions encountered in practice and seek solutions by rapidly answering clinical questions with best evidence 2) Build skills and comfort in knowledge transfer. Attitude component: 1) Facilitate appreciation and enthusiasm for the judicious use of current evidence to optimize patient care, and Maintenance of skills and knowledge through life-long self-directed learning. | Formulating clinical question; Search skills; Critical appraisal skills; Evaluating the evidence skills; Applying the evidence | Critical appraisal skills; EBM concepts based on study design | Formulating clinical question, finding evidence, critical appraisal | 1) PICO, search skills, critical appraisal 2) PICO applied, CAT presentation, CAT abstract generation 3) EBM teaching and mentorship | PICO, Formulating clinical question; Search skills; Critical appraisal skills; Evaluating the evidence skills; Applying the evidence | ||
| Curriculum development process described | Yes, quite extensive. Comittee formed to investigate concerns regarding previous curriculum, goals and objectives developed | Not discussed | Not discussed | Not discussed | Not discussed | Yes, Kerns six step approach used | ||
| Outcome measures | Self-assessed skills, past training, and objectives in EBM | EBM knowledge and skills; Attitude toward EBM | EBM knowledge and skills; attitude toward EBM | Participation during EBCP sessions, applicability to daily clinical practice, and foreseeable future practice, how likely clinical practice was influenced, if trainees were teaching the concepts to other residents/students | a. Knowledge and skills b. Attitudes and behavior | Practice norms and attitudes regarding EBM; knowledge, skills | ||
| Evaluation method | pre-curriculum survey, quiz; post survey, quiz | Validated questionnaire (Fresno test); Questionnaire; Focus group | validated questionnaire (Berlin Questionnaire BQ) | questionnaire/survey | a. Pre-post knowledge questionnaire (1st year), b. self assessment survey (years 2 and 3), c. focus groups | survey | ||
| Results | Statistically significant improvement in all areas of comfort with evidence-based practice and opinions regarding the curriculum; general opinion (e.g., The practice of EBM results in better patient care) showed improvement as well, albeit to a lesser degree; | Fresno pre-test 110.5, post-test 115 (p=0. 60); pre-post therapy questionnaire (p=0. 006), diagnosis (p=0.006), systematic review (p=0.002), harm (p=0.004) | BQ median pre scores 40%, median post scores 73.3% (p=0.002) | On a scale of 1 to 10, they rated the influence to include EBCP concepts in their daily clinical practice as high (mean 6.8, S.D. 1.5). All graduates believed the EBCP concepts were useful, but only applied them when time allowed. On a 1-to-10 scale, they rated the influence to include EBCP concepts in their daily clinical practice higher than the residents (mean 8.5, S. D. 1.2). | pre 6.9, post 9.1 (p=0.02); institutional practice changes reported, such as creating new care plans; residents were better able to appraise the literature (p=0. 02); improved reported searching skills, appraisal, clinical application, repeated practice in the peer mentor role improved knowledge and skills retention | No significant change in the number of subjects that anticipated further practice changes as a result of their experience; 16.3% change in management by the primary team based on the literature searches performed; the change in management was a perceived change as reported by the subjects and was not reviewed by an attending physician. | ||
| Author (year) | Gehlbach et al. (1980) | George et al. (2012) | Green and Ellis (1997) | Halalau et al. (2016) | Keddis et al. (2011) | Kenefick et al. (2013) | ||
| Study design | Prospective cohort | Pre-post study | Pre-post study | Pre-post study | Prospective cohort | Pre-post study | ||
| Country | USA | USA | USA | USA | USA | USA | ||
| Residency type | Family Medicine | Family Medicine | Internal Medicine | Internal Medicine | Internal Medicine | Pediatrics | ||
| Number of Participants, n | 23 PGY-2 and PGY-3 | 26 PGY-2 | 34 (PGY-2 and PGY-3) | 23 | 45 PGY-3, 42 PGY-1 | 4 x PGY-1 | ||
| Study period | 2 years | 2 years | 1 year | 1 year | 5 months | 1 month | ||
| EBM Curriculum Structure | Seminar sessions, resident presentations | One-on-one 1-hour monthly meetings | Based on adult learning theory, the educational strategy included a resident- directed tutorial format, use of real clinical encounters, and specific EBM facilitating techniques for faculty | Based on adult learning theory; resident-directed teaching, small-group sessions, open discussion sessions and direct one-on-one teaching of EBM concept | PGY-1 residents participate in 8 interactive 1-hour sessions during a 1-month rotation. Residents apply their knowledge by presenting their EBM analysis of a clinical case and pertinent literature to a group of residents and faculty members at a 1-hour weekly conference called Clinical Decision Making Journal Club | 2 sessions | ||
| Length/frequency | 8 x 1 hr weelky sessions | 1h monthly meetings (30 min/ meeting reserved for EBM) x 1 year | 7-week, weekly 1 h tutorials, for 5-14 residents | Over 10 hours of EBM training each month, for 12 months | 8 x 1 hour sessions | 2 x 2h sessions during a month | ||
| Teachers/ facilitators | Faculty | Learning coach = family physician and recent residency graduate | General medicine faculty, librarian | Faculty, librarians, residents | Not reported | Medical librarian and hospitalist | ||
| Clinically integrated (lectures versus point of care) | No | No | Yes | Yes, partially | No | Yes | ||
| Curriculum content | First 3 seminars: clinical trials, sensitivity, specificity, predictive value, statistical significance; journal handouts for each session; Next 4 seminars: resident presentations evaluating articles; Last seminar: take-home test | EBM curriculum topics: Introduction to EBM, introduction to online EBM resources, introduction to PubMed, basic biostatistics, Introduction to guidelines, formulating PICO Questions, study design, clinical question search strategies | Each of the seven tutorials guides the residents through a real clinical scenario, representative of one of six prototypical clinical questions: therapy, prognosis, harm, diagnosis, prevention, and decision making. In the seventh, they can choose any question type, but must use a systematic review as evidence. | 1) Fundamentals of EBM and their application to patient care; study designs and simple statistics and their interpretation in patient care; 2) sessions on using and searching EBM resources; learn the hierarchy of evidence and to gain searching skills in finding the best evidence to inform decision making for patient care; 3) skills in critical appraisal of any study design; 4) critical appraisal of a study that would answer their clinical question and receive direct peer to peer and EBM faculty feedback; 5) basic study design and statistical analysis interpretation of therapy studies, diagnosis studies, harm studies and systematic reviews and meta- analysis | Sessions address the following topics: the EBM cycle; the patient, intervention, comparison, outcome, type of study design, type of question (PICOTT) format for structuring clinical questions; hierarchy of evidence; study design; and critical appraisal of therapy, diagnosis, prognosis, meta-analysis, and harm articles. | PICO format to help form and organize a clinical question; search of the literature to answer their PICO question, levels of evidence, optimal study types | ||
| Curriculum development process described | Not discussed | Yes, minimal components included (needs assessment, implementation, evaluation) | Yes, most components (needs assessment, goals and objectives, implementation, assessment) | Yes, fairly robust (questionable needs assessment. Yes implementation, assessment) | Not discussed, pre-existing curriculum | Not discussed | ||
| Outcome measures | Skills to interpret medical literature critically | Attitudes regarding EBM; EBM knowledge and skills | Attitudes toward EBM, prior critical appraisal training, self-assessed EBM competence, medical reading habits, and preferences for information sources. | Residents' comfort level with basic statistics, literature search and EBM concepts, residents' satisfaction with the EBM curriculum | Resident confidence with and knowledge of EBM topics | Scores on in-class, independent searches, and posttest cases, each of which were scored independently by three faculty members | ||
| Evaluation method | take-home test w multiple choice questions centered around some article samples; a practical problem given to all residents at the in training exam; comparison between 23 PGY 2-3 who took curriculum and 12 PGY-1 who did not (for the in training exam problem) | attitudes - survey; knowledge - 18 item quiz, 6 questions adapted from Fresno test of competence; qualitative - coach ratings after every coaching session | survey of EBM behaviors, a survey of self- assessed EBM competences, and an EBM skills test - modified from Stern et al. (free text responses to questions based on a clinical vignette and a redacted journal article) | 27 item survey | Likert- scale type survey (which included questions from the validated Berlin questionnaire) | pre-test and posttest of four cases to evaluate the residents' skill and efficiency in asking a clinical question and finding an appropriate answer | ||
| Results | Participants' % scores on the article analysis questions ranged from 42-93%; nonparticipants scored from 34-100%. Participants had a mean score of 73.8 %, nonparticipants had a mean 63.8 % (stat signif). When median percent scores were compared, a larger difference was observed between the resident groups (79 participants, 62 nonparticipants) | Favorable attitudes; significant improvement in quiz scores by 31.8% (p<0.001); using wider range of electronic resources; greater confidence and comfort in finding clinical information; ability to find best medical evidence for patients in real time / point of care; EBM valued; importance of one-on-one format | Scores on the EBM skills test (8.5 to 11.0, p =.001) versus control (8.5 to 7.1, p=0. 09). Posttest scores for case and control subjects, the mean difference was 3.9 points (p =.001, 95% confidence interval [CI] 1.9, 5.9). | Self-reported conceptual understanding improved for: relative risk 14%, odds ratio 14%, confidence intervals 27%, and number needed to treat 12%. Comfort with meta-analysis appraisal improved, from 30% to 38%. Routine appraisal sheet use increased by 31%. A 17% increase in satisfaction with the EBM curriculum was reported. | There was no relationship between confidence with and actual knowledge of EBM topics (PGY-1 pre-curriculum vs PGY- 3). Lower confidence among PGY-3 than among PGY-1 internal medicine residents for several EBM topics. PGY-3 residents demonstrated poor knowledge of several core topics taught during internship. | PICO: pre-curriculum test score 29%, post- curriculum test score 87%; Searching: Pre- curriculum test scores 46%, post- curriculum 99%; average time to complete cases pre-curriculum 12 min, versus post-curriculum 8.10 min. | ||
| Author (year) | Kim et al. (2008) | Kitchens and Pfeifer (1989) | Kohlwes et al. (2006) | Konen and Fromm (1990) | Kortekaas et al. (2016) | Lentscher and Batig (2017) | ||
| Study design | RCT | Pre-post study | Prospective cohort | Survey | RCT | Pre-post study | ||
| Country | USA | USA | USA | USA | Netherlands | USA | ||
| Residency type | Internal Medicine | Internal Medicine | Internal Medicine | Family Medicine | General Practice | Ob/Gyn | ||
| Number of Participants, n | 50 (PGY-2 and PGY-3) | 83 (PGY-1, PGY-2, PGY-3) | 32 PRIME residents | 12 | 79 | 17 | ||
| Study period | 1 year | unclear | 2001-2004 | 5 years (1983-1988) | 2 years | 1 year | ||
| EBM Curriculum Structure | EBM teaching workshops (intervention) vs weekly resident journal clubs without a formal or explicit curriculum (control) | Group A - literature-based curriculum in clinical appraisal; Group B - ambulatory care medicine topics | 4 components: (1) didactic lecture, (2) frequent journal clubs, (3) work-in-progress sessions, and active mentoring to enable residents to "try out" a clinical research project during residency, (4) research project supervision | PGY-1: 2 seminars, a journal club, quarterly research gatherings; PGY-2: presentation at grand rounds; PGY-3: seminars | Intervention: integrated EBM training and teaching sessions based on dilemmas from actual patient consultations; Comparison: stand-alone EBM training at the institute only | Structured journal club curriculum | ||
| Length/frequency | 6 x 2h EBM teaching workshops during a month | weekly preclinic conference 30-45 min; 17 weeks for Group A, 8 weeks for group B | (1) 12 didactic lectures in consecutive weekly 90- minute sessions; (2) weekly afternoon small journal clubs; (3) PRIME projects - residents present their projects quarterly in 90 min work-in-progress sessions (4) research project supervision: 1-2 x 1 h meetings/month | 3 year curriculum; PGY-1: 2 seminars and a journal club, quarterly research gatherings (length not clear); PGY-2: 1 hour presentation at grand rounds; PGY-3: monthly 2 h seminars | a) Clinical practice: 1 h x 4 days/week; integration of EBM 1/week; critical appraisal 1/month b) institute: 1 day/week; EBM course 5 days/year, 2.5 hours; exchange of last week's experiences in clinical practice 1 h/week, integration of EBM for 15 min | Monthly | ||
| Teachers/ facilitators | Medical librarians, resident-led | Faculty preceptors | Members of clinical investigator faculty, program director, program directors, clinician-investigator faculty, faculty | Faculty members, physician research director, nonphysician biostatistician, research assistant | Experienced GPs | Not reported | ||
| Clinically integrated (lectures versus point of care) | Yes | No | No | No | Yes, partially | No | ||
| Curriculum content | Formulating clinical questions using PICO; medical librarians tought evidence-based synopses and summary resources; topics of therapy, prevention, diagnosis, and prognosis - led by a resident who identified questions from actual patient encounters and performed literature searches using both bibliographic databases as well as evidence-based summary resources to find articles that addressed their question | Series of articles published by the Department of Clinical Epidemiology and Biostatistics at McMaster University that provides a reader with practical guidelines to help read the clinical literature critically. Each week one of these articles was paired with a current article from the clinical literature. Each resident was expected to read both articles, and one resident was assigned to lead a discussion of the content of the clinical article as well as the methodological strengths and weaknesses of that article | (1) PRIME epidemiology curriculum: A. overview of epidemiology study types, B. How epidemiology data are presented; C. How to design a research question; D. Study design and sampling; E. Issues in measurement; F. Causal inference; G, H. Qualitative biostatistics; I. Computer skills; J. Research ethics; K. Systematic review; L. Meta-analysis (2) Weekly journal clubs: diagnostic test evaluation, case- control studies, cohort studies, randomized control trials, meta-analysis, decision analysis, cost- effectiveness analysis, practice guidelines, and clinical overviews are covered, and articles that they frequently hear quoted on the medical wards. | 1) Critically evaluate research studies in the professional literature and translate valid conclusions into medical practice; 2) identify areas of potential research interest; 3) embark on a lifelong program of continuing education and professional growth. | Integration of EBM, critical appraisal of an article, EBM course focusing on translation of evidence into clinical practice, patient-related preassignments and postassignments to perform with supervisor; exchange of last week's experiences in clinical practice; discussion of barriers; presentation of a critically appraised topic; possibility for e-learning, online coaching, participation in research | Monthly curriculum that paired a chapter/topic from 2006 edition of The Lancet Handbook of Essential Concepts in Clinical Research with a contemporary journal article selected to highlight the month's topic: overview of clinical research, descriptive studies, bias and causal associations, cohort studies, case-control studies, finding controls for case-control studies, overview of clinical research, uses and abuses of screening tests, refining clinical diagnosis with likelihood ratios, sample size calculations in RCTs, general allocation sequences in RCT, unequal group sizes in RCT | ||
| Curriculum development process described | Not discussed | Not discussed | Not discussed | Not discussed | Not discussed | Not discussed | ||
| Outcome measures | EBM knowledge, use of evidence-based resources, performance on web-based clinical vignettes | Basic knowledge of clinical epidemiology | Clinical competence, board passing rates, asked to be chiefs, PRIME program popularity, research output | Attendance (as the best measure of interest); resident ratings (attitudes) of each session | EBM behavior, assessed by measuring guideline adherence (incorporating rational, motivated deviation) and information-seeking behavior; EBM attitude, EBM knowledge | Attitudes and self-rated knowledge; EBM knowledge | ||
| Evaluation method | Fresno test pre- and post-intervention. Post intervention, residents twice completed a web-based, multiple-choice instrument (15 items) comprised of clinical vignettes, first without then with access to electronic resources. | clinical epidemiology test 1 month post- intervention for group A and group B; intervention changed for group A and group B (cross over); then test again; test with 22 questions on clinical epidemiology principles derived from the articles by the McMaster University group | clinical competence scores | pre-test taken from Reigelman | 1. EBM behavior: information-seeking behavior - logbooks; guideline adherence - validated instrument; 2. EBM attitude: McColl validated questionnaire; 3. EBM knowledge: MF Kortekaas validated questionnaire | survey: 6 initial questions to assess individual attitudes and self-rated knowledge; additional questions to assess individual knowledge related to basic study design, bias, and levels of medical evidence. administered pre- and post-intervention | ||
| Results | Posttest scores improved for both groups, but were significantly higher for the EBM teaching group (mean score increase 22 (SD=13.8) for teaching group vs. 12 (SD= 12.2) for control group, p=0. 012). There was more improvement in EBM knowledge (100-point scale) for the intervention group compared to the control group (mean score increase 22 vs. 12, p=0.012). | Group B performed significantly better on the second test (post) than on the first test (pre), 68.5% vs 63.3% (p=0.034), while group A did not improve (64.5% va 65.9% - group who received the intervention from the beginning). | The overall clinical competence scores evaluating the 32 PRIME residents gave an overall average score of 8.23 on a 9.0 point scale. This was significantly better than the average of 8.09 for the rest of the internal medicine program (p<0.001). | Attendance greatly improved after the first year (p<0.001); seminar topic preferences (computers in medical practice 0.99; how to write a research article 0.98; different study designs 0.91; reading a research article 0.88; biostatistics 0.86; conducting a literature review 0.75; clinical trials 0.71) | No significant differences in outcomes between the 2 groups, with relative risks for guideline adherence varying between 0.96 and 0.99 (95% CI 0.86 to 1.11) at the end of the third year, and 0.99 and 1.10 (95% CI 0.92 to 1.25) at 1 year after graduation, and for information-seeking behaviour between 0.97 and 1.16 (95% CI 0.70 to 1.91) and 0.90 and 1.10 (95% CI 0.70 to 1.32), respectively. | There was no significant improvement in resident self-assessed knowledge following curriculum implementation. There was a trend toward improved objective knowledge pertaining to study design and interpretation after curriculum completion, but this was not statistically significant | ||
| Author (year) | Letterie and Morgenstern (2000) | Luciano et al. (2016) | Mohr et al. (2015) | Nelson et al. (2017) | Nicholson and Shieh (2005) | Ross and Verdieck (2003) | ||
| Study design | Survey | Prospective cohort | Pre-post study | Pre-post study | Survey | Pre-post study | ||
| Country | USA | USA | USA | USA | USA | USA | ||
| Residency type | Ob/Gyn | Internal Medicine | Emergency Medicine | Pediatrics (PGY-1 and PGY-2) | Internal Medicine | Family medicine | ||
| Number of Participants, n | not provided | 61 | 31 | 60 | 36 | 18 - intervention; 30 - control | ||
| Study period | 1 year | 2 months | 2 years | 1 year | 1 year | 1 year | ||
| EBM Curriculum Structure | Journal club: 2 sets of articles distributed for each session; 1 - literature on topics of epidemiology, biostats, experimental design (2-6 articles); 2 - clinical literature for review (illustrate and emphasize prior concepts) | Patient-centered ambulatory morning report (AMR), which combines the User's Guides to the Medical Literature approach to EBM and the Kolb experiential learning theory | Journal club | Sessions grounded in adult learning theory principles, flaw catching exercises, multiple choice questions using audience responses and interactive review of abstracts and articles | Daily EBM session during rounds with the hospitalist team | Interactive workshops | ||
| Length/frequency | One 2h session/ month x 12 months | Morning report: 2-3 months/year, weekdays, 30 min/day; + 5h basic EBM concepts; + 4h interactive didactic EBM content (most of these 9h take place within morning report) | monthly | 4 x 90 min sessions | 1 month | 10 sessions; 1-2 hour sessions, brief lecture (30- 40 min) followed by practical application | ||
| Teachers/ facilitators | Resident, staff physician | Residents, facilitators (not specified) | Faculty | Faculty | Hospitalist | Preceptors | ||
| Clinically integrated (lectures versus point of care) | No | No | No | No | Yes | Yes | ||
| Curriculum content | 24 concepts descriptive of topics considered critical in the assessment and evaluation of literature descriptive of clinical practice and patient care: topics of epidemiology, biostatistics, and experimental design (p values -> meta analysis), literature on clinical topics (illustrate and emphasize concepts of experimental design and statistical analysis). | EBM skills: identify a clinical problem, define the problem as a structured clinical question, acquire new knowledge; EBM competencies: evaluate the validity of evidence cited, evaluate the results; apply and use new knowledge; EBM competency: apply the results to the clinical scenario | 1) Dedicated 20-min monthly discussion led by a faculty member during monthly journal club focused on an EBM-focused topic. 2) a single faculty member being grouped with three coordinating residents to review selected articles from an EBM perspective prior to journal club to discuss the teaching points of each in detail. 3) a period of peer-to-peer teaching during journal club, 4) a dedicated core of EBM faculty dedicated to directing the journal club curriculum and establishing expertise in EBM concepts | Session 1: class - EBM test, flaw catching exercise, Introduction to MAARIE framework; homework - multiple choice questions covering session 1 information; Session 2: class - review multiple choice questions in teams using audience response systems; introduction to MAARIE framework; flaw catching exercise; homework - read randomized controlled trial; Session 3: multiple choice questions covering session 2, apply MAARIE framework to abstracts; apply MAARIE framework to RCT #1; homework - read RCT #2; Session 4: review MAARIE framework and key concepts; apply MAARIE framework to RCT #2, EBM test. | Definitions and principles of EBM, literature appraisal, EBM internet resources; formulate focused clinical questions for each patient admitted, do literature searches, use EBM principles to evaluate search results, and present findings during subsequent rounds. | 1. Multiple-choice pretest, overview and constructing focused questions: interactive formulation of good, researchable EBM questions for patient care; 2. categorizing articles, reading review articles: discussion of recognizing focused reviews, how to sort articles into groups (i.e., therapy, diagnostic tests, reviews); 3. reading review articles: a discussion regarding key features of well-researched and written reviews; 4. evidence for therapy; 5. evidence for diagnostic tests; 6. evidence for history and physical exam; 7. evidence for prevention; 8. evidence for practice guidelines; 9. applying quality improvement | ||
| Curriculum development process described | Not discussed | Not discussed | Not discussed | Not discussed | Yes, everything but discussing needs assessment | Yes, Kerns six step approach used | ||
| Outcome measures | EBM knowledge and skills | EBM knowledge: learn to use medical literature to guide patient care | EBM skills and knowledge | Baseline knowledge, efficacy of the curriculum in improving knowledge | Learning experience, understanding of EBM terms or practice skills | Knowledge and application of EBM during continuity clinic patient care | ||
| Evaluation method | questionnaire | 18-question EBM test (of the 18 questions, 1 required formulating a structured clinical question, 6 addressed evaluating validity of an article, 5 addressed evaluating results, and 6 addressed applying results to a clinical scenario). | Fresno test | validated 20 question evidence based medicine multiple choice test was administered on three separate occasions | questionnaire on which they were asked to rate the impact of the curriculum on their understanding of 20 EBM terms or practice skills | a 50-item, multiple-choice examination was administered before and after the workshop series; residents at another FP residency at the same university served as a control group. Resident–preceptor interactions during outpatient continuity clinic were tape recorded prior to and six months following introduction of the curriculum | ||
| Results | 85% residents wanted to continue the format without much change; 15% wanted to restrict literature to Ob/Gyn content | Average score increased during the training period, most strongly following internship. Mean scores stabilized after internship; however, the range of scores successively narrowed; EBM scores improved in time in the following domains: formulating a structured clinical question, assessing the validity, and evaluating the results | Total test scores did not increase significantly (105.4 vs. 120.9, p = 0.058) in the before–after analysis; the only subscore showing improvement was interpretation of study validity (32.1 vs. 40.4 points, p = 0.03); attendance was significantly associated with Fresno test score, with those attending more than 6/11 sessions (55%) scoring 28.2 points higher (p = 0.003) | Post curriculum, the fall group’s scores improved 23% from baseline (M=10.3, SD=2.4) to (M=12.7, SD=3.0) students (t(26)=-3.29, p=0.0018) while the spring group improved by 41% (M=10.0, SD=2.8) to (M=14.1, SD=2.2) students (t(32)=- 6.46, p<0.0001). There was an association between number of sessions attended and increase in post curriculum score (χ2 (3, N=60) =11.75, p=0.0083). | Results were very positive with average effect of more than 4 (somewhat strong effect/impact) for 16 of the 20 questions. | Pre-intervention multiple-choice test results were similar (control mean 56%, experimental 53%, p>. 22 NS); post-intervention test scores for the experimental group were significantly improved (mean 72%, p<.001); there was no significant improvement in test results among members of the control group (p>.05 NS); in the recorded resident–preceptor interactions, a marked increase in the use of EBM terms indicated awareness and/or use of EBM in the experimental group | ||
| Author (year) | Shaughnessy et al. (2012) | Thom et al. (2004) | Trickey et al. (2014) | Windish (2011) | Zeblisky et al. (2015) | |||
| Study design | Pre-post study | Pre-post study | Pre-post study | Pre-post study | Survey | |||
| Country | USA | USA | USA | USA | USA | |||
| Residency type | Family Medicine | Family Medicine | General surgery | Internal medicine | Pediatrics | |||
| Number of Participants, n | 23 | 13 PGY-1 | 40 | 52 | 46 | |||
| Study period | 5 years | 3 months | 6 years | 2 years | 6 months | |||
| EBM Curriculum Structure | 1) Block format - intensive instruction over the course of 1 month (30 hours); 2) longitudinal series of ongoing conferences: a modified journal club, group session at the beginning of the year, weekly subspecialist conferences; teaching was also integrated into day-to- day clinical activities via precepting interactions | Individual EBM rotation for interns; online tutorial, workshop, journal club | Lectures, tutorials, practice questions, examples | Journal articles reading, seminars, presentations by learners | 1) Technology sessions, 2) increased librarian support for EBM presenters, 3) increased interaction during conferences, 4) standardized curriculum to reflect progression of the academic year, and 5) small group learning; | |||
| Length/frequency | 1) 30 hours face-to-face intensive EBM education during orientation; 2) longitudinal: unclear length & frequency | 2 weeks (3 half-day clinics per week, with the remainder of the time available for EBM) | 5 x 1-hour monthly research and statistics lectures | 4 x 1 hour weekly sessions | Monthly conference | |||
| Teachers/ facilitators | Faculty, residents | Faculty, medical librarian | Research program coordinator | Teacher with expertise in epidemiology and biostatistics | Librarians, pharmacists, others | |||
| Clinically integrated (lectures versus point of care) | Yes | Yes | No | No | No | |||
| Curriculum content | 1) Introduction to Information Mastery; 2) group exercise: is It a POEM; 3) don't panic: statistics you can understand; 4) power reading of journals; 5) expert- based information delivery systems; 6) diagnostic testing: Bayes' Theorem, evaluating studies of diagnostic tests; 7) evaluation of Grand Rounds using the CME evaluation form; 8) obtaining useful information from pharmaceutical representatives; 9) using Clinical Decision Rules; 10) clinical Jazz: harmonizing clinical experience with Evidence-based Medicine | Critical evaluation of articles about diagnosis, therapy, prognosis, meta- analysis and decision making, and how to ask clinical questions. PICO, formulate questions, generate searches, developing searching strategies, high quality EBM resources, information mastery; essential EBM concepts, including clinical question development, levels of evidence search strategies, and appraisal techniques; critically appraised topics - each CAT is structured to include the 'clinical bottom line' answer to the question | Hypothesis testing, variable types, study designs, clinical trial phases, bias and confounding, odds and risk ratios, sensitivity and specificity, 2 x 2 tables, incidence and prevalence, representative descriptive statistic, standard deviation and measures of variability, multiple comparisons, types of statistical tests, power and sample size calculations, number needed to harm/treat, confounding assessment and adjustment, patient safety and quality improvement methods, national surgical quality improvement program, surgical care improvement project measures | 1) hypothesis testing and exploratory data analysis; 2) confirmatory data analysis; 3) study designs; 4) each resident had to prepare a hypothetical study in advance which would answer a clinical or research question | 1) Introduction and PICO questions; 2) live literature search and analysis of the results; 3) article analysis focusing on assigned key concepts of analyzing different types of articles such as therapy, diagnosis, etc., so that each month focused on a different concept in depth; and 4) article application to patient population; 5) modified journal club format. | |||
| Curriculum development process described | Yes, Kerns six step approach used | Not discussed | Not discussed | Yes, Kerns six step approach used | Yes, Kerns six step approach used | |||
| Outcome measures | Mastery of residents' evidence-based medicine knowledge and skills as well as their confidence at critically appraising medical literature and using evidence to inform clinical decisions. | Level of confidence in basic EBM skills | Improvement in research and statistics ABSITE scores | Assessment of resident performance, assessment of curriculum | Impact that the curriculum changes (including librarian involvement) had on the residents' perceived abilities to integrate knowledge gained from the EBM conference into meaningful literature searches. | |||
| Evaluation method | Fresno Test of Evidence-based Medicine; an attitude questionnaire at the start of the curriculum and then again before graduation | self-assessment of EBM skills (interns) - adapted from Fresno test | ABSITE examinations | 20-item multiple-choice knowledge test; written survey about the curriculum | survey | |||
| Results | Modified Fresno Test scores significantly improved from 104.0 to 121.5. Using a pass/fail approach, nine residents (40.1%) passed the test at the start of training, increasing to 17 (73.4%) at the end of the intervention. Confidence in critical appraisal scores increased from an average 17.90 (95% CI=16.55–19.25) to 21.10 (95% CI=19.49–22.71), out of a possible score of 25. Attitudes regarding confidence in the use of evidence and a decreased reliance on experts were also improved following the curriculum. | Interns significantly increased their confidence over the course of the rotation; scores improved postrotation in all 3 areas tested: EBM terms and concepts 81% to 97%; quantitative skills 51% to 80%; question formulation and searching 71% to 92%, with the total score increasing from 63% to 87%; residents reported applying the EBM skills they learned to patient care (86%) and that these skills were reinforced in the teaching they received outside of the rotation (81%); all residents felt that the EBM curriculum had improved patient care. | Residents demonstrated significant improvement in postcurriculum examination scores for research and statistics items, correct responses increased 27% (p< .001), residents were 5 times more likely to achieve a perfect score on research and statistics items postcurriculum (p< .001). | Mean±SD proportion of correct answers for the 52 course participants was 58±16%; historical controls – 26 residents in the same programme and 277 residents from other residency programmes – scored less well (48±14%, p=0.01 and 41±15%, p<0.0001, respectively). | Overall, residents found the optional technology session helpful, appreciated librarian involvement during EBM conferences, increased their knowledge of library resources, reported improved knowledge and comfort using electronic library resources after the curriculum changes were implemented, and felt that they could integrate knowledge learned during the EBM conference series into meaningful literature searches. | |||
| Author (year) | Al Jalbout et al. (2019) | Gupta et al. (2014) | Kao et al. (2012) | Kluesner (2015) | Merritt (2015) | ||||
|---|---|---|---|---|---|---|---|---|---|
| Study design | Pre-post design | Prospective cohort | Pre-post study | Pre-post study | Pre-post study | ||||
| Country | USA | USA | USA | USA | USA | ||||
| Specialty | Emergency Medicine | Internal Medicine | Internal Medicine | not reported | Internal Medicine | ||||
| Number of Participants, n | 14 | not reported | 11 | 26 | 108 (10-12/session) | ||||
| Study period | 16 months | Not reported | 1 year | 2 years | 6 months | ||||
| EBM Curriculum Structure | 8 structured interactive online modules and a librarian meeting | 5 modules: 1) introduction to EBM; 2) randomized control trials; 3) cohort and case-control studies 4) diagnostic tests. 5) meta-analyses | 2 week block rotation for first and second year residents | Journal Club integrated with EBM principles | Interactive lectures and case discussion | ||||
| Length/frequency | 2 weeks | Not reported | 2 weeks | 18 months | Once a month for 30 minutes | ||||
| Teachers/ facilitators | Librarian | Physician, EBM expert, instructional designer | Internal medicine faculty, librarians | Teaching resident, core EBM faculty | Not reported | ||||
| Curriculum content | Process of EMB, study design, validity, and biostatics, PICO and search strategies | 1) introduction to EBM. 2) randomized control trials. 3) cohort and case control studies. 4) diagnostic tests. 5) meta-analyses | Hands-on sessions to teach literature search and critical appraisal | 1) peer instruction model and peer to peer discussion coordinated by a teaching resident, 2) dedicated EBM lecture delivered at the begining of each journal club, 3) identification of | 1) assessment of risks and benefits of treatment. 2) rational use of diagnostic testing | ||||
| Outcome measures | EBM knowledge and skills | Satisfaction with the course, milestones, correctness of responses, number of attempts, need for remediation | EBM skills and knowledge | EBM skills and knowledge | Comfort in understanding of material | ||||
| Evaluation method | Fresno test | survey | Fresno test; Resident self assessed comfort with EBM resources | Fresno test administered at the end of each year | comfort via 5pt Likert scale; critical risk interpretation test (CRIT) | ||||
| Results | 1) Fresno median pre-test 80, median post-test 103 (p=0.002). 2) PICO question (p=0.009), internal validity (p=0.018), study design (p=0.004); no significant improvement among search strategy or biostatistics knowledge | 97% of residents were satisfied or very satisfied with the course | 1) Fresno mean pre-test 114.3, mean post-test 154.6 (p<0.001). 2) Likert scale (p<0.001) for: searching and using MeSH terms, advanced limits, subheadings in Ovid, and conducting and presenting a literature review | Fresno pretest 110.16, year 1 posttest 127.82, year 2 posttest 127.07, (p=0.011) | 1) "furthered understanding of how to practice EBM" 4.6/5 of Likert. 2) "use skills learned in these sessions during regular practice" 4.5/5 on Likert | ||||
| Author (year) | Moll (2012) | Mroz and Carroll (2019) | Schwartz et al. (2000) | Vesbianu and Rodriguez (2018) | Vogel (2000) | ||||
| Study design | Pre-post study | Prospective cohort | Pre-post study | Pre-post study | Pre-post study | ||||
| Country | USA | USA | USA | USA | USA | ||||
| Specialty | Emergency Medicine | Internal Medicine | Primary Care | internal medicine | internal medicine | ||||
| Number of Participants, n | not specified | 68 | 14 | 30 | 42 | ||||
| Study period | 3 years | 1 year | 6 weeks | 6 months | One year | ||||
| EBM Curriculum Structure | Modules | Interactive lectures and problem based learning | Self-reported critical appraisal on 13 items, and self-reported electronic searching skills on 17 items | Small group ambulatory didactics | Didactic, with residents giving presentations at the end of the course | ||||
| Length/frequency | Four 6-month modules | Six 1-hour sessions over one year | 60 hours over 6 weeks | 1 hour sessions every week | 8 weeks | ||||
| Teachers/ facilitators | Faculty | Not reported | Not reported | Faculty | Unclear | ||||
| Curriculum content | 1) introduction and tools needed; 2) small groups formulate a clinical question, research and rate literature; 3) implementation module, knowledge translation and set | 1) recognition of overconfidence and understanding probability; 2) test interpretation; 3) study design; 4) asking a clinical question; 5) abstract review; 6) feedback and skills | 1) critical appraisal 2) electronic searching skills | 1) appraise and understand medical literature. 2) incorporate learned EBM knowledge into patient care | 1) developing clinical questions from patient encounters 2) finding best- quality information sources 3) critical appraisal | ||||
| Outcome measures | EBM sources, knowledge translation tool use and retention | EBM skills and knowledge; opinions on the new curriculum | Attitudes towards EBM curriculum | Level of confidence | Survey of EBM attitudes and abilities, computer searching skills, opinions on curriculum, confidence in skills learned | ||||
| Evaluation method | self-reported questionnaire | Fresno, resident evaluation | survey, 6-item 4pt nihilism scale to address distrust of research methods and applicability of research articles | self-reported EBM skills questionnaire | self-reports, surveys, and written tests | ||||
| Results | EBM sources are the first query in 67% compared to 50% at the start, and 63% self-identify a change in how they obtain information | 1) Fresno post-test 124; 2) resident course rating 7.8 (SD 1.8) on a 10 point Likert scale | 1) critical appraisal before study 71%, after study 85%, (p=0.002). 2) electronic searching skills before study 68%, after 83%, (p=0.001). | perceived level of confidence improved from 3.62 to 3.89, (p=0.05) | 1st written pretest 49% posttest 86%, 2nd written pretest 42% posttest 57%. (p<0.001) for each | ||||
| Author (year) | Vom Eigen (2002) | Vu et al. (2017) | Walkey and Fairchild (2006) | Zipkin et al. (2010) | |||||
| Study design | Pre-post study | Pre-post study | Pre-post study | Pre-post study | |||||
| Country | USA | USA | USA | USA | |||||
| Specialty | Primary Care Internal Medicine | Pediatric | primary care | Internal Medicine | |||||
| Number of Participants, n | 46 | 23 | 21 | 18 | |||||
| Study period | 3 years | 1 year | Not reported | Not reported | |||||
| EBM Curriculum Structure | interactive didactic small groups | Small groups, didactics | Didactic, interactive | Small, interactive groups | |||||
| Length/frequency | 10-12 weekly sessions for 1 hour over the course of the 3 month curriculum | Monthly conferences | 1.5 hr lecture on didactics followed by 15 min residents presentation on | 4 half-day sessions over one week | |||||
| Teachers/ facilitators | Instructor | Residents and faculty | More senior residents residents | Not reported | |||||
| Curriculum content | Biostatics, informatics, clinical epidemiology, patient care issues, search for relevant research articles, discussion lead by instructor | Formulate a clinical question, selected and appraised a single article, present an overview, and facilitated discussions | 1) didactic session, sample patient encounters. 2) patient contact, generating and question and presentation based on encounter. 3) opportunity to teach other residents | Understanding study design, bias and random error, diagnostics, screening, treatments, harm, and prognostics | |||||
| Outcome measures | Pre/post test of knowledge as well as self-evaluated changes in clinical knowledge, skills, practice, and ability to evaluate medical literature | level of EBM education, ability to appraise articles on Likert scale | Ability to form clinical questions | Knowledge, quality of education, written feedback | |||||
| Evaluation method | 38 item test | Berlin Questionnaire and Assessing Competency in EBM (ACE) | adapted Fresno test (72 pt scale) | 25 point multiple choice test | |||||
| Results | mean score of pre and post tests improved from 65.3% to 73.7% (p<0. 001). | 1) Berlin Questionnaire pretest 63%, posttest 76%, (p<0.001). 2) ACE pretest 63%, posttest 67%, (p=0.27). 3) 2.9/5 Likert for quality of education in EBM during medical school. Rated ability to appraise articles pretest was 2.8 and posttest was 3.2 on Likert (p=0.03) | 1) Fresno pre/post scores increased by 55% (p<0.0001). 2) key subsections: ability to identify EBM sources (120% increase in correctness, p<0.0001) and ability to form questions in PICO format (36% increase, p=0.03) | pretest mean 14.17 (SD 4), posttest mean 17.11 (2.4), (p=0.008) | |||||
Eighteen studies had a moderate risk of bias, and six studies had a low risk of bias. After applying the NOS criteria, 2 of the four observational prospective cohort studies were assessed as having a low risk of bias
Details regarding the EBM curriculum structure, length, frequency, teachers, content, outcome measures, evaluation method, and results can be found in
The EBM curricular structure was found to be highly variable from study to study. EBM was taught through multiple venues: workshops,
The majority of the EBM curricula was longitudinal with different lengths, from weekly 1-hour sessions for 4 weeks
The majority of the reported EBM curricula address all five of the EBM competencies.
Kern's six-step approach to design an experiential curriculum in knowledge translation has been a well-established model for the curriculum development process that has been shown to lead to successful implementation and long-term sustainability.
We defined clinical integration of the EBM curriculum if the EBM principles, steps, or concepts were taught in the context of patient care. The EBM curriculum was clinically integrated into 29.3% (10 out of 29) of the full-text articles. Ten percent of the EBM curricula were only partially clinically integrated
Most of the EBM curricula reported on level 2 (EBM skills and knowledge), 45% (13 out of the 29 full-text articles) and 57% (8 out of the 14 abstracts). Fewer articles reported on level 1 (attitude toward EBM and participants' satisfaction): 17% (5 out of 29 full-text studies) and 35.7% (5 out of 14 abstracts). A total of 10 articles (9 full texts and one abstract) reported on both level 1 and level 2. We found 2 full-text articles that also reported on level 3 (behavior, the degree to which participants apply what they learned during training when they are back on the job). Kortekaas and colleagues
From the 29 full-text articles, only 4 used the validated Fresno
Barriers reported during the EBM curriculum implementation were categorized into three main subsets: barriers to teach, barriers to learning, and barriers to practice EBM. (
| Barriers to teach | |
|---|---|
| - Learners not completing pre-reading48 | |
| - Lack of dedicated teaching time13,44,46,60 | |
| - Learner fatigue with repetition52 | |
| - Varying resident abilities/knowledge40 | |
| - Difficult to recruit teachers32,46,53,60 | |
| - Lack of consensus on teaching strategies/curriculum topics13 | |
| Barriers to learning | |
| - Learner fatigue with repetition52 | |
| Barriers to practice | |
| - Time restraints10,27,48,50 | |
| - Lack of knowledge10,48 | |
| - Lack of access to resources48,50 | |
| - Subjects not taught frequently enough38 | |
| - Poor search skills and database knowledge13,42,5 | |
The heterogeneity of the studies interventions and outcome assessments precludes any attempt to combine the results quantitatively. All the published studies and abstracts, independent of the EBM curriculum structure and the evaluation method used, found an improvement in the residents' attitudes and/or EBM skills and knowledge. Bentley and colleagues
We reviewed a total of 43 articles, from which 29 full-text articles and 14 abstracts reporting on the EBM curriculum. Overall the body of evidence regarding EBM curriculum structure, efficacy, and barriers is small and of limited quality. The studies included were considerably heterogeneous with respect to their EBM curriculum structure, content, length, evaluation method, and outcome assessment. A common finding within the studies was that multiple teaching strategies were implemented simultaneously. All studies showed an improvement in educational outcomes across different EBM curricula. However, no specific curriculum was found to be superior because there was no direct comparison.
The curricular structure in each article included was described in adequate detail that would allow reproduction and generalizability. This is in contrast with what Green and colleagues
Moving EBM instruction out of the classroom and into the clinical arena is crucial for the application.
Our findings are consistent with Green and colleagues
Very few studies reported on the Kirkpatrick' evaluation model level 3, behavioral change, and no studies reported on level 4, results/patient outcomes. These findings are similar to previous systematic reviews data on medical education interventions.
Ilic and colleagues
Time has been mentioned as the most frequent barrier to teach and practice EBM. This finding is consistent with prior literature. Dijk and colleagues
The strengths of our scopic review include systematic reproducible search of the literature, explicit methods which limit bias, reliable and accurate conclusions and the ability to bring a vast body of literature to an update review. One of the main limitations of this review is the likely publication bias. Because negative studies are less likely to be written, submitted or published, especially in medical education, the studies' results might be skewed toward reporting the positive effects of the interventions. We also noticed that the published data on EBM curriculum is geographically limited and thought that it might be due to publication bias and might further bias the results by not fully representing the EBM curricula from many other countries. It is also possible that there are many innovative and effective EBM curricula that have never been reported. Another major limitation is the quality of the study design of the full manuscripts included. The heterogeneity of the interventions and outcomes constitutes a limitation in the ability to draw a conclusion based on this data. One other limitation worth noting is the small sample size in the majority of these studies, as a convenience sample was most often used, and the studies were single centre—none of the studies reported on patient care impact assessment.
This is the first scoping review to answer the question regarding the structure of EBM curricula and barriers found during the EBM curriculum implementation process. EBM competencies are necessary for providing high-quality patient care within physicians in training education. The current body of literature available to guide educators in EBM curriculum development and barriers to implement and teach EBM is enough to constitute a strong scaffold for developing any EBM curriculum. Teaching EBM concepts can use any method and can be integrated into almost any setting. Special attention should be paid to the potential barriers during the curriculum implementation, and ways to overcome them should be identified early in the curriculum development process. Given the amount of time and resources needed to develop and implement an EBM curriculum, it is imperative to follow the curriculum development steps and use validated assessment tools. During that development, medical educators should design rigorous evaluation strategies with more meaningful clinical outcomes that would evaluate Kirkpatrick's learning levels 3 and 4. Further studies should focus on studying long term retention of the EBM concepts, skills and knowledge as they remain unknown for now.
The authors declare that they have no conflict of interest.
Appendix 1. Quality of Pre/Post studies
Appendix 2. Quality of prospective cohort studies and randomized controlled trials