Continuing medical education (CME) conferences are an integral part of health care. CME conferences are widely regarded as essential by clinicians, trainees, and the patients they serve as they support critical activities such as knowledge exchange, networking, and scholarly initiatives like research.^1-4 The importance of CME conferences is further highlighted by their prominence in physician maintenance of certification⁴ and correlation between lack of opportunities to attend conferences and increased risk of burnout and feelings of inadequate knowledge or isolation.⁵  As an example, in a longitudinal study of emergency physicians opportunity to attend conferences was associated with a 3 times lower risk of burnout.⁵ Given the significance to health care and academia, there has been rapid growth and global expansion in the conference industry over the past century, with some estimates suggesting hundreds of thousands of events hosted globally each year.^6-8 While this growth has benefits, it also presents significant downsides, including substantial time and financial investments (organizers and attendees) with increasingly recognized environmental consequences. A study of a single mid-sized American conference estimated that more than 10 000 tonnes of carbon dioxide were generated by air travel alone – equivalent to the annual amount produced by 550 US citizens.⁹ Moving forward, it is critical the field consider the costs and environmental impact of conferences, and strive to maximize value to attendees, patients, and the healthcare system.

Despite their importance and cost, there is no standardized means for conference evaluation, leading to several issues. First, with such a large number of conference options available within all specialities, attendees have no objective means of knowing which conferences provide the greatest value and/or best suit their individual needs.  Available evidence suggests the approach to conference design and implementation can significantly influence impact.  As an example, multiple studies show conferences that utilize both interactive and didactic seminars have more learning when compared to solely didactic or interactive seminars.^{1, 10-13} The lack of standardized evaluation tools makes it challenging for academic and industry researchers to demonstrate and quantify the value of new approaches, innovations and technologies. Consequently, conference organizers and their financers must make decisions about how to spend the limited resource (time and money) when designing their conference without access to this data.

To begin addressing the gap in high quality conference evaluation methodology, we sought to perform a scoping review of the published research evaluating CME conferences. Objective one was to comprehensively identify research studies evaluating conference experience, with the goal of identifying and examining the tools and frameworks utilized. Objective two was to compile a repository of frequently observed evaluation domains and subdomains based on information extracted from the studies. The findings of this scoping review will be of immediate use to individuals or organizations seeking to design or evaluate a conference and represents a critical first step in developing a standardized tool for conference evaluation.

We prepared a scoping review protocol guided by established methodology¹⁴ and published the protocol on Open Science Framework 04-May-2021. The project was completed at a tertiary care pediatric hospital associated with the University of Ottawa (Ottawa, Canada).  Results are reported according to the PRISMA Scoping Review checklist (see supplemental digital appendix 1).

This scoping review explored the published literature on CME conference evaluation with the goals of identifying validated instruments and relevant evaluation domains through content analysis. This work identified 83 studies originating from a range of medical fields, but no broadly applicable validated tools. While inspection of individual studies demonstrated that only a small minority described following recommended methodology for survey development (for instance, pilot testing), the extraction and analysis of over 600 individual items allowed for the identification of several domains and subdomains directly useful to future research in this area.

As expected, conference evaluation research was confirmed to be of widespread interest, spanning over two dozen medical fields and originating worldwide. While interest was widespread, the field of radiology and diagnostic imaging produced three times as many publications (n=10) as the average in all other fields (n=3). The higher volume may be linked to interventional radiology’s (IR) recent recognition as a primary specialty by the American Board of Medical Specialties in 2012 and need to recruit trainees into dedicated IR residencies.⁵⁹ This is consistent with the observation that all 10 of the radiology studies reported on conferences specifically held for trainees, with several tracking conference attendance over time and student attraction to the program.

Analyzing study location identified that the majority of studies (72%) originated from North America. This proportion may be explained in part by study methodology factors (decision to only included English and French articles) or regional/cultural differences in approach to scholarship/publication or continuing medical education (CME); CME and professional development are highly regulated within Canada, the United States, and most Western European and Australasian countries, but vary more globally in terms of policies, infrastructure, and enforcement.^{2, 102-104} Additionally, the reality of ‘conference inequity’ – where most global health conferences (and therefore evaluations on these) are held in higher income countries¹⁰⁵ – is also borne out here, as only 16% of studies came from outside of North America and Europe. While the publication rate appeared to plateau or only minimally grow between 2014-2019, at an average of seven per year, a spike was observed in 2020 to 11 publications. This spike timed with the onset of the COVID-19 pandemic, and as shown in Figure 2, these studies focused on assessing the impact of the necessary transition from in-person to virtual conferences. Although not implemented on a global scale prior to COVID-19, virtual conferences and webinars have been an important component of medical education prior to COVID-19 with proven success in knowledge transfer.

For instance, studies have shown improved standardized test scores following virtual lectures provided by first-world academic institutions to smaller hospitals in developing countries.^106,107 While investigating the specific impact of virtual conferences is outside the scope of this research, seven of the 12 publications that evaluated virtual conferences reported attendee assessment as positive (i.e., the majority of participants reported equivalent or higher preference for the virtual format).^{60,61, 68, 71, 79, 81, 94} This was attributed to improved attendance, greater accessibility, and decreased environmental impact. Of the remaining publications, 3 of 12 did not specifically ask the participants about format preference, although respondents indicated that they had enjoyed the conference and were willing to continue meeting virtually;^{42, 89,96} 1 of 12 reported a majority preference for in-person meetings;⁴⁴ and the final study reported approximately equal preference for in-person and hybrid/virtual.⁸²

Some studies attempted to evaluate conference impact using objective metrics – such as examining links between attendance and performance on the American Board of Emergency Medicine In-Training Examination and U.S. Medical Licensing Examination,³⁹ or distributing case study questionnaires to conference participants and non-participants to determine “whether the diagnostic and therapeutic choices of program participants were consistent with evidence-based guidelines”.³⁶ Of the studies that used surveys, only one⁹⁷ described using validation processes such as iterative revisions, factor analysis, and Cronbach’s alpha methods to assess internal consistency.^108,109 While of clear value, this tool may have limited general applicability as it was specifically designed to measure a mobile device app’s impact on conference experience. In the absence of validated tools, some of the studies (n=13) sought out and described the consideration of previously published conference evaluation frameworks as part of tool development. Finally, only two studies^{42, 54} mentioned performing any tool refinement or pilot testing prior to implementation, widely considered essential steps in survey development.^110,111 Despite the inability to formally assess the quality of each instrument included in our review, the lack of validity evidence supporting these instruments raises concerns about their methodological quality, as do other aspects of our general quality assessment (such as response rate reporting and clear sample population descriptors).

Our content analysis identified eight major evaluation domains. The traditional conference format is geared toward bringing individuals together, usually physically, for the purpose of shared learning – so the observed heavy weighting in these domains as well as in satisfaction and logistics supports the assumption that evaluation weighting parallels conference goals. This format often leads to new mentorship and professional development opportunities for those who attend, and there are well-documented challenges for those who do not or cannot attend.^105,112-115 The four domains identified inductively addressed value-satisfaction, logistics, EDI, and career influences. Items assessing logistics and EDI were primarily found in more recent publications. More recent studies also tended to highlight concerns surrounding in-person conferences, such as the environmental impacts and attendance inequity. Both the identified studies and broader literature suggest factors like funding, inability to travel to conference location, limited speaking opportunities/representation, family/clinical commitments, and intrinsic feelings of belonging as barriers that disproportionately affect in-person conference attendance of women, minorities, and residents of lower income countries.^{32,47,98,105,116-118} Items addressing conference environmental impact and gender-related conference inequity were primarily found in studies^42,98 published after 2019, indicating these to be emerging priorities within the scientific community. Virtual conferences have the potential to reduce environmental impacts and provide more equitable and convenient opportunities for networking, learning, and collaboration to all attendees.

Patients and caregivers are another group for whom inclusion has been a growing priority and seven studies within this review specifically included these individuals as stakeholders, potentially reflecting the growing importance their inclusion in conference planning and implementation has on preventing discrepancies between patient and health professional priorities.^38,41,63,69 Patient and caregiver conference participation avenues varied, ranging from being the primary audience for improved education and involvement in medical and scientific discussions,^63,101 to inclusion as planners and speakers to better incorporate their feedback into research, health care, and policy.^38,41 This trend reflects a similar shift in broader health care and research toward patient inclusion.^119-121 While this is demonstrably valuable and multiple organizations (e.g., Stanford Medicine X, Patients Included, European Patients Forum) have created charters for ideal methods of inclusion in conferences, further discussion within the medical community of how to meaningfully incorporate patients and caregivers from an EDI standpoint is warranted. The Stanford Framework for Patient Partnership, which was written to guide patient inclusion in CME conferences and “could also be used by prospective delegates to evaluate conferences they are contemplating attending,”¹¹⁹ suggests that accommodation, co-design, engagement, and education and mentorship should be guiding principles in meaningful inclusion.

This scoping review has strengths and limitations to be considered. One major strength is our application of a widely-accepted methodological framework^14,122 for conducting scoping reviews. Through this approach we were able to thoroughly capture trends in CME conference evaluation research including the recent emergence of EDI, environmental concerns, logistics and patient/caregivers as important considerations. One major study limitation was our search restriction to MEDLINE®, deemed necessary given the absence of terms related to congress or conferences in other relevant databases. As recommended for difficult-to-search topics (in this case by the research topic and feasibility of a primary database search¹²³), we used ancillary search methods and, in particular, citation searching.¹²⁴ While limiting to a single electronic database may have reduced the number of eligible studies included we anticipate it to be without major effect as only 5 additional eligible citations were identified through citation searching that were not in our original MEDLINE® search, and these were conference abstracts, yielding little reliable evidence. A second potential limitation was the inclusion of only English and French articles, which may have reduced the number of conference evaluation studies outside of North American and Europe, and potentially limit generalizability to other regions and cultures.

Through this scoping review we were able to map the published conference evaluation literature across many medical fields. This review did not identify a validated tool intended for conference evaluation, which suggests that organizers and research teams are developing their own instruments. While formal quality assessment was not performed, general quality assessment indicated that while study methodology was strong, tool development and recruitment techniques/reporting were weaker. This work confirmed the use of longstanding evaluation domains (e.g., education, networking) and revealed newer domains (e.g., EDI, found in studies published in 2019 or later) used in conference evaluations. The identification of domains, subdomains, and their relative weight may be useful to researchers seeking to evaluate future conferences, and to conference organizers to inform objectives, activities, and select indicators of success and impact. Additionally, by identifying widely-used domains (and subdomains) as well as trends in in-person vs virtual conference format, and by creating a database of sample items, this work helps set the stage for future projects aimed at developing more standardized evaluation instruments which can ultimately improve conference quality.