Critical aspects in developing curriculum-based assessment for emerging surgical procedures

Introduction

The past century has seen an enormous rise in new surgical procedures. The general surgery landscape has changed markedly, from predominantly open procedures to laparoscopic and now robotic surgery.  Surgical training has been far slower to adjust to change, as evidenced by the haphazard method of introduction of laparoscopic surgery.  The introduction of laparoscopy was marked by procedures being performed by surgeons without a structured fellowship training program, in addition to questionable legitimacy of procedures due to the lack of supportive data.¹

New surgical methods and new technologies, such as robotic surgery, have been accompanied by a greater emphasis upon educational efforts prior to surgical "practice".  Capabilities such as dual consoles to maximise the teaching potential of the platform in addition to exposure during training lends some recognition to this modern requirement.² Whilst adjuncts such as certified proctorship and simulation exposures have arisen, there remains a need for academic rigour via validity testing to ensure robust and defensible curricula are developed.³ This is especially important when the curriculum articulates into an assessment tool with a rating scale and associated rubric.

To date, the most common methods of assessing a surgeon for accreditation by an institution have relied upon surrogate competency measures. The two most common are Years of experience^4,5 and Case volume-based measures. 6-8 Both may be problematic, especially when assessing surgeons who wish to undertake new procedures as time in practice do not necessarily equate to transferability of skillset; likewise, volume-based criteria require some degree of overlap of skill from one surgical procedure to another.  The use of simulator-based teaching and assessment shows great promise in its ability to be used for de-identified, reproducible and but the method by which curricular documents and subsequent assessment matrices are developed is not clearly defined.  In this paper, we describe a number of critical considerations in creating credentialing guidelines for surgeons in novel or emerging procedures.

Novel procedures and emerging technologies

Refinement of existing surgical procedures is ubiquitous by surgeons and device technology developers alike. However, the past half-century has seen an unprecedented rise in new operations. In general surgery, the development of laparoscopy has led to a legion of new ways to perform existing procedures and a paradigm shift in skillset, almost unrecognisable to that of a traditionally-trained "open" surgeon.  In hindsight, the methodology of learning laparoscopic surgery was at best haphazard; at worst, dangerous with a concomitant rise in morbidity and mortality.

Similarly, the introduction of robotic surgery has shown benefits of shortened hospital stays and complication rates, tempered by increased intraoperative times, cost and increased training burden of the new technique.⁹

The belated introduction of structured courses such as Fundamentals of Laparoscopic Surgery course¹⁰ followed the more routine use of laparoscopy in gynaecology and general surgery, such that it is now the normative approach worldwide. More recently, the introduction of robotic surgery had been accompanied by a greater awareness of the difficulties in introducing surgeons to new technology requiring novel skills and an increased reliance upon simulation in addition to proctorship for training and credentialing.¹¹

Curriculum processes – modified Delphi methodology

Proceduralists benefit from standardised, validated curriculums for teaching competence focused upon specific skills.¹² There are a number of methods wherein a structured curriculum for teaching a new skill or procedure can be developed.  The most ubiquitous is the "expert opinion" method, with variations upon the rigour with which it is created. The older "hot-tubbing" approach involved a group of experts discussing and creating a curriculum within a set time frame.  The more formal, validated approach requires a defined expert-level opinion formed via an iterative forecasting method via multiple rounds seeking consensus, also known as the Delphi method.  For example, utilisation of the Delphi process to develop a proficiency-based progression course in training for robotic surgery.¹³ Benefits of the Delphi approach include the validation of a formal consensus opinion as well as the statistical significance of reaching consensus via Cronbach's alpha. However, in addition to defining the curriculum, a further demonstration of the use of a skills curriculum over existing teaching paradigms is required.¹⁴ A second limitation is the lack of a readily accessible group of experts when the technology is only newly developed or a novel surgical procedure wherein the number of experts may not be enough to utilise Delphi methodology¹⁵; thus, the option of informal expert consensus is likely to continue in future with caveats in place.

Content validity

The creation of a formal, structured curriculum for a novel surgical procedure must be adjusted to suit the local environment.  The curriculum must be structured to suit local nuances of device availability, preference of technique, and patient demographic. Validating the content of a curriculum towards a particular region is often conducted via Delphi methodology similar to above, with the expert group derived entirely from the defined area.^16,17 In doing so, the teaching method and assessment are more relevant for the country or region with greater applicability when extended into the credentialing environment.  However, with greater specificity, there is a loss of generalisability; the potential for bias dependent upon ethnicity and place of qualification must be recognised by the assessor.¹⁸ Such deficiencies in procedure would need to be noted by the credentialing body.

Construct validity

One of the most powerful aspects of a validated curriculum is the ability to discern a novice from an expert surgeon. The definitions of these are highly debatable and is most notably determined by surrogate competency markers such as case-volume load or time-based (years in practice).  There is a move towards standardised, reproducible content assessment, which often requires the use of a simulator.  Via direct observation or retrospective video review of a surgical operation upon a simulator, a trained assessor can rate them upon predefined metrics, utilising construct validity to determine whether the assessment score relates to the allocated group of experts or surgeons.  The preferred approach of a simulator removes the requirement for statistical adjustment for patient comorbidities such as body habitus, previous surgery or adhesions.  However, its transferability to "real-life surgery" is an extra consideration that often needs to be validated prior to use as a high-stakes assessment methodology. Defining criteria for credentialing to determine "expert status" utilising consensus methodology has been applied in robotic¹⁹ and laparoscopic surgery.  Development of learning curve phases from "competent" to "proficient", then "mastery" of procedures²⁰ allows for a more nuanced comparison of the level of expertise.

Conclusions

The determination of a curriculum-based assessment for novel surgical procedures is an exemplary modality of reproducible, quality assessment.  The rise of high-stakes assessment requires greater rigour in the development of the underlying curriculum with which surgeons are being trained to ensure competency upon the simulator as well as ensuring patient safety. Our paper describes the key educational and clinical concerns regarding the development of construct and content validity as well as components of transferability for assessment.  Future assessment of surgeons in novel surgical procedures will likely require credentialing bodies to be cognizant of similar considerations of defensible curricular and assessment development.

Conflict of Interest

The authors declare that they have no conflict of interest.

References

1. Rock JA and Warshaw JR. The history and future of operative laparoscopy. American Journal of Obstetrics and Gynecology. 1994; 170: 7-11.
   Full Text
2. Zhao B, Lam J, Hollandsworth HM, Lee AM, Lopez NE, Abbadessa B, Eisenstein S, Cosman BC, Ramamoorthy SL and Parry LA. General surgery training in the era of robotic surgery: a qualitative analysis of perceptions from resident and attending surgeons. Surg Endosc. 2020; 34: 1712-1721.
   Full Text PubMed
3. Eppich W, Howard V, Vozenilek J and Curran I. Simulation-based team training in healthcare. Simul Healthc. 2011; 14-19.
   Full Text PubMed
4. Novais EN, Carry PM, Kestel LA, Ketterman B, Brusalis CM and Sankar WN. Does Surgeon Experience Impact the Risk of Complications After Bernese Periacetabular Osteotomy? Clin Orthop Relat Res. 2017; 475: 1110-1117.
   Full Text PubMed
5. Abdullah F, Salazar JH, Gause CD, Gadepalli S, Biester TW, Azarow KS, Brandt ML, Chung DH, Lund DP, Rescorla FJ, Waldhausen JH, Tracy TF, Fallat ME, Klein MD, Lewis FR and Hirschl RB. Understanding the Operative Experience of the Practicing Pediatric Surgeon: Implications for Training and Maintaining Competency. JAMA Surg. 2016; 151: 735-741.
   Full Text PubMed
6. Aspinall S, Oweis D and Chadwick D. Effect of surgeons' annual operative volume on the risk of permanent Hypoparathyroidism, recurrent laryngeal nerve palsy and Haematoma following thyroidectomy: analysis of United Kingdom registry of endocrine and thyroid surgery (UKRETS). Langenbecks Arch Surg. 2019; 404: 421-430.
   Full Text PubMed
7. Pace D, Borgaonkar M, Evans B, Marcoux C, Lougheed M, Falk V, Hickey N, O'Leary M, McGrath J and Boone D. Annual colonoscopy volume and maintenance of competency for surgeons. Surg Endosc. 2017; 31: 2630-2635.
   Full Text PubMed
8. Doll KM, Milad MP and Gossett DR. Surgeon volume and outcomes in benign hysterectomy. J Minim Invasive Gynecol. 2013; 20: 554-561.
   Full Text PubMed
9. Hussain A, Malik A, Halim MU and Ali AM. The use of robotics in surgery: a review. Int J Clin Pract. 2014; 68: 1376-1382.
   Full Text PubMed
10. Ritter EM and Scott DJ. Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov. 2007; 14: 107-112.
    Full Text PubMed
11. Zorn KC, Gautam G, Shalhav AL, Clayman RV, Ahlering TE, Albala DM, Lee DI, Sundaram CP, Matin SF, Castle EP, Winfield HN, Gettman MT, Lee BR, Thomas R, Patel VR, Leveillee RJ, Wong C, Badlani GH, Rha KH, Eggener SE, Wiklund P, Mottrie A, Atug F, Kural AR and Joseph JV. Training, credentialing, proctoring and medicolegal risks of robotic urological surgery: recommendations of the society of urologic robotic surgeons. J Urol. 2009; 182: 1126-1132.
    Full Text PubMed
12. Grantcharov TP and Reznick RK. Teaching procedural skills. BMJ. 2008; 336: 1129-1131.
    Full Text PubMed
13. Collins JW, Levy J, Stefanidis D, Gallagher A, Coleman M, Cecil T, Ericsson A, Mottrie A, Wiklund P, Ahmed K, Pratschke J, Casali G, Ghazi A, Gomez M, Hung A, Arnold A, Dunning J, Martino M, Vaz C, Friedman E, Baste JM, Bergamaschi R, Feins R, Earle D, Pusic M, Montgomery O, Pugh C and Satava RM. Utilising the Delphi Process to Develop a Proficiency-based Progression Train-the-trainer Course for Robotic Surgery Training. Eur Urol. 2019; 75: 775-785.
    Full Text PubMed
14. Satava RM, Stefanidis D, Levy JS, Smith R, Martin JR, Monfared S, Timsina LR, Darzi AW, Moglia A, Brand TC, Dorin RP, Dumon KR, Francone TD, Georgiou E, Goh AC, Marcet JE, Martino MA, Sudan R, Vale J and Gallagher AG. Proving the Effectiveness of the Fundamentals of Robotic Surgery (FRS) Skills Curriculum: A Single-blinded, Multispecialty, Multi-institutional Randomized Control Trial. Ann Surg. 2020; 272: 384-392.
    Full Text PubMed
15. Gordon AG and Magos AL. The development of laparoscopic surgery. Baillieres Clin Obstet Gynaecol. 1989; 3: 429-449.
    Full Text PubMed
16. von Buchwald JH, Frendø M, Guldager MJ, Melchiors J and Andersen SAW. Content validity evidence for a simulation-based test of handheld otoscopy skills. Eur Arch Otorhinolaryngol. 2021; 278: 2313-2320.
    Full Text PubMed
17. Angelo RL, Ryu RK, Pedowitz RA and Gallagher AG. Metric Development for an Arthroscopic Bankart Procedure: Assessment of Face and Content Validity. Arthroscopy. 2015; 31: 1430-1440.
    Full Text PubMed
18. Tiffin PA and Paton LW. Differential attainment in the MRCPsych according to ethnicity and place of qualification between 2013 and 2018: a UK cohort study. Postgrad Med J. 2021; 97: 764-776.
    Full Text PubMed
19. Stefanidis D, Huffman EM, Collins JW, Martino MA, Satava RM and Levy JS. Expert Consensus Recommendations for Robotic Surgery Credentialing. Ann Surg. 2020; .
    Full Text PubMed
20. Wehrtmann FS, de la Garza JR, Kowalewski KF, Schmidt MW, Müller K, Tapking C, Probst P, Diener MK, Fischer L, Müller-Stich BP and Nickel F. Learning Curves of Laparoscopic Roux-en-Y Gastric Bypass and Sleeve Gastrectomy in Bariatric Surgery: a Systematic Review and Introduction of a Standardization. Obes Surg. 2020; 30: 640-656.
    Full Text PubMed