Predictors of success on the written anesthesiology board certification examination

Objectives: To determine whether information available prior to and during anesthesia residency training can predict American Board of Anesthesiology (ABA) Part 1 (Written) performance and help identify anesthesia residents at risk for failure. Methods: Retrospective analysis of 97 anesthesiology residents’ academic files at a single Midwestern residency program. ABA Part 1 score was used as the dependent variable. Categorical demographic predictor variables included gender, under-represented minority status, and type of medical degree. Quantitative academic predictor variables included first-attempt scores on United States Medical Licensing Examination (USMLE) Steps 1 and 2, annual ABA In-Training Exam (ITE) scores, and biannual Basic Science Examination (BSE) scores. Predictors that correlated significantly with the dependent variable were entered into stepwise linear regression analyses. Results: In our analyses, only the USMLE Step 2 score was a significant pre-residency predictor (B=0.169; SE=0.031, p<0.001). Scores on ITEs were significant predictors at every time point. In the final regression model using all available measures during residency, the most recent ITE (B = 0.627; SE=0.090; p<0.001) and BSE (B=0.096; SE = 0.047; p=0.046) scores were significant predictors. The model accounted for 52% of the variance in the outcome variable, with 50% of the variance explained by ITE score alone. The model had a specificity of 0.83 and a sensitivity of 0.82 for predicting passing the ABA Part 1 exam. Conclusions: First-attempt scores on the USMLE Step 2 examination and the CA-2 ITE are moderately strong predictors of anesthesiology board examination performance, and may help target residents who are at risk of


Introduction
Passing the American Board of Anesthesiology (ABA) Part 1 (Written) examination is one requirement for obtaining board certification in anesthesiology.Anesthesiology residency in the United States is four years in duration, generally consisting either of one year as a preliminary or transitional intern followed by three years of focused anesthesiology training, or as a four-year program, with non-anesthesia rotations comprising the vast majority of the first year.The ABA Part 1 examination is taken following completion of these four years of residency training.Candidates who fail this examination must retake it and often feel compelled to invest in supplemental board preparation courses or materials.Failing the ABA Part 1 examination may also affect their career plans or delay advancement.Passing the Part 1 examination is required in order to register and sit for the Part 2 board examination, which is an oral examination that typically occurs several months to over a year after residency completion.Additionally, low pass rates can negatively impact a residency training program's ability to maintain accreditation through the Accreditation Council for Graduate Medical Education (ACGME) and compromise its ability to attract competitive medical students applying for residency.Therefore, identifying predictors of success on the ABA Part 1 examination Correspondence: Michael D. Altose, Department of Anesthesiology and Perioperative Medicine, University Hospitals Case Medical Center/Case Western Reserve University School of Medicine, USA.Email: Michael.Altose@UHhospitals.org would be valuable, in order to help identify residents at risk of failure and provide them with additional support and educational intervention.Predictors of the Part 2 examination have been more elusive, as it is quantitatively and qualitatively very different from the Part 1 examination.In addition, since the Part 2 examination is taken at least several months after residency is completed, the residency program usually does not receive the results of their residents' Part 2 exam scores.
Residents' demographic information and scores on various quantitative academic assessments are available both prior to and during residency training; these data may be useful as predictors of a passing score on the ABA Part 1 examination.Gender has not been shown to be a reliable predictor of specialty certification in other specialties, 1,2 though a recent study with anesthesiology residents showed that men performed better than women on the ABA Part 1 examination. 3In that same study, graduates of allopathic medical schools who completed their third resident year of clinical anesthesia between 2002-2004 and who took the ABA Part 1 examination the same year that they graduated performed better than graduates of osteopathic medical schools on the ABA Part 1 examination. 3Other studies have suggested that members of under-represented minority (URM) groups may have more difficulty in medical school and on the United States Medical Licensing Examinations (USMLE), 4,5 and this phenomenon may also apply to board specialty examinations.[8][9][10][11][12][13][14][15][16][17] However, one study involving ophthalmology residents failed to find a correlation between USMLE Step 1 and the ophthalmology specialty board examination. 18uantitative academic assessments of resident performance are typically collected during the course of residency.These measures may also serve as useful predictors of performance on the ABA Part 1 examination.11][12][13][14][15][17][18][19][20][21][22][23][24][25] ITEs occur yearly during residency training and are typically mandatory except during extenuating circumstances such as illness or a leave of absence.6 Their results from nine residency programs suggest that scores greater than the 50 th percentile on the ABA-ASA ITE were highly predictive of passing their written board specialty examination. 26McClintock and Gravlee examined ABA-ASA ITE scores from 2458 residents at the end of their first clinical anesthesia year (CA-1 ITE) as a predictor of ABA Part 1 performance and found that the CA-1 ITE was a significant, moderately strong predictor of passing the ABA Part 1 examination. 3Guffey et al. found that the USMLE Step 1 and Step 2 examinations were significant predictors of performance on the ABA Part 1 examination, though they found the Step 1 examination to be a stronger predictor than Step 2. 15 Residents' performance on departmental, in-house multiple choice question examinations that were written by departmental faculty may also be a means of objective assessment with which to identify at-risk residents during training.One study found that pediatric resident performance on these exams significantly correlated with success on the pediatric ITEs. 27This finding suggests that a departmental, in-house examination may also serve as a predictor of ABMS board examination performance.
The objectives of this study, therefore, were: (1) to determine whether information available prior to anesthesia residency training, such as gender, type of medical degree, USMLE Step 1 scores, USMLE Step 2 scores, or URM status could be used as predictors of ABA Part 1 performance in our department; and (2) to determine whether information collected during clinical anesthesiology residency training, such as the annual ABA-ASA ITEs and our biannual, departmental, faculty-written, in-house BSEs can predict ABA Part 1 performance and help identify anesthesia residents at risk for failure.

Study design and participant selection
Prior to the start of data collection, the University Hospitals Case Medical Center Institutional Review Board (IRB) approved this study protocol and the request for exemption was granted.This study was a retrospective analysis of anesthesiology resident files containing information that was collected prior to starting residency, during their residency training, and after successful completion of their anesthesia residency.The study sample included 97 anesthesiology residents who graduated from a single Midwestern residency training program (Case Western Reserve University/University Hospitals Case Medical Center) between 1995 and 2007.This sample represented all graduates during this time period, except for one resident whose information was misplaced and could not be located.The ABA Part 1 scaled scores were used as the dependent variable; these scores were available for 88 residents, who comprised the final study group.

Independent Variables
Categorical demographic variables used as possible predictors included gender (male or female), URM status (African-American/Hispanic/Native American), and type of medical degree completed (DO, MD, or ECFMG [Educational Commission for Foreign Medical Graduates]).Quantitative academic variables used as possible predictors included first-attempt scores on USMLE Steps 1 and 2, annual ABA-ASA ITE scores obtained during residency, and biannual BSE scores.The USMLE Step 1 exam is a multiple choice examination covering basic science topics.The USMLE Step 2 exam is also a multiple choice exam, but this test mostly focuses on clinical diagnosis and disease management.Three-digit scores for both of these exams vary slightly from year to year, but the typical range of scores is from 140 to 260, with a standard deviation of 20. 15 Passing score cutoffs are provided to the test taker with their exam score and published on the NBME website each year.For example, in 2010, the passing scores for the Step 1 and Step 2 exams were 188 and 184, respectively. 15The annual ABA-ASA ITEs taken by anesthesiology residents during the period of this study were categorized as follows: at the beginning of the intern year, which was designed to test baseline knowledge (PGY-0 ITE); at the beginning of the CA-1 year, which was designed to test knowledge gained during the clinical base intern year (CB ITE); at the beginning of the CA-2 year, designed to test knowledge gained through the CA-1 year (CA-1 ITE); and at the beginning of the CA-3 year, designed to test knowledge gained through the CA-2 year (CA-2 ITE).Two-digit scaled scores on the ABA-ASA ITE were used throughout this study; we used a scaled score of 32 as equivalent to passing the ABA Part 1 examination, as recommended by the ABA-ASA in materials provided to residency program directors.The BSE, which is administered twice each year during the CA-1, CA-2, and CA-3 years, is a multiple choice question examination designed to assess residents' mastery of topics presented during weekly didactic lectures over the past semester.Each faculty member who lectured during that semester submitted five multiple-choice questions; each examination consisted of approximately 50 questions.The curriculum was designed to be completed over three years, so each resident took a total of six BSEs during their clinical anesthesia residency.

Dependent Variable
The ABA Part 1 examination scaled scores were used as the dependent variable in this study.During the study period, the ABA Part 1 examination for candidates that graduated from an ACGME-approved anesthesia residency training program was administered simultaneously with the ABA-ASA ITEs for PGY-0, CA-1, CA-2 and CA-3 residents on the second Saturday in July.

Data analysis
Stepwise linear regression analyses were conducted using SPSS® software (SPSS, Chicago, IL), in order to evaluate the contribution of demographic and academic variables for predicting ABA Part 1 scores.A p-value < 0.05 was regarded as statistically significant.First-order bivariate Pearson correlations (for quantitative variables) and comparisons of means using one-way analyses of variance (ANOVAs) and independent-samples t-tests (for categorical variables) were used to select candidate predictors; only those which were statistically significant in the first-order analyses were entered into the regression analysis.
All predictors available at the time of admission and that significantly correlated with the outcome variable were entered into a step-wise regression analysis with ABA Part 1 scores as the dependent variable.Nationally available predictors (PGY-0 ITE score, CB ITE score, CA-1 ITE score, and CA-2 ITE score) were entered into a series of regression equations predicting ABA Part 1 scores as the dependent variable with the goal of creating model equations that could provide a means of predicting, at each point in time, which residents may be at risk of failure.This was done by first using scores on the PGY-0 ITE as the sole predictor, then using scores on the CB ITE score as the sole predictor.Finally, stepwise regression analyses were performed using CB ITE and CA-1 ITE scores and then CB ITE, CA-1 ITE, and CA-2 ITE scores.All quantitative academic factors available prior to the ABA Part 1 examination (ITE and BSE scores) and found to have a positive correlation with the outcome measure were entered into a step-wise regression analysis with ABA Part 1 scores as the dependent variable.This model equation was then used to predict the ABA Part 1 score for each candidate, and the predicted score was compared to both the actual score and the passing cutoff score to determine the accuracy of the model.

Scores on USMLE Step 1 and
Step 2 examinations, all four ABA-ASA ITEs and all six BSEs positively correlated with the outcome measure (ABA Part 1 scores).The scores for the two ABA-ASA ITEs representing knowledge gained throughout the CA-1 and CA-2 years and the USMLE Step 2 examination scores showed the highest correlation with ABA Part 1 scores of all predictors studied.These data are summarized in Table 1.

Regression analysis -predictors available at time of application
When all predictors available at the time of admission that significantly correlated with the outcome variable (URM status, USMLE Step 1 score, USMLE Step 2 score) were entered into a step-wise regression analysis with ABA Part 1 scores as the dependent variable, the regression model was significant and produced an adjusted R 2 of 0.44 (F(1,37)= 30.23; p<0.001).Only the USMLE Step 2 score was a significant predictor (B=0.169,Standard Error = 0.031; p < 0.001).

The regression model for predicting ABA Part 1 scores from
Step 2 scores is: 1.530 + 0.169 × (USMLE Step 2 score).Thus, a failing score on the ABA Part 1 examination was predicted by a USMLE Step 2 score of less than 181.In our sample, 30.0% of the candidates who had a Step 2 score of less than 181 passed the ABA Part 1 examination, compared to 79.3% of those with a Step 2 score of 181 or higher.This regression equation had a sensitivity of 0.793 (79.3% of those predicted to pass the ABA Part 1 examination did in fact pass) and a specificity of 0.700 (70% of those predicted to fail the exam did in fact fail).

Beginning of intern year
When scores on the PGY-0 ITE were entered as the sole predictor in this regression analysis, with ABA Part 1 scores as the dependent variable, the regression model was significant and produced an adjusted R 2 of 0.26 (F( 1,36) = 14.13; p = 0.001).The regression model for predicting ABA Part 1 score at the end of PGY-0 year is: 27.109 + 0.651 × (PGY-0 ITE score).These results are summarized in Table 2. Thus, a failing score on the ABA Part 1 examination was predicted by a PGY-0 ITE score of 7 or less.In our sample, 66.7% of the candidates who had a PGY-0 ITE score of 7 or less passed the ABA Part 1 examination, compared to 77.1% of those with a PGY-0 ITE score of higher than 7.This regression equation had a sensitivity of 0.771 and a specificity of 0.333.
Because the residency program in question had both a transitional (3-year) anesthesia residency program and a categorical (4-year) anesthesia residency program, only some of the residents had data available for the internship year; PGY-0 ITE scores were only available for a limited number of candidates (N=41).The PGY-0 ITE score was, therefore, not used as a predictor in any subsequent analyses.

Beginning of CA-1 Year
When scores on the CB ITE were entered as the sole predictor, with ABA Part 1 scores as the dependent variable, the regression model was significant and produced an adjusted R 2 of 0.28 (F (1,72)=29.90;p<0.001).The regression model for predicting ABA Part 1 score at the beginning of CA-1 year is: 24.488 + 0.683 × (CB ITE score).These results are summarized in Table 2. Thus, a failing score on the ABA Part 1 examination was predicted by a CB ITE score of 10 or less.In our sample, 61.5% of the candidates who had a CB ITE score of 10 or less passed the ABA Part 1 examination, compared to 80.3% of those with a CB ITE score of higher than 10.This regression equation had a sensitivity of 0.803 and a specificity of 0.385.
In our sample, neither the PGY-0 ITE nor the CB ITE scores proved to be very useful for identifying residents who were at risk of failing the ABA Part 1 examination.Over half of the residents who would have been identified as having a high risk of failure based on the regression equations for these predictors did, in fact, pass the ABA Part 1 examination on the first attempt.Thus, scores on intraining examinations taken after the beginning of internship year but before the completion of the first year of clinical anesthesia residency appear to be of limited predictive value.

Beginning of CA-2 Year
When CB ITE and CA-1 ITE scores were entered into a step-wise regression analysis with ABA Part 1 scores as the dependent variable, the regression model was significant and produced an adjusted R 2 of 0.45 (F(1,72)=61.73;p< 0.001).Only the CA-1 ITE score was a significant predictor.The regression model for predicting the ABA Part 1 score at the beginning of CA-2 year is: 18.994 +0.717 × (CA-1 ITE score).These results are summarized in Table 2. Thus, a failing score on the ABA Part 1 examination was predicted by a CA-1 ITE score of 18 or less.In our sample, 33.3% of the candidates who had a CA-1 ITE score 18 or less passed the ABA Part 1 examination, compared to 84.4% of those with a CA-1 ITE score of higher than 18, as shown in Figure 1.This regression equation had a sensitivity of 0.844 and specificity of 0.667.

Beginning of CA-3 Year
When CB ITE, CA-1 ITE, and CA-2 ITE scores were entered into a step-wise regression analysis with ABA Part 1 scores as the dependent variable, the regression model was significant and produced an adjusted R 2 of 0.50 (F (1,72) =73.93; p<0.001).2. This regression equation had a sensitivity of 0.879 and specificity of 0.727.

Regression analysis -predictors measuring performance during clinical anesthesia residency
The following quantitative academic factors available prior to the ABA Part 1 examination were found to have a positive correlation with the outcome measure: CA-1 ITE, CA-2 ITE, and the six biannual BSE scores for years 1, 2, and 3. When these factors were entered into a step-wise regression analysis with ABA Part 1 scores as the dependent variable, the regression model was significant and produced an adjusted R 2 of 0.52 (F (2, 61) = 35.50;p<0.001).These data are summarized in Table 3.Each candidate's predicted ABA Part 1 score was compared to both the actual score and the passing cutoff score to determine the accuracy of the model.When the model predicted a failing score, the actual score was equivalent to passing 17% of the time (specificity=0.83).When the regression model predicted a passing score, the actual score was equivalent to passing 82% of the time (sensitivity = 0.82).Thus, a prediction from this model of a failing score is a fairly strong indicator that the resident is at increased risk of failure on the ABA Part 1 examination.

Discussion
2][3][4][5] In our study, a significant difference in mean scores on the ABA Part 1 examination between residents belonging to URM groups and non-URM residents was identified.However, when URM status was included as a possible predictor in the regression model, it did not explain any additional variance beyond that explained by USMLE Step 2 scores.In addition, care should be used in interpreting this result, due to the limited number of URM residents in our sample.This finding suggests that the difference in mean scores may have reflected a preexisting effect of lower standardized test scores, as members of URM groups also had lower USMLE Step 1 and Step 2 scores.This finding is consistent with other studies, which have suggested that members of URM groups may have more difficulty in medical school and on the USMLE Step examinations, 4,5 and this effect may also apply to board specialty examinations.Gender is another demographic variable available prior to the start of residency training.In our study, we did not find it to be a significant predictor of performance on the ABA Part 1 examination in our study.The literature contains mixed results regarding gender as a predictor of performance on specialty board examinations.Consistent with our results, one study of internal medicine residents at four internal medicine programs with internal medicine ITE pass rates of less than 100% did not show a difference in pass rates on the internal medicine board certifying examination for men versus women. 2 An older study, examining scores of family medicine residents taking the American Board of Family Practice ITE in 1987-1988, found that male family medicine residents performed better on different subsections of the family medicine certifying examination than females; women scored significantly higher on the obstetrics, pediatrics, and gynecology subsections, and men performed significantly better on the surgery and community medicine subsections. 1 A recent study of 2458 anesthesiology residents who completed their CA-3 year between 2002-2004 showed that men performed better than women on the ABA Part 1 examination. 3

Table 2. Stepwise regression analyses predictors from ITEs by year for anesthesiology Residents (1995-2007) at a Midwestern anesthesiology residency program (N=88)
A recent study of anesthesiology residents found that graduates of allopathic medical schools performed better on the ABA Part 1 examination than graduates of osteopathic medical schools, and graduates from medical schools outside the US were at higher risk of failing. 3Contrary to their results, the type of medical degree obtained (DO/MD/ECFMG) was not found to be a significant predictor in our study.
Our results are in agreement with several other studies showing that performance on written, multiple choice question examinations correlates with performance on subsequent written, multiple choice question examinations from multiple specialties in the literature. 3,6, Forexample, the USMLE Step 3 examination has been shown to correlate with previous USMLE examinations. 28 15 However, some studies suggest that the Step 2 score might be a better predictor in other specialties. 29,31 15ur findings indicate that the USMLE Step 2 examination is a moderately strong predictor of passing the ABA Part 1 examination.According to our results, a USMLE Step 2 score of 181 or higher predicts passing the ABA Part 1 examination: 79.3% of residents who scored 181 or higher passed on the first attempt.Therefore, our findings extend the existing literature and confirm that the USMLE Step 2 may be used as an early predictor of success on the ABA Part 1 examination.
11][12][13][14][15][17][18][19][20][21][22][23][24][25] The results from our study also indicate that predictors such as those ITEs that test knowledge gained over the first two years of anesthesiology training can be used as predictors of passing the ABA Part 1 examination.However, we did not find either the PGY-0 ITE (adminis-tered at the beginning of the intern year) or the CB ITE (administered at the beginning of the CA-1/first anesthesiology residency year) to be particularly good predictors of passing the ABA Part 1 examination.The PGY-0 ITE and the CB ITE explained only approximately half as much of the variance as the CA-1 ITE and the CA-2 ITE.Furthermore, regression models based on these two predictors only achieved 55-59% accuracy in predicting whether or not residents would pass the ABA Part 1 examination, which is only slightly better than chance (50%).
In our regression model that included all the ABA-ASA ITE results available at the beginning of the CA-2 year, the CA-1 ITE was the only significant predictor of passing the ABA Part 1 examination (see Table 2).The CA-1 ITE was a moderately strong predictor of passing the ABA Part 1 examination, with an R 2 of 0.45 and an overall model prediction accuracy of 75.5%.Our regression model predicts that residents scoring 19 or higher on the CA-1 ITE will pass the ABA Part 1 examination.In our sample, 84.4% of those scoring 19 or higher did in fact pass the ABA Part 1 examination.However, 33.3% of residents who were predicted to fail the ABA Part 1 examination also passed, so caution is advised when applying these findings.Thus, the CA-1 ITE may be useful to help identify CA-2 residents at increased risk of failing the ABA Part 1 examination.Our regression model that included all the ABA-ASA ITE results available at the beginning of the CA-3 year identified the CA-2 ITE as the best predictor of passing the ABA Part 1 examination.The CA-2 ITE was found to be a moderately strong predictor of passing the ABA Part 1 examination, explaining 50% of the variance.The model prediction accuracy of the CA-2 ITE was 80.3%.Our regression model predicts that residents scoring 22 or higher on the CA-2 ITE will pass the ABA Part 1 examination.In our sample, 87.9% of those scoring 22 or higher passed the ABA Part 1 examination.In addition, 27.3% of residents who were predicted Other studies have shown that the ABA ITEs correlated significantly with the ABA Part 1 examination 3 and the Canadian anesthesiology certification examination. 26uffey et al., in a secondary analysis found that ITEs were moderate predictors of ABA Part 1 performance. 15cClintock and Gravlee investigated the use of the CA-1 ITE as a predictor of the ABA Part 1 examination and found it to be a moderately strong predictor of passing the ABA Part 1 examination. 3However, they did not examine the use of the CA-2 ITE as a predictor of passing the ABA Part 1 examination in their study.In our analysis, when the CA-1 ITE is the only predictor, our results are consistent in that the CA-1 ITE is a good predictor of passing the ABA Part 1 examination. 3They reported that 46% of the variance was accounted for by the CA-1 ITE score in predicting scores for the ABA Part 1 examination. 3In our study, when the CA-1 ITE score was the only ITE score in the regression model, the CA-1 ITE score predicted 45% of the variance, which is very similar to their results.However, when the CA-2 ITE is included as a predictor, our results indicate that it is a stronger predictor of passing the ABA Part 1 examination than the CA-1 ITE alone, explaining 50% of the variance.Since the results of the CA-1 ITE and the CA-2 ITE are available well before the candidate takes the ABA Part 1 examination, there is ample time for residents to increase the priority of their studying and for the training program to provide additional educational support, if necessary.
There is a paucity of literature regarding past performance on in-house multiple choice examinations by candidates and their subsequent performance on specialty board certification examinations.Davis et al. found that pediatric residents' performance on their departmentally-created, written multiple choice examination strongly correlated with their subsequent performance on the American Board of Pediatrics ITE and was at least 74-80% accurate in predicting failure on the ITE. 27Our institution's in-house BSE, administered twice a year during the CA-1 through CA-3 years, also provides additional opportunities for assessment and feedback regarding knowledge of core anesthesiology topics.Our results indicate that the scores from all six of the BSEs correlated with the ABA Part 1 examination scores, but the BSE administered in May of the CA-3 year was the best predictor out of all of the BSEs that were given over the three years of training.This last BSE was a positive predictor in the regression model using all available information immediately prior to taking the ABA Part 1 examination, but only explained a small amount of variance (2%) beyond that predicted by the CA-2 ITE.Therefore, our findings suggest that in-house examinations can serve as an additional valid predictor when attempting to identify residents who are at risk of failing the ABA Part 1 examination, especially in the final months of residency training.
During the study period, the CA-2 ITE was administered on the second Saturday in July, one year before the ABA Part 1 examination would be taken.In 2008, the annual ABA-ASA ITE administration date was changed to the first Saturday in March, and the ABA Part 1 examination was moved to the first Monday or Tuesday in August.Thus, the CA-2 ITE is now taken about 5 months prior to the ABA Part 1 examination.Although shortening the interval of time between the CA-2 ITE and the ABA Part 1 examination provides less time during residency training to provide educational intervention, we speculate that moving the CA-2 ITE temporally closer to the ABA Part 1 examination will make it an even better predictor of ABA Part 1 performance than when it was administered one year prior to the ABA Part 1 examination.Residents receiving their CA-2 ITE score with the associated list of missed keyword concepts a few months prior to the ABA Part 1 examinations might be able to use this valuable information to redirect their studying efforts as necessary to potentially improve their scores.
The primary limitation to our study was sample size.Initially, data for other potential variables such as age, country of origin, prior degrees earned, Alpha Omega Alpha (AOA) status, chief resident status, clinical evaluations, academic probationary status, and atypical stop/start dates for residency training were collected, but due to small sample size or missing data, we were unable to examine these factors as possible predictors.Analysis was further limited by missing data for individual predictor variables that resulted in a reduced sample size, (i.e., USMLE Step 1 and Step 2 scores, which were unavailable for some residents).Although all residents in this study were required to have passed both Step 1 and Step 2 prior to starting their residency training, these exam scores were not consistently filed with the rest of the resident's academic file and this missing data reduced the power of our study.However, we have no reason to believe that the missing data due to clerical filing systematically biased our sample.The use of multiple comparisons could have increased the probability of Type 1 Error in our results, but adopting a more conservative criterion of statistical significance (such as p<0.01) would not have altered the primary conclusions concerning the usefulness of ITE scores for predicting ABA Part 1 performance.Another limitation was that these results represent data from a single residency training program and may not generalize to all programs.Despite these limitations, we achieved remarkable consistency in the amount of variance explained by the CA-1 ITE with McClintock and Gravlee's study (R 2 = 0.46) 3 as compared to our study (R 2 = 0.45).An additional limitation involves the generalizability of our in-house departmental exam, the BSE.These exams were comprised of questions submitted by faculty who had lectured to residents within the past six months, but they had no training or guidance in writing these questions.If the BSEs had been more standardized, we might have seen a stronger correlation with Part 1 exam scores.In addition, residents were only given feedback regarding their performance on the BSEs; if there had been actual consequences for poor performance, residents might have assigned a higher priority to their performance on the BSEs which in turn might have increased the correlation with their Part 1 exam scores.However, despite these limitations, we still showed a moderate correlation of the BSEs with the Part 1 exam scores, suggesting they may be useful additional predictors of resident performance.Other departments that create their own in-house examinations could use them as another tool to help identify at-risk residents.
A multi-center study with a larger sample size could overcome our limitations of missing data and generalizability, allowing investigation into other factors potentially affecting anesthesiology residents' performance on the ABA Part 1 examination.Based on the current literature, factors such as lecture attendance, study methods, teaching methods, moonlighting, and fatigue may be used to give additional educational support to those residents identified as being at increased risk of failing the ABA Part 1 examination, as discussed below.
There are mixed results on the correlation of lecture attendance and performance on the ITE.In two studies, attendance at lectures was positively correlated with higher scores on the surgery ITE. 33,34The data for the internal medicine ITE performance was mixed; there was a positive correlation of lecture attendance with ITE scores of internal medicine residents in two studies, 35,36 but there was a lack of correlation in two other studies. 37,38In another study, there was no correlation between lecture attendance and the ITE scores of PM&R residents. 6To our knowledge, no such studies have been conducted with anesthesiology residents.
Although physical medicine and rehabilitation (PM&R) residents did not score differently based on whether or not they had attended a review course for their certification examination, 6 there may be other more aggressive educational measures that could be implemented for residents with poor scores on the ITE and in-house departmental examinations.For example, requiring more frequent meetings with program directors and faculty mentors to create an individualized educational plan was shown to raise surgery ITE scores at one program. 39,40Emphasizing adequate study time is clearly important; residents must be encouraged to read every day and use the results of prior ITEs and other self-assessment examinations to guide study topics, as this may help improve performance on ITEs. 41In fact, in one study, anesthesiology residents needed a minimum of 10.5 hours/week of self-study in order to achieve the equivalent of a passing score on the ABA Part 1 examination. 42Both faculty-and resident-directed study (involving textbook/literature reviews on assigned topics) resulted in higher scores on the surgical ITE. 43High levels of anxiety have been shown to correlate with poor academic performance in anesthesiology residents, 44 and poor study habits have been significantly correlated with lower performance on the ABSITE. 45These points can be addressed by creating an individualized plan that includes meeting with an educational psychologist, which was done in one successful academic interventional plan for surgery residents. 39ptimal didactic teaching methods within the department could also help improve residents' exam scores; thoracic surgery residents who engaged in self-study and the presentation of topics instead of attending didactic lectures exhibited a greater improvement of scores from the first to second year of thoracic surgery residency. 46Another study found that a weekly conference focusing on ITE topics resulted in higher performance on the surgical specialty boards. 47e were not able to investigate moonlighting as a predictor.Some believe that moonlighting decreases residents' study time, while others have suggested that the additional clinical experience might be beneficial.One study with internal medicine residents failed to show any correlation between moonlighting and their specialty certification examination scores. 48esident fatigue at the time of taking the ITE may decrease performance and thus limit its predictive value.Although this point may seem intuitive, two studies with surgery residents showed no correlation between being on call the night before the ITE and their ITE scores. 49,50It is unknown whether this finding holds true for anesthesiology residents.
A larger, multi-center investigation would enable us to identify at-risk residents more reliably, and a carefully designed follow-up study could look at the effects of implementing some the interventions discussed above.

Conclusions
Our study demonstrated that data available prior to residency training (USMLE Step 2 scores), as well as information obtained during the course of residency (ABA-ASA ITE scores and our in-house multiple choice BSE scores), may be promising predictors of performance on the ABA Part 1 examination.Our results indicated that first-attempt scores on the USMLE Step 2 examination were a moderately strong predictor of ABA Part 1 examination performance and were the best predictor available prior to the beginning of residency.
During residency, ABA-ASA ITE scores were positively correlated with ABA Part 1 scores.The CA-1 and CA-2 ITEs were moderately strong predictors of passing scores on the ABA Part 1 examination, with the CA-2 ITE score accounting for most of the variance in our regression model.Our findings provide an empirical basis for intervention, with specific scores provided which suggest a higher likelihood of passing, allowing programs to indicate to residents when additional educational measures may be required.
The BSE was also found to be a predictor of ABA Part 1 scores, particularly the last administration two months prior to the certification examination.Thus, at our program, the BSE can be used as a proximal assessment of knowledge and could serve to guide any adjustments to a resident's study schedule or mode of study in the final months before taking this examination.This finding also suggests that other departments with their own in-house routine assessments may find them useful in identifying at-risk residents.
Only the CA-2 ITE score and the second BSE score from the third year (BSE CA-3 Spring) were significant predictors in this regression model.The CA-2 ITE score alone accounted for most of the explained variance (adjusted R 2 = 0.50).The regression model for predicting the ABA Part 1 score immediately prior to the examination is: 12.592 + 0.627× (CA-2 ITE score) + 0.096 × (BSE CA-3 Spring score).

Figure 1 .
Figure 1.ABA Part 1 Exam Scores (first attempt) as a function of ITE CA-1 Scores.The vertical line indicates the ITE CA-1 score cutoff that predicts a passing score on the first attempt of ABA Part 1 Exam [R 2 = 0.45; F (1, 72) = 61.73;p < 0.001].
ITE CA-1 and ABA Part 1 Exam Scores Passed ABA Part 1 Exam Failed ABA Part 1 Exam positive predictors of specialty board certification.

Figure 2 .
Figure 2. ABA Part 1 Exam Scores (first attempt) as a function of ITE CA-2 Scores.The vertical line indicates the ITE CA-2 score cutoff that predicts a passing score on the first attempt of ABA Part 1 Exam [R 2 = 0.50; F (1, 72) = 73.93;p < 0.001].

Failed ABA Part 1
Examto fail the ABA Part 1 examination also passed.Of all the variables examined in our study, the CA-2 ITE was the single best predictor of passing the ABA Part 1 examination.
Kearney et al. investigated whether or not the American Board of Anesthesiology-American Society of Anesthesiologists In-Training Examination (ABA-ASA ITE) was a good predictor of performance on the Royal College of Physicians and Surgeons of Canada certification examination, which is comparable to the ABA Part 1 examination.

Table 1 .
First-order Pearson correlations among outcome and predictor variables for Anesthesiology residents (1995-2007) at a Midwestern Anesthesiology residency program (N=97) passed the ABA Part 1 examination, compared to 87.9% of those with a CA-2 ITE score of higher than 21, as shown in Figure Studies in other specialties have also demonstrated that USMLE scores were 4,[6][7][8][9][10][11][12][13][14][15][16][17]In 1986, Warrick and Crumrine 32 actually found a negative correlation between the National Board of Medical Examiners (NBME) Part 1 examination, which the USMLE Step 1 examination has replaced, and the ABA-ASA ITE.In that study, higher scores on the NBME Part 1 examination were associated with lower ABA-ASA ITE scores.A previous study has found that the USMLE Step 1 examination is a better predictor of the ABA Part 1 examination score than the Step 2 examination for anesthesiology residents.