The Orthopaedic In-Training Examination (OITE) is a standardized test that is administered annually by the American Academy of Orthopaedic Surgeon (AAOS) to residents across the United States and throughout the world.¹ The OITE allows program directors to objectively measure their residents’ medical knowledge, one of the six core competencies established in 1999 by the Accreditation Council for Graduate Medical Education (ACGME).²

In 2017, the Harvard Combined Orthopaedic Residency Program (HCORP) performed in the 6th percentile among Allopathic programs and in the 22nd percentile among all programs. As a result, during the 2018-2019 academic year, we implemented a new OITE review curriculum with the emphasis of improving residents’ overall performance on this exam. This study detailed the curriculum, surveyed residents’ study habits and resources used for exam preparation, and compared OITE performance before and after implementation of the curriculum. We hypothesized that the curriculum would significantly improve OITE performance among residents.

Participants

All postgraduate year (PGY) 2 to 5 residents in the 2017-2018 and 2018-2019 academic years at a single orthopaedic residency program were enrolled in the study. PGY1 residents did not consistently attend the program’s weekly didactic conferences and therefore were excluded. PGY2-5 residents who took the 2017 OITE (2017 cohort; before implementation of the curriculum) were compared to residents who took the 2018 OITE (2018 cohort; after implementation of the curriculum).

Curriculum

The OITE review curriculum was a resident-led curriculum that took place weekly from June of 2018 to November of 2018. This was purely a question-based review course using prior exam questions from the 2014-2017 OITEs. These review sessions were typically between 30 and 60 minutes long and were presented in PowerPoint (Microsoft, Redmond, WA) format by senior (PGY4 or 5) residents. The curriculum covered all 11 orthopaedic knowledge domains tested on the OITE, which consist of Basic Science, Foot and Ankle, Hand, Hip and Knee, Oncology, Pediatrics, Shoulder and Elbow, Spine, Sports Medicine, Trauma, and Practice Management.

Outcomes

Measuring the outcomes of an educational intervention, in this case a new curriculum, involved three phases: Input, Process, and Outcomes.³ During the Input phase, a survey was administered to obtain residents’ basic demographics and study habits with respect to the 2017 OITE (Table 1). Specific parameters of interest included PGY level, personal emphasis placed on the OITE, number of questions completed in preparation for the OITE, and resources used to study for the OITE. Residents’ official 2017 OITE scores were also obtained. During the Process phase, the curriculum was implemented. During the Outcomes phase, a similar survey was administered with respect to the 2018 OITE (Table 1). Finally, the official 2018 OITE scores were then obtained for comparison. Paired sample t-tests were utilized for the nominal data with a discriminatory alpha value of 95%. Chi-square analysis was utilized for the categorical data.

TABLE 1

Summary of the two surveys that were administered to orthopaedic residents to evaluate their study habits and resources used in preparation for the 2017 and 2018 OITEs.

All Rights Reserved. Permission For Use Required.

-View Large-

Forty residents completed the pre-curriculum survey (40/46; 87% response rate), and 41 residents completed the post-curriculum survey (41/46; 89% response rate). Before the curriculum was implemented, most residents placed little to no emphasis on the OITE (26/40, 65%). Most residents also completed fewer than 500 questions in preparation for the OITE (25/40, 62.5%). The main content resource used was Orthobullets,⁴ and the main question bank used was similarly distributed among Orthobullets, ResStudy,⁵ and prior OITE questions. After the curriculum was implemented, all residents placed moderate to strong emphasis on the OITE (41/41, 100%). The majority of residents completed more than 750 questions in preparation for the OITE (32/41, 78%). The main content resource used was High-Yield Orthopaedics (HYO)⁶ and Orthobullets, and the main question bank used was ResStudy (Table 1).

Within the residency program, 52.5% of residents (21/40) believed that OITE performance does predict American Board of Orthopaedic Surgery (ABOS) Part I performance while 47.5% did not (19/40). All but one of the residents (39/40, 97.5%) believed they would pass the ABOS Part I on their first attempt. Half of the residents (20/40; 50%) did not believe any repercussion should result for poor OITE performance, while the other half believed residents should receive formal reprimand (18/40; 45%) or academic probation (2/40; 5%). Half of the residents (20/40; 50%) believed that residents who performed well on the OITE should receive a financial stipend and/or scholarship. Regarding the OITE, 55% of the residents (22/40) believed it should be mandatory, 25% (10/40) believed it should be optional, while 20% (8/40) were in favor of eliminating the OITE altogether. Most residents (31/40, 77.5%) would be interested in taking the ABOS Part I during their PGY4 and/or PGY5 year before graduation (Table 2).

TABLE 2

Summary of the survey that was administered to orthopaedic residents to evaluate their beliefs on the OITE.

All Rights Reserved. Permission For Use Required.

-View Large-

Forty-nine PGY2-5 residents took the 2017 OITE (2017 cohort), while 46 residents took the 2018 OITE (2018 cohort) (Table 3). Of the 275 questions on the OITE, four questions were omitted from the 2017 OITE (271 questions remained), and six questions were omitted from the 2018 OITE (269 questions remained). The 2017 cohort answered 64.9% of the questions correctly (175.8 of 271 questions), compared to 70.2% in the 2018 cohort (188.9 of 269 questions). The 2017 cohort scored in the 6th percentile among Allopathic programs (157 programs) and the 22nd percentile among all programs (237 programs). The 2018 cohort scored in the 73rd percentile among Allopathic programs (160 programs) and 76th percentile among all programs (238 programs). OITE performance by PGY class is summarized in Table 4.

TABLE 3

The orthopaedic residency program’s overall OITE performance before (2017 OITE) and after (2018 OITE) implementation of the focused OITE review curriculum

All Rights Reserved. Permission For Use Required.

-View Large-

TABLE 4

OITE performance before (2017 OITE) and after (2018 OITE) implementation of the focused OITE review curriculum by PGY year

All Rights Reserved. Permission For Use Required.

-View Large-

According to a multicenter study published in 2010, residents who “averaged in the 27th percentile or lower on the OITE had a 57% chance of failing the ABOS Part I examination”.⁷ Subsequent studies have confirmed the utility of OITE scores in predicting performance on the ABOS Part I.^8,9 In 2018, 3% of first-time test takers (19 of 717 examinees) and 9% of all test takers (75 of 835 examinees) failed the ABOS Part I.¹⁰ These studies and exam statistics suggest that residents who consistently score low on the OITE are at risk for failing the written board exam. Despite this, more than half of residents (21/40; 52.5%) at our program did not consider OITE performance a reliable predictor of ABOS Part I performance. Furthermore, all but one (39/40; 97.5%) were confident they would pass the ABOS written board during their first time taking the exam. Thus, the main challenge to improving OITE scores was resident buy-in regarding the importance of the in-training exam.

Various effective strategies have been described to maximize OITE performance among residents. Weglein et al. (2015) instituted a weekly reading and written exam program based on review articles from the Journal of the American Academy of Orthopaedic Surgeons (JAAOS) and noted a significant improvement in OITE performance at their program.¹¹ Franklin et al. (2016) implemented a weekly pediatric didactic lecture and observed improvement in OITE scores and percentiles on the pediatric domain.¹² Both in-person and web-based comprehensive curricula have been shown to be helpful for OITE preparation.^13,14 At our institution, prior to the initiation of the new OITE review curriculum, there was already a weekly didactic conference in place, and residents had independent access to multiple OITE review resources including AAOS ResStudy, Orthobullets, and JBJS Clinical Classroom.¹⁵ Despite this, our residents routinely scored below the national means. This prompted the development of a structured OITE review curriculum to change the learning culture at our orthopaedic surgery residency program. This curriculum had two unique features. First, it was entirely resident-initiated and resident-run. A national survey in 2017 revealed that OITE preparatory courses led by residents were more effective than those led by faculty members.¹⁶ Second, the review content was entirely based on actual questions from the 2014-2017 OITEs. A focused, question-based approach to exam review has been shown to be successful at improving test scores in general.¹⁷

This study showed that implementation of a formal OITE review curriculum significantly improved OITE performance. Specifically, our residents increased the mean percentage of questions answered correctly (64.9% to 70.2%; p<0.001) as well as the program’s percentile among Allopathic programs (6th to 73rd) and all programs (22nd to 76th). Of note, since a program’s performance is compared to the performances of other Allopathic programs, a small improvement in the number of questions answered correctly can represent a significant improvement in percentile ranking. OITE scores were improved by initiating resident leadership in education, changing residents’ study habits, and increasing emphasis placed on the in-training exam. The 2018 cohort (post-curriculum) placed greater importance on the OITE and completed more review questions than the 2017 cohort (pre-curriculum). Previous studies have demonstrated that residents who assigned greater value to the OITE unsurprisingly performed better.^16,18 Likewise, residents who dedicated more time to exam preparation scored higher.^16,19 In addition to dedicating more time, utilizing the right study materials to prepare for the OITE is equally important. Miyamoto et al. (2007) found that residents who concentrated on prior OITEs or self-assessment examinations performed better. A focus on review books (e.g., Miller’s Review) or lecture notes, on the other hand, was not associated with higher scores.¹⁹ In a web-based survey, residents found past exam questions to be the most useful educational resource for the OITE.²⁰ The significant improvement in OITE performance at our program from 2017 to 2018 affirmed that a review curriculum based on prior OITE questions works. There is real value in doing prior questions in preparation for the in-training exam.

This study has several limitations worth noting. This study is not prospectively randomized and therefore bears the limitations of a non-randomized, retrospective cohort study. A major component of this study was survey-based, and therefore the results could be subject to recall bias or response bias of the participants. The study did achieve high response rates for the two surveys administered (i.e, 87% for the pre- and 89% for the post-curriculum survey). Additionally, the post-curriculum survey was conducted immediately after the completion of the curriculum to mitigate recall bias. In addition, the PGY2-5 residents in 2017 were not identical to the PGY2-5 residents in 2018, so there may be inherent differences between the two cohorts. While the curriculum certainly played an important part in improving resident performance on the OITE, there were potential confounding factors that might have influenced the outcomes of this study. During the 2018-2019 academic year, our residency program instituted a modified policy wherein residents who score below the 50th percentile compared to Allopathic residents at the same PGY level are required to complete remediation questions and lose moonlighting privileges. Residents who score below the 15th percentile additionally forfeit their elective rotation and eligibility to request travel grants for the academic year. These repercussions, we speculate, did not affect PGY5 residents as much, which may account for the lack of improvement in this group’s scores from 2017 to 2018. Camp et al. (2017) found that programs which enforced negative consequences for poor OITE performance scored higher on the in-training exam.¹⁶ Thus, our program’s policy with the negative consequences discussed above may have confounded the results of this study and separately contributed to improving test scores. Nonetheless, the curriculum contributed to an overall impetus to improve performance on the OITE.

In summary, orthopaedic residency programs striving to improve resident performance on the OITE should consider establishing a focused OITE review curriculum. Its effectiveness was demonstrated in this study as well as in previous studies.¹⁶ To the best of our knowledge, this curriculum is the first OITE review curriculum reported in the literature that is based solely on prior OITE questions. Moving forward, further research should be carried out to examine whether the benefits of this OITE review curriculum are long-lasting (i.e., whether our residents continue to maintain or improve OITE scores under this curriculum) and replicable at other orthopaedic residency programs.

In 2017, residents at a single orthopaedic residency program performed at a historically-low percentile on the OITE compared to other Allopathic residency programs. Consequently, a resident-led, question-based OITE review curriculum was implemented the following academic year. The curriculum contributed to significantly increasing the residents’ mean percentage of questions answered correctly while improving the program’s performance from the 6th to the 73rd percentile among Allopathic orthopaedic programs. Instituting a focused OITE review curriculum may be an effective strategy to improving OITE scores.