Rank List Formation Among Orthopaedic Surgery Applicants and Program Directors During the COVID-19 Pandemic
Max R. Haffner, MD, John-Allen Smith, MD, Joseph B. Wick, MD, Adam Bacon, BS, Thomas Shen, MS, Christopher T. Holland, MD, Mark A. Lee, MD, Steven W. Thorpe, MD, Cassandra A. Lee, MD, Hai V. Le, MD
The author reports no conflict of interest related to this work.
©2022 by The Orthopaedic Journal at Harvard Medical School
BACKGROUND The COVID-19 pandemic has necessitated widespread use of virtual rotations and interviews during this past orthopaedic residency application cycle. This study aimed to identify factors that were associated with applicant and program rank-list formation in the setting of a change in application cycle environment and to determine whether virtual platforms may have long-term utility.
METHODS Surveys were sent to program directors (PDs) from Accreditation Council for Graduate Medical Education accredited orthopaedic residency programs and applicants who matched into orthopaedic surgery in 2021. Surveys consisted of questions assessing basic information about each residency program or applicant, factors associated with rank list formation, and virtual interview experiences. Descriptive statistical analyses were performed.
RESULTS Most PDs found letters of recommendation (74%), resident (64%) and faculty recommendations (64%), and interview performance (62%) to be more important in making their rank lists as compared to previous years (Figure 1). Mentor recommendations, geographic location, proximity to family, and national reputation were important factors to applicants. For applicants, 48 of 52 (92.3%) rated the interview experience as important in the ranking of programs. 74% of PDs agreed that they were able to adequately assess applicants using a virtual platform. 58% of applicants would recommend virtual interviews in the future.
CONCLUSIONS irtual interviews and rotations were positively perceived by most orthopaedic residency PDs and applicants. Virtual interviews and rotations may represent a cost- and time-saving primary or adjunctive option for future orthopaedic residency selection cycles.
LEVEL OF EVIDENCE Level IV Survey
KEYWORDSVirtual interview; COVID-19; residency application; orthopaedic surgery; orthopaedics
The COVID-19 pandemic created unprecedented challenges for program directors (PDs) and applicants during the 2021 orthopaedic residency application season, requiring many alterations to the traditional application cycle and interview format.1,2,3 Away rotations were restricted, and in-person interviews were discouraged in accordance with the Association of American Medical Colleges (AAMC) and the American Orthopaedic Association Counsel of Orthopaedic Residency Directors (CORD).1,2,3 Programs mitigated the loss of these in-person encounters by conducting virtual rotations (VRs), informational sessions, and virtual interviews (VIs).3
Traditionally, applicants to orthopaedic residency programs are ranked primarily by United States Medical Licensing Examination (USMLE) Step 1 scores, Alpha Omega Alpha (AOA) status, third-year clerkship grades, research experiences, and letters of recommendation (LORs).4,5 Performance of applicants during in-person orthopaedic rotations has been especially important.4 In fact, PDs have previously indicated that rotating with their program is the most important criterion in ranking applicants.5,6 In the past, over half of matched applicants match at their home institution or a program where they have rotated.5,7,8 PDs and applicants clearly value away rotations and in-person interactions to determine whether applicants will be successful in their programs, and there is concern that ability to assess applicant compatibility may be limited in virtual interactions.4,5,9
This pandemic has inspired changes to the residency application and interview process, which may persist even after the COVID-19 era, including VRs and VIs.10 To inform and improve the application and applicant ranking process in future application cycles, we sought to understand the impact of virtual interactions during the 2021 application season. As such, the primary purpose of this study was to assess the impact of VRs and VIs on program and applicant rank-list formation. We also evaluated the efficacy of VIs in PDs’ ability to evaluate residency candidates, applicants’ ability to evaluate residency programs, and PD and applicant satisfaction with the virtual application cycle.
This study was an electronic survey conducted in the United States at a national level in May 2021. The surveys were sent to PDs from Accreditation Council for Graduate Medical Education (ACGME) accredited orthopaedic residency programs, as well as to applicants who matched into orthopaedic surgery in March 2021. Unmatched applicants were excluded.
This study was approved by our Institutional Review Board (IRB). Two surveys were administered electronically in May 2021, including one for PDs and one for applicants to participate in voluntarily. For confidentiality, the survey contained no identifying personal information, and responses were anonymous. PD emails were obtained from the ACGME website.2 For the applicant survey, since there was no email list of matched applicants, program coordinators from ACGME-accredited residency programs were emailed and asked to forward the corresponding survey to matched applicants. Program coordinator emails were also obtained from the ACGME website.11 Weekly reminders were sent to improve response rates.
Program Director Survey
The PD survey was a 41-item, three-part electronic survey. Part I included 8 questions assessing basic information about each residency program. Part II included 21 questions assessing factors impacting PD rank list formation during the 2021 application process. Part III included 14 questions assessing each program’s VI experience.
Applicant Survey
The applicant survey was a 31-item, three-part electronic survey. Part I included 11 questions assessing basic demographic and applicant information. Part II included 14 questions assessing factors impacting applicants’ rank lists. Part III included 8 questions assessing applicants’ VI experiences. Applicants that did not list the rank number of the program they matched into were considered unmatched and were excluded. In order to mitigate response bias, all applicant surveys were anonymous and did not include personal identifying information.
Survey responses were downloaded onto Excel spreadsheet (Microsoft Corporation, Redmond, WA). Descriptive statistics were subsequently performed using Excel. 50 PD surveys were completed, and 52 applicant surveys were completed.
Fifty PDs from ACGME-accredited orthopaedic residency programs responded to the online PD survey, with a response rate of 42% (50/119). Most programs offered in-person rotations for home applicants (92%), with fewer offering in-person rotations to away applicants (18%). Overall, 84% of PDs reported offering virtual informational sessions to prospective applicants, but less than half (44%) offered virtual rotations. Half of PDs reported receiving more applications for the 2021 application cycle, while one-third reported interviewing more candidates than in previous years. All but three programs reported offering the same or fewer interview days. Zoom (60%) and Thalamus (34%) were the most common virtual platforms for conducting interviews (Table 1).
Fifty-two applicant surveys were completed. Table 2 lists basic demographic information from respondents. According to the National Resident Matching Program (NRMP), this past application cycle, there were 866 applicants who matched into orthopaedic residency programs.12 Assuming the survey was forwarded by program coordinators to all matched applicants, the participation rate for applicants is 6%. Of these, 87% attended an in-person home rotation, and 81% attended virtual rotations at other institutions. The average applicant submitted 70.3 applications and participated in 13.3 interviews. Although most respondents matched to their top ranked program, applicants who attended virtual away rotations averaged a match rank of 1.7, while those who did not averaged a match rank of 3.2.
Figure 1 shows factors considered by PDs when assembling rank lists. Most PDs found LORs (74%), resident (64%) and faculty recommendations (64%), and interview performance (62%) to be more important in making their rank lists as compared to previous years. Personal statement (70%), medical school performance evaluation (80%), applicant racial/ethnic (76%) and gender diversity (82%), medical school transcript (84%), AOA selection (90%), medical school reputation (94%), USMLE Step 1 scores (90%), USMLE Step 2 scores (94%), honors and awards (88%), research experience (90%), publications (94%), leadership experience (82%), and extracurricular activities (94%) were all weighted similarly by PDs when forming rank lists.
Figure 2 shows important factors considered by applicants in assembling their rank lists during the 2021 application cycle. Mentor recommendations, city/geographic location, proximity to family, and national reputation were highly important factors to applicants. The interview experience was also highly valued, with 48 of 52 (92.3%) respondents rating the interview experience as either important or very important in ranking programs. Ethnic/racial and gender diversity were most often ranked unimportant/less important by applicants (Figure 3). However, only 10% of female and nonbinary applicants, and 23% of non-white applicants considered ethnic/racial and gender diversity of faculty and residents less important or not important at all, compared to 48% of white male applicants.
All 50 PD respondents reported conducting VIs during the 2021 application season. Virtual interview experience responses for program directors can be found in Figure. PDs expressed strong satisfaction with audio, video, internet connection, and platform used, and largely agreed that VIs were cost-saving and convenient. Overall, 74% of PDs agreed that they were able to adequately assess applicants using a virtual medium. Eighty-four percent agreed that VIs were timesaving, 82% agreed that VIs were effective, and 82% agreed that VIs were easy to organize. Despite 62% of PDs agreeing that they were more likely to rank home applicants higher because of VIs and lack of in-person rotations, 64% said they would consider implementing VIs in the future. PDs who used Thalamus (n=17) generally expressed greater positivity regarding VIs and a higher degree of confidence with rank lists derived from VIs, although this trend does not approach statistical significance due to the small sample size.
Figure 5 summarizes the virtual interview experience for applicants. Of the 52 applicants surveyed, 58% would recommend VIs in the future, with the majority reporting satisfaction with their interview day and ability to get to know the residency programs. Most applicants were neutral (52%) when asked if they would change their rank order if interviews were in-person. However, less than 50% of applicants felt they got to know residents or faculty members well with the use of a VI platform.
Figure 6 shows the geographic distribution of applicants by state of the applicant’s medical school. Of the 52 applicants surveyed, 22 different states were represented.
This study is among the first to evaluate factors influencing rank-list creation among orthopaedic surgery residency PDs and applicants during the COVID-19 pandemic. PDs indicated increased importance of LORs, resident and faculty recommendations, and interview performance, as compared to previous application cycles (Figure 1). This was likely due to limited in-person interactions. Applicants considered mentor recommendations, city/geographic location, proximity to family, national reputation, and VI experiences as highly important factors when ranking programs. Most PDs and applicants were satisfied with VIs and felt able to adequately assess prospective applicants and residency programs, respectively.
This study has important limitations, including its survey-based design and low participation rates, which prohibited statistical comparisons. However, such a study is only possible via survey format. Applicant participation was limited to those who successfully matched, which may have influenced results. Unmatched applicants were excluded as they may be susceptible to social-desirability bias (i.e. survey responses influencing future ability to match into orthopaedics). Additionally, this cohort may have been biased against virtual interactions due to their inability to match. Many of the survey questions were designed using a Likert scale, which may distort data due to participants avoiding extreme responses, representing a central-tendency bias and social-desirability bias. The low applicant response rate was an important limitation which limited generalizability, however the overall applicant sample was diverse with respect to geographic distribution, gender, race, and age of applicants, which increased generalizability. Low response rates in survey studies have traditionally been thought of as indicative of a higher probability of nonresponse bias, however recent studies have shown that response rate and nonresponse bias are poorly related.13,14,15 A low response rate does not by itself indicate nonresponse bias. Factors which are known to increase nonresponse bias include interviewer administered questionnaires, general population surveys, and surveys in which the respondent does not have a relationship with the organization. These factors were minimized by having respondents complete surveys in the setting of their choice, survey of a specific group (orthopaedic surgery applicants), and by having the solicitation of survey completion come from the matched applicant’s program coordinator.
Importantly, while PDs were able to compare their experience with VIs to in-person interviews, most applicants surveyed did not have in-person orthopaedic residency interview experience with which they could compare their VI experience. Lastly, it may be too early to fully appraise PD and applicant satisfaction with the resident selection process, as incoming residents have yet to fully assess their fit with programs they matched at, and satisfaction may change over time. Despite these limitations, this is the first study to report PD and applicant satisfaction with VR and VI platforms and can therefore play an important role in informing rotation and interview practices during future application cycles. PD and applicant satisfaction after completion of intern year may change and could an area for future investigation.
Despite success of virtual interactions, traditional factors in applicant ranking will likely remain important in applicant selection, as PDs were more likely to rank home applicants higher due to VIs and lack of in-person rotations (Figure 4). This may place applicants without home orthopaedic programs at a substantial disadvantage. This also highlights the issue regarding regional and institutional bias with the application match process.16 Furthermore, while LORs have traditionally been important factors in PD evaluation of applicants, LORs obtained during away rotations may normalize against home program LORs, which are met with a greater degree of skepticism.17,18 As such, lack of away rotations and LORs from outside institutions may represent loss of a critical metric for evaluating applicants.
Away rotations have historically been utilized by PDs to demonstrate the character of their programs and hospital facilities in order to attract the best applicants. Additionally, applicants utilize away rotations to evaluate resident collegiality, hospital facilities, and current resident satisfaction, critical factors in rank-list formation.9,19 Without away rotations, applicants may turn to their mentors to advise where they should be pursuing their surgical training.20
Results of this study suggest that VRs and VIs may be implemented successfully in orthopaedic surgery residency applications, especially as 64% of PDs reported considering future use of VIs, 82% of PDs felt VIs were effective, and 58% of applicants would recommend VIs in the future. Virtual away rotations may also play a role in the future, as applicants participating in VRs averaged a substantially higher average rank of 1.7, as compared to 3.2 for applicants who did not participate in VRs. Higher ranking of applicants completing VRs may be interpreted as those applicants showing greater interest in programs. Our results demonstrating the importance of in-person interactions are consistent with prior studies. For example, away rotations have been cited among PDs as the most important factor in assembling rank-lists, while Seifi et al’s 2020 survey of medical students and residents showed a preference toward in-person interviews.6,21
While our results demonstrate the importance of in-person interactions, they do not necessarily negate the potential role VIs and VRs may play in future application cycles. Future implementation of VIs and VRs may reduce costs associated with in-person interactions while allowing applicants to assess a wider array of programs.22 This may be especially important for applicants from disadvantaged backgrounds and those underrepresented in orthopaedic programs, who may lack the financial resources necessary to travel for away rotations and multiple in-person interviews. Indeed, previous studies have shown that orthopaedic interviews can be prohibitively expensive to applicants, with interview costs estimated to average $7,119. Because of this high expense, 28% of applicants cancel interviews for financial reasons, and 72% borrow money to finance interviews.23 High costs associated with orthopaedic applications are consistent with costs reported amongst applicants in other surgical fields.24,25 As such, by reducing financial burdens associated with in-person interviews and away rotations, VIs and VRs may provide an additional benefit of diversifying the orthopaedic applicant pool. However, if programs offer both virtual and in-person options for rotations and interviews, it is important to recognize that applicants choosing virtual options may be penalized if they are perceived as being less interested than those choosing to attend in-person interviews or rotations.
In the future, VIs may also fill a niche role by limiting the need for applicants to return for a second in-person interview after having previously completed an in-person away rotation. In years past, programs have often asked that students who completed away rotations return for a subsequent in-person interview. However, as these students are presumably quite familiar with the programs, VIs may help reduce the financial and time expenditures necessary for a second in-person interview.
irtual assessment of applicants in medicine has been effectively implemented pre-COVID, during this pandemic, and will continue to play an important role in the selection process post-COVID.26,27,28 Full assessment of the impacts of VIs and VRs on orthopaedic residency applications will require longitudinal, comparative studies evaluating clinical performance of residents and PD satisfaction with the 2021 orthopaedic resident intern classes, especially in comparison to prior years. Additionally, more information is needed regarding prospective resident participation levels in virtual curricula and whether participation influenced PD decisions to interview and rank participants versus nonparticipants. Reassessing the value of VRs and VIs in an application cycle where in-person interactions are permitted will be of value. Lastly, unmatched applicants and those without home rotations may have been disproportionately impacted by curtailment of in-person interviews and rotations. More information is needed regarding this specific cohort’s experience with the virtual format to understand its impact on their matching prospects.
In conclusion, VIs were positively perceived by most orthopaedic surgery residency PDs and matched applicants included in our study. While VIs and VRs cannot fully replace in-person interactions, they may represent a cost- and time-saving option for future residency selection cycles, may help increase diversity within the applicant pool by improving access among disadvantaged applicants, and may provide adjunctive value in the evaluation of prospective applicants who have previously completed in-person away rotations.
