Scientific Article

Volume 21 • October 2020

What Factors Influence Online Ratings for Academic Orthopaedic Surgeons?

Brandon E. Earp MD, Nattaly E. Greene BSN, Kyra A. Benavent, BS, Tamara D. Rozental, MD

The authors report no conflict of interest related to this work.

©2020 by The Orthopaedic Journal at Harvard Medical School

Online Ratings Orthopaedic Surgeons PDF

ABSTRACT

Many physician-rating websites exist for patients to evaluate their health care providers. While studies have examined online ratings of orthopaedic surgeons, few have been able to identify factors that may directly contribute to patient ratings and reviews. Our aim was to identify specific factors: academic, demographic or professional, that may influence patient ratings of orthopaedic surgeons. Online searches of three popular physician rating websites were performed for faculty orthopaedic surgeons from 4 academic, tertiary care centers, and patient ratings were obtained. Additionally, demographic and professional data were recorded. The ratings were compared using Student’s t-test and Chi square tests for physicians with low (<4) and high (≥4) ratings on each site. Among 165 physicians, there were 14,569 reviews, with 28 women (17%) and 137 men (83%); 137 (83%) categorized their race as Caucasian. All orthopaedic subspecialties were represented, and all providers had at least one rating. Age, sex, race, years in practice, education and subspecialty did not influence the overall physician rating or the number of ratings. Senior academic ranks were associated with higher ratings at two out of three sites. Physicians with consistently low ratings across three sites had a lower academic rank and were less likely to have attended a top medical school. In our physician population, education and academic rank seem to have the most impact. Most importantly, reviews are not consistent across sites. Physicians should be aware of factors that influence ratings and continue to explore measures to best evaluate patient satisfaction.

LEVEL OF EVIDENCE Level V Prognostic Study

KEYWORDS Patient reviews, physician rating website, orthopedic surgeon, social media, online ratings

 

Online physician ratings and social media are evolving and becoming increasingly common tools for patients selecting members of their health care team.1 A plethora of physician-rating websites (PRWs) has emerged, allowing patients to evaluate their physician and review ratings provided by prior patients. A survey of the U.S population found that 65% of respondents were aware of PRWs, with 19% of respondents reporting these sites as “very important” to choosing a physician; similar findings have been demonstrated in European countries, with almost 75% of respondents reporting awareness of PRWs.2-4 Healthgrades, a leading PRW, boasts over 30 million visits per month.5

Several orthopaedic specialties have sought to understand how online ratings influence patient satisfaction.6,7 A study among spine surgeons identified trustworthiness as the most significant predictor of positive ratings from patients.8 Similarly, a review of 250 hand surgeon ratings found a positive correlation between a higher number of ratings and stronger online presence with positive reviews.9 Neither study was able to identify a relationship between specific physician factors such as age or experience and patient satisfaction. Furthermore, the link between online ratings and level of expertise was not explored. Many studies have found that patient reviews can be inconsistent across different websites.2-4,6,9

We reviewed ratings from the three most popular PRWs for 165 faculty affiliated with a large academic orthopaedic surgery program. Our null hypothesis was that there would be no identifiable factors associated with online physician ratings.

METHODS

The list of physician faculty members affiliated with an orthopaedic residency program was obtained from the residency website. This included full time staff at 4 large tertiary care centers, as well as community affiliates. Three popular online rating websites were subsequently searched for each individual physician:

These sites were selected because they are among the most visited for physician ratings and provide detailed information without the need for a paid subscription. All searches were conducted in October of 2018.

Demographic information such as age, sex, race, years in practice, subspecialty, academic rank and leadership positions was collected. The education profile for each faculty member was reviewed to determine whether the physician attended a top 10 medical school according to the US News and World Report (USNWR) rankings and whether they completed post-graduate training in a top 20 USNWR hospital.10,11

The overall rating for each physician was recorded from each website and physicians were separated into those with low ratings (<4 out of 5, on average) or high ratings (≥4 out of 5, on average). The ratings for each physician were then compared across rating sites.

Statistical methods

Data are shown as mean ±standard deviation (SD) for continuous and n (%) for categorical. We compared the physicians with low ratings vs. high ratings using Student’s t-test and Chi square tests. We then used the group of physicians with all 3 ratings and compared the physicians who were “low” (<4) on each rating vs. those who were high on at least 1 rating. Next, we computed all pair-wise concordance among the 3 ratings on our high/low dichotomy with the Kappa statistic. Values <0.2 indicate minimal agreement, 0.2-0.4 fair, 0.41-0.60 as moderate and >0.6 as substantial.12 We then tested the agreement among all 3 ratings using the publicly available % magree macro in SAS using a multi-rater kappa statistic. All analysis was used with SAS version 9.4 (Cary, NC).

RESULTS

The demographics for our patient population are presented in Table 1. There were 165 physicians, with 28 women (17%) and 137 men (83%). One hundred and thirty-seven (83%) categorized their race as Caucasian. All orthopaedic specialties were represented. Thirty-eight (23.1%) physicians graduated from a top 10 medical school and 80 (48.4%) completed post-graduate training at a top 20 hospital according to the USWNR rankings for 2018 Best Hospitals Honor Roll.10,11

TABLE 1
Demographic Characteristics of the study population
Online Ratings Orthopaedic Surgeons Table 1
All Rights Reserved. Permission For Use Required.

 

All providers had at least one rating on one of the PRWs. Vitals.com is specifically targeted to physician reviews, Healthgrades.com allows reviews of both hospitals and providers, and USnews.com is even broader, allowing reviews within and outside of healthcare.

There was a total of 14,569 reviews across all three sites. On Healthgrades.com, 156 (94.5%) of the physicians were rated, on USnews.com 142 (86%) were rated, and on Vitals.com 156 (94.5%) were rated. On Healthgrades.com, 55 physicians (35.3%) had an average score <4 and 101 (64.7%) had an average score of ≥4. On USnews.com, 111 (78.2%) had a low average score and 31 (21.8%) had a high average score. On Vitals.com, 50 (32.1%) had a low average rating and 106 (67.9%) had a high average rating. The mean rating was 4.2 on Healthgrades.com, 4.1 on Vitals.com and 3.1 on USnews.com. Table 2 presents the comparisons between physicians with low and high ratings for each of the three sites. Age, sex, race, years in practice, education and subspecialty did not influence the overall physician rating or the number of low or high ratings. Senior academic ranks (associate professor and professor) were associated with higher ratings at two out of three sites.

TABLE 2
Group comparison for physicians with low and high ratings for Healthgrades.com, Vitals.com and USnews.com
Online Ratings Orthopaedic Surgeons Table 1
All Rights Reserved. Permission For Use Required.

 

We subsequently compared the 140 physicians who were rated at all three PRWs. Those physicians with low ratings (n=27) across all three sites were compared to those with at least one high rating (n=113) at one of the sites. Physicians with consistent low ratings had a lower academic rank (P=0.03) and were less likely to have attended a top medical school (P=0.01) (Table 3).

TABLE 3
Group comparison for physicians with consistently low and high ratings for all three sites
Online Ratings Orthopaedic Surgeons Table 1
All Rights Reserved. Permission For Use Required.

 

The agreement between ratings at the three sites was assessed. While Healthgrades.com and Vitals.com had a similar number of physicians with low ratings (n=55 and n=50, respectively, P=0.55), USnews.com had a much higher proportion of physicians with low ratings (n=111, P<0.0001). Comparing ratings ≥4 between sites, agreement ranged from 29 to 50% and kappa values revealed weak correlation in ratings between sites. Healthgrades.com and Vitals.com demonstrated the best correlation (κ=0.34) (Table 4).

TABLE 4
Disagreement for physicians with high ratings across all three sites
Online Ratings Orthopaedic Surgeons Table 1
All Rights Reserved. Permission For Use Required.

 

DISCUSSION

Online evaluations and reviews have been utilized for aiding consumers in the purchasing and selection of goods and non-health care services, with rates reported as high as 87% and 71%, respectively.2 While prior work by Lagu et al. in 2010 demonstrated that few physicians were reviewed via online ratings sites,13 more recent publications have demonstrated increasing numbers of physicians rated and increasing number of ratings per physician, as rating websites have spread rapidly into the health care sector.14 In our study, ratings were ubiquitous, with all providers having at least one rating on one of the three PRWs chosen.

Although several past publications noted low rates of awareness and usage of online rating sites,15-17 our findings parallel more recent work by Hanauer et al. in 2014, demonstrating that patients are increasingly aware of online rating sites.2 Indeed 59% of respondents reported that ratings sites are either “somewhat important” or “very important” when making choices about physicians. These sites thus have significant impact and influence; of respondents who reported using online sites, 35% selected a provider based on good ratings and 37% chose to not see a provider with bad ratings.

Our study found that age, sex, race, years in practice, education and subspecialty were not significantly associated with the overall physician rating or the number of low or high ratings. Senior academic ranks, however, were associated with higher ratings on two of the three sites investigated. Furthermore, lower academic rank and having attended a medical school outside the top 10 ranking were significantly associated with low ratings across all three rating sites, compared to those who had a high rating at one of the three sites. These findings may be a reflection of the regional economy being heavily weighted towards and influenced by medicine, technology, and higher education.18,19 Prior publications in the orthopaedic literature have explored the influence of a variety of factors on online physician ratings. Several largely non-modifiable surgeon characteristics have been assessed. Frost et al. in 2013 reported on a national sampling of orthopedic surgeons’ ratings and found that surgeons in academic practice had significantly higher ratings, but found no differences based on gender or geography.20 Similarly, within the spine subspecialty, Zhang et al. evaluated 219 spine surgeons and found that being in practice for 20 years or less and being in academic practice were associated with higher ratings.21 Others have demonstrated disparate findings: Ramkumar et al. showed no differences in online ratings between academic vs. nonacademic providers and between arthroplasty vs. non-arthroplasty surgeons.22

PRWs incorporate multiple variables in their ratings. Burn et al. evaluated how accurately 14 different websites rated physicians based on physician-specific characteristics.23 Their study found that of the questions used to determine ratings, only 28% directly rated the physician, 48% rated both the physician and the office, and 24% rated the office alone. Trehan et al. evaluated the ratings and comments of 250 hand surgeons in 2016 and demonstrated that positive overall ratings were associated with a higher number of ratings, Castle Connolly status, and increased online surgeon presence.9 Donnally et al. also evaluated the association of online comments with online ratings and found that spine surgeons were more likely to receive favorable reviews for factors pertaining to outcomes, likeability/character, and negative reviews based on ancillary staff interactions, billing, and office environment.4 They recommended that surgeons take an active role in modifying factors patients perceive as negative, even if not directly related to physicians themselves.

Interestingly, we found little consistency for physician ratings across the three websites. Indeed, the agreement between ratings at the three sites was also assessed and revealed generally poor agreement. Though relatively weak, Healthgrades.com and Vitals.com demonstrated the best correlation (κ=0.34), with a similar number of physicians with low ratings, USnews.com had a much higher proportion of physicians with low ratings. Of the ratings ≥4 between the three sites, agreement ranged from 29 to 50% and kappa values revealed weak correlation. This variability certainly adds another challenge to the validity of PRWs. Potential explanations for this discrepancy between sites include variable methodologies used in determining the ratings, underlying differences in each site’s participating audience, and the relatively low numbers of reviews for some providers.

According to Ramkumar et al., hospital rating systems may be able to solve this issue. By using variations of standard surveys such as Press Ganey, patient reviews may be more accurately obtained.22 Recently, some providers and institutions have begun making internal patient satisfaction ratings publicly available. This may have the benefit of ensuring that only actual patients are participating in the ratings, and that the questions asked are measuring factors which institutions and providers deem important. Ricciardi et al. compared provider-initiated patient satisfaction ratings with commercial online physician rating websites for orthopaedic surgeons and found that provider-initiated ratings showed a greater number of responses, were higher in overall ratings of patient satisfaction, and had a lower percentage of negative comments.24 Although patients may worry about the anonymity of their responses, it seems that hospitals and providers should take a more active role in collecting meaningful patient satisfaction data.

There are several limitations with our study. Our sample of physicians was limited to providers associated with the orthopaedic surgery departments of several academic medical centers in a single metropolitan area and findings may not be generalizable to all geographic regions, surgeons or clinical practice scenarios. The data were collected during a single month and are only reflective of that time period; ratings systems are continually being updated and a different time period might reveal different data. Second, we chose to focus on the overall physician rating per site rather than the individual categories and sub-ratings for each individual. Third, given that the websites are public and anonymous, some of the reviews may be inaccurate, and there is no feature to “verify” that these reviews are from real patients. Fourth, the overall physician rating categories were narrow and varied according to website. Finally, we did not have a minimum or maximum number of reviews required for each surgeon included in this study.

In conclusion, online rating sites have become a pervasive tool utilized throughout society as people make decisions on purchasing and selection of services, including health care. In our patient population, education and academic rank seem to have the most impact. Perhaps most importantly, reviews are not consistent across sites. To combat this, a standard survey may be useful for validating and collecting patient reviews. Physicians should develop more awareness of the factors that influence ratings and contribute to the evolving discussion of how best to measure patient satisfaction.

REFERENCES
1. Rozental TD, George TM, Chacko AT. Social networking among upper extremity patients. Hand Surg Am. 2010 May;35(5):819-823.e1. doi: 10.1016/j.jhsa.2009.12.043. Epub 2010 Mar 15. PMID: 20227837.
2. Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Public awareness, perception, and use of online physician rating sites. JAMA. 2014 Feb 19;311(7):734-735. doi: 10.1001/jama.2013.283194. PMID: 24549555.
3. McLennan S, Strech D, Meyer A, Kahrass H. Public Awareness and Use of German Physician Ratings Websites: Cross-Sectional Survey of Four North German Cities. J Med Internet Res. 2017 Nov 9;19(11):e387. doi: 10.2196/jmir.7581. PMID: 29122739; PMCID: PMC5701087.
4. Patel S, Cain R, Neailey K, Hooberman L. Public Awareness, Usage, and Predictors for the Use of Doctor Rating Websites: Cross-Sectional Study in England. J Med Internet Res. 2018 Jul 25;20(7):e243. doi: 10.2196/jmir.9523. PMID: 30045831; PMCID: PMC6083046.
5. Healthgrades: 2018 Statistics [Internet]. USA: Healthgrades Operating Company; 2020 [cited October 23, 2018]. Available from: https://www.healthgrades.com/doctor-reviews.
6. Donnally CJ 3rd, Roth ES, Li DJ, Maguire JA Jr, McCormick JR, Barker GP, Rivera S, Lebwohl NH. Analysis of Internet Review Site Comments for Spine Surgeons: How Office Staff, Physician Likeability, and Patient Outcome Are Associated With Online Evaluations. Spine (Phila Pa 1976). 2018 Dec 15;43(24):1725-1730. doi: 10.1097/BRS.0000000000002740. PMID: 29975328.
7. Donnally CJ 3rd, Li DJ, Maguire JA Jr, Roth ES, Barker GP, McCormick JR, Rush AJ 3rd, Lebwohl NH. How social media, training, and demographics influence online reviews across three leading review websites for spine surgeons. Spine J. 2018 Nov;18(11):2081-2090. doi: 10.1016/j.spinee.2018.04.023. Epub 2018 Apr 27. PMID: 29709552.
8. Kalagara S, Eltorai AEM, DePasse JM, Daniels AH. Predictive factors of positive online patient ratings of spine surgeons. Spine J. 2019 Jan;19(1):182-185. doi: 10.1016/j.spinee.2018.07.024. Epub 2018 Aug 1. PMID: 30077043.
9. Trehan SK, Daluiski A. Online Patient Ratings: Why They Matter and What They Mean. J Hand Surg Am. 2016 Feb;41(2):316-9. doi: 10.1016/j.jhsa.2015.04.018. Epub 2016 Jan 1. PMID: 26754195.
10. Harder B. US World & News Report: 2018-19 Best Hospitals Honor Roll and Medical Specialties Rankings [Internet]. USA: US World & News Report LP; 2020 [cited October 23, 2018]. Available from: https://health.usnews.com/health-care/best-hospitals/articles/best-hospitals-honor-roll-and-overview
11. US World & News Report: Best Medical Schools, 2018 [Internet]. USA: US World & News Report LP; 2020 [cited October 23, 2018]. Available from: https://www.usnews.com/best-graduate-schools/top-medical-schools.
12. McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012; 22(3): 276-82. PMID: 23092060; PMCID: PMC3900052.
13. Lagu T, Hannon NS, Rothberg MB, Lindenauer PK. Patients’ evaluations of health care providers in the era of social networking: an analysis of physician-rating websites. J Gen Intern Med. 2010 Sep;25(9):942–946. doi: 10.1007/s11606-010-1383-0. Epub 2010 May 13. PMID: 20464523; PMCIDPMC2917672.
14. Gao GG, McCullough JS, Agarwal R, Jha AK. A changing landscape of physician quality reporting: analysis of patients’ online ratings of their physicians over a 5-year period. J Med Internet Res. 2012 Feb 24;14(1): e38. doi: 10.2196/jmir.2003. PMID: 22366336; PMCID: PMC3374528.
15. Association of American Medical Colleges. Summary of MCA15. Emmert M, Meier F, Pisch F, Sander U. Physician choice making and characteristics associated with using physician-rating websites: cross-sectional study. J Med Internet Res. 2013; Aug 28;15(8):e187. doi: 10.2196/jmir.2702. PMID: 23985220; PMCID: PMC3758064.
16. Galizzi MM, Miraldo M, Stavropoulou C, Desai M, Jayatunga W, Joshi M, Parikh S. Who is more likely to use doctor-rating websites, and why? A cross-sectional study in London. BMJ Open. 2012 Nov 12;2(6):e001493. PMID: 23148340; PMCID: PMC3532991.
17. Kaiser Family Foundation:2008 Update on consumers’ views of patient safety and quality information [Internet].2020 [cited October 23, 2018]. Available from: http://kaiserfamilyfoundation.files.wordpress.com/2013/01/7819.pdf
18. Abdulkadiroğlu A, Angrist J, Pathak P. The Elite Illusion: Achievement Effects at Boston and New York Exam Schools. Econometrica. 2014 Jan;82(1):137-96.
19. Statistical Atlas: Educational Attainment in Boston, Massachusetts [Internet]. Excelsior, MN: Cedar Lake Ventures, Inc.; 2020 [cited March 5, 2019]. Available from: https://statisticalatlas.com/place/Massachusetts/Boston/Educational-Attainment
20. Frost C, Mesfin A. Online reviews of orthopedic surgeons: an emerging trend. Orthopedics. 2015 Apr;38(4):e257-62. doi: 10.3928/01477447-20150402-52. PMID: 25901617.
21. Zhang J, Omar A, Mesfin A. Online Ratings of Spine Surgeons: Analysis of 208 Surgeons. Spine (Phila Pa 1976). 2018 Jun 15;43(12):E722-E726. doi: 10.1097/BRS.0000000000002488. PMID: 29846366.
22. Ramkumar PN, Navarro SM, Chughtai M, La T Jr, Fisch E, Mont MA. The Patient Experience: An Analysis of Orthopedic Surgeon Quality on Physician-Rating Sites. J Arthroplasty. 2017 Sep;32(9):2905-2910. doi: 10.1016/j.arth.2017.03.053. Epub 2017 Apr 4. PMID: 28455178.
23. Burn MB, Lintner DM, Cosculluela PE, Varner KE, Liberman SR, McCulloch PC, Harris JD. Physician Rating Scales Do Not Accurately Rate Physicians. Orthopedics. 2018 Jul 1;41(4):e445-e456. doi: 10.3928/01477447-20180409-06. Epub 2018 Apr 16. PMID: 29658974.
24. Ricciardi BF, Waddell BS, Nodzo SR, Lange J, Nocon AA, Amundsen S, Tarity TD, McLawhorn AS. Provider-Initiated Patient Satisfaction Reporting Yields Improved Physician Ratings Relative to Online Rating Websites. Orthopedics. 2017 Sep 1;40(5):304-310. doi: 10.3928/01477447-20170810-03. Epub 2017 Aug 18. PMID: 28817163.