Scientific Article

Volume 22 • December 2021


Cost and Quality-of-Care Website Utilization by the Orthopaedic New Patient Population

Alexander M. Crawford, MD, William Ciurylo, DO, MA, Jeremy T. Smith, MD, Eric M. Bluman, MD, PhD, Brandon E. Earp, MD, Christopher P. Chiodo, MD

©2021 by The Orthopaedic Journal at Harvard Medical School

Online Ratings Orthopaedics PDF


BACKGROUND Efforts to increase transparency of healthcare quality and cost reporting has resulted in the proliferation of cost and quality-of-care websites (CQCWs). Little is known, however, about the rate of public utilization of these websites. The purpose of this study is to investigate the prevalence of use and perceived value of CQCWs by patients seeing an orthopaedic provider for the first time.

METHODS Three hundred adult orthopaedic outpatients were surveyed prior to their first ambulatory orthopaedic encounter. Participants were asked if they had accessed specific CQCWs prior to selecting their orthopaedic surgeon. Participants were also surveyed about their perceptions regarding the utility of CQCWs compared to other factors when selecting their surgeon.

RESULTS Overall, 167 of 300 patients (55.7%) reported knowing of at least one CQCW. However, only 83 patients (27.7%) actually consulted a CQCW prior to choosing their orthopaedic surgeon. Though Blue Cross Blue Shield’s Cost and Quality Tools was the most popular website investigated, it was used by only 9% of our patient population. Patients insured by Medicare were significantly less likely to use CQCWs, as were patients older than 70 years of age. Those who used CQCWs valued other information (such as surgeon reputation and insurance coverage) more than information found on CQCWs.

CONCLUSIONS A slight majority of orthopaedic outpatients reported being aware of CQCWs. However, relatively few patients found utility in consulting these websites. These results highlight a potential opportunity for improvement within the domain of cost and quality-of-care reporting.

LEVEL OF EVIDENCE Level V Cross-Sectional Analysis

KEYWORDS Cost and quality-of-care website; physician-rating-website, orthopaedic surgery


Cost effectiveness and quality of care are integral to healthcare reform strategies. In attempts to improve value, private business ventures, government agencies, and commercial insurance companies have begun to report cost and quality metrics. Private businesses have created what are known as physician-rating websites (PRWs) such as RateMDs, HealthGrades, and Yelp. Meanwhile, government agencies such as the Centers for Medicare and Medicaid Services (CMS) and commercial insurance companies have created what our group calls cost and quality-of-care websites (CQCWs). Examples of CQCWs include Hospital Compare, Physician Compare, Blue Cross Blue Shield’s Cost and Quality Tools, and Aetna’s Member Payment Estimator.

PRWs have been an intense focus of recent literature. These websites allow anonymous users to comment on and rate physicians.1,2 The prevalence of PRW use in the general population has been found to range from 23% to 28%.3-7 It has been reported that between 3% and 32% of physicians are rated on any individual PRW and that the vast majority of ratings are positive.8-14 While PRWs do provide increased transparency in the healthcare system, this transparency comes with several shortcomings. Ratings may not come from verified patients,1,2 may be based on a low number of reviews,9-11 and have not been associated with better outcomes.15,16

In contrast, little has been written about CQCWs. CQCWs, simply stated, are governmental or commercial insurance websites that attempt to provide information regarding cost and/or quality-of-care metrics from verified patient encounters.17,18 For example, Physician Compare displays performance data from the Physician Quality Rating System, which tracks over 200 quality of care metrics as well as biographic information about physicians who participate in Medicare.18

The focus of this study was to evaluate the use of CQCWs within the orthopaedic new patient population, and to examine patients’ perceptions regarding the utility of these websites. Specifically, we examined the frequency with which new orthopaedic surgery patients use CQCWs, which websites were utilized, and whether patient demographics affected website use. Additionally, we evaluated patients’ perceptions about the utility of information reported on these websites with regard to their selection of the orthopaedic surgeon they saw.




Following approval from the hospital’s Institutional Review Board, 300 adult orthopaedic outpatients were prospectively recruited at a tertiary academic medical center. All patients were seeing their provider for the first time. Patients were excluded only if they were younger than 18 or did not speak English. All participants gave informed consent before participating. Enrollment occurred during June and July of 2015 and compensation was a $5 gift card.


The CQCWs eligible for inclusion were determined a priori. Websites were included in analysis if:

  1. a publicized cost or quality-of-care website affiliated with governmental agencies, or
  2. a cost or quality-of-care website offered through the most common commercial insurers within our patient population

Generally, websites affiliated with commercial insurers required users to create a profile and were restricted to only those patients of the same insurance type. The final list of websites included was agreed upon by a consensus among the authors and is shown in Table 1. These websites contain a variety of information that is either factual (e.g. reports from the most recent Joint Commission visit in the case of Joint Commission Quality Check) or subjective but based on verified patient encounters (e.g. physician ratings within one of the commercial insurance websites).

Prevalence of individual CQCW use amongst the 83 patients who reported to use at least one CQCW prior to their appointment. All CQCWs considered for this study are included. Some patients visited several websites prior to their appointment.
Orthopaedic Website Ratings Table 1
All Rights Reserved. Permission For Use Required.



Participants were asked to complete a 19-question written survey (Appendix 1*). This survey included questions about patient awareness of and use of individual CQCWs, as well as their perception regarding the importance of information provided by such websites when selecting an orthopaedic surgeon. The survey also included questions about other factors that may have influenced their choice of provider (e.g. provider’s reputation, physician referral, location, and friend/family recommendation). Use of CQCWs was determined both for CQCWs as a whole and for each individual website. Demographic cohorts were compared on the basis of those who used one or more CQCWs versus those who used none. Perceived importance results were stratified using a 5-point Likert scale, anchored with Unimportant and Very Important. Responses of Important or Very Important were grouped together as top-box responses and used to determine relative importance of these factors. Responses to perceived importance questions were considered only from participants who had used one or more CQCWs.


Two-tailed Fisher’s Exact Tests were used to determine whether use of CQCWs differed by patient demographic factors. 95% confidence intervals were calculated for comparative importance of factors in selecting a new orthopaedic surgeon. Results with P<0.05 were considered statistically significant.



Three hundred of the 344 new orthopaedic surgery patients approached agreed to participate in the study (87.2% response rate). The study cohort was comprised of 93 males and 207 females with an average age of 55 years (Table 2). Seventy-two patients (24.0%) were insured by Medicare. Five (1.7%) were insured by Medicaid.

Demographics of our orthopaedic new patient population
Orthopaedic Website Ratings Table 2
All Rights Reserved. Permission For Use Required.


Overall, 167 patients (55.7%) reported being aware of at least one CQCW while 133 (44.3%) had never heard of any. Eighty-three patients (27.7%) reported using at least one CQCW in selecting their new orthopaedic surgeon.

Individual website use by the total cohort ranged from 1 to 9%, with Blue Cross Blue Shield’s Cost and Quality Tools being the most frequently used (Table 1). The most frequently used government sponsored CQCW was the Commonwealth of Massachusetts Board of Registration in Medicine, although this was only used by 4.3% of our patients (Table 1).

Use of CQCWs by demographic is reported in Table 3. Those insured by Medicare, those older than 70, and females were significantly less likely to use CQCWs than their respective comparative cohorts (P=0.011, 0.019, 0.026 respectively (Table 3). Race, orthopaedic subspecialty, and income were not significantly associated with CQCW use.

CQCW use by various demographics in our patient population
Orthopaedic Website Ratings Table 3
All Rights Reserved. Permission For Use Required.


Factors that influenced a patient’s decision to select a new orthopaedic surgeon are listed in Figure 1. Overall, 83 patients used at least one CQCW and were therefore selected for this analysis. Reputation of a surgeon, insurance coverage, physician ratings (such as those found on PRWs) and referral from another provider were considered the most important factors in selecting a new orthopaedic surgeon. Quality-of-care information provided by commercial and government websites was deemed important by patients, with 73% and 68% of patients indicating that these factors were important or very important, respectively. Fewer patients reported that cost-of-care information was important. Cost to insurance company was deemed important by the fewest number of patients.

Importance of various factors in selecting a new orthopaedic surgeon amongst those who accessed CQCWs. The percentage represents the number of patients who selected top-box responses of Important or Very Important for a given factor.
Orthopaedic Website Ratings Figure 1
All Rights Reserved. Permission For Use Required.




While much has been written about physician rating websites (PRWs), less is known about cost and quality-of-care websites (CQCWs). This distinction is important because recent literature has suggested that the ratings found on PRWs may not come from verified patients,1,2 may be based on a low number of reviews,9-11 and have not been associated with better outcomes.15,16 By contrast, CQCWs offer the potential to convey either factual information (such as in the case of CMS Provider Charge Data) or subjective data that is based on confirmed patient encounters (such as in the physician rating portions of commercial insurance websites). As such, CQCWs may deliver meaningfully different information to patients than they would receive on PRWs. Within this context, we sought to characterize the use and perceived value of CQCWs as defined by patients seeing an orthopaedic provider for the first time.

In the population we studied, 55.7% of patients were aware of and 27.7% had actually used a CQCW. To our knowledge, the rates of awareness and use of these websites among orthopaedic patients has not been previously investigated. Meanwhile, it has been documented that approximately 65-74% of patients have heard of and 23-28% of patients have used PRWs when selecting a physician.3-7 Thus, while slightly more patients have heard of PRWs than CQCWs, the prevalence of use of both are roughly equivalent at least based on historical standards.

Individual CQCW use was fairly infrequent across our patient population. With the exception of BCBS’s Cost and Quality Tools, no individual website was used by more than 13 (4.3%) of our patients. Although BCBS’s Cost and Quality Tools website was used by 27 (9.0%) of our patients, BCBS did insure a plurality (30.3%) of our patient population. This is important context because most commercial insurance websites are only offered to those patients with that specific insurance. Subgroup analysis amongst commercial insurers revealed wide heterogeneity in the utilization of their websites by their respective patient populations. For example, Cigna’s Find Doctors and Services website performed comparatively well as it was used by 100% of Cigna patients, whereas Harvard Pilgrim’s Now iKnow website performed relatively poorly as it was used by only 25.6% of those insured by Harvard Pilgrim. (Table 1). We suspect this heterogeneity may be due to a combination of websites being marketed differently by individual insurers, website quality dissimilarities, different market dynamics (i.e. co-pays) amongst insurers, and disparate plans (i.e. health maintenance organizations versus preferred provider organizations) utilized by differently insured subgroups.

However, even amongst the open access CQCWs (e.g. Physician Compare) utilization remained quite low (ranging from 1 to 4.3% of our patient population). We believe this could be due to lack of advertising of open access CQCWs, though the evidence of this is currently lacking. We also suspect that the substantial number of different open access CQCWs makes it difficult for patients to keep up with all of the available information, which is often unique on each website. The implication of this finding is that patients who search for cost and quality-of-care information online are likely accessing different information based on the website they visit. We believe the large number of unique CQCWs may be exhaustive to patients. Perhaps one centralized open access CQCW with all appropriate information would achieve more traction.

We found that patients insured by Medicare accessed CQCWs at a significantly lower rate than did all of our other patients (P=0.011). One possible explanation for this is that the Medicare patient population skews older. Prior literature has demonstrated that elderly patients are less Internet proficient19 and that they use PRWs at a lower rate5,6 than younger patients. Indeed, we found this to be the case amongst our patient population; those who were 70 years old or older used these websites significantly less frequently than the rest of the patient population (P=0.019). Therefore, some of the difference between CQCW use by Medicare patients versus the rest of our patient population might be explained by age. Additionally, although no Medicaid patients used CQCWs, too few were surveyed to determine the significance of this finding (P=0.327).

We additionally found men to use CQCWs significantly more than women (P=0.026). This finding is not well documented in the existing literature. In fact, studies have found that women access PRWs6 and general health websites at significantly higher rates than men.20 None of these studies, however, investigated how gender influenced website use amongst the orthopaedic new patient population.

Among those who accessed the websites investigated, factors other than CQCWs were reported by many patients to be more important than CQCWs. These factors included surgeon’s reputation, insurance coverage, physician ratings, and referral. Physician ratings, which can be found on some hospitals’ websites but are mostly found on PRWs, were still considered important or very important by the majority (80.3%) of our patients who used CQCWs1,21 (Figure 1). This is concerning because it has been documented that ratings on PRWs are determined by very few reviews, are not from verified patients, and are not associated with better outcome measures.1,2,9-11,15,16 Though aspects of CQCWs have been called into question, they do provide mostly objective measures from verified patient encounters and may provide a better alternative to PRWs.17,22-24 Further investigation into the public’s perception of PRWs versus CQCWs is recommended.

Our study was limited to new patients visiting an academic tertiary care facility. No attempts were made to normalize our study population or to weight results based on representative demographic. Moreover, some of the websites investigated are relevant only to the Massachusetts market. Therefore, the generalizability of our findings is not known. Nevertheless, we believe the data presented here highlight the need for more, and larger studies to be performed across the country regarding patient behavior as it relates to cost and quality-of-care information.



In summary, most orthopaedic new patients have heard of CQCWs but a majority-particularly those insured by Medicare, those older than 70, and females-do not use them. Even amongst those who actively seek out CQCWs before selecting a new orthopaedic surgeon, other factors are considered more important than much of the information found on CQCWs. Improving quality and minimizing cost are desirable goals for healthcare reform. However, patient behavior as it relates to cost and quality-of-care information should be considered prior to designing further reform initiatives.

1. Lee V. Transparency and Trust - Online Patient Reviews of Physicians. N Engl J Med. 2017 Jan 19;376(3):197-199. PMID: 28099823.
2. Trehan SK, Daluiski A. Online Patient Ratings: Why They Matter and What They Mean. J Hand Surg Am. 2016 Feb;41(2):316-9. PMID: 26754195.
3. Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Public awareness, perception, and use of online physician rating sites. JAMA. 2014 Feb 19;311(7):734-5. PMID: 24549555.
4. Hanauer DA, Zheng K, Singer DC, Gebremariam A, Davis MM. Parental awareness and use of online physician rating sites. Pediatrics. 2014 Oct;134(4):e966-75. PMID: 25246629.
5. Curry E, Li X, Nguyen J, Matzkin E. Prevalence of internet and social media usage in orthopedic surgery. Orthop Rev (Pavia). 2014 Aug 13;6(3):5483. PMID: 25317312; PMCID: PMC4195991.
6. Terlutter R, Bidmon S, Röttl J. Who uses physician-rating websites? Differences in sociodemographic variables, psychographic variables, and health status of users and nonusers of physician-rating websites. J Med Internet Res. 2014 Mar 31;16(3):e97. PMID: 24686918; PMCID: PMC4004145.
7. Emmert M, Meier F, Pisch F, Sander U. Physician choice making and characteristics associated with using physician-rating websites: cross-sectional study. J Med Internet Res. 2013 Aug 28;15(8):e187. PMID: 23985220; PMCID: PMC3758064.
8. Lagu T, Hannon NS, Rothberg MB, Lindenauer PK. Patients' evaluations of health care providers in the era of social networking: an analysis of physician-rating websites. J Gen Intern Med. 2010 Sep;25(9):942-6. PMID: 20464523; PMCID: PMC2917672.
9. Gao GG, McCullough JS, Agarwal R, Jha AK. A changing landscape of physician quality reporting: analysis of patients' online ratings of their physicians over a 5-year period. J Med Internet Res. 2012 Feb 24;14(1):e38. PMID: 22366336; PMCID: PMC3374528.
10. Strech D, Reimann S. [German language physician rating sites]. Gesundheitswesen. 2012 Aug;74(8-9):e61-7. PMID: 22189671.
11. Emmert M, Sander U, Pisch F. Eight questions about physician-rating websites: a systematic review. J Med Internet Res. 2013 Feb 1;15(2):e24. PMID: 23372115; PMCID: PMC3636311.
12. Black EW, Thompson LA, Saliba H, Dawson K, Black NM. An analysis of healthcare providers' online ratings. Inform Prim Care. 2009;17(4):249-53. PMID: 20359403.
13. Kadry B, Chu LF, Kadry B, Gammas D, Macario A. Analysis of 4999 online physician ratings indicates that most patients give physicians a favorable rating. J Med Internet Res. 2011 Nov 16;13(4):e95. PMID: 22088924; PMCID: PMC3222200.
14. López A, Detz A, Ratanawongsa N, Sarkar U. What patients say about their doctors online: a qualitative content analysis. J Gen Intern Med. 2012 Jun;27(6):685-92. PMID: 22215270; PMCID: PMC3358396.
15. Okike K, Peter-Bibb TK, Xie KC, Okike ON. Association Between Physician Online Rating and Quality of Care. J Med Internet Res. 2016 Dec 13;18(12):e324. PMID: 27965191; PMCID: PMC5192234.
16. Emmert M, Sander U, Esslinger AS, Maryschok M, Schöffski O. Public reporting in Germany: the content of physician rating websites. Methods Inf Med. 2012;51(2):112-20. PMID: 22101427.
17. Razmaria AA, Livingston EH. JAMA PATIENT PAGE How to Use Online Clinician Rating Systems. JAMA. 2015 Oct 6;314(13):1418. PMID: 26441197.
18. Frost C, Mesfin A. Online reviews of orthopedic surgeons: an emerging trend. Orthopedics. 2015 Apr;38(4):e257-62. PMID: 25901617.
19. Pew Research Center. Older Adults And Technology. [3 April 2014; cited 2017 April]. Available from:
20. Pew Research Center. Older Adults And Technology. [28 December 2005; cited 2017 April]. Available from:
21. Lee VS, Miller T, Daniels C, Paine M, Gresh B, et al. Creating the Exceptional Patient Experience in One Academic Health System. Acad Med. 2016 Mar;91(3):338-44. PMID: 26606723; PMCID: PMC4885533.
22. Robeznieks A. Still a work in progress: providers skeptical of accuracy of CMS' physician compare website. Mod Healthc. 2013 Jul 1;43(26):10. PMID: 23878900.
23. Missios S, Bekelis K. How well do subjective Hospital Compare metrics reflect objective outcomes in spine surgery? J Neurosurg Spine. 2016 Aug;25(2):264-70. PMID: 26989975.
24. Ritchie A. CMS Launches 'Open Payments' Site Despite Physician Concerns. Med Econ. 2014 Oct 25;91(20):50. PMID: 26242084.