Telemedicine is rapidly becoming a powerful tool in today’s healthcare system as it has the potential to improve access and costs while simultaneously improving the quality of care. While telemedicine has experienced tremendous growth in recent years, with over 300,000 patients making use of telemedical services in 2012, the majority of this growth has been in non-surgical fields.¹ In a 2015 PubMed search for “telemedicine telehealth,” 18,416 results are found, of which only 10% (1,862) pertain to surgery and less than 1% (100) pertain to orthopaedics. The promise of telemedicine in surgery has been limited in part by the tactile component of surgery compared to other fields of medicine where remote observation alone may be adequate to provide assistance. However, the emerging technology of augmented reality, which allows surgeons to “reach into” the surgical field, is opening surgery to the telemedicine revolution. The purpose of this review is to define, review, and comment on the use of augmented reality in orthopaedics.

Generally defined as the use of medical information exchanged from one location to another through electronic communications for the betterment of a patient’s clinical status, telemedicine in surgery can be divided into the concepts of “telesurgery” and “telepresence.”^2,3 Telesurgery refers to a remotely located surgeon performing a procedure on a patient through a robotic surrogate.³ The first telesurgery case was reported in 2001 with a surgeon located in New York, NY performing a cholecystectomy on a patient in Strasbourg, France.⁴ The limitations of telesurgery include cost burden, inefficiency (not scalable operation as a surgeon is able to only perform one case at a time) and medical legal and reimbursement concerns.⁵ Telepresence is an audiovisual interaction with a remote surgeon supervising a local surgeon.³ Examples of this are common in the literature with cases of experienced specialists providing guidance to a surgeon with either less experience or in an austere environment. Novel implementation of the wearable device, Google Glass™ (GG) (Google Inc., Mountain Vista, CA), as the local interface for surgical telepresence has captured public interest. Indeed, multiple recent case reports have highlighted the success and potential utility this wearable interface.^6-8

The addition of augmented reality to surgical telepresence is beginning to expand the impact telemedicine has in surgery. Augmented reality is the real time fusion of a user’s visually perceived physical environment with dynamic virtual elements (Figure 1). In surgical telepresence, this technology allows a remote surgeon to directly interact with the local surgeon’s visual field, allowing visual cues such as pointing, outlining, and circling to communicate spatial information. Thus, augmented reality unites the point of view of the remote and local users, allowing intuitive communication of spatial information. This technology is particularly well suited to video-based procedures such as endoscopy, arthroscopy, and endovascular surgery, as the point of view can be easily unified since both local and remote surgeons are viewing the surgical field through a live camera feed. For open cases, augmented reality requires an interface that closely mimics the local surgeon’s visual perspective, and can display virtual elements with spatial accuracy. However, with the advent of wearable audiovisual transmitting devices, even open cases can be assisted with augmented reality technology. In short, augmented reality shows great promise in surgery as an intuitive means of communicating abstract spatial information.

FIGURE 1

Augmented reality during cadaveric dissection

The remote surgeon’s virtually projected hand is seen in the center providing instruction (yellow arrow). The operating surgeon’s hands are seen on the right and left (red arrows).

All Rights Reserved. Permission For Use Required.

-View Large-

Simulation has been a mainstay in medical education for over a century, with obstetrical simulations first being mentioned in the Flexner report of 1910.⁹ As many of the ACGME’s milestones for orthopaedic residents involve progression in the technical aspects of surgery, allowing residents to learn valuable surgical skills without the requirement of patients is extremely valuable. Simulation has the potential to reduce costs and patient harm associated with learning these techniques.¹⁰ The general types of orthopaedic simulators are synthetic, animal, and cadaveric bench top models, low-fidelity skills trainers such as the Fundamentals of Arthroscopic Surgery Trainer (FAST), and more recently, high definition virtual reality simulators.

A simulation’s fidelity, the qualitative degree in which it resembles the model, can vary significantly. While it may be expected that higher-fidelity simulations would be more beneficial to trainees, studies have shown benefits with low-fidelity simulators in surgical training. For example, in general surgery the low-fidelity simulator, Fundamentals of Laparoscopic Surgery (FLS) (Society of Endoscopic and Gastrointestinal Surgeons, Los Angeles, CA), has been shown to be superior to a comparable high-fidelity simulator in skills transfer to the operating room.¹¹ In orthopaedics, a model has been created to practice basic knot tying skills with proposed cost savings of greater than $10,000 over high-fidelity training systems.¹²

In general surgery, laparoscopic and endoscopic virtual reality trainers have become an integral part of residency training, allowing objective feedback to improve resident skill acquisition.^13,14 The American Board of Surgery requires completion of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES) fundamentals of laparoscopic surgery (FLS) program for board certification in general surgery.¹⁵ Examples of virtual reality simulation in general surgery resident education include full-procedure models of laparoscopic cholecystectomy, appendectomy, sigmoidectomy, and gastric bypass.¹⁶ Virtual reality simulation has also been used in neurosurgery¹⁷, ophthalomogy¹⁸, otolaryngology¹⁹, and urology resident education.²⁰

Various virtual reality simulators have been used in orthopaedic surgery, with generally positive results. The first of these was a knee arthroscopy haptic feedback simulator developed in the late 1990’s using funding from the American Board of Orthopaedic Surgery.²¹ Since that time, various arthroscopic simulation systems have been developed for the knee and the shoulder. Studies have demonstrated construct validity of these simulators, as they are able to differentiate between surgeons at various levels of their training.^22,23 A recent study has even demonstrated transfer validity, as residents who practiced on high fidelity virtual reality knee simulators had better probing and visualization skills than those who did not.²⁴ Similar results were seen in a high fidelity shoulder model, as resident learning was accelerated after simulation training.²⁵ Much of the existing literature on virtual reality in orthopaedic training has been related to arthroscopy with less attention given to open procedures. However, virtual reality models of both hip and ulnar fractures have been shown to have construct validity when attending surgeons and residents were compared.^26,27 The transfer validity of these simulations has not yet been evaluated.

The potential roles of augmented reality in surgical training are numerous; augmented reality can be utilized in animal, cadaveric, high and low fidelity simulations in addition to actual procedures. Augmented reality has been utilized in resident training of complex procedures with small surgical fields, often in endoscopy, laparoscopy, and arthroscopy.^17-20 In FLS trainers, the addition of augmented reality instruction was found to shorten the learning curve for novices learning intracorporeal suturing techniques.28 Augmented reality technology has also successfully allowed remote instruction of complex surgical procedures in cadavers.³ In that study the attending neurosurgeon provided instruction to a resident physician by virtually placing his hands into the resident’s viewing field during either a carotid endarterectomy or pterional craniotomy (Figure 2). The authors concluded that the augmented reality technology was well-suited for remote instruction, as anatomic structures could be identified adequately and the attending surgeon’s virtual projections contributed to the resident’s learning. A similar study from researchers at Bangor University in the United Kingdom using a separate augmented reality system, the Bangor Augmented Reality Education Tool for Anatomy (BARETA), has been used to instruct medical students in neuroanatomical dissection.²⁹ This system projected virtual overlays onto the cadaver to aid in student learning of the ventricular system (Figure 3).

FIGURE 2

Augmented reality-aided cadaveric dissection. The local surgeon is able to see the remote surgeon’s hands in the microscopic image.

All Rights Reserved. Permission For Use Required.

-View Large-

FIGURE 3

The BARETA system. Virtual 3D images of the ventricular system are projected onto an anatomical specimen.

All Rights Reserved. Permission For Use Required.

-View Large-

Augmented reality in orthopaedic training is a relatively unexplored area. We recently performed a study in which augmented reality was taken beyond the realm of simulators and cadaveric dissection with introduction of this technology into the operating room. Fifteen patients undergoing arthroscopic shoulder surgery were enrolled in this study.6During the surgeries, a remote station was placed in a viewing room for the attending orthopaedic surgeon to instruct a resident through the operation until his presence was required in the operating room. The remote augmented reality station consisted of a monitor displaying the arthroscopic image, a two-way conference call phone, a camera and a computer that digitally overlaid the surgeon’s hand into the surgical field monitor viewed by the resident (Figure 4). Following the case, the attending and residents were given Likert scaled surveys to rate their opinion of augmented reality in efficiency, safety, and utility in teaching. Both attending and residents gave favorable ratings of the technology, with insignificant differences seen between the two. There was no difference in surgical times compared to a non-augmented reality comparison cohort suggesting that addition of augmented reality did not decrease operating room efficiency. A frequent finding among the resident comments was the benefit of increased autonomy with the comfort of having adequate supervision to assure patient safety. The potential benefit of augmented reality to shorten the learning curve in resident education needs additional investigation especially as resident training is transitioning towards competency-based milestones.

FIGURE 4

A surgeon remotely assists in arthroscopic shoulder surgery

Using an augmented reality-equipped camera, the attending surgeon projects his hands into the arthroscopic field of the resident who is in the OR (image insert). He is then able to highlight anatomy and provide instruction without physically being present.

All Rights Reserved. Permission For Use Required.

-View Large-

While augmented reality has logically found utility in education due to the need for simulation of and real time constructive feedback on procedural training, other uses in surgery have been explored. The augmented reality component of robotic surgery is well known; it has also shown potential in non-robotic surgeries that rely on image guidance. Recently, during a pancreatico-duodenectomy, the surgical team used augmented reality with an exoscope and 3D rendering of the patient’s anatomy from CT scan for guidance in visualizing important vascular anatomy during mesenteric artery dissection.³⁰ The VITOM exoscope (KARL STORZ, Tuttlingen, Germany) was positioned above the patient and served as a projector, with accurate representations of mesenteric vessels projected onto the pancreas. In orthopaedic surgery, augmented reality has been used in image-guided procedures such as percutaneous vertebroplasty.³¹ Using a head-mount display and a marker sheet placed on the patient, images were generated which showed the surgeon the correct angle of needle insertion. Results from this study showed decreased error when inserting the needle in the augmented reality-guided procedures when compared to cases in which it was not used. Augmented reality has also been used in small joint arthroscopy. During wrist arthroscopy surgeons were able to visualize instrument position within the joint and accurately navigate the arthroscopic field.⁷

The need for exportation of knowledge and transfer of skills between surgeons in the face of a global shortage surgical specialists has previously been described.³² One study, although still performed as a cadaveric dissection, demonstrated the possibility for augmented reality to serve in teleconsulting. This study involved two neurosurgeons, one at the University of Alabama at Birmingham and the other at the University of Indiana collaborating on a microsurgical dissection of the pineal region of the brain.³² The study was a departure from previous works which involved a resident and attending surgeon in the same hospital, as the two collaborators were located nearly 500 miles apart. In spite of the distance, the surgeons reported that they could effectively communicate with each other, while the augmented reality system showed only a 760 msec delay between audio and video signals.³²

More recently, the authors performed a similar study in orthopaedics while also incorporating a wearable audiovisual transmitting device in conjunction with augmented reality software.³³ Contrary to previous augmented reality reports where interaction was via an image projection of the surgical field, such as arthroscopy, Google Glass (GG) (Google Inc., Mountain Vista, CA) was used to unify the digital elements with the surgeon’s point of view for a shoulder arthroplasty case. Using commercially available augmented reality software, the operating surgeon in Birmingham, Alabama was able to interact with a remote surgeon in Atlanta, Georgia through GG. While GG has been used for live recording, its pairing with augmented reality was new territory in wearable devices. Despite the success of this case and the promise of combining several technological advancements, several issues with the wearable device were encountered in surgery. The GG battery life was considerably shorter than the length of the surgery, which necessitated the use of a portable battery pack by the surgeon. The remote surgeon considered the image quality to below what was needed to comfortably identify small anatomic details. Network delays were experienced, with video lagging behind audio signal. Lastly, the line of sight of the GG could not be adjusted to match the operating surgeon’s line of sight; this led to the remote surgeon to see a noticeably different view. In spite of the technical limitations, GG did not interfere with the surgery and allowed for unprecedented real-time interaction between the surgeons.

The results of these recent applications involving augmented reality to communicate with remote surgeons during surgery suggest the viability of a teleconsulting model in which more experienced surgeons may virtually provide real-time assistance to a less experienced surgeon in a rural or international setting. Additionally, implant companies are beginning to take interest in this technology as a way to improve the return on investment of education courses with providing surgeons remote expert assistance when learning to use their products. The implication, in both cases, is that augmented reality helps to facilitate the communication of abstract spatial information through pointing and visual gestures without requiring the teacher to be physically present (Figure 5). This could likely results in savings of cost and time and for both the company and the surgeon who wishes to attend the course, as a single investment in an augmented reality system could circumvent the need for future travel and missed time at work.

FIGURE 5

Augmented reality with Google Glass during a shoulder arthroplasty

The local surgeon wearing Google Glass is shown on the left. The remote surgeon (right) projects his hands into the local surgeon’s viewing field. The composite image is shown in the center.

All Rights Reserved. Permission For Use Required.

-View Large-

In January 2015, GG was officially pulled from the market after an extended run of the Google Glass Explorer program provided both positive and negative feedback on the product. In spite of what could be considered an unsuccessful run due to the short length of time that it was available, similar products to GG remain on the market, and the GG experience may have served to open the door to a much larger audience with its use in the operating room. Augmented reality technology, however, has recently undergone a technologic shift to make it portable. Whereas one of the earliest augmented reality system used, Vipaar (Vipaar Inc., Birmingham, AL) relied upon station-based platforms similar in size to mobile arthroscopy towers, the ability now exists to use an iOS-based application. This transition allows readily available hand held devices to display augmented reality inserted images into the surgical field. Despite technological advancements and the potential promise of augmented reality in orthopaedics, additional research into its utility and benefits is required.

The technical challenges experienced with the hardware to interface with augmented reality technology are not insurmountable. It is not unrealistic to suggest that with continued progressive technological upgrades that a commercially available device designed for an operating room setting will be available in the next few years. However, beyond the needed technical advancements to mainstream augmented reality into common surgical practice, are the issues of security and privacy. Due to GG system constraints preventing username and password authentication for the hospital’s network, special considerations were required for our institution to gain network access. More significant was communication in a Google Hangout, which was the only means of making a connection using GG. Google Hangout is not HIPPA compliant and required additional provisions to protect patient confidentiality. While it is conceivable that special networks could be created that are hospital and HIPAA compliant, it is clear that hurdles beyond hardware technology must be overcome to allow augmented reality to be increasingly utilized in the orthopaedic operating room.

Following the experiences of combining GG and augmented reality to train residents in shoulder arthroscopy and the shoulder replacement, the authors sought to determine obstacles to adopting emerging technology such as GG in healthcare. This unpublished study surveyed physicians who have reported experience with using GG in medicine. The survey focused upon perceptions regarding technological limitations and administrative obstacles. The majority of the respondents were in the United States, male, assistant professors, and worked in academic institutions. Surgeons accounted for roughly 50% of users. Respondents reported technical issues similar to those previously reported including: ergonomics, display location, field of view, audio and video quality, and battery life.⁸ The majority of users purchased the GG with their own resources and none of the users reported receiving compensation for using GG in their medical practice. Outside of any potential efficiency improvements, questions arise regarding who and how such technology will be paid in the future. A significant concern among a majority of these physician technophiles was security and confidentiality concerns surrounding use of this technology.

It is clear that as technology continues to open the operating room to the world, obstacles beyond technology such as the technical security of hospitals, patient confidentiality, and cost issues such as purchase and reimbursement, will need to be remedied.

Technology is impacting how residents and surgeons learn the practice of orthopaedics. Evidence is mounting that augmented reality can be used in low and high fidelity simulation settings as well as in surgery to improve resident training. Potential benefits of faster skill acquisition, cost savings and risk reduction of patient injury are all desired and require additional investigation. Augmented reality is a potential technology to help usher the telemedicine revolution into the surgical domain. While advances in wearable technology are required along with reimbursement clarity, hospital safety and patient confidentiality concerns, the growing need to bring expert care to remote places in a cost efficient manner remains a driving motivator for this technology.

The authors would like to thank Dr. Barton Guthrie of the University of Alabama at Birmingham and Dr. Rhys Thomas of the University of Leeds for supplying us with images of their respective augmented reality experiences.