AAOS Webinar Surveys the State of Augmented Reality
In an AAOS webinar titled “The Current State of Augmented Reality (AR) in Orthopaedics,” clinicians and representatives from the FDA and from the industry offered information and insights on the use of AR and their perspectives on the benefits and risks of the technology.
Leave Feedback: aaosnow@aaos.org
Following an introduction from program director David Jevsevar, MD, MBA, FAAOS, chair of the AAOS Committee on Devices, Biologics, and Technology, Christopher Harner, MD, FAAOS, FAOA, an FDA clinical deputy director in the Office of Orthopedic Devices, said the October webinar—offered free to AAOS members—was intended to “increase awareness of AR technology in orthopaedic stereotaxic—sometimes referred to as 'stereotactic'—navigation systems,” which make use of a three-dimensional coordinate system to locate small targets inside the body and to perform on them some action such as ablation, biopsy, lesion, injection, stimulation, implantation, or radiosurgery. Among the topics to be covered were the clinical and technological challenges associated with orthopaedic AR applications, the scope of possible adverse events and reporting limitations, and the process for implementing adverse-event reporting by surgeons in a timely and accurate way.
The basics
Dr. Harner clarified the distinctions between virtual reality (VR)—“a completely artificial environment accepted as real by users”—and AR, which is “an integrated technique of image processing.” In an AR system, he explained, “Real objects and virtual (computer-generated) objects are combined in a real environment.” Furthermore, in AR, “Real and virtual objects are aligned with each other and run interactively in real-time.”
Currently, VR is not used in surgical practice, Dr. Harner noted; its main orthopaedic use is in training and education for residents, fellows, practicing surgeons, and other allied health professionals.
AR is in active use for surgical procedures (Table 1), he explained, with some 10 AR device submissions having been cleared by the FDA in the past four years. The literature now includes numerous publications on AR, although most studies are smaller clinical trials (<50 patients), and most were conducted outside the United States. Collected data are on device usage, techniques, and accuracy, but not patient-recorded outcomes, and therefore “are not truly clinical studies,” Dr. Harner said. To this point, reported complications are rare, and the most common documented consequence is increased surgical time.
Describing the technological setup for AR (Fig. 1), Dr. Harner said that obtaining an image to be projected in the OR for the end user, the surgeon, requires multiple steps. Failure at any step of the process—from radiographic imaging to visualization, whether due to hardware, software, OR inadequacies, or failure of the user—can adversely affect patient outcomes.
He noted that the Office of Orthopedic Devices has seen a marked increase in the number of stereotaxic “system” submissions over the past five years.
The challenges
Following Dr. Harner was another FDA representative, CDR Michel Janda, MS, of the agency’s Stereotaxic, Bone Growth Stimulators, and Fracture Fixation Devices Team in the Office of Orthopedic Devices. Mr. Janda addressed the “Regulatory Paradigm and Challenges.”
He explained that the product code “OLO” was established to cover stereotaxic guidance during orthopaedic surgery procedures for joints and spine; this code is the current regulatory basis for which limited applications of AR technology are introduced, including robotics and AR systems. Stereotaxic navigation systems, he noted, may use various technologies, including physical, optical, or inertial patient and instrument tracking. From these technologies arrived robotically assisted surgical systems, including robotic arms that connect passively or actively to directly affect the patient.
The FDA, Mr. Janda said, has identified technical and clinical challenges attendant to AR. Among these are technical characteristics, including:
non-rigid patient tracking
lack of continuous patient tracking
completely immersive VR
excessive obstruction of the user’s field of view
soft-tissue navigation/robotics
AR technology expands on these systems, with the simplest iteration using a head-mounted display to replicate the same information as traditionally shown to the user via a TV monitor. Some technical characteristics are considered outside of the governing regulation—including luminance, contrast, temporal and spatial resolution, the field of view, dynamic range, refresh rate, latency, transmission, and optical aberrations such as distortion—as well as acceptance criteria for technical characterization parameters. The limited number of current AR systems and the limited literature on the technology contribute to the challenge. Gauging the accuracy of the displayed information (i.e., the heads-up display) is another aspect of the evaluation process, with the FDA requesting the same validation as it would for a stereotaxic system.
For a system intended to provide an anchored 3D model corresponding to the patient’s anatomy, developers are asked to provide additional quantitative validation of the model’s spatial accuracy.
“Developing this validation protocol for anchored 3D models is challenging, as these virtual models are intended to be observed and interpreted from a user’s perspective,” Mr. Janda said. “Therefore, objective validation requires observation from an independent exterior method that has the same perspective as the user.”
Emerging technical challenges and considerations that the FDA is observing, Mr. Janda said, include user interaction issues with control mechanisms, such as the technician using a mouse on a separate station, foot pedals, hand gestures, vocal commands, and head movements (dwell pointer). “For example, systems that use a commercial off-the-shelf head-mounted display with hand gestures as a control interface may experience difficulties when used in a medical environment,” he said. “This [difficulty] may be due to the required learning curve and also to use of the interface in this environment” versus a commercial setting.
The FDA is also interested in learning about possible negative effects on a user (patient or surgeon), including eye and neck strain, surgery time, and potentially compromised sterility and biocompatibility.
In general, Mr. Janda noted, novel technologies are associated with a training and learning curve, and the FDA expects challenges to arise in this regard.
In the category of higher-level testing, for stereotaxic systems, the FDA reviews quantitative system-level validation testing which compares the accuracy of a planned implant placement and the final implant placement as executed by the system in a cadaver model; this requirement is also applied to AR systems. Additionally, the agency evaluates human factors and usability testing, including adjunctive use versus integral use of the system, as well as “the ability to discontinue use in a convenient way,” Mr. Janda said.
A new FDA request for approval of an AR system, he said, must be accompanied by a submission of a video or step-by-step illustration of the projected imagery to demonstrate what is intended to be seen by the user. Clinical data, or a redefinition of clinically meaningful endpoints, “has not been found to be routinely necessary as of yet but maybe warranted upon future applications of the technology.”
Finally, Mr. Janda summarized the FDA’s protocol and processes for reporting adverse events. The FDA uses post-market surveillance to monitor device performance. Mandatory reporters (i.e., manufacturers, device user facilities, and importers) are required to submit to the FDA certain types of reports for adverse events and product problems that may cause or contribute to adverse events.
“The FDA also encourages healthcare professionals, patients, caregivers, and consumers to submit voluntary reports about serious adverse events that may be associated with a medical device, as well as use errors, product quality issues, and therapeutic failures,” he said. “With regard to AR technology, the agency is interested in using medical device reporting to assess real-world performance.” The FDA looks to monitor medical device reporting directly associated with AR systems, such as that detailing patient reference movement, loss of virtual image registration, and implant malalignment.
The agency is also concerned that adverse effects may be underreported; these may include user fatigue or distraction, surgical delay, and cognitive overload. These occurrences “may not be reported to us because they may not be obviously associated with an adverse effect,” Mr. Janda said. “We encourage those attending this webinar to use the FDA’s online voluntary reporting program to report real-world experience with this technology.”
The other speakers at the webinar were Francesco Siccardi, MS; Barton Sachs, MD, MBA, FAAOS; and Laurence Coyne, PhD. The webinar is accessible to AAOS members at learn.aaos.org.
