JPOSNA® Special Edition
Advances in Pediatric Orthopaedic Education and Technical Training
The Use of Immersive Virtual Reality (IVR) in Pediatric Orthopaedic Education
1Massachusetts General Hospital Department of Orthopedics, Boston, MA; 2Boston Children’s Hospital/Harvard Medical School Department of Orthopedics, Boston, MA; 3University of British Columbia Department of Orthopaedics, PrecisionOS, Vancouver, BC
Correspondence: Daniel Hedequist, MD, Boston Children’s Hospital, Orthopedic Center, 300 Longwood Ave., Boston, MA 02115. E-mail: [email protected]
Received: June 7, 2022; Accepted: June 13, 2022; Published: August 15, 2022
Volume 4, Number S1, August 2022
The traditional postgraduate surgical master-apprenticeship education model is undergoing a significant paradigm shift. For the past century, the transition from trainee to orthopaedic surgeon has been accomplished through stepwise integration of theoretical knowledge, observation of senior faculty, basic surgical skill acquisition, and gradual autonomy in complex procedures. However, increased pressure to promote patient safety, improve operating room cost-efficiency, and the Accreditation Council for Graduate Medical Education (ACGME) duty-hour restrictions raise concerns for decreased resident autonomy and competency at the end of training. One study on a national level found that only 27% of residents and 4% of program directors felt that the ACGME 80-hour work week restriction would prepare residents for life as an attending.1
Recently, the novel coronavirus 2019 (COVID-19) global pandemic created additional constraints on resident education due to suspended on-campus teaching, decreased elective surgeries, off-service deployment, burnout, and skill decay.2 A restructuring of orthopaedic surgery residency programs and implementation of surgical simulation training modalities are necessary to counter the effects of decreased resident autotomy and surgical case volumes.3
The core principle of simulation training is “learning by doing” for the acquisition and retention of surgical skills.4 It allows trainees to integrate theoretical knowledge with procedural skills in a risk-free setting prior to application in the operating room. Significant data supports the use of surgical simulation and virtual reality training modalities for skill acquisition in nearly every surgical specialty.5 Successful completion of surgical training modalities does not guarantee long-term competence, and skill “retention,” or maintenance, is also required.
Atesok et al. divided the concept of skill retention into three periods: immediate (minutes-hours), short-term (<3 months), and long-term (3 months-years).4 Putman et al. demonstrated the disconnect between skill acquisition and retention in a distal radius fracture fixation model.6 In that study, 10/35 residents (29%) who underwent consecutive testing failed after successfully passing the previous year. This skill decay can occur within days to months when motor skills and knowledge are not actively maintained. The accessibility of simulation alleviates the need for expensive cadavers and allows for readily available, repetitive, unlimited, and deliberate practice in a “safe” environment. An additional benefit of surgical simulation provides immediate feedback to monitor progress or weaknesses.7 A recent review of surgical skill decay for orthopaedic surgeons and trainees highlighted the importance of simulator and specifically IVR use. Lohre et al. illustrated the learning capacities of traditional and IVR simulator teaching modalities using a “virtual ladder of orthopaedic learning” with an endpoint of trainee proficiency.8 The authors provide an additional concrete example of an IVR curriculum that aligns with international graduate medical education formats, such as ACGME and Canadian Medical Education Directives for Specialists (CANMeds).
It is evident that alternative training modalities using simulation and virtual reality will become increasingly implemented in surgical education. The American Board of Orthopaedic Surgery (ABOS) and Residency Review Committee (RRC) created a simulation-based training for PGY-1 residents for orthopaedic surgery in 2013. Since this time, many institutions have employed IVR in their resident and fellow teaching, and large professional organizations such as the American Shoulder and Elbow Surgeons (ASES) have endorsed its use. Still, ongoing critical evaluation of skill transferability from simulation education to “real-life” scenarios/patients and standardized curricula are required as orthopaedic training adapts to changing paradigms and technological advances.
The advances of “reality” technologies in medicine has exploded since the early 2000s as is evident by the exponential rise in the number of articles published in PubMed over the last 20 years (Figure 1). There is no doubt that the expansion of this technology in surgical education and obtaining a basic understanding of each modality is important to readily understand the applications in this ever-developing field.
Virtual reality (VR) is a catch-all term for an artificially created, realistic, and interactive world. Early publications prior to 2019 utilize this term to describe interactive, computer-generated models while contemporary uses describe more highly interactive forms of simulation. Using various hardware interfaces, a user is immersed in a digital environment by receiving multisensory feedback from their virtual interactions with objects in digital space. In orthopaedic surgery, this has been employed by creating realistic operating rooms with instrumentation made to engineering specifications as scaled models with realistic visuals supplemented by equally realistic sounds and sense of touch feedback (haptic feedback). Software designers are also able to utilize three-dimensional volume renderings of patient anatomy using advanced cross-sectional imaging to create interactive anatomical models with both bone and soft-tissue layers. Interaction with these digital worlds and objects are enabled using head-mounted displays (HMD) and position tracking handheld controllers, such as the Oculus Rift (Meta, Menlo Park, CA) (Figure 2).
Immersive VR (IVR) is a contemporary term for VR that incorporates an immersive, digital environment. Lohre et al. first utilized this term to assist in distinguishing the contemporary VR systems from older, non-immersive formats.7 IVR allows surgeons to train in a completely simulated operating room environment with visual, auditory, and haptic (sense of touch) feedback. This multisensory feedback allows for high-level surgical simulation to promote both cognitive understanding and motor skill acquisition. The unique properties of these devices allow for complete immersion into the 3-D world without any interaction in the outside world. The handheld controllers allow for image manipulation in a variety of ways including “flying” through the anatomy, viewing the anatomy through a multitude of angles and orientations, and bone clipping or opacity adjustments in any plane to allow for a greater understanding of anatomy. Virtual surgical instruments are controlled by the handheld devices to rehearse surgery in virtual space. Emergence of CT scans and MRI scans can create exquisite three-dimensional models allowing for a greater breadth of anatomical study (Figure 3).
Virtual reality systems also allow multiple users to be present in the same virtual space, such as an operating room. Social VR or Multi-User Virtual Environment (MUVE) is a virtual environment with more than one user (Figure 4). This allows for either didactic or interactive discussions between instructors and trainees with relevant models and surgical procedures. For example, a trainee may perform pinning of a slipped capital femoral epiphysis (SCFE) in a safe, virtual space, free of time constraints while a pediatric orthopaedic surgeon observes and provides immediate feedback through their virtual avatars. These systems are also able to produce and incorporate patient-specific anatomic models into their procedural training software through uploaded CT scans, allowing surgeons and trainees to practice forthcoming surgeries to better understand relevant anatomy.
Augmented reality (AR) technology allows reconstructed three-dimensional images to be overlaid in real time over a patient’s anatomy using see-through hardware displays. A surgeon, therefore, may interact with both the overlaid hologram and the operative patient, with headsets recognizing surgeon hand motions to rotate, zoom, and move interactive holograms in relation to their field of view. As opposed to the immersive, virtual-only world afforded by VR in preoperative settings, AR differs in the overlay of digital structures onto the real world; therefore, it is used during procedures. As with VR, CT, MRI, or x-rays may be used with software programs to convert two-dimensional images into three-dimensional interactive and holographic models. Commonly used see-through AR displays include the Microsoft HoloLens (Microsoft Corporation, Redmond, WA) and Magic Leap One (Magic Leap, Plantation, FL) (Figure 5).
Virtual and augmented reality lie on a spectrum of Mixed Reality (MR). Between the virtual-only world of VR and the holographic overlay of imaging in AR lies various degrees of either technology in a “mix” of extended reality technologies. MR technology allows visualization of both real-time anatomy and three-dimensional images and allows for interaction with both. These technologies may or may not be superimposed on each other. Further forms of interaction with immersive technology including voice, gaze, and gestures with both the real world and virtual images make the term MR more favorable than just a strict definition of AR/VR, and many publications use this term to describe proprietary software or hardware used for study.
Extended reality (XR) has become an umbrella term encompassing all these technologies. The use of virtual images, real-time anatomy, and interactions of vision, hearing, speech, and touch will continue to merge. The ability to have real-time availability of imaging tests, patient medical records, and patient information at the immediate hands-free request of the surgeon makes XR a broader-based definition of technology.
The Metaverse is the sum of all virtual environments. In its original definition, the metaverse was envisioned a single global virtual space, not unlike the internet or the web. Today, the word is being applied to nearly everything on the internet with any interpersonal interaction and some graphics. Even AR apps are being called “metaverse,” which is a non-sequitur. “VR” suffered similar hype cycles in the 90s and early 00s, leaving a trail of confusion in the medical literature.
Basics of the Technology
XR Hardware and Software
The fundamentals of VR and AR have remained stable for years. However, we cannot recommend any brand or model for VR and AR training because the devices rapidly become obsolete as cheaper, faster, and better hardware enters the market. Instead, we will describe the key features of the technology and its variations to give the reader a strategy to decide what is needed.
The headset covers the wearer’s eyes, which are looking directly into two lenses. If the users require glasses, they should wear them inside the headset, between the eyes and headset lenses. The headset shows you the virtual world in every direction you look. Each lens has a tiny high-resolution screen behind it, one showing the user’s view for the left eye and the other for the right eye. In real life, the separation between your eyes gives each one a slightly different view, and your brain uses that to locate objects in space. This effect is called stereopsis, and a VR headset is a stereoscopic display.
The display resolution refers to the number of pixels shown in each eye of the display. For example, the Oculus Quest 2 is a popular headset in 2022, with 1832×1920 resolution per eye (Meta, Menlo Park, CA). That resolution is adequate for simulations that involve teamwork, basic props, a virtual patient with moderate to obvious visible problems, the ability to move props, and medical monitors. VR headsets completely immerse your optical field with no visual reference or sight of the outside world.
Headsets are also necessary for surgeons (or trainees) using AR, as these allow the surgeon free use of her or his hands during the procedure. Perhaps the best known is the Microsoft HoloLens 2 (Microsoft, Redmond, WA) but there are many options on the market, with more being developed in this expanding field. Nevertheless, they all share certain characteristics. All AR headsets have a particular resolution, which determines the level of detail it can present to the user. For example, the HoloLens 2 resolution is 2048 x 1080 per eye. Just like the VR headset, each lens presents the object as seen from a slightly different angle, which creates the illusion of it being a particular distance from the viewer, stereopsis again.
Unlike the VR headset, the user also sees physical objects in the real world. For the illusion to work, the distance to the real objects and the distance to the virtual object must make sense, creating a single visual narrative. For example, a virtual instrument that is supposed to be lying on a physical table must appear to be on its surface and not floating above the table or sinking into it. Alternatively, AR can give the user a simulated “x-Ray vision,” such as visualizing the patient’s skeleton. In that case, bones must appear inside/within the body and in the correct anatomical location. These are both examples of registration—placing the virtual object into the physical space in a convincing way.
There are two ways to do this. First, a fiduciary marker is a printed symbol that the AR display device can recognize with its camera. The device sees the location of the marker and places the three-dimensional illusions within some distance and orientation to the marker. Second, the device can map the room (referred to as tracking) to then ask the user to place the virtual object in the correct location. The device (a headset usually) keeps the object virtually fixed in space in relation to the room as seen by the user. If an object appears to drift when the user moves their head, the headset probably needs to be recalibrated and/or draw a new, fresh map of the room.
Hand Tracking and Controllers
Most VR headsets come with hand controllers, much like the one shown in Figure 6. It uses a variety of sensors and close communication with the headset to track the controllers with a remarkable degree of accuracy. In most VR applications, the user sees a virtual hand at the same location.
Pressing different buttons on the controller make the hand grip, point, palm-flat, and so on. The best tracking is accomplished with external devices called lighthouses, which are arrayed around the room. They are used by headsets such as the Valve Index or the Vive Pro, which achieve accurate tracking below one millimeter. However, not many applications need that degree of accuracy. Headsets like the HoloLens (AR) and the Quest (VR) can track a person’s actual hand. It uses advanced AI algorithms to interpret the camaras’ view of the hand into a skeleton or outline with which the user can interact with virtual objects and controls. Depending on the software, interaction can be fairly limited to things like pushing buttons. The HoloLens lets the user employ specific hand gestures as commands and often presents virtual control panels the user can touch.
Current Role of IVR, AR, and MR in Orthopaedic Education and Training
It is important to delineate learning in domains of efficiency, quality, and most importantly, effect on outcomes through transfer of learned skill. Studies into simulation technology attempt to ascertain this through various validity measures such as realism, ability to teach, ability to distinguish between levels of skill, and the ability to transfer skills to real-operative scenarios (Figure 7). IVR technology has been shown in multiple studies to be realistic and effective in teaching. Systematic reviews of IVR technology have also demonstrated the cumulative effectiveness of IVR training on orthopaedic combined skill acquisition.9
Cost efficiency is also an important consideration for technology incorporation. By optimizing quality and costs through improved skill transfer, learning efficiency, and proficiency-based metrics, Lohr et al. demonstrated that IVR directly improves the value equation for all shareholders (healthcare systems, institutions, trainees, and patients).8 Providing surgical refresher training and preoperative practice using IVR could reduce operative time and improve patient outcomes, thereby reducing morbidity and opportunity costs across the spectrum of shareholders.
An IVR system has been shown to be cost-effective in teaching surgical residents when compared to traveling for a traditional course. The system studied at the time of publication was $4800 USD which included a software subscription and the hardware system (Oculus Rift 2, Meta, Menlo Park, CA).9 For individuals, this cost may be a detriment. However, at an institutional level, this is quite cost-conscious when compared to other simulators or cadavers when one considers that multiple users can utilize the system over time. Factors to consider when calculating costs are subscription-based software models and periodic hardware upgrades. Currently, the hardware systems are quite cost-conscious, with an Oculus Quest 2 system priced at around $300 USD (Meta, Menlo Park, CA).
Case Example: IVR Use for Teaching Screw Fixation of a Slipped Capital Femoral Epiphysis
Consider a surgeon using the IVR system prior to their surgical case in which they must pin a slipped capital femoral epiphysis (SCFE). The user chooses a relevant module based on preoperative imaging and can have uninterrupted access and practice prior to the real-world scenario. In the module, the user can remove the bone from the patient model, rotate and examine the anatomy, and perform guidewire and screw placement with simulated fluoroscopy (Figures 8A–C).
The module can be performed with a virtual guide including written and verbal instruction of key steps, or this can be turned off for more advanced practice. The user hears the anesthesia machine, can change the table position, move the fluoroscopy arm, rest their hands on the patient, grab an instrument, and hear auditory feedback as the wire spins through cortical to cancellous bone. At the end of the module, a Precision Score™ is provided which tallies performance in key areas of the procedure, and this can be tracked over time or be used as a proficiency reference relative to other users or experts. These modules are also efficiently performed, realistic, and intuitive based on published user feedback.
Recently, a descriptive case report of direct skill transfer from IVR to a real surgical scenario has been published.10 A senior orthopaedic surgical resident with limited experience learned and practiced screw fixation for a severe slipped capital femoral epiphysis (SCFE) prior to performing the procedure under guidance by a pediatric fellowship-trained orthopaedic surgeon in real life. The scenario was quite unique in that the patient had an attempt at pinning at an outside institution and required revision. Using the IVR system, the resident utilized a representative severe grade SCFE and practiced wire localization and screw fixation with simulated intraoperative fluoroscopy. The resident’s Precision Score™ improved over training sessions from 70% to 93% (maximum 100%). The score is calculated based off the overall procedure time, fluoroscopy use, guidewire position, number of wire attempts, screw depth, errors in implantation, and amount of soft tissue damage. The resident reduced their virtual case time from 8 minutes 44 seconds to 3 minutes 46 seconds (232% reduction in training time) and reduced the number of intraoperative fluoroscopy images from 42 to 5 (840% reduction) over multiple training iterations. During the actual operative procedure, the resident was able to place two screws across the physis in a proper orientation and used 7.6x less radiation than the failed index procedure, with single attempts at wire placement to avoid cortical perforations. This case demonstrates how preoperative planning and practice using IVR can be a powerful tool for improving patient outcomes in real operative procedures.
Limitations and Pitfalls
Though high-level evidence exists for specific skill acquisition using IVR, it is important to consider the current limitations in evidence and technology. Surgical skill is a difficult metric to define. Our best approximations come from combined experiences, largely in outcomes from cadaveric procedures, though real-life outcomes have been shown in SCFE pinning and spine surgery after IVR training.
Surgical simulation is a spectrum that includes simple task trainers and bench top apparatus, IVR, and cadavers. Though IVR demonstrates clear theoretical advantages over other simulation formats, it is likely that mixed educational models employing multiple avenues of simulation will be the norm. It is imperative that educators understand the inherent advantages and disadvantages of each learning modality and employ them in the proper learning context.
The current level of interactivity and realism in IVR is limited by available technology in audiovisuals and haptic (sense of touch) feedback. Current commercially available IVR systems have improved significantly from initial offerings, with improved screen resolution, refresh rates, and stereoscopic imaging to prevent motion sickness. Controllers, though ergonomic, function to provide tactile sensation through vibration and function in an on/off or gradient response. Current offerings do not provide the same sense of light or deep touch or hot and cold sensation that humans are used to in the real world. Currently, our brains coalesce a mix of audio and visuals with the rudimentary vibration sense to illustrate, for instance, a pin traveling from cortical to cancellous bone. All attempts are made by the designers to simulate the pitch change and feel that occurs in real life. To this end, IVR cannot perfectly replicate a real operative scenario or replace surgical apprenticeship. However, ongoing work is occurring to produce novel hardware interfaces such as gloves with improved tactile responses, higher resolution headsets, or even body suits that monitor sympathetic nervous system conditions to better replicate the surgical experience in the VR environment.
Further study and data collection of multiple tiers of experienced users is also required to produce construct validity of the systems. Each system and learning module are unique and the cognitive and technical skill sets may differ. Determining construct validity allows for the understanding that the system can distinguish between novice, intermediates, and experts, and therefore provide benchmarks of performance. As trainees gain measurable skill using IVR, the longitudinal retention of these skills will need further study. Current studies demonstrate rapid skill acquisition, though retention beyond the 6-week period has not been studied. As most surgical skills, it is likely that ongoing use and refresher training will be required for maintenance.
Future Role in Pediatric Orthopaedic Education and Training
The capabilities of skill acquisition, metric tracking, and connectivity will be leveraged to provide more robust training solutions for both orthopaedic trainees and surgeons. Preoperative planning is a tenet of orthopaedic surgery, and the unique interactive world of IVR allows for surgeons of all levels to practice infrequently performed procedures or to optimize and refine technique in common scenarios. Recently, FDA approval has been obtained for the use of patient scans in preoperative planning and practice using IVR. As three-dimensional reformats have proven to be more reliable in surgeon classifications, directly importing these images and interacting with them will provide an even further level of understanding of the surgical anatomy. Interactive renderings will provide a safe and unparalleled avenue for preparation for more complex surgical procedures, such as pelvic osteotomies for hip dysplasia, limb deformity correction, or complex spinal deformities in various types of scoliosis.
Surgical preparation using IVR to help offset learning curves for more complex procedures by shifting the burden of error to three-dimensional models rather than real patients is a powerful learning adjunct. IVR software is consistently scalable to user desires, and as popularity grows, so too will the number of procedures and available reference data. Cloud-based data accumulation in this manner can provide insight into which modules are routinely used for practice and VR performance outcomes of varying levels of expertise. Comparing individual user scores to combined performance metrics of experts can help distinguish readiness for performing real surgery. Though other learning modalities are available such as cadavers, the ease of use, affordability, and interactivity of IVR may prove to be the preferred training method in the future.
IVR could also be leveraged to expand orthopaedic training internationally. The portability and connectivity of IVR can allow surgeons or practitioners in remote settings to practice these procedures in a simulated environment with surgeons or other experts from around the world. This unique, multi-user collaborative learning environment can heighten the learning of IVR modules by providing discussion points and key pearls while foregoing the travel burden of instructors. Imagine, for instance, a remote, pediatric treatment group receiving a headset and learning with surgeons from multiple centers, practicing their skills, and subsequently returning for further discussion and learning ad infinitum.
In the world of academic medicine, surgeons are constantly trying to balance teaching technical skills to the next generation of surgeons with optimizing patient care. Surgical simulation training plays a crucial role in this skill development by allowing the surgeon and/or trainee to gain experience in a protected space prior to “practicing” on a patient. Advances in technology within the realm of IVR and AR have and will continue to allow us to dramatically expand our capabilities with preoperative planning and surgical training.
R. Lohre, B. Verhofste, D. Hedequist, and J. Jacobson have no conflicts of interest to report. D. Goel: CEO and Founder of PrecisionOS.
- Levine WN, Spang RC. ACGME duty hour requirements: perceptions and impact on resident training and patient care. J Am Acad Orthop Surg. 2014;22(9):535–544.
- Higginbotham DO, Zalikha AK, Stoker SK, et al. The impact of COVID-19 on the orthopaedic surgery residency experience. Spartan Med Res J. 2021;6(2):25963.
- Dougherty PJ, Cannada LK, Murray P, et al. Progressive autonomy in the era of increased supervision: AOA critical issues. J Bone Joint Surg Am. 2018;100(18):e122.
- Atesok K, Satava RM, Van Heest A, et al. Retention of skills after simulation-based training in orthopaedic surgery. J Am Acad Orthop Surg. 2016;24(8):505-514.
- Mao RQ, Lan L, Kay J, et al. Immersive virtual reality for surgical training: a systematic review. J Surg Res. 2021;268:40–58.
- Putnam MD, Adams JE, Lender P, et al. Examination of skill acquisition and grader bias in a distal radius fracture fixation model. J Surg Educ. 2018;75(5):1299-1308.
- Lohre R, Warner JJP, Athwal GS, et al. The evolution of virtual reality in shoulder and elbow surgery. JSES Int. 2020;4(2):215-223.
- Lohre R, Warner JJP, Morrey BR, et al. Mitigating surgical skill decay in orthopaedics using virtual simulation learning. J Am Acad Orthop Surg Glob Res Rev. 2021;5(10). doi: 10.5435/JAAOSGlobal-D-21-00193
- Lohre R, Bois AJ, Pollock JW, et al. Effectiveness of immersive virtual reality on orthopedic surgical skills and knowledge acquisition among senior surgical residents: a randomized clinical trial. JAMA Netw Open. 2020;3(12):e2031217.
- Lohre R, Leveille L, Goel DP. Novel application of immersive virtual reality simulation training: a case report. J Am Acad Orthop Surg Glob Res Rev. 2021;5(11). doi: 10.5435/JAAOSGlobal-D-21-00114.