A new computational medical XR discipline
by George Papagiannakis
Computational medical XR is a new interdisciplinary field, bridging life sciences, with mathematics, engineering and computer science. It unifies computational science (scientific computing) with intelligent extended reality and spatial computing for the medical field. It extends significantly clinical XR by bringing on computer simulation and other forms of computation from numerical analysis, computational geometry, computational vision and computer graphics with theoretical computer science and machine learning, in order to solve hard problems in medicine: from low-code/no-code authoring medical XR platforms to deep learning systems for diagnostics and therapeutics and from surgical planning to real-time operative navigation in XR.
5 billion people today lack access to proper surgical and anesthesia care and current training models cannot meet this demand. Furthermore, more than 1 billion jobs, almost one-third of all jobs worldwide, are likely to be transformed by technology in the next decade, according to OECD estimates. According to WHO, 40 million healthcare professionals are needed by 2030! This growing need for continuous upskill and reskill becomes even more crucial in the post COVID-19 pandemic era. Virtual Reality (VR) together with 5G spatial computing technologies can pose as the next final computing frontier regarding medical psychomotor/cognitive training, education, and empowerment.
Virtual/augmented/mixed reality (VR/AR/MR)technologies (grouped by the industry as XR) are now transforming medical training. Very recent articles and case studies in industry as well as academia report this fact. Only in 2020, more than 20 clinical trials were published in key scientific journals, measuring and testifying the efficacy of medical VR training and skills transfer from virtual to real. It has also been recently reported that XR technologies can offer significant boost in experiential and collaborative learning of healthcare professionals.
VR/AR can provide the means for remote qualitative education (knowledge) and training (skills), using affordable technology with personalized, on-demand and smooth learning curves. Based on recent major advances in the field of 5G edge computing, neuroscience, and spatial computing:
“VR/AR shares with our brain the same basic mechanism: embodied simulations”.
Such immersive technologies can facilitate continuous learning, provide curriculum programs and self-improvement opportunities that both expand medical professionals’ abilities as well as minimize skill gaps in their training. These spatial computing applications are being based on latest advances on real-time 3D graphics, computer vision, novel algebraic representations, simulation models, gamification, analytics, deep learning and 5G cloud edge computing, that allow intelligent education & training and further optimization/maximization of the desired learning outcomes.
After more than 30 years of intense R&D and intense validation by early-adopters, medical VR technologies are now moving into mainstream.
We know from neuroscientists as well as computer scientists and educators that the feeling of Presence and the affordances of agency (body, hand interaction) drive neuroplasticity and long-term potentiation, which refers to the strengthening of connections between neurons in the brain based on recent patterns of activity. With each repetition, cell-to-cell signaling improves and the neural connections tighten, profoundly affecting learning, muscle memory as well as spatial memory retention. Latest advances in affordable VR hardware coupled with 5G-edge mobile and ubiquitous connectivity with massive scale/outreach are further driving this proliferation. Therapeutic VR, is also nowadays a field were the use of VR in the treatment of traumas, phobias, stress, various mental illnesses as well as in rehabilitation, is now a fact with clinically proven results.
Covid-19 has forever accelerated advances in this field and elevated these spatial computing technologies from a mere “nice to have” to a “must have” for the roughly 3000 medical schools worldwide, the 8400 medical device companies and the 600 surgical training centers, that were all challenged by covid-19 lockdowns. The use of medical VR training is now a new reality and part of the “quiet revolution”, as termed in a recent Accenture industry report, where enterprise VR vastly outnumbers consumer VR applications. Furthermore, healthcare professionals have now acknowledged that training simulators are the number one immediate application of VR/AR in healthcare. The forefront of this revolution is the unexpected new insight that “the rising demand for virtual training across industries drives the VR market”, a $15B market in 2020, expected to grow to $57B, as was also recently reported.
However, the looming question is: ‘who is going to “VRify”(translate into VR) the 400+ basic medical procedures and their countless variations?’
These very basic medical procedures are constantly being updated and/or being extended. Furthermore, as medical device equipment for diagnostic or therapeutic procedures such as robotic surgery get updated, so too do the variations of these basic medical procedures thereby exponentially increasing the number of procedures requiring advanced and extensive training. For healthcare professionals, it’s a constant challenge to upskill / reskill following latest advances in the field as well as for the institutions to train enough of them to meet the increasing demand. Every other week/month/quarter, medical professionals must complete self-training courses or attend lectures/workshops/conferences to provide the best treatment for their patients and remain compliant with regulatory agencies. This is especially true if you’re a med-school student, resident, young or experienced surgeon or physician of any specialty or nurse. The traditional method of training for physicians has been to train on cadavers or on-the-job training, but of course, cadavers are scarce, expensive and logistically limited while training on real patients poses other issues (limited opportunity, repeatability not to mention possible adverse outcomes). With capped OR times for trainees/residents and increasingly fewer of them entering the workforce, the problem only gets worse over time. Similar needs/problems we also witness in the adjacent verticals of vocational or corporate skills training.
Despite the above strides, the question of who will finally meet the vast needs of “VRification” of medical training content, both in extensive depth as well as exhaustive breadth, remains unanswered. This problem is not just a “lack of VR content”, as many believe, but mainly a “lack of dedicated VR content authoring tools”, specifically for medical training. The excellent de facto 3D/XR content creation industrial standards of Unity and Unreal Engines act as the essential enabling technologies and general authoring platforms. However, their general-purpose coverage (from games to automotive, engineering, construction to simulation and film/broadcasting) requires highly skilled developer teams from various technological disciplines (computer graphics, deep learning, computer networking, human-computer-interaction, gamification, game development, UI/UX design, affective computing) coupled with teams of education, medicine, psychology professionals and ofcourse 3D artists/designers.
I believe that new integrated software platforms and simulation-training authoring tools are needed. They will allow rapid prototyping of medical VR/XR training, as part of new “ computational medical XR” discipline that tackles medical education, preoperative planning and real-time operative navigation under a new, holistic integrated systems approach. Medical schools, dental schools, nursing academies, medical device companies and surgical training centers should employ teams in-house or outsource parts of the “VRification” of their curricula but keeping ownership of the final source code that will enable themselves to easily update/extend/modify this XR training content, hence driving themselves further the adoption and customization of their medical VR modules. Through the ability to control and develop their own XR training material, they can ensure their medical professionals are properly and (continuously) trained while ensuring optimal patient outcomes and fewer medical errors/complications. I believe that such new tools will act as the new ‘WordPress’, or ‘Office suite’ of medical VR/XR training, so that eventually the subject-matter-experts themselves, can create their own VR/AR learning modules, in the same manner they create their presentation slides today. Ofcourse there is still work to be done from our side as technologists to provide these low-code/no-code authoring platforms. An amazing recent testament to this trend, is the in-house VR content creation department of the International Committee of the Red Cross and the state-of-the-art content they are creating.
The example of aerospace and aviation, which is similar in regulatory complexity as the healthcare industry, testifies to the unique benefits of simulation-based training. Pilots must undergo mandatory simulation training for several types of aircrafts from various aviation authorities worldwide and this has led to increased safety in aviation worldwide. Several mandates for simulation-based training have also been put forward in medicine, and it is only a matter of time till VR simulation education labs for training and competency assessment will be mandatory on a worldwide basis as well. Medical education will be significantly boosted via experiential learning, since learning by doing has proved to be much more effective than the traditional master-apprentice model and the “see one, do one, teach one” concept.
For the above transformation to be realized, it is important that medical institutions drive themselves the medical VR knowledge transformation via XR content curriculum simulators and continuous education of their professionals, for the immediate benefit of their patients with significant impact to humanity in general.
Let’s make the world a better place by advancing XR computational medicine with novel content authoring platforms!
This is a first article on a series of medical XR articles, ranging from computer graphics methodologies, to VR simulation algorithms, from research and innovation, to academic entrepreneurship. Parts of this article have been presented in the 2nd Swiss Conference for Telemedicine and Digital Health 2021, Bern, Switzerland. For an extended version of this article, with complete references please refer to: https://arxiv.org/abs/2108.04136
