There and back again: ACM SIGGRAPH 2023 highlights

from a computational medical XR perspective

ACM SIGGRAPH 2023 in Los Angeles California, celebrating the 50th annual conference

This August 2023, I had the privilege to attend as speaker the 50th ACM SIGGRAPH conference in Los Angeles, as well as visit collaborators and partners in the Bay Area, which I have not seen up close since quite sometime, due to the pandemic.

ACM SIGGRAPH is a special interest group within ACM, and SIGGRAPH 2023 is the premier conference for computer graphics, VR, AR, XR and interactive techniques worldwide. This year, we celebrated the 50th conference and reflected on half a century of discovery and advancement while contributing in charting a course for the bold and limitless future ahead.

In this post I am sharing some updates from the workshop and invited talk I gave at SIGGRAPH, together with a fantastic team of academics and entrepreneurs worldwide, as well as highlights from the amazing interactions I had with truly smart people, all from a computational medical XR perspective.

Computational Medical XR: The Next Frontier in Healthcare Innovation based on deep-learning, generative AI and extended reality

The marriage of scientific computing and the enchanting world of extended reality (XR) is revolutionizing the medical field. In this new discipline we go beyond the confines of “Clinical XR” and are diving deep into a universe where neural simulations meet computational geometry and geometric algebra, where computer graphics dance with computational vision, and where theoretical computer science and deep learning team up to address some of the most pressing challenges of life science and neuroscience.

From user-friendly, almost intuitive medical XR platforms to advanced deep learning systems that revolutionize training, education, diagnostics, therapeutics, and rehabilitation — we discussed in the SIGGRAPH Frontiers workshop how to redefine these boundaries: a world where surgical planning is seamlessly integrated with real-time operative navigation within the realm of XR and nurse education and training meets up post-operative patient rehabilitation and treatment. I presented the latest ground-breaking research and development work we do at ORamaVR, as well as examples of our collaboration with expert partners such as Virtamed, Nonnocere, Vired, Sofmedica and Medical School of Cologne, SFITS and Centre of Virtual Medicine at the University Hospital of Geneva, amongst a few.

Computational medical XR Frontiers workshop at SIGGRAPH 2023 with Prof. Mike Cole (University of Michigan Medical School), Dr. Mark Zhang (Brigham and Women’s Hospital, Harvard University), Dr. Walter Greenleaf (Stanford University), Dr. Oliver Kannape (Mindmaze SA and University hospital of Geneva), me and Gabe Jones (co-founder and CEO of Proprio Vision)

In the invited SIGGRAPH Frontiers Talk, together with Dr. Walter Greenleaf from Stanford University, we attempted to shine further light on how the triad of computational medical XR — Spatial, Neural, and Wearable computing — is joining forces to reshape the very essence of healthcare.

Computational medical XR Frontiers Talk: "Spatial, Neural and Wearable computing converging to transform healthcare" together with Dr. Walter Greenleaf (Stanford University)

The Computational Medical XR workshop run for half a day with a fully packed room, split in 3 panels:

Computational XR panel:

1. “State of the art in Computational Medical XR education, experiential learning, and training”, Dr. George Papagiannakis
2. “Multi-modal imaging systems to enhance surgical performance”, Gabe Jones

Neuroscience XR panel:

1. “Convergence of VR, AR, deep learning and biosensing for medical applications”, Dr. Walter Greenleaf
2. “VR-based interventions for cognitive neurorehabilitation”, Dr. Oliver Kannape

Medical XR case studies panel:

1. “Clinical trials and study protocols for XR-based training”, Prof. Michael Cole
2. “Frameworks for rapid creation of subject matter expertise in extended reality for computational medical XR”, Dr. Mark Zhang

Computational Medical XR continuum and wordwide experts from Harvard, Stanford, Michigan, Geneva university hospitals & medical schools as well as ORamaVR, Mindmaze, Proprio companies. More slides are provided here.

The invited talk on Computational Medical XR run for 90 minutes the following day. Some more highlights of the Computational Medical XR invited Frontiers Talk are provided below. Kudos and special thanks to the Frontiers chair Prof. Ginger Alford for the impeccable organisation as well as kind invitation to present at this forum, together with Walter:

Computational Medical XR talk excerpts (talk was live-streamed and recorded for all SIGGRAPH registered participants and more slides are available here )

One of the top highlights of this year’s SIGGRAPH was of course the keynote speech by the Nvidia CEO Jensen Huang:

“Graphics and artificial intelligence are inseparable, graphics needs AI, and AI needs graphics,” Huang said, explaining that AI will learn skills in virtual worlds, and that AI will help create virtual worlds.”

J. Huang, Nvidia, ACM SIGGRAPH 2023"

Jensen Huang's keynote SIGGRAPH 2023 speech was nothing short of spectacular as well as inspiring

Meeting up with some of the smartest people I know in the field, is a major advantage of top conferences like SIGGRAPH, CGI, Eurographics etc. The technical papers session, courses, posters, exhibition as well as all keynote speeches, such as the one of the ACM president Prof. Yannis Ioannidis were truly remarkable moments:

Testing the new Flamera VR HMD by Meta (top left), with Dr. Kanappe from Mindmaze and University hospital of Geneva (top middle), Prof. Ioannidis keynote (top right), with Prof. Chadhuri from IIT, India (bottom left) and the Graphics history exhibition (bottom right)

In the last part of this trip I had the chance to visit our partners and collaborators in the Bay Area: Stanford University, Apple HQ, Plug-and-Play HQ as well as Swissnex.

At the Apple Park HQ, currently in discussions with Apple Health (left, middle) and early version of the ORamaVR support of Apple's Vision Pro Spatial Computer (right)

Emilia Pasquier, the Swissnex CEO trying the ORamVR demo at the Swissnex HQ (left), our advisor Steve Ciesinski together with the CEO of Plug-and-Play Saeed Amidi (right), one of our closest partners in Silicon-Valley

I am grateful for these intense weeks in California, USA this year and all the truly amazing people I had the rare chance to meet.

It is now more clear to me than ever before, that AI and XR are poised to revolutionise both healthcare and education fields in one go!

AI with XR can solve both industries major pain points: (1) access — there aren’t enough good doctors or educators to provide timely services to all who need it (and clinicians/teachers/professors are leaving the field in droves due to burn out), and (2) cost — the cost of high-quality, personalised healthcare and education has skyrocketed, largely due to increasing labor costs, despite the abundance of online material.

However, in both industries, novel science-based tools are needed in order to empower the medical and education professionals to prototype and author themselves the material they desperately need.

Generative AI interfaced through XR and spatial computing, can be the next final frontier for both healthcare and education.

As a final parting thought that inspires our innovative science and science-based generative AI tools we are now building at ORamaVR , so that anyone can create high-fidelity medical XR training simulations (based on our best in the market low-code medical XR training SDK), is this inscription from the Computer History Museum in Mountain-View, next to the first VR HMD by Ivan Sutherland in 1968:

"We become what we behold. We shape our tools, and then our tools shape us."

Marshall McLuhan

First VR Head-mounted-display from 1968 and Ivan Sutherland when he was at Harvard University (left) and an inspiring software tools inscription from the Computer History Musem (right)