The winners of #IBM's $5k prize at #atthack for use of IBM Cloud and Watson Services, AvatarRescue. They utilized a #MagicLeapOne and #Watson Speech to Text, Text to Speech, and Watson Assistant to provide animations for first responders via VR/AR. Very cool solution! pic.twitter.com/RtwiNiiUdj— Derek Teay (@derekteay) April 15, 2019
Had a great time hanging out with my team, Magic Leap developer relations squad and all the other rad participants of ATTHack 2019.
I attended Virtual Medicine conference at Cedars Sinai in LA and was struck by the reality of virtual reality in healthcare. It seems that 2019 is a landmark year, when VR in healthcare was less about ‘is this useful’ and more about specific science and technology needs.
Hearing Dr. Rafael Grossmann speak and give overview of available look-through mixed reality headsets really drove home the point that this tech is here now so start building today. The chances to radically advance both training and practice of medicine are immense. His emphasis on Haptic technologies also keep pushing us forward to consider multisensory experiences.
Dr. Grossmann is part of #LeapNation and I am honored to get to chat with him on occasion as our Magic Leap developer/enthusiast community explores the potential of Spatial Computing.
Skip Rizzo of USC presented their work with digital agents, aka Virtual Humans, in therapy. Very compelling to see their research that humans will more likely be honest with a purely software agent than with a character with a human behind the face. A purely digital character can assuage their fears of judgement and opens up therapy pipeline to billions of people who would not otherwise have access, or courage to seek access.
Dr Shafi Ahmed presented an inspiring talk about using immersive tech to bridge the massive 5 billion person surgery access gap through training and direct intervention. He even is involved now in constructing a hospital. Dr. Ahmed is thinking massive scale, exponential tech to get education and collaborative surgery into the normal working reality.
This strikes a chord in me, as ever since a trip to Barcelona to see the Sagrada Familia I have become fascinated by a vision of hospitals becoming more architectural, more like churches: custom buildings designed to facilitate healing through their very structure, moving beyond florescent lights and particle board and toward deeply meaningful architecture.
Was compelling to hear from Dr David Rhew, head of Healthcare and Fitness at Samsung. Connecting the dots between visuals, tracking and even physical interventions via exoskeleton is a fascinating nexus that can yield impressive results. I continue to regret that exoskeleton technology was not available for my now deceased grandmother whom retained mental acuity as she was every year more and more confined to wheelchairs and seated postures. Directed walking interventions are a rich frontier, coupled with cognitive XR therapies we could see people living not just longer lives but better lives for longer times.
Heard a great talk from Dr. Brandon Birckhead on the scientific and economic realities of testing all this innovation. Innovation in the basic science side will be needed as well as the more ‘flashy’ applications arena. I am hoping that more companies will step up and help fund this research as it will create markets for their systems and hardware.
There were so many great talks and panels, I only got to see a handful, the full speakers list is available on the website, and hopefully they will make the videos available! Here are a collection of pictures from panels:
Closing out the conference, Dr. Brennan Spiegel showed a very useful spectrum to help situate what kind of experiences we can create and how to utilize them in a holistic manner. Along with a call to respect the real suffering that this new technology can address, and not over-promise. Immersive tech for medicine needs grounded science and respectful approaches, keep the hype in check even as the excitement grows!
Got to experience many different vendors displaying. Of note was just how many pain relief or relaxation VR experience companies there were, mostly using the Oculus Go system and 360º videos for content. This is a hot field and it will be fascinating to see what brands stick around by differentiating themselves through features, support, standardization, etc. Hospitals want to purchase VR for pain relief, patient comfort, and also training and simulation. Speaking with different vendors, particularly simulation vendors, was very enlightening. VR is a massive leap forward for training, as it will enable practitioners to get beyond just having tidbits and facts and toward practical readiness. VR training is a revolution already here, while VR therapeutics is still emerging.
Saw some new hardware too.
Experienced a surgical simulation inside of HP’s latest Windows Mixed Reality headset Reverb (specs here) and can confirm that the visuals are amazing. The quality of the display is finally good enough to let me totally forget that I’m looking at screens and focus instead on the content. This is the ‘retina display’ moment for VR hardware.
What really caught my eye was a 180º projector called Broomx capable of covering an entire room with projected images. This is an exciting development that could drastically reduce the cost of creating immersive CAVE systems.
CAVE systems have an immense benefit in being literal environments, allowing participants to use normal human social cues and gestures ‘for free’ without needing to build in avatars or any such things, and without the synchronization challenges and cost of getting each participant to wear individual spatial computing glasses like hololens or magic leap. Yet they are often constructed of many panels, either back projected or LCD screens. Having just a few Broomx projectors be able to cover a full 360º area drastically reduces cost and complexity, perhaps even brings portability to this powerful tool once reserved for labs, see below for example:
In closing, immersive technologies are here to stay for medicine, in both training and therapy.
A nexus of science, art and human practices is needed to move this possibility space forward into useful elements, networks and standards. There is still research needed and much work to do.
I aim to push into new frontiers with holohealing.com focusing on bringing ancient traditions and deep symbolism into look-through spatial computing headsets like Magic Leap and Hololens.
Exciting revolutionary change is happening right now, 2019 is immersive technology's arrival moment; from the labs and experiments to the real world.
Read more about related research conference:
For me, the most wonderful possibility of mixed reality headsets is the chance to draw in mid-air in 3D with multiple people. This was my great dream for over 8 years. Now it is becoming available soon in Spatiate on Magic Leap One (also on phones all shared together, whoa!) Still in beta, yet already so full featured and solid, such an exciting time!
Since the dawn of language (probably) we have been speaking with our hands to add in flare, expression, illustration and beyond. Now with mixed reality headsets like Magic Leap we have a form factor that allows mid-air drawing to become a natural part of talking.
The first thing we can do, before speaking, is usually finger painting. Now, with mixed reality and spatial computing, we can return to a natural form of expression in a totally new way. Imagine a time before there were words, we are in the time just before another explosion of expressive possibility.
Moreover, with the power of avatars, we can now talk with anyone anywhere at any time while sharing this new mid air drawing layer.
I am powerfully fascinated by this affordance as part of the future of language. Drawing in the space between us can open up a new era, beyond mere high bandwidth information and into true high bandwidth meaning.
The videos below are of an amazing jam session that happened over the internet. It was so freaking cool to reconnect with Steve Lukas as an avatar. Steve started the Spatiate project and was my first demo of Magic Leap (and the source of my ability to alpha test Spatiate). At one point there were 8 people in the same shared drawing space all split between 8 different locations. This quick draw jam session is a historic moment for me, and perhaps for the entire Spatial Computing world: the era of shared 3D mixed reality drawing across space is here, and it is so freaking fun!!
Sketches done between Spatiate participants can be saved as a 3D model, totally changing the nature of ‘notes’ in conversation. Perhaps we can move beyond napkin sketches to go back to the workstation and turn our daily lives and conversations into powerful generative design sessions and art making exercises, with no more effort than it takes to just be as we are.
This medium of ‘projected light’ is what I have been waiting for as an artist for many years. Unlike 3D modeling in a computer, we now have a medium that allows for 3D expression at the scale of being in a direct way. I will explore VR drawing as well, yet something about the mixed reality drawing is truly the most captivating. To draw with the world in this new way is hypnotic and beautiful as an experience, and the results are fascinating even at this early beta stage.
Back in November of 2018 I used an alpha version of Spatiate to make my first of what I hope will be many (much more polished haha) videos using the power of this expressive real time graphic affordance for communication. The old adage of ‘show don’t tell’ can take on a real-time meaning and has the chance to totally upend the traditional workflow of shooting video content, in addition to eventually evolving the nature of language itself.
OMG what a fun opportunity!! I got to have a chat with Savannah Niles, an MIT Media Lab alumni who has been leading Magic Leap’s efforts to create Social and Avatar experiences. We talked inside her own thing!!
She is very brilliant, I got to catch her after her talk at LeapCon, a blog about that time and wider thoughts is linked below. This next chat we got deeper on what she has worked on, how the Magic Leap team approached designing avatars and then went deep on some areas she wanted to hear from me about: environmental AI
Talking with Savannah as an avatar was a beautiful and surreal experience. It really felt like we were sharing space together, because we were! We stood up, shifted weight, moved around and even high fived and ended with a fist bump! She is in Florida and I’m in California and we hung out in shared space together for over 40 minutes and could have kept going.
Social on Magic Leap has already changed everything for me. Video calling is a weird outlier as far as Im concerned: the experience of being avatars is that much better. I don’t want to sit and stare at a screen I want to inhabit the same space as a friend, and that’s what Social on Magic Leap lets us do.
This conversation inspired an upcoming blog about embodied AI and Environmental AI that will feature clips of our conversation to help explain the ideas.
Here is the full conversation all the way through cut up as segments so you can browse. I’d recommend just letting it play through, we were on fire!! ;)
Let it play through, choose videos or scroll below as I’ve also pulled out some segments with context so scroll down if you desire!
We were just talking so the fact that it went into such nice conceptual bins is a testament to how intelligent Savannah is and how engaging the Avatar experience really is on Magic Leap. Just keep in mind that the ‘camera’ for this is tied directly to my head so things might get a little shaky as I nod in agreement or do other natural things ;)
Exploring how the Avatars were designed for Magic Leap Social and finding the “Millenial” art style!
Asking after how shared context will be handled in Magic Leap Social was too on the nose for her to discuss currently, so let’s just wait and see what they are cooking up ;)
We talked about XR as a spacesuit (had never heard that metaphor and now the ML marketing materials make a lot more sense!) and also how mixed reality might be a kind of antidote to the fracturing of reality that can come when everyone has their own virtual playgrounds.
Savannah was curious to hear more about what I had been thinking since our first quick chat and so I downloaded some of my thoughts. Stay till the end of this clip when she walks across the shared space to write down notes, that was a holy grail moment for me in terms of UX because it drove home that we were both spatially present. and inspired the next set of discussion below.
I got to see real Mike Seymour be Virtual Mike Seymour to interview the venerable Ken Perlin as an avatar discussing with a photorealistic avatar MeetMike at Siggraph 2017, very good insights from Perlin on what an Avatar’s job is in communication and how it can best achieve those goals given constraints. I think the Magic Leap team succeeded along his line of thinking with their avatar system.
Yet Magic Leap is trying ‘Digital Human’ fidelity level avatars as well. See Magic Leap’s work on MICA, in addition to the Avatar design and work going on in Social, it is beyond fascinating watching Mike Seymour’s recent talk “What Happens When Technology Has a Human Face?”
As an end little note, this video below was a reflection after a quick chat with Savannah after her design panel talk at LeapCon 2018 where I was working out some ideas and seeing her feedback. How the avatar intelligence will understand how to be it self is an ongoing research area that enables all this, her a short little musing about how to conceptualize that in technical terms:
See you around the MagicVerse everyone.