The winners of #IBM's $5k prize at #atthack for use of IBM Cloud and Watson Services, AvatarRescue. They utilized a #MagicLeapOne and #Watson Speech to Text, Text to Speech, and Watson Assistant to provide animations for first responders via VR/AR. Very cool solution! pic.twitter.com/RtwiNiiUdj— Derek Teay (@derekteay) April 15, 2019
I attended Virtual Medicine conference at Cedars Sinai in LA and was struck by the reality of virtual reality in healthcare. It seems that 2019 is a landmark year, when VR in healthcare was less about ‘is this useful’ and more about specific science and technology needs.
Hearing Dr. Rafael Grossmann speak and give overview of available look-through mixed reality headsets really drove home the point that this tech is here now so start building today. The chances to radically advance both training and practice of medicine are immense. His emphasis on Haptic technologies also keep pushing us forward to consider multisensory experiences.
Dr. Grossmann is part of #LeapNation and I am honored to get to chat with him on occasion as our Magic Leap developer/enthusiast community explores the potential of Spatial Computing.
Skip Rizzo of USC presented their work with digital agents, aka Virtual Humans, in therapy. Very compelling to see their research that humans will more likely be honest with a purely software agent than with a character with a human behind the face. A purely digital character can assuage their fears of judgement and opens up therapy pipeline to billions of people who would not otherwise have access, or courage to seek access.
Dr Shafi Ahmed presented an inspiring talk about using immersive tech to bridge the massive 5 billion person surgery access gap through training and direct intervention. He even is involved now in constructing a hospital. Dr. Ahmed is thinking massive scale, exponential tech to get education and collaborative surgery into the normal working reality.
This strikes a chord in me, as ever since a trip to Barcelona to see the Sagrada Familia I have become fascinated by a vision of hospitals becoming more architectural, more like churches: custom buildings designed to facilitate healing through their very structure, moving beyond florescent lights and particle board and toward deeply meaningful architecture.
Was compelling to hear from Dr David Rhew, head of Healthcare and Fitness at Samsung. Connecting the dots between visuals, tracking and even physical interventions via exoskeleton is a fascinating nexus that can yield impressive results. I continue to regret that exoskeleton technology was not available for my now deceased grandmother whom retained mental acuity as she was every year more and more confined to wheelchairs and seated postures. Directed walking interventions are a rich frontier, coupled with cognitive XR therapies we could see people living not just longer lives but better lives for longer times.
Heard a great talk from Dr. Brandon Birckhead on the scientific and economic realities of testing all this innovation. Innovation in the basic science side will be needed as well as the more ‘flashy’ applications arena. I am hoping that more companies will step up and help fund this research as it will create markets for their systems and hardware.
There were so many great talks and panels, I only got to see a handful, the full speakers list is available on the website, and hopefully they will make the videos available! Here are a collection of pictures from panels:
Closing out the conference, Dr. Brennan Spiegel showed a very useful spectrum to help situate what kind of experiences we can create and how to utilize them in a holistic manner. Along with a call to respect the real suffering that this new technology can address, and not over-promise. Immersive tech for medicine needs grounded science and respectful approaches, keep the hype in check even as the excitement grows!
Got to experience many different vendors displaying. Of note was just how many pain relief or relaxation VR experience companies there were, mostly using the Oculus Go system and 360º videos for content. This is a hot field and it will be fascinating to see what brands stick around by differentiating themselves through features, support, standardization, etc. Hospitals want to purchase VR for pain relief, patient comfort, and also training and simulation. Speaking with different vendors, particularly simulation vendors, was very enlightening. VR is a massive leap forward for training, as it will enable practitioners to get beyond just having tidbits and facts and toward practical readiness. VR training is a revolution already here, while VR therapeutics is still emerging.
Saw some new hardware too.
Experienced a surgical simulation inside of HP’s latest Windows Mixed Reality headset Reverb (specs here) and can confirm that the visuals are amazing. The quality of the display is finally good enough to let me totally forget that I’m looking at screens and focus instead on the content. This is the ‘retina display’ moment for VR hardware.
What really caught my eye was a 180º projector called Broomx capable of covering an entire room with projected images. This is an exciting development that could drastically reduce the cost of creating immersive CAVE systems.
CAVE systems have an immense benefit in being literal environments, allowing participants to use normal human social cues and gestures ‘for free’ without needing to build in avatars or any such things, and without the synchronization challenges and cost of getting each participant to wear individual spatial computing glasses like hololens or magic leap. Yet they are often constructed of many panels, either back projected or LCD screens. Having just a few Broomx projectors be able to cover a full 360º area drastically reduces cost and complexity, perhaps even brings portability to this powerful tool once reserved for labs, see below for example:
In closing, immersive technologies are here to stay for medicine, in both training and therapy.
A nexus of science, art and human practices is needed to move this possibility space forward into useful elements, networks and standards. There is still research needed and much work to do.
I aim to push into new frontiers with holohealing.com focusing on bringing ancient traditions and deep symbolism into look-through spatial computing headsets like Magic Leap and Hololens.
Exciting revolutionary change is happening right now, 2019 is immersive technology's arrival moment; from the labs and experiments to the real world.
Read more about related research conference:
For me, the most wonderful possibility of mixed reality headsets is the chance to draw in mid-air in 3D with multiple people. This was my great dream for over 8 years. Now it is becoming available soon in Spatiate on Magic Leap One (also on phones all shared together, whoa!) Still in beta, yet already so full featured and solid, such an exciting time!
Since the dawn of language (probably) we have been speaking with our hands to add in flare, expression, illustration and beyond. Now with mixed reality headsets like Magic Leap we have a form factor that allows mid-air drawing to become a natural part of talking.
The first thing we can do, before speaking, is usually finger painting. Now, with mixed reality and spatial computing, we can return to a natural form of expression in a totally new way. Imagine a time before there were words, we are in the time just before another explosion of expressive possibility.
Moreover, with the power of avatars, we can now talk with anyone anywhere at any time while sharing this new mid air drawing layer.
I am powerfully fascinated by this affordance as part of the future of language. Drawing in the space between us can open up a new era, beyond mere high bandwidth information and into true high bandwidth meaning.
The videos below are of an amazing jam session that happened over the internet. It was so freaking cool to reconnect with Steve Lukas as an avatar. Steve started the Spatiate project and was my first demo of Magic Leap (and the source of my ability to alpha test Spatiate). At one point there were 8 people in the same shared drawing space all split between 8 different locations. This quick draw jam session is a historic moment for me, and perhaps for the entire Spatial Computing world: the era of shared 3D mixed reality drawing across space is here, and it is so freaking fun!!
Sketches done between Spatiate participants can be saved as a 3D model, totally changing the nature of ‘notes’ in conversation. Perhaps we can move beyond napkin sketches to go back to the workstation and turn our daily lives and conversations into powerful generative design sessions and art making exercises, with no more effort than it takes to just be as we are.
This medium of ‘projected light’ is what I have been waiting for as an artist for many years. Unlike 3D modeling in a computer, we now have a medium that allows for 3D expression at the scale of being in a direct way. I will explore VR drawing as well, yet something about the mixed reality drawing is truly the most captivating. To draw with the world in this new way is hypnotic and beautiful as an experience, and the results are fascinating even at this early beta stage.
Back in November of 2018 I used an alpha version of Spatiate to make my first of what I hope will be many (much more polished haha) videos using the power of this expressive real time graphic affordance for communication. The old adage of ‘show don’t tell’ can take on a real-time meaning and has the chance to totally upend the traditional workflow of shooting video content, in addition to eventually evolving the nature of language itself.
OMG what a fun opportunity!! I got to have a chat with Savannah Niles, an MIT Media Lab alumni who has been leading Magic Leap’s efforts to create Social and Avatar experiences. We talked inside her own thing!!
She is very brilliant, I got to catch her after her talk at LeapCon, a blog about that time and wider thoughts is linked below. This next chat we got deeper on what she has worked on, how the Magic Leap team approached designing avatars and then went deep on some areas she wanted to hear from me about: environmental AI
Talking with Savannah as an avatar was a beautiful and surreal experience. It really felt like we were sharing space together, because we were! We stood up, shifted weight, moved around and even high fived and ended with a fist bump! She is in Florida and I’m in California and we hung out in shared space together for over 40 minutes and could have kept going.
Social on Magic Leap has already changed everything for me. Video calling is a weird outlier as far as Im concerned: the experience of being avatars is that much better. I don’t want to sit and stare at a screen I want to inhabit the same space as a friend, and that’s what Social on Magic Leap lets us do.
This conversation inspired an upcoming blog about embodied AI and Environmental AI that will feature clips of our conversation to help explain the ideas.
Here is the full conversation all the way through cut up as segments so you can browse. I’d recommend just letting it play through, we were on fire!! ;)
Let it play through, choose videos or scroll below as I’ve also pulled out some segments with context so scroll down if you desire!
We were just talking so the fact that it went into such nice conceptual bins is a testament to how intelligent Savannah is and how engaging the Avatar experience really is on Magic Leap. Just keep in mind that the ‘camera’ for this is tied directly to my head so things might get a little shaky as I nod in agreement or do other natural things ;)
Exploring how the Avatars were designed for Magic Leap Social and finding the “Millenial” art style!
Asking after how shared context will be handled in Magic Leap Social was too on the nose for her to discuss currently, so let’s just wait and see what they are cooking up ;)
We talked about XR as a spacesuit (had never heard that metaphor and now the ML marketing materials make a lot more sense!) and also how mixed reality might be a kind of antidote to the fracturing of reality that can come when everyone has their own virtual playgrounds.
Savannah was curious to hear more about what I had been thinking since our first quick chat and so I downloaded some of my thoughts. Stay till the end of this clip when she walks across the shared space to write down notes, that was a holy grail moment for me in terms of UX because it drove home that we were both spatially present. and inspired the next set of discussion below.
I got to see real Mike Seymour be Virtual Mike Seymour to interview the venerable Ken Perlin as an avatar discussing with a photorealistic avatar MeetMike at Siggraph 2017, very good insights from Perlin on what an Avatar’s job is in communication and how it can best achieve those goals given constraints. I think the Magic Leap team succeeded along his line of thinking with their avatar system.
Yet Magic Leap is trying ‘Digital Human’ fidelity level avatars as well. See Magic Leap’s work on MICA, in addition to the Avatar design and work going on in Social, it is beyond fascinating watching Mike Seymour’s recent talk “What Happens When Technology Has a Human Face?”
As an end little note, this video below was a reflection after a quick chat with Savannah after her design panel talk at LeapCon 2018 where I was working out some ideas and seeing her feedback. How the avatar intelligence will understand how to be it self is an ongoing research area that enables all this, her a short little musing about how to conceptualize that in technical terms:
See you around the MagicVerse everyone.
The augmented reality glasses that were talking like they had it, the real deal, the holy smokes, the bonafide extended enhanced amazing powerful joyous reality altering machinery of the finest computation on your face!
Did they deliver with Magic Leap One?
You can count, right?
We’re on version one of version one version of new reality. So yes indeed does it have bugs and foibles. The FOV is just big enough to start getting a taste for proper synthesis of graphics and view. Yet it’s still a floating rectangle of perception. A big rectangle, yet edged nevertheless.
For the voices getting upset that this tech is not good enough, I’ll just say that it’s good enough for everyone using it. Magic Leap One is an overall rad device with lots of computing power, solid inputs and some excellent touches that make it a really wonderful development platform. So good you could even live with it! Magic Leap’s whole team did a stellar job here, thank you!
We’re undergoing a big shift in a tech capability space that was just 7 years ago a gimmicky and searching approach to make goofy experiences of changing the world! Not the whole world of course, just the world of the computer sensor. We had lots of dreams and visions, but mostly showed each other with powerpoint and hopeful hand gestures. Times were dark yet hopeful.
Fast forward to now. Magic Leap is a Cambrian Explosion moment for the new ways of computing, now called XR. Magic Leap and Hololens are enough to catalyze the new ways of computing among us who research, create and build. And there are other models from ODG and the tech modules are shuffling around searching for form as new conglomerate companies.
It’s been funny watching the term “XR” come into use as it seemed to emerge spontaneously to signify a merging, or crossing, of all concepts related to computer mediation as applied to human perception.
The Magic Leap One is a very slick device and also a very alpha device. The displays have limited field of view, there are some fun differences between the color balance of each eye’s display and even between their focal planes. It washes out totally outside.
None of that changes the fact: Magic Leap One WORKS! It works well. Well enough to where we are seeing great things fast from an extremely excited community.
I have been dreaming of the launch of a generally useful, robust look-through spatial computer headset. I did not and still do not care what ‘brand’ it is, as long as the dang thing lets me do my visions and stories!
So when Magic Leap was announced I started befriending Twitter people who had an ML1 or were getting one. Anyone in the XR space doing fun and meaningful things. And it’s been so fun watching the people, techniques and demos emerge out of the woodwork around the next phase of getting XR from gimmick to holy shit.
At LEAPcon 1 I met amazing people. The Magic Leap talks were inspiring, then AT&T got up on stage and I laughed loud enough that I bet you heard me if you were in that tiny presentation room. If you’ve read my blog at all, or know me, you know that I can often see the very real dark side of any given wonderful thing. Convenience often has a dark side. So I’m watching this space and tech closely, yet also am finding reasons for optimism. Specifically I’m meeting reasons for optimism.
The people give me the hope, especially my newest tribe #LeapNation which emerged on Twitter right after I met Steve Lukas of AcrossXR now Magic Leap who started the group chat. The “Magic Leap” has been very literal for me, suddenly collaborating and discussing with over 50 different talented developers pushing the boundaries and laughing the whole way.
Was great to see a real-life Lumin Runtime app from Sven Mesecke and his team at Rebel Camp in Oregon @dachsghund
Basically every app you can get from other developers on Magic Leap currently is a fully ‘immersive’ experience that does not utilize the UI/UX framework and OS layer that Magic Leap has with Lumin Runtime. Rebel Camp has a real app in that OS environment and it was great to see, even as a proof of concept. This is big and I’m very glad that Rebel Camp is getting the expertise to make Lumin OS apps because these are amazing test-beds to collaborate with Magic Leap on adding intelligence into the operating system while developers get that ‘for free’. If Android can evolve into a XR operating system, I’m all for helping. Hats off to Rebel Camp for diving into the Lumin rabbit hole and emerging with a rabbit.
Sven told me that jhana.space had proceeded me a little bit with him having found it in the forums as I asked the same questions he assured me weren’t crazy: how the hell do I get models with textures into ML1 reliably?!
Developing for Helio browser is very fun, and I’m excited to see what official WebXR support looks like as it comes to the headset in 2019.
I have a special interest in the medical and therapeutic applications of XR, and it was my great pleasure to talk with two surgeons, Rafael Grossman and Alex Johnson at John’s Hopkins.
I sketched out with them a prototype idea that will keep brewing as many forms of knowing around energies, health and ‘trauma’ continue intersecting: contact-less surgery using vibrations and intention tracking channeled through the hands.
They are largely not satisfied with the current surgical robots, and so a vision of direct hands-on manipulation of tissues with a fidelity down to microns without the need for held tools excited them to say the least!
At the Medical talks at LEAPcon I was very impressed by the quality of research and solutions people have been developing.
After the talk I learned from a Leaper that the Magic Leap One is getting heart-beat data and just using it for noise cancelling. She seemed coy as the Cheshire cat as to whether we’d get access to that data any time soon, so make sure to add your voice to the “we need heart beat please!” chorus haha
My interest in medical is currently channeling through as holohealing.com seeking to unite a knowledge and toolbase, with a special interest in human bioenergy visualization and work.
Important note about the reflectivity of Magic Leap One for first responders & therapists:
During training as a California State Beach Lifeguard we were told specifically not to wear reflective sunglasses on the job. If the victim sees their wound in the reflection of your lenses they can go into shock, which just made your job as first responder almost impossibly difficult. Most of lifeguarding first aid is to keep people out of shock. In my years as a guard I experienced the immense health benefit of ‘tactical lying’ to victims about the state of their condition. If someone can’t move their arms and legs, they don’t need you to freak out and tell them that. If someone has a massive gash on their head and they ask “how bad is it?” you can tell them it’s fine and they'll be fine. But that won’t work if they see their blood in your reflective lenses! So for all emerging medical prototypes with ML1 just remember that you have reflective lenses on, so be careful how you insert the ML1 into healthcare situations!
I’m finding humor, curiosity and heart are the keys to making this new medium emerge as something useful. Don’t take it too seriously, yet take it seriously enough to get passionate.
First time I used the Magic Leap was with Steve Lukas (@slukas) showing me his freshly received headset then I received mine on 9/9/18. Every day with it has been a horizon-widening experience. The form factor of ‘holographic headset’ is just singing with my imagination, and then comes all the learning of how to translate ideas to code!
This is a hugely exciting time to get in and shape the medium of ARVRXR whatever, you’re shaping R reality so stay open and find where everything intersects. My research goal is to use Magic Leap and other similar headsets to bring together many different knowledge threads in new ways of visualizing. We are to have a medium for thinking the unthinkable, as Bret Victor (worrydream.com) says.
We seem further along in some areas than I thought we would be and on pace for others.
Overall the thing that draws me in most about Magic Leap and this look through style of Mixed Reality is that I enjoy high quality physical environments and things. I want to expand and merge new aspects into a consensus reality more so than exist in many walled off realities. The best part about reality prime that we share by default is that no one owns the light that bounces off a painting or the sound that leaves my vocal chords as we speak and breathe.
The XR future that I care about is characterized by balance and by a revival of sacred aspects, such as plants being amazing beyond our tech, and real paintings made of atoms being wicked cool. It’s so fun to develop for a new holographic medium while sitting in my partner Kyra V Brandt’s oil painting artwork. A feeling of elevating and grounding of the computer medium in being something humane, and fundamentally artistic over specific. Emergent over prescribed. Gestural and free is how we can be with computers helping our minds and hearts to sing our most authentic forms and dance our most authentic songs.
The Deep Potentials of XR
Everything that follows here are resources for understanding the magnitude of the shift that the human experience can undergo through our passionate utilization of technology for ‘story being’.
I’ve gotten to meet Ken Perlin, who has been an inspiration to my ambition for what the ‘computer’ as a medium really ought to behave like.
The excellent Ramez Naam rushes ahead to a world where XR is in our very minds to explore the remapping of this conservative old world onto radically new possibilities in Water. This is a wake-up call to all of us working on this that behavior and choices are more fundamental then technology; we are responsible for a ‘good’ world.
The merger of VR and AR into a functional whole seems more likely to occur than when I wrote these stories in 2015 of a world where VR as used for school and AR happened in the background. It was easier for me to ground VR in stories back then. Now I feel I can ground XR and even make demos. I am working on new visions where the world itself is the school. You are encouraged to view the world as school regardless of the technology, we just seek to augment the learning potential in any given moment with best alignment.