Avatar Chat with Savannah Niles Leading Social & Avatars at Magic Leap + More!

OMG what a fun opportunity!! I got to have a chat with Savannah Niles, an MIT Media Lab alumni who has been leading Magic Leap’s efforts to create Social and Avatar experiences. We talked inside her own thing!!

She is very brilliant, I got to catch her after her talk at LeapCon, a blog about that time and wider thoughts is linked below. This next chat we got deeper on what she has worked on, how the Magic Leap team approached designing avatars and then went deep on some areas she wanted to hear from me about: environmental AI

Talking with Savannah as an avatar was a beautiful and surreal experience. It really felt like we were sharing space together, because we were! We stood up, shifted weight, moved around and even high fived and ended with a fist bump! She is in Florida and I’m in California and we hung out in shared space together for over 40 minutes and could have kept going.

Social on Magic Leap has already changed everything for me. Video calling is a weird outlier as far as Im concerned: the experience of being avatars is that much better. I don’t want to sit and stare at a screen I want to inhabit the same space as a friend, and that’s what Social on Magic Leap lets us do.

This conversation inspired an upcoming blog about embodied AI and Environmental AI that will feature clips of our conversation to help explain the ideas.

Here is the full conversation all the way through cut up as segments so you can browse. I’d recommend just letting it play through, we were on fire!! ;)

Let it play through, choose videos or scroll below as I’ve also pulled out some segments with context so scroll down if you desire!

We were just talking so the fact that it went into such nice conceptual bins is a testament to how intelligent Savannah is and how engaging the Avatar experience really is on Magic Leap. Just keep in mind that the ‘camera’ for this is tied directly to my head so things might get a little shaky as I nod in agreement or do other natural things ;)

Exploring how the Avatars were designed for Magic Leap Social and finding the “Millenial” art style!

Asking after how shared context will be handled in Magic Leap Social was too on the nose for her to discuss currently, so let’s just wait and see what they are cooking up ;)

We talked about XR as a spacesuit (had never heard that metaphor and now the ML marketing materials make a lot more sense!) and also how mixed reality might be a kind of antidote to the fracturing of reality that can come when everyone has their own virtual playgrounds.

Savannah was curious to hear more about what I had been thinking since our first quick chat and so I downloaded some of my thoughts. Stay till the end of this clip when she walks across the shared space to write down notes, that was a holy grail moment for me in terms of UX because it drove home that we were both spatially present. and inspired the next set of discussion below.

I was very interested to find that Savannah has visited DynamicLand, and we had a fun discussion around using paper in XR.

I have researched stylus based drawing input and am excited to bring more gestural free-hand interpretation into the MagicVerse.

 

I got to see real Mike Seymour be Virtual Mike Seymour to interview the venerable Ken Perlin as an avatar discussing with a photorealistic avatar MeetMike at Siggraph 2017, very good insights from Perlin on what an Avatar’s job is in communication and how it can best achieve those goals given constraints. I think the Magic Leap team succeeded along his line of thinking with their avatar system.

Yet Magic Leap is trying ‘Digital Human’ fidelity level avatars as well. See Magic Leap’s work on MICA, in addition to the Avatar design and work going on in Social, it is beyond fascinating watching Mike Seymour’s recent talk “What Happens When Technology Has a Human Face?”

As an end little note, this video below was a reflection after a quick chat with Savannah after her design panel talk at LeapCon 2018 where I was working out some ideas and seeing her feedback. How the avatar intelligence will understand how to be it self is an ongoing research area that enables all this, her a short little musing about how to conceptualize that in technical terms:

See you around the MagicVerse everyone.

Bridging Rigid Logics with Blurry Imaginations

The tension between ‘what is’ and ‘what can be’ is omnipresent in technological design. The ‘what can be’ side of the tension further striates into what ‘ought be’ and ‘what can afford to be’ in an industrial economic setting. To me, nowhere is the tension between ‘what is’ and ‘what can be’ more apparent than with digital computers. These devices are substrates for logical operations, and as increasingly diverse communities of people have integrated them into their practices we see a flowering of implementations in software. Yet the initial boundary conditions of the history of computing powerfully shape what it is – where computing has been is the ground for us as we stretch to search for what it can be.

“The devices and systems of technology are not natural phenomena but the products of human design, that is, they are the result of matching available means to desired ends at acceptable cost. The available means ultimately do rest on natural laws, which define the possibilities and limits of the technology. But desired ends and acceptable costs are matters of society.” (Mahoney, 122)

So far ‘desired ends’ of the computational society have been seeded with industrial concerns and perspectives.

“the computer industry was, more than anything else, a continuation of the pre-1945 office equipment industry and in particular of the punched card machine industry.” (Mahoney, 126; quoting Haigh) “But making it universal, or general purpose, also made it indeterminate. Capable of calculating any logical function, it could become anything but was in itself nothing (well, as designed, it could always do arithmetic).” (Mahoney, 123)

Thus, in the 1970s, humanist artists began wading into computation, and we have witnessed an explosion of ‘high level’ creativity as to what the metamedium of ‘computation’ can actually do for us as meaning-makers. Ideas flourished that saw the computer as not just a machine for counting, but a substrate for human imagination. Yet the histories of computing set the devices we compute with on a path that has shaped its form: a device with baked-in logics that we recombine. The histories of computing feature engineering, science and data analysis as the kernel of the computer’s unfolding into the wider sociotechnical ecosystem. Art was tacked on later as an affordance of having enough 1/0s to spare. Computer programs are precise manipulations of the state of an electro-atomic system we call a computer. Yet human language too manipulates other electro-atomic systems (aka, other humans) in a much more blurry and imprecise way - yet this blurriness leaves room for emergence, and this I think is the key to the future direction of computing itself.

I am struck more and more each day by the 20th century origins of computing, and harden my resolve to lean more and more into what the 21st century of computing looks like. The future will see the “front” and “back” of computation merge into a holistic loop where generative logics allow computers to learn as they are used. The loops in our minds will be further augmented by loops through machines that begin to not just manipulate saved libraries, but increasingly generate new forms. We are, I think, at a profound crossroads in the path: will computing be continually defined by linear “processing”, or can we move it toward continuous relational inference? I think we must move to the latter, for the affordances of the future will enable and demand new human-scale ways to program computers. We are in the midst of a latent programming revolution.

This thinking has been culminating for me with the input of this class and my continued experience with the Microsoft Surface. The Surface device that I am typing this on is perhaps the perfect symbol for the crossroads that personal computing is currently in. The Surface has two distinct interface modes: the touchscreen/pen digitizer, and the keyboard. The mouse is unified with the digitizer pen decently well, but the keyboard remains a realm unto itself.

I am finding it increasingly jarring to coexist in free-flowing writing inside of digital inking applications and interfacing with programming.

To this day when writing to a computer at the level of its logical comprehension we are forced to bring our hands together and cramp over an un-changing keyboard. We input 1/0 commands into the machine through keys that correspond to symbols, which in sequence will (when interpreted) illicit the electrical state of the computer to evolve step by step as fast as the system clock allows.

The more I use a pen on a grid, the more I believe that there is potentially another way to program.

The work of von Neumann and others who pioneered the study of cellular automata has shown me that computing does not have to be about direct control using predefined symbol sets, but rather can be about boundary conditions and evolution.

I wonder if we cannot use digitizer grids and pens to allow human operators to sketch with computers. Already much of the power of the computer comes to us via adding abstraction. To edit a photo with machine code directly would be impossibly tedious, but thanks to many layers of abstraction I can use a tool like photoshop to move around thousands of pixels and billions of transistors in large strokes.

Programming languages have been path dependent upon 20th century paradigms. To me, programming a digital computer feels like playing with a near-infinite movable type: there are libraries of modules that I arrange in patterns to produce sequences which instruct the machine and can even mean something to a person.

Yet I wonder, is that the only way to program computers? Must we only use rigid pre-delineated symbols?

I think we can begin to write higher level programming environments that allow us to write to our computers, not type, but actually write.

I discovered a groundbreaking paper recently which shows that a unification between the way humans reason and the way computers process might be increasingly possible and fruitful.

Researchers Lake, Salakutdinov and Tenenbaum instantiated a “machine learning” concept by creating a “Bayesian program learning (BPL) framework, capable of learning a large class of visual concepts from just a single example and generalizing in ways that are mostly indistinguishable from people.” Using digital inking they developed a technique to parse drawn symbols via vector and temporal relational information and allow the computer to generate further symbols from these inputs.

“Concepts are represented as simple probabilistic programs—that is, probabilistic generative models expressed as structured procedures in an abstract description language.” Their framework brings together compositionality, causality and learning to learn. “As programs, rich concepts can be built ‘compositionally’ from simpler primitives. Their probabilistic semantics handle noise and support creative generalizations in a procedural form that (unlike other probabilistic models) naturally captures the abstract “causal” structure of the real-world processes that produce examples of a category.”

“Learning proceeds by constructing programs that best explain the observations under a Bayesian criterion, and the model “learns to learn” (23, 24) by developing hierarchical priors that allow previous experience with related concepts to ease learning of new concepts (25, 26). These priors represent a learned inductive bias (27) that abstracts the key regularities and dimensions of variation holding across both types of concepts and across instances (or tokens) of a concept in a given domain.”

“In short, BPL can construct new programs by reusing the pieces of existing ones, capturing the causal and compositional properties of real-world generative processes operating on multiple scales.”

Finding this paper feels profound to me. Lake et al have been able to create a learning system that does not need huge amounts of data, but rather using smaller stochastic programs to represent concepts and building them compositionally from parts, subparts and spatial/temporal relations.

BPL is a generative model for generative models.

The BPL approach gets us away from the traditional histories of computing with their emphasis on large datasets and toward smaller evolutionary rules-based generative computing.

Using the BPL method, concepts are represented as probalistic relational programs, so anything entered by the human operator (or theoretically by other BPL-taught machines) becomes instantly absorbed into a formal logic and is combinatorial at a mathematically grounded and sound level.

The key of BPL is that, like human beings, it allows the computer to start working on relational categorization after just one example. This is how “machine learning” can go from tool of the corporation toward tool of the individual. We individuals do not have thousands or millions of datapoints to give to our personal computers, but we do have individual ideas that we can sketch to them.

I truly think that computer science is going through a revolution in understanding: no longer will computing be about “business machines” and cracking cyphercodes and massive datasets, but instead will increasingly feature generative creative inference and blurry conversation.

The BPL approach, if embedded into the OS of modern personal computing could enable humans to converse with designed emergent libraries of recombinatorial mathematical artifacts. BPL is much more “as we may think” than any of the ‘neural net’ approaches that require astronomically large datasets and vast number crunching. Programming can evolve from reading “tapes” with rigid logics into sketching blurry ideas and creating relational inferences. This is not a replacement, but rather a welcome addition. The BPL approach is still “grounded” in piles of 1/0, but the way that BPL structures the 1/0s is much more modular and inherently combinatorial than previous approaches (from my limited perspective at least).

I think this approach is a keystone I have been seeking to merge ‘symbols that mean’ with ‘symbols that do’ into a unified mathematically complete “metasymbology” that will allow us to merge programming with language. Going further, the authors (and I) see no limits to using a BPL style approach to allow computers to engage with all forms of human symbolism, from language to gestures to dance moves. Even engineered devices and natural complexity, all the way to abstract knowledge such as natural number, natural language semantics and intuitive physical theories. (Lake et al, 1337)

In their history computers have been substrates for enacting human logic, moving forward computers will also become ever better substrates for enacting human dreams.

--

Sources:

Michael S. Mahoney, "The Histories of Computing(s)." Interdisciplinary Science Reviews 30, no. 2 (June 2005).

Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum “Human-level concept learning through probabilistic program induction” https://www.cs.cmu.edu/~rsalakhu/papers/LakeEtAl2015Science.pdf

 

originally for Georgetown CCT class CCTP-820: Leading by Design – Principles of Technical and Social Systems 

"The Dynamics of Ecological Crisis"

 

I came across a brilliantly ambitious model that spoke to me. Gregory Bateson's attempt spoke to that urge in so many of us humans, "if only we could model catastrophe we could avoid it!"

 
 
 
 

I love "hubris" as a module in this system. Classic political science! I wanted to accentuate the strength of "hubris", since in my studies hubris seems like an ultimate driver of humanity and deserves more attention when invoked. So I recreated the diagram in vector form to explore: 

 

Now it occurs to me that the model should be alive and that the central modules should have tunable parameters. So for example, hydrocarbon-dominant technology increases pollution, but fusion-dominant technology reduces pollution. Then the size of the outer modules either grows or shrinks. If a module becomes too big it will eat the model and the sim is over. Getting rid of them can become a goal of a gamified form of this model. The inner modules touching or overlapping could add further complexity over time to better express the nature of the listed dynamics.

I'm still thinking of all the parameters, but I think this could be a fascinating model to animate as a simulation of itself! I'm hoping I could do it in p5.js or D3.js.

This will be a fun experiment!