3D Shared Drawing in Magic Leap & Across devices with Spatiate!

For me, the most wonderful possibility of mixed reality headsets is the chance to draw in mid-air in 3D with multiple people. This was my great dream for over 8 years. Now it is becoming available soon in Spatiate on Magic Leap One (also on phones all shared together, whoa!) Still in beta, yet already so full featured and solid, such an exciting time!

Since the dawn of language (probably) we have been speaking with our hands to add in flare, expression, illustration and beyond. Now with mixed reality headsets like Magic Leap we have a form factor that allows mid-air drawing to become a natural part of talking.

The first thing we can do, before speaking, is usually finger painting. Now, with mixed reality and spatial computing, we can return to a natural form of expression in a totally new way. Imagine a time before there were words, we are in the time just before another explosion of expressive possibility.

Moreover, with the power of avatars, we can now talk with anyone anywhere at any time while sharing this new mid air drawing layer.

I am powerfully fascinated by this affordance as part of the future of language. Drawing in the space between us can open up a new era, beyond mere high bandwidth information and into true high bandwidth meaning.

The videos below are of an amazing jam session that happened over the internet. It was so freaking cool to reconnect with Steve Lukas as an avatar. Steve started the Spatiate project and was my first demo of Magic Leap (and the source of my ability to alpha test Spatiate). At one point there were 8 people in the same shared drawing space all split between 8 different locations. This quick draw jam session is a historic moment for me, and perhaps for the entire Spatial Computing world: the era of shared 3D mixed reality drawing across space is here, and it is so freaking fun!!

Sketches done between Spatiate participants can be saved as a 3D model, totally changing the nature of ‘notes’ in conversation. Perhaps we can move beyond napkin sketches to go back to the workstation and turn our daily lives and conversations into powerful generative design sessions and art making exercises, with no more effort than it takes to just be as we are.

This medium of ‘projected light’ is what I have been waiting for as an artist for many years. Unlike 3D modeling in a computer, we now have a medium that allows for 3D expression at the scale of being in a direct way. I will explore VR drawing as well, yet something about the mixed reality drawing is truly the most captivating. To draw with the world in this new way is hypnotic and beautiful as an experience, and the results are fascinating even at this early beta stage.

Back in November of 2018 I used an alpha version of Spatiate to make my first of what I hope will be many (much more polished haha) videos using the power of this expressive real time graphic affordance for communication. The old adage of ‘show don’t tell’ can take on a real-time meaning and has the chance to totally upend the traditional workflow of shooting video content, in addition to eventually evolving the nature of language itself.

The Exponentials Are Here

Here they are, The Exponentials, the generation once weakly called 'Z', the ones behind the millennials. For the Exponentials technology is not interesting in and of itself. They grew up in an Aether of internet with phones faster than supercomputers of old. Starting in the 90s we began to separate. Each year of being alive in the Exponential generation looks different from the last in enough profound socio-technical ways that lumping them is impossible. When my 10 year old next door neighbor came over with a robot that balances on its own that you drive with a phone and asked me (27) and my brother (23) if we ‘had this when we were kids’ we could only laugh and explain that when we were kids computers weighed 15 pounds and gyroscopes cost thousands of dollars. He didn’t really understand.

Exponential technological change has created a segmentation of experience between generations that takes years to change, not decades; and it’s getting faster.

Let’s quickly compare a 6 year old’s general technological landscape since 1990:

A child born in 1990 (me) used a computer plugged into the wall that was a heavy box when they were 6 years old and a very text-heavy internet at 56k dialup. A child born in 1995 could have had broadband internet allowing them to upload and download rich media. A child born in 2000 could have had wireless internet and a laptop to work with and a few more videos with some social media blossoming up. A child born in 2005 had wireless internet by default with multiple communication and networking channels all in rich media. A child born in 2010 can potentially access virtual reality and definitely has omnipresent smartphone access with infinite communication channels, 3D printing access, and even access to entirely new kinds of money. A child born in 2015 will have mixed reality headsets and holographics, intelligent inferential computing, potentially even 4D reactive printing materials, quantum computing access, ‘action voxels’ such as synthetic biology and swarm robotics, their real-world becoming programmatic.

This is why the only generational definition that matters is Exponential. Each ‘childhood’ experience is now defined by vast leaps in technological capability. Each specific experience looks different in what tools and toys are around. Yet each childhood experience is unified by a dark thread: the unaddressed trauma and cost of the weight of human action on the world.

The Exponentials are here, they swim in the Internet and they think with each other in rich hypermedia as their default way of being. There is no lost dream for them, no economy that should be working better, only the tools to build a better one. They care about the real, deep issues and they have little luxury for distractions on rat races and exploitation. They can’t be told to stop because they have too many resources for moving forward. They don’t just appear on talk shows, they make them daily for fun. For them technology is used in context, yet not marveled at as its own achievement. Only the actions matter. There is no other option than deep social reform.

The teens of Parkland are our signal. The Exponentials have arrived. These brave humans are riding the ripples of convergent technological ability to fix real and very old issues. We waited for super heros when they are already among us. The Exponentials are already working, picking up on the threads left behind by the previous generations. The Exponentials have no distinctions between what is allowed and what should be done. They care not for titles and honors, only for the safety of their friends and loved ones. You cannot convince them that the world is ‘ok’ and they should just wait it out because they know better, in their hearts and minds. They cannot and will not stop until the world has been cleaned up. They are articulate, able to query global expertise at the tap of some fingers, and soon with merely looking and thinking into EEG readers on their headsets.

Each year the technological tools at their disposal will only get exponentially more powerful. Their networks will only get exponentially more interconnected. The movement is here. The Exponentials have arrived.

It is all other generations’ duty to help them and invite them into the meetings. They’ll find their way eventually, so let’s network with them now. Millennials, drop your cynicism, let go of your shields. Forget the faded vision of capitalism past. Join your brothers and sisters of the Exponentials to create a golden chain into the new future. One where we do not hide from trauma, nor wallow in it, rather network with each other to transform trauma into action. A world where we know only two things: problems are inevitable, and problems are solvable, together.

The real exponential change of our time is in the people ourselves: we have the means to communicate with light and radio. There is no more need to stay in the shadows. Rise up and find the others. Rise up and we are uplifting the world.

The jewel of our age, the ones who swim in cyberspace as fish, unaware of the novelty of their medium, these are the ones who will connect the dots. The Exponentials are here. Let’s get to work brothers and sisters, the next phase has begun.

Deep dream of a CNN placeholder
TeensDreamWebblue.png

If you are an Exponential, check out Teens Dream video contest & social platform as an avenue to get your goal out there and connect with other change-makers.

I helped start this contest back in 2014 under the Global Co Lab which I'm on the board of. Connect with me for more info.

SIGGRAPH 2017 & The Birthing Of Interactive External Imagination

uf1.jpg

I attended SIGGRAPH 2017 in LA and was struck by how mature the technologies are. It felt like a capstone moment, when so much of what I had dreamt of over the recent years of my life started to look tantalizingly possible. This post is a collection links from myself and others, please click around and explore the future As We May Think - There has long existed a dream of what an augmenting medium could be about, and a profound new chapter feels close at hand if we choose to build it...

 

At SIGGRAPH I was delighted to see Ken Perlin in the interesting demo #MeetMike. I recorded his talk on my phone, so it's not the best quality, but ironically the main thing I want to share is not the visuals, although they are incredibly impressive, but rather Perlin's message: we need to work with humanity as we build social computer experiences. Perlin would rather see a cartoon with great expressivity than a photo-realistic avatar with rigid muscles. Perlin is a pioneering researcher and thinker currently working on HoloJam and Chalktalk.

That said, the #MeetMike demo was extremely impressive in showing a glimmer of what a photo-realistic avatar could be like. The possibility has been lurking, but Mike Seymour and his team brought it to life and gave it an application, and they did an amazing job!

In my touring around the booths I encountered Imverse in the back of the room with a cool first take on making the creation of Mixed Reality really easy. Talking with Javier the CEO was really fun, his tool already has some of the elements of Perlin's HoloJam, but could use a lot more interactivity in creating real-time advanced animations, as Javier agreed and is working on. Unlike HoloJom though, Imverse is a commercially-ready application that works on more standard VR gear, without the need of a fully tracked room, making it ripe for experimentation by us mere mortals without lab-space!

Another highlight: I was really blown away to wear ODG's Mixed Reality glasses because they were so light! They run Android and will cost a bit over $1,000. Their form factor made me think, "yup, this one will actually get used in the field like it should." They're a self-contained computer, which limits graphics but mobility is my favorite element of what makes Mixed and Augmented Reality so special - computing out in the real world, with our hands free! 

With Apple's AR kit and small glasses like this we are getting ready for Mixed (or Augmented) reality sooner than I thought! 

But before any of this we also need to think hard about computation and what we're trying to do - what is the Center Of "Why"?

And I'm intent to explore where we've already been to make sure we're always breaking new ground and/or refining the garden we're already in. A recent ACM panel gives a great crash history through the present with the ethics.

Most of the improvements we need are in conceptual understanding of computation - SIGGRAPH showed me that the raw technology is ready for anything! So it's up to us to imagine a more robust version of just what it means to "use a computer". We need smarter interfaces and a more expressive form of programming, and we can begin experimenting today even before all the gadgets are ready.

If I talked with you at SIGGRAPH I probably told you I'm tired of clicking little buttons and using inert spaces which I must instruct EVERY element of the action I wish to perform. I mean it. I want to paint with math and math with paint. I believe that computing is a shared substrate for all of human endeavor, and I know it needs to let us be fully human within it. 

In graduate school I started exploring the research that is currently present for making a much more expressive interface for computing based on the the mighty pen, hypercharged by the affordances of a digital substrate: As We May Sketch

I am fascinated by the dream that we might eventually Converse with Computers in new forms of conversation, even if it is just to make graphics in a more user-friendly way.

There are already methods for approaching "artificial" intelligence in computers that will let us switch from thinking of machine learning as a data problem to having machine learning become an expressive experience allowing us to bridge rigid logics with blurry imaginations.

I am struck to my core that computing can and ought to move beyond us as explicit instructors and toward us as cooperative participants with an increasingly dynamic and intelligent substrate ready to hold our creations and let our minds soar. We in our physical bodies and full humanity are at the center of the vision of the future.

Still beyond our wonderful maturing pen-based tablets with 2D motion screens, we are entering into a time when graphics can move outside of rectangles - a time to explore more Humane Representations of Thought. We do need to keep humanity in mind, both our soaring potentials and our consistent ambitions and nagging flaws, books like Rainbow's End help situate the affordances against stories.

Back in 2015 I wrote a collection of three scenes from a School Astride the Metaverse, an attempt to envision a school that bridges material and virtual reality into a kind of whole:

This vision is not complete, it is playground to think about possible styles of school in the future. The first two are grounded, and the last one is more fanciful. There's an entire arena to go into with mixed reality and even more simulation space games - I'm working on it, but for now want to re-surface these since they are starting to look almost conservative in light of this years' tech I witnessed!

This is not meant as a description of the future, but rather a constellation of ideas about how the future of education might look in a world where technology keeps getting better and cheaper, yet overall amounts of money spent on education remain relatively constant. This is not a utopia, this is meant to explore the feeling of what certain compromises and perspectives might manifest as. You certainly do not have to agree with or love or even like this vision, but I do hope that you want to discuss it and the ideas within since this is a world not too far away. Mere decades really for some parts of it; other parts are less clear...

It's important to think and converse about holistic visions for the future of education, not just individual technology artifacts in the classroom and moderate systems deployments. This is my attempt to help add a little flavor to the conversation.

Let's make a future where learning is an adventure and allow new generations to soar past our wildest dreams.

Bridging Rigid Logics with Blurry Imaginations

The tension between ‘what is’ and ‘what can be’ is omnipresent in technological design. The ‘what can be’ side of the tension further striates into what ‘ought be’ and ‘what can afford to be’ in an industrial economic setting. To me, nowhere is the tension between ‘what is’ and ‘what can be’ more apparent than with digital computers. These devices are substrates for logical operations, and as increasingly diverse communities of people have integrated them into their practices we see a flowering of implementations in software. Yet the initial boundary conditions of the history of computing powerfully shape what it is – where computing has been is the ground for us as we stretch to search for what it can be.

“The devices and systems of technology are not natural phenomena but the products of human design, that is, they are the result of matching available means to desired ends at acceptable cost. The available means ultimately do rest on natural laws, which define the possibilities and limits of the technology. But desired ends and acceptable costs are matters of society.” (Mahoney, 122)

So far ‘desired ends’ of the computational society have been seeded with industrial concerns and perspectives.

“the computer industry was, more than anything else, a continuation of the pre-1945 office equipment industry and in particular of the punched card machine industry.” (Mahoney, 126; quoting Haigh) “But making it universal, or general purpose, also made it indeterminate. Capable of calculating any logical function, it could become anything but was in itself nothing (well, as designed, it could always do arithmetic).” (Mahoney, 123)

Thus, in the 1970s, humanist artists began wading into computation, and we have witnessed an explosion of ‘high level’ creativity as to what the metamedium of ‘computation’ can actually do for us as meaning-makers. Ideas flourished that saw the computer as not just a machine for counting, but a substrate for human imagination. Yet the histories of computing set the devices we compute with on a path that has shaped its form: a device with baked-in logics that we recombine. The histories of computing feature engineering, science and data analysis as the kernel of the computer’s unfolding into the wider sociotechnical ecosystem. Art was tacked on later as an affordance of having enough 1/0s to spare. Computer programs are precise manipulations of the state of an electro-atomic system we call a computer. Yet human language too manipulates other electro-atomic systems (aka, other humans) in a much more blurry and imprecise way - yet this blurriness leaves room for emergence, and this I think is the key to the future direction of computing itself.

I am struck more and more each day by the 20th century origins of computing, and harden my resolve to lean more and more into what the 21st century of computing looks like. The future will see the “front” and “back” of computation merge into a holistic loop where generative logics allow computers to learn as they are used. The loops in our minds will be further augmented by loops through machines that begin to not just manipulate saved libraries, but increasingly generate new forms. We are, I think, at a profound crossroads in the path: will computing be continually defined by linear “processing”, or can we move it toward continuous relational inference? I think we must move to the latter, for the affordances of the future will enable and demand new human-scale ways to program computers. We are in the midst of a latent programming revolution.

This thinking has been culminating for me with the input of this class and my continued experience with the Microsoft Surface. The Surface device that I am typing this on is perhaps the perfect symbol for the crossroads that personal computing is currently in. The Surface has two distinct interface modes: the touchscreen/pen digitizer, and the keyboard. The mouse is unified with the digitizer pen decently well, but the keyboard remains a realm unto itself.

I am finding it increasingly jarring to coexist in free-flowing writing inside of digital inking applications and interfacing with programming.

To this day when writing to a computer at the level of its logical comprehension we are forced to bring our hands together and cramp over an un-changing keyboard. We input 1/0 commands into the machine through keys that correspond to symbols, which in sequence will (when interpreted) illicit the electrical state of the computer to evolve step by step as fast as the system clock allows.

The more I use a pen on a grid, the more I believe that there is potentially another way to program.

The work of von Neumann and others who pioneered the study of cellular automata has shown me that computing does not have to be about direct control using predefined symbol sets, but rather can be about boundary conditions and evolution.

I wonder if we cannot use digitizer grids and pens to allow human operators to sketch with computers. Already much of the power of the computer comes to us via adding abstraction. To edit a photo with machine code directly would be impossibly tedious, but thanks to many layers of abstraction I can use a tool like photoshop to move around thousands of pixels and billions of transistors in large strokes.

Programming languages have been path dependent upon 20th century paradigms. To me, programming a digital computer feels like playing with a near-infinite movable type: there are libraries of modules that I arrange in patterns to produce sequences which instruct the machine and can even mean something to a person.

Yet I wonder, is that the only way to program computers? Must we only use rigid pre-delineated symbols?

I think we can begin to write higher level programming environments that allow us to write to our computers, not type, but actually write.

I discovered a groundbreaking paper recently which shows that a unification between the way humans reason and the way computers process might be increasingly possible and fruitful.

Researchers Lake, Salakutdinov and Tenenbaum instantiated a “machine learning” concept by creating a “Bayesian program learning (BPL) framework, capable of learning a large class of visual concepts from just a single example and generalizing in ways that are mostly indistinguishable from people.” Using digital inking they developed a technique to parse drawn symbols via vector and temporal relational information and allow the computer to generate further symbols from these inputs.

“Concepts are represented as simple probabilistic programs—that is, probabilistic generative models expressed as structured procedures in an abstract description language.” Their framework brings together compositionality, causality and learning to learn. “As programs, rich concepts can be built ‘compositionally’ from simpler primitives. Their probabilistic semantics handle noise and support creative generalizations in a procedural form that (unlike other probabilistic models) naturally captures the abstract “causal” structure of the real-world processes that produce examples of a category.”

“Learning proceeds by constructing programs that best explain the observations under a Bayesian criterion, and the model “learns to learn” (23, 24) by developing hierarchical priors that allow previous experience with related concepts to ease learning of new concepts (25, 26). These priors represent a learned inductive bias (27) that abstracts the key regularities and dimensions of variation holding across both types of concepts and across instances (or tokens) of a concept in a given domain.”

“In short, BPL can construct new programs by reusing the pieces of existing ones, capturing the causal and compositional properties of real-world generative processes operating on multiple scales.”

Finding this paper feels profound to me. Lake et al have been able to create a learning system that does not need huge amounts of data, but rather using smaller stochastic programs to represent concepts and building them compositionally from parts, subparts and spatial/temporal relations.

BPL is a generative model for generative models.

The BPL approach gets us away from the traditional histories of computing with their emphasis on large datasets and toward smaller evolutionary rules-based generative computing.

Using the BPL method, concepts are represented as probalistic relational programs, so anything entered by the human operator (or theoretically by other BPL-taught machines) becomes instantly absorbed into a formal logic and is combinatorial at a mathematically grounded and sound level.

The key of BPL is that, like human beings, it allows the computer to start working on relational categorization after just one example. This is how “machine learning” can go from tool of the corporation toward tool of the individual. We individuals do not have thousands or millions of datapoints to give to our personal computers, but we do have individual ideas that we can sketch to them.

I truly think that computer science is going through a revolution in understanding: no longer will computing be about “business machines” and cracking cyphercodes and massive datasets, but instead will increasingly feature generative creative inference and blurry conversation.

The BPL approach, if embedded into the OS of modern personal computing could enable humans to converse with designed emergent libraries of recombinatorial mathematical artifacts. BPL is much more “as we may think” than any of the ‘neural net’ approaches that require astronomically large datasets and vast number crunching. Programming can evolve from reading “tapes” with rigid logics into sketching blurry ideas and creating relational inferences. This is not a replacement, but rather a welcome addition. The BPL approach is still “grounded” in piles of 1/0, but the way that BPL structures the 1/0s is much more modular and inherently combinatorial than previous approaches (from my limited perspective at least).

I think this approach is a keystone I have been seeking to merge ‘symbols that mean’ with ‘symbols that do’ into a unified mathematically complete “metasymbology” that will allow us to merge programming with language. Going further, the authors (and I) see no limits to using a BPL style approach to allow computers to engage with all forms of human symbolism, from language to gestures to dance moves. Even engineered devices and natural complexity, all the way to abstract knowledge such as natural number, natural language semantics and intuitive physical theories. (Lake et al, 1337)

In their history computers have been substrates for enacting human logic, moving forward computers will also become ever better substrates for enacting human dreams.

--

Sources:

Michael S. Mahoney, "The Histories of Computing(s)." Interdisciplinary Science Reviews 30, no. 2 (June 2005).

Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum “Human-level concept learning through probabilistic program induction” https://www.cs.cmu.edu/~rsalakhu/papers/LakeEtAl2015Science.pdf

 

originally for Georgetown CCT class CCTP-820: Leading by Design – Principles of Technical and Social Systems 

The Blockchain is So Meta, Maybe That's Why I'm So Excited By It

The blockchain seems magical: a system that can verify itself. Distributed trust.

Any problem we have today with high transaction costs to bridge it is easy to imagine bridging the gap with some kind of proof of stake, or work, or reference technology. 

This in many ways is the ultimate continuation of the “control revolution”: creating a money that is not a token so much as a network. Something that knows about itself at all times, allowing transparency of a level never before possible. Truly cybernetic society with feedback loops abounding. Imagine a future built environment more like a forest than a static city: where the environment knows how to grow and adapt to what we need.

Yet this world will not simply happen. And it will not allow us to just escape from existing theoretical and historical realities. We are in complex times and complexity is path dependent. We search the fitness landscape from where we are not where we wish we were - unless we take a random shot at mutation.

As we have seen as the internet has matured, promises of utopia are merely waiting for the other shoe to drop. We live in a Topia, the jungle of reality. Both Dys- and U- versions of topia miss the mark on the full truth, yet they do help us shape our trajectory.

{Toward the Answer Engine: Promise & Peril}

We will have to be on guard for both opportunities and downsides as we navigate the future. Through history we have learned that once systems are in place it is difficult to uproot them. Even new shiny systems emerge from the old.

{Capitalism, The Internet & Network Power}

It's important to ask when trying to use technology for "good" what is it that we mean?

{Interview on Millenial Universe - Using Technology for Good}

 

 

 

There is much to consider as we turn the internet into the cyberspace of fiction.

{A New Economy From AR Visors}

A nose for guiding theories and scientific investigation of base principles will be required. I was fortunate to study at Georgetown's Communication Culture & Technology MA program where I explored standards setting, new economic theory and the limits of the calculable, social network analysis and the evolution of technology, among lots of other amazing trans-disciplinary pursuits.

{Lit Review: Information Flow & Link Formation in Networks - Hidden Metric Spaces}

As our understanding grows we see how to avoid the lazy monopoly scenarios - after all simple experiments show us that inequality is a natural emergent behavior of any economy with unequal starting points. Indeed power laws abound in many areas of our world...

{Beyond Network Feudalism} 

If we do this right, we’ll get to somewhere wonderful:

{100 years from now}

Figures from "The Origin of Wealth" by Beinhocker