Bridging Rigid Logics with Blurry Imaginations

The tension between ‘what is’ and ‘what can be’ is omnipresent in technological design. The ‘what can be’ side of the tension further striates into what ‘ought be’ and ‘what can afford to be’ in an industrial economic setting. To me, nowhere is the tension between ‘what is’ and ‘what can be’ more apparent than with digital computers. These devices are substrates for logical operations, and as increasingly diverse communities of people have integrated them into their practices we see a flowering of implementations in software. Yet the initial boundary conditions of the history of computing powerfully shape what it is – where computing has been is the ground for us as we stretch to search for what it can be.

“The devices and systems of technology are not natural phenomena but the products of human design, that is, they are the result of matching available means to desired ends at acceptable cost. The available means ultimately do rest on natural laws, which define the possibilities and limits of the technology. But desired ends and acceptable costs are matters of society.” (Mahoney, 122)

So far ‘desired ends’ of the computational society have been seeded with industrial concerns and perspectives.

“the computer industry was, more than anything else, a continuation of the pre-1945 office equipment industry and in particular of the punched card machine industry.” (Mahoney, 126; quoting Haigh) “But making it universal, or general purpose, also made it indeterminate. Capable of calculating any logical function, it could become anything but was in itself nothing (well, as designed, it could always do arithmetic).” (Mahoney, 123)

Thus, in the 1970s, humanist artists began wading into computation, and we have witnessed an explosion of ‘high level’ creativity as to what the metamedium of ‘computation’ can actually do for us as meaning-makers. Ideas flourished that saw the computer as not just a machine for counting, but a substrate for human imagination. Yet the histories of computing set the devices we compute with on a path that has shaped its form: a device with baked-in logics that we recombine. The histories of computing feature engineering, science and data analysis as the kernel of the computer’s unfolding into the wider sociotechnical ecosystem. Art was tacked on later as an affordance of having enough 1/0s to spare. Computer programs are precise manipulations of the state of an electro-atomic system we call a computer. Yet human language too manipulates other electro-atomic systems (aka, other humans) in a much more blurry and imprecise way - yet this blurriness leaves room for emergence, and this I think is the key to the future direction of computing itself.

I am struck more and more each day by the 20th century origins of computing, and harden my resolve to lean more and more into what the 21st century of computing looks like. The future will see the “front” and “back” of computation merge into a holistic loop where generative logics allow computers to learn as they are used. The loops in our minds will be further augmented by loops through machines that begin to not just manipulate saved libraries, but increasingly generate new forms. We are, I think, at a profound crossroads in the path: will computing be continually defined by linear “processing”, or can we move it toward continuous relational inference? I think we must move to the latter, for the affordances of the future will enable and demand new human-scale ways to program computers. We are in the midst of a latent programming revolution.

This thinking has been culminating for me with the input of this class and my continued experience with the Microsoft Surface. The Surface device that I am typing this on is perhaps the perfect symbol for the crossroads that personal computing is currently in. The Surface has two distinct interface modes: the touchscreen/pen digitizer, and the keyboard. The mouse is unified with the digitizer pen decently well, but the keyboard remains a realm unto itself.

I am finding it increasingly jarring to coexist in free-flowing writing inside of digital inking applications and interfacing with programming.

To this day when writing to a computer at the level of its logical comprehension we are forced to bring our hands together and cramp over an un-changing keyboard. We input 1/0 commands into the machine through keys that correspond to symbols, which in sequence will (when interpreted) illicit the electrical state of the computer to evolve step by step as fast as the system clock allows.

The more I use a pen on a grid, the more I believe that there is potentially another way to program.

The work of von Neumann and others who pioneered the study of cellular automata has shown me that computing does not have to be about direct control using predefined symbol sets, but rather can be about boundary conditions and evolution.

I wonder if we cannot use digitizer grids and pens to allow human operators to sketch with computers. Already much of the power of the computer comes to us via adding abstraction. To edit a photo with machine code directly would be impossibly tedious, but thanks to many layers of abstraction I can use a tool like photoshop to move around thousands of pixels and billions of transistors in large strokes.

Programming languages have been path dependent upon 20th century paradigms. To me, programming a digital computer feels like playing with a near-infinite movable type: there are libraries of modules that I arrange in patterns to produce sequences which instruct the machine and can even mean something to a person.

Yet I wonder, is that the only way to program computers? Must we only use rigid pre-delineated symbols?

I think we can begin to write higher level programming environments that allow us to write to our computers, not type, but actually write.

I discovered a groundbreaking paper recently which shows that a unification between the way humans reason and the way computers process might be increasingly possible and fruitful.

Researchers Lake, Salakutdinov and Tenenbaum instantiated a “machine learning” concept by creating a “Bayesian program learning (BPL) framework, capable of learning a large class of visual concepts from just a single example and generalizing in ways that are mostly indistinguishable from people.” Using digital inking they developed a technique to parse drawn symbols via vector and temporal relational information and allow the computer to generate further symbols from these inputs.

“Concepts are represented as simple probabilistic programs—that is, probabilistic generative models expressed as structured procedures in an abstract description language.” Their framework brings together compositionality, causality and learning to learn. “As programs, rich concepts can be built ‘compositionally’ from simpler primitives. Their probabilistic semantics handle noise and support creative generalizations in a procedural form that (unlike other probabilistic models) naturally captures the abstract “causal” structure of the real-world processes that produce examples of a category.”

“Learning proceeds by constructing programs that best explain the observations under a Bayesian criterion, and the model “learns to learn” (23, 24) by developing hierarchical priors that allow previous experience with related concepts to ease learning of new concepts (25, 26). These priors represent a learned inductive bias (27) that abstracts the key regularities and dimensions of variation holding across both types of concepts and across instances (or tokens) of a concept in a given domain.”

“In short, BPL can construct new programs by reusing the pieces of existing ones, capturing the causal and compositional properties of real-world generative processes operating on multiple scales.”

Finding this paper feels profound to me. Lake et al have been able to create a learning system that does not need huge amounts of data, but rather using smaller stochastic programs to represent concepts and building them compositionally from parts, subparts and spatial/temporal relations.

BPL is a generative model for generative models.

The BPL approach gets us away from the traditional histories of computing with their emphasis on large datasets and toward smaller evolutionary rules-based generative computing.

Using the BPL method, concepts are represented as probalistic relational programs, so anything entered by the human operator (or theoretically by other BPL-taught machines) becomes instantly absorbed into a formal logic and is combinatorial at a mathematically grounded and sound level.

The key of BPL is that, like human beings, it allows the computer to start working on relational categorization after just one example. This is how “machine learning” can go from tool of the corporation toward tool of the individual. We individuals do not have thousands or millions of datapoints to give to our personal computers, but we do have individual ideas that we can sketch to them.

I truly think that computer science is going through a revolution in understanding: no longer will computing be about “business machines” and cracking cyphercodes and massive datasets, but instead will increasingly feature generative creative inference and blurry conversation.

The BPL approach, if embedded into the OS of modern personal computing could enable humans to converse with designed emergent libraries of recombinatorial mathematical artifacts. BPL is much more “as we may think” than any of the ‘neural net’ approaches that require astronomically large datasets and vast number crunching. Programming can evolve from reading “tapes” with rigid logics into sketching blurry ideas and creating relational inferences. This is not a replacement, but rather a welcome addition. The BPL approach is still “grounded” in piles of 1/0, but the way that BPL structures the 1/0s is much more modular and inherently combinatorial than previous approaches (from my limited perspective at least).

I think this approach is a keystone I have been seeking to merge ‘symbols that mean’ with ‘symbols that do’ into a unified mathematically complete “metasymbology” that will allow us to merge programming with language. Going further, the authors (and I) see no limits to using a BPL style approach to allow computers to engage with all forms of human symbolism, from language to gestures to dance moves. Even engineered devices and natural complexity, all the way to abstract knowledge such as natural number, natural language semantics and intuitive physical theories. (Lake et al, 1337)

In their history computers have been substrates for enacting human logic, moving forward computers will also become ever better substrates for enacting human dreams.



Michael S. Mahoney, "The Histories of Computing(s)." Interdisciplinary Science Reviews 30, no. 2 (June 2005).

Brenden M. Lake, Ruslan Salakhutdinov, Joshua B. Tenenbaum “Human-level concept learning through probabilistic program induction”


originally for Georgetown CCT class CCTP-820: Leading by Design – Principles of Technical and Social Systems 

Toward the "Answer Engine" - 1 of 2


Part 1 - The Promise

Also published on Atlantic Council's FutureSource Blog


The Internet is moving beyond a portal to another world and becoming part of the very world itself, meshing with and augmenting physical reality. The idea of the “Alive Web” sees the Internet becoming an increasingly real-time affair.  We are in the beginning. Apps like Snapchat and Twitter already represent the new pace and face of the new web: it’s nearing real-time, and it involves us directly. No longer about stale ‘feeds’ and static data, but experiences and emotions in the present. Less about user names and pseudonyms and more about identity strewn across the world, ready to interact with anything. Not snippets of text and culled media, but a nuanced and raucous interactive conversation happening across the globe. Forget myriad disparate websites and links; imagine an ever-growing pool of everything being fed by a never-ending deluge of data. This is the Internet of tomorrow. 

Currently, when you have a question that cannot be answered within your brain, you must take it upon yourself to seek the answer. This has become so ubiquitous and easy that it merely takes a few taps on a device in your pocket and dozens of answers appear before you. Yet the burden is still on the user, to format the question within guidelines, to cull the answer from myriad sources and arrive at a conclusion. It is still very much a process of actively seeking, which often constitutes a break in flow, if only just a brief one. 

Imagine if you didn’t have to seek. Imagine if you didn’t have to cull. Imagine if the answer to your question was delivered right as you desired it, maybe even before. No break in flow, just ubiquitous information when you need it.

The future will be powerfully defined by the emergence of servile computing, enabled by powerful contextual information. No more asking for help, future computing will simply provide it -from sensors on the device itself, sensors on your body reporting your bodily state, and the coming wealth of ambient information available from the “Internet of Things” along with, of course, the huge amounts of data exhaust we all leave behind in living our cyber lives. Combined with clever and powerful processing, these huge swaths of data can transcend noise and become salient real-time answers. The push toward ubiquitous mobile computing all but demands the emergence of the next generation of search engines, or more accurately, Answer Engines. No more clunky queries, no more digging through links, no more breaking flow, just the answer you want when you need it.

I use the term “Answer Engine” as a catch-all phrase for powerful artificial semi-intelligences—think IBM’s Watson Jeopardy champion— fed by immense sources of data from individuals and environment, coupled with the fluidity of interaction that will characterize the coming wearable modalities of computing. Already the search engine-powered Internet has become part of working cognition, and the Answer Engine-powered Internet likely will see this role increase to the limits of our current imagination.

Interacting with the Answer Engine Internet will be characterized primarily by very little actual interaction. The Answer Engine -- if you choose to enable its features -- can listen to your conversations and do searches on key terms or concepts automatically. The Answer Engine hears you humming along to some tune and can find it for you for later listening. The Answer Engine knows where you are and knows what you’re interested in. It can make suggestions about where you might like to be, or where your schedule says you’re supposed to be. The Answer Engine might even have access to where your gaze is pointed and can infer interest, querying relevant details automatically based on objects, people, locations – anything it thinks you’re interested in. It might even have access to detailed information about your bodily state and help guide you in regulating your health. If you allow it, it might even have access to do this via small wearable medication injection systems or more exotic Nano-machines, biological or otherwise. This is no longer merely about search and ‘social’ applications; this is an extension of self, of mind.

Aside from answers to daily problems and questions, the Answer Engine’s powers will scale exponentially for providing more complex answers. It will have increasing access to networked artifacts and sensors via the “Internet of Things.” The Answer Engine can be used to provide real-time answers to queries about the current state of almost the entire Earth, scaling elegantly from a single room to the entire planet and beyond. It will also have access to networks of connected cameras, terrestrial, and orbital. On a lark, you could have it deliver answers to queries that are almost impossible to address today.

For example, combining its access to sensors, location data for every connected human and artifact, computer vision, and the global mesh of cameras, you could ask it–in natural language–something like, “when it next rains in Paris, count the number of yellow umbrellas seen from beginning of rainfall out 5 minutes; do this for 5 rain days and report the average number of Parisians with yellow umbrellas for this period. Correlate observed location of umbrella with individuals’ locations and ignore duplicates. Report margin of uncertainty.” Useful? Perhaps not. Representative of statistical inference abilities unheard of at any cost today? Very much yes. Of course the data that a badly worded query like this would generate would be full of problems, but think what it represents. Imagine what kinds of questions you might like to ask it, and imagine if you could ask as many as you would like; think of the questions such answers would lead to!

The Answer Engine is about smashing together data from any and all sources available, and future sources of data are set to increase exponentially. Its powers of inference will only grow, and every question it answers or query it receives will be catalogued and recombined to provide better answers in the future, to hone its intelligence further.

While the Answer Engine Internet itself is a powerful amalgamation of artificial intelligence (AI), it also knows humans who might have an answer for you. If you require complex assistance beyond its scope, it might facilitate a connection to someone who can help. The Internet of the future knows where everyone is and can even coordinate them in "meat-space" if so desired. This is the longer-term future of the Answer Engine Internet, presenting all users with a seamless river of relevant information built from unifying human and software abilities and knowledge into one useful whole. 

For example, a little marker might appear in your field of view pointing you to a human near you who shares your interest on a topic or can help you with a 3D-space project. Or you could instantly give an expert access to your current view and have her help you through the problem, likely for a small fee. If today’s networked youth are any indication, many users will opt-in to this system. Providing a way to seamlessly engage with new people anywhere on Earth, requiring only the desire to do so; perhaps only coming together to solve a problem or provide an answer, never to see each other again. As the Internet and physical reality merge, we’ll see online paradigms and modalities of community and interaction rub off on the physical world and vice versa.

Collaboration as fluid and dynamic as meat-space interaction, but no longer limited by physical barriers or confines. The online/offline distinction is dying and in its place a new kind of hybrid reality is emerging. No longer just running parallel to the physical world, but intricately woven into it; responsive and aware, but also affective. The Internet of the future is ubiquitous. The fluidity of querying your own mind applied to the global mind. Software and human voices aggregated together into a seamless whole. As search engines enabled the indexing of data, Answer Engines will enable us to do something with it.

Continue to Part 2: The Peril