One of the most talked-about Ted Talks in recent years was the one delivered by John Underkoffler in February 2010. Underkoffler, best known as the science advisor and computer interface designer for the 2002 movie “Minority Report,” showed an astonished audience that the futuristic spatial operating environment in which Tom Cruise’s character manipulated data across displays with hand gestures isn’t just the stuff of sci-fi movies. It’s real.
As he walked around the stage, Underkoffler, co-founder and CEO of Oblong Industries in Los Angeles, the company that’s commercializing this technology in the form of its core product, Mezzanine, demonstrated that computers can now understand real-world space, so that the room you’re in becomes the operating environment, and you become the pointing device. In an interview last week, I spoke at length with Underkoffler about this technology and its application. I began by asking him to explain what he meant by something he said during that Ted Talk that appears to be one of the underlying premises of his work: “The OS is the interface; the interface is the OS.” Underkoffler, who also demonstrated that amiability and humility can coexist in perfect harmony with brilliance, said there are two important meanings or interpretations of it:
One is the very literal one — part of what I was saying is that the UI, the way that you interact with the machine, essentially defines the technology all the way down. They’re really synonymous. You have to build new kinds of basic technology in order to provide a new kind of user experience. Simply that, and nothing more. And I think I called out in the talk that this is exactly what the Macintosh team had found in the early 80s when they were building the world’s first commercial GUI. They were taking the world from the command line — text in, text out, one-dimensional, progressing from right to left and then carriage return, for those of us old enough to remember what a carriage return is — to a two-dimensional pictorial interface, where it’s pictures, not words. It’s the visual part of the human brain, not the logical and linguistic part. They were all huge, important steps forward, but that technical team at Apple in the early 80s couldn’t build on what was there — they had to start essentially from scratch. Because the UI asks so much of the technology that unless you’ve designed the core operating system with a particular UI in mind, you’re not going to be able to get there. So that’s the first meaning of that phrase.
The other meaning, Underkoffler said, in terms of our experience as human beings, is that the UI really is everything:
You can have as much RAM and cloud and Hadoop and whatever else you want behind the scenes — silicon wires, whatever it is, it doesn’t matter. Because the only thing you can actually see and feel and touch and experience is the UI — kind of by definition. On the one hand, that seems obvious, or tautological. But I think it’s really, really important. We go through phases, as technologically fascinated human beings, of being obsessed with stuff that we can’t touch — obsessed with the cloud, obsessed with this and that that’s literally and figuratively distant from us. But my mantra has always been, “Let’s pay a little more attention to the stuff that people are actually using.”
At the end of that Ted Talk, when Underkoffler was asked when all of this was going to happen, he responded, “I think in five years’ time, when you buy a computer, you’ll get this.” I asked Underkoffler if he wouldn’t agree that now, seven years later, what we actually get when we buy a computer really hasn’t changed all that much after all. He said he absolutely agreed:
Of course, optimism, and maybe a little bit of over-optimism, are part and parcel of being an entrepreneur, so I plead guilty to the calendar misfire there. I still essentially believe what I said, but you’re right, we’ve gone sailing right past the five-year horizon that I predicted in the talk. On the one hand, I think that you’re right, the interface we have on the most capable machines that we use, which is to say laptop machines and desktop machines, has not changed. It really hasn’t changed, and I was feeling indignant about that when I got started that it was still the same, and here it is all these years later, and we’ve still seen no change.
That understood, Underkoffler said what’s important to note is that the horizon has opened up, and people no longer assume that that mode of using a computer is the only one:
That really is the big change. People conduct big chunks of their daily digital lives not typing into a laptop machine and using that UI, but rather using a smartphone or a tablet — I guess less and less a tablet these days, but a smartphone, for sure. There are a couple of billion of those out there, right? So I think that’s the critical event, and it’s as important as it might have been if we’d all moved over to a gestural interface on our computers. Because that really paves the way for the possibility of something really excitingly new. Now, the UI that you get on your smartphone is not the equal of the UI you get on your laptop. As a journalist I’m guessing you probably don’t do 120 words per minute on any smartphone — the UI isn’t for that, and it’s not that good. But what’s valuable about it is that it’s portable — you take a little bit of computation with you. And so, we’ve gone from a view of the world that was really pre-Copernican, which said that the entire universe revolves around this one GUI, the GUI that we know from our laptop and desktop machines, to saying, actually there’s a constellation out there, and we now know at least two major UIs: There’s the GUI on your laptop, and there’s the touch-based interface on your phone. And if there’s two — and this is what’s important to us at Oblong — there can be more.
I asked Underkoffler what Oblong Industries will be delivering five years from now that it’s not delivering today. His response:
That’s an easy one. The answer is ubiquity — the collaborative computing experience, the “I see what you see, we all see the same thing” mantra, which is at the core of Mezzanine, is going to be universal. That Mezzanine experience will be accessible on every device in your life: your laptops, your desktop machines, your tablets, your smartphones, big Mezzanine rooms that are constructed for that purpose, but also in the home. The flat-screen TV in your living room should operate that same way — why wouldn’t it? The idea is that you can transfer visual content and applications and data and documents between any of these devices just as smoothly as you would pick up a piece of paper from a table and tape it to a wall. All that’s just going to be part of our understanding of how the digital world works.
Of course, I had to ask the obvious question: He got the five years wrong in the Ted talk, so what’s to say he’s going to get the five years right this time? Underkoffler said it was a totally fair question:
The difference is, we know a huge amount more now at Oblong than we did back then. We were just starting in 2010 to build Mezzanine, so we had a good run building raw technology and building domain-specific applications with really interesting customers, like Boeing and GE and Saudi Aramco, up through 2010, 2011. But by 2010, by the time that I’m standing there on stage in Long Beach, we’ve got this idea and we’re starting to build Mezzanine. Now that we’ve had it out in the world for three-and-a-half or four years as a commercial offering, and as an instantiation of these ideas, we know what’s important. We know what’s not important. We know which direction we have to go in.
For example, the gestural stuff that’s very much on display in “Minority Report” and in the Ted Talk — I’ve never said this before to anyone, and certainly not to a journalist, but this is an interesting moment — that stuff is important, but it’s hardly the most important thing. The collaborative element is way more important. The idea that pixels become a universal interchange format, that your application can hop across to my screen, and some data from my device can hop across to your screen, all of that is radically more important. When you know where the value lies, you can drive toward it very quickly. And that’s why I’m confident that when you and I talk again in 365 times 5 days, this time you won’t be saying, “Well, you got it wrong again, John!”
Underkoffler wrapped up this part of our conversation by highlighting his fascination with the phrase, “see more, do more”:
I’d be really interested to ask your readers if they feel like the digital systems that they use today get done what they need. Can they see enough of the problem they’re working on, and does the UI let them reach in and really get done what they need to get done? I think the answer is no, and I think we’re going to fix that pretty rapidly. But it’s one of those iceberg things, where there’s very little above the surface, what people are talking about, and a huge amount under the surface. And it’s either a danger if you don’t do anything about it, or it’s a huge opportunity if you recognize it and really drive right toward it. Is the world seeing more and doing more in the digital realm? Let’s find out.
I also spoke at some length with Underkoffler about his work on “Minority Report” and the second movie he was involved with, “Iron Man,” and about how that technology is being used in the real world. I’ll cover those topics in a forthcoming post.
A contributing writer on IT management and career topics with IT Business Edge since 2009, Don Tennant began his technology journalism career in 1990 in Hong Kong, where he served as editor of the Hong Kong edition of Computerworld. After returning to the U.S. in 2000, he became Editor in Chief of the U.S. edition of Computerworld, and later assumed the editorial directorship of Computerworld and InfoWorld. Don was presented with the 2007 Timothy White Award for Editorial Integrity by American Business Media, and he is a recipient of the Jesse H. Neal National Business Journalism Award for editorial excellence in news coverage. Follow him on Twitter @dontennant.
