This is ubicomp

in digital anima mundi

Saturday, March 17th, 2012

My SxSW session with Sally Applin, PolySocial Reality and the Enspirited World, seemed to be well received. The group that attended was well-engaged and we had a fertile Q&A discussion. Sally focused her keen anthropological lens on the study of our increasingly complex communications with her model of PolySocial Reality; for more on PoSR see Sally’s site. [Update 3/20: Sally posted her slides on PolySocial Reality]. My bit was about the proximate future of pervasive computing, as seen from a particular viewpoint. These ideas are not especially original here in 02012, but hopefully they can serve as a useful nudge toward awareness, insight and mindful action.

What follows is a somewhat pixelated re-rendering of my part of the talk.

This talk is titled “in digital anima mundi (the digital soul of the world).” As far as I know Latin doesn’t have a direct translation for ‘digital’, so this might not be perfect usage. Suggestions welcomed. Anyway, “the digital soul of the world” is my attempt to put a name to the thing that is emerging, as the Net begins to seep into the very fabric of the physical world. I’m using terms like ‘soul’ and ‘enspirited’ deliberately — not because I want to invoke a sacred or supernatural connection, but rather to stand in sharp contrast to technological formulations like “the Internet of Things”, “smart cities”, “information shadows” and the like.

The image here is from Transcendenz, the brilliant thesis project of Michaël Harboun. Don’t miss it.


The idea of anima mundi, a world soul, has been with us for a long time. Here’s Plato in the 4th century BC.


Fast forward to 1969. This is a wonderful passage from P.K. Dick’s novel Ubik, where the protagonist Joe Chip has a spirited argument with his apartment door. So here’s a vision of a world where physical things are animated with some kind of lifelike force. Think also of the dancing brooms and talking candlesticks from Disney’s animated films.


In 1982, William Gibson coined the term ‘cyberspace’ in his short story Burning Chrome, later elaborated in his novel Neuromancer. Cyberspace was a new kind of destination, a place you went to through the gateway of a console and into the network. We thought about cyberspace in terms of…


Cities of data…


Worlds of Warcraft…


A Second Life.


Around 1988, Mark Weiser and a team of researchers at Xerox PARC invented a new computing paradigm they called ubiquitous computing, or ubicomp. The idea was that computing technologies would become ubiquitous, embedded in the physical world around us. Weiser’s group conceived of and built systems of inch-scale, foot-scale and yard-scale computers; these tabs, pads and boards have come to life in today’s iPods, smartphones, tablets and flat panel displays, in form factor if not entirely in function.


In 1992 Rich Gold, a member of the PARC research team, gave a talk titled Art in the Age of Ubicomp. This sketch from Gold’s talk describes a world of everyday objects enspirited with ubicomp. More talking candlesticks, but with a very specific technological architecture in mind.


Recently, Gibson described things this way: cyberspace has everted. It has turned inside out, and we no longer go “into the network”.


Instead, the network has gone into us. Digital data and services are embedded in the fabric of the physical world.


Digital is emerging as a new dimension of reality, an integral property of the physical world. Length, width, height, time, digital.


Since we only have this world, It’s worth exploring the question of whether this is the kind of world we want to live in.


A good place to begin is with augmented reality, the idea that digital data and services are overlaid on the physical world in context, visible only when you look through the right kind of electronic window. Today that’s smartphones and tablets; at some point that might be through a heads-up display, the long-anticipated AR glasses.


Game designers are populating AR space around us with ghosts and zombies.


Geolocative data are being visualized in AR, like this crime database from SpotCrime.


History is implicit in our world; historical photos and media can make these stories explicit and visible, like this project on the Stanford University quad.


Here’s a 3D reconstruction, a simulation of the Berlin Wall as it ran through the city of Berlin.


Of course AR has been applied to a lot of brand marketing campaigns in the last year or two, like this holiday cups app from Starbucks.


AR is also being adopted by artists and culture jammers, in part as a way to reclaim visual space from the already pervasive brand encroachment we are familiar with.


We also have the Internet of Things, the notion that in just a few years there will be 20, 30, 50 billion devices connected to the Net. Companies like Cisco and Intel see huge commercial opportunities and a wide range of new applications.


An Internet of Things needs hyperlinks, and you can think of RFID tags, QR codes and the like as physical hyperlinks. You “click” on them  in some way, and they invoke a nominally relevant digital service.


RFID and NFC have seen significant uptake in transit and transportation. In London, your Will and Kate commemorative Oyster card is your ticket to ride the Underground. In Japan, your Octopus or Suica card not only lets you ride the trains, but also purchase items from vending machines and pay for your on-street parking. In California we have FasTrak for our cars, allowing automated payment at toll booths. These systems improve efficiency of the infrastructure sevices and provide convenience to citizens. However, they are also becoming control points for access to public resources, and vast amounts of data are generated and mined based on the digital footprints we leave behind.


Sensors are key to the IoT. Botanicalls is a product from a few years ago, a communicating moisture sensor for your houseplants. When the soil gets dry, the Botanicall sends you a tweet to let you know your plant is thirsty.


More recently, the EOS Talking Tree is an instrumented tree that has a Facebook page and a Twitter account with more than 4000 followers. That’s way more than me.


This little gadget is the Rymble, billed by its creators as an emotional Internet device. You connect it with your Facebook profile, and it responds to activity by spinning around, playing sounds and flashing lights in nominally meaningful ways. This is interesting; not only are physical things routinely connected to services, but services are sprouting physical manifestations.


This is a MEMS sensor, about 1mm across, an accelerometer & gyroscope that measures motion. If you have a smartphone or tablet, you have these inside to track the tilt, rotation and translation of the device. These chips are showing up in a lot of places.


Some of you probably have a FitBit, Nike+, FuelBand, WiThings scale. Welcome to the ‘quantified self’ movement. These devices sense your physical activity, your sleep and so on, and feed the data into services and dashboards. They can be useful, fun and motivating, but know also that your physical activities are being tracked, recorded, gamified, shared and monetized.


Insurance companies are now offering sensor modules you can install on your car. They will provide you with metered, pay-as-you-drive insurance, with variable pricing based on the risk of when, where and how safely you drive.


Green Goose wants you to brush your teeth. If you do a good job, you’ll get a nice badge.


How about the Internet of Babies? This is a real product, announced a couple of weeks ago at Mobile World Congress. Sensors inside the onesie detect baby’s motion and moisture content.


Here’s a different wearable concept from Philips Design, the Bubelle Dress that senses your mood and changes colors and light patterns in response.


So physical things, places and people are becoming gateways to services, and services are colonizing the physical world. Microsoft’s Kinect is a great example of a sensor that bridges physical and digital; the image is from a Kinect depth camera stream. This is how robots see us.


If a was a service, I think I’d choose some of these robots for my physical instantiation. You’ve probably seen these — DARPA’s Alpha Dog all-terrain robotic pack horse, DARPA’s robot hummingbird, Google’s self-driving cars. You might not think of cars as robots, but these are pretty much the same kinds of things.


Robots also come in swarms. This is a project called Electronic Countermeasures by Liam Young. A swarm of quadrotor drones forms a dynamic pirate wireless network, bringing connectivity to spaces where the network has failed or been jammed. When the police drones come to shoot them down, they disperse and re-form elsewhere in the city.


A team at Harvard is creating Robobees. This is a flat multilayer design that can be stamped out in volume. It is designed so that the robot bee pops up and folds like origami into the shape at top right. I wonder what kind of service wants to be a swarm of robotic bees?


On a larger scale, IBM wants to build you a smarter city. There are large smart city projects around the globe, being built by companies like IBM, Cisco and Siemens. They view the city as a collection of networks and systems – energy, utilities, transportation etc – to be measured, monitored, managed and optimized. Operational efficiency for the city, and convenience for citizens.


But we as individuals don’t experience the city as a stack of infrastructures to be managed. Here’s Italo Calvino in his lovely book Invisible Cities. “Cities, like dreams, are made of desires and fears…the thread of their discourse is secret, their rules absurd.”


Back at ground level in the not-so-smart city of today, displays are proliferating. Everywhere you turn, public screens are beaming messages from storefronts, billboards and elevators.


We’re getting close to the point where inexpensive, flexible plastic electronics and displays will be available. When that happens, every surface will be a potential site for displays.


We’re also seeing cameras becoming pervasive in public places. When you see a surveillance camera, do you think it’s being monitored by a security guard sitting in front of a bank of monitors as seen in so many movies? More likely, what’s behind the camera is a sophisticated computer vision system like this one from Quividi, that is constantly analyzing the scene to determine things like the gender, age and attention of people passing by.


A similar system from Intel called Cognovision is being used in a service called SceneTap, which monitors the activity in local nightclubs  to let you know where the hottest spots are at any given moment.


You’ve probably seen something like this. It’s worth remembering that our technologies are all too brittle, and you should expect to see more of this kind of less-than-graceful degradation.


In case the city isn’t big enough, IBM wants to bring us a smarter planet. HP wants to deploy a trillion sensors to create a central nervous system for the earth. “The planet will be instrumented, interconnected and intelligent. People want it.” But do we? Maybe yes, maybe no?


So we come back to the question, what kind of world do you want to live in? Almost everything I’ve talked about is happening today. The world is becoming digitally transformed through technology.


Many of these technologies hold great promise and will add tremendous value to our lives. But digital technology is not neutral — it has inherent affordances and biases that influence what gets built. These technologies are extremely good at concrete, objective tasks: calculating, connecting, distributing and storing, measuring and analyzing, transactions and notifications, control and optimization. So these are often fundamental characteristics of the systems that we see deployed; they reflect the materials from which they are made.


We are bringing the Internet into the physical world. Will the Internet of people, places and things be open like the Net, a commons for the good of all? Or will it be more like a collection of app stores? Will there be the physical equivalents of spam, cookies, click-wrap licensing and contextual advertising? Will Apple, Google, Facebook and Amazon own your pocket, your wallet and your identity?


And what about the abstract, subjective qualities that we value in our lives? Technology does not do empathy well. What about reflection, emotion, trust and nuance? What about beauty, grace and soul? In digital anima mundi?


In conclusion, I’d like to share two quotes. First, something Bruce Sterling said at an AR conference two years ago. You are the world’s first pure play experience designers. We are remaking our world, and this a very different sort of design than we are used to.


What it is, is up to us. Howard first said it more than 25 years ago, and it has never been more true than today.


I want to acknowledge these sources for many of the images herein.


@ubistudio: Introducing the Ubiquitous Media Studio

Tuesday, July 13th, 2010

As promised during my talk at ARE2010, I’m launching a new project called the Ubiquitous Media Studio, a.k.a. @ubistudio. The idea is to gather an open network of technologists, artists, experience designers, social scientists and other interested folks, to explore the question “If the world is our platform, then what is our creative medium?” I’m provisionally calling this notion “ubiquitous media”, building on initial research I did in this area several years back. The idea is also very much inspired and influenced by my friends at the most excellent Pervasive Media Studio in Bristol England, who you should know as well.
button-ubi So what is ubiquitous media? I don’t know exactly, thus the exploration. But it seems to me that its outlines can be sensed in the choppy confluence of ubicomp, social networks, augmented reality, physical computing, personal sensing, transmedia and urban systems. It’s like that ancient parable of the blind monks trying to describe an elephant; the parts all feel very weird and different, and we’re trying to catch a glimpse of the future in its entirety. When you look through an AR magic lens, ubiquitous media is in there. When your kid went crazy over the Pokemon and Yu-Gi-Oh story-game universes, it was in there too. When you snap your Nike+ sensor into your running shoe, you’re soaking in it. When you go on a soundwalk or play a mediascape, there’s more than a bit of ubiquitous media in the experience.


Anyway, we are going to investigate this, with the goals of learning new creative tools and applying them in creative projects. And “we” includes you. If you’re in the Bay Area and you think you might be interested, just jump right in! We’re having a little get-together in Palo Alto:

@ubistudio: Ubiquitous Media Studio #1
Thursday July 22, 2010 5:30-8:30PM
Venue: The Institute for the Future
Details & RSVP:

I hope you’ll join us. You can also stay connected through @ubistudio on Twitter, and a soon-to-be-more-than-a-placeholder website at

Nokia’s approach to mobile augmented reality

Tuesday, August 25th, 2009

We had good AR-related fun at last night’s talk by Kari Pulli and Radek Grzeszczuk from Nokia Research, “Nokia Augmented Reality” hosted by SDForum. It was basically a survey of AR-related work done at Nokia in the last few years, with special emphasis on their research work in image-based recognition.

Kari presented an overview of several research projects, including:

MARA (Mobile Augmented Reality Applications) — A GPS+compass style AR prototype, also using accelerometers as part of the user interaction model. See also this Technology Review article from Nov06.

Image Space — “A user-created mirror world”, or less romantically, a social photo capture & sharing system, using GPS+compass to locate and orient photos you take and upload, and allowing you to browse others’ photos taken nearby.

Landmark-Based Pedestrian Navigation from Collections of Geotagged Photos — A bit hard to describe, best to have a scan of the research paper (pdf).

Point & Find — Mobile service that uses image-based recognition to tag and identify physical objects such as products, movie posters and buildings and provide links to relevant content and services. This is being incubated as a commercial service and is currently in public beta.


Radek did a technical dive into their approach to image-based recognition, touching on a variety of image processing techniques and algorithms for efficiently extracting the salient geometric features from a photograph, and identifying exact matching images from a database of millions of images. The algorithms were sufficiently lightweight to run well on a smartphone-class processor, although matching against large image collections obviously requires a client-server partitioning. This work seems to be an important part of NRC’s approach to mobile AR, and Radek noted that their current research includes extending the approach to 3D geometry as well as extracting features from streaming images. Capturing a comprehensive database of images of items and structures in the world is one barrier they face, and they are considering ways to use existing collections like Google’s Street View as well as urban 3D geometry datasets such as being created by Earthmine. Another area for further work is matching images that do not contain strong, consistent geometric features; the algorithms described here are not useful for faces or trees, for example.

Update: DJ Cline was there and has pictures of the event, the slides and the demo.

Related links:
SDForum Virtual Worlds SIG
SDForum Emerging Technologies SIG
Gathering of the AR tribes: ISMAR09 in Orlando

a brief response re: web squared

Friday, July 10th, 2009

Tim O’Reilly and John Battelle recently proposed the term “Web Squared” to describe the next phase of the web, where “web meets world” in a melange of collective intelligence, data utilities, pervasive sensing, real time feedback, visualization, emergent semantic structure, and information infusing the physical world. For what it’s worth, I quite like it. We needed a new handle for the remarkable confluence of technologies we are experiencing, and I think Web Squared nicely captures the exponential expansion of possibilities while reaffirming that the web is the only plausible distributed systems infrastructure to build the new world on.

I was also intrigued by the authors’ conclusion, which moves the discussion beyond the realm of technology and into “the stuff that matters”:

All of this is in many ways a preamble to what may be the most important part of the Web Squared opportunity. The new direction for the Web, its collision course with the physical world, opens enormous new possibilities for business, and enormous new possibilities to make a difference on the world’s most pressing problems.

As a techno-optimist by nature, I’m pretty susceptible to visions of enormous new possibilities. I’ve even generated a few of those lovely consensual hallucinations myself, and they can be very exciting to be in the middle of. And it’s almost certainly true – the potential implications are huge. However, I think we also need to examine this vision more critically as part of the ongoing discussion, for example giving serious attention to Adam Greenfield’s design principles for Everyware, and to John Thackara’s concerns when he writes:

Connected environments…and the Internet of Things as a whole, are not a step forwards if they guzzle matter and energy as profligately as the internet of emails does

and echoes Patricia de Martelaere’s caution against

“wasting our lives by continuously watching images of world-processes, or processes of our own body, and desperately trying to interfere – like a man chasing his own shadow.”

After all, in the era of Web Squared we are not just creating new business opportunities; we are talking about cyberspace seeping out of the very fabric of reality. I’m thinking that we don’t want to screw that up.

what is ubiquitous media?

Friday, June 26th, 2009

In the 2003 short paper “Creating and Experiencing Ubimedia“, members of my research group sketched a new conceptual model for interconnected media experiences in a ubiquitous computing environment. At the time, we observed that media was evolving from single content objects in a single format (e.g., a movie or a book), to collections of related content objects across several formats. This was exemplified by media properties like Pokemon and Star Wars, which manifested as coherent fictional universes of character and story across TV, movies, books, games, physical action figures, clothing and toys, and American Idol which harnessed large-scale participatory engagement across TV, phones/text, live concerts and the web. Along the same lines, social scientist Mimi Ito wrote about her study of Japanese media mix culture in “Technologies of the Childhood Imagination: Yugioh, Media Mixes, and Otaku” in 2004, and Henry Jenkins published his notable Convergence Culture in 2006. We know this phenomenon today as cross-media, transmedia, or any of dozens of related terms.

Coming from a ubicomp perspective, our view was that the implicit semantic linkages between media objects would also become explicit connections, through digital and physical hyperlinking. Any single media object would become a connected facet of a larger interlinked media structure that spanned the physical and digital worlds. Further, the creation and experience of these ubimedia structures would take place in the context of a ubiquitous computing technology platform combining fixed, mobile, embedded and cloud computing with a wide range of physical sensing and actuating technologies. So this is the sense in which I use the term ubiquitous media; it is hypermedia that is made for and experienced on a ubicomp platform in the blended physical/digital world.

Of course the definitions of ubicomp and transmedia are already quite fuzzy, and the boundaries are constantly expanding as more research and creative development occur. A few examples of ubiquitous media might help demonstrate the range of possibilities:


An interesting commercial application is the Nike+ running system, jointly developed between Nike and Apple. A small wireless pressure sensor installed in a running shoe sends footfall data to the runner’s iPod, which also plays music selected for the workout. The data from the run is later uploaded to an online service for analysis and display. The online service includes social components, game mechanics, and the ability to mashup running data with maps. Nike-sponsored professional athletes endorse Nike-branded music playlists on Apple’s iTunes store. A recent feature extends Nike+ connectivity to specially-designed exercise machines in selected gyms. Nike+ is a simple but elegant example of embodied ubicomp-based media that integrates sensing, networking, mobility, embedded computing, cloud services, and digital representations of people, places and things. Nike+ creates new kinds of experiences for runners, and gives Nike new ways to extend their value proposition, expand their brand footprint, and build customer loyalty. Nike+ has been around since 2006, but with the recent buzz about personal sensing and quantified selves it is receiving renewed attention including a solid article in the latest Wired.


A good pre-commercial example is HP Labs’ mscape system for creating and playing a media type called mediascapes. These are interactive experiences that overlay audio, visual and embodied media interactions onto a physical landscape. Elements of the experience are triggered by player actions and sensor readings, especially location-based sensing via GPS. In the current generation, mscape includes authoring tools for creating mediascapes on a standard PC, player software for running the pieces on mobile devices, and a community website for sharing user-created mediascapes. Hundreds of artists and authors are actively using mscape, creating a wide variety of experiences including treasure hunts, biofeedback games, walking tours of cities, historical sites and national parks, educational tools, and artistic pieces. Mscape enables individuals and teams to produce sophisticated, expressive media experiences, and its open innovation model gives HP access to a vibrant and engaged creative community beyond the walls of the laboratory.

These two examples demonstrate an essential point about ubiquitous media: in a ubicomp world, anything – a shoe, a city, your own body – can become a touchpoint for engaging people with media. The potential for new experiences is quite literally everywhere. At the same time, the production of ubiquitous media pushes us out of our comfort zones – asking us to embrace new technologies, new collaborators, new ways of engaging with our customers and our publics, new business ecologies, and new skill sets. It seems there’s a lot to do, so let’s get to it.

a few remarks about augmented reality and layar

Wednesday, June 24th, 2009

I genuinely enjoyed the demo videos from last week’s launch of the Layar AR browser platform. The team has made a nice looking app with some interesting features, and I’m excited about the prospects of an iPhone 3GS version and of course some local Silicon Valley layarage.

At a technical level, I was reminded of my Cooltown colleagues’ Websign project, which had the very similar core functionality of a mobile device with integrated GPS and magnetometer, plus a set of web services and a markup language for binding web resources (URLs) to locations with control parameters (see also: Websigns: Hyperlinking Physical Locations to the Web in IEEE Computer, August 2001). It was a sweet prototype system, but it never made it out of the lab because there was no practical device with a digital compass until the G1 arrived. Now that we have location and direction support in production platforms, I’m pretty sure this concept will take off. Watch out for the patents in this area though, I think there was closely related prior art that even predated our work.

Anyway I looked carefully at all the demos from Layar and the various online coverage, and wondered about a few things:

  • Layar’s graphical overlay of points of interest appears to be derived entirely from the user’s location and the direction the phone is pointed. There is no attempt to do real-time registration of the AR graphics with objects in the camera image, which is the kind of AR that currently requires markers or a super-duper 3D point cloud like Earthmine. That’s fine for many applications, and it is definitely an advantage for hyperlinks bound to locations that are out of the user’s line of sight (behind a nearby building, for example). Given this, I don’t understand why Layar uses the camera at all. The interaction model seems wrong; rather than using Layar as a viewfinder held vertically in my line of sight, I want to use it like a compass — horizontally like a map, and the phone pointed axially toward my direction of interest. This is most obvious in the Engadget video, where they are sitting in a room and the links from across town are overlaid on images of the bookshelves ;-) Also, it seems a bit unwieldy and socially awkward to be walking down the street holding the phone in front of you. Just my $0.02 there.
  • How will Layar handle the navigation problem of large numbers of active items? The concept of separate “layars” obviously helps, but in a densely augmented location you might have hundreds or even thousands of different layers. Yes this is a hard UI/UX problem, but I guess it’s a problem we would love to have, too much geowebby goodness to sort through. I suppose it will require some nicely intuitive search/filtering capability in the browser, maybe with hints from your personal history and intent profile.
  • Will Layar enable participatory geoweb media creation? I’d be surprised if they don’t plan to do this, and I hope it comes quickly. There will be plenty of official corporate and institutional voices in the geoweb, but a vibrant and creative ecosystem will only emerge from public participation in the commons. This will demand another layer of media literacy, and this will take time and experimentation to develop. I say the sooner we get started, the better.

In any case, good luck to the Layar team!

a remarkable confluence of technologies

Tuesday, June 23rd, 2009


I have used versions of this picture in many of my talks over the last 10 years, and it just keeps getting more interesting. The overarching message is that we are in the midst of a remarkable wave of innovation, with major advances coming to fruition across many different technology domains, at more or less the same time. This is creating fertile conditions for all manner of new products, services and experiences to emerge through what economist Hal Varian calls “combinatorial innovation” (a nicely refined phrase for an incredibly messy process). Another way to put it is, this is the ubiquitous computing + digital media + internet + physical world supercollider, and we are starting to see the results of a very big, long-running experiment.

It has become somewhat of a cliche to promise radical transformation of businesses through technology, but just because it’s a cliche doesn’t mean it’s wrong. Over the next several years, much of the physical world will become connected into an Internet of people, places & things. This will fundamentally change the nature of our experiences with products, services, environments, organizations, and each other. There is no industry or institution that will be untouched; in retrospect we will see that traditional media companies were simply the low-hanging fruit. Just as the Nike+ running system transforms a shoe into a networked sensor node and an apparel company into the host of a worldwide social media community, combinatorial innovation will plant seeds of opportunity and disruption widely.

(Click image for ridiculously large version, cc-by-sa)