Archive for the ‘mobility media ubicomp’ Category

Google’s Project Glass is a new model for personal computing

Friday, April 6th, 2012

The concept video of Google’s Project Glass has whipped up an Internet frenzy since it was released earlier this week, with breathless coverage (and more than a little skepticism) about the alpha-stage prototype wearable devices. Most of the reporting has focused on the ‘AR glasses’ angle with headlines like “Google Shows Off, Teases Augmented Reality Spectacles“, but I don’t think Project Glass is about augmented reality at all. The way I see it, Glass is actually about creating a new model for personal computing.

Think about it. In the concept video, you see none of the typical AR tropes like 3D animated characters, pop-up object callouts and video-textured building facades. And tellingly, there’s not even a hint of Google’s own Goggles AR/visual search product. Instead, what we see is a heads-up, hands-free, continuous computing experience tightly integrated with the user’s physical and social context. Glass posits a new use model based on a novel hardware platform, new interaction modalities and new design patterns, and it fundamentally alters our relationship to digital information and the physical environment.

This is a much more ambitious idea than AR or visual search. I think we’re looking at Sergey’s answer to Apple’s touch-based model of personal computing. It’s audacious, provocative and it seems nearly impossible that Google could pull it off, which puts it squarely in the realm of things Google most loves to do. Unfortunately in this case I believe they have tipped their hand too soon.

Photo: Thomas Hawk / Flickr

Let’s suspend disbelief for a moment and consider some of the implications of Glass-style computing. There’s a long list of quite difficult engineering, design, cultural and business challenges that Google has to resolve. Of these, I’m particularly interested in the aspects related to experience design:

Continuous computing

The rapid adoption of smartphones is ample evidence that people want to have their digital environment with them constantly. We pull them out in almost any circumstance, we allow people and services to interrupt us frequently and we feed them with a steady stream of photos, check-ins, status updates and digital footprints. An unconsciously wearable heads-up device such as Glass takes the next step, enabling a continuous computing experience interwoven with our physical senses and situation. It’s a model that is very much in the Bush/Engelbart tradition of augmenting human capabilities, but it also has the potential to exacerbate the problematic complexity of interactions as described by polysocial reality.

A continuous computing model needs to be designed in a way that complements human sensing and cognition. Transferring focus of attention between cognitive contexts must be effortless; in the Glass video, the subject shifts his attention between physical and digital environments dozens of times in a few short vignettes. Applications must also respect the unique properties of the human visual system. Foveal interaction must co-exist and not interfere with natural vision. Our peripheral vision is highly sensitive to motion, and frequent graphical activity will be an undesirable distraction. The Glass video presents a simplistic visual model that would likely fail as a continuous interface.

Continuous heads-up computing has the potential to enable useful new capabilities such as large virtual displays, telepresent collaboration, and enhanced multi-screen interactions. It might also be the long-awaited catalyst for adoption of locative and contextual media. I see continuous computing as having enormous potential and demanding deep insight and innovation; it could easily spur a new wave of creativity and economic value.

Heads-up, hands-free interaction

The interaction models and mechanisms for heads-up, hands-free computing will be make-or-break for Glass. Speech recognition, eye tracking and head motion modalities are on display in the concept video, and their accuracy and responsiveness is idealized. The actual state of these technologies is somewhat less than ideal today, although much progress has been made in the last few years. Our non-shiny-happy world of noisy environments, sweaty brows and unreliable network performance will present significant challenges here.

Assuming the baseline I/O technologies can be made to work, Glass will need an interaction language. What are the hands-free equivalents of select, click, scroll, drag, pinch, swipe, copy/paste, show/hide and quit? How does the system differentiate between an interface command and a nod, a word, a glance meant for a friend?


Physical and social context can add richness to the experience of Glass. But contextual computing is a hard problem, and again the Glass video treats context in a naïve and idealized way. We know from AR that the accuracy of device location and orientation is limited and can vary unpredictably in urban settings, and indoor location is still an unsolved problem. We also know that geographic location (i.e., latitude & longitude) does not translate to semantic location (e.g., “in Strand Books”).

On the other hand, simple contextual information such as time, velocity of travel, day/night, in/outdoors is available and has not been exploited by most apps. Google’s work in sensor fusion and recognition of text, images, sounds & objects could also be brought to bear on the continuous computing model.

Continuous capture

With its omnipresent camera, mic and sensors, Glass could be the first viable life recorder, enabling the “life TiVo” and total recall capabilities explored by researchers such as Steve Mann and Gordon Bell. Continuous capture will be a tremendous accelerant for participatory media services, from YouTube and Instagram-style apps to citizen journalism. It will also fuel the already heated discussions about privacy and the implications of mediated interpersonal relationships.

Of course there are many other unanswered questions here. Will Glass be an open development platform or a closed Google garden? What is the software model — are we looking at custom apps? Some kind of HTML5/JS/CSS rendering? Will there be a Glass equivalent to CocoaTouch? Is it Android under the hood? How much of the hard optical and electrical engineering work has already been done? And of course, would we accept an even more intimate relationship with a company that exists to monetize our every act and intention?

The idea of a heads-up, hands-free, continuous model of personal computing is very interesting, and done well it could be a compelling advance. But even if we allow that Google might have the sophistication and taste required, it feels like there’s a good 3-5+ years of work to be done before Glass could evolve from concept prototype into a credible new computing experience. And that’s why I think Google has tipped their hand far too soon.

in digital anima mundi

Saturday, March 17th, 2012

My SxSW session with Sally Applin, PolySocial Reality and the Enspirited World, seemed to be well received. The group that attended was well-engaged and we had a fertile Q&A discussion. Sally focused her keen anthropological lens on the study of our increasingly complex communications with her model of PolySocial Reality; for more on PoSR see Sally’s site. [Update 3/20: Sally posted her slides on PolySocial Reality]. My bit was about the proximate future of pervasive computing, as seen from a particular viewpoint. These ideas are not especially original here in 02012, but hopefully they can serve as a useful nudge toward awareness, insight and mindful action.

What follows is a somewhat pixelated re-rendering of my part of the talk.

This talk is titled “in digital anima mundi (the digital soul of the world).” As far as I know Latin doesn’t have a direct translation for ‘digital’, so this might not be perfect usage. Suggestions welcomed. Anyway, “the digital soul of the world” is my attempt to put a name to the thing that is emerging, as the Net begins to seep into the very fabric of the physical world. I’m using terms like ‘soul’ and ‘enspirited’ deliberately — not because I want to invoke a sacred or supernatural connection, but rather to stand in sharp contrast to technological formulations like “the Internet of Things”, “smart cities”, “information shadows” and the like.

The image here is from Transcendenz, the brilliant thesis project of Michaël Harboun. Don’t miss it.


The idea of anima mundi, a world soul, has been with us for a long time. Here’s Plato in the 4th century BC.


Fast forward to 1969. This is a wonderful passage from P.K. Dick’s novel Ubik, where the protagonist Joe Chip has a spirited argument with his apartment door. So here’s a vision of a world where physical things are animated with some kind of lifelike force. Think also of the dancing brooms and talking candlesticks from Disney’s animated films.


In 1982, William Gibson coined the term ‘cyberspace’ in his short story Burning Chrome, later elaborated in his novel Neuromancer. Cyberspace was a new kind of destination, a place you went to through the gateway of a console and into the network. We thought about cyberspace in terms of…


Cities of data…


Worlds of Warcraft…


A Second Life.


Around 1988, Mark Weiser and a team of researchers at Xerox PARC invented a new computing paradigm they called ubiquitous computing, or ubicomp. The idea was that computing technologies would become ubiquitous, embedded in the physical world around us. Weiser’s group conceived of and built systems of inch-scale, foot-scale and yard-scale computers; these tabs, pads and boards have come to life in today’s iPods, smartphones, tablets and flat panel displays, in form factor if not entirely in function.


In 1992 Rich Gold, a member of the PARC research team, gave a talk titled Art in the Age of Ubicomp. This sketch from Gold’s talk describes a world of everyday objects enspirited with ubicomp. More talking candlesticks, but with a very specific technological architecture in mind.


Recently, Gibson described things this way: cyberspace has everted. It has turned inside out, and we no longer go “into the network”.


Instead, the network has gone into us. Digital data and services are embedded in the fabric of the physical world.


Digital is emerging as a new dimension of reality, an integral property of the physical world. Length, width, height, time, digital.


Since we only have this world, It’s worth exploring the question of whether this is the kind of world we want to live in.


A good place to begin is with augmented reality, the idea that digital data and services are overlaid on the physical world in context, visible only when you look through the right kind of electronic window. Today that’s smartphones and tablets; at some point that might be through a heads-up display, the long-anticipated AR glasses.


Game designers are populating AR space around us with ghosts and zombies.


Geolocative data are being visualized in AR, like this crime database from SpotCrime.


History is implicit in our world; historical photos and media can make these stories explicit and visible, like this project on the Stanford University quad.


Here’s a 3D reconstruction, a simulation of the Berlin Wall as it ran through the city of Berlin.


Of course AR has been applied to a lot of brand marketing campaigns in the last year or two, like this holiday cups app from Starbucks.


AR is also being adopted by artists and culture jammers, in part as a way to reclaim visual space from the already pervasive brand encroachment we are familiar with.


We also have the Internet of Things, the notion that in just a few years there will be 20, 30, 50 billion devices connected to the Net. Companies like Cisco and Intel see huge commercial opportunities and a wide range of new applications.


An Internet of Things needs hyperlinks, and you can think of RFID tags, QR codes and the like as physical hyperlinks. You “click” on them  in some way, and they invoke a nominally relevant digital service.


RFID and NFC have seen significant uptake in transit and transportation. In London, your Will and Kate commemorative Oyster card is your ticket to ride the Underground. In Japan, your Octopus or Suica card not only lets you ride the trains, but also purchase items from vending machines and pay for your on-street parking. In California we have FasTrak for our cars, allowing automated payment at toll booths. These systems improve efficiency of the infrastructure sevices and provide convenience to citizens. However, they are also becoming control points for access to public resources, and vast amounts of data are generated and mined based on the digital footprints we leave behind.


Sensors are key to the IoT. Botanicalls is a product from a few years ago, a communicating moisture sensor for your houseplants. When the soil gets dry, the Botanicall sends you a tweet to let you know your plant is thirsty.


More recently, the EOS Talking Tree is an instrumented tree that has a Facebook page and a Twitter account with more than 4000 followers. That’s way more than me.


This little gadget is the Rymble, billed by its creators as an emotional Internet device. You connect it with your Facebook profile, and it responds to activity by spinning around, playing sounds and flashing lights in nominally meaningful ways. This is interesting; not only are physical things routinely connected to services, but services are sprouting physical manifestations.


This is a MEMS sensor, about 1mm across, an accelerometer & gyroscope that measures motion. If you have a smartphone or tablet, you have these inside to track the tilt, rotation and translation of the device. These chips are showing up in a lot of places.


Some of you probably have a FitBit, Nike+, FuelBand, WiThings scale. Welcome to the ‘quantified self’ movement. These devices sense your physical activity, your sleep and so on, and feed the data into services and dashboards. They can be useful, fun and motivating, but know also that your physical activities are being tracked, recorded, gamified, shared and monetized.


Insurance companies are now offering sensor modules you can install on your car. They will provide you with metered, pay-as-you-drive insurance, with variable pricing based on the risk of when, where and how safely you drive.


Green Goose wants you to brush your teeth. If you do a good job, you’ll get a nice badge.


How about the Internet of Babies? This is a real product, announced a couple of weeks ago at Mobile World Congress. Sensors inside the onesie detect baby’s motion and moisture content.


Here’s a different wearable concept from Philips Design, the Bubelle Dress that senses your mood and changes colors and light patterns in response.


So physical things, places and people are becoming gateways to services, and services are colonizing the physical world. Microsoft’s Kinect is a great example of a sensor that bridges physical and digital; the image is from a Kinect depth camera stream. This is how robots see us.


If a was a service, I think I’d choose some of these robots for my physical instantiation. You’ve probably seen these — DARPA’s Alpha Dog all-terrain robotic pack horse, DARPA’s robot hummingbird, Google’s self-driving cars. You might not think of cars as robots, but these are pretty much the same kinds of things.


Robots also come in swarms. This is a project called Electronic Countermeasures by Liam Young. A swarm of quadrotor drones forms a dynamic pirate wireless network, bringing connectivity to spaces where the network has failed or been jammed. When the police drones come to shoot them down, they disperse and re-form elsewhere in the city.


A team at Harvard is creating Robobees. This is a flat multilayer design that can be stamped out in volume. It is designed so that the robot bee pops up and folds like origami into the shape at top right. I wonder what kind of service wants to be a swarm of robotic bees?


On a larger scale, IBM wants to build you a smarter city. There are large smart city projects around the globe, being built by companies like IBM, Cisco and Siemens. They view the city as a collection of networks and systems – energy, utilities, transportation etc – to be measured, monitored, managed and optimized. Operational efficiency for the city, and convenience for citizens.


But we as individuals don’t experience the city as a stack of infrastructures to be managed. Here’s Italo Calvino in his lovely book Invisible Cities. “Cities, like dreams, are made of desires and fears…the thread of their discourse is secret, their rules absurd.”


Back at ground level in the not-so-smart city of today, displays are proliferating. Everywhere you turn, public screens are beaming messages from storefronts, billboards and elevators.


We’re getting close to the point where inexpensive, flexible plastic electronics and displays will be available. When that happens, every surface will be a potential site for displays.


We’re also seeing cameras becoming pervasive in public places. When you see a surveillance camera, do you think it’s being monitored by a security guard sitting in front of a bank of monitors as seen in so many movies? More likely, what’s behind the camera is a sophisticated computer vision system like this one from Quividi, that is constantly analyzing the scene to determine things like the gender, age and attention of people passing by.


A similar system from Intel called Cognovision is being used in a service called SceneTap, which monitors the activity in local nightclubs  to let you know where the hottest spots are at any given moment.


You’ve probably seen something like this. It’s worth remembering that our technologies are all too brittle, and you should expect to see more of this kind of less-than-graceful degradation.


In case the city isn’t big enough, IBM wants to bring us a smarter planet. HP wants to deploy a trillion sensors to create a central nervous system for the earth. “The planet will be instrumented, interconnected and intelligent. People want it.” But do we? Maybe yes, maybe no?


So we come back to the question, what kind of world do you want to live in? Almost everything I’ve talked about is happening today. The world is becoming digitally transformed through technology.


Many of these technologies hold great promise and will add tremendous value to our lives. But digital technology is not neutral — it has inherent affordances and biases that influence what gets built. These technologies are extremely good at concrete, objective tasks: calculating, connecting, distributing and storing, measuring and analyzing, transactions and notifications, control and optimization. So these are often fundamental characteristics of the systems that we see deployed; they reflect the materials from which they are made.


We are bringing the Internet into the physical world. Will the Internet of people, places and things be open like the Net, a commons for the good of all? Or will it be more like a collection of app stores? Will there be the physical equivalents of spam, cookies, click-wrap licensing and contextual advertising? Will Apple, Google, Facebook and Amazon own your pocket, your wallet and your identity?


And what about the abstract, subjective qualities that we value in our lives? Technology does not do empathy well. What about reflection, emotion, trust and nuance? What about beauty, grace and soul? In digital anima mundi?


In conclusion, I’d like to share two quotes. First, something Bruce Sterling said at an AR conference two years ago. You are the world’s first pure play experience designers. We are remaking our world, and this a very different sort of design than we are used to.


What it is, is up to us. Howard first said it more than 25 years ago, and it has never been more true than today.


I want to acknowledge these sources for many of the images herein.


toward human-centered mobility

Thursday, November 4th, 2010

zuckerberg-bi-250pxAt yesterday’s Facebook press event launching new mobile features, Mark Zuckerberg stirred up a minor tempest in the pundit-o-sphere. When asked about when there would be a Facebook mobile app for the iPad, he responded glibly:

“iPad’s not mobile. Next question.” [laughter] “It’s not mobile, it’s a computer.” (watch the video)

This of course spawned dozens of blog posts and hundreds of tweets, basically saying either “he’s nuts!” or “validates what I’ve said all along.” And in this post-PC connected world, it’s interesting that Zuckerberg sees a distinction between mobiles and computers. But as you might imagine, I have a somewhat different take on this: We need to stop thinking that ‘mobile’ is defined by boxes.

Boxes vs. Humans

The entire mobile industry is built around the idea that boxes — handsets, tablets, netbooks, e-readers and so on — are the defining entity of mobility. Apps run on boxes. Content is formatted and licensed for boxes. Websites are (sometimes) box-aware. Network services are provisioned and paid for on a box-by-box basis. And of course, we happily buy the latest shiny boxes from the box makers, so that we can carry them with us everywhere we go.

And there’s the thing: boxes aren’t mobile. Until we pick them up and take them with us, they just sit there. Mobility is not a fundamental property of devices; mobility is a fundamental property of us. We humans are what’s mobile — we walk, run, drive and fly, moving through space and also through the contexts of our lives. We live in houses and apartments and favelas, we go to offices and shops and cities and the wilderness, and we pass through interstitial spaces like airports and highways and bus stations. Humans are mobile; you know this intuitively. We move fluidly through the physical and social contexts of our lives, but our boxes are little silos of identity, apps, services and data, and our apps are even smaller silos inside the boxes. Closed, box-centric systems are the dominant model of the mobile industry, and this is only getting worse in the exploding diversity of the embedded, embodied, connected world.

So why doesn’t our technology support human-centered mobility?

One big reason is, it’s a hard problem to solve. Or rather, it’s a collection of hard problems that interact. A platform for human-centered mobility might have the following dimensions:

* Personal identity, credentials, access rights, context store, data store, and services you provide, independent of devices. Something like a mash-up of Facebook profiles, Connect, Groups, and Apps with Skype, Evernote and Dropbox, for example.

* Device & object identities, credentials, rights, relationships, data & services; a Facebook of Things, if you will.

* Place identities, credentials, rights, relationships, data & services. Think of this as a Future Foursquare that provides place-based services to the people and objects within.

* Device & service interaction models, such that devices you are carrying could discover and interact with your other devices/services, other people’s devices/services, and public devices/services in the local environment. For example, your iPod could spontaneously discover and act as a remote controller for your friend’s connected social TV when you are at her house, but your tweets sent via her TV would originate from your own account.

* Physical context models that provide raw sensor data (location, motion, time, temperature, biometrics, physiological data etc) and outputs of sensor fusion algorithms (“Gene’s phone is in his pocket [p=75%] in his car [p=100%] on Hwy 101 [p=95%] in Palo Alto, CA USA [p=98%] at 19.05 UTC [p=97%] on 02010 Nov 4 [p=100%]”).

* Social context models that map individuals and groups based on their relationships and memberships in various communities. Personal, family, friendship, professional and public spheres, is one way to think of this.

Each of these is a complex and difficult problem in its own right, and while we can see signs of progress, it is going to take quite a few years for much of this to reach maturity.

The second big reason is, human-centered mobility is not in the interests of the mobile boxes industry. The box makers are battling for architectural control, for developer mindshare, for unique and distinctive end user experiences. Network operators are fighting for subscribers, for value added services, for regulatory relaxation. Mobile content creators are fighting for their lives. And everyone is fighting for advertising revenue. Interoperability, open standards, and sharing data across devices, apps and services are given lip service, but only just barely. Nice ideas, but not good for business.

So here’s a closing thought for you. Maybe human-centered mobility won’t come from the mobile industry at all. Maybe, despite his kidding about the iPad, it will come from Mark Zuckerberg.  Maybe Facebook will be the mobile platform that transcends boxes and puts humans at the center. Wouldn’t that be interesting?

toward virtuosity, reflection and a conscious computing experience

Friday, October 22nd, 2010

@lindastone published this short post titled The Look & Feel of Conscious Computing, which I found compelling and resonant with thoughts that have been rattling around in my head for awhile:

“With a musical instrument, it’s awkward at first. All thumbs. Uncomfortable. Noise. With practice, instrument and musician become as one. Co-creating music. So it will be with personal technology. Now, a prosthetic of mind, it will become a prosthetic of being. A violinist with a violin. Us with our gadgets, embodied, attending as we choose.”

For context, Linda also pointed me toward another of her posts, A new era of post-productivity computing? where she closes with the question

“How do we usher in an era of Conscious Computing? What tools, technologies, and techniques will it take for personal technologies to become prosthetics of our full human potential?”

I’ve wrestled with similar questions in the past:

“In the arts, we speak of a talented and communicative practitioner as a virtuoso. The virtuous performer combines technical mastery of her medium with a great depth of human expressiveness, communicating with her audience at symbolic, intuitive and emotional levels. Can we imagine a similar kind of virtuosity of communication, applied to domains that are not traditionally considered art? Can we further make this possibility accessible to more people, allowing a richer level of discourse in the walks of everyday life?

“When groups of musicians play together, they establish communication channels among themselves through the give and take of listening and leading. Great ensemble players know how to establish a state of flow, a groove, where the music takes on a vitality and life of its own, greater than the sum of the individual rhythms, pitches and timbres. What are the conditions that make such a group ‘chemistry’ possible? Could we capture that essence and apply it to the work of organizations, the building of communities, the life of families?

“As information technologies increasingly become integral to our activities, the information we use, even to our ways of thinking and perceiving, we must confront some difficult, elusive notions about the relationships between people and their tools. For instance, in what sense can the technology enhance creative, playful thinking — are we having fun? What about beauty, inspiration, spirituality, mystery? These are qualities for which humans have striven over our entire history; shall we subjugate them in the name of efficiency, convenience and immediacy? Do the artifacts we make allow people space for reflection and insight, or merely add to the numbing cacophony of digital voices demanding our attention? Is it strange to ask such questions? Not at all. The economics of information technologies seem to dictate a future where more and more of our lives will be mediated by networks and interfaces and assorted other paraphernalia of progress. We must recognize the importance of such uniquely human concerns and integrate them into our vision, or risk further dehumanization in our already fractured society.”

Okay, that was from 1994, so where are we on this? I have to say, it seems like mainstream computing has advanced very little in these areas. Apple has good intentions, and the iPad actually does a nice job of getting out of the way, letting you interact directly and physically with individually embodied apps. It’s the best of the bunch, but the iPad is no violin, no instrument of human expression. Certainly the current crop of PCs, netbooks and phones are no better.

There are a few non-mainstream computing paradigms that give me hope for a conscious computing experience. The Nike+ running system, my favorite example of embodied ubi-media, creates an inherently physical experience augmented with media and social play. Nike+ doesn’t have a broadly expressive vocabulary, but it does bring your whole body into the equation and closes the feedback loop with contextually suitable music and audio prompts.

At its best, Twitter starts to feel like a global jam session between connected minds. The rapid fire exchange of ideas, the riffing off others’ posts, the flow of a well-curated stream can sometimes feel uniquely expressive. Yes, it is primarily a mental activity and mostly disembodied, but the occasional flashes of genuine group chemistry are wonderfully suggestive of the potential for an interconnected humanity.

For me, the most interesting possibilities arise from games. There are the obvious examples of physical interaction and expression that the Wii and Kinect deliver primarily for action games now, but with time a broader range of immersive and reflective experiences (is Wii Yoga any good?). I’m also thinking of the emerging genre of out-in-the-world games like SF0, SCVNGR and Top Secret Dance Off that send you on creative, social missions involving physicality, play, performance and discovery. Finally there is the next generation of “gameful” games, as proposed by Jane McGonigal:

What is Gameful?

We invented the word gameful! It means to have the spirit, or mindset, of a gamer: someone who is optimistic, curious, motivated, and always up for a tough challenge. It’s like the word “playful” — but gamier! Gameful games are games that have a positive impact on our real lives, or on the real world. They’re games that make us:

  • happier
  • smarter
  • stronger
  • healthier
  • more collaborative
  • more creative
  • better connected to our friends and family
  • more resilient
  • better problem-solvers
  • and better at WHATEVER we love to do when we’re not playing games.

I think the future of expressive, improvisational, conscious computing will be found at the intersection of personal sensing tools like Nike+ and Kinect, collective action tools like Twitter, and the playful engagement of gameful games. It won’t look like computing, and it won’t come in a box. It won’t be dumbed down for ‘ease of use’, it will be flowing experiences designed to make us more complex, capable and creative. It will augment our humanity, as embodied individuals embedded in a physical and social world.

Beyond Augmented Reality: Ubiquitous Media

Saturday, June 19th, 2010

Here are the slides I presented during my talk at ARE2010, the first Augmented Reality Event on June 3, 2010 in Santa Clara. Many thanks to all who attended, asked questions and gave feedback. For interested Bay Area folks, I will be organizing some face to face gatherings of the Ubiquitous Media Studio to explore the ideas raised here. The first one will be in July; follow @ubistudio on Twitter for further details.

will the HP Slate be a killer AR device?

Monday, April 5th, 2010
The HP Slate in live video mode

The HP Slate in live video mode

Augmented reality enthusiasts and developers got the shaft (again) from Apple when the iPad launched without an integrated camera, thus becoming a dead platform for AR purposes. Well it looks like the little  computer company on the other side of Hwy 280 might pull a little auggie magic out of their hat, just in time for the AR summer of love in Silicon Valley. HP has been teasing their forthcoming Slate for a few months, and they just posted another video clip that clearly shows live video from a forward-facing camera. We already know the slate will run Windows 7, and we have heard public rumblings about Android from various quarters, so it’s likely to be reasonably developer-friendly.

With the horsepower to run object recognition and tracking plus high quality 3D graphics, the Slate will definitely blur the line between webcam AR and mobile AR experiences. You know all those marker-based AR toys that feel so gimmicky when you have to use them in front of a PC with a webcam? I guarantee they are going to seem 1000% cooler when you pull out your Magic Internet Magnifying Glass and look through it into an alternate universe. And if the Slate ends up shipping with a GPS and digital compass, just watch all the mobile AR guys scrambling to learn Win7 and Silverlight. Oh yeah SLAR toolkit dude, better get a bigger server ;-)

the massively multiplayer magazine

Friday, February 19th, 2010

This idea is a quick brainstorming sketch that brings together several threads in the spirit of combinatorial innovation. I’d love to have your feedback in the comments.

The future of magazines in a connected world

I love magazines, and I’ll bet you do too. Magazines are perhaps the most vibrant and culturally relevant form of print media, and their diversity mirrors the staggering range of human interests and obsessions. In the connected world, they have the potential to evolve into an incredibly interesting and engaging networked medium. In recent months we have seen two inspiring future design concepts: the lushTime/Sports Illustrated video, and the poeticMag+ conceptcreated by design firm@BERGLondonand publisherBonnier. With high performance, connected e-reader and tablet platforms finally coming to market, it’s clear we are going to see some very exciting developments in this space. However, we should remember that the fundamental nature of connected media is very different from that of print media, and we should be careful about bringing a print-oriented mindset to a new networked medium. The features of electronic magazines should not simply be incremental extensions of the printed version, even if the physical artifacts are roughly similar in size, shape and appearance. With that in mind, I’d like to engage you in a thought experiment about what could happen when we collide digital magazines together with the global social Internet. One possibility we might imagine ismassively multiplayer magazines.


Massively multiplayer what?

You’re probably familiar with massively multiplayer online games like World of Warcraft. The massively multiplayer magazine re-imagines the traditional periodical in the context of an online social game environment involving thousands of people. In this scenario, the magazine becomes a gateway into a universe of intertwined stories, knowledge, people and play, with experience design and game mechanics drawn from MMOGs, ARGs and social games. Readers become players who have profiles, scores, achievements and abilities. Players self-assemble into clans, guilds and communities. Game elements encompass traditional magazine fare such as stories, images, features and advertisements, alongside new aspects including collaborative quests, mini-games, social streams, location awareness, augmented reality and physical hyperlinks. Gameplay involves completing missions, defeating bosses, unlocking hidden features, and participating in experiences that add richness, engagement and dimensionality to the magazine’s thematic center.

Magazines are like printed Usenet

Magazines are like printed Usenet

Magazines are like printed Usenet

You may be thinking this is a pretty strange idea, because magazines and online games seem like completely different media with little apparent synergy.  But consider: A well-stocked newsstand’s magazine rack is a glossy reflection of a world of enthusiast niches, each one incredibly narrow and deep. From heavy metal music to needlepoint; from luxury island living to body modification culture, magazines are the proto-Usenet of publishing. Furthermore, every special interest grouping you can imagine has already established itself in some form of online presence, be it a mailing list, web forum, or social network. The inherently social, topical milieu of magazines and their enthusiast communities has much in common with the ecosystem of social, story-oriented worlds and deeply invested players of many online games. It’s not hard to imagine that online gamers and magazine readers would each be attracted to a medium that combined the best of both genres. In fact, there is ample precedent for communities passionately following and participating in stories and games across multiple media. Think of Pokemon, Survivor, and the Star Wars Universe as examples of huge cultural phenomena with stories that span books, television, games, the web and more. For more on that, you might enjoy going down the deep rabbit hole of transmedia storytelling. But let’s continue.

Possible user stories

Clearly, this new kind of magazine/game would be primarily a digital medium. Mobile tablet computers like the just-announced Apple iPad would be excellent platforms to build optimized experiences around. Moreover, a massively multiplayer magazine would also play out across websites, social forums and physical locations, in much the same way that many alternate reality games have done. With the addition of ‘clickable’ links via QR codes and similar physical links, even readers of printed magazines could be drawn into the game through their mobile phones.

So what would a massively multiplayer magazine be like? Here are a few possible user stories that begin to explore the concept; you should definitely add your own ideas in the comments:

* Each story is a context that you “check into”, much like a Foursquare location. This might show up in your Twitter stream as “I’m reading <article> with 5 other people“, with a shortened link directly to the article. As you check in and comment about the article in your social stream, you accumulate points in your profile for each new reader that clicks through your link. If you are leafing through a paper copy of the magazine, you might find a QR code printed on the page, and scan it with your smartphone to “check in” and connect to the social stream about the article.

* Stories are customized based on your location. When you are physically in Paris, stories and games with a Parisian context are revealed. Reading those stories in their intended locations around the city earns you a special achievement badge for Paris. Meeting local players face to face grows your social circle and adds to your in-game reputation.

* A rock music magazine works with bands and concert promoters to place printed QR codes on posters at live shows, and readers earn badges by going to the show and scanning the codes.

* A pop culture fan magazine creates a series of 12 monthly challenges, each building on the previous one and taking players progressively deeper into a complex storyline. The challenges can only be worked out through large-scale cooperation by fans; the resolution leads to a hidden plot device in the upcoming season of a hit reality TV series.

* An advertiser sponsors a global treasure hunt, with rabbit holes, missions and puzzles embedded in the digital and print versions of a travel magazine. The prize is significant and the story engaging enough to attract tens of thousands of players and drive millions of social media mentions and impressions over the entire duration. For inspiration, take a look at the ARG calledPerplex City, which offered a $200,000 prize for finding a game artifact called the Receda Cube.

* Collaboratively generated story/soundtrack pairings are recommended by your friends and other readers of the same stories. “One of your friends recommended the Cowboy Junkies channel on Pandora, to accompany this story on musician Townes Van Zandt.” Alternatively, writers and photographers offer their own musical pairings to convey mood and contextual cues for their work, similar to sound design for cinema.

* A media literacy foundation challenges teams to create an entirely new magazine, organized around crowdsourced recommendations and contributions for the best stories, photographs, video, audio and even advertisements. The contributors earn achievements and reputation scores based on readers’ ratings and social metrics. The winning team receives a grant funding the creation of their next 6 issues, and featured placement on a popular media blog.

These are only a few examples of the possibilities of a new kind of massively multiplayer media. There are many open questions here, obviously. Would publishers find this concept attractive? Would readers make the leap to become players in a worldwide game? How hard would they be to develop [see note 2 below], and at what cost? It’s no sure thing, but I am inclined to believe that the well-established cultural familiarity and affection for magazines, combined with the addictive and viral nature of online games like Farmville and Foursquare, and built on the mobile, social, contextual platform of the connected world, would make an incredible creative genre and a very interesting business opportunity.

What do you think? Leave comments, send me email, or tweet some feedback to @genebecker. Thanks for reading, and YMMV as always.

[1] Photo credit: mannobhai

[2] It’s worth noting that designing media to be massively multiplayer will require new skills, tools and workflows, well beyond those employed in today’s magazine publishing ecosystem; our hypothetical project will surely need to address the authoring process. If you are interested, Ben Hammersley makes this point well in a series of eloquent posts that are worth your time.

augmentation overload

Friday, February 5th, 2010

Watching Keiichi Matsuda’s superb short video imagining a hyper-branded augmented reality environment, for some reason I was reminded of this classic bit of dystopia from PK Dick’s 1969 classic novel Ubik.

The door refused to open. It said, “Five cents, please.”

He searched his pockets. No more coins; nothing. “I’ll pay you tomorrow,” he told the door. Again he tried the knob. Again it remained locked tight. “What I pay you,” he informed it, “is in the nature of a gratuity; I don’t have to pay you.”

“I think otherwise,” the door said. “Look in the purchase contract you signed when you bought this conapt.”

In his desk drawer he found the contract; since signing it he had found it necessary to refer to the document many times. Sure enough; payment to his door for opening and shutting constituted a mandatory fee. Not a tip.

“You discover I’m right,” the door said. It sounded smug.

From the drawer beside the sink Joe Chip got a stainless steel knife; with it he began systematically to unscrew the bolt assembly of his apt’s money-gulping door.

“I’ll sue you,” the door said as the first screw fell out.

Joe Chip said, “I’ve never been sued by a door. But I guess I can live through it.”

Ubik, Philip K. Dick 1969

let’s bury the electronic newspaper

Friday, January 22nd, 2010

In technology, use model and interface metaphors exert a powerful influence on the rhetorical framing innovators adopt for their work. These metaphors can become entrenched schools of thought in product and experience design, making it difficult to imagine alternative approaches. Consider the longevity of the desktop metaphor for personal computing – it has been more than 35 years since the original Xerox PARC Alto, and we are still looking at desktops on many of our screens.

Similarly, the idea of electronic newspapers has been around since at least the 1970s, and now that the print newspaper industry is in dire straits we are seeing that notion thrown around rather more freqently. For example, LG Display is currently showing off an impressive lab prototype of a 19” flexible e-paper display that is 0.3mm thick and weighs just 4 ounces. The prototype measures 40x25cm or around 16×10 inches, making it about the size of a small tabloid newspaper. LG are touting it as “optimized for an e-newspaper and able to convey the feeling of reading an actual newspaper”


LG Display 19" e-paper prototype

As I see it, there’s a fundamental problem with attempts to transplant the design of physical newspapers into an “electronic newspaper” interaction metaphor. The design of print newspapers, and our interaction with them – where, when, what and how we read – arises from the intrinsic properties of the medium. The size of pages, the multicolumn tiled layout, the length of stories, the variety and separation of sections, advertising, subscriptions, deadlines, distribution, local geographic focus, national syndication, editorial viewpoint – all of these factors evolved to their current state largely due to the physical properties and economics of paper. When you remove the paper and substitute a dynamic networked display appliance, you have changed the underlying properties and constraints so radically that the entire newspaper metaphor collapses.

Furthermore, as Clay Shirky and many others have observed, the Internet has spawned a host of disaggregated alternatives to all of the major functions of print newspapers. From online news sites, blogs and social media to Craigslist, Google, and the Sunlight Foundation, we are evolving new structures, business logic and user experiences based on the properties and economics of connected world technologies. As Shirky wrote in March 2009:

Round and round this goes, with the people committed to saving newspapers demanding to know “If the old model is broken, what will work in its place?” To which the answer is: Nothing. Nothing will work. There is no general model for newspapers to replace the one the internet just broke.

With the old economics destroyed, organizational forms perfected for industrial production have to be replaced with structures optimized for digital data. It makes increasingly less sense even to talk about a publishing industry, because the core problem publishing solves — the incredible difficulty, complexity, and expense of making something available to the public — has stopped being a problem.

When I see electronic versions of print newspapers being sold for Amazon’s Kindle, demonstrated on the Plastic Logic QUE, and mocked up in demos like LG’s flexible plastic e-paper, I see designers and marketers indulging in nostalgia for a bygone era. Newspapers as we know them are pretty much dead. Let’s bury the electronic newspaper metaphor with them.



While researching this post I came across Hakon Lie’s 1990 MSc thesis from the MIT Media Lab, titled The Electronic Broadsheet – all the news that fits the display. Lie describes the design and implementation of a broadsheet-sized electronic newspaper on a large high resolution display. Although some of the leading edge technology from 1990 (pre-WWW, pre-flat panel monitors) seems quaint now, Lie’s overview of the Newspaper Metaphor remains relevant and worth reviewing. Lie sought to maintain the best qualities and practices of newspaper reading while augmenting them with the affordances of networked digital media, reifying the whole into a new kind of newspaper. At the time, he did not anticipate the breadth and depth of disruption that would begin in just a few years with the advent of the web. Of course Lie went on to develop CSS in 1994, demonstrating somewhat greater adaptability than the newspaper industry he sought to transform.

experience design for locative media & AR

Thursday, December 3rd, 2009

The next generation of mobile Augmented Reality applications will go well beyond simply overlaying points of interest, floating post-its and 3D models on the video display of your phone. Mobile AR is becoming a sophisticated medium for immersive games, situated storytelling, large-scale visualization, and artistic expression. The combination of physical presence, visual and audio media, sensor datastreams and social environments blended together with web services offers tremendous new creative possibilities. However, the design challenges of creating engaging, exciting and compelling experiences are quite significant. AR experience designers will draw fruitful inspiration and practical lessons from game design, 3D graphics, architecture and stagecraft, as well as the structure, linking, protocols and openness of the web.

Some of the best research to date on experience design for locative media experiences, was done at HP Labs as part of the Mobile Bristol collaboration. You might find these papers useful and applicable to AR design, as I have.

Technology Experiences: What Makes Them Compelling? Alison Kidd, 2000

Experience Design for Pervasive Computing, Richard Hull & Jo Reid

Experience Design Guidelines for Creating Situated Mediascapes, Reid et al, 2005

Magic Moments in Situated Mediascapes, Reid et al