Larry and Mark weren’t physically at Where 2.0, but their perceived presence loomed large over the proceedings. Those are clever mashups on the Obey Giant theme as well; what are they trying to say here?

At Wherecamp on the Stanford campus, locative social media were very much in evidence. Here, camp organizer @anselm and AR developer @pmark were spotted in physical/digital space.

The freespace cabal apparently thought the geo community would be receptive to their work, although it seemed some of the messages were aimed at a different audience. The detention of Chinese artist Ai Wei Wei is a charged topic, certainly.

So you’ll note that although these are all screenshots from the AR view in Layar, I’m referring to them as photographs in their own right. It’s a subtle shift, but an interesting one. For me, this new perspective is driven by several factors: the emergence of visually interesting and contextually relevant AR content, the idea that AR objects are vectors for targeted messages, and the new screenshot and share functions which make Layar seem more like a social digital camera app. I’m finding myself actively composing AR photos, and thinking about what content I could create that would make good AR pictures other people would want to take. Oh, and that awkward AR holding-your-phone-up gesture? I’m taking pictures, what could be more natural?

AR photography feels like it might be important. What do you think?

It started when I found @jewelia‘s Instagram pic from the Where 2.0 stage in the new Instagram AR layer in Layar. I took a screenshot:

and shared it on Twitter:

A bit later, I saw my tweet in the TweepsAround layer, and I took a screenshot:

and shared that one to Twitter too:

Then Christopher @endurablegoods got in on the fun:

Of course that was bait, so I snapped a photo in Color:

and shared it on Twitter:

But Christopher was not to be outdone:

And in the end:

We live in interesting times.

In my recent Ignite talk Hijacking the Here and Now: Adventures in Augmented Reality, I showed examples of how creative people are using AR in ways that modify our perceptions about time and space. Now, Ignite talks are only 5 minutes long and I think this is a big idea that’s worth a deeper look. So here’s my claim: I assert that one of the most natural and important uses of AR as a creative medium is hacking space and time to explore and make sense of the emerging physical+digital world.

When you look at who the true AR enthusiasts are, who is doing the cutting edge creative work in AR today, it’s artists, activists and digital humanities geeks. Their projects explore and challenge the ideas of ownership and exclusivity of physical space, and the flowing irreversibility of time. They are starting to see AR as the emergence of a new construction of reality, where the physical and digital are no longer distinct but instead are irreversibly blended. Artist Sander Veenhof is attracted to the “infinite dimensions” of AR. Stanford Knight Fellow Adriano Farano sees AR ushering in an era of “multi-layer journalism”. Archivist Rick Prelinger says “History should be like air,” immersive, omnipresent and free. And in their recent paper Augmented Reality and the Museum Experience, Schavemaker et al write:

In the 21st century the media are going ambient. TV, as Anna McCarthy pointed out in Ambient Television (2001), started this great escape from domesticity via the manifold urban screens and the endless flat screens in shops and public transportation. Currently the Internet is going through a similar phase as GPS technology and our mobile devices offer via the digital highway a move from the purely virtual domain to the ‘real’ world. We can collect our data everywhere we desire, and thus at any given moment transform the world around us into a sort of media hybrid, or ‘augmented reality’. [emphasis mine]

When the team behind PhillyHistory.org augments the city of Philadelphia with nearly 90,000 historical photographs in AR, they are actively modifying our experience of the city’s space and connecting us to moments in time long past. With its ambitious scope and scale, this seems a particularly apt example of transforming the world into a media hybrid.

In their AR piece US/Iraq War Memorial, artists Mark Skwarek and John Craig Freeman transpose the locative datascape of casualties in the Iraq War from Wikileaks onto the northeastern United States, with the location of Baghdad mapped onto the coordinates of Washington DC. In addition to spatial hackery evocative of Situationist psychogeographic play, this work makes a strong political statement about control of information, nationalist perspectives and the cultural abstraction of war.

Now let’s talk about this word, ‘hacking’. Actually, you’ll note that I used the term ‘hijacking’ as well, so let’s include that too. My intent is to evoke the tension of multiple meanings: Hacking in the sense of gaining deep understanding and mastery of a system in order to modify and improve it, and as a visible demonstration of a high degree of proficiency. Also, hacking in the sense of making unauthorized intrusions into a system, including both white hat and black hat variations. I use ‘hijacking’ in the sense of a mock takeover, like the Black Eyed Peas playfully hijacking the myspace.com website for publicity purposes, but also hijacking as an antagonistic, possibly malign, and potentially unlawful attack. In the physical+digital augmented world, I expect we will see a wide variety of hacking and hijacking behaviors, with both positive and negative effects. For example, in Skwarek’s piece with Joseph Hocking, the leak in your hometown, the corporate logo of BP becomes the trigger for an animated re-creation of the iconic broken pipe at the Macondo wellhead, spewing AR oil into your location. It is possible to see this as an inspired spatial hack and a biting social commentary, but I have no doubt BP executives would consider it a hijacking of their brand in the worst way.

In his book Smart Things, ubicomp experience designer Mike Kuniavsky asks us to think of digital media about physical entities as ‘information shadows’; I believe the work of these AR pioneers points us toward a future where digital information is not a subordinate ‘shadow’ of the physical, but rather a first-class element of our experience of the world. Even at this early stage in the development of the underlying technology, AR is a consequential medium of expression that is being used to tell meaningful stories, make critical statements, and explore the new dimensionality of a blended physical+digital world. Something important is happening here, and hacking space and time through AR is how we’re going to understand and make sense of it.

I paraphrase, obviously. But seriously, did you read Ray Ozzie’s Dawn of a New Day? It’s his manifesto for the post-PC era, and a poignant farewell letter to Microsoft executives as he unwinds himself from the company. In Ozzie’s post, frequent readers of this space will recognize what I’ve been calling ‘the new revolution in personal computing’, the rise of a connected world of mobile, embedded and ubiquitous devices, services, sensors & actuators, and contextual transmedia; a physical, social, immersive Internet of People, Places & Things.

“All these new services will be cloud-centric ‘continuous services’ built in a way that we can all rely upon.  As such, cloud computing will become pervasive for developers and IT – a shift that’ll catalyze the transformation of infrastructure, systems & business processes across all major organizations worldwide.  And all these new services will work hand-in-hand with an unimaginably fascinating world of devices-to-come.  Today’s PC’s, phones & pads are just the very beginning; we’ll see decades to come of incredible innovation from which will emerge all sorts of ‘connected companions’ that we’ll wear, we’ll carry, we’ll use on our desks & walls and the environment all around us.  Service-connected devices going far beyond just the ‘screen, keyboard and mouse’:  humanly-natural ‘conscious’ devices that’ll see, recognize, hear & listen to you and what’s around you, that’ll feel your touch and gestures and movement, that’ll detect your proximity to others; that’ll sense your location, direction, altitude, temperature, heartbeat & health.”

- Ray Ozzie, Dawn of a New Day

Frankly, there’s nothing especially surprising about this vision of the future; many of us (including Gates and Ozzie) have been working toward similar ideas for at least 20 years. Former HP Labs head Joel Birnbaum was predicting a world of appliance/utility computing (pdf) in the ’90s. I’m sure that many of these ideas are actively being researched in Microsoft’s own labs.

What I find really interesting is that Ozzie is speaking to (and for) Microsoft, one of the largest companies in tech and also the one company that stands to be most transformed and disrupted by the future he describes. He’s giving them a wake-up call, and letting them know that no matter how disruptive the last 5 years may have seemed to the core Windows and Office franchises, despite the wrenching transition to a web-centric world, the future is here and you ain’t seen nothing yet.

And now at “the dawn of a new day – the sun having now arisen on a world of continuous services and connected devices”, Ray Ozzie is riding off into the sunset. I don’t see how that can be interpreted as a good sign.

(photo credit: WIRED)

Innovation is happening at every level of personal systems, from processor architecture and devices to social media and advertising. The very idea of a personal system is broadening rapidly to encompass mobile, embedded and cloud systems, identity, context and the physical world.

In core hardware, Qualcomm’s Snapdragon platform garnered significant design wins at HP, Google and many others, while Apple has developed their own ARM-based A4 system on a chip. Nvidia also apparently gained support for their new Tegra2 mobile processor.

In system software, Google released the open source Chromium OS while their Android platform continued to gather design wins. Microsoft announced a completely redesigned Windows Phone 7 OS, Intel and Nokia merged Moblin and Maemo into the MeeGo linux platform, Symbian3 launched, and Apple extended their iPhone OS to a major new platform.

In devices, Apple announced the iPad, Google jumped into the hardware business with the Nexus One, HP and others previewed tablet PCs, and a stack of new E-readers launched from Barnes & Noble, Hearst, Plastic Logic, and more.

Application ecosystems continued to heat up as Apple’s AppStore crossed 100,000 apps and 3 billion downloads. Intel introduced the AppUp store for netbooks, and a broad consortium of carriers and device makers launched the oddly named Wholesale Applications Community for mobile apps.

The social media frenzy continued unabated, with Facebook hitting 400 million users (third in country population behind China and India!), Twitter passing 1 billion tweets per month, and Google’s Buzz launching like a rocket with millions of users before running into a buzzsaw of criticism for their tone-deaf approach to privacy and usability. A recent analysis showed Facebook driving more traffic to major web destinations than Google, signalling a dramatic shift from organic search to friend recommendations for finding information online.

Google acquired AdMob while Apple bought Quattro Wireless, pointing to a major battle for mobile advertising as well as a very provocative business model play for Apple.

Mobile social location-based gamers Foursquare, a favorite of the early-adopter tribe, inked deals with major media properties including Bravo TV, Conde Nast’s Lucky Magazine, Zagat guides, HBO, Warner and the New York Times.

The race to capture, index and augment the physical world further intensified. Microsoft’s Bing Maps and Google’s Street View each showed major new features, including integrating users’ photographs seamlessly into their visual canvases. Street View now has capture operations in 30 countries on 6 continents, and they are managing a fast-growing multi-petabyte store of image and lidar data (1 PB = 1 million GB). Meanwhile NYC startup Everyscape raised $6M from SK Telecom to expand their real-world capture into Asia, and SF-based Earthmine opened their high-resolution 3D city point cloud database to developers.

Google also released Goggles, a mobile app for Android devices that provides visual recognition, identification, OCR and search for physical world objects such as books, products, and landmarks. Nokia began a pilot of their mobile Point & Find service with bus shelter advertising in Colchester UK. Augmented reality startup Layar added $3.4M in funding and a global mobile phone distribution deal, signalling growing commercial interest in overlaying the real world with digital media and experiences.

In the realm of open innovation we saw grass-roots networks mount a groundswell of response to the disastrous earthquake in Haiti. Open source platform Ushahidi, mapping and geoweb experts from Open Street Map, and hackers at worldwide self-organizing Crisis Camps provided tools and expertise to support a wide range of relief efforts on the ground in Port au Prince.

Lastly, in two fascinating signs that the future is upon us, HP announced that it was getting into 3D printers through a deal with Stratasys, while San Diego outfit Organovo announced the first commercial 3D bio-printer for manufacturing human tissue and organs. It really doesn’t get much more personal than that.

In the 40-plus years since Douglas Engelbart created the mother of all demos, the personal computer has fundamentally transformed the way we work, play, create, communicate, shop, learn and live. Now we find ourselves at the cusp of a new revolution, where personal computing is no longer synonymous with the personal computer. The new personal computing is mobile, embedded, networked, virtual, social, contextual, wearable and physical. And it’s here. Are you ready?

Readme

Daniel Suarez’ novels Daemon and Freedom (TM), and Cory Doctorow’s Little Brother and Makers are four excellent near-future science fiction novels that I recommend highly. Both Suarez and Doctorow are savvy observers of today’s high tech scene, and they use their knowledge of technology to extrapolate our common experience of the Internet and personal computing into imaginative and entertaining stories of the future to come.

(The links above are to Amazon; if you buy there I receive a small commission which I donate to a reputable charity. You can also download Makers for free from Cory’s site. Little Brother too).

GOTO

O’Reilly’s Where 2.0 (3/30 – 4/1, 2010 in San Jose, CA) is the best conference to intersect with experts in mapping, mobile social location services, geoweb, GIS, and more. Also don’t miss the open unconference WhereCamp SF 2010 on April 3&4 hosted by Google.

New thinktank Council have declared April 9th Global Internet of Things Day, an unstructured, self-organizing event aimed at discussion of the notion of an Internet of Things. If you’re in Silicon Valley that day, I’m organizing an informal workshop focused on the Internet of People, Places and Things. If interested, ping me on twitter or email info@lightninglaboratories.com.

If you’re in Europe in May, Lift10 will convene a delightful community of future thinkers with a definite slant toward humanistic design. Geneva, May 5-7, 2010.

Augmented Reality Event 2010 is an industry conference about, well, augmented reality. It runs June 2-3, 2010 at the Santa Clara Convention Center in Silicon Valley. The always provocative Bruce Sterling will keynote. I’ll be speaking there, along with a very bright roster of technical and business folks. If you’re going, get in touch and we’ll find the best parties together.

Get Your Game On

EVOKE is a game about learning to change the world through social innovation. It was developed by the World Bank Institute, the learning and knowledge arm of the World Bank Group, and directed by alternate reality game master Jane McGonigal.

EVOKE is free to play and open to anyone, anywhere. The game begins on March 3, 2010, and players can join the game at any time. Players who successfully complete ten game challenges in ten weeks will be able to claim their honors: Certified World Bank Institute Social Innovator – Class of 2010. Top players will also earn online mentorships with experienced social innovators and business leaders from around the world, and scholarships to share their vision for the future at the EVOKE Summit in Washington DC.

I’m part of the team helping to run the game, along with some truly amazing people from around the world. I hope you’ll check it out, sign up to play, and see firsthand how games might just change the world.

Similarly, the idea of electronic newspapers has been around since at least the 1970s, and now that the print newspaper industry is in dire straits we are seeing that notion thrown around rather more freqently. For example, LG Display is currently showing off an impressive lab prototype of a 19” flexible e-paper display that is 0.3mm thick and weighs just 4 ounces. The prototype measures 40x25cm or around 16×10 inches, making it about the size of a small tabloid newspaper. LG are touting it as “optimized for an e-newspaper and able to convey the feeling of reading an actual newspaper”

LG Display 19" e-paper prototype

As I see it, there’s a fundamental problem with attempts to transplant the design of physical newspapers into an “electronic newspaper” interaction metaphor. The design of print newspapers, and our interaction with them – where, when, what and how we read – arises from the intrinsic properties of the medium. The size of pages, the multicolumn tiled layout, the length of stories, the variety and separation of sections, advertising, subscriptions, deadlines, distribution, local geographic focus, national syndication, editorial viewpoint – all of these factors evolved to their current state largely due to the physical properties and economics of paper. When you remove the paper and substitute a dynamic networked display appliance, you have changed the underlying properties and constraints so radically that the entire newspaper metaphor collapses.

Furthermore, as Clay Shirky and many others have observed, the Internet has spawned a host of disaggregated alternatives to all of the major functions of print newspapers. From online news sites, blogs and social media to Craigslist, Google, spot.us and the Sunlight Foundation, we are evolving new structures, business logic and user experiences based on the properties and economics of connected world technologies. As Shirky wrote in March 2009:

Round and round this goes, with the people committed to saving newspapers demanding to know “If the old model is broken, what will work in its place?” To which the answer is: Nothing. Nothing will work. There is no general model for newspapers to replace the one the internet just broke.

With the old economics destroyed, organizational forms perfected for industrial production have to be replaced with structures optimized for digital data. It makes increasingly less sense even to talk about a publishing industry, because the core problem publishing solves — the incredible difficulty, complexity, and expense of making something available to the public — has stopped being a problem.

When I see electronic versions of print newspapers being sold for Amazon’s Kindle, demonstrated on the Plastic Logic QUE, and mocked up in demos like LG’s flexible plastic e-paper, I see designers and marketers indulging in nostalgia for a bygone era. Newspapers as we know them are pretty much dead. Let’s bury the electronic newspaper metaphor with them.

Postscript:

While researching this post I came across Hakon Lie’s 1990 MSc thesis from the MIT Media Lab, titled The Electronic Broadsheet – all the news that fits the display. Lie describes the design and implementation of a broadsheet-sized electronic newspaper on a large high resolution display. Although some of the leading edge technology from 1990 (pre-WWW, pre-flat panel monitors) seems quaint now, Lie’s overview of the Newspaper Metaphor remains relevant and worth reviewing. Lie sought to maintain the best qualities and practices of newspaper reading while augmenting them with the affordances of networked digital media, reifying the whole into a new kind of newspaper. At the time, he did not anticipate the breadth and depth of disruption that would begin in just a few years with the advent of the web. Of course Lie went on to develop CSS in 1994, demonstrating somewhat greater adaptability than the newspaper industry he sought to transform.

Geographic AR Points

Geographic AR systems like Layar, Geovector, Wikitude, Robotvision, Gamaray etc, use a spheroid-based coordinate system of latitude, longitude and (sometimes) altitude to specify the point locations of the observer and georeferenced content. POIs (points of interest) consisting of a single (lat,lon,alt) coordinate tuple plus various metadata, are commonly used to represent physical entities such as restaurants, monuments and attractions. Unfortunately even in this extremely simple case, there is no agreement on specifications for a single point in space. For example, if altitude is used, is it the height of the point above the topographic surface at that location, the height above the observer’s location, or the height above the WGS-84 reference ellipsoid approximating mean sea level, as a GPS would measure it? Does a point also have accuracy metrics? And what metadata are required or optional for each point?  Each of the companies mentioned above is doing something a bit different, and so are their upstream POI data providers. So far, and despite recent announcements, openness is not really happening yet.

3D AR Points

AR has its roots in computer graphics & vision technologies, and these approaches primarily use 3D cartesian (xyz) coordinate systems. A 3D model of a teapot might have a local xyz coordinate system; the teapot rests on a 3D model of a table which in turn has its own reference coordinate system; the observer of the scene has their own reference coordinate system; the screen that the scene is displayed on has its own 2D pixel coordinates, and a set of mathematical transformations (e.g., translation, scaling, rotation & projection) ties them all together. A 3D graphics scene is not inherently tied to any physical world reference point; in marker-based AR, the fiducial marker provides an anchor that binds the 3D augmented scene to a physical world location. However, the data structure for the scene’s location is entirely relative, which makes the location of 3D models fairly portable.

Simple Geo + 3D AR

Of course, one simple and obvious thing we want is to enable 3D graphics models to be placed in geographic locations. If we truly think open AR is important, we are going to want to agree on which kinds of coordinate systems to use. This is not a trivial question. Do we want the 3D model to be on a local or global coordinate system? A fixed position relative to the world and regardless of viewpoint, or always located relative to the observer? What if the model and the observer are on boats? What if the model is something like an entire city? Different choices for coordinate systems and schema will impact computational costs and accuracy. In Google Earth, KML allows use of static COLLADA models which are then imported/transformed to the GE geographic coodinate system. Planet9′s virtual cities have a single reference coordinate system for the entire city, and use UTM WGS-84 in order to keep their building models square. The Web3D Consortium’s X3D framework supports georeferencing models in geodetic, UTM and geocentric reference frames, appropriate for a variety of use cases. What approach(es) makes sense for mobile AR? Can we leverage & extend existing standards, or will we have to create new ones from the ground up?

Start simple, but start now

Okay, so clearly things can get messy, even for the simple case of specifying a point in space. And it is also clear that multiple constituencies are going to be very interested in the geographic and 3D graphic aspects of AR. I think it’s time to have serious discussions about open standards for mobile AR, starting with the basic question of representing POIs and static 3D objects. I realize it is hard for small, fast moving teams to spend precious energy on this kind of discussion, but to me it seems like a critical thing for the community to establish a common foundation for the mobile AR experience. Do you agree? If not, why not? If so, then where should this discussion happen and who should be involved? Perhaps the recently formed AR Consortium can play a role here? Maybe it is already happening somewhere?

I’m very interested in your thoughts on this topic. Please share in the comments below, link here from your own blog, or respond @genebecker. YMMV as always.

For further reading

* Augmented Reality Should Be Open by Joe Ludwig
* Augmented Reality: Open, Closed, Walled or What? by Robert Rice
* Wikitude API
* Layar API
* Gamaray formats
* Garmin GPX POI schema
* WGS-84
* UTM
* A Discussion of Various Measures of Altitude
* GeoRSS
* GeoJSON
* W3C Geolocation API
* KML
* COLLADA
* X3D
* CityGML
* OGC GML

This journal offers disruptive ideas, spirited perspectives and tools to think with. My goals are to help you make sense of the emerging landscape, engage you in an ongoing conversation about technology, strategy and society, and work with you to envision and build the kind of future we actually want to live in. Please explore the archives, immerse yourself in new ideas, and share your reactions in the comments.