Augmented Reality and Games

Although my schedule this semester allows me to work from home on Fridays, I decided to head into school for what may have been the last presentation given by candidates of a gaming faculty position here at Carleton (two of the other talks were covered here and here). This talk was given by Mark Fiala, currently of the NRC, regarding his vision of the future of augmented reality gaming.

Dr. Fiala began his talk with this loaded question: "Is gaming a valid research area?" While it seems strange to suggest that it is, it does in fact have a lot of value in the academic world. For instance, it does a good job of motivating students to learn computer science in the first place. But beyond that, one must consider the research results beyond their application to gaming. Techniques developed for the purpose of games have benefits in other areas as well, including medical computing, computer aided drawing, robotics, artificial intelligence, and even military training. Or, if that's not enough to convince, then think of the application of games as just another way to validate the theory.

In The World According To Mark, gaming will move more toward console applications and away from the PC. Multi-player games will gain momentum, and players will expect full mobility (including being able to play outdoors). And, the big one, is that augmented reality will play a major part in the games of the future. This means that it is an exciting time to be a researcher in computer vision in general and AR in particular. If one is lucky, he can ride the wave to where the future is probably headed anyway.

So what is augmented reality, anyway? Although I wrote a bit about augmented reality here, here's a little reminder: a bit like virtual reality, augmented reality mixes real video with computer generated objects and information. This could be seen as a new kind of improvement to the human-computer interface.

Back to games. There are two main paradigms for AR games. First, the magic lens makes use of a portable device with a front facing camera. You see the real world as per usual on the screen with some computer generated goodies added on top. Second, the magic mirror reflects footage of you back at you as if looking in a mirror. You might have some specially recognized tags strategically positioned on your body that can be used to augment your image with just about anything. Some examples shown included putting a scuba suit on someone and turning another person into a robot.

The magic lens concept was used in the popular exhibit that traveled (is still traveling?) across the US in various science museums called "Star Wars: Where Science Meets Imagination." The photo below shows a user interacting with the game, including a view of the augmented portion.

(Image from here)

From the OMSI web site describing the exhibits:

Building Communities and Augmented Reality. Visitors build a spaceport, moisture farm community and walled Jawa town. Placing cards on a table - the physical landscape - a computer superimposes a building on a site in virtual reality and real time.

This magic lens paradigm has a lot of potential, particularly for tabletop games. Action and strategy titles could be shipped with a specialized mat with the appropriate tag marker system to base the augmentations off of. These tags can even be hidden into game art if some cleverness is applied. Perhaps one day even World of Warcraft will be played as a tabletop game in 3D!

To take things to the next level, imagine playing Halo in real life. All you would need is a series of rooms or hallways marked up with tags, and some kind of head mounted camera and display unit. You would be able to see where obstacles were, though not necessarily as they are in the real world. It would be like laser tag, but with animated violence!

The final step, at least for now, would be to bring it all outside into the world's playground. Here, of course, there are no tags, so natural features are required. Extra sensors would be required, like GPS and orientation. Some computer vision magic would help pinpoint the exact location of a player once the GPS gives a rough enough idea to retrieve local context information.

I'm happy to call gaming a research area with all these exciting opportunities related to augmented reality. I guess we'll have to wait and see who gets hired for this new games faculty position, but if someone into AR and computer vision ends up on board, I will certainly be watching their research, and maybe even working with them one day!