Monday, January 21, 2008

Augmented Reality and Games

Although my schedule this semester allows me to work from home on Fridays, I decided to head into school for what may have been the last presentation given by candidates of a gaming faculty position here at Carleton (two of the other talks were covered here and here). This talk was given by Mark Fiala, currently of the NRC, regarding his vision of the future of augmented reality gaming.

Dr. Fiala began his talk with this loaded question: "Is gaming a valid research area?" While it seems strange to suggest that it is, it does in fact have a lot of value in the academic world. For instance, it does a good job of motivating students to learn computer science in the first place. But beyond that, one must consider the research results beyond their application to gaming. Techniques developed for the purpose of games have benefits in other areas as well, including medical computing, computer aided drawing, robotics, artificial intelligence, and even military training. Or, if that's not enough to convince, then think of the application of games as just another way to validate the theory.

In The World According To Mark, gaming will move more toward console applications and away from the PC. Multi-player games will gain momentum, and players will expect full mobility (including being able to play outdoors). And, the big one, is that augmented reality will play a major part in the games of the future. This means that it is an exciting time to be a researcher in computer vision in general and AR in particular. If one is lucky, he can ride the wave to where the future is probably headed anyway.

So what is augmented reality, anyway? Although I wrote a bit about augmented reality here, here's a little reminder: a bit like virtual reality, augmented reality mixes real video with computer generated objects and information. This could be seen as a new kind of improvement to the human-computer interface.

Back to games. There are two main paradigms for AR games. First, the magic lens makes use of a portable device with a front facing camera. You see the real world as per usual on the screen with some computer generated goodies added on top. Second, the magic mirror reflects footage of you back at you as if looking in a mirror. You might have some specially recognized tags strategically positioned on your body that can be used to augment your image with just about anything. Some examples shown included putting a scuba suit on someone and turning another person into a robot.

The magic lens concept was used in the popular exhibit that traveled (is still traveling?) across the US in various science museums called "Star Wars: Where Science Meets Imagination." The photo below shows a user interacting with the game, including a view of the augmented portion.

(Image from here)

From the OMSI web site describing the exhibits:
Building Communities and Augmented Reality. Visitors build a spaceport, moisture farm community and walled Jawa town. Placing cards on a table - the physical landscape - a computer superimposes a building on a site in virtual reality and real time.
This magic lens paradigm has a lot of potential, particularly for tabletop games. Action and strategy titles could be shipped with a specialized mat with the appropriate tag marker system to base the augmentations off of. These tags can even be hidden into game art if some cleverness is applied. Perhaps one day even World of Warcraft will be played as a tabletop game in 3D!

To take things to the next level, imagine playing Halo in real life. All you would need is a series of rooms or hallways marked up with tags, and some kind of head mounted camera and display unit. You would be able to see where obstacles were, though not necessarily as they are in the real world. It would be like laser tag, but with animated violence!

The final step, at least for now, would be to bring it all outside into the world's playground. Here, of course, there are no tags, so natural features are required. Extra sensors would be required, like GPS and orientation. Some computer vision magic would help pinpoint the exact location of a player once the GPS gives a rough enough idea to retrieve local context information.

I'm happy to call gaming a research area with all these exciting opportunities related to augmented reality. I guess we'll have to wait and see who gets hired for this new games faculty position, but if someone into AR and computer vision ends up on board, I will certainly be watching their research, and maybe even working with them one day!


Haz said...

I'd still say all the things you listed are research areas in their own right and Game Development is only an extra application. Mind you, great applications for a research area are always good...typically means $$.

On that note, can AR do the following:

Imagine a cube with 2d bit vector markers (typical AR style) on 5 of the 6 faces. A stick connected to the unmarked face. If I wield it around in front of a webcam with AR technology behind it, can the system detect the source/orientation of the hilt? If so, how hard is that to implement?

...and no, this isn't for a sword fighting game ;).

fbeeper said...

You must know that a big amount of money on R&D comes from military areas, but these R&D results are something that we can't see :P

On the "real world" (calling "real world" to non military areas :P) we can see apps related with Games and those can attract so much to us.

But if you can, i encourage you to follow searching more AR articles and you will see that gaming is one of a big mount of apps related with AR... I'm not disregarding gaming, but you'll see that other apps are impressive!

See you :)

Sorry about my poor English, I'm learning yet :P

Note: AR isn't only synthetic objects on video, AR is real world with synthetic objects on it (you must consider see-thu head mounted displays too :P)

Haz said...

Gail said...

Haz, on the question of your (not) sword fighting game, the answer is a resounding yes! In fact this idea was included in Mark's examples - one scenario has the cube as a sword, and the other as a shark to go with the scuba suit. Check out Mark's upcoming book here, which should give you enough code and info to set it all up for yourself!

As for the 3D mouse, someone presented that in a class recently (perhaps the exact same thing, perhaps something related). I thought it was neat but in reality wasn't sure how practical it would be yet - for example, I recall that scrolling was pretty impractical and slow using this method.

On all the other stuff regarding research, great comments. At this point I'm just repeating more or less what was discussed in the talk, and haven't formed a really strong opinion either way. To me, the important thing is how much fun research related to games can be ;)

Gail said...

PS to Haz: I think this is the stuff of the future for great museum exhibits - tell K! Together we'd make a great team! ;)

Gail said...

Whoops, correction on the mouse thing (didn't look closely enough): the talk I saw was about using your cell phone as a mouse for the cell phone rather than the desktop. This does look intriguing...

Post a Comment

Comments are moderated - please be patient while I approve yours.