ISMAR09: A Few Demos

There were many cool demos at ISMAR this year, and you can check out the description of them in the conference schedule. In this post, I've included a few photos and a video of some of my favourites. Click through the images to get a brief description. EDIT: You can also visit this blog post by Tom Carpenter to get a more detailed description of what you see below.





Object depth and shape extraction for Augmented Reality Interaction






Put a Spell: Learn to Spell with Augmented Reality






A Mixed Reality Painting Experience for Physical Rehabilitation







Computing Alpha Mattes in Real-time for Noisy Mixed Reality Video Sources






ProFORMA: Probabilistic Feature-based On-line Rapid Model Acquisition




Animatronic Shader Lamps Avatars

ISMAR09: Human Factors and User Interfaces

There were more than a few good papers presented at ISMAR this year on human factors and user interfaces. Here's just a taste of them. See the conference schedule for information about authors and their affiliations.


Using Augmented Reality to Support Cross-Organizational Collaboration in Dynamic Tasks

This student paper was an honourable mention for the best paper awards. It was all about a crisis management system designed for use by commanders with different backgrounds. Augmented reality is intended to give each user a personalized view that they can most easily understand based on their culture and so on.

The scenario used for the user study - the first such study for joint realtime operations - was planning the fight again forest fires. Rescue, police, and military helicopter units are all involved.

The initial brainstorming stage with field experts in these areas suggested that hand held displays should be used to give individualized views of a command map. But we all know how important it is to ask the real users what works best for them, not their managers; it turned out that the field workers couldn't use the handhelds. They were too clumsy and took away their ability to use their hands freely. They wanted a shared map that they could point to and have the others see. In other words, they wanted a heads up display with joystick control.

When compared with a paper based map, the AR system with custom markers for each type of field worker performed significantly better.


Interference Avoidance in Multi-User Handheld Augmented Reality

Have you ever wondered how safe multi-user augmented reality games really are? I mean, when you're competing furiously while looking through your mobile device, it seems like it'd be pretty easy to knock into each other as you move around in the virtual world in front of you, right? Well, trying to avoid this is what this paper is all about.

The concept is pretty simple. As you move closer to your opponent, the virtual objects in your view shift slightly away from them. The key is to make sure that you as a user don't notice thisvhappening, so certain compensations are needed, such as covering the playing surface with a flat texture that can also shift with the virtual objects.

The amazing thing is how effective this approach is compared to other proximity warnings, like dimming the screen, beeping, and disabling user actions when they get too close to another user. Users perceived the shifting to be less distracting but also less effective than other methods. However, the real distance maintained between players in a competitive two-player game was significantly better than the other methods, making it quite effective in reality.


Interaction and Presentation Techniques for Shake Menus in Tangible Augmented Reality

The investigation in this paper sought to find a way to interact with objects directly in the environment using some kind of menu system. Objects should not require any kind of tags or electronics added to them beforehand, and hands should be able to manipulate the object freely without having to pick up something else as well.

The idea of a shake menu was inspired by shaking a gift to see what's inside. So you shake an object to open a menu, and then move the object to the desired menu selection and hold it there to make the choice. But what's the best way to present the menu items in relation to the object?

A user study looked at a clipboard paradigm in which menu items (which look like cubes) are aligned along the right of the object, and "stick" to it as it moves around in the camera's view. Other layouts include aligning the choices surrounding the object (this seems very similar to the clipboard version), aligning relative to the display only (so it sticks to the screen and doesn't move again), and aligning to the world coordinated, but not the object's.

The hypothesis was that the object alignment would be the fastest and most intuitive, and would be appreciated for the ability to examine the menu choice from different angles (after all, it could be any 3D object). However, user studies proved this wrong. The object was almost tied with the display alignment for the best speed, but display had far fewer errors than any other method. The display choice was also the best in terms of perceived intuitiveness, with object in second place.

ISMAR09: Workshop on Handheld Augmented Reality Games

This was a great workshop given by Blair MacIntyre from Georgia Tech on mobile augmented reality games. I got a lot out of it, from being reminded of some solid game design topics to getting new ideas about the game I want to make for my PhD.

The goal of augmented reality in this context is to embody social interaction in the physical world, enabled by a tight integration of the physical and virtual world. In terms of games, it's important to remember that design is more than just form and function - it needs context, too (which AR can give). Game design is about solving a problem within a set of constraints, and making something fun, challenging, awe inspiring, and captivating.

In augmented reality, mobility is usually assumed. But it's not just a combination of the physical and virtual world - there should also be registration between the virtual and physical worlds and real time interaction.

It's worth remembering that there are two classes of AR systems: task based and experiential. Task-based AR is perhaps not as well suited to handhelds, since your hands aren't totally free to complete the task, and it's hard to hold something light up for a long time. This is one of the areas that give head mounted displays (HMDs) an advantage: they can provide zero-effort, hands-free interaction and continuous peripheral information. Both interfaces provide some privacy, an in-place display, and per-user customization.

So what makes "good" AR anyway? What is unique about it that can be leveraged? Multiple people can work in a shared space, for one. Each person gets a unique view of the world while not giving up the global perspective. It allows for direct and natural interaction, and the physical world can be leveraged with props, spatial understanding, and dexterity.

Some of the graphics issues to consider when determining a platform for your game (cell phone or something more advanced?) include lighting, shadows, occlusions, and physics capabilities. Graphics don't always have to be real, either - non-photorealistic effects can help alleviate the processing power needed. Remember that latency is a bigger issue with AR.

Back to game design. We, as computer scientists, have to think like a game designer when coming up with new ideas. AR games shouldn't be all about the technology, turning them into demos, essentially. We need to create something that's fun to play. "The designer needs to envision how a game will work during play ... planning everything necessary to create a compelling player experience." In other words, you need to decide first what you want the player to experience, not what they will do, or learn, or whatever. This is a key point for me in thinking about my educational game.

The structure of a game includes the following components:

• players
• objectives
• rules
• resources (making the game not too easy, not too hard)
• boundaries
• outcome

Games are dynamic systems. They have a 'magic circle' defined by the boundaries. Dramatic elements make it emotionally engaging. Design is about balancing all these elements.

Remember that to make a game something more than a toy, there must be goals, and interesting and meaningful choices to reach those goals. The story and characters are brought out through actions.

Some questions to ask when making a handheld AR game:

• Who is your target player?
• When or where are they playing?
• Single or groups?
• Will there be props? How comfortable and easy to use are they?
• What exactly will the player do while playing the game?

Some of the constraints to consider when designing a handheld AR game:

• Fast motions are a problem.
• How will having the device (phone) in the player's hand affect things?
• It's tiring to hold up a relatively light device for long stretches of time.
• Awareness of other players.
• Small screens are tiring to look at for a long time.
• Vibrations and sounds to give feedback, especially when looking elsewhere.

The future of mobile AR gaming looks pretty exciting. We first need to make some good progress on tracking, and when we do, there will be some amazing opportunities to be immersed into a terra-scaled world. Plus, we will be able to create more social experiences and casual games, possibly hooking into social networks. How will you contribute to this future?

ISMAR09: Experiential Learning 3 of 3 - Group Discussion

This is the last of three posts on the ISMAR09 experiential learning workshop. Post one and post two covered the morning presentations on current applications, while this one will attempt to capture the excellent group discussion that took place in the afternoon.

The afternoon's format was to look at three main questions about education and augmented reality, each one building on the last. For each question, we broke ourselves into three groups, discussed the topic for 15 minutes (or more, in most cases), and then shared our thoughts with the group. My notes below will consist of our own group's findings first, which will naturally have more detail. Points from the others groups will follow - members of these groups are most definitely invited to add more insight or links to their own blog posts in the comments.


What are the Key Elements of Mixed and Augmented Reality that Create a Meaningful Experience?

I got this one started by explaining something I tell my friends and family when they want to know about augmented reality. I feel that one of the big benefits of AR is that you essentially reduce the number of levels of indirection required to do something. For example, consider a traditional map. You have a bird's eye, (usually) non-photorealistic view of the world that you must rotate and project onto the real world in front of you. What if that information was augmented for you in the first place? You can free up all that cognitive power for the actual task at hand (such as learning).

Another key element suggested was the idea that augmented reality should not provide the entire story - the imagination should have the ability to work its magic, too. You should also be able to bring in other senses beyond vision, making the presence of the physical world so important. Having an EyePet in a completely virtual world is somehow different than playing with it in your living room - in the latter case, the broader context of your own culture is included in the gameplay.

Augmented reality allows non-experts to participate in and understand tasks outside their field. For example, it seems unlikely that Disney could have succeeded in getting permission to build Disney World here in Orlando today. But if the city council (or whoever needed to vote) were able to see with their own eyes exactly how it would all look, and how, say, emergency evacuations would work, things might be different.

Our group also believed that the most meaningful experiences would come from free-range AR, where much larger environments can become immersive sandboxes for learning. This setup could also lead to a more social experience.

Another key point was on the adaptability of software. Ideally, AR programs would learn you as you learned them. Of course, this requires much more advanced artificial intelligence than what is available today, but we do get better all the time in mimicking this ability.

Finally, we decided that AR would be most meaningful when it was personalized. This refers to not just the changing viewpoint of the virtual objects, but also the content of the virtual portion of the environment itself. This, among other things, will help avoid information overload.

Points from other groups:

• AR needs to be consistent with what's expected in the real world (it has to "make sense").
• There must be an element of surprise and magic.
• It should be social, approachable, and easy to use.
• Users should enjoy being tricked/surprised.
• The end user experience is key (not the technology itself).
• There should be some degree of being novel or special.
• It should be scalable in terms of time, space, size, and orientation.
• It will provide the ability to experiment where it was once impossible.
• It must be reliable enough to reflect realism.

How Do We Continue the Learning Experience Once the User Leaves

The first example our group discussed was the idea of capturing information about the experience that can then be used later in various ways. For instance, a military training exercise might record the decisions made for a particular scenario, and the user can bring that home and show his or her family what they experienced. They can compare their stats to others who have done the same scenario, and so on. The question then becomes: what is the best way to present the data? Whatever it is, it shouldn't replace the original experience. Otherwise, there's no reason to use the augmented reality again (or, for instance, no reason to go to a museum again).

An interesting discussion started about whether doing a good enough job in creating the experience is enough to spark interest in a certain topic such that the user will go home and learn more about it. The example of the Louvre was that most visitors look at art for only 30 seconds or so, when you need at least two full minutes to fully appreciate the details. If the proper viewing was encouraged by the AR experience, perhaps this is enough to want to find out more. What if the Mona Lisa had an augmentation of da Vinci putting on the finishing touches after acting out some story related to life in that era? Would you be more inclined to find out more about da Vinci?

Finally, we felt it was key to avoid making it about the technology - the tech needs to be invisible. This way, the focus will be on the topic at hand, which again will make for an easier transition to, say, a follow up activity to be done at home.

Points from other groups:

• A museum exhibit can have a take-home piece so the adventure can be continued (for example, your own fish from the main giant fish tank exhibit). The individual experience is sparked thanks to the larger context of the exhibit.
  
• Make the follow-up activity viral. Share with friends and family.
• Allow learners to finish the story at home when they run out of time.
• Provide networking opportunities online.
• Create physical activities later on.

What is Novel When it Comes to Augmented Reality and Learning?

We agreed that augmented reality isn't a new paradigm shift, but rather another tool in a teacher's toolkit. However, this tool might benefit a teacher in many ways. For instance, it may be easier to employ than other computer-based demonstrations if it's as easy to use as we insisted it be in the earlier questions. Furthermore, the exploratory nature makes for an environment that allows a teacher to say "I don't know, let's find out," avoiding the fear of teaching a topic they don't understand well themselves. Finally, it might be something that much better than just Googling a topic because it would certainly be more immersive.

Other advantages of AR in the classroom are that it would be more repeatable than more free-form techniques, making it possible to standardize the content (though not the experiences) of AR scenarios across the board.

It may also open up opportunities for standardized learning at home. This might help capture the attention of the gifted students and help the struggling students catch up. It would even be possible to have distributed study groups who could interact with the same virtual object.

In a training context, augmented and mixed reality has already proven to be very effective. Apparently many commercial pilots take their first flight in a real jet because the simulators are just that good.

Thinking more simply to see what could be done now, it's clear that printed material can be augmented with markers and cell phones used to view them (and kids would love getting permission to pull out their phones in class!).

Points from other groups:

• Will AR be a revolution or just an evolution? Can we truly improve learning with AR? Perhaps we won't truly know for another few decades.
• AR provides a different dimension related to creativity and self-reflection. It can be about exploration, not necessarily just making abstract concepts concrete.
• Main barrier: How will it improve peoples' lives? We just don't know - there is a lack of understanding that won't be solved until we start getting more products in peoples' hands.
• What are trying to accomplish with AR? Connection, relevance, and perspective? How?

Conclusion

That concludes the workshop on experiential learning. I will be taking away the excellent thoughts and insights from the three posts on this blog, as well as a better appreciation for the big picture. I hate to admit it, but when thinking about the game I want to build for my PhD research, I got stuck in thinking of a basic marker based interaction. There's so much more to AR that it would be tragic to miss considering it all.

ISMAR09: Experiential Learning 2 of 3 - Current Applications 2

This is the second of three posts on the experiential learning workshop held Monday at ISMAR09. The first post introduced the topic and summarized the first three presentations given in the morning on current AR applications. This post will summarize the last three speakers, and the last post will be on the group discussion held in the afternoon.


Infinite Story, Finite Space

Chris Stapleton, co-chair of ISMAR09, gave us his vision for augmented reality and told us about the projects he's worked on. He says "we think that if we deal with physical space, we can only deal with one story." But if the augmentations can change, this is no longer true. Using augmented reality, we can allow users to add their imagination, rather than just give them the story - imagination is the third reality.

A project that really intrigued me was a memory scape for the Maitland Holocaust Museum. The idea was to recreate stories told in children's diaries of the Holocaust so visitors could understand what happened in terms of humanity. A physical space would be created, and embedded projections used to bring the space to life. Bits and pieces of the story can be told through these augmentations, and imagination can fill in the rest. Even better, you could experience a different story each time you visited the museum.

Chris goes on to lay out the spectrum of levels of engagement:

• Passive : Absorb the media (TV)
• Engaging : Think and feel (film)
• Active : Participate (amusement park rides)
• Reactive : Choose (games)
• Interactive : Contribute (Second Life)
• Experiential : Live (enhanced media? augmented reality?)

Finally, he notes that we should bring problems from the community to academia, where prototypes will be built and passed on to industry. Industry will then create products, and complete the cycle by passing them on to the community.


Total Immersion

Unfortunately I didn't catch the presenter's name (wasn't on slides, didn't see it in the schedule), but he was from Total Immersion. He pointed out that 20-30% of the population are auditory learners, 40% are visual, and 30-40% are kinaesthetic. Naturally, augmented reality helps the kinaesthetic learning type unlike many other mediums. Experiences will allow for engagement, reflection, insight, and of course, learning.

Also touched on was a set of best practices for augmented reality. It needs to attract users, be easy to use, and give instant access to the experience.

Finally, a comparison between entertainment and education was made to highlight some of the differences.

Entertainment:

• Audience: Groups
• Duration: Quick
• Engagement: Immediate gratification
• Outcome: Something immersive or magical

Education:

• Audience: Small groups or individuals
• Duration: Long term engagement
• Engagement: Deep exploration
• Outcome: A new visual perspective on topic of study
  

Museum Learning and AR

Last but not least, we have Kate Haley Goldman from the Institute for Learning Innovation. She's all about informal and free choice learning, creating voluntary, non-sequential learning experiences like Wolf Quest. Though Wolf Quest is not a space-based game, it's interesting nonetheless for its great success; kids played it much longer than the expected two to three hours it was designed for.

Kate explained that personal, sociocultural, and physical context are all factors that help influence learning. These are all things that can be employed in, say, museum exhibits. But why do people visit museums? Research has shown that reasons range from fun and entertainment, to social activity, to being a site of interest (a 'must-see' while on vacation), or even specifically to be challenged or learn something. How might augmented reality help with all this?

One project Kate talked about was an AR system that augmented the wing of a plane. Visitors could adjust various settings or move the wing, and see the resulting forces in the augmentation. They could then figure out whether the plane would actually fly under those conditions. This system helped with some of the above goals (especially learning, as tests showed), but not all. For example, the system was too separated and thus lacked the social aspect of a museum visit. Users couldn't really talk to or interact with each other.

Some of the aspirations of augmented reality that Kate mentioned were:

• creating conversations
• making the abstract tangible
• helping visualize change
• adding sensory capabilities
• supporting critical thinking skills
• ability to act like a scientist (collect data, form and test theories)

These ideas along with those of the other five presenters are all very insightful, and give me much excitement about the future of AR - both in terms of learning and in general!