Grad school's different from pretty much all school before it.

In high school, everything was very structured, with the same classes happening every day. You always knew what homework you had to do because you either had to hand it in the next day, or your teacher reminded you about it.

In university during undergrad, things are a little less structured than in high school, but it was still possible to keep track of everything. You could easily write down your assignments in a paper planner, and in computer science, there weren't usually many large term projects until fourth year. Even then, it was possible to finish in a few days at the end of the term (ahem ... you didn't hear it from me).

Then comes grad school. You can forget everything you used to know about time management, because while it might seem like it's working, it probably won't for long. Courses in grad school are really different from undergrad courses. There are still set assignments, but these assignments are usually more open ended and take longer to complete. Now you also have to read papers, give presentations, and do a big term project. These projects are often more research-oriented, and tend not to be the types you can finish up the night before.

I have always been very good at time management, thanks in part to my organizational skills. Things kind of went ok when I started my Masters, though I think I was thrown off by having fewer courses with more work each. But when it came time to direct my own research project... it got a lot harder. That's when things are really open ended, and it can be easy to lose track of yourself.

I did a lot of things wrong during my Masters, but lo and behold, I've learned from my mistakes. Here's how I've been keeping things straight so far in my PhD adventure (which, mind you, only began September 2009, but seems to be going much better so far).


(1) Carpool

When I drove myself to school, I often figured I'd go in after rush hour; after all, sitting in traffic for more than twice the normal off-peak commuting time actually really stinks. But if I do that, I may as well not show up to campus until my class actually starts. Then, to avoid the rush home, I may as well leave at 3pm. I could work from home before and after being on campus.

Except I don't.

I have no idea why, but this strategy seemed to lead to sleeping in, finding chores to do, generally being distracted... The amount of actual work I did wasn't what it could have been.

So this year, I started carpooling with my husband, since he works pretty close to campus. Now I have to get up, I have to be on campus all day, and I may as well do something useful while I'm stuck here. Oh, and I don't get to leave until 5:30pm. It's amazing how much my productivity has gone up - I even have weekends free (at least for now)!


(2) Keep a time sheet

This is going to sound more annoying than useful, but trust me. This works.

Tracking all your work hours can be an incredible way to make yourself accountable to... yourself. Oddly enough, even though nobody else will ever see how my time breaks down, I feel that I must push myself to meet my own goals in terms of time spent on certain activities. This can also mean not spending too much time on other things that are much more enjoyable than that annoying bug in the ol' research project (you know, like CU-WISE stuff - much more fun).

When I was trying to write my thesis for my Masters, I used a time sheet that tracked number of pages written rather than hours spent. I even used a formula in the spreadsheet to set a goal and see how many page I had to do that day to reach it. You can modify what you track to suit whatever it is you need to get done.

I currently use a Google Docs spreadsheet for my time sheet. This is the template that I copy every week and fill in. This is a nice easy-access, flexible solution. Give a try, and keep honest!


(3) Keep research organized

This has suddenly become even more important to me than ever. I'm trying to juggle a survey-based psychology class with a data structures class, finishing up a paper for my Masters research, and working on ideas for upcoming projects. If I can't keep it all straight, I'll spend more time trying to remember where I left off than moving forward.

I've written about research tools and organization here and here. My current setup involves Mendeley for reading papers and taking initial notes, a Google Notebook for a list of tools and interesting links (since Notebook is no longer supported, you might also try Delicious, which I'm starting to use now), and Google Docs for jotting down research ideas and keeping a dynamic to-do list.


There are just some of the things I do to manage my time effectively; what are some of your tricks?

There were many cool demos at ISMAR this year, and you can check out the description of them in the conference schedule. In this post, I've included a few photos and a video of some of my favourites. Click through the images to get a brief description. EDIT: You can also visit this blog post by Tom Carpenter to get a more detailed description of what you see below.





Object depth and shape extraction for Augmented Reality Interaction






Put a Spell: Learn to Spell with Augmented Reality






A Mixed Reality Painting Experience for Physical Rehabilitation







Computing Alpha Mattes in Real-time for Noisy Mixed Reality Video Sources






Augmented Reality Pop-Up Dollhouse




ProFORMA: Probabilistic Feature-based On-line Rapid Model Acquisition




Animatronic Shader Lamps Avatars

There were more than a few good papers presented at ISMAR this year on human factors and user interfaces. Here's just a taste of them. See the conference schedule for information about authors and their affiliations.


Using Augmented Reality to Support Cross-Organizational Collaboration in Dynamic Tasks

This student paper was an honourable mention for the best paper awards. It was all about a crisis management system designed for use by commanders with different backgrounds. Augmented reality is intended to give each user a personalized view that they can most easily understand based on their culture and so on.

The scenario used for the user study - the first such study for joint realtime operations - was planning the fight again forest fires. Rescue, police, and military helicopter units are all involved.

The initial brainstorming stage with field experts in these areas suggested that hand held displays should be used to give individualized views of a command map. But we all know how important it is to ask the real users what works best for them, not their managers; it turned out that the field workers couldn't use the handhelds. They were too clumsy and took away their ability to use their hands freely. They wanted a shared map that they could point to and have the others see. In other words, they wanted a heads up display with joystick control.

When compared with a paper based map, the AR system with custom markers for each type of field worker performed significantly better.


Interference Avoidance in Multi-User Handheld Augmented Reality

Have you ever wondered how safe multi-user augmented reality games really are? I mean, when you're competing furiously while looking through your mobile device, it seems like it'd be pretty easy to knock into each other as you move around in the virtual world in front of you, right? Well, trying to avoid this is what this paper is all about.

The concept is pretty simple. As you move closer to your opponent, the virtual objects in your view shift slightly away from them. The key is to make sure that you as a user don't notice thisvhappening, so certain compensations are needed, such as covering the playing surface with a flat texture that can also shift with the virtual objects.

The amazing thing is how effective this approach is compared to other proximity warnings, like dimming the screen, beeping, and disabling user actions when they get too close to another user. Users perceived the shifting to be less distracting but also less effective than other methods. However, the real distance maintained between players in a competitive two-player game was significantly better than the other methods, making it quite effective in reality.


Interaction and Presentation Techniques for Shake Menus in Tangible Augmented Reality

The investigation in this paper sought to find a way to interact with objects directly in the environment using some kind of menu system. Objects should not require any kind of tags or electronics added to them beforehand, and hands should be able to manipulate the object freely without having to pick up something else as well.

The idea of a shake menu was inspired by shaking a gift to see what's inside. So you shake an object to open a menu, and then move the object to the desired menu selection and hold it there to make the choice. But what's the best way to present the menu items in relation to the object?

A user study looked at a clipboard paradigm in which menu items (which look like cubes) are aligned along the right of the object, and "stick" to it as it moves around in the camera's view. Other layouts include aligning the choices surrounding the object (this seems very similar to the clipboard version), aligning relative to the display only (so it sticks to the screen and doesn't move again), and aligning to the world coordinated, but not the object's.

The hypothesis was that the object alignment would be the fastest and most intuitive, and would be appreciated for the ability to examine the menu choice from different angles (after all, it could be any 3D object). However, user studies proved this wrong. The object was almost tied with the display alignment for the best speed, but display had far fewer errors than any other method. The display choice was also the best in terms of perceived intuitiveness, with object in second place.

This was a great workshop given by Blair MacIntyre from Georgia Tech on mobile augmented reality games. I got a lot out of it, from being reminded of some solid game design topics to getting new ideas about the game I want to make for my PhD.

The goal of augmented reality in this context is to embody social interaction in the physical world, enabled by a tight integration of the physical and virtual world. In terms of games, it's important to remember that design is more than just form and function - it needs context, too (which AR can give). Game design is about solving a problem within a set of constraints, and making something fun, challenging, awe inspiring, and captivating.

In augmented reality, mobility is usually assumed. But it's not just a combination of the physical and virtual world - there should also be registration between the virtual and physical worlds and real time interaction.

It's worth remembering that there are two classes of AR systems: task based and experiential. Task-based AR is perhaps not as well suited to handhelds, since your hands aren't totally free to complete the task, and it's hard to hold something light up for a long time. This is one of the areas that give head mounted displays (HMDs) an advantage: they can provide zero-effort, hands-free interaction and continuous peripheral information. Both interfaces provide some privacy, an in-place display, and per-user customization.

So what makes "good" AR anyway? What is unique about it that can be leveraged? Multiple people can work in a shared space, for one. Each person gets a unique view of the world while not giving up the global perspective. It allows for direct and natural interaction, and the physical world can be leveraged with props, spatial understanding, and dexterity.

Some of the graphics issues to consider when determining a platform for your game (cell phone or something more advanced?) include lighting, shadows, occlusions, and physics capabilities. Graphics don't always have to be real, either - non-photorealistic effects can help alleviate the processing power needed. Remember that latency is a bigger issue with AR.

Back to game design. We, as computer scientists, have to think like a game designer when coming up with new ideas. AR games shouldn't be all about the technology, turning them into demos, essentially. We need to create something that's fun to play. "The designer needs to envision how a game will work during play ... planning everything necessary to create a compelling player experience." In other words, you need to decide first what you want the player to experience, not what they will do, or learn, or whatever. This is a key point for me in thinking about my educational game.

The structure of a game includes the following components:

• players
• objectives
• rules
• resources (making the game not too easy, not too hard)
• boundaries
• outcome

Games are dynamic systems. They have a 'magic circle' defined by the boundaries. Dramatic elements make it emotionally engaging. Design is about balancing all these elements.

Remember that to make a game something more than a toy, there must be goals, and interesting and meaningful choices to reach those goals. The story and characters are brought out through actions.

Some questions to ask when making a handheld AR game:

• Who is your target player?
• When or where are they playing?
• Single or groups?
• Will there be props? How comfortable and easy to use are they?
• What exactly will the player do while playing the game?

Some of the constraints to consider when designing a handheld AR game:

• Fast motions are a problem.
• How will having the device (phone) in the player's hand affect things?
• It's tiring to hold up a relatively light device for long stretches of time.
• Awareness of other players.
• Small screens are tiring to look at for a long time.
• Vibrations and sounds to give feedback, especially when looking elsewhere.

The future of mobile AR gaming looks pretty exciting. We first need to make some good progress on tracking, and when we do, there will be some amazing opportunities to be immersed into a terra-scaled world. Plus, we will be able to create more social experiences and casual games, possibly hooking into social networks. How will you contribute to this future?

This is the last of three posts on the ISMAR09 experiential learning workshop. Post one and post two covered the morning presentations on current applications, while this one will attempt to capture the excellent group discussion that took place in the afternoon.

The afternoon's format was to look at three main questions about education and augmented reality, each one building on the last. For each question, we broke ourselves into three groups, discussed the topic for 15 minutes (or more, in most cases), and then shared our thoughts with the group. My notes below will consist of our own group's findings first, which will naturally have more detail. Points from the others groups will follow - members of these groups are most definitely invited to add more insight or links to their own blog posts in the comments.


What are the Key Elements of Mixed and Augmented Reality that Create a Meaningful Experience?

I got this one started by explaining something I tell my friends and family when they want to know about augmented reality. I feel that one of the big benefits of AR is that you essentially reduce the number of levels of indirection required to do something. For example, consider a traditional map. You have a bird's eye, (usually) non-photorealistic view of the world that you must rotate and project onto the real world in front of you. What if that information was augmented for you in the first place? You can free up all that cognitive power for the actual task at hand (such as learning).

Another key element suggested was the idea that augmented reality should not provide the entire story - the imagination should have the ability to work its magic, too. You should also be able to bring in other senses beyond vision, making the presence of the physical world so important. Having an EyePet in a completely virtual world is somehow different than playing with it in your living room - in the latter case, the broader context of your own culture is included in the gameplay.

Augmented reality allows non-experts to participate in and understand tasks outside their field. For example, it seems unlikely that Disney could have succeeded in getting permission to build Disney World here in Orlando today. But if the city council (or whoever needed to vote) were able to see with their own eyes exactly how it would all look, and how, say, emergency evacuations would work, things might be different.

Our group also believed that the most meaningful experiences would come from free-range AR, where much larger environments can become immersive sandboxes for learning. This setup could also lead to a more social experience.

Another key point was on the adaptability of software. Ideally, AR programs would learn you as you learned them. Of course, this requires much more advanced artificial intelligence than what is available today, but we do get better all the time in mimicking this ability.

Finally, we decided that AR would be most meaningful when it was personalized. This refers to not just the changing viewpoint of the virtual objects, but also the content of the virtual portion of the environment itself. This, among other things, will help avoid information overload.

Points from other groups:

• AR needs to be consistent with what's expected in the real world (it has to "make sense").
• There must be an element of surprise and magic.
• It should be social, approachable, and easy to use.
• Users should enjoy being tricked/surprised.
• The end user experience is key (not the technology itself).
• There should be some degree of being novel or special.
• It should be scalable in terms of time, space, size, and orientation.
• It will provide the ability to experiment where it was once impossible.
• It must be reliable enough to reflect realism.

How Do We Continue the Learning Experience Once the User Leaves

The first example our group discussed was the idea of capturing information about the experience that can then be used later in various ways. For instance, a military training exercise might record the decisions made for a particular scenario, and the user can bring that home and show his or her family what they experienced. They can compare their stats to others who have done the same scenario, and so on. The question then becomes: what is the best way to present the data? Whatever it is, it shouldn't replace the original experience. Otherwise, there's no reason to use the augmented reality again (or, for instance, no reason to go to a museum again).


An interesting discussion started about whether doing a good enough job in creating the experience is enough to spark interest in a certain topic such that the user will go home and learn more about it. The example of the Louvre was that most visitors look at art for only 30 seconds or so, when you need at least two full minutes to fully appreciate the details. If the proper viewing was encouraged by the AR experience, perhaps this is enough to want to find out more. What if the Mona Lisa had an augmentation of da Vinci putting on the finishing touches after acting out some story related to life in that era? Would you be more inclined to find out more about da Vinci?

Finally, we felt it was key to avoid making it about the technology - the tech needs to be invisible. This way, the focus will be on the topic at hand, which again will make for an easier transition to, say, a follow up activity to be done at home.

Points from other groups:

• A museum exhibit can have a take-home piece so the adventure can be continued (for example, your own fish from the main giant fish tank exhibit). The individual experience is sparked thanks to the larger context of the exhibit.
  
• Make the follow-up activity viral. Share with friends and family.
• Allow learners to finish the story at home when they run out of time.
• Provide networking opportunities online.
• Create physical activities later on.

What is Novel When it Comes to Augmented Reality and Learning?

We agreed that augmented reality isn't a new paradigm shift, but rather another tool in a teacher's toolkit. However, this tool might benefit a teacher in many ways. For instance, it may be easier to employ than other computer-based demonstrations if it's as easy to use as we insisted it be in the earlier questions. Furthermore, the exploratory nature makes for an environment that allows a teacher to say "I don't know, let's find out," avoiding the fear of teaching a topic they don't understand well themselves. Finally, it might be something that much better than just Googling a topic because it would certainly be more immersive.

Other advantages of AR in the classroom are that it would be more repeatable than more free-form techniques, making it possible to standardize the content (though not the experiences) of AR scenarios across the board.

It may also open up opportunities for standardized learning at home. This might help capture the attention of the gifted students and help the struggling students catch up. It would even be possible to have distributed study groups who could interact with the same virtual object.

In a training context, augmented and mixed reality has already proven to be very effective. Apparently many commercial pilots take their first flight in a real jet because the simulators are just that good.

Thinking more simply to see what could be done now, it's clear that printed material can be augmented with markers and cell phones used to view them (and kids would love getting permission to pull out their phones in class!).

Points from other groups:

• Will AR be a revolution or just an evolution? Can we truly improve learning with AR? Perhaps we won't truly know for another few decades.
• AR provides a different dimension related to creativity and self-reflection. It can be about exploration, not necessarily just making abstract concepts concrete.
• Main barrier: How will it improve peoples' lives? We just don't know - there is a lack of understanding that won't be solved until we start getting more products in peoples' hands.
• What are trying to accomplish with AR? Connection, relevance, and perspective? How?

Conclusion

That concludes the workshop on experiential learning. I will be taking away the excellent thoughts and insights from the three posts on this blog, as well as a better appreciation for the big picture. I hate to admit it, but when thinking about the game I want to build for my PhD research, I got stuck in thinking of a basic marker based interaction. There's so much more to AR that it would be tragic to miss considering it all.

This is the second of three posts on the experiential learning workshop held Monday at ISMAR09. The first post introduced the topic and summarized the first three presentations given in the morning on current AR applications. This post will summarize the last three speakers, and the last post will be on the group discussion held in the afternoon.


Infinite Story, Finite Space

Chris Stapleton, co-chair of ISMAR09, gave us his vision for augmented reality and told us about the projects he's worked on. He says "we think that if we deal with physical space, we can only deal with one story." But if the augmentations can change, this is no longer true. Using augmented reality, we can allow users to add their imagination, rather than just give them the story - imagination is the third reality.

A project that really intrigued me was a memory scape for the Maitland Holocaust Museum. The idea was to recreate stories told in children's diaries of the Holocaust so visitors could understand what happened in terms of humanity. A physical space would be created, and embedded projections used to bring the space to life. Bits and pieces of the story can be told through these augmentations, and imagination can fill in the rest. Even better, you could experience a different story each time you visited the museum.

Chris goes on to lay out the spectrum of levels of engagement:

• Passive : Absorb the media (TV)
• Engaging : Think and feel (film)
• Active : Participate (amusement park rides)
• Reactive : Choose (games)
• Interactive : Contribute (Second Life)
• Experiential : Live (enhanced media? augmented reality?)

Finally, he notes that we should bring problems from the community to academia, where prototypes will be built and passed on to industry. Industry will then create products, and complete the cycle by passing them on to the community.


Total Immersion

Unfortunately I didn't catch the presenter's name (wasn't on slides, didn't see it in the schedule), but he was from Total Immersion. He pointed out that 20-30% of the population are auditory learners, 40% are visual, and 30-40% are kinaesthetic. Naturally, augmented reality helps the kinaesthetic learning type unlike many other mediums. Experiences will allow for engagement, reflection, insight, and of course, learning.


Also touched on was a set of best practices for augmented reality. It needs to attract users, be easy to use, and give instant access to the experience.

Finally, a comparison between entertainment and education was made to highlight some of the differences.

Entertainment:

• Audience: Groups
• Duration: Quick
• Engagement: Immediate gratification
• Outcome: Something immersive or magical

Education:

• Audience: Small groups or individuals
• Duration: Long term engagement
• Engagement: Deep exploration
• Outcome: A new visual perspective on topic of study
  

Museum Learning and AR

Last but not least, we have Kate Haley Goldman from the Institute for Learning Innovation. She's all about informal and free choice learning, creating voluntary, non-sequential learning experiences like Wolf Quest. Though Wolf Quest is not a space-based game, it's interesting nonetheless for its great success; kids played it much longer than the expected two to three hours it was designed for.


Kate explained that personal, sociocultural, and physical context are all factors that help influence learning. These are all things that can be employed in, say, museum exhibits. But why do people visit museums? Research has shown that reasons range from fun and entertainment, to social activity, to being a site of interest (a 'must-see' while on vacation), or even specifically to be challenged or learn something. How might augmented reality help with all this?

One project Kate talked about was an AR system that augmented the wing of a plane. Visitors could adjust various settings or move the wing, and see the resulting forces in the augmentation. They could then figure out whether the plane would actually fly under those conditions. This system helped with some of the above goals (especially learning, as tests showed), but not all. For example, the system was too separated and thus lacked the social aspect of a museum visit. Users couldn't really talk to or interact with each other.

Some of the aspirations of augmented reality that Kate mentioned were:

• creating conversations
• making the abstract tangible
• helping visualize change
• adding sensory capabilities
• supporting critical thinking skills
• ability to act like a scientist (collect data, form and test theories)

These ideas along with those of the other five presenters are all very insightful, and give me much excitement about the future of AR - both in terms of learning and in general!

The International Symposium on Mixed and Augmented Reality (ISMAR) held workshops today, and I was lucky enough to be assigned as student volunteer to exactly the workshop I wanted to attend.

From the program (though it was referred to as experiential learning on the conference signs):

Falling in Love with Learning: Education and Entertainment Converge with Learning Landscapes is designed to meet the needs of people who are currently designing memorable and lasting experiences for visitors and students through AR technology. These include professionals in the areas of:

• cultural heritage preservation
• education and in-situ learning
• entertainment and games for learning
• museum curation and design

The leaders of this workshop will discuss how they are currently using Mixed and Augmented Reality for education and entertainment and the challenges they face or most wish to tackle in the future.

First Post of Three

This is the first of three posts covering this workshop. Here, I will summarize the first three of six presentations given in the morning by those already using augmented reality for their particular purposes. In the next post, I will cover the remaining talks. Finally, the third post will cover the afternoon's discussions that sought to answer three main questions about augmented reality's place in education.


What Is Augmented Reality?

If you really want to know, check out the Wikipedia article. The points mentioned before the six presentations began include:

• AR gives context to the situation. It's not an out-of-body experience or a separate thing from the world we know.
• Blends the real and the synthetic.
• When the technology disappears, the imagination is enhanced.
• Involves multiple senses.
• Can record experiences in detail (such as high scores, stress of learners, etc).

What's Happening at UCF

Eileen Smith, director at the Institute for Simulation and Training at the University of Central Florida, spoke first, telling us about some of the projects surrounding experiential learning going on at UCF. Some examples include informal learning at museums, teacher training, recreating the World Fair, and military training.


One of the most interesting and unique uses of AR was, for me, the green kitchen. This is a reconfigurable set of cabinetry that can be arranged as per anyone's kitchen. Someone requiring cognitive rehabilitation can then wear a head mounted display and see what looks a lot like their own house, and then practice performing simple tasks like making cereal.

Another neat project was Journey With the Sea Creatures. A magic window into a fossil exhibit that would otherwise never change made the museum worth visiting more than once. This particular program filled the room with water and brought in the amazing creatures alive many years ago. Apparently once the children discovered this feature, they would go back into the main exhibit area and start swimming around for their friends and family to see on the magic window.

Eileen closed with a suggestion on when to use augmented reality. Don't use it when the real world will do just fine (in other words, if you can just do what you are trying to simulate, why bother with the simulation?). Instead, employ AR when you want to explore space, time, and scale, or to collect data you can then use or display to others later.


Museum Exploration, DNP Digitalcom

Next up was Tsutomu Miyashita from DNP Digitalcom [Japanese]. He discussed the AR projects intended for use in the Louvre to encourage better appreciation of art by visitors, and route guidance.


His group wanted to use markerless tracking at first, since they felt that the 2D bar codes would probably detract from the art itself, not being terribly attractive. Visitors using this technology were surprised and gleeful, but because they were not familiar with the concept of AR, they did not use it as expected. Furthermore, the weight and battery life of the devices used were a problem. (Something that may not be as important in research, but crucial in the real world!)

The next iteration used cell phones and markers instead. In the interface, a computer animated character taught users how to view art and properly appreciate it in addition to showing them where to go next. They understood the marker-based system much better, and the system also performed better in terms of recognition accuracy.

The key takeaway was that users feel surprised when they see augmented reality for the first time, leading to strong attention. But if they don't really know how to use it, then engaging them is really important so that they actually want to figure it out. Finally, once their attention is obtained, retention, understanding, and satisfaction become the aim.


EyePet

Istvan Siklossy spoke next, mainly showing us the new EyePet game for Playstation 3. He explains that in camera based games, you typically see yourself and use motions and gestures to interact. Player actions generally map to the game action, making the games accessible to everyone.


In EyePet, an adorable creature comes to life on your living room floor. Your interaction with it, which occurs by gestures as well as with a special marker, is robust and responsive. It's quite impressive! To get the robust tracking even in low-lighting (noisy images), the group took the usual tracking algorithms and made some improvements, such as rapid multiple thresholding to find many contours and locate the marker. It's crucial in the skill-based games that the tracking accuracy is no less than excellent.

In terms of learning environments, the EyePet allows for experimentation in that some basic sketches drawn by players are interpreted and transformed into toys for the pet. Players learn how the pet reacts, get a personalized experience, and have an opportunity to record and share videos of their experience.