Learning Computer Science and Algorithms with Augmented Reality

This is the third and last post in a three-part series of design proposals for augmented reality learning applications. These are from a paper I wrote in my computers and cognition class. I'll be reworking the main ideas in the paper for a future submission, but probably won't include these, so I figured I'd share!

Algorithms -- lists of well defined steps that can be used to solve a particular type of problem -- are a fundamental part of computer science education, but the concepts can be abstract and are consequently not always easy to teach. Various applications and techniques have been proposed to aid this process. For instance, the SHALEX system was built to be adaptable to a learner's behaviour, providing spatial and temporal links, textual descriptions of algorithms, and supplemental animations. This kind of system still presents the algorithms in their abstract form, though, and does not take advantage of situated, embedded, or embodied cognition theory.

Situating algorithm instruction in the context of real and specific problems that can be solved with particular techniques can be helpful to students in terms of motivation and understanding, which has lead to some instructors to advocate the use of puzzles. Taking this idea even further, kinaesthetic activities have been proposed in the place of paper-based puzzles. For instance, Sivilotti and Pike described activities that can be used in upper level undergraduate and even graduate courses in computer science to teach algorithms for distributed computing, and Snapp showed how graph algorithms could be introduced to beginners using a corn maze. The CS Unplugged set of activities is designed for use with elementary and middle school students, and contains a wide range of activities. Most are situated to a specific real world problem, and there is a mix of paper-based exploration and group kinaesthetic activities.

While these examples do take advantage of the aforementioned cognitive theories, there are still reasons to consider implementing the activities using augmented reality. For instance, the paper-based activities might be brought to life, or corn mazes brought to the classroom or school yard to avoid the need for a field trip. Some of the group activities might be adapted for use by one person for independent study. Having any activity driven by software would allow for more complex examples, and teachers who aren't as familiar with computer science are more likely to be able to bring the activities into their classrooms.

With this in mind, the first design proposition is based on a CS Unplugged activity about sorting (Activity 8: Beat the Clock -- Sorting Networks). The goal of the activity is to show that some tasks can be completed more quickly when they can be broken up into individual parts which can then be completed by different people at the same time. A diagram is drawn on the ground with chalk (p. 72, or p. 2 of the individual PDF). Students are divided into teams of six and positioned at the starting squares of the diagram. Each student will have a random number given to them. They follow the arrow leaving their current position until they reach a circle. There, they must wait for another student to reach them. When there are two, the person with the smaller number follows the left arrow out, and the other person the right arrow. When everyone reaches the ending squares, the numbers assigned to them will be in sorted order. The idea is to time the students so they can see what networks can sort the fastest, then think about why.

This activity gives a good embodied experience in that students physically move through a real network and as a result get a sense of how the sorting work is distributed. However, it requires the ability to take a class outside where the chalk diagrams can be drawn, and is limited to a small number of networks because of the time is takes to reproduce them. Also, the problem they are solving is still somewhat abstract in that there isn't a context the students can connect with beyond the given goal of sorting numbers (though the competitive nature of timing the students surely doesn't hurt). The proposed augmented reality version of the activity will seek to improve on these downsides.

To give the activity some more context, a problem that students can relate to is devised and the activity is modified so students will start without knowing they are sorting numbers. Each student is given a picture of a toy with a house number and street name. They are told that they have been assigned to help Santa deliver Christmas gifts to children on that particular street (note that the street name could be chosen by the particular group running the activity to be one the students are familiar with, but the house numbers must always be associated with a particular toy). Because it has snowed recently, nobody can read the house numbers, but luckily, there are elves available to help the students find their way.

At the beginning of the activity, students will line up on one side of the classroom and look at the pictures of the toys to see where to go next. They might use a handheld device as a magic lens, or a head mounted display. A camera will be used to recognize the image of the toy, and an arrow or perhaps a trail of bubbles or pixie dust will lead the student to the next location, where they must wait for another student to join them. When there are two students waiting, they each hold up the images of their toys for an animated elf to examine. The images must be held close enough for a single camera to recognize them together. The elf then indicates which path each student will take based on the relative order of the house numbers associated to the toys. This continues until each student reaches a house, their final destination. When all students have made it through the sorting network, a discussion on what happened has the students figure out that they managed to quickly sort the street addresses using a sorting network.

In addition to situating the problem in the toy delivery story, augmented reality makes it possible to do this activity anywhere, anytime. It could be designed to include an arbitrary number of students, and have a mode in which the entire sorting network is projected onto the ground so students can see it all at once, just as was the case for the CS Unplugged activity. The embodied nature of the original activity is retained. Students physically move through the network, and can take advantage of the intuition that says going left often means something is smaller, and going right means something is larger. This is useful when developing an understanding of how the algorithm works.

Augmented reality is the only technology that can accomplish all of this. Virtual reality comes close, but does not have the advantage of using the real world as a base, and would not allow for the use of mobile devices instead of heads up displays. A similar scenario might be possible with mobile devices that don't support augmented reality, but it would be difficult to direct students to their next locations without a virtual or real network that can be visualized. Recreating the activity on any other device, such as a desktop computer, would remove the advantages that come from the embodiment found in the other versions.