Serious Games challenging us to play inclusiveness
Via: MathSigner and The Smile Project
Overview
Two Purdue University professors are developing a highly interactive learning tool to improve the mathematical abilities of deaf children.
The reason for the project is that, for deaf children, reading instruction is delayed; therefore most kids in this age group cannot use traditional textbooks.
They also aren't able to take full advantage of secondary learning opportunities that non-disabled kids have, such as television and dinner table conversation, and their parents may not be fluent enough in sign language to teach them about math.
The first phase of the project focused on creating an interactive PC game called Mathsigner, in which 3D avatars communicate with the user in sign language
The Purdue project hopes to alleviate these challenges by allowing deaf children to learn math concepts from animated characters, who use sign language to communicate with the kids in a computer-game setting.
Even more effective is extending the program into virtual reality, where the characters and the children can communicate with each other in sign language in real time, in a fully immersive environment. SMILE, the project’s VR game, features a robot virtual signer and a virtual clock store. The game is projected onto three walls and the floor of a special room using Fakespace Systems’ FLEX.
Project Background
Three years ago, Nicoletta Adamo-Villani, assistant professor in Purdue’s Department of Computer Graphics Technology, who teaches 2D and 3D animation and specializes in character animation and design, joined with Ronnie Wilbur, professor and chair of linguistics in Purdue’s Department of Speech, Language and Hearing Science.
Mathsigner Game
The first phase of their joint project focused on creating an interactive PC game called Mathsigner, in which 3D avatars communicate with the user in sign language. The team recently earned a grant from the National Science Foundation to proceed with this aspect of their work; it is currently in prototype form and in use at the Indianapolis School for the Deaf as a web/CD-deliverable desktop application.
The first phase of their joint project focused on creating an interactive PC game called Mathsigner, in which 3D avatars communicate with the user in sign language. The team recently earned a grant from the National Science Foundation to proceed with this aspect of their work; it is currently in prototype form and in use at the Indianapolis School for the Deaf as a web/CD-deliverable desktop application.
Mathsigner interface
Smile - An Immersive 3D VR Game
A more recent phase of the project extends the software into an immersive, virtual-reality world. Working with Purdue’s Envision Center for Data Perceptualization, the researchers are creating a game that is projected onto three walls and the floor of a special room, using Fakespace Systems’ FLEX (the VR game can be played on a PC as well but it is not immersive in that format). When using the application, the child wears a special headset, lightweight stereoscopic glasses, pinch gloves and a wrist tracker. These devices allow the child to communicate with the avatars and redraws the animation to follow the child’s perspective as he or she moves.
“Research shows that immersive games are more effective than non-immersive games,” says Adamo-Villani. Studies have found that humans process visual information 60,000 times faster than text-based information, she reports, noting that students can raise their math scores 16% after eight weeks using an immersive VR program.
Such a system is an effective teaching tool because it gives the students control over the environment, offers self-paced and repetitive learning, presents concepts in a concrete rather than abstract manner, features an environment that feels safe, and, as a game, is motivational. All of these characteristics help the student overcome challenges that are common to people with learning disabilities.
The Animation Process
The game features four stores in which students learn a different math concept: a bakery where they learn about weight, a clock store where they learn to tell time, a toy store where they learn how to count money and a candy store where they learn addition and subtraction. The avatar and the child can communicate; for example, one of the avatars can ask a question and the child can sign a simple answer, such as a number, or the child can pick up a certain number of candies and the avatar can make the sign for that number.
Motion-capture is used to record the signs, so the avatars — characters such as rabbits, robots and pigs that serve as the storekeepers — can communicate with the students in real time; the remaining animation, including facial expressions, is key-frame. All models were created in Maya.
In order for the Mathsigner characters to be displayed in the FLEX system, the polygon count had to be reduced to maintain the fast frame rate and real-time interaction, and new facial rigs had to be created. Polygons are concentrated in areas where detail is needed most, namely parts of the body that bend or twist. This keeps the signing realistic-looking.
“It requires an extremely fluid motion,” says Adamo-Villani. “We’re working with signers who are very picky. The motion has to be very realistic.” At the same time, the animation look is colorful, cartoony and stylized. “We’re using Disney’s Toontown as inspiration,” she explains, noting that while many of the challenges of this project are similar to any animation production, a unique challenge is to make the stylized characters do the signing motions organically, in real time.
“It requires an extremely fluid motion,” says Adamo-Villani. “We’re working with signers who are very picky. The motion has to be very realistic.” At the same time, the animation look is colorful, cartoony and stylized. “We’re using Disney’s Toontown as inspiration,” she explains, noting that while many of the challenges of this project are similar to any animation production, a unique challenge is to make the stylized characters do the signing motions organically, in real time.
The project is popular with students in computer animation at Purdue. They see it as a good opportunity to learn both motion capture and key-frame animation, and they end up with a strong portfolio piece. Graduate, masters and undergraduate students are working on it, led by David Jones, a graduate student in the department of computer graphics technology.
The application is being refined to add more gaming elements. Currently, users can go into individual stores, one at a time. The creators are looking at ways to link the stores so the child can move from one to another as he or she completes a task, making the action more like a videogame.
Some improvements are under consideration on the technology side, too. The researchers plan to look into using more sensitive gloves, such as those employed for mo-cap, which would enable more complex signing by the children as they use the immersive system.
In addition, the current FLEX system is expensive and must be used in a special room, which makes it impractical for both institutions and individuals. The researchers are exploring different options for a portable yet still immersive system.