Via: 3D SIGN LANGUAGE MATHEMATICS IN IMMERSIVE ENVIRONMENT
Following my recent post dated 21 September 2007 Serious Games Improve Mathematical Abilities Of Deaf Children, I've come across Futurelab Blog's article Immersive 3d VR Games for Deaf Children, on the 27th, which makes a reference to Adamo-Villani reports, where we may find the research abstracts that show that immersive games are far more effective than non-immersive games, noting that students can raise their math scores 16% after eight weeks using an immersive VR program.
Excerpts
In the paper, Nicoletta Adamo-Villani, Edward Carpenter and Laura Arns describe the development of the immersive 3D learning environment to increase the mathematical skills of deaf children.
The application teaches mathematical concepts and ASL (American Sign Language) math terminology through user interaction with fantasy 3D virtual signers and environments. The program can be displayed in immersive devices and includes a gesture control system comprised of a pair of pinch gloves and a 6-degrees-of-freedom wrist tracker.
Their application would improve on existing examples of virtual learning environments for the hearing impaired in terms of: (1) high realism/fluidity of the 3D characters’ signing motion; (2) complexity of real time interaction between student and 3D avatars and environments; and (3) natural communication between user and application via a simple glove-based gesture control system.
A Unique In-Depth Approach: 3D Immersive Animated Signing
Deaf education, and specifically math/science education, is a pressing national problem. The project addresses the need to increase the abilities of young deaf children in math with a unique approach: 3D immersive animated signing.
The general goal of the research is the development of an immersive virtual learning environment in which deaf children (age K-3) interact with fantasy 3D signers and learn basic ASL math terminology and concepts. The interactive application can be displayed in immersive devices such as the ‘Fakespace Labs FLEX’ [3] and is designed to engage deaf learners in "hands-on, minds-on" experiences, leading to deeper understanding of fundamental ideas in accordance with current educational guidelines.
Initially they created a highly interactive computer animation program (Mathsigner™) for classroom and home learning of K-3 arithmetic skills, aimed at deaf children.
Because several research findings suggest that immersive learning applications are more effective than non-immersive ones, they have adapted the Mathsigner™ characters for display in a total-immersion environment.
Research demonstrates that deaf individuals are significantly underrepresented in the fields of science and engineering. Historically, it has been difficult for them to gain entry into higher education that leads to STEM careers.
The documented effectiveness of Virtual Reality (and games) as a learning tool and the potential of immersive Virtual Reality Environments to help disabled students are the motivating factors behind the research.
Recently, there has been noticeable progress in development of VR applications for people with different types of disabilities. In the area of hearing impairments in particular, efforts have been directed primarily to creation of sign language recognition and synthesis systems.
Survey of existing VR applications for the Hearing Impaired
As far as development of virtual learning environments to assist in Deaf education, they have found two noticeable examples of virtual environments for deaf/speech-impaired students: the ‘Virtual Supermarket’ developed at the University of Nottingham in England, and the VREAL (Virtual Reality Education for Assisted Living) project, funded by the U.S. Department of Education.
Researchers at the University of Nottingham have developed a series of desktop VEs to assist in teaching of daily life skills, as well as ‘Makaton’, a sign language used in the UK by people with learning disabilities. Assessment studies showed that disabled students who used the VE were able to perform everyday life activities (i.e., a shopping task at the supermarket) in a shorter time and with a lower number of mistakes. Students also showed a significant increase in the number of Makaton signs that they knew.
In the VREAL project, engineers from Veridian have teamed up with six schools for the Deaf and have adapted the technology used in military simulations to create an elaborate immersive virtual learning environment for deaf children. The environment includes replicas of an elementary school, supermarket, farm, and post office. Students use joysticks and head mounted display glasses with motion sensors to navigate the virtual world and learn life skills, language arts, math, and science. The software has embedded video of the teacher signing the instructions; the teacher can also stand in front of a video camera and sign to the student while the image appears in real-time on the student’s display screen. Five schools across the US used the program in 2004 and assessments show that students improved their test scores by an average of 35%.
Implementation
The interactive content is so far limited to grades K-1; they are currently programming math activities for grades 2 and 3. The majority of the interactive activities are based on Mathsigner™ and have been redesigned and reprogrammed to function with the immersive application and the specialized input devices.
The program teaches mathematics symbols and ASL signs for the numbers one
to twenty, and mathematics skills for the four operations.
Three representations of number ‘two’: ‘2’ candies on the counter,
Bunny signing ‘2’; math symbol for number ‘2’ on cash register
Candy store model: interior with ‘bunny’ character, left; exterior, right
The application has been evaluated throughout its development by deaf adults, Purdue faculty and students knowledgeable in sign language and deaf related issues who have provided positive feedback on the readability of the signs and the effectiveness of the program.