[ skip to content ]

ODU Researchers Give Children with Autism a Smartphone-Image Communication System

De Leo with a PixTalk smartphone

Many children with autism communicate very little, even with the family members and teachers who are around them the most. But researchers at Old Dominion University have shown that a novel approach using smartphones and the World Wide Web can help these children express their thoughts and desires.

To Gianluca De Leo, an assistant professor who specializes in bioinformatics and virtual reality at ODU's College of Health Sciences, the project is yet another example of how information technology (IT) can improve human health and wellbeing. He previously has adapted smartphones to help diabetics manage their health and employed virtual reality to promote treadmill training for children with cerebral palsy.

The autism project, which was funded by Microsoft's External Research Group, allows children to communicate by calling up images on Windows mobile devices. A child who wants chicken for dinner, for example, can call up one image of a dinner table and another of a chicken leg.

Children with autism often are trained to express themselves by picking one or more laminated picture flashcards from a folder or binder and affixing their message or "sentence" to a strip of Velcro. A popular adaptation of this is called PECS for Picture Exchange Communication System.

There are also existing electronic communication devices such as the Cyrano Communicator and Pocket Reader that can be used to help children express themselves with images, but these devices often cost $5,000 or more.

De Leo says smartphones are cheap, easy to use and offer numerous other advantages over the laminated picture system and existing electronic communicators.

He leads the development of the PixTalk Communications System together with Gondy Leroy, associate professor of information systems and technology at Claremont Graduate University in California. Another member of the project team is Padmaja Battagiri, who received a master of science degree in electrical and computer engineering from ODU in December 2009. Her advisers included De Leo and her master's thesis focused on smartphone use by children with autism.

The team's paper, "A Smart-Phone Application and a Companion Website for the Improvement of the Communication Skills of Children with Autism: Clinical Rationale, Technical Development and Preliminary Results," was published earlier this month by the Journal of Medical Systems.

PixTalk software - which is available under an open-source license - can run on any Windows Mobile Smartphone, and is designed to work with a Web site that the researchers have fashioned to serve as project central. A child's teacher or caregiver can access the Web site and select from an online library of images to be downloaded to the child's phone. The downloaded file can be personalized to the child.

Furthermore, a PixTalk usage log can be downloaded from the smartphone to the Web site, where the data are crunched and a graphic representation of the child's progress with communication skills is posted. This tracking feature also allows the teacher or caregiver to fine-tune a child's personalized portfolio.

Perhaps the biggest advantage of the smartphone scheme is the ease with which a teacher or caregiver can add photographs to a child's portfolio. "It's difficult to represent by cartoon images a child's sister or the child's house," De Leo explained. "But if you can take pictures of the sister and house and load them, there is more chance that the child will use them."

Photos customized to the child also can be incorporated when the Velcro board and laminated cards are used, De Leo said. But teachers have told the researchers that the physical work involved in the lamination and in replacing lost cards tends to discourage the customization.

The researchers relied heavily upon teachers of autism spectrum disorder (ASD) children when they were developing PixTalk, and this user-centered design seems to have worked out well, De Leo said. The "preliminary results" referred to in the title of the research paper came from an in-depth study of how children with autism used PixTalk as compared to PECS. Each child and each child's teacher were previously trained in PECS. The teachers were given PixTalk training before the experiment began.

Questionnaires filled out by the teachers as the experiment progressed showed that they judged PixTalk to be the better communications tool. The children's "motivation, attention, response rate and referential communication … increased when computers were used," the researchers conclude in their paper.

Another interesting finding: the children preferred images taken with a camera to cartoon images. "This is a good indication that for children with severe autism real images may work better than cartoon images," the paper added.

Objective data collected during the experiment from the PixTalk logs showed that the children's usage of images was complex enough to make it difficult for teachers using PECS to accurately keep a manual log. This means that communications progress information churned out automatically by PixTalk would likely be superior to reports filled out by the teachers or caregivers who use PECS.

Smartphones are easier to use than laminated cards and Velcro, De Leo said. "It takes a long time for a child to select the laminated pictures from the binder and arrange them on the Velcro strip. They may want to express happiness when they start the process, but by the time they get their message posted, they're frustrated and not so happy."

Still, the researchers report that the children with autism in their study could get frustrated with computers, as well. Their results indicated that some children with ASD "may react poorly to a sudden turn-off of the device due to its battery life." Their recommendation to PixTalk users was to pay special attention to battery power and power conservation.

In addition, the researchers say they are continuing to work on ways to customize mobile devices for use by children. For instance, teachers have recommended a delayed central button response to minimize instances of accidental changes between "operational" mode - in which children search for images and arrange them - and "display" mode - in which the message is locked in for display.

De Leo sees a lot of work ahead on the project, not the least of which will be customizing PixTalk for use in other countries and in other languages. The Italian-born researcher pointed out that "a picture of french fries might not work very well in Italy."

The doctorate that De Leo earned from the University of Genova is in bioengineering and bioelectronics, and he has worked on medical modeling projects as an affiliate of ODU's Virginia Modeling, Analysis and Simulation Center (VMASC).

He is part of an international research team that is helping elderly diabetics in China manage their health and learn more about their condition. One aspect of the project provides computer games that are fun, but also educational about diabetes. De Leo's role in that project, which was presented last October in Washington, D.C., at a health summit sponsored by the Foundation for the National Institutes of Health, involves the game-based tools for smartphones. He also is designing a project Web site that can help transition the work being done in China to other countries, such as the United States.

De Leo was part of a third team that last year published "Combining a Virtual Reality System with Treadmill Training for Children with Cerebral Palsy" in the Journal of CyberTherapy & Rehabilitation. Karen Kott, associate professor of physical therapy at ODU's College of Health Sciences, led that project.

Gaming again was central to the treadmill training project. The researchers reported that children exercised longer when they took a virtual role in a "save the princess" video drama unfolding on a screen in front of them.

This article was posted on: February 10, 2010

Old Dominion University
Office of University Relations

Room 100 Koch Hall Norfolk, Virginia 23529-0018
Telephone: 757-683-3114

Old Dominion University is an equal opportunity, affirmative action institution.