Contact Us
Helping the Hearing Impaired Through Research at UWC

UWC Researchers: Helping the Deaf Communicate

Estimates vary but it is believed that there are up to 1.5 million people with hearing disabilities in South Africa, and 360 million or more worldwide have disabling hearing loss - a significant part of the population that is frequently overlooked as a potentially economically active workforce. To help, a number of researchers at the University of the Western Cape are developing innovative technology applications to enable Deaf and hearing hearing-impaired people to communicate more effectively with hearing people, especially for users in the developing world.

Professor Isabella Venter, Head of the Computer Science Department, says more than 45 postgraduate UWC students are currently working on projects relating to technology applications.

Researchers at the Telkom/Cisco/Aria Technologies/THRIP Centre of Excellence for IP and Internet Computing, including scientists from Bridging Application and Network Gaps (BANG) and Integration of Signed and Verbal Communication: South African Sign Language Recognition, Animation and Translation (SASL) groups in the UWC computer science department, are working on several interconnected projects and programmes focusing on cellphone or mobile applications with the ability to translate spoken language into sign language and vice versa.

A key reason for the success of these projects, Professor Venter believes, is that they are inclusive – involving large industry partners and small NGOs as well as active engagement with ordinary South Africans with hearing impairments.

Signing Support: How the Deaf can be heard

SignSupport is a video application for cellphones, in which pre-recorded videos for specific situations are loaded onto the phones, enabling Deaf people to interact with, for example, a pharmacist and to get medicine without the help of an interpreter or third person.

BANG leader and UWC Associate Professor in Computer Science Bill Tucker says that their research and collaboration with a  local Deaf community has helped uncover the real needs of Deaf people. One of the big misconceptions about Deaf people is that they are able to read and write. But many Deaf people are in fact, functionally illiterate and unable to read, literate only in sign language.

“Scientists have for years been building things for them that they don’t need. Often it was a case of computer geeks looking for problems they could think up cool solutions for, but which weren’t necessarily helpful to people in the real world,” he says. “SignSupport is video-based to make it more accessible. So far, the Deaf people we have involved in the research think it’s great.”

Real-Time Intelligent Translator On a Mobile Phone

In parallel with this, the Assistive Technologies (formerly SASL group), led by Mehrdad Ghaziasgar, is working on a system to translate between South African Sign Language and English. Sign language interpreters are scarce and expensive, and such a system can provide the currently marginalised Deaf with much-needed access to the world of the hearing.

This research has focused, over a number of years, on recognising sign language in ordinary video by locating the signer in a video, tracking both of the signer’s hands, and recognising the smaller sub-units of sign language gestures, namely, hand shape, orientation, location and motion, as well as facial expressions, all using a normal inexpensive webcam. Work has also been done on rendering high quality realistic sign language using 3D avatars. This work will eventually be integrated into a fully-fledged translation system that will allow Deaf and hearing users to communicate using mobile phones.

“Considerable research has been conducted in this field, but none of the research has managed to produce full-fledged, or even partial, translation systems,” notes Ghaziasgar. “Our most recent work allows us to provide hand tracking capabilities that are comparable to that of the Microsoft Kinect camera using only a simple webcam. We are working with two Deaf schools and have received very positive feedback on our system thus far.”

Ultimately, the hearing user will use the camera on their phone to record a person performing sign language, and this will be translated and played back in English speech in real-time on the phone. In reverse, the Deaf user will use the microphone on the mobile phone to record what is being said in English and the sign language equivalent will be rendered by a 3D avatar on the display of the phone.

“This work is already at the cutting-edge of research in the field, but when it’s finished, it will truly be a world-first. In this day and age, there simply should be no reason for Deaf people to remain ostracised from the hearing world - we want to provide the ability to communicate fully.”

Local Solutions to Global Problems

The mobile technology applications have been designed in such a way that they can be used by people in other countries and enable oral recordings as well, making it internationally competitive – but it is right here in the developing world where the technologies stand to have the greatest impact.

“Cellphones have become ubiquitous, even in poor communities. So this has implications for especially rural areas, where people have amazing phones but where literacy levels are low. As one goes north on the continent, this will probably only become more pronounced,” says Professor Tucker.

This is where UWC has real potential to make a difference. While considerable work is being done all over the world regarding technology applications for Deaf people, there is very little for people with disabilities in developing regions.

Professor Venter says: “Alternative forms of communication as demonstrated by these projects can make a real and measurable difference for individuals and more broadly for economies and countries.”

So on this World Hearing Day, spare a thought for those who have trouble experiencing the simple joys of sound - and the work that’s being done to make that less of a problem.