The Smart Interpreter, the intelligent viewer created in Peru that seeks to connect deaf people with the hearing world

Not being able to communicate with another person because they do not speak sign language is an obstacle for deaf people. And it is that in an ideal society this could be taught in schools, universities and even in workplaces in order to eradicate that difference; however, that is not the reality. Fortunately, the technology It has allowed a new generation of creators to be able to develop inventions and functionalities that bridge this communication gap.

LOOK: uDocz, the Peruvian study note exchange platform that has received US$2 million from Silicon Valley

William Bastianan industrial engineer from the University of Lima who works as a technological consultant, is one of the people who used technology, specifically artificial intelligenceto create a new alternative for deaf people.

According to the World Federation of the Deaf, there are more than 70 million deaf people in the world. In Peru, on the other hand, there are 232,136 people who cannot hear, according to the INEI. Not being able to hold a conversation with a person who doesn’t know sign language not only keeps them away from the possibility of establishing new bonds, but also from access basic services such as hospitals and supermarkets.

LOOK: Freakit, the new dating app that aims to be the “Tinder of otakus”

Faced with this need, Bastian created The Smart Interpretera viewer that, thanks to artificial intelligence, works as an interpreter between the person who listens and the other who is not able to hear. The viewer can automatically convert the spoken word into written text and vice versa.

The creation of this viewer caused the young man to be named as one of the 35 Innovators Under 35 Latin America 2022 of MIT Technology Reviewan event in which 4 other Peruvians were also recognized.

Trade spoke with Guillermo Bastian, who told us a little more about his creation and the experience in this event.

READ ALSO: Images created with AI: how long would such a job take a professional illustrator?

—What is the story behind the creation of The Smart Interpreter?

I work as a technology consultant and I met a person who was very talented, he had many degrees, he was very intelligent. But the moment we work together in a meeting, let’s say a group conversation, it was a bit difficult for him to follow the topic, actually show all the talent he hadEven though he was a very intelligent person.

So from there I began to see a bit of the need for deaf or hard of hearing people to be able to connect with the hearing world at work or even in personal life. It is not only working and being successful at work, but also connecting with people, making friends, falling in love with someone. So that connection was missing, missing in society.

LOOK: This is ART, the turtle robot that is capable of moving on land and water

—Keeping in mind that sign language is quite gestural, how exactly does the viewer work and what technology does it use?

These glasses have two main functions, the first is to “listen”. The viewer has microphones that pick up information when a person speaks. This audio, with a model ‘speech to text’, is converted to text. From there, that text will be transformed by various grammar engines into a sequence of images.

So, what people speak to you will be seen on a screen through a sequence of images and different signs that make up that sentence. And a little the second function that it has is that of “speak”. The glasses have a camera positioned on the forehead and the idea is that by looking at the signs you make with your hands, they can be translated into voice. And it does this through artificial intelligence techniques, deep learningto learn how signs relate to words.

LOOK: This AI chatbot lets you “talk” with Elon Musk, Donald Trump and more

The viewer has a camera, a screen, microphones and a speaker. (Photo: William Bastian)

“How many words has the viewer learned so far?”

have learned about 70 words and it just needs a lot more data to reinforce and to expand the vocabulary.

—How long does it take approximately to interpret sentences in a conversation?

It depends a lot on the sentence, but it does take a few extra seconds for two reasons. One, because of the nature of sign language, and two, because the solution has to read, interpret, audio and play. So you do have a few more seconds that it can take.

LOOK: This is NOT Mark Zuckerberg, it is a hyper-realistic avatar that has left us perplexed

—There are more than 300 sign languages ​​in the world, which one is used in the viewer?

Use the one from Peru.

—Is there a possibility of expanding the viewer to more sign languages ​​in the future?

Yeah right if. The tool is parameterizable and trainable, It has those two characteristics. Parameterizable is that, as you yourself said, sign language is very gestural, so there are things that you won’t be able to see with glasses, which is the way you put your mouth, the way you put your eyes. So that’s why you have certain parameters that you have to know how to use these glasses, to tell you when they’re done in the right order.

And, on the other hand, it is trainable because by using artificial intelligence, one can teach him the different languages. So, the first time you teach a different language, a different sign language, it can be a bit expensive to teach the whole language to the tool, but once you’ve learned the language of a country it’s easily scalable because it’s already copy and paste that model in the different glasses.

LOOK: Goodbye Zoom? They test the first holographic video calls by 5G in Europe

—Currently it is in the second prototype, how soon could we see the final design?

My idea is that there In a year I will already have a final design with many words used by the deaf.

There the issue is that one of the needs that I would like to cover is that already with the visibility that this award has given, to have more contact with universities, for example, that can help me train this device. Training an artificial intelligence is a very expensive process, it takes time and several people. I think that with the support of an educational institution I could do it much faster.

View this post on Instagram

A post shared by Intérprete Inteligente / Smart Interpreter (@speakinghandsproject)

—That is precisely what I was going to ask you, is there any company that has shown any interest either in buying the algorithm or perhaps starting to work as an alliance?

For the time being still, and my idea is that after a little while with the visibility of the award, I can have easier contact with universities or other entities.

LOOK: The shocking images that two AIs created with the futuristic version of Lima in the year 2050

—Now, how was the experience at MIT Technology Review? How does it feel to be part of the five Peruvian winners of this event?

Well the truth is exciting because apart from the work that one does day to day, because I also work as a consultant in a technology company, in my day to day, in my free time I dedicated many hours to this project. so for me it is quite gratifying to be recognized for all the effort and hours that I have invested of my personal life in a project like this.

And above all that it is a project that has a social affinity, So it’s gratifying to know that I can have this visibility and that I can go much further with this project. So, really, very good experience. And the organization that has had the event has been quite agile. The questions that we have asked, well at least me, to the organizers have always been answered immediately. Super friendly, super organized and very excited. And also very excited to meet the other winners, who are surely very talented people.

—In addition to seeking that alliance with universities as you mentioned, is there any other step that awaits The Smart Interpreter?

I think that a little, apart from the universities, alliances with deaf associations here in Peru, which I am sure would be delighted to initiate more user tests to vary the solution. So I think it’s a bit of the following steps.

On the one hand, as you rightly said, there is contact with universities to train, to expand the capacity of the product, of the solution. And on the other hand, when the solution reaches a prototype three, surely the next step will be the association with these deaf associations to vary the solution and to give it the shape of a commercial product.

LOOK: This AI “imagines” video game characters as if they were real people

—We are waiting then that the next step, in the future, will be the commercialization of the viewfinder…

Yes, but now, a bit is the reality of Peru. The ideal would be to do it with the universities or the State because, from what I have found out, deaf people in Peru have it quite complicated.

Then, this project was never for an economic purpose because the deaf really have difficulty having an education and therefore have income that can sustain the purchase of this type of product. So I would wait to make contacts to make this solution more accessible to Peruvian men and women who have complete deafness or hearing loss.

Source: Elcomercio