Sony Unveils In-Game Sign Language Translator

Sony Patents ASL to JSL Translator for Video GamesProposed to Use VR Devices and Work Over Cloud Gaming

Sony has filed a patent that adds a real-time sign language translator to video games. The patent, titled "TRANSLATION OF SIGN LANGUAGE IN A VIRTUAL ENVIRONMENT," illustrates a technology wherein American Sign Language (ASL) could be communicated to a Japanese-speaking user using Japanese Sign Language (JSL).

Sony said it aims to put in place a system that can assist deaf gamers via real-time translations of sign languages during in-game conversations. The technology described in the patent would allow virtual indicators or avatars that are displayed on-screen to communicate sign language in real-time. The system will initially translate the sign gestures of one language into text, then convert the text to another specified language, and finally translate the data received into the sign gestures of the other language.

"Implementations of the present disclosure relate to methods and systems for capturing sign language of one user (e.g., Japanese), and translating the sign language to another user (e.g., English)," Sony described in the patent. "Because sign languages vary depending on geographical origins, sign language is not universal. This provides a need for appropriately capturing the sign language of one user, understanding the native language, and generating new sign language as output for another user in their native sign language."

Sony further proposed that one user device could seamlessly communicate with another user device over a network with a game server. "In some implementations, the game server executes a shared session of a video game, maintaining the canonical state of the video game and its virtual environment," Sony said, "and to which the user devices are synchronized regarding the state of the virtual environment."

With this setup, users can share and interact with each other in the same virtual environment, aka game, over a shared network or server. Sony additionally said that in some implementations of the system, the game server could be part of a cloud gaming system, which "renders and streams video" between each user device.