Home » News » Audio in the Metaverse, Experience The Future

Audio in the Metaverse, Experience The Future

fb twitter pinterest linkedin
Audio in the Metaverse, Experience The Future-GadgetAny
Audio in the metaverse

Reports say that Meta and a group of researchers from the University of Texas at Austin (UT Austin) are experimenting with bringing realistic audio to the metaverse. Kristen Garuman, Research Director at Meta AI, states that AR and VR are not only visuals. She explains that audio is a significant part of the metaverse, “shaped by the surrounding environment.” Thus, various factors influence how sound behaves, like the geometry of a room, what’s being said in it, and from how far. Hence, audio is a significant part of the virtual world experience. 

Meta plans to achieve this by using AR glasses to record audio and video. Then, it will use three AI models to transform and clean the recording for users to get an immersive experience. Interestingly, the AIs will consider the room the user is to match the environment. Similarly, with VR headsets, Meta wants to replicate the sights and sounds of an environment. 

Audio in the metaverse

Furthermore, Meta made these AI models open source so third-party developers can work on the audio in the metaverse tech. However, it did not reveal how users can listen to the enhanced audio or further details. 

AI models to bring audio into the metaverse 

Here’s how Meta will record audio with AR glasses and have it mimic an environment –

1. The first AI model to bring audio into the metaverse is AViTAR or the “Visual Acoustic Matching” model. It will transform audio to match a new environment. Researchers claim that users can take a recording with AR glasses and play it in their homes, where the AI will morph the audio. AViTAR will scan the environment, consider any obstacles, and have the recording sound like it’s happening in real-time. According to the researchers, the audio will come from the glasses. 

2. The second model is “Visually-Informed Dereverberation.” It will clean up the audio in the recording by removing distracting reverb. 

Audio in the metaverse

3. The last AI model is “VisualVoice.” It will use a combination of visual and audio signs to separate voices from other noises. Meta believes visual cues are essential as AI must determine who’s talking by understanding specific nuances. So for visual cues, the company will bring in video and other signs to enhance AI-drive audio. 

Lastly, the technology used by both these AI models is pretty new. Hence, it is unclear when(or if) Meta will bring audio into the metaverse.

Fakun Gram

By Fakun Gram

Well, I am here to keep you up to date so together we can summarize the whole world.

Related news