Search Explore How it Works Login Sign Up

Audio in the Metaverse, Experience The Future

September 30, 2022 By Fakun Gram

(Image Credit Google)

Reports say that Meta and a group of researchers from the University of Texas at Austin (UT Austin) are experimenting with bringing realistic audio to the metaverse. Kristen Garuman, Research Director at Meta AI, states that AR and VR are not only visuals. She explains that audio is a significant part of the metaverse, "shaped by the surrounding environment." Thus, various factors influence how sound behaves, like the geometry of a room, what's being said in it, and from how far. Hence, audio is a significant part of the virtual world experience. Meta plans to achieve this by using AR glasses to record audio and video. Then, it will use three AI models to transform and clean the recording for users to get an immersive experience. Interestingly, the AIs will consider the room the user is to match the environment. Similarly, with VR headsets, Meta wants to replicate the sights and sounds of an environment.

Furthermore, Meta made these AI models open source so third-party developers can work on the audio in the metaverse tech. However, it did not reveal how users can listen to the enhanced audio or further details.

AI models to bring audio into the metaverse

Here's how Meta will record audio with AR glasses and have it mimic an environment - 1. The first AI model to bring audio into the metaverse is AViTAR or the "Visual Acoustic Matching" model. It will transform audio to match a new environment. Researchers claim that users can take a recording with AR glasses and play it in their homes, where the AI will morph the audio. AViTAR will scan the environment, consider any obstacles, and have the recording sound like it's happening in real-time. According to the researchers, the audio will come from the glasses. 2. The second model is "Visually-Informed Dereverberation." It will clean up the audio in the recording by removing distracting reverb. Audio in the metaverse

3. The last AI model is "VisualVoice." It will use a combination of visual and audio signs to separate voices from other noises. Meta believes visual cues are essential as AI must determine who's talking by understanding specific nuances. So for visual cues, the company will bring in video and other signs to enhance AI-drive audio. Lastly, the technology used by both these AI models is pretty new. Hence, it is unclear when(or if) Meta will bring audio into the metaverse.

By Fakun Gram

Well, I am here to keep you up to date so together we can summarize the whole world.

Audio in the Metaverse, Experience The Future

AI models to bring audio into the metaverse

Walmart's GenAI Search Engine: Challenging Google's Dominance in Retail Search

Introducing the MSI Claw Handheld: Your Ultimate Gaming Companion

Uber Increases In-App Ads, Prompting Mixed Reactions from Customers

Apple's iOS 18: A Leap into the AI Era

Google's Regular Pixel 8 Won't Get Gemini Nano AI

MacBook Air M3 Makes Amends for M2's Storage Blunder

Samsung Unveils the Galaxy M15 5G

Elon Musk's xAI to Open-Source Chatbot Grok

Contra: Operation Galuga - A Modern Run-and-Gun Classic

Musk Confirms X's TV App Arrives This Week

This DIY Dream Gaming PC Machine Is Made Of Glass

Mythic Quest 3 Review: VR Headset From Meta Has Mixed Reactions