In a recent research update, the BMW Group Technology Office USA in Mountain View unveiled their successful demonstration of accurately showcasing VR and MR content to passengers, even when experiencing dynamic movements such as turns, speed bumps, and acceleration. This achievement marks the first time such stability has been achieved in a fast-moving automotive setting.
Claus Dorrer, Head of BMW Group Technology Office USA in Mountain View, emphasized the potential of extended reality (XR) devices in vehicles, envisioning numerous use cases, including assisting drivers in crowded parking lots, providing road hazard alerts, and delivering crucial information about the vehicle’s condition. Dorrer highlighted the promising implications of future augmented reality (AR) glasses and VR devices for both drivers and passengers. The research partnership with Meta allows them to explore immersive XR experiences within vehicles and drive the seamless integration of such devices into BMW cars.
One of the key technical challenges addressed in this collaboration is the conflict between the headset’s sensors and cameras when used in fast-moving vehicles. VR headsets rely on motion sensors and cameras to maintain stable virtual content as users move their heads or look around. However, within a moving vehicle, these sensors can become confused by conflicting signals from the headset’s movement and the vehicle’s motion. As a result, current VR headsets struggle to provide stable virtual content during fast travel, including accelerations, sharp turns, or encountering road bumps.
To overcome this challenge, BMW and Meta’s researchers worked together to integrate information from BMW’s sensor array into the tracking system of a Meta Quest Pro headset. This integration enabled the tracking system to accurately calculate the headset’s location relative to the moving car and anchor virtual objects to the car’s interior, ensuring stable and captivating VR and MR experiences for passengers.
Richard Newcombe, Vice President of Research Science at Meta’s Reality Labs Research, expressed excitement about the research prototype’s potential to revolutionize passenger experiences within vehicles. The technology showcased in the prototype opens up possibilities for immersive gaming, entertainment, productivity, and even meditation that are seamlessly connected to the car itself. As advancements progress towards reliable world-locked content and AR glasses, passengers may soon have access to additional information such as landmarks, restaurants, and points of interest during their travels.
The successful integration of vehicle movement data with XR headset sensor capabilities allows for precise tracking of the headset’s movements within the car. This enables real-time rendering of content that remains synchronized with the vehicle’s motion, whether driving on a straight road or navigating sharp turns and bumpy terrain.
The proof-of-concept prototype not only demonstrates the full range of a person’s movements within the car (six degrees of freedom) but also showcases an XR experience that accurately calculates the headset’s location relative to the car and the car’s position in the external world by leveraging the car’s sensors.
As a pioneer in digital technologies, the BMW Group has a history of driving industry standardization, including innovations like vehicle access based on the Digital Key. The company’s global network of technology offices plays a vital role in initiating and fostering such innovation projects. Additionally, BMW Group remains open to collaborating with additional partners to establish an industry standard for connecting XR consumer electronics devices to vehicles. Privacy and data protection are fundamental principles guiding the BMW Group’s digital feature development, ensuring customer data is securely processed.
