2 New prototypes demonstrated by the Meta research team

The Reality Labs Research team specialising in display systems will be presenting two new demonstration prototypes at the SIGGRAPH 2023 conference in Los Angeles from 6 to 10 August.

First prototype "Butterscotch Varifocal": This headset combines the varifocal technology of the Half Dome series of prototypes with a retina-resolution screen.

Second prototype "Flamera": A computational camera using light field technology for a virtual reality experience without reprojection.

Both devices are at the research stage and are not intended to be marketed. The aim here is to test new technologies to advance research.

Butterscotch Varifocal: focusing on the finest details

Today's headsets immerse us perfectly in virtual worlds, whether highly stylised or photorealistic. However, they are currently limited by a fixed focal length. Anything within about 1 metre of your eyes appears clearly, but if you're holding something close to your face, most people are unable to focus with complete clarity. Fortunately, most VR developers take this into account, keeping the action at a comfortable distance to keep everything in focus.

But what if you're using VR for work and need to read text on a virtual screen? Or if there's an interesting object you'd like to inspect up close to appreciate all the little details? That's where variable focus comes in.

By taking advantage of eye-tracking technology and moving the screen closer or further away from your eyes depending on where you're looking, the system allows you to focus at different depths for a more natural, realistic and comfortable experience. Combined with a retina-resolution screen, you get crisp, clear images that rival what you can see with the naked eye.

Flamera: A new approach to Passthrough

Virtual reality is ideal when you want a totally immersive experience, like being cut off from the real world to live in new worlds, enjoying a totally virtual interactive story or watching a film on the big screen like at the cinema.

But there are also times when it's useful to be a little more connected to the outside world, for example when something unexpectedly enters your game space or if you want to introduce virtual content into your physical environment. This is the case with mixed reality (MR) and "passthrough", which allows you to see a digital recreation of the physical world inside your headset.

This is something that modern headsets like Quest Pro do well (and Quest 3 will do even better). However, today's passthrough relies on helmet-mounted cameras, usually a few centimetres away from where your eyes are. This means that the cameras capture a different view to the one you would have if you weren't wearing a helmet. The images can be computer reprojected to get the 'correct' view, but this can lead to visual artefacts. And even if the cameras were placed directly in front of the eyes, the view would still be offset because of the thickness of the headset.

"To meet this challenge, we thought about optical architectures capable of directly capturing the same rays of light as those seen by the naked eye. By starting from scratch to design our helmet instead of modifying an existing model, we ended up with a camera that looks completely unique, but which delivers better image quality and lower latency." | Grace Kuo - Research Scientist

Unlike a traditional light-field camera with an array of lenses, Flamera (think 'flat camera') strategically places an aperture behind each lens in the array. These apertures physically block unwanted rays of light, so that only the desired rays reach the eyes (whereas a traditional light-field camera would capture more than these rays of light, resulting in unacceptably low image resolution). The architecture used also concentrates the sensor pixels on the relevant parts of the light field, resulting in much higher resolution images.

The raw sensor data ends up looking like little circles of light, each containing only part of the desired view of the physical world outside the helmet. Flamera reorganises the pixels, estimating a coarse depth map to enable depth-based reconstruction.

All this results in a view of the physical world through the helmet lens that is closer to what the eye would naturally see, with fewer artefacts than commercial helmets available on the market today and with higher resolution than traditional light-field cameras. This paves the way for even more realistic MR experiences that seamlessly blend virtual content with our view of the physical world in the future.

Source : https://www.meta.com/en-gb/blog/quest/reality-labs-research-display-systems-siggraph-2023-butterscotch-varifocal-flamera