Facebook wants to bring your entire body into virtual reality

Posted at 6:15 PM, May 01, 2019
and last updated 2019-05-01 21:23:51-04

Facebook is working on full-body virtual reality avatars that look and move just like real people, in hopes of making the technology feel even more engaging and immersive.

The company is making and animating anatomically correct models of the human skeleton and overlaying muscle and skin, it announced at its annual F8 developer conference Wednesday.

It is tracking how humans move in real time and replicating their movements in VR. The company is also reconstructing and simulating the actual clothing a person is wearing, using physics-based software to figure out how clothing should move virtually when they do things such as dance or stretch.

Current avatars in VR apps are quite limited, though they have improved in recent years. Your virtual persona likely only consists of a head and (if you’re lucky) hands, since those are the two areas of the body whose motion can be tracked with existing consumer VR headsets and controllers.

On Facebook’s Oculus Quest VR headset, for example, even if you look down and see an entire body, you can only control the head and hands. Customization options, including those that Oculus offers in its own apps, lean more toward making your animated persona a fantasy or cartoon version of yourself.

Oculus research supervisor Ronald Mallet showed off a video on Wednesday of what he called an early prototype of the next generation of hyper-realistic VR avatars. In it, a man and a woman moved around a big room wearing VR headsets, while nearly identical, three-dimensional, virtual versions of them — down to their jeans and t-shirts — played soccer on a virtual field with a digital soccer ball. As they raised their hands and kicked their legs in real life, their VR avatars did the same with what appeared in the video to be only a slight lag.

Mallet pointed out that this kind of fully-tracked full-body avatar is still far off in the future. One challenge (among many) is that there isn’t currently a way for people to generate these digital versions of themselves with off-the-shelf sensors.

Facebook will also need to determine how best to keep these avatars secure. Mallet said this may mean using facial or fingerprint recognition to connect a realistic avatar to a person.

“We’re still years away from this kind of technology in consumer headsets,” Mallet said.

Facebook said in March that it was making progress at creating three-dimensional, realistic-looking avatar facesthat can match the motions and expressions of their human counterpart in real time; adding the rest of the body would be a logical next step. A video demonstration on Wednesday showed animated faces that appeared almost human.

The peek at the future came a day after the company announced that it would begin shipping its two newest virtual-reality headsets, Oculus Questand Rift S, on May 21. Quest is a wireless, self-contained device, while Rift S tethers to a PC for a more powerful experience.