Hand and Full Body Tracking in Virtual Reality

Last week was focused on preparing a demonstration showing full body tracking. We took time to go through multiple iterations, test different configurations, and learned a lot. Today this post is about sharing this knowledge with you !

 

 

What is Full Body Tracking ?

Full body tracking in Virtual Reality often means tracking the feet, on top of the hands, usually tracked with the controllers, and the VR headset.
The main question you should ask yourself is “do I need full body tracking ?”. And some subquestions like “do I need hand tracking ?”, “do I need to see my body ?”. Here are some pointers to help you find answers !

I see my feet !

Watching your feet move as you move them is, 95% of the time, simply a “Nice to have” feature.
Think about it : in life, how often do you look down on your feet ? And how often do you use them for anything else than walking ? In my case, almost never. Therefore being able to look at your foot in VR is often something you play around with for 30 seconds… but not much longer. Is it worth this extra work : setting up the feet tracker on each player, making sure they are always charged ? You’re the judge !

Still, my comment doesn’t always apply: you will absolutely want foot tracking in games that requires to interact with your feet (a football game or a dance game). Same for something we often see in VR Arcades : play with the players balance and vertigo by having them walk on a plank, high in the air, which is by the way a very nice experience. We have tried this last experience with and without foot tracking in VR LBE, and for this special case foot tracking is a must !

Note that you could still be able to see your avatar’s feet, or at least show them to other players using kinematic only, without foot tracker. Of the course the feet won’t be at the right place, for the user it can feel a bit weird but from experience we have been using it in Toys Uprising game and it plays very nicely, and it is great to have body awarness, even if your feet are misplaced. Once again you don’t look down on your feet that often. And showing misplaced feet to other players is ok as they won’t even know !

We have done two different tests here, one simply with a tracker on each feet, the second with orientation tracker on the knees too. Mostly to find out that trackers on knees are pretty useless, they add very little precision. The knees position can be very well inferred with kinematic equations and feet orientations.

Using your hands

We have been thinking about developing controllers, like Oculus or Vive have their own. Because interacting with the environment is key. But when it comes to VR Arcades, you will often encounter first time users. And a controller is not a natural way to interact with the VR world : player have to learn the controls…

Your hands on the other… hand (best joke of the all post, I know) are the most natural way to interact. This is what we use every day in real world, so when it comes to using them in VR, the learning curve is almost non existent.

We have been working with the LeapMotion in our demo, it works great, as long as it doesn’t gets too close to a tracker’s light. Sadly it is for PC only, and requires a backpack (no wireless option). They are other options available : gloves, though you have to equip the player and keep them clean, but also some equivalent to the LeapMotion with android headset compatibility that we will try out soon 😉

Using the LeapMotion for Hand Tracking

Another issue we encountered with the LeapMotion is that is has a hard time tracking forearms. And extra orientation tracker here, or better kinematic to avoid the weird movements it sometimes outputs would help.

Body Tracker

On the chest we set a tracker without position tracking, just the orientation. That was actually important to have, more than I initially thought. It totally makes sense in the end : you can move your head a lot independently to your chest and pelvis, therefore you need to keep track of both for accurate body tracking.

Kinematic

What make body tracking work is having a good kinematic plugin to calculate the positions and rotations of each body parts from the trackers position. One of the most famous being IKinema, as we were on Unity we preferred using Final IK, and especially its sub plugin VRIK. It took us at least a week to get a proper configuration. It is a big plugin so there is a lot to learn, and we had to integrate different sources of tracking (trackers, leap, orientation only trackers). The result can be seen in the video !

Final Ik Kinematic plugin for Unity

 

Conclusion

I hope this post helped you learning more about body tracking in VR. If you agree, disagree, or want to add something to our comments don’t hesitate to say so in the comments below or start a discussion in the Discord =)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>