Kinect retargeting

The Kinect sensor promised us a higher level of immersion, with one-to-one control over the motion of our game character, a concept Microsoft dubbed ‘avateering’.

In practice few titles provide this kind of direct mapping. Partly this is because gamers don’t always want hero characters to track their motions – they want them to track their intentions. But it’s also because of the difficulty in retargeting the tracked player, with all their variability in size and shape, performance style and living room clutter, to the very different game character shape and posture.

In the Kinect integration for morpheme, we at NaturalMotion think we’ve shown that it is possible to realise the avateering concept as originally envisaged. Perhaps equally important, quality retargeting onto a uniform character opens up a much more natural and flexible way of defining gestures to be used to control your title – as poses and animations of the game character itself, rather than the infinitely variable player skeleton. Here are some of the main techniques we used.

FILTERING

We use a Butterworth filter to smooth away noise, both on the original tracking data and on the joint rotations of the retargeted animation. Just as important are tools to transition into default animation when the tracking quality becomes poor, which can be used to handle glitches – as well as to subtly train the player to limit their actions to those the sensor can track well. You can also clean up glitches by ignoring large displacement of a tracked joint if it is too sudden.

OFFSETS

There are signifant discrepancies between the points on the player’s body tracked by Kinect and the location of the equivalent joints on a typical game rig. In particular, the ‘hips’ joint Kinect tracks is actually somewhere in the player’s abdomen. To avoid the need to change the game rig to fit the tracker, we define offset geometry which transforms the game character’s joints to the position of their Kinect equivalents. Retargeting then takes place on the transformed ‘offset rig’, which is converted back afterwards.

Offsets are essential for getting a good quality mapping without having to change your rig. They are also useful for handling rigs with different postures. For instance, when playing as a dinosaur character we don’t want the dinosaur standing tall like the player; the offsets mean the retargeter correctly interprets the player standing upright as the dinosaur’s natural stooped posture.

MAPPING

The underlying algorithm is to match the orientation of the tracked bones (lines between joints) to your game character – we can’t match joint orientations because Kinect only provides world joint positions, not rotations.

Treat the retargeter as a filter on an incoming animation stream, adjusting each bone in turn to the right orientation. Where there is ambiguity, blend the orientations implied by each child bone – the hips are oriented by looking at the relative positions of the player’s root, spine, and left and right hips, for instance.

One important point to note is to treat the shoulders and neck of the tracked skeleton as parented to the spine joint. Bundled software always draws the shoulders connected to the neck, which is misleading.

WEIGHTS

Do your mapping with geometry represented in the world frame, and convert back to local space at the end, starting at the root and working towards the leaf joints. This allows you to use weights to shift rotation from a joint to its children. This is particularly useful for the hips, which will otherwise rotate when the spine ought to be bending instead, due to the way the hips are positioned on the Kinect skeleton.

RESAMPLING

To handle different character topologies we use interpolation to spread rotation to unmapped joints. For instance, since Kinect tracks only one ‘bone’ for the torso we would get out rotation only at the base of the spine; but with resampling this is spread out along all the spine joints in a user-controlled way.

Morpheme 3 available now for PS3, Xbox360, Wii, PC and NGP. For more information contact paul.topping@naturalmotion.com.

www.naturalmotion.com