Pulling Animation’s Strings

Puppetry is a truly ancient craft. With some five millennia of history as a practiced art fom, it can make the history of the video game seem rather trivial in scope.

It’s a form of entertainment traditionally free from technological complexity. As such, it doesn’t seem like game developers have much to learn from puppeteers, beyond universal insights into characters and storytelling.

Inside a vast studio space in Los Angeles, however, an iconic entertainment company is doing a great deal to close the gap between the approaches of game development and puppetry.

The Creature Shop – itself part of the family run Jim Henson Company – has been perfecting an approach it has named digital puppetry for many years. In combining mo-cap, puppetry, performance and game technology, it hopes to bring a little more of the human touch in to the digital realm.

The Henson Digital Puppetry Studio is a virtual production facility. Here motion capture convention is split in two, to allow teams of puppeteers to perform in unison and have their movements captured by a game engine before being sent out to various other digital production tools. That split divides a character at about the neckline, with one performer enacting body movements, while a separate puppeteer operates facial gestures.

We’ve had to create input devices very similar to game controllers 

Steffan Wild, Henson Digital Puppetry

STANDARD OPERATING PROCEDURE

In a typical shoot, a performer – who may or may not be strapped into a physical pupper– moves as directed on stage, watched by an array of motion capture cameras. Meanwhile, offstage, a puppeteer uses two custom hand controllers – usualy a flight type stick and a waldo – to manipulate the facial muscles and head movements, just as they would on a physical contemporary puppet. Ultimately the concept offers a mo-cap method where body performer and head puppeteer movement is captured as date fed into a custom Henson game engine.

“The digital puppetry concept comes from the fact that a traditional puppeteer works in a real-time environment,” explains Steffen Wild, head of digital production and director at the Henson Digital Puppetry Studio. “What that means, of course, is that they are working with a physical puppet in front of a camera. It’s a live performance, with spontaneity and so on, where the puppeteer gets immediate real-time feedback.

“What we at Henson have had to do over time is bring that tradition of puppetry into the digital realm, but retain that real-time feedback for the performer. 

"So we’ve had to create different input devices very similar to sophisticated game controllers. But more than that, we’ve written our own game engine to work with the puppeteer.”

Ultimately the technique is a distinct spin on the conventions of motion capture. But, says Wild, it offers something more. Something no amount of key framing and rigging can rival. Great things can be done by a sole motion capture performer impersonating a creature with a wildly distinct physique to their own, of course, but that isn’t the only approach available today.

“The advantages of digital puppetry goes back to traditional puppetry,” Wild offers. “Back when Jim Henson crafted his approach to the medium in general, the consensus and the paradigm was ‘anything that moves in front of the camera only moves because it was moved or performed by a human hand’. That same concept extends into the digital space. Anything that moves only moves because a person touched it. The human element is integral to digital puppetry.”

And, says Wild, digital puppetry can apply that in a rather unique way, lending the puppeteer’s craft and the finesse of the human hand where desk-bound animators or an actor’s facial limitations have different contributions to make.

“This concept allows for human spontaneity in the digital realm,” he continues. “Because of the human element, digital puppetry is a very spontaneous and creative process, that can lead to things that we didn’t anticipate. That can be great for the director, who can decide on the day what works. Something that was a mistake in a puppeteer’s performance might turn out to be integral for a shot, and that’s the human element we hope to capture.

“Now we can capture something else through puppetry,” Wild later adds. “I can always see – even if the same puppeteer would perform two different characters – characteristics of an individual puppeteer’s movement and their style. Whether conveying a feeling of happiness or of disgust, the puppeteer behind the creature shines through in their own way. Digital puppetry can capture that, and it’s human and heart-warming to for the audience to connect with that. That emotional connection is a big part of what Henson does, and what visual effects tries to do.” 

Henson was one of the first companies that embraced game technology wholeheartedly. 

IS IT A GAMES ENGINE?

A means to harness the skills that have let The Jim Henson Company connect so meaningfully with so many audiences for almost 60 years makes for a powerful proposition. But that doesn’t explain away all that talk of a ‘game engine’.

Let’s be clear. The Henson team didn’t develop this technique or technology with games specifically in mind. But they stop short of branding their solution a ‘puppetry engine’. In part, the answer to that naming convention is that Henson have long strived to transcend the divisions that separate disciplines and mediums.

“Digital puppetry is de facto a virtual production stage that is powered by a game engine,” Wild asserts. “So in that respect it could be applied to all the disciplines in this space, from traditional visual effects to games. I think Henson was one of the first companies – because we’ve been working with approaches like this for over a decade – that embraced game technology wholeheartedly. A game engine is a core part of what we do. I have many conversations with other industry professionals and leaders, in terms of looking at where the animation or visual effects or games business might go, and one thing that doesn’t take me or the Creature Shop any real thought is the idea that we are all moving towards real-time more and more.”

But the engine created at Henson doesn’t just bear a loose comparison with game-specific middleware. The heart of digital puppetry is what Wild calls a ‘full-blown game engine’.

“We wrote our own, starting about a decade ago,” he says. “The main reason we did have to write our own game engine was the fact that we wanted to get data streams out of the game engine into traditional packages such as Maya in very specific ways. Outside of that, the technology is similar to what you might find in Unity or Unreal. But from the company standpoint, we completely acknowledge that we’re not in the business of yet writing – or offering to the market – another game engine.”

However, that doesn’t mean that Henson isn’t able to use its technology to make games.

MARIONETTE WORTH

One of the other advantages of Henson’s digital approach – which also supports the long-established puppetry method of using several performers to operate one large creature – is that the content it spits out can be applied universally to many media forms.

“This stage that we have – that we’ve called a digital puppetry virtual production stage – is a melting pot for so many disciplines coming together. We see it influencing the types of characters we are creating. In the past we had characters that used different resolutions, so a character geared for films and a character geared for games and so on. Now we have one character that can move across all the different areas, whether it’s being used in marketing or in consumer products or in games – because we do our own apps and games – or if it is needed in a feature film. We’re doing that right now. This way we can use one character everywhere.”

That defining ability of building a character shaped by the human hand that is ready to leap from medium to medium is indeed a powerful idea, especially at a time when developers large and small are increasingly looking to build brands that transcend the game form. 

As such, digital puppetry is a technique every games maker with an interest in animation should investigate. Henson isn’t known to be preparing to serve game developers with this approach, but the are a company committed to knowledge sharing across creative disciplines. And considering how long those involved have been charming the world with their character creations, it seems game studio have as much to learn from puppeteers as Henson have taken from game middleware.