When Kinect was first revealed to the public, it was seen by many as a way of opening the doors of a traditionally hardcore gaming platform to the casual masses. The device’s launch line-up, dominated by dancing and fitness titles, compounded the sense that Microsoft’s motion tracker was one for mainstream games design.
Then, come release, something happened. All over the world homebrew frontiersmen got under the casing of the device, eager to see how such a cheap peripheral could seemingly do what was previously the reserve of well-equipped laboratories with enormous budgets. Quickly researchers and hackers joined their ranks, taking a microscope to Kinect’s dismantled innards.
INTO THE VOID
It turned out cracking open the outer shell and voiding all those warrantees was worth the effort. When open-source software solutions – which allow Kinect output to be interpreted through a USB connection – were made public, the floodgates were blown apart.
Since, new Kinect ‘hacks’ are an almost daily occurrence. From bedroom coders augmenting reality to academics using the device to give flying machines eyes, the breadth and scope of what has been done with a standard Kinect and a bit of ingenuity is quite incredible. Somehow, a device aimed at living rooms the world over has become a poster child for natural user interaction (NUI) researchers, roboticists, artists, game developers and hackers out to have some fun.
Josh Blake is a man who knows why Kinect has been embraced by this diverse range of technophiles. He is the founder of the OpenKinect ‘open source community’, a collective of likeminded innovators focused on the ongoing development and maintenance of the libfreenect software, which is a core library for accessing Kinect
through its standard USB-out.
Currently, libfreenect supports Kinect’s RGB and depth images, motors, accelerometer and LED, and by the time you read this, it may even have access to the device’s audio functionality.
Blake, who is also a proactive NUI evangelist, has come up with a phrase to the appeal of Kinect; ‘democratic magic’.
“To put it another way, it’s the democratisation of magical technology,” says Blake. “The magic of the how the sensor works is just beyond our intuition, even when the other side of our brains understand the underlying math and science.
“In addition, anyone can buy a Kinect from their local gaming store, download free and open source drivers, and start creating. It’s an extension of the personal computing movement where anyone can publish and be heard, but in this case anyone can experiment with innovative interactions. It gives us hope that we can create something amazing and that the people have a chance to make a difference in the future. That’s democratic magic.
It’s an enchanting notion, and it sheds light on the fact that the Kinect hacks scene is about more than playful re-appropriation.
While it’s great to see a programmer controlling Angry Birds with his hands, or a YouTube video of a virtual invisibility suit, those achievements are in fact part of a far greater good that could eventually serve to better the mainstream game design that inspired Kinect.
But before we get ahead of ourselves, what exactly is going on under Kinect’s plastic skin that makes it such an appealing prospect for the experimental?
UNDER THE BONNET
Mark Bolas believes it is because Kinect offers an available, affordable solution that replaces the expensive equipment previously needed for the exploration of NUI interfaces.
Like many of those branded as Kinect hackers, Bolas is in fact an academic. The associate professor at the University of Southern California’s School of Cinematic Arts, Interactive Media Division and director of the Mixed Reality Research Lab at the Institute for Creative Technologies has, together with the lab’s post Doctoral researcher Evan Suma, been using high-end motion capture technologies to create virtual environments for training applications.
When the researchers adopted Kinect, and released a video where they used the peripheral to control World of WarCraft as a demonstration, the footage became one of the most famous Kinect hacks.
“It is interesting, on one hand a stodgy academic could say that there is nothing new here – the algorithms have been developed using high-end motion sensing technologies” says Bolas of Kinect’s unique appeal.
“On the other hand, the price point and form-factor change everything.”
Garrat Gallagher is another researcher whose work with Microsoft’s platform has become one of the most renowned Kinect hacks, despite its academic grounding. The systems robotics engineer at MIT’s Computer Science and Artificial Intelligence Laboratory has created a system that apes the infamous technology from the movie Minority Report to allow remarkably fluid, intricate gesture finger control that far exceeds anything seen in a game, and he’s done it with an affordable game controller.
“The accuracy is quite amazing for the price and, within the ‘sweet spot’, Kinect has resolution that rivals or surpasses any other ranging system on the market,” suggests Gallagher.
“Part of the reason Kinect is so good is that it does a lot of processing on board. This lets Kinect deliver an amazing out of data: around 300,000 points at 30 fps. This amount of data puts new challenges to researchers – to use this data in a timely manner, algorithms must run extremely fast.”
Kinect also offers researchers something perhaps no other device does; a factor so simple it’s easy to ignore.
“Because it’s a device that is already present in millions of living rooms, this has fantastic implications on the impact of our work as researchers,” says Suma.
“For example, it’s one thing to produce a game-based rehabilitation tool that patients can use when they visit the lab, but when they can take it home and augment their treatment by playing with their friends and family, that’s really revolutionary.”
Whether you’re researching NUI at a world leading academic research lab, or conceiving an interactive installation in a public space, the fact is that Kinect’s user-friendly nature means the people you’re trying to reach don’t need to be engineers or programmers, and that makes it all the more appealing as a subject for hacks.
The very word ‘hacks’, however, may be holding back the progress being made by the likes of Gallagher and Bolas. The reality is that a huge number of the so-called hacks are academic endeavours, or homebrew contributions to technological progress. Yet despite this, Microsoft declined to comment when contacted for this article.
Similarly, PrimeSense, which worked with Microsoft on designing Kinect and currently contributes to Open NI, is keen to make clear it does not support, cater for or condone Kinect ‘hacking’.
Open NI is an industry-led, not-for-profit organisation formed to certify and promote the compatibility and interoperability of natural interaction devices, applications and middleware, and makes available the open source OpenNI framework, an API for writing applications utilising natural interaction.
The parallels with the Kinect hacks scene are obvious, but the distance between the two is made absolutely clear by PrimeSense.
A BLIND EYE
Elsewhere, Microsoft has also previously recognised OpenKinect’s Blake as a ‘Most
Valuable Professional’ for his work in the Microsoft Surface and NUI communities, but seems keen not to be seen as approving of the extra-curricular work done with Kinect.
Still, without officially recognised support the Kinect hacks scene thrives. Affordable, accessible and supported by a community of creators happy to share the fruits of their labour, Kinect is a device perfect for a hacks scene. It’s so perfect, in fact, that those large scale developers making boxed games for the device have been watching.
“I think the Kinect hack community has leapt on this for the same reason we’ve been singing its virtues for the last 18 months,” says Blitz Game Studios design director John Nash.
“It’s the flexibility, the innovation, the potential for doing genuinely new and creative things with it. The fact that the device itself is software driven has obviously helped that process but some of the hacks we’ve seen have amazed us all.”
Which begs the question, will studios be impressed enough by what they see from the Kinect hacking community to adopt some of the techniques for their own titles? Perhaps the scene will do the unthinkable and take things full circle, influencing mainstream game design.
“I bet it already has,” suggests Blake.
“Figuring out how to make these interactions work is a very hard problem. If you look at the first wave of Kinect games, there are couple aspects of the menus and interactions that don’t work well and a couple that work really well.”
With the OpenKinect community experimenting and innovating with many new and different types of interactions, it doesn’t matter that not everything they conceive will work in reality. In ironing out the kinks, and test the boundaries of possibility, Kinect hackers are doing much of the work for traditional games developers, and all in plain sight.
“I’m sure that mainstream game designers who are working on the next wave of Kinect games have been watching the community closely and have already incorporated some new ideas into their games,” says Blake.
“We ignore the gamers and enthusiasts at our peril,” says Nash, confirming Blake’s suspicion.
“It’s easier than it’s been for decades for those at the grass roots level to get their hands dirty and create fun content for their favourite hardware, and we’ve always tried to listen to and engage with that indie community.
“It’s much easier for them to break down gameplay conventions, and although we have other considerations or restrictions to contend with when we develop big-budget commercial games, I think it’s inevitable that we’ll see the mainstream influenced by some of this work.”
THE INNER LIMITS
To paint too optimistic a picture, however, would be unrealistic. Kinect imposes numerous limitations on the hackers and academics that have embraced it.
Patrick Bouffard, a researcher at the University of California, Berkeley’s Electrical Engineering and Computer Sciences department, has adapted Kinect to provide the eyes for a quadrotor helicopter, which has captured the attention of the world’s media.
But it isn’t just a cool flying robot. It serves as an experimental test-bed for more theoretically oriented research in autonomous control systems.
Bouffard’s creation, which uses Willow Garage’s open source Robot Operating System, has also taught the researcher a thing or two about the challenges of working with Kinect.
“One thing is that the field of view is limited,” he explains. “It isn’t designed to give you a wraparound 3D view of the world. So if you only have a single Kinect onboard it’s like your robot is flying around a darkened room and can only see what’s in the beam of a flashlight that points forward. But limitations are also a good thing in the sense that they drive advances.
That particular limitation, for example, might motivate us to ask how we can make the quadrotor fly so it looks around itself in different directions, often enough so that it remains safe.”
Another limitation also comes from one of Kinect’s greatest strengths. As mentioned previously, the device produces lots of data.
While providing a fantastic opportunity for those harnessing the power of Kinect, using all that data requires a lot of CPU cycles.
“It also is hard to separate the Kinect from the graphics engine and software,” says Suma.
"Speed and accuracy are always concerns. In terms of accuracy, I’ve been impressed with how good the tracking can be, though it’s still not going to measure up to the sophisticated motion capture systems used by the film and video game industries, which can get submillimetre precision. I hope that this will be improved in next-gen sensors.”
There are also other areas where Kinect struggles that those familiar with its inner workings are tackling with software solutions; issues such as improving tracking of the parts of the body that the Kinect has trouble with, such as head orientation and arm twisting.
Suma and Bolas are tackling these very issues as they improve FAAST, a general purpose toolkit that enables anyone to harness the potential of low cost body tracking that, in the spirit of the community spirit that defines the Kinect hacking scene, is free.
So far FAAST has been downloaded 40,000 times. With figures like that, it’s easy to imagine the Kinect hackers are starting a revolution. There’s eight million of the game gadgets in the public domain, and some are already being used to operate surgical machinery, read sign language, guide robots, project art installations onto walls and control high-end creative software.
The potential is enormous, and considering the current rate of progress, in a few short years it could be possible that just about all of our interaction with technology takes place under the watchful eye of Kinect and its inevitable imitators. You might even be using it to interface with your game engine.
Click here to see our our top-5 Kinect hacks in pictures.