So, since the holiday season, have you been flailing around in your living room, throwing imaginary footballs, jumping over invisible obstacles or throwing punches in the air? If you have, you probably got Microsoft’s Kinect for Xbox under your tree, the gesture based gaming system that requires no controller to play.
The Kinect’s sensor uses a camera, a depth sensor and a microphone array to track your movements, scan your environment and listen to spoken commands. The special software employs face and voice recognition and 3D motion capture, transforming you into an onscreen avatar that can fully interact with the characters and action of the game.
While the Kinect was still in development under the name Project Natal, I knew it wouldn’t take too long for programmers and/or artists to figure out a way to make use of its open USB port and create other kinds of drivers that would read input from the sensors. Microsoft says it basically welcomes the experiments that started cropping up so quickly after the system’s release last fall. Beyond gaming and other media industries, the possibilities for Kinect’s use in the creative process and in multimedia performance seem pretty clear. Let’s take a look at what some of the Kinect hackers are up to besides virtual bowling:
Last month at a meetup of the Boston Ableton users group, the crowd got a demo of Kinect being used to send midi information to Ableton Live. Here, the user waves his arm up and down to control a “wobble” effect in the music:
Hirofumi Kaneko made this line drawing program using OpenFrameworks and the Kinect, creating a pencil sketched avatar of himself that moves as he does:
Ryan Monk developed some painting software for use with the Kinect. Here you can watch him moving his “brush” in the air as the artwork takes shape onscreen:
Here’s the next step in VJing with Kinect, controlling images or graphics by dancing along with the music. This example uses an open application called TUIO, which was originally created for interactive or multi-touch surfaces. It can now track specific hand gestures, and helps the Kinect “speak” with the visual software, OpenSoundControl:
Here is some digital puppetry created with the Kinect connected to a MacBook and real-time animation software called Animata. It’s part of the Virtual Marionette research project from grifu.com. :
Looking at all these Kinect hacks sprouting up all over the place, it seems that creating or editing digital media won’t just mean butt spread and carpal tunnel syndrome. We’ll rise up from our workstations and learn to control the arts with the rest of our bodies in some new, though sometimes silly looking, ways.
– Jocelyn
The MMiXdown 