I can still remember my grade school self hunched low in the glow of the TV screen, muscles braced & straining, thumbs aching as I defended the world from space invaders and incoming asteroid attacks. Atari’s joystick controller was my small body’s extension into the video-game universe, while the rest of me remained slouched in the living room cushions. Although I graduated to a Sega Genesis, my brother’s first Nintendo, brief dalliances with Myst, and a few frustrating marathon sessions with Sony Playstations, I can’t say I ever did become a hardcore gamer. But I certainly knew the pain of controller induced carpal tunnel syndrome.

I took note of cool developments in gaming technology and culture as years went by. I admired the perseverance and ingenuity of both designers and players, but weren’t we all still stuck at a desk or in an easy chair, tethered to the console by a wire and a controller?

When the Wii game system with its wireless Wiimote was introduced, gamers finally got off the couch to physically interact with their favorite games. You didn’t just press a button to swing a digital golf club, you had assume the STANDING position, put the device in hand and ACTUALLY SWING YOUR ARM. The Wii remote uses an accelerometer that detects and measures speed of motion and infrared sensors to gauge the device’s position in space when you point it at the sensor bar on the console. Wii’s peripheral controllers allow players to use more than their thumbs to manipulate objects, shoot lasers, play sports, dance, do yoga or conduct an orchestra.

Although the Wii remote’s been hacked to perform non-gaming tasks, like controlling a robot lawn mower, I’ve been noticing how programmers and artists exploit the Wii’s motion control capabilities in their work. The Wii remote devices seem to be especially popular with those who use Ableton Live, Max/MSP/Jitter, Serato Live or similar programs for live audio and visual mixing.

For instance, take one of my favorite digital artists out there right now: Daito Manabe, is a composer, DJ, visual artist and programmer based in Japan. His work is hip and high tech, infused with playfulness & beauty. Collaborating with dancers, museums and all kind of designers, he’s made faces dance and buildings sing. In 2006, Manabe developed a method of using the Wii remote to trigger audio and visual samples on the computer, which he uses in his DJ/VJ sets. Here’s an example using his laptop’s built-in iSight camera:

To me, NYU’s ITP Program is the place for cooking up interactive devices, and no doubt the Wii controller has figured in the work being done there. At the department’s 2008 Winter Show, Si Chan and Ioana Staicut allowed users to release their inner Jackson Pollock by using a Wii controller on a projected canvas on the floor. The piece, called “Dra”wii”ing Jackson Pollock”, is demonstrated in this clip.

In an odd mix of disciplines, Jason Cayabyab uses the Wii remote to combine skateboard culture and video art in a project called “Wii Are Board”. He embedded the Wii controller in an actual skateboard and uses it to manipulate Quartz Composer, and the Open Sound Control program, Osculator. The user’s steering, jumping and balancing movements on the skateboard control various video elements, such as camera angles, filters and a drawing program using colored points of light.  Check out his project demo HERE. Bet Tony Hawk can’t do that.

Visual and performance artist, Rashaad Newsome, premiered a work called “Shade Compositions” at The Kitchen earlier this year. The Kitchen described the piece thusly:

“..a live performance featuring a chorus of more than twenty black women…Newsome divides his performers into groups akin to instrumental sections as they enact his choreographed sound score made up of repeated sequences of culturally specific or stereotypical gestures, movements, and vocalizations. Newsome simultaneously records, loops, edits, and remixes in real-time the audio and video documentation of the performers using a hacked Nintendo® Wii™ game controller…”

Just Google “Wii remote” and you’ll find all sorts of multimedia experiments like these. But if the Wii and other controllers like Sony’s Magic Wand can be used by artists in a variety of applications, what about a gesture and voice based gaming system that uses NO controller at all?

Xbox’s Project Natal employs advanced camera technology, 3D depth sensors, voice & face recognition – people have been calling it “the Wii-killer”. There are no remotes, no wands or dance pads. YOU are the controller. The body need only move and speak to manipulate images and sound. So I imagine Natal’s future artist/hackers will take us far and away from those old joystick memories.

– Jocelyn