Realtime Streaming of Mocap Avatars

My fascination with avatars is at the union of motion capture and real-time streaming. The use of motion capture for recorded character animation is well established (see Golem and his long line of predecessors and followers), but I find it most fascinating when motion capture is used for live control of remote characters, in one to one settings and in broadcast mode.

At SGI, we pioneered this type of character teleoperation. One project was the advent of "Captain Stanley Vermle." Our mascot for the launch of Cosmo Player 2.0, we created the a teleoperated version of the character in cooperation with SimGraphics and Viewpoint. Primarily for tradeshow use, we remotely operated Stanley using a facial motion capture rig and a 3D head tracker along with streaming audio. The effect of Stanley's head addressing the crowd interactively was striking.

Not wanting to stop there, we were inspired by the idea that streaming 3D characters could become a new, more efficient broadcast medium - including the ability to broadcast live! To that end, we worked with Mark Pesce and Jan Mallis, then of Blitcom, to take their Bliss.com character and send her streaming across the Internet. First stop was Siggraph 97, where the actor in a mocap suit and mic would control the character and voice. The motion data and audio were then processed, encoded, and broadcast out to multiple stations around the show. 

Concurrent with Siggraph 97, Bliss made a stop in Santa Monica for a live collaborative performance with a human dancer at the Electronic Cafe. With Bliss controlled from across Los Angeles at the convention center, and Mona Jean Cedars dancing on video at the E-Cafe, the two intertwined virtually breaking down the barriers of distance and reality.

Not wanting to stop there, we pushed for another first - a Trans-Pacific 3D broadcast. In what was dubbed the "VRML Matsuri," around 100 devotees gathered at the Cybernet Cafe in Harajuku in the evening of September 20, 1997. I was in Tokyo managing the receive side with Shinya Matsuoka, while Mark, Jan, and the rest of the team worked pre-dawn in Mountain View on the sending side. Since this was still early days for broadband and long-haul Internet, we used an ISDN line for the last mile connection to increase reliability (with a whopping 56Kbps connection). The result was a great success.