🤐 SPOILER ALERT 🤐

In January 2019, a creative team held a development meeting in North London with the stated aim of brainstorming new approaches to performance capture. Due to its complex setup and lengthy editing process, performance capture has mostly been the plaything of scripted films or television shows. But this team wanted to apply it to an unusual medium: reality television, a format that requires near-constant filming and tight timelines. Could they pull it off?

The team quickly settled on creating a show around dance, recognizing that its emphasis on precision and elegant movement would mean an exciting application for that technology. But how could performance capture augment an already thrilling art form? Appreciating that performance capture inherently anonymizes the people behind the movement, the creative team cooked up a unique competition show.

In Dance Monsters, amateur dancers execute impressive routines without the fear of being judged on their appearance alone. The three judges (Ne-Yo, Ashley Banjo and Lele Pons) sit in front of a live studio audience and a real stage. As the dancers perform their choreography on a separate, custom-built performance capture stage, video representations of the monsters appear on the live stage flanked by in-person backup dancers. All of this is possible thanks to a cutting-edge, virtual production setup.

In order to create Dance Monsters, producers relied on two renowned production companies: Imaginarium Studios, or IMS, and Mo-Sys. IMS is a performance capture studio that primarily works with film, TV and video game projects, and it provided the motion and performance capture for the show. Mo-Sys brought the monsters to life on a stage in front of a live audience and judging panel.

IMS first built a “custom volume,” or a custom stage, for Dance Monsters that exactly matched the stage in front of the live audience. It was critical they got both the stage’s size and shape right, otherwise the movement of the computer-generated monsters would’ve looked unnatural. The decision to build matching stages overcame “massive issues with going from real to digital,” says Glenn Kelly, IMS’ head of production.

IMS employed a Vicon setup with 73 small cameras mounted around the performance capture studio. Those cameras tracked small sensors, or markers, that producers placed all over catsuits worn by the dancers. Donning headsets outfitted with custom-built, head-mounted cameras and dotted with performance capture sensors, the dancers were able to translate their exact physical movements into code; by sensing the spatial relationship between those sensors, the Vicon cameras were able to re-create the dancer’s body in 3D.

Five large cameras filmed the show, and those cameras were not static: They swooped around the performance capture stage, recording every pirouette and two-step. Mo-Sys needed to track the locations of those cameras exactly, and in order to accomplish that, they used a system dubbed “star tracker technology.” Each of the cameras had a sensor mounted to its top, and those sensors communicated with reflective markers scattered across the performance capture studio’s ceiling (“stars”). The “star tracker,” a digital feed attached to the camera, received highly precise locational information from the stars, and editors used that information to pinpoint the position of the camera within the motion-capture studio.

All of this information was then poured into a motion builder, in which editors made necessary corrections to the monster’s interaction with the space. (Those corrections could be anything, from the way the monster interacted with the floor under its “feet” to smoothing out the movements of the monster’s body.) The resulting data then was processed with Epic Games’ Unreal Engine, a graphics engine originally developed as the foundational technology for video games. All of this meant that the production team rendered footage in real time, so the dancers, judges and crew could see a rough version of the final product as it happened.

“We were essentially doing what we would call post [production] instantly throughout the show,” Kelly says, maintaining that the streamlined performance capture process was the real innovation of Dance Monsters. Many of the dancers were outspoken about the way the monsters impacted their dancing. The dancer who took the stage as Roberta, a boxy but smooth-moving robot, described feeling like she had an “alter ego” during the competition. “You can push more, you can express yourself, you can be risky,” she said. “You can be over the top and really animated.” When many of them were starting out, the dancers had felt that they were judged on their appearances or body types. But taking the stage as their monsters freed them from those feelings of insecurity and self-consciousness. “When I’m dancing as Jam,” said one contestant, whose avatar is a furry red behemoth with curling horns, thick glasses and a friendly smile, “I’m the most confident I’ve ever felt in my entire life.”