Earlier this month, passersby were greeted outside the northwest lobby windows of 41 Cooper Square by a dynamic virtual sculpture, generated in real time based on their movement. The exhibition, displayed on a large digital screen facing the street, is the first of a two-part project titled Marionette, developed through a collaboration between Richard Yurewitch, a senior in the School of Art; Ricky Lam, a mechanical engineering sophomore; and Mili Shah, associate professor of mathematics and the 2021 C.V. Starr Distinguished Professor of Engineering.

The interactive installation combines the group’s work across art and engineering as part of the Autonomy Lab and is an extension of the lab’s Vertically Integrated Project research on body-tracking technology, led by Shah. The Marionette project also received support from the 2021 Cooper Union Grant program, which awards funding to student, faculty, and staff initiatives that help advance The Cooper Union’s goals and strategic priorities, including sustained multidisciplinary collaborations.

“Working with two people from the engineering school helped me realize the importance of teamwork,” says Richard Yurewitch, who arrived at Cooper not expecting to write code and was surprised when Shah approached him about the idea of translating movement to data in the digital space. “Mili’s prompt and the promise of commingling my art practice with code motivated me to begin sketching ideas,” Richard says. Ricky Lam found similar opportunities to broaden his thinking: “Working on this project has extended my knowledge of robotics beyond anything else I have worked on so far at Coper Union and has provided an avenue for me to explore creative expression.”

Professor Shah and the students designed the weeklong public exhibition to showcase their research using a body tracking system called BlazePose. Capturing bodily motion through a webcam, the system mirrors viewers’ movements via a 3-D rendering of black lines and spheres against a white background displayed on the screen. As people in front of the camera move their bodies, the digital sculpture responds in real-time. In addition to the interactive visualization facing the street, the Marionette exhibition was livestreamed to external viewers through a video feed website. “Adding this information draws attention to the way our data is used and transmitted by third parties,” the group explains. “It will hopefully cause surprise and bring engagement to the space.” The group has also created an online simulation of the body tracking project.

The second part of the Marionette project, which is still being planned, will consist of a physical construction akin to the virtual configuration of spheres generated in their exhibition. Like the digital version, it will respond to movements using BlazePose, changing the height of the spheres to match the body positions of its viewers.

Both students see this work as a testament to the immense potential for cross-disciplinary exploration at Cooper. “Being supported by two people from backgrounds vastly different from my own—an engineer and a mathematician—helped me realize the versatility, presence, and importance of various perspectives in my work, and how crucial they are in informing the ideas, practice, and functionality of a work of art that is resolved through code,” Richard says. His mechanical engineering collaborator, Ricky, agrees: “An engineer can help an artist realize their ideas, no matter how ludicrous, while an artist can help create a product with both form and function. There needs to be more interaction between the schools. I’m so grateful to Professor Shah for facilitating this. I hope projects like this become a trend and I see more of this in the future.”