Chef, Shark and Melody

Hand Gesture Recognition and Interaction

Qiao Li
MDes Mediums 2025
Beiqi Wang
MDes Mediums 2025
Sijia Ma
MDes Mediums 2025
Image
1 Hand gesture recognition for three-game scenes reflecting daily activities.

James J. Gibson once proposed the pioneering concept of affordance, in which he said, “The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill.” Our project draws inspiration from it and utilizes the Teachable Machine to recognize hand gestures through a webcam and integrate it into Unity to create various interactive scenarios, each unfolding unique interactions based on the recognized hand postures. Aims to explore non-tangible interactive postures and invite the users to redefine the conventionally perceived functionalities behind them.

Project video
Image
2 Game scene in Unity with hand gesture control.
Image
3 Our technical pipeline.