Open your hand and a menu appears, then follows your hand. Hold still over an option and it is selected. Move your hands apart and a value goes up. The person in front of the camera is the controller, and the software behind the screen is ComfyUI. No mouse, no keyboard, no game pad.
This is the project Alexandre Grégoire, Technical Director, built during cohort 1 of our Generative AI Bootcamp for Film and TV and presented to the final jury on 11 September 2026. This article is based on that presentation.
🖐️ The experience
Picture a room with a large screen and a camera. The screen invites you to raise your hand. A menu opens and follows it. You choose an ambience (fire, ice, storm or dust), an avatar style (cyberpunk, manga, a dark film look in black and white, or zombie), an environment (town, forest, mountain or desk) and two details that combine, such as rain and mist, or neon lights and rust. Then you generate, with a button or with a gesture. The camera takes your picture, and the model turns you into a character in the world you picked.
Two controls change the result. Transformation sets how much the character still looks like you: at low values it keeps your features, at high values the model takes over. Precision sets the number of sampling steps: more steps take longer and give a sharper image, fewer steps are faster and softer. An album collects the generated pictures, and you browse it with the same gestures to keep or remove them.
A hidden menu, called by a gesture the public does not know, is for the operator. It shows the tracking overlay, a diagnostics panel with the frame rates (about 60 frames per second for the tracking), the recording toggle, and the list of ComfyUI workflows saved for the installation, so one setup can switch between several experiences. A separate operator view shows the camera, the generated image and the menus, and lets the person running the event change the generation settings live.
🧩 How it is built
Everything runs inside ComfyUI. Two custom nodes, written by Alexandre, do the tracking. The first manages the camera and the hand tracking. The second picks one feature out of that tracking and turns it into a number between 0 and 1: whether a hand faces the camera, the distance between two hands, whether two hands mirror each other, or whether a gesture such as the OK sign is being made. Both the input and the output ranges can be remapped, so a node downstream receives whatever range it needs.
The menus are built from nodes too. Sliders, menu entries and the logic that connects them are ComfyUI nodes wired together in the graph. The image comes from an image-to-image workflow on the webcam frame. Alexandre chose DreamShaper because it is fast enough for a live experience on his RTX 3060: two to three images per second.
🎛️ One rule for the whole interface
Tracking a hand is a solved problem, Alexandre said. The hard part was knowing when a person means to select something. Hands shake, and the machine cannot be certain that a movement was a choice. So he wrote one rule and applied it to the whole interface:
- A continuous gesture sets a value. It never chooses between options.
- To choose, you hold your hand still over the option for a moment.
- Anything that validates a choice shows its progress while it validates.
⚖️ Bias and guardrails
Alexandre was open about the limits of the model. DreamShaper carries a strong bias: left to itself, it keeps drifting toward one type of subject and toward images you would not want on a public screen. He raised the CFG so that negative prompts take effect against unwanted content, which is also why the demo images look saturated, and he plans to test other models before the event.
The jury pushed the point further: stress-test the installation with adversarial prompts before it meets the public, and put a light filtering layer in front of the model so that unwanted inputs are caught fast, rather than relying on the model alone. Alexandre's own idea goes in the same direction: an agent between generation and display that checks the image before it reaches the screen.
🔭 What comes next
The installation is built for a public event that Alexandre's Fab Lab, a maker space, takes part in this autumn. Next on his list: connect physical objects through boards such as ESP32 or Arduino, so that changing a real light in the room changes the generated picture; let people speak instead of choosing from menus; and finish the prompts behind the avatar styles, which are placeholders for now. The jury saw a strong appeal for public-facing events and suggested looking at how live generation is wired into tools such as TouchDesigner.
🎓 Built in the Bootcamp
Cohort 1 of the Generative AI Bootcamp for Film and TV finished on 11 September 2026 with a group of six participants and six projects, each built in ComfyUI across the sessions, with a midpoint review and a final review by a jury of working AI professionals. Alexandre's installation is one of the six presented that day.
Cohorts 2 (from 2 October 2026) and 3 (from 6 January 2027) are open for enrollment. You bring one production case, and you build it. Learn more about the Bootcamp.