Shot-to-Set: from one film shot to a set you can visit in VR, built in the Bootcamp

Give it one shot from a film. Shot-to-Set breaks the shot down, works out who is in it and what surrounds them, rebuilds all of it in 3D and assembles the scene in Unreal Engine. Then you put on a VR headset and you are standing on the set.

Ersan Musa, Previs Supervisor, built it during cohort 1 of our Generative AI Bootcamp for Film and TV and presented it to the final jury on 11 September 2026.

🎬 The idea

Ersan has worked in previs for about ten years, and he also teaches. Previs, short for previsualization, is the rough 3D version of a sequence that is made before the shoot to plan it. For him, one of the best feelings in making a film is being on the set. Shot-to-Set is a way to give that feeling to the people who only ever see the finished film: take a shot they know, and let them walk into it.

He also chose it because it is hard. Rebuilding an entire shot from a single clip touches almost everything the Bootcamp covers: image analysis, character generation, 3D, scene assembly. His words on the day: what could be more complicated than recreating an entire shot and putting the viewer inside it?

🔗 How it works

Shot-to-Set is a chain of more than a dozen steps, each one a ComfyUI workflow. The steps are connected with n8n, an automation tool that passes the result of one step to the next, and everything runs locally on Ersan's machine. The chain has three phases.

Reading. The tool splits the shot into its frames and its audio, then analyses the frames.

Research. The tool goes on the web. Qwen, a model that understands images and text, looks at the frames and at the web pages it finds, and works out which actors are on screen, what they wear, which vehicles and props are in the shot, and what the set looks like. It fills in a breakdown sheet: characters, sets, actions, vehicles, props.

Rebuilding. For each character, the tool generates a portrait, then several views of that portrait, then a 3D model from those views. Sets are captured as Gaussian splats, a 3D format that renders in real time and that you can move around in. Everything is then assembled in Unreal Engine, where the VR headset comes in.

A viewer shows the shot next to the result of each step, so you can compare them, and a right-click on any step opens it in ComfyUI. If something needs a correction, you fix that one step and continue.

🧑‍🔧 A human stays in control

The tool has three modes. In automatic mode it runs the whole chain by itself. In assisted mode, the one Ersan prefers, it stops at two or three points marked in the flow and asks: is this the right actor, is this what is happening in the shot? You confirm or correct, and it continues. In manual mode you drive every step. You can also ask Qwen questions along the way, such as what is missing before the next step.

🔭 What comes next

Two parts were still in progress on the day of the jury: the wardrobe, and the final scene in Unreal Engine. Generating full 3D bodies for the child characters in his test shot turned out to be the hardest part. Ersan's answer is to generate only the heads, use MetaHuman bodies from Unreal Engine, and combine the two before rigging. That is where the project goes next.

🎓 Built in the Bootcamp

Cohort 1 of the Generative AI Bootcamp for Film and TV finished on 11 September 2026 with a group of six participants, each presenting their own project, built in ComfyUI across the sessions, with a midpoint review and a final review by a jury of working AI professionals. Shot-to-Set is one of them.

Cohorts 2 (from 2 October 2026) and 3 (from 6 January 2027) are open for enrollment. You bring one production case, and you build it. Learn more about the Bootcamp.

🔗 Sources & Links

🎬 The project:

🛠️ Tools:

🎓 The Bootcamp:

📄 Related Posts:

Join the conversation

Have thoughts on this article? We'd love to hear from you!

Let's work together

Inquire now