A shot is approved, then a note arrives: add smoke in the background. Stock footage almost never sits where the frame needs it. A simulation means setup, cache and iteration time you no longer have. So we trained a model that takes the drawing your supervisor already made on the frame and turns it into a moving element.
ainvfx-fluid is an IC-LoRA for LTX 2.5. You paint where the volume starts and where it ends, in flat colours, and the model replaces the paint with smoke, steam or fire and animates between your keyframes. A 5-second, 512x512 element takes about 50 seconds on a laptop 4090. The weights, the ComfyUI workflow and the painted example frames are free on Hugging Face.
🎯 What an IC-LoRA does here
An IC-LoRA (in-context LoRA) learns a relationship between two videos rather than a style. During training the model sees a pair: a control video and the result it should produce from it. At inference you supply a new control video and it produces the matching result.
For this model the pair is simple:
- Control: flat colour blobs on the frames you paint, black everywhere else.
- Result: the same shot with real smoke, steam or fire where the blobs were.
The trigger word is ainvfxfluid. Everything else about the shot — the shape of the plume, its colour, where it is in frame, how it changes over time — comes from what you paint, not from the prompt.
We trained it during cohort 1 of our Generative AI Bootcamp for Film and TV as the teaching example for IC-LoRA training, and it turned out useful enough to release.
🧩 The control video rules
This is the part that decides whether the model behaves. A control video is a normal video file, the same size and length as the shot you want:
- Sizes are multiples of 64: 512x512, 960x512, 512x960, 1024x576, 1920x1088. The reference is encoded at half resolution, so anything off the grid degrades fast.
- 121 frames is the tested length, at 24, 25 or 50 fps. Other 8k+1 lengths should work.
- First frame at index 0, last frame at index 120, black frames in between.
- Extra keyframes sit at a multiple of 8, plus 1: frame 1, 9, 17, ... 113. The LTX VAE groups frames in blocks of 8, so a keyframe placed off that grid is smeared across a block instead of being read cleanly.
- A keyframe can be a single frame or held across its 8 frames. Both were trained, so both are read.
- Paint flat colours: one blob per volume, light grey for smoke, white for steam, orange with a yellow core for fire. A black background, or a few flat zones for sky and ground.
The shipped workflow builds all of this with core ComfyUI nodes: Empty Image for the black frames, Batch Images to stack first frame, black frames, last frame, and Create Video to make the clip. No custom node beyond the LTX pack.
⚡ Running it in ComfyUI
- Node pack: install ComfyUI-LTXVideo through the Manager. On a recent ComfyUI you may hit an import error on
padfrom Kornia: open the file named in the traceback, remove thepadimport, and addpad = F.padbelow. Restart. - Models: the LTX 2.5 transformer, the two VAEs and the Gemma text encoders are gated on Hugging Face. Accept the licence with a free account, then let the workflow download them or place them by hand in
diffusion_models,vaeandtext_encoders.ainvfx-fluid.safetensorsgoes inmodels/loras. - Painted frames: copy the two PNGs from the repo's
workflow/inputs/intoComfyUI/input/so the two Painter nodes open with the example already painted. - Load and queue. The LoRA loads through
LTX IC-LoRA Loader Model Onlyat strength 1.0; the loader reads the reference downscale factor from the file, so there is nothing to set.
The workflow uses the distilled LTX 2.5 transformer at 8 steps, CFG 1, euler_ancestral, seed 42. With LTX there is no separate scheduler: the step distribution is set directly through sigmas, and the shipped values put most steps in the high-noise range where composition is decided.
The two Painter nodes require Nodes 2.0, which you turn on in the ComfyUI settings. They are piped one into the other, so the second one opens on your first frame and you paint the change on top. One bug to know: if a Painter's output stops reaching the graph, toggle Nodes 2.0 off and on again.
🎨 Prompting
Keep it short and start with the trigger word:
ainvfxfluid, smoke plume
That is often enough. Longer prompts help for two things:
- Removing what you did not ask for. The model sometimes invents an emitter or a foreground.
ainvfxfluid, smoke plume over a black backgroundcleans that up and gives you an element you can composite. - Describing material and light, which the doodle cannot say:
wildfire and smoke plume, brown soil in the foreground, trees behind the fire, blue sky at the top of frame.
Colour and shading carry over from the paint. A second, darker grey along one edge of the plume comes back as shading in the generation, not as a flat white cloud.
🎞️ More than two keyframes
Two frames give you a start and an end. A third lets you change direction mid-shot, a gust of wind for example. The maths for a keyframe at frame 57, in a 121-frame clip:
| Slot | Content |
|---|---|
| image0 | painted first frame (index 0) |
| image1 | 55 black frames |
| image2 | painted keyframe (index 57 = 8 × 7 + 1) |
| image3 | 63 black frames |
| image4 | painted last frame (index 120) |
Add as many as you want, as long as each one lands on the 8n+1 grid.
The result follows the painted shapes closely and still leaks a little at the edges, which is what you want: over-controlling a fluid is how a simulation stops looking like a fluid.
🏙️ Working with a real plate
You can paint your smoke over a photograph and use that as first and last frame. The result keeps the composition and the colours of the plate, and puts the fluid where you painted it, but it is a video inspired by the plate, not a pixel-exact match. The model was never trained to reproduce a plate.
For production, treat it as a stock shot generator:
- Generate over black, then composite the element over your plate in Nuke or your compositing software of choice.
- If you need the perspective, run two generations: one with the plate as background to get the perspective right, one over black to patch it.
⚖️ Limitations
- Trained on smoke, steam and fire only. Water, ink and clouds are hit and miss. Prompting recovers some of it.
- Not a plate matcher, as above.
- Off the 8-frame grid, or off multiples of 64, quality drops quickly.
- 121 frames at 24, 25 and 50 fps were tested.
- The base model's own limits apply.
Two ways to push quality: generate at a higher resolution, up to 1920x1088 which is what the last training stage used, and swap the distilled transformer for the non-distilled LTX 2.5. The LoRA was trained against the non-distilled base, so it responds well to it. Slower, nicer.
🧪 How it was trained
Everything here is reproducible with the LTX-2 fork of Musubi-tuner (ltx-2-dev branch).
Dataset: 52 stock clips from Pexels — bonfires, wildfires, grass fires, smoke over forests, chimney steam. 44 landscape, 8 portrait, at 24, 25, 30 and 60 fps. The 30 fps clips were resampled to 25 and the 60 fps clips to 50, by timestamp, so the motion speed is unchanged. Captions by Qwen3-VL, trigger word first.
Control videos were generated from the clips themselves: each keyframe reduced to 6 flat colours, with the palette chosen among the colours actually present in the frame rather than by surface area, so a flame covering 2% of the frame keeps its own colour. Between 2 and 5 keyframes per clip on the 8-frame grid, first and last frame always, each held for 1 to 8 frames, black everywhere else. That variety is why the model reads both a single painted frame and a held one.
Three-stage resolution chain, each stage warm-started from the previous one, with one dataset block per frame rate so no clip is ever resampled:
| Stage | Pairs | Geometry | Steps | Learning rate | Time |
|---|---|---|---|---|---|
| 1 · motion | 40 | 576 x 320 x 321 frames | 2000 | 1e-4 | 3 h 48 |
| 2 · shape and material | 94 | 1024 x 576 x 121 | 1000 | 5e-5 | 2 h 26 |
| 3 · fine detail | 206 | 1920 x 1088 x 33 | 500 | 2e-5 | 1 h 32 |
LoRA rank 32, alpha 32, v2v target preset, reference downscale 2, batch size 1, 8-bit AdamW, FP8 base weights. Total: 7 h 47 min on a single RTX PRO 6000, caching and previews included, peak 24 GB of VRAM. Three pairs were held out of stage 2 and used as sample controls, so every preview during training was on material the model had never seen.
📜 License
The LoRA is a derivative of LTX-2.5 and ships under the LTX-2.x Community License, the same licence as the base model, with a copy in the repository. Section 3.5 matters if you work at scale: a Commercial Entity with annual revenue of 10 M USD or more needs a paid agreement with Lightricks before using LTX-2.x or any derivative. Read the file before you ship. This is a summary, not legal advice.
📥 Downloads
Everything is on the Hugging Face model card:
- ainvfx-fluid.safetensors — the IC-LoRA, ComfyUI format, 654 MB
- Example workflow — paints both frames inside ComfyUI, core nodes plus the LTX pack
- Two painted frames — so the workflow opens ready to run
- Six sample pairs — control video and result, side by side
🚀 The takeaway
The interesting part is not that a model can make smoke. It is that a rough drawing, the kind already attached to a supervisor's note, is enough input to direct it. You keep the art direction, the machine does the solve, and the whole round trip fits inside an iteration instead of a day.
If the element you need is not smoke or fire, the recipe above is the whole method. Swap the dataset, keep the control video rules, and train your own.