Request access to the LayerGen weights

Access is reviewed manually. Please tell us who you are and how you intend to use these research checkpoints.

LayerGen — Decompose (grounded) · `decompose_grounded_ema_gbs32_4n_b200`

Single-task video layer decomposition checkpoint: given a composite video + object mask, it predicts the foreground and background layers (object removal / clean-plate + extracted foreground). Built on Wan2.2-14B as a multi-stream diffusion transformer.

Task: decompose (composite + mask → foreground + background)
Base model: Wan2.2-14B diffusers WanPipeline (40-layer / 5120-dim DiT, AutoencoderKLWan VAE, UMT5EncoderModel)
Trained at: 480×832, 41 frames (11 latent frames), flow-shift 3.0 · 4-node B200, gbs32 · grounded editcap2 captions

Checkpoints in this repo

Path	Which weights	Step
`checkpoint-4000/`	EMA (exponential moving average)	4000
`nonema/checkpoint-4000/`	raw / non-EMA (the "pure" training weights)	4000

Both are model-only DCP exports (roles.* = the trained DiT only, ~53 GB fp32 each; the frozen Wan VAE / text encoder come from init_from). Each dir is a self-contained checkpoint-<step>/ with dcp/ inside, plus a metadata.json (embedded training config). EMA weights are usually the better choice for sampling; the non-EMA weights are provided for completeness / ablation.

Requirements

FastVideo with the layer-decomp stack (fastvideo/train/entrypoint/infer_layer_decomp.py, fastvideo.train.models.layer_decomp.LayerDecompWanModel). Reference: branch prompt-fix-infer, commit b7ccac76. (Currently internal research code — you need access to it to run inference.)
A local Wan2.2-14B diffusers model for the VAE + architecture; point init_from in the config at it.
One 80 GB GPU (A100-80G / H100 / H200 / B200). FlashAttention recommended (TORCH_SDPA also works).

Inference

# Point configs/infer_decompose.yaml:  init_from -> your local Wan2.2-14B diffusers dir,
#                                       data_path -> your encoded latent pool (see encode_evals.py).
export FASTVIDEO_ATTENTION_BACKEND=FLASH_ATTN        # or TORCH_SDPA
export FASTVIDEO_COMBINED_INFER=0                    # single-task checkpoint
export FASTVIDEO_FG_CORRUPT_TYPE_ID=2                # default 4-type stream layout (this model has no fg-corrupt stream)
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export TOKENIZERS_PARALLELISM=false WANDB_MODE=disabled

# EMA weights:      --checkpoint ./checkpoint-4000
# non-EMA weights:  --checkpoint ./nonema/checkpoint-4000
torchrun --standalone --nnodes 1 --nproc_per_node 1 \
  -m fastvideo.train.entrypoint.infer_layer_decomp \
  --config configs/infer_decompose.yaml \
  --checkpoint ./checkpoint-4000 \
  --out ./out --steps 30 --model-only \
  --training.distributed.num_gpus 1 \
  --training.distributed.hsdp_replicate_dim 1 \
  --training.distributed.hsdp_shard_dim 1 \
  --training.distributed.sp_size 1

Outputs are written under ./out/samples/<clip_id>/ (per-stream mp4s: composite input, predicted fg, predicted bg, plus a per-clip record.json).

Longer clips: trained at 41 frames. For 61f/81f set FASTVIDEO_ROPE_T_INTERP=(41-1)/(T_infer-1) and bump num_latent_t (16 for 61f, 21 for 81f).

Provenance

Run decompose_grounded_ema_gbs32_4n_b200 (wandb dec_grnd_ema_b200), checkpoints at step 4000 (EMA + non-EMA). Part of the LayerGen layer-decomposition project.

License

Research preview released for reproducibility. Built on Wan2.2 (see the Wan-AI license for the base model). Contact the authors for reuse terms.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support