MPEG-Diffusion: Mixed-Resolution Video Generation

Prompt

Applications

Show tokens high-res region low-res background

More examples ↓Collapse examples ↑

Comparison with prior and concurrent methods

Method

2.Self-selected HR region

Prior methods need a preset HR region: The early denoising steps are too blurry to externally detect the HR region from the video, so it must be given before generation.

Fix: The DiT already knows where the important subject is. A single linear layer on its intermediate features picks the HR region during denoising, so no region input is needed (examples).

4.Dual-resolution attention

Curse of mixed resolution: When HR and LR tokens share one K/V cache, we prove that no single LR token can match the attention of the four HR tokens it replaces, even with fine-tuning.

Fix: Keep separate HR and LR K/V caches so each query reads K/V at its own resolution, using a linear map to fill in the missing K/V from the other resolution (examples).

5.Mixed-resolution decoder

Resolution mismatch: Decoding the HR and LR latents separately and pasting them together causes visible distortion and a seam along the resolution boundary, even for latents encoded from real videos.

Fix: Merge the HR and LR latents into a single unified latent before decoding, so the VAE decodes a seamless video (examples).

Experimental results

Human evaluation

ours preferred baseline preferred

Quantitative benchmark (Wan2.1 14B, 720p, 81 frames; MovieGen and VBench prompts)

Method	Tokens	Speed-up	MANIQA ↑	Imaging ↑	DOVER ↑	VBench Q. ↑	Extra condition
Full resolution	100%	1.0×	0.445	0.671	0.819	85.3	–
Phase-Aligned RoPE	30%	5.2×	0.293	0.622	0.757	83.8	region
Foveated Diffusion	30%	5.2×	0.300	0.619	0.743	83.9	region
MPEG-Diffusion (ours)	30%	4.8×	0.397	0.644	0.794	83.9	none

Text-to-image

MPEG-Diffusion also applies to image generation; here we use FLUX.2 4B (1024×1024) as an example.

Show tokens high-res region low-res background

More examples ↓Collapse examples ↑

Limitations

For fast-moving backgrounds, the transition between the low- and high-resolution regions can occasionally be noticeable (examples).
We currently train only a linear head to pick the self-selected region; end-to-end fine-tuning is a promising next step.

Citation

@inproceedings{mpegdiffusion2027,
  title     = {MPEG-Diffusion: Mixed-Resolution Video Generation},
  author    = {Menn, Dennis and Gu, Tianpei and He, Guande and V S, Aryan and Song, Yanke and
               Liu, Nan and Xie, Desai and Huang, Xun and Xu, Chenfeng and Marculescu, Diana},
  booktitle = {Under review},
  year      = {2027}
}