Most top video generators first squeeze footage into a compact code with a separate network called an autoencoder, then generate in that code space. That two-step setup loses detail and ties results to a compressor trained for a different goal.
The alternative is generating raw pixels directly. Earlier attempts with plain transformers, the standard building block of modern neural networks, struggled: video is too high-dimensional for a patch of pixels to fit the model's internal width.
The fix varies patch size with network depth, down to single pixels at the finest level. On a standard video benchmark it improves FVD, a realism score where lower is better, by 27% over the best published latent model.