← Bookmarks 📄 Article

Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling

Kling's team reveals that video generation quality isn't about model scale—it's about understanding propagating correctly through a chain of specialized components, from captioning as dense semantic contracts to dual reward models that treat perceptual bandwidth as a first-class design parameter.

· ai ml
Read Original
Summary used for search

• Video generation failures are diagnostic puzzles: the same artifact could originate from bad captions, poor conditioning, misaligned rewards, or broken evaluation—you can't fix it by just scaling the model
• Captions are semantic contracts, not descriptions: they explicitly enumerate spatial relations, camera movement, shot boundaries, and temporal order, while generic "cinematic" descriptions are useless for control
• Prompt enhancement works backward: start with captions of good videos and generate the user prompts that could have produced them, ensuring output stays in the distribution the generator was trained on
• Text encoders need perceptual training, not stronger language modeling: candidates can be screened using text-only tasks before expensive end-to-end training
• Dual reward models balance explainability and speed: a General Reward Model provides localized critiques while a Decision Reward Model enables fast multidimensional scoring

The Kling team presents a systems-level perspective on video generation: quality is the projection of multimodal understanding through the entire data–conditioning–alignment–evaluation chain. This means generation failures aren't self-diagnosing—the same visual artifact could stem from captions that omit supervision, conditioning that discards temporal structure, rewards that favor static motion, or evaluators that never test the requested event. Their solution is to treat understanding as a property of the entire pipeline and design each component to preserve the information the next one needs.

The core technical contributions span the full chain. Captioning becomes a semantic contract with explicit requirements: spatial relations (left-right, front-back), camera movement, shot boundaries, temporal order, and visible text must all be captured, because generic aesthetic descriptors like "cinematic lighting" don't give the generator executable instructions. Reference-grounded captioning (RGC) extends this with explicit binding—which entity gets which reference, what authority each reference has, and how conflicts are resolved. Prompt enhancement works backward: instead of training a model to "improve" prompts, they start with captions of compelling videos and generate the user prompts that could have produced them, then add text-level verification (faithfulness, reasonableness, vividness) and DiT-in-the-loop RL to align with product objectives.

For the text encoder, they show that perceptual training (aligning text with visual/temporal features) matters more than language modeling strength, and candidates can be screened using text-only tasks before expensive end-to-end training. Their dual reward model architecture pairs a General Reward Model for explainable, localized critiques with a Decision Reward Model for fast multidimensional scoring, explicitly treating perceptual bandwidth as a design parameter. The evaluation framework routes each property to the simplest tool that can measure it reliably—signal-based metrics for global quality, specialist perception models for localized evidence, and MLLM checklist judges for semantic requirements that resist simple measurement.

Listen to Summary
0:000:00