Physion Labs Arc 1.0: First Benchmark for Evaluating AI Video Agents on Minute-Long Narrative Coherence

• Arc 1.0 introduces a 3-dimensional framework (narrative coherence, cinematic language, production quality) with 16 metrics that combine objective technical checks with subjective aesthetic judgments grounded in film theory
• Testing 6 commercial video agents on 100 screenplays revealed that cross-chapter consistency is the hardest challenge—identity consistency drops sharply across scene boundaries while spatial consistency holds up better
• Runway Agent 2.0 leads all agents, with the biggest performance gap in "cinematic language" (shot selection, pacing, composition) rather than narrative coherence, suggesting directorial taste is the key differentiator
• The benchmark decomposes videos into "chapters" and "beats" to evaluate at different temporal granularities, since some failures only emerge when scenes are stitched together
• Points toward future "world critics"—AI systems that can evaluate whether generated content maintains coherence, intention, and meaning over time, not just detect localized technical glitches

Physion Labs introduces Arc 1.0, the first benchmark designed to evaluate AI video generation agents on their ability to create coherent minute-long videos across multiple scenes, moving beyond existing benchmarks that only test short single-scene clips for technical correctness. The core insight is that creating a real video requires more than generating plausible footage—it demands maintaining a consistent world, narrative logic, and cinematic quality across time, which requires both objective technical evaluation and subjective aesthetic judgment.

The benchmark introduces a hierarchical framework grounded in film theory (Aristotle on coherent action, Eisenstein on shot relationships, Bordwell on cinematic form) with three dimensions: narrative coherence (story continuity across scenes), cinematic language (directorial choices in shot selection, pacing, composition), and production quality (technical stability). These break down into 16 metrics evaluated at different temporal granularities—within chapters, across chapters, and full video—because different failures emerge at different scales. They decompose each screenplay into narrative "beats" (smallest meaningful story units) and segment generated videos into "chapters" (semantically cohesive segments), then evaluate whether beats are realized and how well chapters maintain consistency.

Testing 6 commercial agents (Runway, Utopai, MiniMax, Luma, TapNow, Kling) on 100 screenplays from T2F-Bench revealed that Runway Agent 2.0 leads across all dimensions, with the largest performance gap in cinematic language rather than narrative coherence—suggesting that "directorial taste" in translating narrative to audiovisual structure is the key differentiator as agents converge on basic planning capabilities. Cross-chapter consistency proved much harder than within-chapter, with identity consistency dropping most sharply across scene boundaries while spatial consistency remained relatively stable, indicating that visual conditioning preserves coarse geometry better than fine-grained instance details. The work points toward a future need for "world critics"—AI systems that can evaluate whether generated content maintains coherence, intention, rhythm, and cinematic meaning over time, not just detect localized technical failures.