FUURAA AI Knowledge Library · Video generation evaluation

How to evaluate AI video generation capability claims

Use six evidence gates to turn “high-quality video generation” into an exact task, input and prompt distribution, system version, temporal settings, frame quality, identity and object persistence, action, interaction, physics, complete clips and human selection, safety, disclosure, latency, cost and a dated deployment boundary.

Enter the six evidence gates ↓Open the failure matrix

Published27 August 2026Evidence statusMethod synthesis grounded in primary video-generation evaluation researchScopeText-to-video, image-to-video, extension, transformation and content-production research

Smooth is not the same as correct, stable or usable

A video-generation claim must keep prompts, source media, temporal configuration, complete clips, time-localised failures, human selection and real use in one evidence chain.

An attractive first frame and an overall average can hide identity drift, object mutation, action mismatch, brief flicker, broken physics, failed retries and post-editing. A clip can move smoothly while failing the requested action or relationship.

Applicability boundaryThis is a public research method, not a FUURAA or FUUVO product-capability claim or an assessment of any model, provider, creator, dataset or leaderboard. It is not a quality guarantee, identity or rights clearance, provenance determination, safety certification, procurement, audit, investment, legal or compliance opinion.

Six rejectable evidence gates

Each gate answers one decision question, requires minimum evidence and stops transfer or narrows the conclusion when material unknowns remain.

Freeze the video task, input and acceptance decision

Decision question: Is the claim about text-to-video, image-to-video, extension, transformation or an edited workflow; for which audience, channel and consequence?
Minimum evidence: Verbatim claim and date; generation mode; source inputs; target duration, frame rate, resolution and shot structure; prompt language; audience, channel, quality floor and acceptance rubric.
Stop condition: Stop when a showreel, model name or generic “cinematic” label replaces a defined task, input distribution and viewing decision.

Preserve the exact generator and temporal pipeline

Decision question: Which model, API or checkpoint produced the clip, through which complete temporal and post-production pipeline?
Minimum evidence: Provider, model and API version; checkpoint and adapters; prompt optimiser; seed, sampler, steps and guidance; duration, frame rate, resolution and aspect ratio; reference image or video; interpolation, extension, upscaling, stabilisation, editing, audio, filters and compute.
Stop condition: Stop when versions, frame rates, hidden reranking, undisclosed edits or post-processing are pooled into one capability claim.

Define prompt and source provenance, coverage and overlap

Decision question: How were prompts and source media sampled, translated, rewritten or licensed; which languages, people, actions, camera moves and difficulty levels are represented?
Minimum evidence: Prompt and source set with hashes; provenance and licence; sampling frame; deduplication; translation and rewriting; strata for subject, action, interaction, language, duration and camera; benchmark overlap and contamination review.
Stop condition: Stop when hand-picked English prompts, curated source images or an undisclosed optimiser stand in for broad multilingual, cultural or open-world generation.

Separate frame quality, temporal consistency, action and physics

Decision question: Does evaluation distinguish visual quality, identity and object persistence, flicker, motion, action and relation binding, physical commonsense, camera behaviour and prompt following?
Minimum evidence: Metric and judge versions; per-dimension and time-localised results; identity, tracking, action, relation, physics, text and camera checks; qualified human ratings, instructions, blinding, agreement, uncertainty and metric–human validation.
Stop condition: Stop when FVD, a CLIP-like score, one multimodal judge or an attractive first frame is treated as proof of every video-quality dimension.

Count every clip, failure, retry, selection and edit

Decision question: How many clips, seeds, extensions, reruns, rejections and edits preceded the displayed or accepted output?
Minimum evidence: All prompts, source inputs, seeds and clips; invalid, blocked, truncated and timed-out runs; candidate count; selection and reranking rules; extension and edit history; acceptance, abstention and failure rates; human time, latency and cost distributions.
Stop condition: Stop when only the best clip survives, brief failures disappear in averages, or success lacks the complete generation and human-work denominator.

Test harms, disclosure, deployment transfer and expiry

Decision question: What happens with identifiable people, protected groups, sensitive actions, misleading realism, real distribution channels and later model or policy changes?
Minimum evidence: Consent and likeness controls; subgroup and red-team prompts; sexual, violent and deceptive-content tests; provenance or disclosure checks; target-channel review; incident and fallback procedures; monitoring, change triggers, owner and expiry.
Stop condition: Stop when a benchmark average or filter label is transferred into rights, cultural, safety or production suitability without direct target-context review.

Minimum video-generation claim failure matrix

Check these conditions deliberately before transferring selected clips, attractive first frames or one average score into a broad video-generation claim.

01
Prompt optimisation or translation silently changes the requested action, camera move, negation or culture
Record the affected prompt, input, time segment, setting, metric, audience and channel; preserve the narrowest conclusion that remains and specify whether to resample, regenerate, review frame by frame, include failures, evaluate manually or withdraw the claim.
02
The first frame is correct but a person, object, count or attribute mutates over time
Record the affected prompt, input, time segment, setting, metric, audience and channel; preserve the narrowest conclusion that remains and specify whether to resample, regenerate, review frame by frame, include failures, evaluate manually or withdraw the claim.
03
Motion looks smooth while the action, interaction, spatial relation or prompt meaning is wrong
Record the affected prompt, input, time segment, setting, metric, audience and channel; preserve the narrowest conclusion that remains and specify whether to resample, regenerate, review frame by frame, include failures, evaluate manually or withdraw the claim.
04
A visually attractive clip violates physical commonsense, causality or object permanence
Record the affected prompt, input, time segment, setting, metric, audience and channel; preserve the narrowest conclusion that remains and specify whether to resample, regenerate, review frame by frame, include failures, evaluate manually or withdraw the claim.
05
An average score hides a brief but severe identity, text, anatomy, safety or continuity failure
Record the affected prompt, input, time segment, setting, metric, audience and channel; preserve the narrowest conclusion that remains and specify whether to resample, regenerate, review frame by frame, include failures, evaluate manually or withdraw the claim.
06
Best-of-many selection and post-editing hide rejected, blocked, truncated or low-quality generations
Record the affected prompt, input, time segment, setting, metric, audience and channel; preserve the narrowest conclusion that remains and specify whether to resample, regenerate, review frame by frame, include failures, evaluate manually or withdraw the claim.
07
Results shift materially across duration, frame rate, aspect ratio, seed, language or API revision
Record the affected prompt, input, time segment, setting, metric, audience and channel; preserve the narrowest conclusion that remains and specify whether to resample, regenerate, review frame by frame, include failures, evaluate manually or withdraw the claim.
08
A benchmark clip is transferred to advertising, editorial, education or sensitive use without channel review
Record the affected prompt, input, time segment, setting, metric, audience and channel; preserve the narrowest conclusion that remains and specify whether to resample, regenerate, review frame by frame, include failures, evaluate manually or withdraw the claim.

Minimum video-generation capability evaluation record

Let the next reviewer reconstruct the conclusion with the same inputs, prompts, version, temporal settings, complete clips, dimensional metrics, human rubric and deployment boundary.

01Exact claim, date, video task, source input, audience, channel, consequence and acceptance rubric
02Provider, model, API or checkpoint version and complete generation and post-production pipeline
03Seed, sampler, steps, guidance, duration, frame rate, resolution, aspect ratio, filters and compute
04Prompt and source-media set, language, provenance, licence, sampling, translation, rewriting and hashes
05Benchmark overlap, contamination review and coverage of subjects, actions, interactions and camera grammar
06Separate frame, temporal, identity, motion, action, relation, physics, camera, text and preference results
07Metric and judge versions, time-localised review, human protocol, agreement and uncertainty
08All clips, invalid runs, blocks, truncations, retries, extensions, candidate counts, selections and edits
09Acceptance, abstention, subgroup, safety, latency, cost and human-effort distributions
10Rights and disclosure review, deployment boundary, monitoring, fallback, change triggers, owner and expiry

Common evidence states

Bind conclusions to the exact generator, video task, input and prompt distribution, temporal pipeline, evaluation dimensions, complete runs, use and date.

Supported

Evidence supports the exact generator, video task, source and prompt distribution, temporal pipeline, dimensional evaluation, complete runs and dated deployment boundary.

Conditional

Evidence supports a narrower mode, input class, prompt language, duration, frame rate, action, selection process, audience or channel.

Mixed

Results vary materially across frame quality, identity, temporal consistency, motion, action, physics, people, languages, seeds, judges, safety or acceptance.

Insufficient

System identity, representative prompts and sources, temporal settings, separate metrics, human validation, complete clips, safety, effort, cost or transfer evidence is missing.

FUURAA analysisThe minimum decision unit for an AI video-generation claim is exact generator and version × video task, input, prompt distribution, language, audience and use × duration, frame rate, resolution, shot design and complete generation pipeline × frame quality, temporal consistency, action, interaction, physics, camera and prompt following × all clips, failures, selection, editing, safety, latency, cost and human effort × deployment boundary and cut-off date. FVD, EvalCrafter, VBench, VideoPhy, T2V-CompBench and VBench++ illuminate different parts; none independently represents real video-production capability.

Primary sources and non-transfer boundaries

These sources constrain distributional quality, visual/content/motion quality, temporal consistency, physical commonsense, compositional action and text-to-video/image-to-video; none independently proves general video-generation capability.

Sources rechecked 27 August 2026. Each retains its publication timing, role in this method and non-transfer boundary.

First submitted 3 December 2018; revised 27 March 2019Unterthiner et al. — Fréchet Video Distance (FVD)

Introduces FVD to compare generated and reference-video distributions while reflecting visual quality, temporal coherence and diversity.

BoundaryA distribution score does not verify an individual clip's prompt fidelity, action correctness, physics, identity consistency, safety or usefulness in a target workflow.

Open primary source ↗First submitted 17 October 2023; revised 23 March 2024EvalCrafter — Benchmarking Large Video Generation Models

Uses 700 prompts, 17 objective metrics and human-alignment analysis across visual quality, content quality, motion quality and text–video alignment.

BoundaryIts prompt set, metric selection, fitted weights and evaluated models are a research snapshot, not a universal production ranking or acceptance test.

Open primary source ↗First submitted 29 November 2023VBench — Comprehensive Benchmark Suite for Video Generative Models

Dissects video generation into 16 fine-grained dimensions including identity consistency, motion smoothness, temporal flicker and spatial relationships, with human-preference validation.

BoundaryResults depend on its prompts, dimension definitions, automated evaluators and human annotation set; they do not cover every duration, language, camera grammar or deployment.

Open primary source ↗First submitted 5 June 2024; revised 3 October 2024VideoPhy — Evaluating Physical Commonsense for Video Generation

Tests text adherence and physical commonsense across interactions among solids and fluids, supported by human evaluation.

BoundarySelected physical interactions do not establish general simulation, causality, long-horizon consistency, safety or production fitness.

Open primary source ↗First submitted 19 July 2024; revised 15 January 2025T2V-CompBench — Compositional Text-to-Video Generation

Provides 1,400 prompts across attribute, spatial, motion and action binding, object interaction and numeracy, with MLLM-, detection- and tracking-based metrics checked against human evaluation.

BoundaryIts compositional taxonomy does not establish aesthetics, camera editing, cultural accuracy, safety, audio quality or real-channel acceptance.

Open primary source ↗First submitted 20 November 2024VBench++ — Versatile Video Generation Benchmark Suite

Extends fine-grained evaluation to text-to-video and image-to-video, adds an adaptive image suite and includes trustworthiness alongside technical quality.

BoundaryBroader benchmark coverage still does not prove every input, identity, duration, policy, audience, disclosure need or downstream workflow.

Open primary source ↗

Continue checking

Move from video generation into image generation, multimodal, factuality and benchmark evaluation, or the full library.

Enter the AI Knowledge Library Evaluate image generation Evaluate multimodal capability Evaluate factuality Read benchmark claims Enter AI Evidence Atlas