NEWTON: Agentic Planning for Physically Grounded Video Generation

Feng, Yuxiang; Wang, Juncheng; Xu, Chao; Qian, Yijie; Wang, Huihan; Hou, Wenlong; Liu, Yang; Sun, Baigui; Liu, Yong; Wang, Shujun

Newton orchestrates external tools to ground video generation in physical laws.

Abstract

Video generation models produce visually compelling results but systematically violate physical commonsense — on VideoPhy-2, the best model achieves only 32.6% joint accuracy. We identify a specification bottleneck: text prompts are lossy compression of the physical world, omitting the parameters that fully determine dynamics, and no amount of model scaling can recover what was never specified. From this diagnosis we derive three properties that physics conditioning must satisfy — sufficiency, dynamism, and verifiability — and show that no existing approach satisfies all three.

We present Newton, in which video generation is demoted from the system output to one action inside an agent's toolbox: a learned planner orchestrates physics-aware tools (keyframe generation, scientific computation, prompt refinement) to construct rich conditioning, and a verifier closes the loop for iterative re-planning. The planner is the sole trainable component, optimized on-policy via Flow-GRPO inside the live multi-turn loop.

On VideoPhy-2, Newton improves joint accuracy from 21.4% to 29.7% on LTX-Video and from 30.7% to 37.4% on Veo-3.1, without modifying either generator.

Why Existing Approaches Fail

Physics failures are a specification problem, not a capacity problem. “A ball rolls off a table” omits mass, friction, height and initial velocity — the parameters that determine the trajectory. Three properties follow, and no existing approach satisfies all three.

Comparison of three paradigms for physically grounded video generation against the sufficiency, dynamism and verifiability properties.
Paradigm	Sufficiency	Dynamism	Verifiability
End-to-end The generator hallucinates all physics from text.	✗Physics embedded implicitly Training cannot guarantee which physical dimensions are covered.	○Not claimed	✗One-shot, no feedback Nothing checks whether the output obeys the intended physics.
Conditional ControlNet-style fixed-modality signals (depth, identity).	○Not claimed	✗Cannot adapt per scene A fixed modality cannot cover physical domains it was not built for.	✗One-shot, no feedback
NEWTON (ours) A trained planner orchestrates physics tools; a verifier closes the loop.	✓Tool library Complementary tools cover complementary physical dimensions.	✓Planner Selects and composes tools per scene, on demand.	✓Verify–correct loop Evaluation feeds back into the next round of re-planning.

Sufficiency — covering enough physical dimensions to determine the dynamics · Dynamism — adapting per scene · Verifiability — checking the output and correcting it
✓ satisfied · ✗ the paper’s stated failure mode for this paradigm · ○ not claimed either way

Method

Given a user prompt, the planner decides which tools to invoke; the video generator becomes one tool among many. The critic evaluates each draft against physical plausibility and feeds language-form feedback back into the next plan–execute–verify cycle.

Plan–Execute–Verify Loop

Vision–language planner chooses tools per turn.
Multi-turn rollouts let the system self-correct.
Treats the video generator as one callable tool.

External Toolkit

Python interpreter for physics simulation.
Search engine for visual reference retrieval.
Image generator for physics-grounded conditioning.

Flow-GRPO Training

Multimodal critic produces language & scalar rewards.
Trajectory-level credit assignment across turns.
Planner self-evolves from live deployment traces.

Flow-GRPO: Training the Planner In the Loop

NEWTON never touches the video generator. The planner is the only trainable part, optimized on-policy via Flow-GRPO inside the live multi-turn loop — it trains on the same plan–execute–verify trajectories it runs at inference.

Trainable

The planner (a vision–language model) — and nothing else.
Optimized on-policy via Flow-GRPO across live rollouts.
Trajectory-level credit assignment spans all turns of a loop.
Reward comes from a multimodal critic emitting both scalar scores and language-form feedback.

Frozen

The video generator — LTX-Video and Veo-3.1 are used as-is.
The external tool library — no tool is ever trained. The library is extensible: new physics tools can be registered alongside the existing ones.
The verifier.
No retraining, no fine-tuning, no architectural change to any generator.

VideoPhy-2 joint accuracy before and after NEWTON on two frozen video generators.
Video generator	Baseline	+ NEWTON	Δ
LTX-Video	21.4%	29.7%	+8.4
Veo-3.1	30.7%	37.4%	+6.7

VideoPhy-2 joint accuracy. Neither generator was modified — the gains stack on the stronger backbone rather than substituting for it.
For reference, the best model on VideoPhy-2 reaches only 32.6% joint accuracy on its own.

The planner discovers scene-dependent tool scheduling nobody specified: trajectories for projectiles, keyframes for spatial constraints, refined prompts for material properties. Physical consistency stops being something you hope emerges and becomes something you engineer.

Real-World Physics

Newton vs. open-source video generators on prompts that require fluid dynamics, deformable cutting, granular pouring, and rigid-object peeling.

FluidA bottle of beer is poured into a mug until it is full.

Newton (Ours)

LTX

HunyuanVideo

Wan 2.2

CuttingA small knife digs a groove into a piece of wood.

Newton (Ours)

LTX

HunyuanVideo

Wan 2.2

GranularSalt is poured from a shaker onto a plate, creating a layer of granules.

Newton (Ours)

LTX

HunyuanVideo

Wan 2.2

PeelingA grapefruit is peeled with a knife; the thick rind separates.

Newton (Ours)

LTX

HunyuanVideo

Wan 2.2

Animated World

Newton transfers its Physically Grounded behavior into stylized domains — Studio Ghibli and LEGO — without sacrificing visual style.

GhibliStudio Ghibli — a girl on a park bench blows a bubble that drifts up into the sky.

Newton (Ours)

HunyuanVideo

Wan 2.2

LEGOLEGO — a quarterback hands off the football; the running back drops it and it bounces.

Newton (Ours)

HunyuanVideo

Wan 2.2

Citation

@article{feng2026NEWTON,
  title         = {NEWTON: Agentic Planning for Physically Grounded Video Generation},
  author        = {Feng, Yuxiang and Wang, Juncheng and Xu, Chao and Qian, Yijie and Wang, Huihan and Hou, Wenlong and Liu, Yang and Sun, Baigui and Liu, Yong and Wang, Shujun},
  journal       = {arXiv preprint arXiv:2605.18396},
  year          = {2026},
  url           = {https://arxiv.org/abs/2605.18396}
}