OmniPhysGS

🧩 Abstract 🧩

Recently, significant advancements have been made in the reconstruction and generation of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically assume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heterogeneous objects in real scenarios and tend to render less physically plausible animation given a wider range of objects. We propose OmniPhysGS for synthesizing a physics-based 3D dynamic scene composed of more general objects.

A key design of OmniPhysGS is treating each 3D asset as a collection of Constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensemble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementation, we define a scene by user-specified prompts and supervise the estimation of material weighting factors via a pretrained video diffusion model. Comprehensive experiments demonstrate that achieves more general and realistic physical dynamics across a broader spectrum of materials, including elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.

🔮 Method 🔮

Constitutive Gaussian

The capability of OmniPhysGS in general physics-based 3D dynamic synthesis is achieved by three key components: (1) a memory-efficient MPM solver that imposes strong physical constraints for the scene; (2) Constitutive Gaussians for flexible and general material modeling; (3) video score distillation to provide prior knowledge for encoded in the pretrained video diffusion model.

Physics-guided Network

The physically guided architecture of Constitutive Gaussians consists of two main components: (1) a 3D feature encoder backbone to extract features of 3D scenes (2) a physical-aware decoder to predict the material properties of Constitutive Gaussians with an ensemble of domain-expert sub-models.

📝 Single Object with Different Material 📝


A swinging ficus	A collapsing ficus	A rubber wolf bouncing	A sand wolf collapsing

A jelly bouncing	Water flowing	A crushed metal can	A rubber duck falling

A collapsing mud pile	A soccer ball hits the ground	A soft ficus swinging heavily in the wind	A ficus swinging slightly

A very soft, bouncy rubber wolf	A hard wolf	A very tender jelly	A firming jelly

🛠️ Multiple Objects with Different Materials 🛠️


A lego excavator is digging soil	A rubber wolf falling on water

A hard duck colliding to create a dent in a breakable metal can	A duck falling on a soft mud pile

A golden rubber wolf bouncing and a grey sand wolf collapsing	Two rubber ducks colliding

Different kinds of materials on the table	Pillows falling into a basket

🕹️ Motion Generalization 🕹️


Falling	Flipping

Spinning	Teasing

🖼️ Real-world Scene 🖼️


Flowers swinging gently	A fox shaking its head

🌐 Related Links 🌐

📚 BibTeX 📚

If you find our work helpful, please consider citing:

@misc{lin2025omniphysgs3dconstitutivegaussians,
  title={OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation}, 
  author={Yuchen Lin and Chenguo Lin and Jianjin Xu and Yadong Mu},
  year={2025},
  eprint={2501.18982},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2501.18982}, 
}

OmniPhysGS: 3D Constitutive Gaussians for General Physics-based Dynamics Generation

OmniPhysGS is a novel framework for general physics-based 3D dynamic scene synthesis, which can automatically and flexibly model various materials with domain-expert Constitutive Gaussians.