Grounding LLMs with JEPA-based world models trained in simulation — has this been tried? [D]
Some users have proposed a solution that trains a world model based on JEPA in a physical simulation environment and connects it to an LLM, aiming to address the issue of LLMs lacking physical intuition. This approach trains models for predicting future states in an abstract embedding space using MuJoCo or a simple 2D environment. It utilizes a mechanism that tolerates physical errors to extract representations of principles such as the constancy of encoded objects, momentum, and trajectories, and attaches these representations as conditional signals to the LLM to accelerate downstream learning. Although V-JEPA and DreamerV3 involve video frame representation predictions or latent world models for reinforcement learning, a combination of JEPA-style prediction with simulation physics representation and connection to LLMs has not been clearly implemented yet. The community is concerned about whether there are any overlooked prior studies, the interface between JEPA representations and language models (such as concatenation or cross-attention), and whether the gap between simulation and reality may lead to failed迁移.