Improving Weak World Models Behind Strong Agents in Atari Pong

The research team reproduced five visual world model agents in the Atari Pong environment (DreamerV3, DIAMOND, TWISTER, Simulus, and STORM) and found significant visual or dynamic failures in their frozen world models, manifested as the disappearance of the ball, incorrect movements, and ineffective interactions between the ball and the paddle. In native zero-sample model-based reinforcement learning, new strategies trained solely using the frozen models performed significantly worse in real environments compared to the original agents: DreamerV3 showed a decline of -5.5 to -20.9, DIAMOND from 19.7 to -9.6, TWISTER from 17.7 to -13.3, Simulus from 20.8 to -11.6, and STORM from 18.7 to -21.0. Inspired by ball-related rolling failures, the researchers proposed Concept-Guided Space Regularization (CGSReg), a improved method for task-critical concept areas.