Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study

A recent study on understanding and generation across tasks in a unified visual language model (VLM) was published. The study constructed two benchmarks, SmartWatch and modified CelebA, including paired question-answering, image description, and text-to-image tasks, and evaluated various LLM architectures based on SigLIP and VQ-VAE. The results showed that hybrid training improved performance in both tasks, but the effect depended highly on the correlation between visual input and output spaces; models with good spatial alignment had stronger generalization, while reversible affine distortions weakened this effect and caused conflicts. Additionally, increasing data for a single task initially was beneficial but excessive imbalance could lead to degradation in complementary tasks, and generative supervision helped restore concepts that were underrepresented in understanding. The analysis indicated that cross-space generalization stemmed mainly from the learning relationships of the base language model rather than visual adapter features, and empirical results with LLaVA further confirmed the benefits of hybrid training for visual understanding.