Training-Free Speech-Centric Omni Understanding with Frozen VLMs

The researchers proposed a training-free framework called TFO (Training-Free Omni), aimed at converting frozen video-language models (VLM) into full-modal models for speech centers. This framework was published by Ankan Deria et al. in August 2026, without the need to modify the architecture or perform multi-modal re-alignment. TFO utilizes Whisper for confidence filtering and timestamped transcription, routing audio information through the existing language interfaces of the VLM while maintaining the visual path unchanged. In 56 benchmark tests and comparisons across 21 languages, this framework was competitive with native full-modal models in audio and video understanding, improving audio performance across all five model settings and achieving significant cross-language speech gains. Additionally, frozen VLM generally retained stronger image/video understanding, visual localization, encoding, mathematical reasoning, and medical question-answering capabilities than corresponding native full-modal checkpoints. The results indicate that powerful full-modal speech center understanding can usually be achieved through modular audio-to-language routing…