Paper page - ShallowStream: Index Shallow then Answer Deep for Streaming Video Understanding

To address the high computational costs associated with processing continuous video streams using multimodal large language models, researchers proposed the ShallowStream framework. This framework utilizes shallow networks to simultaneously perform frame encoding and index construction, and during the query phase, evidence is retrieved through shallow attention scores combined with diversity selection strategies. Experimental results show that ShallowStream’s performance is on par with the strongest existing methods, while reducing the pre-filling delay per frame and the end-to-end delay by approximately one-fifth and one-tenth of their values, respectively.