Bridging Modalities and Tasks: A Unified Hierarchical ViT for SAR-to-Optical Translation and Semantic Segmentation

The researchers proposed a unified hierarchical visual Transformer (ViT) framework called BMT, aimed at simultaneously addressing the tasks of translating SAR images into optical images and performing semantic segmentation. This framework optimizes both tasks through a shared hierarchical visual Transformer, integrating local ViT blocks, an enhanced output module, a control network-based conditional injection mechanism, and a bounded Kendall uncertainty weighting scheme. Experimental evaluations were conducted on the WHU-OPT-SAR paired dataset and the antidual datasets built based on HRSID and DIOR, and the results show that this method is competitive in terms of image translation quality and segmentation performance. Currently, the relevant datasets and source code are available publicly.