SA-LanSeg: Bridging semi-automatic interaction and language guidance for medical image segmentation

초록

Medical image segmentation plays an essential role in clinical workflows, including diagnosis, surgical planning and treatment response assessment. However, achieving reliable segmentation in challenging cases, where lesions exhibit ambiguous localization or visually similar patterns, remains difficult. Semi-automatic segmentation has become a widely adopted approach to handle these cases by utilizing user-guided hints for improved segmentation. Recently, language-guided segmentation methods have emerged as another important paradigm for medical image segmentation by leveraging clinical textual reports to provide semantic guidance, such as lesion attributes and locations. Although both strategies outperform image-based methods, they continue to remain limited. Language-guided methods mainly address semantic ambiguity (what the target is) but struggle to capture precise spatial context, while semi-automatic methods reduce spatial ambiguity (where the target is) through user-guided hints but lack sufficient semantic understanding from clinical context. To overcome these limitations, in this study, we propose a unified Semi-Automatic Language-Guided Segmentation (SA-LanSeg) framework, which synergistically integrates semantic guidance from clinical reports with location cues obtained from user-guided hints. SA-LanSeg employs a multi-branch architecture to separately model image features, clinical report semantics, and user-guided hints, and introduces a cosine-similarity fusion mechanism to adaptively integrate these complementary feature streams for accurate segmentation. We comprehensively evaluated our SA-LanSeg on four public benchmark datasets including QaTa-COV19, MosMedData, Kvasir-SEG, and Bkai-polyp, totaling 13,987 images. Our framework consistently outperforms existing image-only, language-guided, and semi-automatic segmentation methods across all benchmark datasets, and is especially effective in segmenting challenging cases that require both accurate spatial and semantic contexts.

키워드