Illumination Adaptation for SAM to Achieve Accurate Segmentation of Images Taken in Low-Light Scenes

Achieving accurate segmentation in low-light scenes is challenging due to 1) severe domain shift encountered when models trained on daylight data are applied to such scenes and 2) lack of large-scale fine-grained labels in low-light conditions. A good idea is to use the generalization capabilities of segmentation foundation models like Segment Anything Model (SAM) to address the scarcity of annotated data. However, applying SAM to low-light scenes faces a severe domain shift issue due to the lack of inductive bias in effectively transforming low-light features into natural-light ones. To address this issue, we propose to adapt SAM for low-light scenes. To reduce the reliance on labels of low-light data, we develop a self-training method that makes SAM generate source-free predictions. To reduce the domain gap between low-light target data and SAM's natural-light trained data, we design a transformation head that enhances low-light features prior to the application of SAM. We further propose a domain shift compensation loss that trains our model to select a domain-adaptation-optimal illumination-enhanced feature map. Experimental results demonstrate that our method well outperforms the state of the art on the Dark Zurich and Nighttime Driving datasets. Code is available at https://github.com/HongminμSALS.