Incorporating depth information into few-shot semantic segmentation

International audience ; Few-shot segmentation presents a significant challengefor semantic scene understanding under limited supervision.Namely, this task targets at generalizing the segmentationability of the model to new categories given a few samples.In order to obtain complete scene information, we extend theRGB-centric methods to take advantage of complementary depthinformation. In this paper, we propose a two-stream deep neuralnetwork based on metric learning. Our method, known as RDNet,learns class-specific prototype representations within RGB anddepth embedding spaces, respectively. The learned prototypesprovide effective semantic guidance on the corresponding RGBand depth query image, leading to more accurate performance.Moreover, we build a novel outdoor scene dataset, known asCityscapes-3i, using labeled RGB images and depth imagesfrom the Cityscapes dataset. We also perform ablation studiesto explore the effective use of depth information in few-shotsegmentation tasks. Experiments on Cityscapes-3i show that ourmethod achieves excellent results with visual and complementarygeometric cues from only a few labeled examples.