Color Diffuser: Photo-Realistic Image Colorization via Diffusion Feature Extractor

Colorizing images is a challenging problem due to its multi-modal uncertainty and high degree of ill-posedness. Directly training a deep neural network often results in inaccurate semantic colors and a lack of color richness. Although recent methods offer improved outcomes, the capability to extract semantic information and the quality of image generation are still limited. To further improve the results, we propose Color Diffuser, a comprehensive approach to recovering vivid colors by leveraging the rich and diverse color priors encapsulated in the pre-trained diffusion model. Specifically, our method comprises a pixel decoder and a query-based color decoder. The pixel decoder integrates the semantic features obtained from the diffusion inversion process. The color decoder utilizes rich semantic features to refine color queries, thus eliminating the need for manually designed priors. Through cross-attention, our two decoders collaborate to establish correlations between color and multi-scale semantic representations, significantly mitigating the color bleeding effect. Extensive experiments demonstrate that Color Diffuser outperforms existing state-of-the-art methods qualitatively.