Referring Image Harmonization
Zhenghui Wang, Xiaoxuan Fan, Zhuangzhuang Li, Fangxiang Feng, Xiaojie Wang · 2023
Image harmonization is the process of modifying the foreground of a composite image in order to achieve a cohesive visual consistency with the background. Existing works viewed image harmonization as a purely visual task, using masks to distinguish between the foreground and background in images. However, distinguishing between foreground and background based on human intention is more intuitive. In this paper, we developed a new multimodal task, named referring image harmonization, which distinguishes between foreground and background based on text prompts to perform image harmonization. To collect the necessary data for this task, we supplement the widely used harmonization dataset iHarmony4 with referring expressions for the foreground region, creating a new harmonization dataset called ReiHarmony4. To cope with this task, we propose a segmentation-harmonization pipeline method, which first segments the foreground region based on referring expressions, and then harmonizes the foreground region. Extensive experiments demonstrate that our baseline method achieve promising results on ReiHarmony4 dataset for the proposed referring image harmonization task.