RealCustom: Narrowing Real Text Word for Real-Time Open-Domain Text-to-Image Customization
Mengqi Huang, Zhendong Mao, Mingcong Liu, Qian He, Yongdong Zhang · 2024
Text-to-image customization, which aims to synthesize text-driven images for the given subjects, has recently rev-olutionized content creation. Existing works follow the pseudo-word paradigm, i.e., represent the given subjects as pseudo-words and then compose them with the given text. However, the inherent entangled influence scope of pseudo-words with the given text results in a dual-optimum para-dox, i.e., the similarity of the given subjects and the con-trollability of the given text could not be optimal simultane-ously. We present RealCustom that, for the first time, dis-entangles similarity from controllability by precisely lim-iting subject influence to relevant parts only, achieved by gradually narrowing real text word from its general conno-tation to the specific subject and using its cross-attention to distinguish relevance. Specifically, RealCustom intro-duces a novel “train-inference” decoupled framework: (1) during training, RealCustom learns general alignment be-tween visual conditions to original textual conditions by a novel adaptive scoring module to adaptively modulate influence quantity; (2) during inference, a novel adaptive mask guidance strategy is proposed to iteratively update the influence scope and influence quantity of the given sub-jects to gradually narrow the generation of the real text word. Comprehensive experiments demonstrate the supe-rior real-time customization ability of RealCustom in the open domain, achieving both unprecedented similarity of the given subjects and controllability of the given text for the first time. The project page is h t tps: / / cor 1 eone-huang.github.io/realcustom