TY - GEN
T1 - Simultaneous Multiple-Prompt Guided Generation Using Differentiable Optimal Transport
AU - Tian, Yingtao
AU - Cuturi, Marco
AU - Ha, David
N1 - Publisher Copyright:
© 2022 Proceedings of the 13th International Conference on Computational Creativity, ICCC 2022. All rights reserved.
PY - 2022/1/1
Y1 - 2022/1/1
N2 - Recent advances in deep learning, such as powerful generative models and joint text-image embeddings, have provided the computational creativity community with new tools, opening new perspectives for artistic pursuits. Text-to-image synthesis approaches that operate by generating images from text cues provide a case in point. These images are generated with a latent vector that is progressively refined to agree with text cues. To do so, patches are sampled within the generated image, and compared with the text prompts in the common text-image embedding space; The latent vector is then updated, using gradient descent, to reduce the mean (average) distance between these patches and text cues. While this approach provides artists with ample freedom to customize the overall appearance of images, through their choice in generative models, the reliance on a simple criterion (mean of distances) often causes mode collapse: The entire image is drawn to the average of all text cues, thereby losing their diversity. To address this issue, we propose using matching techniques found in the optimal transport (OT) literature, resulting in images that are able to reflect faithfully a wide diversity of prompts. We provide numerous illustrations showing that OT avoids some of the pitfalls arising from estimating vectors with mean distances, and demonstrate the capacity of our proposed method to perform better in experiments, qualitatively and quantitatively.
AB - Recent advances in deep learning, such as powerful generative models and joint text-image embeddings, have provided the computational creativity community with new tools, opening new perspectives for artistic pursuits. Text-to-image synthesis approaches that operate by generating images from text cues provide a case in point. These images are generated with a latent vector that is progressively refined to agree with text cues. To do so, patches are sampled within the generated image, and compared with the text prompts in the common text-image embedding space; The latent vector is then updated, using gradient descent, to reduce the mean (average) distance between these patches and text cues. While this approach provides artists with ample freedom to customize the overall appearance of images, through their choice in generative models, the reliance on a simple criterion (mean of distances) often causes mode collapse: The entire image is drawn to the average of all text cues, thereby losing their diversity. To address this issue, we propose using matching techniques found in the optimal transport (OT) literature, resulting in images that are able to reflect faithfully a wide diversity of prompts. We provide numerous illustrations showing that OT avoids some of the pitfalls arising from estimating vectors with mean distances, and demonstrate the capacity of our proposed method to perform better in experiments, qualitatively and quantitatively.
M3 - Conference contribution
AN - SCOPUS:85171877216
T3 - Proceedings of the 13th International Conference on Computational Creativity, ICCC 2022
SP - 324
EP - 332
BT - Proceedings of the 13th International Conference on Computational Creativity, ICCC 2022
A2 - Hedblom, Maria M.
A2 - Kantosalo, Anna Aurora
A2 - Confalonieri, Roberto
A2 - Kutz, Oliver
A2 - Veale, Tony
PB - Association for Computational Creativity (ACC)
T2 - 13th International Conference on Computational Creativity, ICCC 2022
Y2 - 27 June 2022 through 1 July 2022
ER -