Skip to main navigation Skip to search Skip to main content

Simultaneous Multiple-Prompt Guided Generation Using Differentiable Optimal Transport

  • Google Brain
  • ENSAE

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (Scopus)

Abstract

Recent advances in deep learning, such as powerful generative models and joint text-image embeddings, have provided the computational creativity community with new tools, opening new perspectives for artistic pursuits. Text-to-image synthesis approaches that operate by generating images from text cues provide a case in point. These images are generated with a latent vector that is progressively refined to agree with text cues. To do so, patches are sampled within the generated image, and compared with the text prompts in the common text-image embedding space; The latent vector is then updated, using gradient descent, to reduce the mean (average) distance between these patches and text cues. While this approach provides artists with ample freedom to customize the overall appearance of images, through their choice in generative models, the reliance on a simple criterion (mean of distances) often causes mode collapse: The entire image is drawn to the average of all text cues, thereby losing their diversity. To address this issue, we propose using matching techniques found in the optimal transport (OT) literature, resulting in images that are able to reflect faithfully a wide diversity of prompts. We provide numerous illustrations showing that OT avoids some of the pitfalls arising from estimating vectors with mean distances, and demonstrate the capacity of our proposed method to perform better in experiments, qualitatively and quantitatively.

Original languageEnglish
Title of host publicationProceedings of the 13th International Conference on Computational Creativity, ICCC 2022
EditorsMaria M. Hedblom, Anna Aurora Kantosalo, Roberto Confalonieri, Oliver Kutz, Tony Veale
PublisherAssociation for Computational Creativity (ACC)
Pages324-332
Number of pages9
ISBN (Electronic)9789895416042
Publication statusPublished - 1 Jan 2022
Externally publishedYes
Event13th International Conference on Computational Creativity, ICCC 2022 - Bozen-Bolzano, Italy
Duration: 27 Jun 2022 → 1 Jul 2022

Publication series

NameProceedings of the 13th International Conference on Computational Creativity, ICCC 2022

Conference

Conference13th International Conference on Computational Creativity, ICCC 2022
Country/TerritoryItaly
CityBozen-Bolzano
Period27/06/22 → 1/07/22

Fingerprint

Dive into the research topics of 'Simultaneous Multiple-Prompt Guided Generation Using Differentiable Optimal Transport'. Together they form a unique fingerprint.

Cite this