My Computer Illustrates Love Poems

Poems (by Humans) with Visuals (by Machine)

{{ Give me a new love carousel }}

For my first love poem carousels, I asked my computer to match each love poem to an image from the Metropolitan Museum of Art. For version 2.0, I asked my computer to illustrate each poem, all by itself.

The images were created with a text-to-image generator called AttnGAN, and I used Spacy, a natural language proccessing tool to help choose which part of the poem to use. For this project, I chose to generate an image based on the longest "noun chunk" in each poem. I hoped the long noun chunk would capture a well-described object, and sometimes, it did. For Edgar Lee Masters's My Light with Yours, the noun chunk my computer identified was "the desert sand." More often, the noun chunks were less concrete. For Frances Ellen Watkins Harper's poem Advice to Girls the noun chunk was "a friendly word." For Rita Dove's American Smooth, "swift and serene magnificence." For Edna St. Vincent Millay's I think I should have loved you presently, "all my pretty follies."

Computer-generated image for My Light with Yours by Edgar Lee Masters
"the desert sand'
Computer-generated image
'a friendly word'
Computer-generated image
'swift and serene magnificence'
Computer-generated image
'all my pretty follies'

I came across a number of text-to-image generators (text2scene, SG2IM, stackGAN, objGAN, HDGan), all of which I hope to try out eventually. I chose AttnGAN for this project because I discovered a hosted version on RunwayML, which is dedicated to bringing machine learning to creators via an "intuitive and simple visual interface." The model was indeed easy to use. For anyone interested in learning more, details below. I hope you enjoy the new carousel!

Here’s What Happened:

Computer-generated image for Want Could Kill Me
Want Could Kill Me by Xandria Phillips
Generated by computer with AttnGAN

Not long ago, a PetaPixel story appeared in my news feed: This AI Generates Photos Using Only Text Captions as a Guide. Naturally, I clicked the link and learned more about Allen Institute for Artificial Intelligence's new machine learning algorithm, LXMERT, which has a fun online demo here.

The LXMERT-generated images had a haunting quality and I knew at once that I wanted to make a set for the love poems. As the LXMERT code has not yet been released, I began searching for other options, which is how I discovered the hosted AttnGAN model on RunwayML.

RunwayML has a web interface, which permitted me to try out the model without doing much more than typing in a caption and hitting run. I just needed to decide which text to use for each poem. The title? First line? First twenty words? Last twenty words? I looked at the exmples in the AttnGAN paper and decided that I'd try to keep things simple.

I used Spacy to parse the text of each poem and identify the noun chunks. The Spacy docs describe noun chunks as "flat phrases that have a noun as their head. You can think of noun chunks as a noun plus the words describing the noun – for example, “the lavish green grass” or “the world’s largest tech fund”)." I then selected the longest chunk from each poem, with hopes that the longer the chunk, the more descriptive.

Once I had the noun chunks, I ran them by the AttnGAN model and generated a picture for each. I chose to use a hosted model and followed the example Frederik De Bleser describes in his six-minute video tutorial to access the results using JavaScript.

Here are a few of my favorites:

Computer-generated image for My Light with Yours by Edgar Lee Masters
'Great, rough-bearded forests'
North Wind by Lola Ridge
Computer-generated image
'Sweet and bitter waters'
Sink Your Fingers into the Darkness of my Fur by Ellen Bass
Computer-generated image
'A strange romance'
The Picnic by the Ocean by Grace Cavalieri
Computer-generated image
'a large yellow parasol'
This Poem is Asking For Your Love by Grace Cavalieri
Computer-generated image for My Light with Yours by Edgar Lee Masters
'the red vervain weed'
Simaetha by H.D.
Computer-generated image
'the sleek deer-hounds circle'
Circe by H.D.
Computer-generated image
"There Passion's glowing crimson hue"
The Rainbow by Effie Waller Smith
Computer-generated image
'all true companions'
The Rats by Keetje Kuipers
Computer-generated image
'the old flat-front desk'
be specific by Fred L. Joiner
Computer-generated image
'that hot sweet hour'
On the Hill-Side by Radclyffe Hall
Computer-generated image
''the rocky broken spurs'
Thera by Brian Turner
Computer-generated image
'before her extended, Dreary and vast and silent, the desert'
Evangeline: A Tale of Acadie by Henry Wadsworth Longfellow
Computer-generated image
'An apothecary bottle'
from "Like Nebraska" by Sophie Klahr
Computer-generated image
'secret and divine signs'
Among the Multitude by Walt Whitman
Computer-generated image
'the strong and arrogant woman'
Unfolded Out of the Folds by Walt Whitman
Computer-generated image
'the autumn love'
Songs (VIII) by E. E. Cummings

Want more? See another computer-generated-image love carousel, a sentiment carousel, a category carousel and more here.


Further reading

Xu, Tao, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. "AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1316-1324. 2018.