For my first love poem carousels, I asked my computer to match each love poem to an image from the Metropolitan Museum of Art. For version 2.0, I asked my computer to illustrate each poem, all by itself.
The images were created with a text-to-image generator called AttnGAN, and I used Spacy, a natural language proccessing tool to help choose which part of the poem to use. For this project, I chose to generate an image based on the longest "noun chunk" in each poem. I hoped the long noun chunk would capture a well-described object, and sometimes, it did. For Edgar Lee Masters's My Light with Yours, the noun chunk my computer identified was "the desert sand." More often, the noun chunks were less concrete. For Frances Ellen Watkins Harper's poem Advice to Girls the noun chunk was "a friendly word." For Rita Dove's American Smooth, "swift and serene magnificence." For Edna St. Vincent Millay's I think I should have loved you presently, "all my pretty follies."
I came across a number of text-to-image generators (text2scene, SG2IM, stackGAN, objGAN, HDGan), all of which I hope to try out eventually. I chose AttnGAN for this project because I discovered a hosted version on RunwayML, which is dedicated to bringing machine learning to creators via an "intuitive and simple visual interface." The model was indeed easy to use. For anyone interested in learning more, details below. I hope you enjoy the new carousel!
Not long ago, a PetaPixel story appeared in my news feed: This AI Generates Photos Using Only Text Captions as a Guide. Naturally, I clicked the link and learned more about Allen Institute for Artificial Intelligence's new machine learning algorithm, LXMERT, which has a fun online demo here.
RunwayML has a web interface, which permitted me to try out the model without doing much more than typing in a caption and hitting run. I just needed to decide which text to use for each poem. The title? First line? First twenty words? Last twenty words? I looked at the exmples in the AttnGAN paper and decided that I'd try to keep things simple.
I used Spacy to parse the text of each poem and identify the noun chunks. The Spacy docs describe noun chunks as "flat phrases that have a noun as their head. You can think of noun chunks as a noun plus the words describing the noun – for example, “the lavish green grass” or “the world’s largest tech fund”)." I then selected the longest chunk from each poem, with hopes that the longer the chunk, the more descriptive.
Once I had the noun chunks, I ran them by the AttnGAN model and generated a picture for each. I chose to use a hosted model and followed the example Frederik De Bleser describes in his six-minute video tutorial to access the results using JavaScript.
Here are a few of my favorites:
Want more? See another computer-generated-image love carousel, a sentiment carousel, a category carousel and more here.
Tweet