Can AI Generate Visual Content for Social Media?

Women Leading Visual Tech: Interview with Dr. Lydia Chilton

LDV Capital invests in people building businesses powered by visual technologies. We thrive on collaborating with deep tech teams leveraging computer vision, machine learning, and artificial intelligence to analyze visual data. We are the only venture capital firm with this thesis.

Our Women Leading Visual Tech series is here to showcase the leading women whose work in visual tech is reshaping business and society.

Dr. Lydia Chilton is an Assistant Professor of Computer Science at Columbia University. Her focus of interest is human-computer interaction. Her research views the design process from a computational standpoint. She is a part of the Computational Design Lab, a research group in the Computer Science Department of Columbia University, where she builds AI tools that enhance people’s productivity. She works on constructing visual metaphors for creative ads and tools that help to write humorous stories and news satire. 

Dr. Chilton received her Ph.D. from the University of Washington in 2015. She received her Master’s in Engineering from MIT in 2009 and her SB in 2007, also from MIT. Before joining Columbia Engineering in 2017, she was a postdoctoral student at Stanford University. 

Abigail Hunter-Syed spoke with Lydia about visual metaphors in creative ads, her career in academia and AI-powered tools that she builds. This topic is relevant to this year’s LDV Insights research on the future of content creation. (Note: After five years with LDV Capital, Abby decided to leave LDV to take a corporate role with fewer responsibilities that will allow her to have more time to focus on her young kids during these crazy times.)

The following is the shortened text version of the interview.

Abby: How would you describe what you do to a two-year-old?

This is how VisiBlends - an AI-powered tool - creates a visual blend for “Starbucks is here for summer”.

Lydia: I help people to make images that contain a message in them. If there are two ideas you want to relate, how do you convey that visually? That's hard and one way to do it is to take symbols of the two things that you want to relate and put them together. You can't mash them together. You make two images fit together so they convey the third message.

Abby: I know that these are called “visual metaphors”. Do you have a favorite metaphor?

Lydia: The one I like the most – “The classroom was a zoo." I saw it in a dictionary. What's so great about a metaphor is that it's up to you to decide in what way the classroom was like a zoo. Is it because the children were animals or is it something else? It's undefined and it invites the listener to imagine that. That's one of the reasons metaphors are more powerful than just saying the classroom was messy or the kids were uncontrolled. It paints a bigger picture and allows you to fill in the details from what you think about children, schools and zoos. That's where some of the richness of language comes into it.

Abby: What made you start looking at visual metaphors and how they were created?

Lydia: For a long time, I've had this obsession with trying to take complicated or what people call “creative tasks”. 

I don't want to try to define “creativity”. That's hard. But people say, “that's too hard for either a computer to do or for even a novice to do”. I don't think that's true. I think there are many layers to it and we have to understand what those layers are, make them explicit and then anyone can do it. 

I knew that this was true because I'd played the piano for a long time. Anyone who's played piano since childhood learned to play sheet music but you are never taught to compose and so you think of people like Beethoven, Mozart and Verdi as geniuses. “This music just came to them”. That's not true or it's not totally true at least... I took a music composition class and I realized there are “rules” like chord progression to make music “likable”. It's not like every song has the same melody but there's an architecture that's almost the same or that varies very little between each song. It made me think, “this is why I've been struggling to compose. I didn't know there was this structure but Mozart and Beethoven knew. These people had been taught the structure”. 

In school, we're taught to write an argumentative essay. A paragraph has a thesis sentence and then a supporting one. If you don't know that, it'll take you a long time to figure out how to write a coherent essay. If someone tells you what the right structure is, you can fit your thoughts into it, you can reinvent this from scratch.

There's architecture in everything. It’s not always exposed to people and it’s not always taught. I have two goals. One, to discover those architectures and then teach them to people. Two, if there are aspects that computers can help with, we'll do that too.

Abby: What had sent you down the computer science pathway?

Lydia: While I was in high school, the internet was just starting. I remember the first webpage I'd ever seen. It was some Star Wars fan page. People were connecting and sharing knowledge. Then we had AOL and we could chat with each other. You could pirate music. Everything was possible and new things were coming up every day. I wanted to learn to create web pages so I created a lot of crappy HTML websites.

Abby: There's a big jump between computer science and AI. How did you get on the AI path? 

Lydia: I've been on and off the AI bandwagon for my entire career. It's sort of a love-hate relationship. 

How do ideas happen? How was X created? Can we use AI to recreate it? This has always been interesting to me. How does the brain work? None of the books I read ever said anything because we don’t know the answer.

I was an economics major. The software that we had for doing data analysis was terrible and I decided that I'm going to make better, more visual software. It was my undergrad thesis. That was rewarding because you put it in front of people and they can be smarter when they use it. 

Abby: To be able to display the information in a meaningful fashion requires you to follow a certain architecture, right?

Lydia: Yes. If it’s an architecture, then there are some rules and it means that you can do it automatically with little or no input from the user. 

I got into making end-user applications. Then I went to an info session from Amazon and learned about Mechanical Turk where to get some data labeled, you put it up on this specific site. Nobody had heard of it at this point.

I also had a colleague who was trying to do some sort of AI and it wasn't working. I was thinking, “if AI can't do it. Why not just plug in a person to do it?” but everyone thought we should keep getting the AI to do it. I thought it’s expensive but it's expensive if you think of yourself as a grad student who has $20,000 a year but not if you're a big agency who has millions or billions. These costs are small from an organizational perspective. 

We convinced them and we did that for a long time but then we realized that the data that we were collecting could be used to train machines to do the same job. 

That's one of the ways that AI started to take off. Now we can get massive amounts of well-labeled and well-organized data because we can pay for people on a much larger scale to provide that data.

Abby: What's brought you to the creative side of AI?

Lydia: I see a great path towards computers doing more exciting or complicated things. We’ve got the data labeling part, what's the next layer? Well, maybe an algorithm can generate a face.

AI-generated photos © 2021 Generated Media, Inc.

I don't know if I believe in moonshots. You have to have a path from where you are to where you want to be. Going to the moon is mostly about having rockets big enough to get you there.

We have to keep building up those layers - building blocks - until we can do creative things with the help of AI. This means you almost always want to have a person in the loop. The person in charge still has a whole lot of contextual knowledge about the world. This person can make decisions and lead the creative work.

Abby: People have been creating visual metaphors for a while but now you are building an algorithm that can automatically create them. What gave you the idea that this was a good application for the things that you were working on?

Lydia: It came from intellectual curiosity. I thought, “I'm not going to try to come up with the next grand unified theory of physics or write a novel. That takes too long. I want something that can be done in a sitting.” I Googled “creative ads” in an image search and I liked them. I'm also always looking for things that have some sort of visual appeal so that the people that I'm working with can understand what it is.

It seemed like you could take any two concepts and try to come up with a message for them. I picked this opportunity because I can solve a lot of possible problems that are easy to evaluate. Not because I thought this was the world's greatest moneymaker. It's an intellectual decision. Just like I want to study creativity. This is a good vehicle to study creativity.

Abby: Interestingly, you didn't think that it could be a big moneymaker because there's a bunch of startups that think that this could indeed be a big moneymaker. They're using AI to generate ads that have better performance. Ads that have more clicks, sell more things.
One of the worries I have is that when computers get involved, they will eventually start to see the same things working well. Are they going to start creating a bunch of clickbait where all of these ads start to look similar to each other? Will our news feeds become full of visual metaphors or do you see it as a window of opportunity for ad creatives to leverage these tools to get back at their jobs?

Lydia: Let’s take AI out of the equation. People will say, “that strategy worked for that person. I'm going to write headlines like that." In journalism, if it bleeds, it leads. You turn on the news and it's all about horrible tragedies. 

This is how people work: someone discovers a pattern, others copy it. That's what we expect. Maybe the world stays like that forever but one day someone will discover another opportunity outside that area that's then been neglected because everyone's been doing the bleeds-leads thing for so long that there are other things that you can do.

That's how markets work. They go in one direction. Someone says, “let's explore that direction”. Then at some point, everyone's in that direction, there's not as much opportunity there. 

In computer science, we call it “a search process”. Everyone's searching together. We could continue to go deep on the thing that works or we could try to go broad, diversify and find something new. As soon as you develop a technology that makes something valuable, everyone's going to flock to that technology but it's not going to be the only thing because there are so many corners of the world.

Abby: Do you see it as a great opportunity for startups to get involved in?

Lydia: Definitely. Creative visuals with motion drive even more clicks. In a world like TikTok and Instagram, if you don't have a short video or an image, no one is seeing your stuff. Everyone wants visual content but you can't use stock photography anymore. Everyone's seen it. It has to be special in some way to grab your attention.

Abby: Making money isn't necessarily your end goal. Where do you want to go with it?

Lydia: It's flattering to have people want to use the tools that I build. 

It's not just creating the visuals that is hard. It's convincing the leadership that these are the right visuals to use, getting everyone to agree that this is our branding message.

For Tabasco, it’s the most obvious visual metaphor. Tabasco's got their brand but startups don't always know exactly who they are... 

We worked with a company called Leilo. They sell a lemon-flavored drink with CBD in it. It's relaxing. It's a pot derivative. So we have “lemon” and “relaxing”. We could make a bunch of blends off those two words. We filled up their Instagram campaign.

Abby: In your career, you've held positions at big tech companies. Why have you decided to do this research at an academic institution? What intrigued you about academia?

Lydia: It's one of those things that's always been inside me. You could probably do great work anywhere. Probably different work. 

In academia, there's this idea that you get more freedom. You get a lot more responsibilities. You have to raise your own money as well. I was attracted to the idea that I could work on things for their pure intellectual value. I didn't have to worry about making money.

In retrospect, I see a lot of things in the tech world are done without making money. Also, a lot of the services that tech companies put out - like search engines - are free. You have to see ads but that technology is free for everyone to use. Imagine if they charged for it. That's so valuable. I'd easily pay a couple of hundred dollars a month to be able to search the Internet.

Abby: Having your lab is a lot like having your startup. You have to get all the funding for it. You have to figure out what your priorities are because there's always going to be the time, cost, quality continuum that you're dealing with.

Lydia: Yes. You do all the hiring and firing. You create the rules. The university does help a lot of things but you still have to make it go.

Abby: Have you ever thought about spinning it out into a startup?

Lydia: I’d love to do that. The goal in academia is to find the right student who wants to take this on because a startup is more than a full-time job.

Abby: Is there another application of AI that makes you incredibly excited at the moment?

Lydia: A lot of language technology is taking off. The ability to summarize is a big deal.

Abby: We have a portfolio company Headroom that's operating in the video conferencing space. The entire premise behind what they want to be able to do is that you can have a conversation and then you get the bullet points from that meeting afterward.

Lydia: I think everyone wants that. It's hard to do but if you can get anywhere close to that, it would be amazing!

---