This artificial intelligence gives you the best emoji tags for your photos

When Instagram launched emoji hashtags last year, the data promised us the best guide to emoji meaning we had seen yet. Now a company called Curalate has taken it a step farther, using the Instagram data set to make Emojini, ‘the internet’s premiere emoji granting genie,’ a tool to which you can upload a photo to find out which emoji best describes it.

Sometimes the results are eerily accurate, such as in this example of a pair of dolphins swimming through someone’s vacationgram.

The Emojini, which uses deep learning to analyze the photos, has no difficulty recognizing those dorsal fins, and the other emoji seem to suggest that it recognizes the activity. It didn’t just post a wave or a drop of water, after all, but swimming and boating.

Still, tangible objects where there are corresponding emoji already in the lexicon hardly seem like a challenge, which is why I was so impressed with Emojini’s analysis of this busy street:

The obvious object emoji would be a taxi , but instead the Emojini delivers abstract associations, including the Statue of Liberty and a big apple. This feels pretty sophisticated, as it’s not just recognizing the content of the photo, but connecting it with colloquialisms. The Emojini is witty!

This makes perfect sense when we remember where the data came from: millions of Instagram users ascribing emoji to their own photos, not just as descriptors, but as hashtags. This last bit is crucial, and I think very much at the root of Emojini’s delight: when we hashtag a photo, it’s often sarcastic commentary as opposed to a straight description.

“We knew it was never going to be 100% accurate, but even when it’s wrong, it’s pretty fun,” said Lou Kratz, Lead Research Engineer at Curalate.

The company is usually focused on its visual marketing platform, which enables retailers to cull photos from users to make their Instagram accounts ‘shoppable.’ Kratz called the Emojini a ‘happy accident,’ a project that provided a way to test out some of the deep learning algorithms the company had already been working on.

Building the bot started with boiling down the data to the 500 most popular emoji, and then feeding over one million photos into a neural net. Once trained on the data, the EmojiNet would spit back the likelihood that an emoji would be related to your photo. If you’re interested in all the nuts and bolts, they can be found on the Curalate website.

Initially I was surprised that Emojini didn’t seem to return many or , but as it was explained to me it became clear that because those emoji are used so often, the images they are associated with are very broad, and therefore less meaningful. “It’s hard to find an image MOST suited for the happy face,” Katz said.

We are likely to use a heart to describe just about anything, while we are unlikely to use  to describe anything other than Autumn color or Thanksgiving pies.

As we look for a guide to emoji meaning, these sorts of tools can be invaluable in linking our use of emoji to non-verbal definitions, allowing this emerging visual language to remain explicitly visual. Deep learning could be the key to our unlocking the complex contextual meaning behind emoji. In the meantime we can all upload our own photos into Emojini for a little fun.

Cara Rose DeFabio is a pop addicted, emoji fluent, transmedia artist, focusing on live events as an experience designer for Real Future.