# How to use a deep learning algorithm to cluster image *styles* in an unlabeled data set?

I have a hard problem, and I'd be interested in hearing people's thoughts. I have a set of images depicting a large variety of phenomena, but only a few styles. The images are already labeled by phenomenon, but not by style. The distinction between image styles is largely based on what technology was used to produce the images. We want to use a clustering algorithm to separate the different styles.

For an analogous problem, imagine that you had a very large dataset of paintings, but with many kinds of different subjects and only a few styles. Further imagine that you don't care what the subject of the painting is, only about its stylistic features (e.g. watercolor vs oil vs acrylic, realistic vs photorealistic vs impressionist vs expressionist). How could you try to go about "telling" a deep learning algorithm to emphasize these stylistic features over what the subject of the painting is, without using any labels?

## 2 Answers

You can use a pretrained model, extract features, and try to cluster images based on these features. The success will be contingent on the "style" you want to distinguish (and if the derived features represent the different styles). I had a similar problem and used Kerans and VGG16 features to cluster images using KNN. Find my Python code on Github.

If you use a pretrained model, it will be normally be trained for object detection/classification, so the features it extracts are precisely optimized to be independent of style. It should detect a car in watercolor as well as in oil paint. So I think that’s unlikely to work.

You need a model that transforms your images to a representation where very different image in the same style are close together. Then you can run a clustering algorithm like K-means to find the centroids, which are your style stereotypes.

To get that model, I don’t see how you can do without some kind of supervision (manual labeling). So you label as many images as you can bear, and treat it as a multi class classification problem, using a network that is part generating a representation (using convolutional layers) and some fully connected layers for the classification. Then you drop the classification layers, and you run k-means on the representations.

• Is there a way to use the style-independence of those features to our advantage? What if we trained a convolutional autoencoder, and between the encoder and decoder, we add information about the type and shape of the object, so that the encoder will be encouraged to learn stylistic features, since the object classification features are already given to the decoder? – Zorgoth Aug 15 '19 at 14:21
• I don’t think so. Imagine you had a linear regression that predicts how nice an image looks. Knowing the features of this regression is useless if you want to do something completely unrelated to “nice-ness”. – Paul Aug 15 '19 at 18:20