I want to train neural network on aesthetics. I am getting confused on how to go about for training data.
Assume, I have large data set of landscapes, portraits, wildlife etc which are aesthetic according to humans. But, I want to train them for the quality, the kind of colours involved, contrast levels, background blur etc.
How do I train images for this criteria? Is there a way for doing this by unsupervised learning?
Modeling aesthetics in media is an example of ordinal classification. One of the most actively maintained datasets for this can be found is Jen Aesthetics A relatively recent paper using deep learning towards aesthetics modeling is this
Prior to deep learning era, research groups were trying to translate methods/guidelines used in the photography community to create/capture good quality pictures. There are several guidelines that you can explore with a bit of search online. One popular example is the 'rule of thirds'. Here the primary subject should not be centered in the image but offset and ideally centered at the intersection of 1/3 and 2/3 horizontal and vertical lines.
This is easy to translate into an algorithm: use salient object recognition or visual attention detection and measure the distance of the center of the salient/attention patch from the 4 rule-of-thirds points. Use this off-set as a feature. The closer the salient patch is to any one of the rule-of-third points, the higher the aesthetic ordinal score for that image.
This is another good paper that explores what makes images popular.
Some researchers have also used the tags or descriptions of photos as features. The objective here is to learn an association between lexical features and image aesthetics. They have sourced their data from online repositories like Digital Photography Challenge.
This subjective task is needless to say very complex. If you plan to address it, I'd recommend beginning with a clear definition of the context within which you aim to address aesthetics.
Ideally, you'd like to map any given image (media) to some value in [0, ..., 1] $\in \mathcal{R}$. However, this is very difficult unless you have access to a lot of training data. I suggest trying instead to simplify the problem. If you can reliably map images to just two classes, good aesthetics and bad aesthetics.
You can successively generalize from binary classification to full fledged multi-class ordinal classification, for which you'll very likely have to keep increasing the depth of your CNN.
Good luck! Since, there is more to aesthetics than meets the eye! :-)
Jen dataset is more useful for me in my experiment. it has scientifically defined characteristics for aesthetics of image. When I personally look at photo, its not just the subject in it, but also how the photographer has used light, colors, background and stitched it perfectly. I was trying to see if that can be replicated. But, hard. However, your pointers are very helpful. :) Thanks a ton again.
$\endgroup$
Commented
Sep 27, 2017 at 18:15
Your question is not properly framed. Deep learning optimizes a function that maps input variables to output variables. Your input variables are the pictures (specifically, the combination of RGB pixel values). What are your output variables? That is, what specifically do you want the network to predict?
If you are trying to get the network to simply rate how aesthetically pleasing an image is on a scale of 1-10, that is fairly straightforward: simply have a large sample of human volunteers rate the images on this scale, and train the network to predict those ratings. If done properly, the network will then be able to give a fairly sophisticated prediction of whether or not a given image is aesthetically pleasing.
If you are trying to get the network to create aesthetically pleasing images, you have a long road ahead of you - this is an unsolved problem and an area of open research. Check out this post.
If you are trying to train a neural network to understand what qualities make an image aesthetically by detecting underlying patterns and communicating them to humans, you are probably using the wrong tool.
