You begin by asking about image normalisation, but then refer to other techniques, which I believe all fall under "image augmentation". So I will answer the more general question:
how can I perform image augmentation to improve my model?
I would generally say that the more augmentation you can apply, the better. A caveat to that statement is that the augmentations must make sense for your target case. I will explain at the end what I mean by that.
Let's list some ways to augment an image, along with some common values:
Starting with perhaps the easiest to implement, you can (easily) flip images:
- horizontally: makes an image of an arrow pointing left point in the opposite direction - to the right, and vice-versa
- vertically: makes an arrow pointing down point up, and vice-versa.
It is more difficult, due the general rectangular shape of images, but you could flip an image diagonally, but I haven't seen anyone do that really. It is getting close to the idea of rotation, I suppose.
In this case, we simply rotate the image about its center by a random number of degrees, usually within a range of
[-5, +5] degrees. Even tiny small angles, which we can maybe hardly perceive, this is a simple trick that can add a lot of rubustness to a model.
We take "chunks" of an image, either at random or using a pre-defined pattern. This helps the model perhaps focus on certain areas of images and not be overwhelmed by perhaps unimportant features. It also simply creates more data.
You might also incorporate this augmentation method as a means to resize your input images to fit into a pre-trained model, which took input images of a different size to your data.
This is more of a numerical optimisation trick that one which can really be interpreted from a visual perspective. Many algorithms will be more stable whilst working with smaller numbers, as values are less likely to explode. Numbers closer together (in the sense of continuous space with a linear scale) are also more liekly to produce a smoother optimisation path.
One way of performing normalisation is:
- first computing a mean and standard deviation of the images, then
- (from all images) we subtract the mean and divide by the standard deviation.
This will produce images whose pixel values that lie over some range, but have a mean value of zero. Have a look here for some other methods and more discussion. You can normalise e.g. over whole images or just over the separate colour channels (RedGreenBlue).
One thing to keep in mind, is that the values used for optimisation must come from the training dataset itself, and must not be computed over the entire dataset including the validation/test datasets. This is because that information should not be passed to the model in any way - it is kind of cheating.
SImply move the image up, down, left or right by some amount, such as 10 pixels, or again within a range of
[-5, +5] % of the image size. This will produce pixels along either one or two axes, which must be filled by another colour because the picture has moved out of the frame. It is common to use black or white for these "empty" parts, but you could also use the average pixel value or even crop the shifted image to remove them.
This is a less obvious augmentation step, but can add value because the model will be able to extract different features (more or less general) from larger and smaller images. A small image without much detail would only allow the model to learn higher level, or hazier features and not be able to focus on specific details that are available in high resolution images.
Methods such as progressive resizing, commonly used for training GAN architectures, start with smaller images and slowly work up to larger images. This can be done to keep training as stable as possible (in GANs it can increase robustness against mode collapse), but also coincides with the notion that early layers in a network learn high-level features, and layers deeper into e.g. a convolutional network learn representations that are more detailed. Why do the early layers require high resolution images? Let's give them low resolution images, train quicker and hopefully generalise slightly better!
7. Be creative!!
I once made a model for a self-driving car. I knew that the track where the model would be tested had many trees and quite a few high walls, which oth created shadows over the road. These shadows looked a lot like straight lines representing the edges of the road! So I added shadows to my images, at random intensities and random angles with random sizes. The car learnt that a change in brightness over a straight line did not necessarily mean the edge of the road and hence stopped freaking out when it reached one!
Think about the artefacts of your training data and the situations your model will face in the validation data, and try to incorporate them. This is really like feature engineering, incorporated into a model via augmentation... and it can help a lot!
Just to add a nice picture, here is a great display of how interesting variations can be made from a single image, taken from the Keras tutorial, linked below. The original image is top left and the remaing 7 are the output of combinations of a set of random augmentations:
You can have a look at an article like this, which explains augmentation quite well, with examples.
Another great example here, part of a series of articles, goes through augmentation possibilities in some detail.
This is a single class, where you can say which augmentation steps to apply and (if relevant) how much. It allows very easy integration of augmentation into a model.
Check out the official Keras tutorial for some great examples with images.
You can build a small pipeline of transformations, gathering them together into a Compose object, to have fine-grained control over each augmentation step, its parameters and with which probability it is applied.
Have a look at this tutorial for an example of combining
This standalone library allows you build up great augmentation pipelines, which then hand off the images into a model. There are a great number of possibilities implemented, such as random warping/distortion, which looks really funky:
It also look as easy to use as the Keras solution shown above!
Going back to the start
Now going back to my opening statement, the reason we want to apply as many augmentations as possible, is because it is synthetically creating more data for our model to learn from. We are trying to teach the model a distribution of possible data inputs and make it learn how to produce the correct output. We usually do this on a training set and then ask for its predictions on a validation set, which contains unseen data. The more data that the model has seen from the distribution/function that generates the data, the less surprised and better it will perform on the "unseen" data. We want our model to be as robust as possible to make kinds of alterations to data, and exposing it to more of that helps.
The caveat, that we only use relevant augmentations, is necessary because as soon as we start using augmentations that skew the input distribution somehow (e.g. showing the model images that make no sense1).
So use as much augmentation as possible, within the constraints of reality for your problem; as well as the time it takes to train the model ;-)
1 - Imagine training a model to detect cars on a road. If we perform a vertical flip, images shown to the model will contain cars that are essentially driving upside down; "on a grey the sky". This really doesn't make sense and will never occur in a realistic validation set or in the real world.