# How can I augment my image data?

What are the correct and common ways to normalize image for CNN? I used to work with text and it was pretty straightforward. Removing stop words, clean text from noise, tokenization, stemming etc. There is no problem with length we can easly add padding. I am not sure how to treat images. Grey scale is much better, less data as input. Should I rotate each image for random angle? How to pick right size of image in my traing dataset? What about adding additional filters?

(my goal is to classify 2 classes with tensorflow)

You begin by asking about image normalisation, but then refer to other techniques, which I believe all fall under "image augmentation". So I will answer the more general question:

how can I perform image augmentation to improve my model?

I would generally say that the more augmentation you can apply, the better. A caveat to that statement is that the augmentations must make sense for your target case. I will explain at the end what I mean by that.

Let's list some ways to augment an image, along with some common values:

# Augmentations

### 1. Flipping

Starting with perhaps the easiest to implement, you can (easily) flip images:

• horizontally: makes an image of an arrow pointing left point in the opposite direction - to the right, and vice-versa
• vertically: makes an arrow pointing down point up, and vice-versa.

It is more difficult, due the general rectangular shape of images, but you could flip an image diagonally, but I haven't seen anyone do that really. It is getting close to the idea of rotation, I suppose.

### 2. Rotation

In this case, we simply rotate the image about its center by a random number of degrees, usually within a range of [-5, +5] degrees. Even tiny small angles, which we can maybe hardly perceive, this is a simple trick that can add a lot of rubustness to a model.

### 3. Crop

We take "chunks" of an image, either at random or using a pre-defined pattern. This helps the model perhaps focus on certain areas of images and not be overwhelmed by perhaps unimportant features. It also simply creates more data.

You might also incorporate this augmentation method as a means to resize your input images to fit into a pre-trained model, which took input images of a different size to your data.

### 4. Normalisation

This is more of a numerical optimisation trick that one which can really be interpreted from a visual perspective. Many algorithms will be more stable whilst working with smaller numbers, as values are less likely to explode. Numbers closer together (in the sense of continuous space with a linear scale) are also more liekly to produce a smoother optimisation path.

One way of performing normalisation is:

• first computing a mean and standard deviation of the images, then
• (from all images) we subtract the mean and divide by the standard deviation.

This will produce images whose pixel values that lie over some range, but have a mean value of zero. Have a look here for some other methods and more discussion. You can normalise e.g. over whole images or just over the separate colour channels (RedGreenBlue).

One thing to keep in mind, is that the values used for optimisation must come from the training dataset itself, and must not be computed over the entire dataset including the validation/test datasets. This is because that information should not be passed to the model in any way - it is kind of cheating.

### 5. Translations

SImply move the image up, down, left or right by some amount, such as 10 pixels, or again within a range of [-5, +5] % of the image size. This will produce pixels along either one or two axes, which must be filled by another colour because the picture has moved out of the frame. It is common to use black or white for these "empty" parts, but you could also use the average pixel value or even crop the shifted image to remove them.

### 6. Rescaling

This is a less obvious augmentation step, but can add value because the model will be able to extract different features (more or less general) from larger and smaller images. A small image without much detail would only allow the model to learn higher level, or hazier features and not be able to focus on specific details that are available in high resolution images.

Methods such as progressive resizing, commonly used for training GAN architectures, start with smaller images and slowly work up to larger images. This can be done to keep training as stable as possible (in GANs it can increase robustness against mode collapse), but also coincides with the notion that early layers in a network learn high-level features, and layers deeper into e.g. a convolutional network learn representations that are more detailed. Why do the early layers require high resolution images? Let's give them low resolution images, train quicker and hopefully generalise slightly better!

### 7. Be creative!!

I once made a model for a self-driving car. I knew that the track where the model would be tested had many trees and quite a few high walls, which oth created shadows over the road. These shadows looked a lot like straight lines representing the edges of the road! So I added shadows to my images, at random intensities and random angles with random sizes. The car learnt that a change in brightness over a straight line did not necessarily mean the edge of the road and hence stopped freaking out when it reached one!

Think about the artefacts of your training data and the situations your model will face in the validation data, and try to incorporate them. This is really like feature engineering, incorporated into a model via augmentation... and it can help a lot!

### 8. Examples

Just to add a nice picture, here is a great display of how interesting variations can be made from a single image, taken from the Keras tutorial, linked below. The original image is top left and the remaing 7 are the output of combinations of a set of random augmentations:

You can have a look at an article like this, which explains augmentation quite well, with examples.

Another great example here, part of a series of articles, goes through augmentation possibilities in some detail.

# Implementations

### 1. Keras: ImageDataGenerator

This is a single class, where you can say which augmentation steps to apply and (if relevant) how much. It allows very easy integration of augmentation into a model.

Check out the official Keras tutorial for some great examples with images.

### 2. PyTorch: Transforms

You can build a small pipeline of transformations, gathering them together into a Compose object, to have fine-grained control over each augmentation step, its parameters and with which probability it is applied.

Have a look at this tutorial for an example of combining Rescale and RandomCrop.

### 3. Augmentor: Augmentation library:

This standalone library allows you build up great augmentation pipelines, which then hand off the images into a model. There are a great number of possibilities implemented, such as random warping/distortion, which looks really funky:

It also look as easy to use as the Keras solution shown above!

# Going back to the start

Now going back to my opening statement, the reason we want to apply as many augmentations as possible, is because it is synthetically creating more data for our model to learn from. We are trying to teach the model a distribution of possible data inputs and make it learn how to produce the correct output. We usually do this on a training set and then ask for its predictions on a validation set, which contains unseen data. The more data that the model has seen from the distribution/function that generates the data, the less surprised and better it will perform on the "unseen" data. We want our model to be as robust as possible to make kinds of alterations to data, and exposing it to more of that helps.

The caveat, that we only use relevant augmentations, is necessary because as soon as we start using augmentations that skew the input distribution somehow (e.g. showing the model images that make no sense1).

So use as much augmentation as possible, within the constraints of reality for your problem; as well as the time it takes to train the model ;-)

1 - Imagine training a model to detect cars on a road. If we perform a vertical flip, images shown to the model will contain cars that are essentially driving upside down; "on a grey the sky". This really doesn't make sense and will never occur in a realistic validation set or in the real world.

Disclaimer: I will try to answer the question but promote Image Augmentation Library Albumentations, which may collaborators and I develop in free time and which we believe is the best image augmentation library at the market :)

There are many ways to augment the image data.

Spatial transforms: Crops, Flips, Transpose, Elastic transform, ShiftScaleRotate, etc.

Pixel level transforms: BrightnessContrast, Gamma, shifts in RGB space, shifts in HSV space, JpegCompression, weather transforms (say add rain to the image), etc.

Which transformations to use and what parameters to choose is art, rather than science.

Those transformations that do not change data distribution: must-have. Those that change: may work may not. If you have a lot of experience they will, if not; they will make models weaker :)

Spatial transformations:

1. If your target is invariant to horizontal flips, say you need to classify dogs versus cats or perform segmentation on a street view imagery - horizontal flips should be used.

2. If you work with satellite imagery, horizontal or vertical flips, transpose, and rotations are good choices.

3. For some medical imagery, you may add elastic, and grid transforms.

Another standard spatial transform is RandomResizedCrop and ShiftScaleROtate. It helps to deal with the fact that your objects may have different sizes.

In your street view imagery, you may have a pedestrian in front of you, or far from the camera => you may zoom in / zoom out from an image and crop a piece that fits your network input. As long as your parameters for zoom are not too extreme, it serves as a good regularizer and improves the quality of your models.

Color Transforms

These are trickier. They solidly move you away from your original data distribution, but in many (but not all) situations, it adds value.

I would recommend trying to play with the parameters of your transform and check if the transformed image looks more or less natural.

One day I was working on a Kaggle competition for which applications of the JpegCompression with different strength was a must-have. Blog post with the solution.

It may happen that for the problem that you are facing some similar not widely used transform may be essential.

But it not the end of the story.

For many problems, we know that your train and test data have different distributions.

Ex: Data is split by the city, and you train on the data collected in Moscow and predict on the data from Paris. Your train comes from one hospital and test set from the other. Or you want extra confidence that your models will be more robust when the distribution changes in the production pipeline.

In these cases, you can go hardcore and augment your images in a bizarre, non-realistic way.

But this is an advanced technique. It is widely used to win Deep Learning Challenges and beat existing State Of The Art. For most problems, you may safely use flips, crops, ShiftScaleRotate, BrightNessContrast, and focus on other parts of your Deep Learning pipeline.

In the Albumentations library, we prepared a list of the jupyter notebooks that show how to apply transforms on your images.

Examples on the applications of albumentations to images, masks, bounding boxes, keypoints, weather transforms and multiple targets problems.

Feel free to check and submit an issue if something is not clear :)

• Thank you for this wonderful library! Can you guys add example of easy integration with Keras, if possible? Ideally something to replace ImageDataGenerator directly. I tried some 3rd party solution but there were problems... – Anatoly Alekseev Oct 3 '20 at 9:06
• Keras Example – Vladimir Oct 5 '20 at 18:40

Keras data argumention could be a right tool for you.

• thank you! I will certain check out keras data augmentation tool. I wish someone could tell more about how does it work under the hood. I'd like to implement some kind normalization with opencv in python (i think its good for learning new stuff) – robinloop Nov 13 '18 at 18:42
• @robinloop - If my answer didn't answer enough, it would probably make sense to thoroughly read the documentation of the ImageDataGenerator and then even start looking into the code itself! – n1k31t4 Nov 14 '18 at 19:04