Data augmentation: rotating images and zero values

Question

A lot of people rotate images to create a larger training set for neural networks. For most nets, all of the inputs have to be the same size so the image rotation function has to crop the newly rotated images to match the input size. So, say you have $32\times32$ resolution images and do 45 degree rotations. Some of the output images will look diamond shaped with black (zero values) in the corners. So, my question is: should you leave these zero values alone or change them in some way and if so, how?

Answer 1

Keeping the values as zeros will introduce some bias to your network. Given, you have this corner effect for the majority of your dataset, you do not want the network to identify a high probability of making the corners black. Thus, you should fill them, you can extend the edge, do a reflection, wrapping. You can also do some more complex function, like take average of a few patches in your image then place them in the missing areas.

Keras has a very nice function that can do all this for you.

import numpy as np
from keras.datasets import mnist
from keras.preprocessing.image import ImageDataGenerator
import matplotlib.pyplot as plt
%matplotlib inline

# load data
(X_train, y_train), (X_test, y_test) = mnist.load_data()
X_train = X_train.astype('float32')

# set up your data generator
datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rotation_range=15,
    width_shift_range=0.2,
    height_shift_range=0.2,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    vertical_flip = False)

# Fit the generator using your data
datagen.fit(X_train.reshape((len(X_train), 28, 28, 1)))

# Black images
image = X_train[5]
plt.imshow(image,  cmap='gray')
plt.show()

plt.figure(figsize=(12,12))
plt.subplot(4, 4, 1)
plt.imshow(image.reshape((28,28)),  cmap='gray')

for j in range(15):
    augmented = datagen.random_transform(image.reshape((28,28,1)))
    plt.subplot(4, 4, j+2)
    plt.imshow(augmented.reshape((28,28)),  cmap='gray')
plt.tight_layout()
plt.show()

# White images
image = -1*X_train[5]
plt.imshow(image,  cmap='gray')
plt.show()

plt.figure(figsize=(12,12))
plt.subplot(4, 4, 1)
plt.imshow(image.reshape((28,28)),  cmap='gray')

for j in range(15):
    augmented = datagen.random_transform(image.reshape((28,28,1)))
    plt.subplot(4, 4, j+2)
    plt.imshow(augmented.reshape((28,28)),  cmap='gray')
plt.tight_layout()
plt.show()