autoencoder for features selection [closed]

Question

I m using a data set with 41 features numerics and nominals the 42 one is the class (normal or not) first I changed all the nominals features to numeric since the autoencoder requires that the imput vector should be numeric. so the number of features incresed from 42 to 122. I removed the class colomn because AE use unlabelled data and I used it to reduce dimensionality from 121 to 10 ( 121> 50->10->50-121) now I want to build a MLP to classify the data I divided the data set into 3 parts: train, validate and test set, I want to put the 10 features selected by the AE instead of the 121 but I dont know how (code?). and how add the class colomn again to the data set to do a supervised classification with MLP?

Answer 1

An autoencoder is meant to do exactly what you are asking. It is a means to take an input feature vector with $m$ values, $X \in \mathbb{R}^m$ and compress it into a vector $z \in \mathbb{R}^n$ when $n < m$. To do this we will design a network that is compressed in the middle such that it looks this.

We train this network by comparing the output $X'$ to the input $X$. This will cause $X'$ to tend towards $X$, thus despite the feature compression in the network, the output will preserve sufficient information about the input such that the input $X$ can be recovered.

Once this network is trained, we can then truncate everything after the layer which outputs the vector $z$. Then you can use the feature vector $z$ as the input features to train a different neural network which you can use to classify your instances as normal or not. During this process you will NOT tune any of the weights of the autoencoder. It will only be used as a feed-forward network.

---

The autoencoder

We will train an autoencoder on the MNIST dataset.

First we will download all the data

from keras.datasets import mnist
import numpy as np

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.


print('Training data shape: ', x_train.shape)
print('Testing data shape : ', x_test.shape)

Training data shape: (60000, 28, 28)
Testing data shape : (10000, 28, 28)

Let us see the distribution of our output classes for the MNIST data

import matplotlib.pyplot as plt
%matplotlib inline

training_counts = [None] * 10 
testing_counts = [None] * 10
for i in range(10):
    training_counts[i] = len(y_train[y_train == i])/len(y_train)
    testing_counts[i] = len(y_test[y_test == i])/len(y_test)

# the histogram of the data
train_bar = plt.bar(np.arange(10)-0.2, training_counts, align='center', color = 'r', alpha=0.75, width = 0.41, label='Training')
test_bar = plt.bar(np.arange(10)+0.2, testing_counts, align='center', color = 'b', alpha=0.75, width = 0.41, label = 'Testing')

plt.xlabel('Labels')
plt.xticks((0,1,2,3,4,5,6,7,8,9))
plt.ylabel('Count (%)')
plt.title('Label distribution in the training and test set')
plt.legend(bbox_to_anchor=(1.05, 1), handles=[train_bar, test_bar], loc=2)
plt.grid(True)
plt.show()

Let us look at some examples of the MNIST dataset

import matplotlib.pyplot as plt
%matplotlib inline

# utility function for showing images
def show_imgs(x_test, decoded_imgs=None, n=10):
    plt.figure(figsize=(20, 4))
    for i in range(n):
        ax = plt.subplot(2, n, i+1)
        plt.imshow(x_test[i].reshape(28,28))
        plt.gray()
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)

        if decoded_imgs is not None:
            ax = plt.subplot(2, n, i+ 1 +n)
            plt.imshow(decoded_imgs[i].reshape(28,28))
            plt.gray()
            ax.get_xaxis().set_visible(False)
            ax.get_yaxis().set_visible(False)
    plt.show()

show_imgs(x_train, x_test)
print('Training labels: ', y_train[0:10])
print('Testing labels : ', y_test[0:10])

Training labels: [5 0 4 1 9 2 1 3 1 4]
Testing labels : [7 2 1 0 4 1 4 9 5 9]

These are some imports we will use or not for making our model. Note: not all of these are needed but I'm too lazy to sift through and pick the useful ones.

from __future__ import print_function
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import ModelCheckpoint
from keras.models import model_from_json
from keras import backend as K
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D, Flatten, Reshape
from keras.models import Model
from keras import backend as K

Let us build our model. Notice we have the encoder, this maps the input from the higher dimension to the constrained dimension in the middle of the network. It goes from a vector of dimension 784 at the input to a vector $z$ of dimension 128. Then we have a decoder that is a mirror of the encoder which will try to decompress the vector $z$.

input_img = Input(shape=(28, 28, 1))  # adapt this if using `channels_first` image data format

input_img = Input(shape=(28, 28, 1)) # adapt this if using channels_first image data format

x = Conv2D(16, (3, 3), activation='relu', padding='same')(input_img)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = MaxPooling2D((2, 2), padding='same')(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
encoded = MaxPooling2D((2, 2), padding='same')(x)

# at this point the representation is (4, 4, 8) i.e. 128-dimensional

x = Conv2D(8, (3, 3), activation='relu', padding='same')(encoded)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(8, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
x = Conv2D(16, (3, 3), activation='relu')(x)
x = UpSampling2D((2, 2))(x)
decoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)

autoencoder = Model(input_img, decoded)
autoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')

You can see a description of the model using

autoencoder.summary()

Now let us train this model

from keras.callbacks import TensorBoard
epochs = 50
batch_size = 128

autoencoder.fit(x_train_reshaped, x_train_reshaped, epochs=epochs, batch_size=batch_size,
               shuffle=True, validation_data=(x_test_reshaped, x_test_reshaped), verbose=1,
               callbacks=[TensorBoard(log_dir='/tmp/autoencoder')])

This will take a while. I will get back to you when it is done training. However, once this is done we will take the autoencoder model and we will separate the encoder and decoder part. We will trash away the decoder and only use the encoder.

Then we will use that as a forefront to any model we want to use to classify these digits. We will pass every image through our encoder, to get this compressed information vector $z$ and we will use that as the input to our classification model.

The classifier

The classification model will then look something like

# The known number of output classes.
num_classes = 10

# Input image dimensions
img_rows, img_cols = 28, 28

# Channels go last for TensorFlow backend
x_train_reshaped = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)
x_test_reshaped = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)
input_shape = (img_rows, img_cols, 1)

# Convert class vectors to binary class matrices. This uses 1 hot encoding.
y_train_binary = keras.utils.to_categorical(y_train, num_classes)
y_test_binary = keras.utils.to_categorical(y_test, num_classes)

The classification model

model  ==  Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

We will train the classification model

# Save the model# Save t 
model_json = model.to_json()
with open("weights/model.json", "w") as json_file:
    json_file.write(model_json)

# Save the weights using a checkpoint.
filepath="weights/weights-improvement-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

epochs = 100
batch_size = 128
# Fit the model weights.
model.fit(x_train_reshaped, y_train_binary,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          callbacks=callbacks_list,
          validation_data=(x_test_reshaped, y_test_binary))

score = model.evaluate(x_test_reshaped, y_test_binary, verbose=0)
print('Model accuracy:')
print('Test loss:', score[0])
print('Test accuracy:', score[1])

print('Predict the classes: ')
prediction = model.predict_classes(x_test_reshaped[0:10])
show_imgs(x_test)
print('Predicted classes: ', prediction)