# How to decide number of hidden layers and number of neurons for Autoencoder for dimensionality reduction function?

I have been looking into deep learning and what caught my attention is the implementation of Autoencoder as a dimensionality reduction function for anomaly detection. I found out about it through the following link: Machine learning for anomaly detection and condition monitoring. I tried to modify my code to fit for my case where I have 5 cases in total with two datasets where the first dataset I have 50 rows and 7 columns and second dataset I have the same number of rows but 10 columns. Every 10 rows represent a case and the first case is used as training and the rest as a test dataframe. The thing I am stuck as to how to decide the number of layers and number of neurons. How can I create Autoencoder to be used for detecting anomalies?

Find below my current approach for my model.

Code:

from numpy.random import seed
from tensorflow.random import set_seed as set_random_seed
from keras.layers import Input, Dropout
from keras.layers.core import Dense
from keras.models import Model, Sequential, load_model
from keras import regularizers
from keras.models import model_from_json
import theano.tensor as tt
import numpy as np
import pandas as pd

try:
## Using AutoEncoder Nueral Network:
data_train_df = dataframe.loc[dataframe['Label'] == 'Training']
data_test_df_C1 = dataframe.loc[dataframe['Label'] == 'Case1']
data_test_df_C2 = dataframe.loc[dataframe['Label'] == 'Case2']
data_test_df_C3 = dataframe.loc[dataframe['Label'] == 'Case3']
data_test_df_C4 = dataframe.loc[dataframe['Label'] == 'Case4']

# Separating and concentrating on features:
data_train_df.drop(['Label', 'Tran_Label'], inplace=True, axis=1)
data_test_df_C1.drop(['Label', 'Tran_Label'], inplace=True, axis=1)
data_test_df_C2.drop(['Label', 'Tran_Label'], inplace=True, axis=1)
data_test_df_C3.drop(['Label', 'Tran_Label'], inplace=True, axis=1)
data_test_df_C4.drop(['Label', 'Tran_Label'], inplace=True, axis=1)

# Getting array of dataframes:
data_train = np.array(data_train_df.values)
data_test_C1 = np.array(data_test_df_C1.values)
data_test_C2 = np.array(data_test_df_C2.values)
data_test_C3 = np.array(data_test_df_C3.values)
data_test_C4 = np.array(data_test_df_C4.values)

# # Getting the covariance, its inverse matrix and mean of the training data for MD:
seed(10)
set_random_seed(10)
act_func = 'elu'

# Input layer:
model=Sequential()
# First hidden layer, connected to input vector X.
# # Test 1:

# Test 2:

# Train model for 100 epochs, batch size of 10:
NUM_EPOCHS=150
BATCH_SIZE=10

# Fitting the model:
history=model.fit(data_train_df,data_train_df, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_split=0.05, verbose = 1)

# Plotting the model's training losses:
plt.plot(history.history['loss'], 'b', label='Training loss')
plt.plot(history.history['val_loss'], 'r', label='Validation loss')
plt.legend(loc='upper right')
plt.xlabel('Epochs')
plt.ylabel('Loss, [mse]')
# plt.ylim([0,.1])
plt.show()

# Distribution of loss function in the training set:
data_pred = model.predict(np.array(data_train))
data_pred = pd.DataFrame(data_pred)
data_pred.index = data_train_df.index

scored = pd.DataFrame(index=data_train_df.index)
scored['Loss_mae'] = np.mean(np.abs(data_pred-data_train), axis = 1)
plt.figure()
sns.distplot(scored['Loss_mae'], bins = 10, kde= True, color = 'blue')
plot = sns.distplot(scored['Loss_mae'], bins = 10, kde= True, color = 'blue').get_lines()[0].get_data()
# plt.xlim(0.0, max(plot[0]))
plt.show()

data_pred = model.predict(np.array(data_test_C1))
data_pred = pd.DataFrame(data_pred)
data_pred.index = data_test_df_C1.index

scored_C1 = pd.DataFrame(index=data_test_df_C1.index)
scored_C1['Loss_mae'] = np.mean(np.abs(data_pred-data_test_C1), axis = 1)
scored_C1['Threshold'] = max(plot[0])
scored_C1['Anomaly'] = scored_C1['Loss_mae'] > scored_C1['Threshold']
scored_C1['Case'] = '1 Case'

data_pred_train = model.predict(np.array(data_train))
data_pred_train = pd.DataFrame(data_pred_train)
data_pred_train.index = data_train_df.index

scored_train = pd.DataFrame(index=data_train_df.index)
scored_train['Loss_mae'] = np.mean(np.abs(data_pred_train-data_train), axis = 1)
scored_train['Threshold'] = max(plot[0])
scored_train['Anomaly'] = scored_train['Loss_mae'] > scored_train['Threshold']
scored_train['Case'] = '0 Case'

data_pred = model.predict(np.array(data_test_C2))
data_pred = pd.DataFrame(data_pred)
data_pred.index = data_test_df_C2.index

scored_C2 = pd.DataFrame(index=data_test_df_C2.index)
scored_C2['Loss_mae'] = np.mean(np.abs(data_pred-data_test_C2), axis = 1)
scored_C2['Threshold'] = max(plot[0])
scored_C2['Anomaly'] = scored_C2['Loss_mae'] > scored_C2['Threshold']
scored_C2['Case'] = '2 Case'

data_pred = model.predict(np.array(data_test_C3))
data_pred = pd.DataFrame(data_pred)
data_pred.index = data_test_df_C3.index

scored_C3 = pd.DataFrame(index=data_test_df_C3.index)
scored_C3['Loss_mae'] = np.mean(np.abs(data_pred-data_test_C3), axis = 1)
scored_C3['Threshold'] = max(plot[0])
scored_C3['Anomaly'] = scored_C3['Loss_mae'] > scored_C3['Threshold']
scored_C3['Case'] = '3 Case'

data_pred = model.predict(np.array(data_test_C4))
data_pred = pd.DataFrame(data_pred)
data_pred.index = data_test_df_C4.index

scored_C4 = pd.DataFrame(index=data_test_df_C4.index)
scored_C4['Loss_mae'] = np.mean(np.abs(data_pred-data_test_C4), axis = 1)
scored_C4['Threshold'] = max(plot[0])
scored_C4['Anomaly'] = scored_C4['Loss_mae'] > scored_C4['Threshold']
scored_C4['Case'] = '4 Case'

# scored = pd.concat([scored_train, scored], copy=False)
# scored = scored_train.append(scored)
# scored_list.append(scored)

final_scored = pd.concat([scored_train, scored_C1, scored_C2, scored_C3, scored_C4])
print(final_scored)

except Exception as e:
print('Cause of the error:')
print(e)
print('Cannot implement Anomaly detection using Autoencoder')
pass


Side Question/s:

• Is there a way to decide the number of epochs and batch size?
• These are hyperparameters that are generally decided upon by feeling and/or trail and error, there is no hard science on what the optimal values are for the number of layers and the number of neurons in each layer. A point to note in addition is that deep learning methods generally are quite data intensive, so don't expect too much when just using 50 rows. Oct 14 at 14:39