I have been looking into deep learning and what caught my attention is the implementation of Autoencoder as a dimensionality reduction function for anomaly detection. I found out about it through the following link: Machine learning for anomaly detection and condition monitoring. I tried to modify my code to fit for my case where I have 5 cases in total with two datasets where the first dataset I have 50 rows and 7 columns and second dataset I have the same number of rows but 10 columns. Every 10 rows represent a case and the first case is used as training and the rest as a test dataframe. The thing I am stuck as to how to decide the number of layers and number of neurons. How can I create Autoencoder to be used for detecting anomalies?

Find below my current approach for my model.


from numpy.random import seed
from tensorflow.random import set_seed as set_random_seed
from keras.layers import Input, Dropout
from keras.layers.core import Dense 
from keras.models import Model, Sequential, load_model
from keras import regularizers
from keras.models import model_from_json
import theano.tensor as tt
import numpy as np
import pandas as pd

    ## Using AutoEncoder Nueral Network:
    data_train_df = dataframe.loc[dataframe['Label'] == 'Training']
    data_test_df_C1 = dataframe.loc[dataframe['Label'] == 'Case1']
    data_test_df_C2 = dataframe.loc[dataframe['Label'] == 'Case2']
    data_test_df_C3 = dataframe.loc[dataframe['Label'] == 'Case3']
    data_test_df_C4 = dataframe.loc[dataframe['Label'] == 'Case4']
    # Separating and concentrating on features:
    data_train_df.drop(['Label', 'Tran_Label'], inplace=True, axis=1)
    data_test_df_C1.drop(['Label', 'Tran_Label'], inplace=True, axis=1)
    data_test_df_C2.drop(['Label', 'Tran_Label'], inplace=True, axis=1)
    data_test_df_C3.drop(['Label', 'Tran_Label'], inplace=True, axis=1)
    data_test_df_C4.drop(['Label', 'Tran_Label'], inplace=True, axis=1)    
    # Getting array of dataframes:
    data_train = np.array(data_train_df.values)
    data_test_C1 = np.array(data_test_df_C1.values)
    data_test_C2 = np.array(data_test_df_C2.values)
    data_test_C3 = np.array(data_test_df_C3.values)
    data_test_C4 = np.array(data_test_df_C4.values)
    # # Getting the covariance, its inverse matrix and mean of the training data for MD:
    act_func = 'elu'
    # Input layer:
    # First hidden layer, connected to input vector X. 
    # # Test 1:
    # model.add(Dense(10,activation=act_func, kernel_initializer='glorot_uniform', kernel_regularizer=regularizers.l2(0.0), input_shape=(data_train.shape[1],)))
    # model.add(Dense(2,activation=act_func, kernel_initializer='glorot_uniform'))
    # model.add(Dense(10,activation=act_func, kernel_initializer='glorot_uniform'))
    # model.add(Dense(data_train.shape[1], kernel_initializer='glorot_uniform'))
    # model.compile(loss='mse',optimizer='adam')
    # Test 2:
    model.add(Dense(6,activation=act_func, kernel_initializer='glorot_uniform', kernel_regularizer=regularizers.l2(0.0), input_shape=(data_train.shape[1],)))
    model.add(Dense(2,activation=act_func, kernel_initializer='glorot_uniform'))
    model.add(Dense(6,activation=act_func, kernel_initializer='glorot_uniform'))
    model.add(Dense(data_train.shape[1], kernel_initializer='glorot_uniform'))
    # Train model for 100 epochs, batch size of 10: 
    # Fitting the model:
    history=model.fit(data_train_df,data_train_df, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_split=0.05, verbose = 1)
    # Plotting the model's training losses:
    plt.plot(history.history['loss'], 'b', label='Training loss')
    plt.plot(history.history['val_loss'], 'r', label='Validation loss')
    plt.legend(loc='upper right')
    plt.ylabel('Loss, [mse]')
    # plt.ylim([0,.1])
    # Distribution of loss function in the training set:
    data_pred = model.predict(np.array(data_train))
    data_pred = pd.DataFrame(data_pred)
    data_pred.index = data_train_df.index
    scored = pd.DataFrame(index=data_train_df.index)
    scored['Loss_mae'] = np.mean(np.abs(data_pred-data_train), axis = 1)
    sns.distplot(scored['Loss_mae'], bins = 10, kde= True, color = 'blue')
    plot = sns.distplot(scored['Loss_mae'], bins = 10, kde= True, color = 'blue').get_lines()[0].get_data()
    # plt.xlim(0.0, max(plot[0]))

    data_pred = model.predict(np.array(data_test_C1))
    data_pred = pd.DataFrame(data_pred)
    data_pred.index = data_test_df_C1.index
    scored_C1 = pd.DataFrame(index=data_test_df_C1.index)
    scored_C1['Loss_mae'] = np.mean(np.abs(data_pred-data_test_C1), axis = 1)
    scored_C1['Threshold'] = max(plot[0])
    scored_C1['Anomaly'] = scored_C1['Loss_mae'] > scored_C1['Threshold']
    scored_C1['Case'] = '1 Case'
    data_pred_train = model.predict(np.array(data_train))
    data_pred_train = pd.DataFrame(data_pred_train)
    data_pred_train.index = data_train_df.index
    scored_train = pd.DataFrame(index=data_train_df.index)
    scored_train['Loss_mae'] = np.mean(np.abs(data_pred_train-data_train), axis = 1)
    scored_train['Threshold'] = max(plot[0])
    scored_train['Anomaly'] = scored_train['Loss_mae'] > scored_train['Threshold']
    scored_train['Case'] = '0 Case'

    data_pred = model.predict(np.array(data_test_C2))
    data_pred = pd.DataFrame(data_pred)
    data_pred.index = data_test_df_C2.index

    scored_C2 = pd.DataFrame(index=data_test_df_C2.index)
    scored_C2['Loss_mae'] = np.mean(np.abs(data_pred-data_test_C2), axis = 1)
    scored_C2['Threshold'] = max(plot[0])
    scored_C2['Anomaly'] = scored_C2['Loss_mae'] > scored_C2['Threshold']
    scored_C2['Case'] = '2 Case'
    data_pred = model.predict(np.array(data_test_C3))
    data_pred = pd.DataFrame(data_pred)
    data_pred.index = data_test_df_C3.index
    scored_C3 = pd.DataFrame(index=data_test_df_C3.index)
    scored_C3['Loss_mae'] = np.mean(np.abs(data_pred-data_test_C3), axis = 1)
    scored_C3['Threshold'] = max(plot[0])
    scored_C3['Anomaly'] = scored_C3['Loss_mae'] > scored_C3['Threshold']
    scored_C3['Case'] = '3 Case'
    data_pred = model.predict(np.array(data_test_C4))
    data_pred = pd.DataFrame(data_pred)
    data_pred.index = data_test_df_C4.index
    scored_C4 = pd.DataFrame(index=data_test_df_C4.index)
    scored_C4['Loss_mae'] = np.mean(np.abs(data_pred-data_test_C4), axis = 1)
    scored_C4['Threshold'] = max(plot[0])
    scored_C4['Anomaly'] = scored_C4['Loss_mae'] > scored_C4['Threshold']
    scored_C4['Case'] = '4 Case'
    # scored = pd.concat([scored_train, scored], copy=False)
    # scored = scored_train.append(scored)    
    # scored_list.append(scored)

    final_scored = pd.concat([scored_train, scored_C1, scored_C2, scored_C3, scored_C4])

except Exception as e:
    print('Cause of the error:')
    print('Cannot implement Anomaly detection using Autoencoder')

Side Question/s:

  • Is there a way to decide the number of epochs and batch size?
  • 1
    $\begingroup$ These are hyperparameters that are generally decided upon by feeling and/or trail and error, there is no hard science on what the optimal values are for the number of layers and the number of neurons in each layer. A point to note in addition is that deep learning methods generally are quite data intensive, so don't expect too much when just using 50 rows. $\endgroup$
    – Oxbowerce
    Oct 14 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.