0
$\begingroup$

For my degree's final project I am working with Keras and trying to build different AI models. I'm having trouble with an MLP. First I preprocess the UNSW-NB15 dataset and later use it as input in a NN. Here is the preprocess:

import pandas as pd
import numpy as np
from tensorflow.keras.utils import get_file

import numpy as np
import tensorflow.keras
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation
from sklearn.model_selection import train_test_split
from tensorflow.keras.callbacks import EarlyStopping
from sklearn import metrics

from sklearn.preprocessing import OrdinalEncoder

dfs = []
for i in range(1,5):
    path = './UNSW-NB15_{}.csv'# There are 4 input csv files
    dfs.append(pd.read_csv(path.format(i),dtype={'attack_cat': str, 'ct_ftp_cmd' : int }, header = None))
all_data = pd.concat(dfs).reset_index(drop=True)  # Concat all to a single df

 This csv file contains names of all the features
df_col = pd.read_csv('./NUSW-NB15_features.csv', encoding='ISO-8859-1')
 Making column names lower case, removing spaces
df_col['Name'] = df_col['Name'].apply(lambda x: x.strip().replace(' ', '').lower())
 Renaming our dataframe with proper column names
all_data.columns = df_col['Name']

all_data.drop(['srcip', 'sport', 'dstip', 'dsport', 'ct_ftp_cmd','state'],axis=1, inplace=True)
all_data['attack_cat'] = all_data['attack_cat'].str.strip()
all_data['attack_cat'] = all_data['attack_cat'].replace(['Backdoors'], 'Backdoor')
all_data.groupby('attack_cat')['attack_cat'].count()
all_data["attack_cat"] = all_data["attack_cat"].fillna('Normal')
all_data.groupby('attack_cat')['attack_cat'].count()
all_data.drop(all_data[all_data['is_ftp_login'] >= 2.0].index, inplace = True)
all_data['ct_flw_http_mthd'] = all_data['ct_flw_http_mthd'].fillna(0)
all_data['is_ftp_login'] = all_data['is_ftp_login'].fillna(0)

ord_enc = OrdinalEncoder()
all_data['encoded'] = ord_enc.fit_transform(all_data[['attack_cat']])
all_data['encoded_int'] = all_data['encoded'].astype(int)
all_data.drop(['encoded'],axis=1)
all_data.drop(['encoded_int'],axis=1)

ord_enc_serv = OrdinalEncoder()
all_data['encoded_serv'] = ord_enc_serv.fit_transform(all_data[['service']])
all_data['int_serv'] = all_data['encoded_serv'].astype(int)
all_data.drop(['encoded_serv'],axis=1)
all_data.drop(['encoded','encoded_serv'],axis=1)

ord_enc_proto = OrdinalEncoder()
all_data['encoded_proto'] = ord_enc.fit_transform(all_data[['proto']])
all_data['int_proto'] = all_data['encoded_proto'].astype(int)
all_data= all_data.drop(['encoded_proto'],axis=1)

encoded_data = all_data.drop(['encoded','encoded_serv','proto','service','attack_cat','label'],axis=1)

multi_data = encoded_data

X = multi_data.drop(columns=['encoded_int'],axis=1)
Y = multi_data['encoded_int']

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,Y, train_size = 0.8, test_size = 0.2, random_state = 0,stratify = Y)

And this is the model:

model = Sequential()
model.add(Dense(X.shape[1], input_dim= X.shape[1], activation= 'relu'))
model.add(Dense(2*X.shape[1]+1, activation= 'relu'))
model.add(Dense(2*X.shape[1]+1,activation= 'relu'))
model.add(Dense(10,activation= 'softmax', kernel_initializer='normal'))
model.compile(loss= 'sparse_categorical_crossentropy', optimizer= 'adam', metrics= ['accuracy'])
monitor = EarlyStopping(monitor='val_loss', min_delta=1e-3, patience=5, 
                        verbose=1, mode='auto', restore_best_weights=True)

model.fit(X_train,y_train,validation_data=(X_test,y_test),
          callbacks=[monitor],batch_size=10000,verbose=2, epochs=100)

 Evaluate the model on the test set
from sklearn.metrics import classification_report
y_pred = model.predict(X_test)
y_pred_classes = y_pred
print(classification_report(y_test, y_pred_classes))

Right now the notebook throws this error:

Classification metrics can't handle a mix of multiclass and continuous-multioutput targets

I need some help as I have made several versions and I can't get to the right one, I don't know if it's something wrong in the model or whatever but I would like to obtain the accuracy score and the metrics to see for each class of attack its values.

Thanks for your help.

Edit:

Finally I've been able to make it work. It now throws an accuracy value of ~71%. What I've made is select the 20 most important features and then normalize the data. I have obtained that 71% with the activation function sigmoid on every layer except the output layer. I'm trying different configurations and activation functions. How can I increase accuracy?

enter image description here

enter image description here

$\endgroup$

1 Answer 1

0
$\begingroup$

The line where you call classification_report is throwing Classification metrics can't handle a mix of multiclass and continuous-multioutput targets because y_preds_classes is output from softmax so it will be a (507973,10) array of floats, and y_test is a (507973) array of ints - i.e. you need to convert the output of softmax back to categories in order to compare to your ground truth labels.

Something like this y_preds = np.argmax(y_pred_classes, axis=1) will allow you to call classification_report without throwing. You can also add target names, so something like:

target_names=["attack0", "attack1", "attack2", "attack3", "attack4", "attack5", "attack6", "attack7", "attack8", "attack9" ]
y_preds = np.argmax(y_pred_classes, axis=1)
print(classification_report(y_test, y_preds, target_names=target_names))

enter image description here

I would recommend looking at the confusion matrix as well.

from sklearn.metrics import multilabel_confusion_matrix
multilabel_confusion_matrix(y_test, y_preds) 

or

import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay
ConfusionMatrixDisplay.from_predictions(y_test, y_preds)
plt.show()

enter image description here

This will tell you that the trained classifier is simply classifying every as class 6. A good starting point would be to understand the data distributions of your classes. There are many approaches to dealing with highly skewed input data including over and undersampling, or weighting classes according to their input distributions. This is a good tutorial on weighting classes in keras: https://www.tensorflow.org/tutorials/structured_data/imbalanced_data

$\endgroup$
3
  • $\begingroup$ I have executed your code and it's returns a similar classification report to the one you attached. I don't understand why for other algorithms that I tried the results are correct but using this NN it's imposible to get a coherent result. $\endgroup$ May 15, 2023 at 10:24
  • $\begingroup$ Yes, handling imbalanced data is a big topic in itself. This stats.stackexchange.com/questions/357466/… is a good discussion of the problem and some solutions. If this fixes the error about classification metrics, please accept+upvote as answer. $\endgroup$ May 16, 2023 at 8:01
  • $\begingroup$ Sorry I hadn't included in the code that I have applied SMOTE. I am trying different optimizers and activation functions but the results are still wrong. $\endgroup$ May 16, 2023 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.