How to improve the accuracy of my MLP (Current benchmark 77%)

Question

My MLP code:

# In[1]:


import pandas as pd
from keras.models import Sequential #Sequential Models
from keras.layers import Dense #Dense Fully Connected Layer Type
from keras.optimizers import SGD #Stochastic Gradient Descent Optimizer
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint


# In[2]:


data = pd.read_csv("doom.csv", delimiter=",")
data = data.drop('id', 1)
data = data.drop('cid', 1)
data = data.drop('alerts', 1)
data = data.drop('hsalerts', 1)
data = data.drop('capacity', 1)
data = data.drop('duplicate', 1)
data.shape


# In[3]:


data.selectedoption.value_counts()


# In[4]:


feature = data.drop('selectedoption',axis=1)
label = data[['selectedoption']]


# In[5]:


from sklearn.model_selection import train_test_split
train_feature, test_feature, train_label, test_label= train_test_split(
 feature, label, test_size=0.3, random_state=42)


# In[ ]:





# In[ ]:





# In[6]:


from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

scaler.fit(train_feature)
train_feature_trans = scaler.transform(train_feature)
test_feature_trans = scaler.transform(test_feature)


# In[7]:


from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

import matplotlib.pyplot as plt 
def show_train_history(train_history,train,validation):
    plt.plot(train_history.history[train])
    plt.plot(train_history.history[validation])
    plt.title('Train History')
    plt.ylabel(train)
    plt.xlabel('Epoch')
    plt.legend(['train', 'validation'], loc='best')
    plt.show()


# In[11]:


from keras import optimizers


model = Sequential() 

model.add(Dense(units=32, 
                input_dim=10, 
                kernel_initializer='uniform', 
                activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(units=32,  
                kernel_initializer='uniform', 
                activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(units=16,  
                kernel_initializer='uniform', 
                activation='softmax'))
model.add(Dropout(0.5))

model.add(Dense(units=1, 
                kernel_initializer='uniform',
                activation='sigmoid'))

print(model.summary()) 

adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)

model.compile(loss='binary_crossentropy',   
              optimizer='adam', metrics=['accuracy'])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
mc = ModelCheckpoint('best_model.h5', monitor='val_acc', mode='max', verbose=1, save_best_only=True)
train_history = model.fit(x=train_feature_trans, y=train_label,  
                          validation_split=0.8, epochs=20000, 
                          batch_size=1000, verbose=2,callbacks=[es, mc]) 

show_train_history(train_history,'acc','val_acc')
show_train_history(train_history,'loss','val_loss')


# In[9]:


scores = model.evaluate(test_feature_trans, test_label)
print('\n')
print('accuracy=',scores[1])

My dataset is this:

doom.csv

I made a few changes like, changing the optimizer, changing the learning rate, changing activation functions, changing the units. Nothing seems to work. The accuracy isn't going above 80%. Is there something that i can do to improve the accuracy over 90%? I am a newbie to this. Thank you.

Answer 1

1. Parameters optimization

As you have not mentioned the strategy, which you use for parameters choosing, I'd suggest to attend to it. For example in this article https://towardsdatascience.com/hyperparameter-optimization-with-keras-b82e6364ca53 you can find, how parameters can be chosen.

The other powerful approach is Bayesian Optimization Methods https://towardsdatascience.com/automated-machine-learning-hyperparameter-tuning-in-python-dfda59b72f8a It allows to move in parameters values space, taking into account previous results of optimization.

Note: it's crucial to consider parameters as a whole, but not just separately (if value $a$ of the parameter $A$ is not the best choice, it doesn't mean, that it won't be the best option in combination with value $b$ of the parameter $B$)

2. Deal with outliers

It would be reasonable to check outliers in the data (objects which appear to be inconsistent with the remaining instances). For this aim you can use box plot approach. It might be the reason, why it's impossible to get particular model performance.

3. Working with feature space

It also might be useful to provide feature analysis.

Answer 2

I would start with dealing with data: it seems from your code that you only have a min-max scaler which seems light.

I will say this: you cannot expect to just throw data at a neural network and expect good performance, you have to do some work on your data (that is the most important part and the hardest I think because there is no universal answer).

You can start by looking at the distribution of your data:

• try to get everything look like a bell shape (if not, maybe apply cox-box transformations)

• if your distribution is very irregular you can try to bucketize - then apply a one hot encoding

• features engineering: maybe your data can be represented another way

• feature selection: maybe all of your features are not really useful for prediction

I would try a few different approaches and compare performance improvement. This is just a start I would advise to look at Kaggle examples to get ideas of how dealing with data. I don't think there is a short answer to this question: but you are already off to a good start!

Then after (and only after) you have worked with your data, you can work more on your model (hyperparameter optimisation as suggested by Lana)