0
$\begingroup$

My MLP code:

# In[1]:


import pandas as pd
from keras.models import Sequential #Sequential Models
from keras.layers import Dense #Dense Fully Connected Layer Type
from keras.optimizers import SGD #Stochastic Gradient Descent Optimizer
from keras.callbacks import EarlyStopping
from keras.callbacks import ModelCheckpoint


# In[2]:


data = pd.read_csv("doom.csv", delimiter=",")
data = data.drop('id', 1)
data = data.drop('cid', 1)
data = data.drop('alerts', 1)
data = data.drop('hsalerts', 1)
data = data.drop('capacity', 1)
data = data.drop('duplicate', 1)
data.shape


# In[3]:


data.selectedoption.value_counts()


# In[4]:


feature = data.drop('selectedoption',axis=1)
label = data[['selectedoption']]


# In[5]:


from sklearn.model_selection import train_test_split
train_feature, test_feature, train_label, test_label= train_test_split(
 feature, label, test_size=0.3, random_state=42)


# In[ ]:





# In[ ]:





# In[6]:


from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

scaler.fit(train_feature)
train_feature_trans = scaler.transform(train_feature)
test_feature_trans = scaler.transform(test_feature)


# In[7]:


from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

import matplotlib.pyplot as plt 
def show_train_history(train_history,train,validation):
    plt.plot(train_history.history[train])
    plt.plot(train_history.history[validation])
    plt.title('Train History')
    plt.ylabel(train)
    plt.xlabel('Epoch')
    plt.legend(['train', 'validation'], loc='best')
    plt.show()


# In[11]:


from keras import optimizers


model = Sequential() 

model.add(Dense(units=32, 
                input_dim=10, 
                kernel_initializer='uniform', 
                activation='relu'))
model.add(Dropout(0.5))

model.add(Dense(units=32,  
                kernel_initializer='uniform', 
                activation='sigmoid'))
model.add(Dropout(0.5))
model.add(Dense(units=16,  
                kernel_initializer='uniform', 
                activation='softmax'))
model.add(Dropout(0.5))

model.add(Dense(units=1, 
                kernel_initializer='uniform',
                activation='sigmoid'))

print(model.summary()) 

adam = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)

model.compile(loss='binary_crossentropy',   
              optimizer='adam', metrics=['accuracy'])
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=200)
mc = ModelCheckpoint('best_model.h5', monitor='val_acc', mode='max', verbose=1, save_best_only=True)
train_history = model.fit(x=train_feature_trans, y=train_label,  
                          validation_split=0.8, epochs=20000, 
                          batch_size=1000, verbose=2,callbacks=[es, mc]) 

show_train_history(train_history,'acc','val_acc')
show_train_history(train_history,'loss','val_loss')


# In[9]:


scores = model.evaluate(test_feature_trans, test_label)
print('\n')
print('accuracy=',scores[1])

My dataset is this:

doom.csv

I made a few changes like, changing the optimizer, changing the learning rate, changing activation functions, changing the units. Nothing seems to work. The accuracy isn't going above 80%. Is there something that i can do to improve the accuracy over 90%? I am a newbie to this. Thank you.

$\endgroup$
6
  • $\begingroup$ Before trying any HPO method, try and understand the problem you are solving and evaluate whether the data can provide an adequate representation for this issue. Also, if possible set an optimal bayes error for this task. Moreover try other methods of machine learning before using neural networks and check if you can reach a better result. After that we can re-evaluate the NN further $\endgroup$
    – Nikos H.
    Sep 5 '19 at 12:33
  • $\begingroup$ hi i have already implemented other machine learning models like random forests, Gradient boosting and Näive Bayes where i got an accuracy of 95%, 93% and 78% respectively. the reason why im so particular about NN is that. im experimenting stuff ie: trying different models. so i just want to know where i am going wrong because what i've seen/heard is that Neural networks should actually work better comparatively. not better than the rest but atleast a very good performance. $\endgroup$ Sep 5 '19 at 12:54
  • $\begingroup$ Could you elaborate on the "heard/seen" part? There are many domains where NN are not performing better-or in order to perform better they require great complexity and computational power. NNs are not panacea for all problems. Since you have metrics that you consider good for your problem I would propose to move on. If you want to even further enhance your results or make more robust predictions try model ensembles. What I going for is that NNs might just not work that well here. Now for the sake of argument you can try a bigger network, training for longer and random search for HPO. $\endgroup$
    – Nikos H.
    Sep 5 '19 at 13:06
  • $\begingroup$ im sorry if i sounded rude. I am a self learning noobie in the world of data science. ive seen many youtube tutorials and thus i cam to a conclusion that nenural networks generally work well when compared to the other approaches. for the matter of fact. NN giving such poor performance just swept me off a little. i know im going wrong somewhere. its just that i learn from these type of silly mistakes. $\endgroup$ Sep 5 '19 at 13:15
  • $\begingroup$ You did not come across as rude :) don't worry. I am just asking so I could check the source myself, since this does not follow state of the art literature. Neural networks can be seen as universal approximators but they are not always the optimal solution to our problems. I am going to write a detailed answer on what you could try later in the day :) $\endgroup$
    – Nikos H.
    Sep 5 '19 at 13:29
1
$\begingroup$

1. Parameters optimization

As you have not mentioned the strategy, which you use for parameters choosing, I'd suggest to attend to it. For example in this article https://towardsdatascience.com/hyperparameter-optimization-with-keras-b82e6364ca53 you can find, how parameters can be chosen.

The other powerful approach is Bayesian Optimization Methods https://towardsdatascience.com/automated-machine-learning-hyperparameter-tuning-in-python-dfda59b72f8a It allows to move in parameters values space, taking into account previous results of optimization.

Note: it's crucial to consider parameters as a whole, but not just separately (if value $a$ of the parameter $A$ is not the best choice, it doesn't mean, that it won't be the best option in combination with value $b$ of the parameter $B$)

2. Deal with outliers

It would be reasonable to check outliers in the data (objects which appear to be inconsistent with the remaining instances). For this aim you can use box plot approach. It might be the reason, why it's impossible to get particular model performance.

3. Working with feature space

It also might be useful to provide feature analysis.

$\endgroup$
1
  • $\begingroup$ sure! thanks :D $\endgroup$ Sep 9 '19 at 7:32
0
$\begingroup$

I would start with dealing with data: it seems from your code that you only have a min-max scaler which seems light.

I will say this: you cannot expect to just throw data at a neural network and expect good performance, you have to do some work on your data (that is the most important part and the hardest I think because there is no universal answer).

You can start by looking at the distribution of your data:

  • try to get everything look like a bell shape (if not, maybe apply cox-box transformations)

  • if your distribution is very irregular you can try to bucketize - then apply a one hot encoding

  • features engineering: maybe your data can be represented another way

  • feature selection: maybe all of your features are not really useful for prediction

I would try a few different approaches and compare performance improvement. This is just a start I would advise to look at Kaggle examples to get ideas of how dealing with data. I don't think there is a short answer to this question: but you are already off to a good start!

Then after (and only after) you have worked with your data, you can work more on your model (hyperparameter optimisation as suggested by Lana)

$\endgroup$
1
  • $\begingroup$ thank you, ill look into my data and work on code a little :D $\endgroup$ Sep 9 '19 at 7:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.