0
$\begingroup$

I'm trying to make MLP based classifier based on numerical and categorical data

The train_X (Input) data that I'm working with is look like that

enter image description here

each data type is 20 numerical and 1 categorical data ,categorical here I've converted that using cat.codes

My train_Y (prediction) data looks like this :

winPlacePerc
0   0.4444
1   0.6400
2   0.7755
3   0.1667
4   0.1875
5   0.0370
6   0.0000
7   0.7368
8   0.3704
9   0.2143
10  0.3929

its between 0 to 1

I am using here keras sequential model :

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dense, Dropout, Activation
from keras.callbacks import EarlyStopping


#create model
model = Sequential()

#get number of columns in training data
n_cols = train_X.shape[1]

#add layers to model
model.add(Dense(500, activation='relu', input_shape=(n_cols,)))
model.add(Dropout(0.5))
model.add(Dense(1000, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1000, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='Adam',
          loss='binary_crossentropy',
          metrics=['accuracy'])




#set early stopping monitor so the model stops training when it won't 
improve anymore
early_stopping_monitor = EarlyStopping(patience=3)
#train model
model.fit(train_X, train_y, validation_split=0.4, epochs=30, callbacks=[early_stopping_monitor]

here's the model summary

Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 500)               11000     
_________________________________________________________________
dropout_1 (Dropout)          (None, 500)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 1000)              501000    
_________________________________________________________________
dropout_2 (Dropout)          (None, 1000)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 1000)              1001000   
_________________________________________________________________
dropout_3 (Dropout)          (None, 1000)              0         
_________________________________________________________________
dense_4 (Dense)              (None, 500)               500500    
_________________________________________________________________
dropout_4 (Dropout)          (None, 500)               0         
_________________________________________________________________
dropout_5 (Dropout)          (None, 500)               0         
_________________________________________________________________
dense_5 (Dense)              (None, 250)               125250    
_________________________________________________________________
dropout_6 (Dropout)          (None, 250)               0         
_________________________________________________________________
dense_6 (Dense)              (None, 250)               62750     
_________________________________________________________________
dropout_7 (Dropout)          (None, 250)               0         
_________________________________________________________________
dense_7 (Dense)              (None, 250)               62750     
_________________________________________________________________
dropout_8 (Dropout)          (None, 250)               0         
_________________________________________________________________
dense_8 (Dense)              (None, 1)                 251       
=================================================================
Total params: 2,264,501
Trainable params: 2,264,501
Non-trainable params: 0
_________________________________________________________________

since the output is between 0 to 1 ,I used sigmoid as my last layer

But the problem is the loss is not converging at all,what's the issue here ?

I also happen to have some 400k training data ,could that be the reason ?

$\endgroup$
1
$\begingroup$

The dense layers will expect numerical features so you have to transform the categorical features to make them numerical, for example using one-hot encoding. One-hot encoding transforms a categorical features into a set of booleans. For example if you have two features: temperature and day of week, and you would one-hot encode the day of the week, then you would get 6 features for that: (is_monday, is_tuesday, is_wednesday, ...). If the day of week is a Monday you get a 1 for that feature and 0's for all the others. If it's a Sunday, you get 0's for all, so you don't need 7 features for to encode 7 values.

Now about your problem:

  • Start with a simpler model, for example, one or two layers and make step-wise improvements from there. Two subsequent drop-out layers make no sense - if you wanted that, use one drop-out layer with the parameter that is the product of the existing two layers, but I don't think you want that.
  • Why are you saying it doesn't converge? Plot the loss as a function of the number of iterations (or after each epoch, that is, after each pass through the data). It should be shrinking. Maybe you are just not training long enough.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.