Keras MLP Classifer Model not converging

Question

I'm trying to make MLP based classifier based on numerical and categorical data

The train_X (Input) data that I'm working with is look like that

each data type is 20 numerical and 1 categorical data ,categorical here I've converted that using cat.codes

My train_Y (prediction) data looks like this :

winPlacePerc
0   0.4444
1   0.6400
2   0.7755
3   0.1667
4   0.1875
5   0.0370
6   0.0000
7   0.7368
8   0.3704
9   0.2143
10  0.3929

its between 0 to 1

I am using here keras sequential model :

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dense, Dropout, Activation
from keras.callbacks import EarlyStopping


#create model
model = Sequential()

#get number of columns in training data
n_cols = train_X.shape[1]

#add layers to model
model.add(Dense(500, activation='relu', input_shape=(n_cols,)))
model.add(Dropout(0.5))
model.add(Dense(1000, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1000, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(500, activation='relu'))
model.add(Dropout(0.5))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(250, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='Adam',
          loss='binary_crossentropy',
          metrics=['accuracy'])




#set early stopping monitor so the model stops training when it won't 
improve anymore
early_stopping_monitor = EarlyStopping(patience=3)
#train model
model.fit(train_X, train_y, validation_split=0.4, epochs=30, callbacks=[early_stopping_monitor]

here's the model summary

Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 500)               11000     
_________________________________________________________________
dropout_1 (Dropout)          (None, 500)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 1000)              501000    
_________________________________________________________________
dropout_2 (Dropout)          (None, 1000)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 1000)              1001000   
_________________________________________________________________
dropout_3 (Dropout)          (None, 1000)              0         
_________________________________________________________________
dense_4 (Dense)              (None, 500)               500500    
_________________________________________________________________
dropout_4 (Dropout)          (None, 500)               0         
_________________________________________________________________
dropout_5 (Dropout)          (None, 500)               0         
_________________________________________________________________
dense_5 (Dense)              (None, 250)               125250    
_________________________________________________________________
dropout_6 (Dropout)          (None, 250)               0         
_________________________________________________________________
dense_6 (Dense)              (None, 250)               62750     
_________________________________________________________________
dropout_7 (Dropout)          (None, 250)               0         
_________________________________________________________________
dense_7 (Dense)              (None, 250)               62750     
_________________________________________________________________
dropout_8 (Dropout)          (None, 250)               0         
_________________________________________________________________
dense_8 (Dense)              (None, 1)                 251       
=================================================================
Total params: 2,264,501
Trainable params: 2,264,501
Non-trainable params: 0
_________________________________________________________________

since the output is between 0 to 1 ,I used sigmoid as my last layer

But the problem is the loss is not converging at all,what's the issue here ?

I also happen to have some 400k training data ,could that be the reason ?

Answer 1

The dense layers will expect numerical features so you have to transform the categorical features to make them numerical, for example using one-hot encoding. One-hot encoding transforms a categorical features into a set of booleans. For example if you have two features: temperature and day of week, and you would one-hot encode the day of the week, then you would get 6 features for that: (is_monday, is_tuesday, is_wednesday, ...). If the day of week is a Monday you get a 1 for that feature and 0's for all the others. If it's a Sunday, you get 0's for all, so you don't need 7 features for to encode 7 values.

Now about your problem:

• Start with a simpler model, for example, one or two layers and make step-wise improvements from there. Two subsequent drop-out layers make no sense - if you wanted that, use one drop-out layer with the parameter that is the product of the existing two layers, but I don't think you want that.
• Why are you saying it doesn't converge? Plot the loss as a function of the number of iterations (or after each epoch, that is, after each pass through the data). It should be shrinking. Maybe you are just not training long enough.