Probabilities in KERAS

Question

I'm struggle with a problem OF class probabilities (binary, 0 and 1). Dont know why but after 100 epoches the probabilities became 0 or 1 (like the class to predict). Maybe something is not working with the code or I missing something?

Train data is 20.000 rows (more or less), test data is 2000 rows. Each row is a match where 0 is lost and 1 is won.

y_train_binary = keras.utils.to_categorical(Y, 2)
y_test_binary = keras.utils.to_categorical(t_Y, 2)
model = Sequential()
model.add(Dense(40, input_dim=45, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(30, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(20, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(10, activation='relu'))
model.add(Dropout(0.05))
model.add(BatchNormalization())
model.add(Dense(2, activation='softmax'))
keras.optimizers.Adam(lr=0.5, beta_1=0.9, beta_2=0.999, epsilon=0.3)
model.compile(loss='categorical_crossentropy', optimizer='Adam', metrics=['accuracy']) 
model.fit(X, y_train_binary, epochs=300, validation_split=0.2, batch_size=10, verbose=0)
prediction_classes = model.predict_proba(t_X)
numpy.savetxt("C:/Users/Megaport/Desktop/foo.csv", prediction_classes, delimiter=",") 

I've tried so many parameters (learning rate, epochs, batch size, epsilon, add layer, less layer, different value of dropout) but the problem is the same: probs are not working.

the probabilities of the class are 1 or 0 and not a value between 1 and 0. This happen after 100 epoches.

data test with prediction after 100 epoches is like this:

**RESULT**   VALUE_A     VALUE_B     VALUE_C    **PRED_0    PRED_1**
   0           4          5          3           1          0
   0           7          4          5           0          1
   1           6          7          6           0          1
   1           2          3          4           0          1

What I'm looking for:

**RESULT**   VALUE_A     VALUE_B     VALUE_C    **PRED_0    PRED_1**
   0           4          5          3           0.65          0.35
   0           7          4          5           0.25          0.75
   1           6          7          6           0.20          0.80
   1           2          3          4           0.30          0.70

Plus, someone could please give me some advice looking at accuracy and loss graph?

• https://i.sstatic.net/vTnz4.png
• https://i.sstatic.net/Zz5cv.png

Answer 1

Try predicting log-probabilities that would give you bigger seperation (i.e. values between -inf and 0) I Would clip these values (so that you dont take log from 0 which would be -inf but something like if 0 than log(0.0001) For example:

y_pred = estimator.predict_proba(X.drop(columns=self.drop_columns))
log_prob = lambda x: np.log(np.clip(x, 0.001, 0.999)).reshape(-1, 1)
y_pred = np.hstack([log_prob(y_pred[:, 0]), log_prob(y_pred[:, 1])])

Answer 2

It does not seem like the Adam optimizer object you created is being used because you write:

    optimizer='Adam'

in the following line. Otherwise it looks fine. Though it is very strange that your loss is not extremely high, which is what you would expect with a log loss and confident wrong predictions. Check that the labels have the right dimensions and try the sparse_categorical_crossentropy loss, where you only need to pass an index.