Strange Behavior for trying to Predict Tennis Millionaires with Keras (Validation Accuracy)

Question

I'm trying to make an NN with Keras to predict the ATP players that will get more than US$1 million in prize money based on their weight and height (from a dataset I mined some weeks ago), but I have been getting a weird behavior especially for the validation accuracy. Sometimes it gets to 84-85%, which is reasonable since SVMs and GaussianNB seem to be able to hit only 83.3% at best (check this post for more info), but sometimes it gets only around 15% accuracy. And the accuracy mostly doesn't change at all if I add capacity or more epochs. I guess one of the weirdest trends in this problem is that the loss is comparably quite small when I get low accuracies:

All of the code is available in this repo (specially in ATP NN.ipynb and the atp_python_2018-08-27_1-1500.json), but the main part of it is:

model = Sequential()
model.add(Dense(20, activation = 'relu', input_shape = (2,)))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(2))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
model_training = model.fit(x = hw_a, y = prize_a_bin, epochs = 4, validation_split = 0.1)

At first, I thought this was happening because the validation split done by Keras would be getting somewhat improper sets that were not very consistent with the rest of the data, since the millionaires are only 10% of the whole dataset, but that doesn't seem to be the case, the weirdness persists. The only pattern I can kind of recognize is that the normal performance and the strange one are complementary, i.e., if one is 85%, the other will be 100% - 85% = 15%. Is that linked to some mistake I've made?

On a sidenote, since I'm a beginner with Keras, should I have specified the last layer's activation function? I assume he is doing either a softmax or a sigmoid by default, but it seems make no difference if I specify either of them. I wonder why with only class (either millionaire or not), you have reshape y into a to_categorical and have two units - instead of two - in the output layer.

Answer 1

I can't find your script hidden in your repo which is your dataset which is full of stuff...you should organize your work better if you plan to ask on SO.

This would be the reply from a veteran but I am not.

You should use binary_crossentropy as loss, and then I think you would be able to not use to_categorical.

Then the last layer should have activation sigmoid.

More suggestion

Try to use only a neuron in the last layer: Dense(1, activation='sigmoid')

The accuracy problem may be on how you split the data: maybe only samples from one class go into the validation or in training data. I suggest you use train_test_split from sklearn with fixed random_state or something similar.

Answer 2

Try these things and see if it works:

1.) Explicitly mention the activation function in the last layer as sigmoid.

2.) The loss should be binary_crossentropy.

3.) Try using a different metric instead of accuracy.

4.) In the input and hidden layer, try using leaky_relu

5.) Also mention the batch_size when fitting the model.

Lemme know if any of these work.

Cheers!