1
$\begingroup$

I'm trying to make an NN with Keras to predict the ATP players that will get more than US$1 million in prize money based on their weight and height (from a dataset I mined some weeks ago), but I have been getting a weird behavior especially for the validation accuracy. Sometimes it gets to 84-85%, which is reasonable since SVMs and GaussianNB seem to be able to hit only 83.3% at best (check this post for more info), but sometimes it gets only around 15% accuracy. And the accuracy mostly doesn't change at all if I add capacity or more epochs. I guess one of the weirdest trends in this problem is that the loss is comparably quite small when I get low accuracies:

The test and validation losses are quite small, but the accuracy is terrible.

All of the code is available in this repo (specially in ATP NN.ipynb and the atp_python_2018-08-27_1-1500.json), but the main part of it is:

model = Sequential()
model.add(Dense(20, activation = 'relu', input_shape = (2,)))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(2))
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
model_training = model.fit(x = hw_a, y = prize_a_bin, epochs = 4, validation_split = 0.1)

At first, I thought this was happening because the validation split done by Keras would be getting somewhat improper sets that were not very consistent with the rest of the data, since the millionaires are only 10% of the whole dataset, but that doesn't seem to be the case, the weirdness persists. The only pattern I can kind of recognize is that the normal performance and the strange one are complementary, i.e., if one is 85%, the other will be 100% - 85% = 15%. Is that linked to some mistake I've made?

On a sidenote, since I'm a beginner with Keras, should I have specified the last layer's activation function? I assume he is doing either a softmax or a sigmoid by default, but it seems make no difference if I specify either of them. I wonder why with only class (either millionaire or not), you have reshape y into a to_categorical and have two units - instead of two - in the output layer.

$\endgroup$
  • $\begingroup$ When you get 85% accuracy is your model guessing no one will be a millionaire and vice versatility when you see 15%? In other words, accuracy is a bad metric. $\endgroup$ – kbrose Oct 22 '18 at 2:24
0
$\begingroup$

I can't find your script hidden in your repo which is your dataset which is full of stuff...you should organize your work better if you plan to ask on SO.

This would be the reply from a veteran but I am not.

You should use binary_crossentropy as loss, and then I think you would be able to not use to_categorical.

Then the last layer should have activation sigmoid.

More suggestion

Try to use only a neuron in the last layer: Dense(1, activation='sigmoid')

The accuracy problem may be on how you split the data: maybe only samples from one class go into the validation or in training data. I suggest you use train_test_split from sklearn with fixed random_state or something similar.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Can't you really find the files I mentioned? They are here, aren't they? Though there is a lot of stuff there, the only files you need is the data (atp_python_2018-08-27_1-1500.json) and the Jupyter Notebook (ATP NN.ipynb). $\endgroup$ – Philippe Fanaro Sep 20 '18 at 22:38
  • $\begingroup$ Your suggestion for using binary_crossentropy does work as an alternative way of doing the same thing, but it doesn't seem to solve the problem, I still get the weird complementary accuracies. Thanks for clarifying the difference between binary_crossentropy and categorical_crossentropy though. $\endgroup$ – Philippe Fanaro Sep 20 '18 at 22:41
  • $\begingroup$ I had already tried to do that, as I specified in the original question. I've tried it again with sklearn's more compact train_test_split and it still doesn't work. $\endgroup$ – Philippe Fanaro Sep 21 '18 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.