I'm trying to make an NN with Keras to predict the ATP players that will get more than US$1 million in prize money based on their weight and height (from a dataset I mined some weeks ago), but I have been getting a weird behavior especially for the validation accuracy. Sometimes it gets to 84-85%, which is reasonable since SVMs and GaussianNB seem to be able to hit only 83.3% at best (check this post for more info), but sometimes it gets only around 15% accuracy. And the accuracy mostly doesn't change at all if I add capacity or more epochs. I guess one of the weirdest trends in this problem is that the loss is comparably quite small when I get low accuracies:
All of the code is available in this repo (specially in
ATP NN.ipynb and the
atp_python_2018-08-27_1-1500.json), but the main part of it is:
model = Sequential() model.add(Dense(20, activation = 'relu', input_shape = (2,))) model.add(Dense(20, activation = 'relu')) model.add(Dense(2)) model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy']) model_training = model.fit(x = hw_a, y = prize_a_bin, epochs = 4, validation_split = 0.1)
At first, I thought this was happening because the validation split done by Keras would be getting somewhat improper sets that were not very consistent with the rest of the data, since the millionaires are only 10% of the whole dataset, but that doesn't seem to be the case, the weirdness persists. The only pattern I can kind of recognize is that the normal performance and the strange one are complementary, i.e., if one is 85%, the other will be 100% - 85% = 15%. Is that linked to some mistake I've made?
On a sidenote, since I'm a beginner with Keras, should I have specified the last layer's activation function? I assume he is doing either a softmax or a sigmoid by default, but it seems make no difference if I specify either of them. I wonder why with only class (either millionaire or not), you have reshape
y into a
to_categorical and have two units - instead of two - in the output layer.