Keras: matching logistic regression performance with sequential neural network?

Assume a binary classification problem and a relatively small dataset ($\sim \mathbb{R}^{5000 \times 39}$). By using common ML techniques, starting with logistic regression, I'm able to reach ~0.76-0.79 AUC on the validation sample, depending on the model. So there clearly are some significant signals to be extracted from the model.

However, when trying even a simplest sequential neural network, it does not learn at all.

I.e., using keras from R: (NB: AFAIK the language shouldn't be a problem, since it's only a wrapper for R)

model <- keras_model_sequential() %>%
layer_dense(units = 48,
activation = "sigmoid",
input_shape = 39) %>%
layer_dense(units = 48, activation = "sigmoid") %>%
layer_dense(units = 2, activation = "sigmoid")

model %>% compile(
optimizer = "rmsprop",
loss = "categorical_crossentropy",
metrics = c("accuracy")
)

history <- model %>% keras::fit(
x_train,
y_train,
epochs = 500,
batch_size = 16,
validation_split = 0.7, shuffle = T
)


However, whatever number of epochs or batch sizes, the model doesn't learn at all, the validation accuracy corresponds to the distribution of the target variable (predicts all as 1).

What am I doing wrong? Shouldn't a small one/two layer NN with sigmoid activations at least reach the performance of logistic regression? Any suggestions of how to at least replicate the logistic regression results? Or maybe I've missed some steps in the code?

Would be greateful for any insight!

Debugging neural networks is quite an empirical task. Have you tried one of the following techniques:

• Reducing the number of layers, neurons of your neural networks. The logistic regression model has 39 features to optimize while given the shape of your artificial networks has 40*48 + 49*48 + 49 * 2 = 4370 parameters.
• Adding some regularization such as dropout.

One more detail, sigmoid activation functions tend to be less and less used these days. More dynamic activation function such as RELU often give better results.

Just one last question to be certain. I suppose your data is not equally distributed between the positive and negative class. Am I right ?

• Yes, the distribution is not equally distributed, but it is not too extreme either, roughly 30/70. About the number of parameters: how do we decide on the number of parameters for each layer? Are there any proper heuristics, or is it only checked empirically, by trial and error approach?
– runr
Jun 26 '18 at 10:54
• Lastly, I've commented this on @n1k31t4 answer, but again -- I'm finding that changing sigmoid's to RELU results in an even worse learning. (By fiddling with the data, I've found that the sigmoids now started to learn, but still not on-par with benchmark models). If this is not expected in practice, does this suggest anything on possible routes to take for constructing additional layers, if any?
– runr
Jun 26 '18 at 11:00
• Well there is no exact heuristics to determine the number of parameters. What's generally recommended is go from a very simple model (ie: 1 layer with few neurons) and see from there. If that's not enough then you add complexity. That makes me think you also got to invest your training curves in order to get insights on what hte problem actually is ? Do you overfit ? Does the training loss goes down correctly ? That's questions you got to have answers to. Jun 26 '18 at 11:28
• About the RELU sigmoid debate. Sigmoid is restricted between 0 and 1 while RELU is not. In case your input is not normalized, that may be a cause of poor performances Jun 26 '18 at 11:30

Some question you might want to think about:

Is your dataset big enough? What kind of data is it? Time series? Should you be shuffling? What learning rate are you using? Can you change it and see the effect on the learning curves: plot(history) output?

With regards to your model, you usually leave the last connected layers joined only by a linear activation (i.e. don't use an activation function, just an identity matrix). You have used the sigmoid all the way through, which is fine, but not for final layer! I have corrected this by not including such a non-linearity, rather the softmax activation, which will squash all values into the range of [0, 1], so they can be interpreted as probabilities.

I have increased the number of layers and neurons in the initial layers, and swapped in the preferred non-linearity: the Rectified Linear Unit (ReLU). I would recommend having a quick read of this intro from Stanford's CS231n course, which covers some of the best practices. Also, have a look here for a sample of a larger network performing classification, to see how R Keras is best used.

Give the following code a test, and also plot the history to get more intuition as to how the training is progressing: how many epochs might be necessary, whether you are over- or underfitting, etc.

model <- keras_model_sequential() %>%
layer_dense(units = 200, activation = "relu", input_shape = 39) %>%
layer_dense(units = 100, activation = "relu") %>%
layer_dense(units = 100, activation = "relu") %>%
layer_dense(units = 50, activation = "relu") %>%
layer_dense(units = 2, activation = "softmax")

model %>% compile(
optimizer = "rmsprop",
loss = "categorical_crossentropy",
metrics = c("accuracy")
)

history <- model %>% keras::fit(
x_train,
y_train,
epochs = 500,
batch_size = 16,
validation_split = 0.7, shuffle = T
)

plot(history)                           # Are we overfitting?


Disclaimer: If you don't have much data, this model might be even worse that your original one - or you will massively overfit and get 100% training accuracy and terrible validation/test accuracy.

It could well be the case that your data is just better modelled with a simpler model, or that you do not have enough data to get a neural network to hone in on a nice optimum in its optimisation.

• Thank you for your answer and links, will definitely have a read. What ive noticed so far is that scaling the variables had a clear impact, which is a common practice, though not an obvious one, especially since many strong variables were dummy in my case. Secondly, relu happens to be the worst learner. Does the fact that sigmoid produce somewhat better results than relu, even though you and @Alexis expect otherwise, suggest some insight on the data and possible structures one can further impose when constructing the layers?
– runr
Jun 26 '18 at 10:35
• While it is possible that the data does actually have a very simple structure, linear boosters allowed for some further accuracy gain, so Im suspecting its not that basic.
– runr
Jun 26 '18 at 10:39