How to Improve Low Accuracy Keras Model Design?

Question

I am trying to train a system that looks at some data points and predicts the quantity of surfers on a surf break. I have labeled the pattern for the past 2 months and I have 1500+ training examples with observations every 15 minutes during the day (excluding night time)

My data is shown below (*kooks = surfers):

I am using Keras and here's the code:

*I am removing the month feature from the input matrix before processing it. I also did the preprocessing.MinMaxScaler() drill.

model = Sequential()
model.add(Dense(64, input_dim=6, activation='relu'))
model.add(Dense(64, activation='relu'))
model.add(Dropout(0.25))
model.add(Dense(64, activation='relu'))
model.add(Dense(1, activation='relu'))

model.compile(optimizer='adam',loss='mse', metrics=['accuracy'])

early_stopping_monitor = EarlyStopping(patience=10)
history = model.fit(X, y, validation_split=0.33, epochs=200, batch_size=15, verbose=0, callbacks=[early_stopping_monitor])

The results I am getting are extremely poor:

Test score: 0.015
Test accuracy: 0.12

I have tried multiple optimizers and multiple activation functions, but haven't landed at a satisfactory model yet.

I have a couple of suspicions:

1. The data is not really predictable, as the system is getting confused as some times many of the features are the same (see lines 0 and 1), but the expected output is completely different.
2. The design of the model is not well suited (I don't really know how to design the hidden layer dimensions)
3. The loss function, optimizer and/or activation functions on each layer (including the output layer) are not well suited.

Am I doing something really wrong or this is just the nature of the beast? Any thoughts / advices?

Answer 1

Looking at the training epochs, it seems to me you set a patience parameter that is too short. Please consider removing early stopping at all, for a model trained on 1500 observations only. Early stopping comes useful for particularly heavy models, but in this you shouldn't need it.

I think the size of each mini-batch is very small. That would make gradient descend very noisy, please consider increasing its size, or using full batch training as well.

Additionally, I think you have implemented a Network that is too big. Your input is very small, therefore you don't need to expand its signal on layers of size 64. There are too many nodes that are trying to "learn" not many things, IMHO. A good architecture could be:

model = Sequential()
model.add(Dense(6, input_dim=6, activation='relu'))
model.add(Dense(6, activation='relu'))
model.add(Dense(1, activation=None))

That would make your model faster to train, and ensure that each node is learning relevant features of your data.

I would also change the output layer. Since you want to predict an outcome, you need an output node with no activation (i.e. linear activation). That is mandatory for regression tasks with unbounded output.

Additional things you can try are:

• change dropout levels (but for such a small network it might not be needed at all),
• try regularization techniques, such as Batchnorm, L1 - L2 regularization, different weights initialization... you name it,
• try alternative activation functions.

Also, for medium-to-small size datasets, it is possible for other ML algorithms such as Random Forests or SVMs to beat Neural Networks in performance.