Neural net only works for small datasets

Question

I have a neural network that attempts to solve some regression problem.

The network works fine when the dataset has a small number of training examples, lets say 20. It overfits of course, but the training loss decreases to 0 which is what I want right now.

However, when I take my entire dataset, which has 5000 training examples in it, the loss plateaus at around 375. It won't decrease to zero, no matter how many hidden layers and neurons and activations I add. I tried everything.

What does this mean? Why does this happen and how can I fix it? Is it plausible that it's simply not possible to train a model that fits the training data?

Answer 1

It could mean, that your data is too complex to be learned by the model using a certain number of observations (what is number of features do you have?). On the plot training loss is almost zero for small dataset, but you should rely on the validation loss, not on training one. And validation loss is lower for bigger number of samples, then learning process performes predictable normal

Answer 2

IMPORTANT EDIT AFTER INSPECTING YOUR PLOTS

Your plots are not showing the same thing… the first one shows the MAE for 1000 epochs and the second one for 100... looking at them, in the first one the MAE is also large for 100 epochs.

A fair comparison is to show the plot for the whole training set for the 1000 epochs

FIRST ANSWER

Imagine you want to fit the following function, $sin(x)$, taking olny as training data points inside the red box. With a linear model $f(x)= Wx + b$ you will perform well on the training set and really bad on the validation set (if it comprises points belonging not only to the red box).

You may be experiencing something similar, as Lana has suggested. A small set of training samples are well fitted by your MLP model, but you are not able to find a suitable model for the whole dataset

It won't decrease to zero, no matter how many hidden layers and neurons and activations I add. I tried everything.

It's almost imposible you have tried everything, regarding Deep Learning…. Have you tuned the learning rate, the optimizer, used regularization, droput and so on…?

What does this mean? Why does this happen and how can I fix it? Is it plausible that it's simply not possible to train a model that fits the training data?

It's really difficult to say, moreover taking into account the high dimensionality of your data, can you tell us anything else about it?

Answer 3

As argued by Lana before, the network probably memorizes the small training data, and the full dataset either does not really have any further patterns to learn or they are simply too complex to improve once you reach the plateau.

You can still improve on your results on the validation set by using some kind of regularization, if you are not using any (try dropout?).

And if you simply want to improve the result on training set, you might want to reduce the learning rate on plateau. Check the following link on how to do it if you are using TensorFlow. It will probably improve your results but I do not think the loss will go to zero.

https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/ReduceLROnPlateau

Answer 4

I guess you need to work on data pre-processing more. You need to explore your data:

• Check for outliers in every column, you can use z-score for that.
• If you have missing data, then you need to see how are you imputing the missing values.
• If you have some categorical variables, you need to see how you can encode them like use one-hot encoding or label encoding or binary encoding, target mean encoding etc..

Try to do feature selection and ranking and study the correlations between them using heatmaps or pearson coefficient or if you have categorical variables that may have some sort of ordinal ranking among them you can go for spearman coefficient.

Because one thing is clear your dataset is too complex for the neural net to make out any pattern or establish any kind of linear or non linear relationship amongst the features in your manifold of data created in some high dimensional plane. So you need features that are in some way related to each other and try to reduce the number of redundant features.

One thing that you can do is use RFECV with any gradient boosted tree algorithm and get a baseline of how many features you actually need, and after that try scaling your data. If it is not normally distributed try normalization or otherwise you can use Min max scaler or standard scaler and see what works best for you. Scaling your data properly is very important while using neural networks because if not done properly neural nets tend to become biased and go kind of awry.

You also should try better resampling strategies or increasing you k-folds might help.

If you can give some explanation about your dataset that might help others to answer you queries in a better way. Like:

• What are you doing?
• What are you predicting?
• What is your problem statement?