# Why cant I overfit this dataset with my neural network?

I have read that given a model is complex enough and I train for enough epochs, my model should at some point overfit the dataset. However I implemented a simple neural network in keras and my validation set loss seems to never go up:

import tensorflow as tf
from tensorflow import keras
import numpy as np
import random
from sklearn import datasets, preprocessing
import matplotlib.pyplot as plt

# import and scale
X = dataset.data
y = dataset.target
X = preprocessing.scale(X)
y = y.reshape((y.shape[0], 1))

# shuffle
shuffle_indices = list(range(X.shape[0]))
random.shuffle(shuffle_indices)
X = X[shuffle_indices]
y = y[shuffle_indices]

# tain-validation split
X_train, y_train  = X[:int(X.shape[0] * 0.7)], y[:int(X.shape[0] * 0.7)]
X_val, y_val = X[int(X.shape[0] * 0.7):], y[int(X.shape[0] * 0.7):]

# define and fit model
model = keras.Sequential([keras.layers.Dense(X.shape[1], use_bias=True, activation="sigmoid"),
keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
keras.layers.Dense(128, use_bias=True, activation="sigmoid"),
keras.layers.Dense(y.shape[1])
])
model.compile(optimizer=tf.keras.optimizers.SGD(
learning_rate=0.0001
), loss='MeanSquaredError')

model.fit(X_train, y_train, validation_data=(X_val, y_val), epochs=500, verbose=1)

# plot data
plt.plot(range(1, len(model.history.history['loss']) + 1), model.history.history['loss'], label='Train Set Cost')
plt.plot(range(1, len(model.history.history['val_loss']) + 1), model.history.history['val_loss'], label='Validation Set Cost')
plt.xlabel("epoch")
plt.ylabel("loss")
plt.legend()
plt.show()


The model is a simple dense neural network with Mean Squared Error as its loss function and gradient descent as it's optimizer. I tried to make the network deeper, but the validation loss only keeps decreasing until it stops at one point.

It is usually good to start with a small model because you can then evaluate the contribution of adding layers, etc. Also, Boston dataset is a popular dataset so there are several tutorials showing good neural network architectures, like this one. Concerning your model, here are some notes.

• The use of sigmoid activation is likely to worsen results, since sigmoid function compresses values between 0 and 1, while you are trying to predict outputs between 5 and 50.
• Instead of sigmoid, you can use ReLU activation, which has better convergence properties for inner layers
• You can standardize your target data to reduce the variance of your data and control the mean, this usually improve a lot regression models. In this case, a sigmoid activation on your final layer would be a good choice
• Thanks! I already standardized my dataset using preprocessing.scale(X). What do you mean with faster convergence properties? Faster loss reduction per epoch? – Tim von Känel Jul 15 '20 at 20:47
• The two main advantages for ReLU are to avoid vanishin gradient and to arise sparsity in the model weights. ReLU does not particularly "speed up" convergence, rather it makes loss converge to lower values because of this. Indeed, you standardized your training data but, you can also standardize your target data, you can take a look at the tutorial I linked in my answer it demonstrates it. – Adam Oudad Jul 16 '20 at 6:50

So, an interesting question asking why a complex model such as the one you have illustrated above is not overfitting (interesting to hear why you would want to achieve this).

Firstly, to make sure we are on the same page, overfitting is typically seen when the training loss decrease (accuracy increases), as validation loss remains the same or increases. So, it is important that note that overfitting is made visible by comparing the trajectories of both training and validation losses (accuracies, etc.).

In response to your question, overfitting happens when it adjusts the parameters such that it fits to the training examples, so therefore examples which are similar to the training examples would theoretically be correctly classified. Where overfitting is really the problem is for when there are values which go beyond the range given in the training examples.

Therefore, one possible reason as to why there might be no explicit sign of overfitting could be that the validation data is very similar to the training data, such that for each dimension, the values from the validation set could be within the range in the training set? Might be worth checking this out by checking feature distributions in both training and validation sets.

• Thanks! It could indeed be that the boston housing dataset has no real outliers and the test and train sets are really similar. I switched to adam and increased the learning rate. I can notice some really small overfitting now. – Tim von Känel Jul 15 '20 at 21:05