I am working on a very basic binary classification problem. For each set of four float numbers $(x,y,z,w)$, I want to check if they fall or not into one category.

I have written a model in Keras with 3 dense layers (ReLU activation function) and an output layer (with sigmoid activation function). The model doesn't overfit, so I tried increasing the hyper-parameters, but still it doesn't overfit. I thought that reaching overfit was easy, as long as you increase the number of nodes. Isn't true?

Initially, I thought the problem was with the data, so I have decided to generate a mock dataset, but still the model doesn't overfit. In the code below, the function generate_pattern() generate a valid pattern that I want to label with the integer 1. I populate a Pandas dataframe with this function, and I add some noise by inserting random generated patterns.

Why is the model not overfitting? What is the best model's architecture for this kind of problems?

enter image description here

import pandas as pd
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from sklearn.model_selection import train_test_split

def generate_pattern():
    [x,y,z,w] = np.random.rand(4)/50
    return [0.35+x,0.45+y,0.7+z,1.32+w]

mock_data = pd.DataFrame(columns=['x','y','z','w','target'])
while i < 10000:
    if np.random.randint(2) == 0:
        mock_data.loc[i] = generate_pattern() +[1]
        if np.random.randint(2) == 0:
            if np.random.randint(2) == 1:
                mock_data.loc[i] = list(np.random.rand(4)) + [0]

df_input = mock_data[['x','y','z','w']]
df_output = mock_data[['target']]
X = df_input.values
Y = df_output.astype(int).values
X_train, X_test, y_train, y_test = train_test_split(X, Y[:,0], test_size=0.33, random_state=52)
X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.33, random_state=52)

# Create the neural network model
model = Sequential()
model.add(Dense(5, activation='relu', input_dim=4))
model.add(Dense(16, activation='relu'))
model.add(Dense(32, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

# Compile the model
model.compile(loss=['binary_crossentropy'], optimizer='rmsprop', metrics=['accuracy','mean_squared_error','binary_crossentropy'])

# Train the model
history = model.fit(X_train, y_train, epochs=150, batch_size=64, validation_data=(X_valid,y_valid))

1 Answer 1


In the pattern, the w column is 1.32 plus a bit of noise. In the noisy data it is 0.0 to 1.0. In other words you can drive a bus through the gap between your two targets, and that is just using one of the 4 inputs.

I thought that reaching overfit was easy, as long as you increase the number of nodes.

Overfitting due to adding more model capacity is when you have random noise in your data that such as a perfect split can't be done except by paying attention to the noise. See figure 1 at https://en.wikipedia.org/wiki/Overfitting for instance. In your case the red and blue points are far enough apart that you can draw the simple black line through them, and not have to resort to the over-fitted green line.

BTW, if I've not mis-read your code 80% of your data has the pattern, and target is 1, 20% is uniform random values, with target 0? In which case a model could also learn just to guess the majority class.

(50% of the time it is a 1, then the other 50% it only does something one quarter of the time, and three times out of four it creates no data, so the data is in a 4:1 ratio.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.