Why is my Neural Network having constant loss and always predicting a singular value?

Question

I am trying to make a neural network on a dataset with 257 features and 1 target variable. My code looks like the following:

    df = pd.read_csv('Training Data.csv', low_memory=False, index_col=0)
    df = df.dropna()
    dataset = df.values
    X = dataset[:, 1:]
    y = dataset[:, 0:1]
    scalerx = MinMaxScaler().fit(X)
    scalery = StandardScaler().fit(y)
    X = scalerx.transform(X)
    y = scalery.transform(y)
    k = 257
    X = SelectKBest(f_regression, k=k).fit_transform(X, y)
    # imputer = IterativeImputer(verbose=2)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
    # X_train = imputer.fit_transform(X_train)
    # X_test = imputer.transform(X_test)
    model = Sequential()
    model.add(Dense(k, input_dim=k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='relu'))
    sgd = tf.keras.optimizers.SGD(learning_rate=0.0001)
    model.compile(loss='hinge', optimizer=sgd)
    model.fit(X_train, y_train, epochs=100, verbose=1, batch_size=16)
    # model = GradientBoostingRegressor(random_state=0)
    # model.fit(X_train, y_train)
    res = model.predict(X_test)
    res = scalery.inverse_transform(res)
    MAE = []
    MAPE = []
    print('Neural Net Results...')
    score = mean_absolute_error(y_test, res)
    MAE.append(score)
    print('MAE: ' + str(score))
    score = r2_score(y_test, res)
    print('r2: ' + str(score))
    prediction = pd.DataFrame(res, columns=['predictions'])
    prediction['actuals'] = scalery.inverse_transform(y_test)
    prediction['percent'] = abs(prediction['predictions'] / prediction['actuals'] - 1)
    print('MAPE: ' + str(np.mean(prediction['percent'])))
    MAPE.append(str(np.mean(prediction['percent'])))
    print(MAE)
    print(MAPE)
    prediction.to_csv('Standardized Prediction.csv')

I've tried different loss function and different learning rates and I continue to get an output that looks like this:

Epoch 1/100
348/348 [==============================] - 1s 677us/step - loss: 0.7974
Epoch 2/100
348/348 [==============================] - 0s 744us/step - loss: 0.7974
Epoch 3/100
348/348 [==============================] - 0s 712us/step - loss: 0.7974
Epoch 4/100
348/348 [==============================] - 0s 823us/step - loss: 0.7974
Epoch 5/100
348/348 [==============================] - 0s 815us/step - loss: 0.7974
Epoch 6/100
348/348 [==============================] - 0s 712us/step - loss: 0.7974
...
Neural Net Results...
MAE: 21.426221199575174
r2: -446.9135260782362
MAPE: 116766355636412.97
[21.426221199575174]
['116766355636412.97']

My output csv looks like this:

...

What might be the issue on why it isn't learning?

Answer 1

I think you have an issue with the number of elements in your dataset for analyzing a neural network with 257 features.

Consider reducing the number of features. Are all of them mandatory? What is the correlation between them? What is the mutual information between all these variables?

Consider adding more data to you dataset. Is that possible? Could you add some synthetic data?