0
$\begingroup$

I am trying to make a neural network on a dataset with 257 features and 1 target variable. My code looks like the following:

    df = pd.read_csv('Training Data.csv', low_memory=False, index_col=0)
    df = df.dropna()
    dataset = df.values
    X = dataset[:, 1:]
    y = dataset[:, 0:1]
    scalerx = MinMaxScaler().fit(X)
    scalery = StandardScaler().fit(y)
    X = scalerx.transform(X)
    y = scalery.transform(y)
    k = 257
    X = SelectKBest(f_regression, k=k).fit_transform(X, y)
    # imputer = IterativeImputer(verbose=2)
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
    # X_train = imputer.fit_transform(X_train)
    # X_test = imputer.transform(X_test)
    model = Sequential()
    model.add(Dense(k, input_dim=k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(k, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal', activation='relu'))
    sgd = tf.keras.optimizers.SGD(learning_rate=0.0001)
    model.compile(loss='hinge', optimizer=sgd)
    model.fit(X_train, y_train, epochs=100, verbose=1, batch_size=16)
    # model = GradientBoostingRegressor(random_state=0)
    # model.fit(X_train, y_train)
    res = model.predict(X_test)
    res = scalery.inverse_transform(res)
    MAE = []
    MAPE = []
    print('Neural Net Results...')
    score = mean_absolute_error(y_test, res)
    MAE.append(score)
    print('MAE: ' + str(score))
    score = r2_score(y_test, res)
    print('r2: ' + str(score))
    prediction = pd.DataFrame(res, columns=['predictions'])
    prediction['actuals'] = scalery.inverse_transform(y_test)
    prediction['percent'] = abs(prediction['predictions'] / prediction['actuals'] - 1)
    print('MAPE: ' + str(np.mean(prediction['percent'])))
    MAPE.append(str(np.mean(prediction['percent'])))
    print(MAE)
    print(MAPE)
    prediction.to_csv('Standardized Prediction.csv')

I've tried different loss function and different learning rates and I continue to get an output that looks like this:

Epoch 1/100
348/348 [==============================] - 1s 677us/step - loss: 0.7974
Epoch 2/100
348/348 [==============================] - 0s 744us/step - loss: 0.7974
Epoch 3/100
348/348 [==============================] - 0s 712us/step - loss: 0.7974
Epoch 4/100
348/348 [==============================] - 0s 823us/step - loss: 0.7974
Epoch 5/100
348/348 [==============================] - 0s 815us/step - loss: 0.7974
Epoch 6/100
348/348 [==============================] - 0s 712us/step - loss: 0.7974
...
Neural Net Results...
MAE: 21.426221199575174
r2: -446.9135260782362
MAPE: 116766355636412.97
[21.426221199575174]
['116766355636412.97']

My output csv looks like this:

Output CSV

...

What might be the issue on why it isn't learning?

$\endgroup$
3
  • $\begingroup$ What hyperparameters have you tried changing? For example, have you tried increasing your learning rate or change the optimizer? $\endgroup$
    – Oxbowerce
    Aug 26, 2021 at 16:08
  • $\begingroup$ Yeah I've changed the optimizer to hinge, mean squared error, and mean absolute error. I've tried using learning rates of 0.01, 0.001, 0.0001 as well as batch sizes of 16, 32, and 64. $\endgroup$
    – bballboy8
    Aug 26, 2021 at 16:13
  • $\begingroup$ are your data imbalanced? $\endgroup$
    – Nikos M.
    Aug 26, 2021 at 18:19

1 Answer 1

1
$\begingroup$

I think you have an issue with the number of elements in your dataset for analyzing a neural network with 257 features.

Consider reducing the number of features. Are all of them mandatory? What is the correlation between them? What is the mutual information between all these variables?

Consider adding more data to you dataset. Is that possible? Could you add some synthetic data?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.