# Neural Network regression negative performance

I have a problem with the performance of a multi layer perceptron regressor (neural network) and I cannot figure out why.

Task: I am trying to improve a time series prediction. I have predictions of a physical parameter of the last 4 years along with the quasi true values. I train the NN with the predictions for -7 days until +1 days around the day I am interested in as features, in order to obtain a better prediction for that day.

Problem: The output of the NN is worse than the feature for the day I am interested in, both for the training and the testing data. Both in terms of RMSE and MAE. I expected the output to be at least on the same level as the feature I input to the NN.

Method: Python with sklearn. I use a grid search with cross validation to get good hyper parameter. I test for different hidden layer configurations, activation functions, learning rate and regularization penalty strengths. I split the data into 66% for training and the remaining data for testing.

I am really grateful for tips how to figure out what my problem here is.

EDIT: I am using sklearn.neural_network.MLPRegressor which provides ‘identity’, ‘logistic’, ‘tanh’ and ‘relu’ as activation functions and I have teseted all of them in the grid search.

I did NOT scale the feature matrix because all features are in the same unit as the desired output and range from about -1 to +1.

EDIT2:

tuned_parameters = [{'hidden_layer_sizes': [int(2/3*number_features),
(int(2/3*number_features), int(4/9*number_features)),
(int(2/3*number_features), int(4/9*number_features), int(8/27*number_features))],
'alpha': 10.0 ** -np.arange(1, 4),
'activation': ["identity", "relu", "logistic", "tanh"],
'solver': ['lbfgs'],
'early_stopping': [True],
'max_iter': [600]}]

regr = GridSearchCV(MLPRegressor(), tuned_parameters, n_jobs=3, verbose=2)
regr.fit(feature_training_matrix, combined_training_target_vector)


Data: The prediction data I use has the following structure: for every day of the last ~4 years there were predictions made for the next 90 days. I have a text file with -90d to +90d data for every day. I try to train the NN to estimate a better prediction for the next 10 days. For this I take -7 up to +1 days around the current prediction day (1-10 days after the currently used starts to predict) as features. This means that the predcition of the day I am interested in is included as a feature.

feature example: [0.16272058, 0.13296574, 0.14213905, 0.25064893, 0.23302285,
0.21019931, 0.20733988, 0.1466959 , 0.17029025, 0.15876942]

corresponding target: 0.174652

• Thank you for your help! I try 4 different activation functions in the grid search including identity (see edit). The code is quite long and complicated. Are you interested in a specific section? I will gladly post it. – Mark Sep 18 '20 at 12:13
• Yout network definition/GridSearch would be great, as well as one sample of your dataset (like 1 row of your X_train) I think that would help me and others better contribute to your issue – mprouveur Sep 18 '20 at 12:28
• Note about the scaling of the feature matrix : my second suggestion is not about scaling features but more about predicting the difference rather than a raw value ( if the NN is random that should at least give your baseline's results). that would be asuivalent I think to a Y = Y - X[:, -1] and X = X - X[:, -1] – mprouveur Sep 18 '20 at 12:30
• I have added the code and an example for a feature set wih the corresponding target value. I try to predict residuals which were generated by removing a priori models. Do you still recommend to work with the difference? – Mark Sep 18 '20 at 13:06