2
$\begingroup$

I am doing my undergrad Dissertation on time series prediction, and use various models (linear /ridge regression, AR(2), Random Forest, SVR, and 4 variations of Neural Networks) to try and 'predict' (for academic only reasons) daily return data, using as input lagged returns and SMA - RSI features (using TA - Lib) built based on those returns. However, I have noticed that my NNs do not learn anything, and upon inspecting the loss graph and the vector of predictions, I noticed it only predicts a single value, with the same applying for the Ridge and AR regressions.

Also, when I try to calculate the correlation between the labels and the predictions (of the NNs) I get 'nan' as a result, no matter what I try, which I suspect has to do with the predictions. I also get wildly varying r2 scores on each re-run (even though I have set multiple seeds, both on Tensorflow backend as well as numpy) and always negative, which I cannot understand as even though my search on the internet and the sklearn's docs say it can be negative, my professor insists it cannot be, and I truly am bewildered.

What can I do about it? Isn't it obviously wrong for an entire NN to predict only a single value? Below I include the code for the ridge / AR regressions as well as the 'Vanilla NN' and a couple of useful graphs. The data itself is quite large, so I don't know if there's much of a point to include it if not asked specifically, given there are no algorithmic errors below.

def vanillaNN(X_train, y_train,X_test,y_test):
    n_cols = X_train.shape[1]
    model = Sequential()
    model.add(Dense(100,activation='relu', input_shape=(n_cols, )))
    model.add(Dropout(0.3))
    model.add(Dense(150, activation='relu'))
    model.add(Dense(50, activation='relu'))
    model.add(Dropout(0.1))
    model.add(Dense(1))

    model.compile(optimizer='adam', loss='mse', metrics=['mse'])

    history = model.fit(X_train,y_train,epochs=100,verbose=0,
        shuffle=False, validation_split=0.1)  

    # Use the last loss as the title
    plt.plot(history.history['loss'])
    plt.title('last loss:' + str(round(history.history['loss'][-1], 6)))
    plt.xlabel('Epoch')
    plt.ylabel('Loss')
    plt.show()

    # Calculate R^2 score and MSE
    # .... Omitted Code ......

    # it returns those for testing purposes in the IPython shell
    return (train_scores, test_scores, y_pred_train, y_pred_test, y_train, y_test)
    VNN_results = vanillaNN(
        train_features,train_targets,
        test_features,test_targets)

Neural Network Loss History - 100 epochs

def AR(X_train, order=2):
    arma_train = np.array(X_train['returns'])
    armodel = ARMA(arma_train, order=(order,0))
    armodel_results = armodel.fit()
    print(armodel_results.summary())
    armodel_results.plot_predict(start=8670, end=8698)
    plt.show()
    ar_pred = armodel_results.predict(start=8699, end=9665)

    # ...r2 and MSE scores omitted code...

    return [mse_ar2, r2_ar2, ar_pred]

AR(2) Model out-of-sample predictions

def Elastic(X_train, y_train, X_test, y_test):
    elastic = ElasticNet()
    param_grid_elastic = {'alpha': [0.001,  0.01, 0.1, 0.5],
        'l1_ratio': [0.001, 0.01, 0.1, 0.5]
                              }
    grid_elastic = GridSearchCV(elastic, param_grid_elastic, 
        cv=tscv.split(X_train),scoring='neg_mean_squared_error')

    grid_elastic.fit(X_train, y_train)

    y_pred_train = grid_elastic.predict(X_train) 
    train_scores = scores(y_train, y_pred_train)

    y_pred_test = grid_elastic.predict(X_test)

    # ...Omitted Code...
    return [train_scores, test_scores, y_pred_train, y_pred_test]

AR(2) sample test and train predictions: Test Set Predictions Train Set Predictions

SAMPLE DATA: Train features and Train Targets (future_returns) 

It looks like a mess, but just copy paste into excel file and it should be good to go!)

Date returns ma14 rsi14 ma30 rsi30 ma50 rsi50 ma200 rsi200 future_returns 10/14/1980 3.49E-05 42.76324407 49.21625218 66.6250545 49.69881565 49.45368438 49.93538688 37.78942977 50.51223405 0.013277481 10/15/1980 0.013277481 0.239711734 53.45799196 0.16387242 51.78260494 0.140801819 51.19194274 0.10545251 50.79944024 -0.011855382 10/16/1980 -0.011855382 -0.306338818 45.66265303 -0.159676722 47.88773425 -0.115851283 48.81901884 -0.107808537 50.24325364 -0.00414208 10/17/1980 -0.00414208 -1.154286743 48.16105108 -0.451328445 49.1031414 -0.3083074 49.55134428 -0.299669528 50.4107189 0.007939494 10/20/1980 0.007939494 0.548806765 51.89223141 0.338253188 50.95654204 0.15403304 50.679274 0.145277255 50.67207082 -0.0050544 10/21/1980 -0.0050544 -0.580692978 47.89906352 -0.443621598 48.97241743 -0.250429072 49.46553614 -0.227539737 50.38503757 -2.93E-05 10/22/1980 -2.93E-05 -85.38681662 49.51695915 -69.94007273 49.7551065 -46.09189634 49.93866124 -37.61970911 50.49403171 -0.018135363 10/23/1980 -0.018135363 -0.020087358 44.19203763 -0.067897836 47.0643223 -0.037135977 48.27685366 -0.05615813 50.09551572 0.000415381 10/24/1980 0.000415381 -2.422576075 50.11149401 3.125882141 49.93407273 1.480210269 50.01580668 2.418672772 50.49781053 -0.013535864 10/27/1980 -0.013535864 0.12834969 46.14718747 -0.056014904 47.91325905 -0.053384853 48.75782925 -0.065375964 50.19198915 0.00337859 10/28/1980 0.00337859 -0.566993349 51.18890074 0.168834456 50.4293269 0.275579337 50.30416736 0.265200747 50.55684672 -0.003396646 10/29/1980 -0.003396646 0.522213275 49.20187924 0.045438303 49.43972414 -0.207144904 49.69125655 -0.266487054 50.40819543 -0.011421006 10/30/1980 -0.011421006 0.185961701 46.88078737 0.029632441 48.27895767 -0.018832701 48.97017418 -0.073584097 50.23238873 0.013350935 10/31/1980 0.013350935 -0.156079209 54.08231943 -0.003988301 51.88647852 0.031510014 51.2008709 0.064478565 50.76515097 0.01758565 11/3/1980 0.01758565 -0.047207787 55.20045808 0.011243586 52.47271385 0.048876357 51.57016088 0.055301429 50.85553721 0.025052611 11/5/1980 0.025052611 0.000435129 57.18044487 0.053812694 53.50605621 0.05421219 52.22072289 0.039977524 51.01490125 -0.017722306 11/6/1980 -0.017722306 0.023031138 44.92992843 -0.03124573 47.39891281 -0.070442975 48.41875471 -0.050855544 50.07992286 0.000581639 11/7/1980 0.000581639 -0.121650134 49.87840955 1.580058713 49.92880444 2.604518867 50.0080164 1.580277943 50.47031631 0.002508371 11/10/1980 0.002508371 -0.182865195 50.38381628 0.640792927 50.18967587 0.608670831 50.17291603 0.34700777 50.51126003 0.012129939 11/11/1980 0.012129939 0.063376989 52.93601236 0.221445976 51.49515928 0.145866445 50.99656886 0.079352102 50.71573089 0.024926655

$\endgroup$
  • $\begingroup$ from your graph it looks like your predictions are not really constant and vary at least a little bit around zero?! or am I reading the blue line incorrectly? $\endgroup$ – oW_ Apr 15 '19 at 23:53
  • $\begingroup$ Could you please provide some sample lines of X_train and y_train? $\endgroup$ – georg-un Apr 16 '19 at 10:12
  • $\begingroup$ oW_ yes, the graph did give me that impression but upon inspecting the test set vector itself I found that it gives a few varying predictions at first but every single one after a point is the same number, which strangely does not apply for in-sample predictions. See sample above $\endgroup$ – Constantine Phoenix Apr 16 '19 at 21:15
0
$\begingroup$

Thanks for providing the sample data. I do not really see any severe problems to pin something down as the definite cause of your problem, but I can give you some advice for improvement that could help.

Standardizing and scaling

Some of your features have larger values and some have smaller values. If you don't standardize and scale your features and targets, it will result in "unbalanced" weights inside your NN which can lead to an unstable model. Therefore, use something like the StandardScaler to standardize and scale your data after splitting it up into a train and a test dataset.

Activation function

It is always worth a try to play around with different activation functions. ReLu is quite simple and computationally inexpensive compared to other activation functions, but in a bad setup, it can lead to many dead neurons. So I would suggest trying out other activation functions like Tanh or Leaky Relu. Note: That does not mean that ReLu is a bad activation function. For many reasons, it is actually a very popular one.

Learning rate

Especially if you stick with ReLu, check what difference it makes if you reduce the learning rate and/or set a learning rate decay.

Neural Network Architecture

Since you are working with a time-series, it would make sense to use a Recurrent Neural Network which was designed for time-series data like GRU or LSTM.

Other

One side note to prevent you from falling into the same trap I did: If you work with TA-Lib, scale your values before you calculate any features. There is an open issue on Github of TA-Lib calculating wrong features if the input-values are too small. I see that your targets also have quite small values, so maybe keep an eye on that.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Hi, thanks a lot for your advice! I do actually use LSTMs later down the road but just wanted to show the problem with the simplest architecture. I actually did start by standardizing my data (via StandardScaler in sklearn) and now I do get different predictions, and the R2 has jumped to 98%, which I think is overfitting, as with standardization even linear regression gets 95%+ R2. I suppose that is a problem? $\endgroup$ – Constantine Phoenix Apr 18 '19 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.