# Tensorflow - simple multi-layer perceptron not stabilizing around mean of normally distributed y-values

I'm building an FX trading model where I'm trying to predict the +/- movement of a currency pair 5 minutes into the future. I've had some promising results adapting the model as a classifier (i.e., buy if the currency pair is expected to increase by more than some threshold, sell if the pair is expected to decrease by some threshold, or do nothing otherwise), but when trying to create even the simplest multi-layer perceptron regression model, the mean of predicted values bounces around the mean of actual y values (as expected, this is very close to zero).

The two images below depict the train predictions (green) versus the actual y_train values (blue) after two different epochs after a number of epochs had already been completed. Rather than move towards the true y mean in some stable way, the prediction mean flies right past it and will seemingly do this over and over with more epochs.

So far, I've tried:

1. Using a smaller learning rate
2. Adding more epochs. The training set has almost 300k items in it, so the classification model is able to learn in only about 5 epochs, but increasing this to 50 or 100 for regression still doesn't seem to help.
3. Changing the loss function from mean squared error to mean absolute error
4. Changing the optimizer from Adam to other options like RMSProp (eventually will predict the same value for all items), Adagrad, Adadelta, Nadam, etc.
5. Using elu or leaky relu instead of relu. Also, if I don't add 1 to the y values before training, the model ends up predicting the same value for every item after only a few epochs, which might be an extension of the same problem.
6. A bias constraint on the output node to keep it close to zero (the true y mean). Ultimately, it doesn't matter and the model still pushes away from zero.
7. Batch normalization

This is happening even with a very simple model such as this:

self.model = models.Sequential()
self.model.compile(optimizer=self.optimizer, loss='mean_squared_error', metrics=["mae"])


Any suggestions as to how I can stabilize the model?

• try predicting log values, than you will have lower margin to fail – Noah Weber Dec 28 '19 at 8:01
• What I’m currently trying to predict is y-values expressed as +/- percentage changes, which I think has the same normalizing effect that predicting logs of the absolute/dollar amount has. Are you suggesting predicting logs of the percentage change? – SuperCodeBrah Dec 28 '19 at 8:41
• Plotting predicted vs observed as a histogram is not useful for understanding the performance of a regression model. It is more useful to plot predicted vs observed as a scatterplot or create a residual plot. – Brian Spiering Dec 28 '19 at 14:27

Try deeper network. Something like this.

self.model = models.Sequential()
self.model.compile(optimizer=self.optimizer, loss='mean_squared_error', metrics=["mae"])

• Doesn't seem to help, although using batch gradient descent with more epochs instead of mini batches does seem to help. The only problem is that the classifier model works best with very small mini batches, which makes me skeptical of how well full batches can work for regression. I will need to experiment more with it. But one question about your answer - why is it that people use the pyramiding structure where the number of nodes decreases as the layers get closer to the output layer (i.e., //2 and //4 per your answer)? – SuperCodeBrah Dec 31 '19 at 2:05
• One reason I can think of is that it makes learning stable. i.e. small change in the input does not affect output much. – Suhas Shastry Jan 2 at 22:16

You may need to spend more time identifying the problem than looking for tricks that have worked in literature. So start with the simplest architecture and see how and on what values your network is converging. It's hard to know all the details but here are a few suggestions that might help.

1. You may try different initializations and also random re-initializations (random restarts).
2. I would also give a try to linear activation first, and later tanh or relu/leaky relu. As the actual (y_train) is distributed over a positive and negative regime. I am not saying that linear will or will not perform better, but it will help you understand some mechanics and identify the problem.
3. If you choose to go deeper, I would suggest add a hidden layer whose dimensions larger than features of input data and shrink back (1) one dimension in preferably multiple (1-2 hidden) layers.

Finally here is a very popular blog to follow best practices. http://karpathy.github.io/2019/04/25/recipe/

Also, the comment form Noah Weber seems handy, log scale will penalize the parameters heavily.