Tensorflow classification - maximize the accuracy of certain classes

Question

I'm doing some experimentation and trying to train a forex trading model to classify based on three classes:

• Buy
• Sell
• No action

Input rows are labeled as buy when the price at some later time is X% greater than the current price and labeled as sell when the future price is X% less than the current price. All other rows are labeled as no action, which is the large majority of rows.

The problem I'm running into is that the no action items represent about 84% of the total dataset, however the model is predicting no action almost 99% of the time. I suppose the reason for this is that the model is aiming for maximum accuracy and it doesn't realize that no action items are of no use.

So what I would like, if possible, is for the model to only care about buy and sell items in trying to maximize accuracy to hopefully increase the number of buy and sell predictions (to the extent that this helps the profitability of the model).

One way that I think I could do this is to increase the number of classes and have each class represent a roughly equal percentage of the total items, however, that would be pretty messy, so I'm curious if there's a way that directly affects the way accuracy is calculated.

---

EDIT: I've added additional details here: Variability in CNN test results

But the focus is still in maximizing the accuracy of the buy/sell categories. Looking through the Tensorflow metrics (https://www.tensorflow.org/api_docs/python/tf/metrics), I think I could use two separate binary classifiers and use the TruePositives metric, however, I'd prefer to keep things simple and get it all in one model.

Answer 1

The best approach for this problem would be to change the value of X. For example, if you are using -%2 and %2 as the classification limit such as sell(<-%2), buy(>%2) and no action otherwise; you can reduce this to %1, which will in turn reduce the number of samples that fall into this class while increasing number of samples in buy and sell classes.

If you want to keep that value, the way to go is undersampling: when training you take equal number of samples from each class, meaning you ignore some portion of the data from no action class. So you training and validation sets will be balanced and you can use accuracy as a proper metric.

If this also is not a good option for you, another way would be to try changing the classification threshold for each output so that their possible outcomes are roughly equal. But this might be a little complicated to implement as you have 3 classes (for 2 classes this can be done in a relatively simple way by changing classification threshold from 0.5).

If none of these suit you, you can find a more general approach to imbalance problems here. In case you wish to use a different metric, the most popular ones are listed and discussed in detail here. You probably don't need to define a custom metric because there is a one for pretty much every choice.

Answer 2

I would suggest you to play with sample weight. My suggestion is try putting more weights on taking actions, this way you can configure penalizing more for not predicting buy or loss. Bear in mind that this might also might cause false positive (you are told to take action when it is not supposed to be taken) so please do some testing of this trade-off.

Maybe another idea is to simply attempt to use binary classification (whether to take action or not) instead and then apply some rule-based action with your domain expertise. People have used algorithmic trading in years and this approach/step will most likely work or even required. Indeed it is not an easy task, but I In real life/deployment of machine learning model you might have to chain programs to automate the process better. You cannot be too greedy and put all the load on one model especially as the task gets more complicated.

Another note, saying your model aiming to maximize accuracy is not entirely correct(since your model is learning based on loss and accuracy is just an evaluation metric, it just happen that minimizing loss will eventually increase accuracy).

Answer 3

Being 84% of the data any model will try to converge to no-action class. I would go with SuperCodeBrah's opinion of using under-sampling for your dataset. I encountered similar situation, but doing over-sampling on lower class's data didn't work, so i went for undersampling & used bidirectional-LSTM. It did converge to an balanced model. Yes CNN works great with numerics but i would suggest you to apply Bi-directional LSTM.

You can apply different metrices with combination of LSTM & Dropout layers to reduce risk of overfitting as it is common with imbalanced data cases.

Hope this little experiment works