2
$\begingroup$

I'm doing some experimentation and trying to train a forex trading model to classify based on three classes:

  • Buy
  • Sell
  • No action

Input rows are labeled as buy when the price at some later time is X% greater than the current price and labeled as sell when the future price is X% less than the current price. All other rows are labeled as no action, which is the large majority of rows.

The problem I'm running into is that the no action items represent about 84% of the total dataset, however the model is predicting no action almost 99% of the time. I suppose the reason for this is that the model is aiming for maximum accuracy and it doesn't realize that no action items are of no use.

So what I would like, if possible, is for the model to only care about buy and sell items in trying to maximize accuracy to hopefully increase the number of buy and sell predictions (to the extent that this helps the profitability of the model).

One way that I think I could do this is to increase the number of classes and have each class represent a roughly equal percentage of the total items, however, that would be pretty messy, so I'm curious if there's a way that directly affects the way accuracy is calculated.


EDIT: I've added additional details here: Variability in CNN test results

But the focus is still in maximizing the accuracy of the buy/sell categories. Looking through the Tensorflow metrics (https://www.tensorflow.org/api_docs/python/tf/metrics), I think I could use two separate binary classifiers and use the TruePositives metric, however, I'd prefer to keep things simple and get it all in one model.

$\endgroup$
1
$\begingroup$

The best approach for this problem would be to change the value of X. For example, if you are using -%2 and %2 as the classification limit such as sell(<-%2), buy(>%2) and no action otherwise; you can reduce this to %1, which will in turn reduce the number of samples that fall into this class while increasing number of samples in buy and sell classes.

If you want to keep that value, the way to go is undersampling: when training you take equal number of samples from each class, meaning you ignore some portion of the data from no action class. So you training and validation sets will be balanced and you can use accuracy as a proper metric.

If this also is not a good option for you, another way would be to try changing the classification threshold for each output so that their possible outcomes are roughly equal. But this might be a little complicated to implement as you have 3 classes (for 2 classes this can be done in a relatively simple way by changing classification threshold from 0.5).

If none of these suit you, you can find a more general approach to imbalance problems here. In case you wish to use a different metric, the most popular ones are listed and discussed in detail here. You probably don't need to define a custom metric because there is a one for pretty much every choice.

| improve this answer | |
$\endgroup$
  • $\begingroup$ That's what I was saying in the last paragraph - to have each class represent a roughly equal percentage, although I'm not sure that that really helps because then I just have more classes that I don't care about. What do you mean about changing the X value? $\endgroup$ – SuperCodeBrah Nov 17 '19 at 10:06
  • $\begingroup$ Your last paragraph is not the same sa undersampling. Keep your classes the same, but use equal number of samples from each class in training. And if you reduce %X value, that will in turn reduce the number of samples in the "no action" class while increasing the other two. $\endgroup$ – serali Nov 17 '19 at 10:26
  • $\begingroup$ Okay, I see. I figured out the X% a minute ago and that did help. I see that there’s a Tensorflow guide on this (tensorflow.org/tutorials/structured_data/imbalanced_data), but I’ll read through it when I get a chance later and report back. Since the idea makes total sense, I’ll go ahead and give you the green check. $\endgroup$ – SuperCodeBrah Nov 17 '19 at 10:44
  • $\begingroup$ I reopened the question with a bounty. Your answer was helpful, but I'm still looking for something that targets the accuracy of the buy/sell categories. There are a number of metrics available through Tensorflow (tensorflow.org/api_docs/python/tf/metrics), but I don't know that any of them work for what I'm trying to achieve. $\endgroup$ – SuperCodeBrah Nov 20 '19 at 8:03
  • $\begingroup$ @SuperCodeBrah I rephrased my answer but I think it would be better if you bountied the other question. There is more information there compared to this one. Anyway, check the link about metrics in above link. That should help $\endgroup$ – serali Nov 20 '19 at 8:33
1
$\begingroup$

I would suggest you to play with sample weight. My suggestion is try putting more weights on taking actions, this way you can configure penalizing more for not predicting buy or loss. Bear in mind that this might also might cause false positive (you are told to take action when it is not supposed to be taken) so please do some testing of this trade-off.

Maybe another idea is to simply attempt to use binary classification (whether to take action or not) instead and then apply some rule-based action with your domain expertise. People have used algorithmic trading in years and this approach/step will most likely work or even required. Indeed it is not an easy task, but I In real life/deployment of machine learning model you might have to chain programs to automate the process better. You cannot be too greedy and put all the load on one model especially as the task gets more complicated.

Another note, saying your model aiming to maximize accuracy is not entirely correct(since your model is learning based on loss and accuracy is just an evaluation metric, it just happen that minimizing loss will eventually increase accuracy).

| improve this answer | |
$\endgroup$
0
$\begingroup$

Being 84% of the data any model will try to converge to no-action class. I would go with SuperCodeBrah's opinion of using under-sampling for your dataset. I encountered similar situation, but doing over-sampling on lower class's data didn't work, so i went for undersampling & used bidirectional-LSTM. It did converge to an balanced model. Yes CNN works great with numerics but i would suggest you to apply Bi-directional LSTM.

You can apply different metrices with combination of LSTM & Dropout layers to reduce risk of overfitting as it is common with imbalanced data cases.

Hope this little experiment works

| improve this answer | |
$\endgroup$
  • $\begingroup$ I was actually planning to do a joint CNN-LSTM. Just based on a little experimentation, the CNN model was doing better than the bi-directional LSTM but I'm going to come back and experiment with it more later. $\endgroup$ – SuperCodeBrah Nov 21 '19 at 6:55
  • $\begingroup$ By the way, if you have any feedback here, I'd be interested: datascience.stackexchange.com/questions/63442/…. I'm kind of stumped by the fact that I keep getting imbalanced predictions (see the matrixes in the question). $\endgroup$ – SuperCodeBrah Nov 21 '19 at 6:59
  • 1
    $\begingroup$ Sure sir, i will see to that & will later post my bilstm architecture here for you to see. Yes CNN-LSTM is what i am doing as of now $\endgroup$ – Akshay Dodhiwala Nov 21 '19 at 7:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.