5
$\begingroup$

I'm training a machine learning algorithm to classify up/down trends in a time series and I'm using an imbalanced feature set. It seems necessary to balance the data since the algorithm could learn a bias towards a particular trend, but this comes at the cost of a non-representative dataset. Should I balance my data? And if so, is random undersampling the right method?

$\endgroup$
5
  • 1
    $\begingroup$ Which types of models do you use? some models are less sensitive to imbalanced datasets $\endgroup$
    – Omri374
    Feb 24, 2018 at 20:41
  • 1
    $\begingroup$ @Omri374: I'm testing an LSTM network, SVM, and Random Forest classifier. $\endgroup$ Feb 25, 2018 at 5:28
  • 1
    $\begingroup$ For SVMs and Random Forests, are you using a sliding window to create samples? If yes, you can then perform sampling on the created windows $\endgroup$
    – Omri374
    Feb 25, 2018 at 9:07
  • 1
    $\begingroup$ For LSTMs, you could tweak the loss function. see here: stats.stackexchange.com/questions/197273/… and stackoverflow.com/questions/35155655/… $\endgroup$
    – Omri374
    Feb 25, 2018 at 9:13
  • $\begingroup$ You might want to read this paper $\endgroup$
    – iso_9001_
    Mar 14, 2019 at 14:09

1 Answer 1

3
$\begingroup$

If you can change the Loss function of the algorithm, It will be very helpful and as a result you don't need to down sample your data. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.