4
$\begingroup$

I'm training a machine learning algorithm to classify up/down trends in a time series and I'm using an imbalanced feature set. It seems necessary to balance the data since the algorithm could learn a bias towards a particular trend, but this comes at the cost of a non-representative dataset. Should I balance my data? And if so, is random undersampling the right method?

$\endgroup$
  • 1
    $\begingroup$ Which types of models do you use? some models are less sensitive to imbalanced datasets $\endgroup$ – Omri374 Feb 24 '18 at 20:41
  • 1
    $\begingroup$ @Omri374: I'm testing an LSTM network, SVM, and Random Forest classifier. $\endgroup$ – Jonathan Shobrook Feb 25 '18 at 5:28
  • 1
    $\begingroup$ For SVMs and Random Forests, are you using a sliding window to create samples? If yes, you can then perform sampling on the created windows $\endgroup$ – Omri374 Feb 25 '18 at 9:07
  • 1
    $\begingroup$ For LSTMs, you could tweak the loss function. see here: stats.stackexchange.com/questions/197273/… and stackoverflow.com/questions/35155655/… $\endgroup$ – Omri374 Feb 25 '18 at 9:13
  • $\begingroup$ You might want to read this paper $\endgroup$ – iso_9001_ Mar 14 '19 at 14:09
2
$\begingroup$

If you can change the Loss function of the algorithm, It will be very helpful and as a result you don't need to down sample your data. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.