I'm training a machine learning algorithm to classify up/down trends in a time series and I'm using an imbalanced feature set. It seems necessary to balance the data since the algorithm could learn a bias towards a particular trend, but this comes at the cost of a non-representative dataset. Should I balance my data? And if so, is random undersampling the right method?
$\begingroup$
$\endgroup$
5
asked Feb 22, 2018 at 18:10
-
1$\begingroup$ Which types of models do you use? some models are less sensitive to imbalanced datasets $\endgroup$– Omri374Feb 24, 2018 at 20:41
-
1$\begingroup$ @Omri374: I'm testing an LSTM network, SVM, and Random Forest classifier. $\endgroup$– Jonathan ShobrookFeb 25, 2018 at 5:28
-
1$\begingroup$ For SVMs and Random Forests, are you using a sliding window to create samples? If yes, you can then perform sampling on the created windows $\endgroup$– Omri374Feb 25, 2018 at 9:07
-
1$\begingroup$ For LSTMs, you could tweak the loss function. see here: stats.stackexchange.com/questions/197273/… and stackoverflow.com/questions/35155655/… $\endgroup$– Omri374Feb 25, 2018 at 9:13
-
$\begingroup$ You might want to read this paper $\endgroup$– iso_9001_Mar 14, 2019 at 14:09
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If you can change the Loss function of the algorithm, It will be very helpful and as a result you don't need to down sample your data. There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.
Disclaimer:
If you use python, PyCM module can help you to find out these metrics.
Here is a simple code to get the recommended parameters from this module:
>>> from pycm import *
>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})
>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]
After that, each of these parameters you want to use as the loss function can be used as follows:
>>> y_pred = model.predict #the prediction of the implemented model
>>> y_actu = data.target #data labels
>>> cm = ConfusionMatrix(y_actu, y_pred)
>>> loss = cm.Kappa #or any other parameter (Example: cm.SOA1)
answered Mar 14, 2019 at 14:47