Problem
fit_resample(X,y)
is taking too long to complete execution for 2million rows.
Dataset specifications
I have a labeled dataset about network features, where the X
(features) and Y
(labels) are of shape (2M, 24)
and (2M,11)
respectively.
i.e. there are over 2million rows in the dataset. there are 24 features, and 11 different classes/labels.
Both X and Y are numpy
arrays of float
dtype.
Motivation for using SVM SMOTE
Due to class imbalance, I realized SVM SMOTE
is a good technique balance it, thereby, giving better classification.
Testing with smaller sub-datasets
To test the performance of my classifier, I started small. I made small subdatasets out of the big 2million row dataset.
It took the following code:-
%%time
sm = SVMSMOTE(random_state=42)
X_res, Y_res = sm.fit_resample(X, Y)
1st dataset contains only 7.5k rows. It took about 800ms to run the cell. 2nd dataset contains 115k rows. It took 20min to execute the cell.
Solution Attempts
My system crashes after running continuously for more than 48hrs, running out of memory.
I've tried some ideas, such as
1. splitting it to run on multiple CPU cores using %%px
No improvement in quicker execution
2. using NVIDIA GPU's
Same as above. Which is more understandable since the _smote.py
library functions aren't built with parallel programming for CUDA in mind.
I'm pretty frustrated by the lack of results, and a warm PC. What should I do?