is taking too long to complete execution for 2million rows.
I have a labeled dataset about network features, where the
Y(labels) are of shape
(2M, 24) and
i.e. there are over 2million rows in the dataset. there are 24 features, and 11 different classes/labels.
Both X and Y are
numpy arrays of
Motivation for using SVM SMOTE
Due to class imbalance, I realized
SVM SMOTE is a good technique balance it, thereby, giving better classification.
Testing with smaller sub-datasets
To test the performance of my classifier, I started small. I made small subdatasets out of the big 2million row dataset.
It took the following code:-
%%time sm = SVMSMOTE(random_state=42) X_res, Y_res = sm.fit_resample(X, Y)
1st dataset contains only 7.5k rows. It took about 800ms to run the cell. 2nd dataset contains 115k rows. It took 20min to execute the cell.
My system crashes after running continuously for more than 48hrs, running out of memory.
I've tried some ideas, such as
1. splitting it to run on multiple CPU cores using %%px
No improvement in quicker execution
2. using NVIDIA GPU's
Same as above. Which is more understandable since the
_smote.py library functions aren't built with parallel programming for CUDA in mind.
I'm pretty frustrated by the lack of results, and a warm PC. What should I do?