Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
In python sklearn, there are multiple algorithms (e.g. regression, random forest ... etc.) that have the class_weight parameter to handle unbalanced data.
However, I do not find such parameter for the MLLib algorithms. Is there a plan of implementing class_weight for some MLLib algorithm? Or is there any approach in MLLib for unbalanced data? Or we actually have to handle all the up/downsampling ourselves in MLLib?
Thanks!
Algorithms in MLLib are always used as baseline in production scenario , and they indeed can not handle some industrial problems , such as label imbalance . So if you want to use them , you have to balance your instances .
Besides , mechanism of BSP in Spark , you can simply see as data parallel , might be the main reason why Spark does not cover that problem . It might be hard for Spark to dispatch instances to all nodes in cluster , while the partial instances of each node share the same label distribution as the whole .
At last , you only have to weight the loss value for every minor labeled instance during your iteration process if you want to implement it .
Hopes this will help you , good luck -)
One of the ways I've handled imbalanced classes in the past has been to build a classifier based on a dataset samples to have a 50/50 sample split. This means using all of the data points associated with your minority class, and randomly sampling the same number of data points from your majority class.
Whether this will work depends on how much data you actually have in your minority class -- if you have extreme class imbalance (<5% minority class instances), then you may want to consider synthetic oversampling.
You could probably look at pydf.rdd.takeSample() in spark, or df.sample in pandas.
The way I handled class imbalances is by following methods: 1. Merging the class that appear least frequently to other classes. Obviously you should use some kind of domain knowledge instead of merging them randomly 2. Use resampling techniques like oversampling, undersampling, SMOTE, ADASYN. I don't recommend using these techniques because they don't actually represent the actual data. But in any case you can certainly take a look at them
In addition to the last post, you could take a look at the imblearn library (https://imbalanced-learn.readthedocs.io/en/stable/index.html) which allows you to do various types of over/ undersampling as well as SMOTE / SMOTENEC
