8
$\begingroup$

In python sklearn, there are multiple algorithms (e.g. regression, random forest ... etc.) that have the class_weight parameter to handle unbalanced data.

However, I do not find such parameter for the MLLib algorithms. Is there a plan of implementing class_weight for some MLLib algorithm? Or is there any approach in MLLib for unbalanced data? Or we actually have to handle all the up/downsampling ourselves in MLLib?

Thanks!

$\endgroup$
2
  • $\begingroup$ Yes, the algorithms in Spark's MLLib are prepared to handle complex problems. Additionally, from my understanding there not a way to perform a stratified split either. Thus, any performance metrics you acquire will not be appropriately represented. $\endgroup$ Jan 6, 2017 at 17:43
  • $\begingroup$ Here is an exampled of weighted logistic regression in MLlib from the 2.2 documentation. $\endgroup$
    – Emre
    Oct 3, 2017 at 22:14

4 Answers 4

1
$\begingroup$

Algorithms in MLLib are always used as baseline in production scenario , and they indeed can not handle some industrial problems , such as label imbalance . So if you want to use them , you have to balance your instances .

Besides , mechanism of BSP in Spark , you can simply see as data parallel , might be the main reason why Spark does not cover that problem . It might be hard for Spark to dispatch instances to all nodes in cluster , while the partial instances of each node share the same label distribution as the whole .

At last , you only have to weight the loss value for every minor labeled instance during your iteration process if you want to implement it .

Hopes this will help you , good luck -)

$\endgroup$
1
$\begingroup$

One of the ways I've handled imbalanced classes in the past has been to build a classifier based on a dataset samples to have a 50/50 sample split. This means using all of the data points associated with your minority class, and randomly sampling the same number of data points from your majority class.

Whether this will work depends on how much data you actually have in your minority class -- if you have extreme class imbalance (<5% minority class instances), then you may want to consider synthetic oversampling.

You could probably look at pydf.rdd.takeSample() in spark, or df.sample in pandas.

$\endgroup$
0
$\begingroup$

The way I handled class imbalances is by following methods: 1. Merging the class that appear least frequently to other classes. Obviously you should use some kind of domain knowledge instead of merging them randomly 2. Use resampling techniques like oversampling, undersampling, SMOTE, ADASYN. I don't recommend using these techniques because they don't actually represent the actual data. But in any case you can certainly take a look at them

$\endgroup$
0
$\begingroup$

In addition to the last post, you could take a look at the imblearn library (https://imbalanced-learn.readthedocs.io/en/stable/index.html) which allows you to do various types of over/ undersampling as well as SMOTE / SMOTENEC

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.