# Unbalanced class: class_weight for ML algorithms in Spark MLLib

In python sklearn, there are multiple algorithms (e.g. regression, random forest ... etc.) that have the class_weight parameter to handle unbalanced data.

However, I do not find such parameter for the MLLib algorithms. Is there a plan of implementing class_weight for some MLLib algorithm? Or is there any approach in MLLib for unbalanced data? Or we actually have to handle all the up/downsampling ourselves in MLLib?

Thanks!

• Yes, the algorithms in Spark's MLLib are prepared to handle complex problems. Additionally, from my understanding there not a way to perform a stratified split either. Thus, any performance metrics you acquire will not be appropriately represented. Jan 6 '17 at 17:43
• Here is an exampled of weighted logistic regression in MLlib from the 2.2 documentation.
– Emre
Oct 3 '17 at 22:14

Algorithms in MLLib are always used as baseline in production scenario , and they indeed can not handle some industrial problems , such as label imbalance . So if you want to use them , you have to balance your instances .

Besides , mechanism of BSP in Spark , you can simply see as data parallel , might be the main reason why Spark does not cover that problem . It might be hard for Spark to dispatch instances to all nodes in cluster , while the partial instances of each node share the same label distribution as the whole .

At last , you only have to weight the loss value for every minor labeled instance during your iteration process if you want to implement it .

You could probably look at pydf.rdd.takeSample() in spark, or df.sample in pandas.