0
$\begingroup$

I am using the Association Rules algorithm using this:

http://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html

I'm have 83945 transactions in my dataset. And I want to filter some products that only appears X times in my dataset. Basically, I want to set my cut-off. My question is: How can I define my cut-off, this is how to define the minimum number of occurences that my products need to have?

Many thanks!

$\endgroup$
  • 1
    $\begingroup$ It's called 'support' and it's right there in the docs. $\endgroup$ – Sean Owen Sep 30 '16 at 5:23
0
$\begingroup$

Like @SeanOwen pointed out its called support.

spark.mllib’s FP-growth implementation takes it as a hyper-parameter under minSupport.

It is the minimum support for an itemset to be identified as frequent, e.g : if an item appears 4 out of 5 transactions, it has a support of 4/5=0.8.

Usage:

import org.apache.spark.mllib.fpm.FPGrowth

val transactions: RDD[Array[String]] = ???

val fpg = new FPGrowth().setMinSupport(0.2)

val model = fpg.run(transactions)

I hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.