Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I am using the Association Rules algorithm using this:
http://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html
I'm have 83945 transactions in my dataset. And I want to filter some products that only appears X times in my dataset. Basically, I want to set my cut-off. My question is: How can I define my cut-off, this is how to define the minimum number of occurences that my products need to have?
Many thanks!
Like @SeanOwen pointed out its called support.
spark.mllib’s FP-growth implementation takes it as a hyper-parameter under minSupport.
It is the minimum support for an itemset to be identified as frequent, e.g : if an item appears 4 out of 5 transactions, it has a support of 4/5=0.8.
Usage:
import org.apache.spark.mllib.fpm.FPGrowth
val transactions: RDD[Array[String]] = ???
val fpg = new FPGrowth().setMinSupport(0.2)
val model = fpg.run(transactions)
I hope this helps.
