Cut-Off using Frequent Pattern Mining - Spark Mllib

Question

I am using the Association Rules algorithm using this:

http://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.html

I'm have 83945 transactions in my dataset. And I want to filter some products that only appears X times in my dataset. Basically, I want to set my cut-off. My question is: How can I define my cut-off, this is how to define the minimum number of occurences that my products need to have?

Many thanks!

Answer 1

Like @SeanOwen pointed out its called support.

spark.mllib’s FP-growth implementation takes it as a hyper-parameter under minSupport.

It is the minimum support for an itemset to be identified as frequent, e.g : if an item appears 4 out of 5 transactions, it has a support of 4/5=0.8.

Usage:

import org.apache.spark.mllib.fpm.FPGrowth

val transactions: RDD[Array[String]] = ???

val fpg = new FPGrowth().setMinSupport(0.2)

val model = fpg.run(transactions)

I hope this helps.