0
$\begingroup$

Is the FPG-Growth an Machine Learning algorithm? Because I'm looking at this code:

import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.rdd.RDD

val data = sc.textFile("data/mllib/sample_fpgrowth.txt")

val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))

val fpg = new FPGrowth()
  .setMinSupport(0.2)
  .setNumPartitions(10)
val model = fpg.run(transactions)

model.freqItemsets.collect().foreach { itemset =>
  println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}

val minConfidence = 0.8
model.generateAssociationRules(minConfidence).collect().foreach { rule =>
  println(
    rule.antecedent.mkString("[", ",", "]")
      + " => " + rule.consequent .mkString("[", ",", "]")
      + ", " + rule.confidence)
}

And I'm not seeing how the algoritm can learn because it don't have an train, test or validation set...

Thanks

$\endgroup$
0
$\begingroup$

FP-growth is a frequent pattern association rule learning algorithm. Thus it's a rule based machine learning algorithm.

When you call the following :

val model = fpg.run(transactions)

You are actually creating and Frequent Pattern model without generating candidates. So actually you'll need to generate afterwards if you need to use them with :

model.generateAssociationRules(minConfidence)

Now, concerning the usual flow of building and validating such models, your code doesn't deal with that. It's usually done through quality measures variations.

It can also be done through Feature extraction techniques.

With such techniques, you should consider the following. For each association rule, you'll need to measure the improvements in accuracy that a commonly used predictor can obtain from an additional feature, constructed according to the exceptions to the rule. In other terms, you'll have to select a reference set of rules that should help your model perform better. I strongly advice you to read this paper about the topic.

So now what does that mean ? This means that you'll need to implement that pipeline yourself because it's not implemented in Spark yet.

I hope that this answers your question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.