I've a dataset with following schema:

Customer_ID - Unique ID 
Product - ID of purchased product
Department - ID of the department that sells the product
Product_Type - The purchased product type
Date - The date of purchase
Quantity - The number of units purchased

I need to do a link analysis project to analyze some consumption patterns of the products and answer the following questions:

"If product B is purchased then customer will also take product A"

I will use Scala/Python to make the link analysis over the datasets but the examples I have seen are dataset with direct links to the project of "Flight Data" in the schema is:


My question is: there I need to prepare my dataset to make the Link Analysis (there exists some best practices to do this?) or can I analyze the dataset with that structure?

Many thanks! Sorry my inexperience on this topic!


3 Answers 3


One option is that you could make a bipartite graph from your TLOG and then implement some link analysis. Depending on the requirements that you need (volume of the data) there are different frameworks that you can use. One which is quite popular for not so large data is networkx where (manual) you can find already implemented algorithms for link analysis and link prediction.

Maybe, the community would be able to help you more if you try to be more specific about what kind of link analysis you want and what kind of problem you try to solve (does it has to do probably with Supervised or Unsupervised learning).

  • $\begingroup$ Hi Philip C. thanks for your response. I will need to make this link analysis using Spark (is one of the requirements). I'm trying to predict which products are associated in this dataset like the "Diapers and Beer" research. As I am young in this topic I'm not sure if the dataset have an appropriate schema for this prediction. $\endgroup$ Aug 23, 2016 at 19:49
  • $\begingroup$ There are other Graph implementations for Spark, however, If I understand right, you want to make use of Association algorithms like Apriori. Is this what you need? $\endgroup$
    – 20-roso
    Aug 23, 2016 at 19:56
  • $\begingroup$ exactly :) Apriori is good example of the graph that I want to return :) $\endgroup$ Aug 23, 2016 at 20:24

When referencing code examples built to analyze phenomena that differs from what I am studying, I find it best to learn as much as I can about the procedures themselves.

In this case, I would look at the procedures defined in the example code you have found, run a few tests on a subset of your own data to make sure you know how the procedures are functioning, and then, if you are satisfied with the results, proceed to use the code to in your analysis (making sure to cite code sources where required or applicable).


Algorithms like Apriori or FPGrowth are specially designed to analyze such datasets (at scale) and infer the association rules between items across all baskets. Each of those will require that data is input in a specific format.

For example, for FPGrowth in Spark it's an RDD of [Array[String]], where each Array represents a market basket (a transaction/purchase), Strings being the transaction's items' names/identifiers. So, you have to transform the data into that format to input it to the algorithm.

There's a good example of this at Spark's website:

import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.rdd.RDD

val data = sc.textFile("data/mllib/sample_fpgrowth.txt")

//prepare the data to use with FPGrowth
val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))

//create and run the model
val fpg = new FPGrowth()
val model = fpg.run(transactions)

//output the frequent itemsets (items frequently bought together)
model.freqItemsets.collect().foreach { itemset =>
  println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.