0
$\begingroup$

I've a dataset with following schema:

Customer_ID - Unique ID 
Product - ID of purchased product
Department - ID of the department that sells the product
Product_Type - The purchased product type
Date - The date of purchase
Quantity - The number of units purchased

I need to do a link analysis project to analyze some consumption patterns of the products and answer the following questions:

"If product B is purchased then customer will also take product A"

I will use Scala/Python to make the link analysis over the datasets but the examples I have seen are dataset with direct links to the project of "Flight Data" in the schema is:

ID
Origin
Destination

My question is: there I need to prepare my dataset to make the Link Analysis (there exists some best practices to do this?) or can I analyze the dataset with that structure?

Many thanks! Sorry my inexperience on this topic!

$\endgroup$
0
$\begingroup$

One option is that you could make a bipartite graph from your TLOG and then implement some link analysis. Depending on the requirements that you need (volume of the data) there are different frameworks that you can use. One which is quite popular for not so large data is networkx where (manual) you can find already implemented algorithms for link analysis and link prediction.

Maybe, the community would be able to help you more if you try to be more specific about what kind of link analysis you want and what kind of problem you try to solve (does it has to do probably with Supervised or Unsupervised learning).

$\endgroup$
  • $\begingroup$ Hi Philip C. thanks for your response. I will need to make this link analysis using Spark (is one of the requirements). I'm trying to predict which products are associated in this dataset like the "Diapers and Beer" research. As I am young in this topic I'm not sure if the dataset have an appropriate schema for this prediction. $\endgroup$ – João_testeSW Aug 23 '16 at 19:49
  • $\begingroup$ There are other Graph implementations for Spark, however, If I understand right, you want to make use of Association algorithms like Apriori. Is this what you need? $\endgroup$ – Grzegorz Aug 23 '16 at 19:56
  • $\begingroup$ exactly :) Apriori is good example of the graph that I want to return :) $\endgroup$ – João_testeSW Aug 23 '16 at 20:24
0
$\begingroup$

When referencing code examples built to analyze phenomena that differs from what I am studying, I find it best to learn as much as I can about the procedures themselves.

In this case, I would look at the procedures defined in the example code you have found, run a few tests on a subset of your own data to make sure you know how the procedures are functioning, and then, if you are satisfied with the results, proceed to use the code to in your analysis (making sure to cite code sources where required or applicable).

$\endgroup$
0
$\begingroup$

Algorithms like Apriori or FPGrowth are specially designed to analyze such datasets (at scale) and infer the association rules between items across all baskets. Each of those will require that data is input in a specific format.

For example, for FPGrowth in Spark it's an RDD of [Array[String]], where each Array represents a market basket (a transaction/purchase), Strings being the transaction's items' names/identifiers. So, you have to transform the data into that format to input it to the algorithm.

There's a good example of this at Spark's website:

import org.apache.spark.mllib.fpm.FPGrowth
import org.apache.spark.rdd.RDD

val data = sc.textFile("data/mllib/sample_fpgrowth.txt")

//prepare the data to use with FPGrowth
val transactions: RDD[Array[String]] = data.map(s => s.trim.split(' '))

//create and run the model
val fpg = new FPGrowth()
  .setMinSupport(0.2)
  .setNumPartitions(10)
val model = fpg.run(transactions)

//output the frequent itemsets (items frequently bought together)
model.freqItemsets.collect().foreach { itemset =>
  println(itemset.items.mkString("[", ",", "]") + ", " + itemset.freq)
}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.