Initially, I have a dataset where for each row there is user_id and product_ids he bought.

In that dataset there are 478191 products bought by different users.

In order to discover frequent items that are bought together, I will use association rules, apriori algorithm. As apriori algorithm expects to have the features one-hot encoded. I need the product_ids one-hot encoded.

Scikit-learn one-Hot encoding (sparse matrix=True) resulted in memory error. What other methods can I try? (using Python)

  • $\begingroup$ Please provide more details. What are you doing with the feature vectors? What did you use, to apply the a priori algorithm? $\endgroup$ – D.W. Jun 10 '18 at 23:25
  • $\begingroup$ I didnt apply apriori algorithm yet because it expects features to be one-hot encoded but I failed to do that. $\endgroup$ – SarahData Jun 11 '18 at 9:38

Apparently, you need an implementation of the a priori algorithm (or whatever algorithm you want to use) that supports sparse feature vectors. If one doesn't already exist, you might need to implement it yourself.

Alternatively, you can use the feature hashing trick. However, beware that it can cause loss of accuracy and loss of interpretability; nothing comes for free.

  • $\begingroup$ I need to be able to do the inverse in order to know what's the product real label. and feature hashing doesn't allow that $\endgroup$ – SarahData Jun 11 '18 at 9:58

When the itemset is very large you can take a representative sample:

1- first you take a sample that your memory can handle and then generate some rules from that sample (apply Apriori algorithm to the sample instead of the population)

2- check the rules against the large dataset (calculating the confidence for a known rule is not heavy computationally)

3- repeat steps 1&2 until your mined rules settle (no new rules or additional rules are rarely mined)

For more methods to handle large itemsets please see https://pdfs.semanticscholar.org/3825/172284e1902f107540a52ae47a656c99eaba.pdf

another possible solution is cloud computing, there are many cloud computing service providers such as Amazon Web services AWS


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.