Initially, I have a dataset where for each row there is user_id and product_ids he bought.
In that dataset there are 478191 products bought by different users.
In order to discover frequent items that are bought together, I will use association rules, apriori algorithm. As apriori algorithm expects to have the features one-hot encoded. I need the product_ids one-hot encoded.
Scikit-learn one-Hot encoding (sparse matrix=True) resulted in memory error. What other methods can I try? (using Python)
Apparently, you need an implementation of the a priori algorithm (or whatever algorithm you want to use) that supports sparse feature vectors. If one doesn't already exist, you might need to implement it yourself.
Alternatively, you can use the feature hashing trick. However, beware that it can cause loss of accuracy and loss of interpretability; nothing comes for free.
When the itemset is very large you can take a representative sample:
1- first you take a sample that your memory can handle and then generate some rules from that sample (apply Apriori algorithm to the sample instead of the population)
2- check the rules against the large dataset (calculating the confidence for a known rule is not heavy computationally)
3- repeat steps 1&2 until your mined rules settle (no new rules or additional rules are rarely mined)
For more methods to handle large itemsets please see https://pdfs.semanticscholar.org/3825/172284e1902f107540a52ae47a656c99eaba.pdf
another possible solution is cloud computing, there are many cloud computing service providers such as Amazon Web services AWS
