1
$\begingroup$

I have a data of a bag of words in a document. The data has 3 coulumns : document number, word number, count of the word in the number. I am supposed to generate frequent itemsets of particular size.

I thought I will make list of all words that appear in a document. Create a table of this list. Then generate frequent itemsets using Mlxtend or Orange . But it doesn’t seem to be efficient.

$\endgroup$
0
$\begingroup$

If the size is reasonable (i.e. not too many documents and not too many words in a document), you could try to build a map for each possible itemset, for instance like this:

// Assuming data is an array of size N containing all the documents
// clusters is a map associating each itemset with a set of documents
for i=0 to N-1
  for j=i+1 to N-1
    group = overlap(data[i], data[j])
    add data[i] to the set clusters[group]
    add data[j] to the set clusters[group]

An alternative version if the number of different values and size of the sets allow it and/or if it's possible to precompute the itemsets of interest:

for i=0 to N-1
  for every subset S of data[i]
    add data[i] to the set clusters[S] 

(adapted from https://datascience.stackexchange.com/a/60609/64377)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.