To cluster label (in a multilabel classification problem) which mostly appear together in a dataframe? For example, I have this dataframe:

text     |   genre
text 1   | [action,mistery,horror,thriller]
text 2   | [drama,romance]
text 3   | [comedy,drama,romance]
text 4   | [scifi,mystery,horror,thriller]
text 5   | [horror,mystery,thriller]

How can I cluster the tag that often appear together? For example, genre "mystery","horror","thriller" often appear together (3 times), genre "drama","romance" often appears together (2 times).


The difficulty here is the design: if one just represents every "class" as an instance with a vector of its co-occurrences with other "classes", then the clustering will only group the full group of instances/classes by how similar they are with other groups.

Instead I think that an instance should represent a pair of "classes", using the PMI as a measure of how strongly they are associated.

An advanced option would be to represent each "class" as a node in a graph and add an edge between any two classes which occur together, weighted by their PMI value. Then use a graph-clustering algorithm to group the "classes".


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.