1
$\begingroup$

How can I perform conceptual clustering in sklearn? My use case is that I have English Wikipedia articles that I'm doing unsupervised learning on (tfidf -> truncated svd -> l2 normalize), and I'd like to create a hierarchy for them such that the nodes at the top are the most general articles (e.g. Programming Languages -> Functional Languages -> Haskell).

I tried using hierarchy.linkage, but it seems that the algorithm uses n^2 space, and I ran out of memory. I also tried using a KDTree on the l2 normalized vectors, and then setting each node to be the normalized sum of its children recursively, but this did not produce desirable results.

What is the right way to perform conceptual clustering with cosine similarity in scikit-learn without using quadratic space?

$\endgroup$

1 Answer 1

0
$\begingroup$

Scikit-learn does not natively support conceptual clustering. You'll have to implement conceptual clustering yourself or find another implementation.

To use a hierarchy of knowledge, you'll have to find an existing one or create your own. Unsupervised learning is not a useful method to create a hierarchical structure based on semantic meaning.

$\endgroup$
1
  • $\begingroup$ on top of @Brain's answer, if you are out-of-memory, check this out. However, Scikit-learn is not designed to for huge data (e.g. no GPU support), so you may better consider alternative frameworks too. $\endgroup$
    – lpounng
    May 16, 2022 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.