2
$\begingroup$

I have a huge amount of documents and every document has its own portrait, where a portrait has this structure (document_id, word, weight). TFIDF, basically.

I want to cluster these documents into different clusters, say, 10.
I'm trying to implement the K-Means algorithm with sklearn, but I have almost zero experience with data science whatsoever. All tutorials that I found get texts as input from Wikipedia or somewhere else, but I don't have access to the texts themselves. I have only their portraits. Hope that makes sense.

Is this something that can be achievable with sklearn and if so, can you guide me where to dig or what to look at

$\endgroup$

2 Answers 2

0
$\begingroup$

You can use these words with their weight as a vector representation of the document. The important point is to make all the documents vectors over the full vocabulary, so that any position $i$ in any vector always represents the same word $w_i$. This means that a vector should contain zeros in all the positions corresponding to a word which is not in the document.

Using these vectors you can indeed use k-means to cluster the documents. Of course the quality of the results depends on the data: if there are very few words in common, it cannot work very well.

$\endgroup$
0
$\begingroup$

I do not know the nature of words you have, but you could start with cosine similarity. It is based on the number of common words between 2 sentences/documents.

You could then extend this to include syntactically similar words using word embeddings which would translate words into numeric vectors and then perform operations on them.

If simple cosine similarity doesn't work, you will have to read about other measures of similarities or word embeddings.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.