1
$\begingroup$

I'm currently working on a project that relies on the clustering of documents into an unknown number of clusters, based on a similarity threshold (ideally using cosine distance between tf-idf vectors).

I'm attracted to elasticsearch for the project due to the 'out-of-the-box' similarity metrics provided when querying by string, however, I'm looking for some guidance as I'm very new to this. If anyone could provide any critique of the following approach, I'd be very grateful.

Is this whole approach horrible and inefficient? How feasible is this? Am I asking 'too much' of ES for this task?

Any help would be greatly appreciated, sorry for the long read. Thanks :)

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
0
$\begingroup$

I'm afraid I have zero experience with ElasticSearch, so I can't help with that but here are a few thoughts about the process you propose.

Overall I think the approach makes sense, especially in terms of efficiency but also check the following points:

  • This is not strictly speaking a clustering task, in the sense that the system doesn't find itself how to separate the clusters since this is based on a predefined threshold.
  • The threshold is crucial and you need a method to determine it. You might also want to plan for updating the threshold as the collection grows. The process is likely to lead to some very big clusters and many very small ones (unique document in a cluster).
  • If you assign a document to the best match cluster, it means that a document belongs to a single cluster. Depending on the application this could be problematic, with some documents expected in a cluster but found in another.
Improve this answer
$\endgroup$
0
$\begingroup$

not sure if its a best approach

1.get the top n similar documents for your new sample doc using cosine similarity in ES

2.get the cluster centroids of your top n document got from ES from the clustering model that you have used

3.If you feel there might be FP in results then to remove false positives do a cosine similarity again between you sample doc vector and cluster centroids (eg 10 vectors in from of a matrix )of those top n documents and sort the scores and chose the cluster whose score is close to 1 .

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.