I'm relatively new to the NLP and DataScience, so apologies for omission or things like this.

I've been trying to use the KMeans to classify a list of 1000 of unique URLs containing several keywords that span across the taxonomy of a site. The intent is to understand how the algorithm would see these URLs fitting the best and - up to a degree - validate what the taxonomy exercise completed by the specialist.

The first step was to vectorise the URLs, obtaining an array of 1416 features.

I could run the KMeans without no problem, but here the first challenge: how can I verify the number of clusters. As far as I understand could be done with the Elbow algorithm, and the right number should be where the curve happens. In my case, it seems there is not a curve but a progressive degradation of the number of items falling in each group. See image for a range 1 to 40. If I have to see a curve, I'd say this is 17.

enter image description here

Assuming for a moment the number is correct, my next challenge is, how can I visualise the clusters and how can I understand about any outliers?

I thought that using a scatterPlot would have been a good idea, but with my data set this doesn't seem an option. I still have an array of 1464 feature each row vs the 2 data points needed.

Any idea on how I can proceed?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.