3
There cannot be a unique answer to your question. There is a discrepancy in your question though -
I am aware that this is a classification problem on which I am working on.
Could you please help me with the right step by step guide that I should follow in order to achieve an efficient clustering at the end?
However, I am assuming that you are ...
3
If you expect that all zeros is a result of error in the measuring of the features (i.e. the observations should not be all 0s but they are), then I would say: Keep all the data, but increase k (from k-means) by 1. This extra one will hopefully become the class of all these wrong observations.
If you expect that all zeros is correct (i.e. these observations ...
3
It's a matter of data quality so it depends how the dataset was built:
Either these instances are meaningful, i.e. it makes sense that an observation would have zeros for all the features and that it would happen that often.
Or these are the result of an error, typically the complete absence of measurement for these observations.
Naturally one wants to ...
2
K-means don't modify the underlying structure of your data. K-means will just provide the 'color' part of your graph.
To answer the question about why do you get a cuboid, it's because your underlying data are a cuboid. Not necessarily by construction, but that's what happen when you cap your data. As an exemple, look at the following code :
X1 = c(rnorm(...
2
Since they are categorical variables, I would cluster them using the k-medoids clustering method. Before applying this method, one-hot encode all the predictors.
See a tutorial here:
https://towardsdatascience.com/k-medoids-clustering-on-iris-data-set-1931bf781e05
Sklearn has an implementation:
https://scikit-learn-extra.readthedocs.io/en/latest/generated/...
2
You can, as an example, create a binned field for the measure. The value range can be specified in the tooltip. I used a single color based continuous palette since (population) total is a continuous field.
[Edited]
Based on your comment, I tested with a fixed dimension based binned field for the continuous measure. At least in my example file, the values ...
2
I'd suggest looking at hierarchical clustering:
It's simple so you could implement and tune your own version
It lets you decide at which level you want to stop grouping elements together, so you could have a maximum distance.
Be careful however that this approach can sometimes lead to unexpected/non-intuitive clusters.
2
Not very familiar with k-medoids, but i guess it's something like k-means, right? If so, the most time-consuming part of the entire model is updating the medoids. We randomly select initial start and update the center of mass to have better cluster results.
I suggest you to pickle final_medoids. When you have new data, compute the pca, pass it to kmedoids ...
2
I would try DBSCAN algorithm first: fairly easy to tune (with, in particular, a notion of distance as you requested), and does not need to know the number of clusters.
There are a few other algorithms that can help you decide the number of clusters: Bayesian Gaussian Mixtures (see sklearn implementation) for instance, but it requires a bit more knowledge ...
1
You could use K-means clustering as well here with euclidian distance measure..
Why I am suggesting euclidian distance because you have all numeric data, if it was mixed then gover distance was better pick and similarly you could pick correct distance measure based on requirements.
Here you can get the optimal number of clusters by nbclust function in R.
1
you can have a look at these suggestions
Clustering categorical data
i havent tried clustering on pure categorical dataset yet however have tried on text data where at the end you end up creating a sparse matrix and have had success there with hierarchical clustering using Wards' method
https://towardsdatascience.com/understanding-the-concept-of-...
1
If you want to visualise the data after K-Means, the better approach would be to reduce the dimensionality to two or three dimensions and visualise using a matplotlib 2D or 3D plot.
You might also try pair plots but I don't think It would be much helpful from clustering stand point.
Only top voted, non community-wiki answers of a minimum length are eligible
