4
They won't necessarily be the same. Consider observations equally distributed over a circle (radius = 1). Depending on the initial centroids, the algorithm will converge on different solutions. For instance, consider the case where two centroids are initially located on each side of one of the circle's diameters. Those can be any pair of points, and the ...
4
The are some techniques to choose the number of clusters K.
The most common ones are The Elbow Method and The Silhouette Method.
Elbow Method
In this method, you calculate a score function with different values for K. You can use the Hamming distance like you proposed, or other scores, like dispersion.
Then, you plot them and where the function creates "...
3
Looking at your different steps, the important thing to do is check which step would be affected by outliers.
Removing missing values is not affected because this step is not dependent on other data points present (or not) in the dataset.
However, normalizing your data is. Indeed, let's say your outliers contain extreme values, this will affect the ...
3
Always use a supervised algorithm when you have labeled data for your problem. Why would you ignore the labels, your most valuable bit of information?
To improve quality, you most likely need to improve your features.
2
If you have outliers, the best way is to use a clustering algorithm that can handle them.
For example DBSCAN clustering is robust against outliers when you choose minpts large enough. Don't use k-means: the squared error approach is sensitive to outliers. But there are variants such as k-means-- for handling outliers.
1
Either can be good or bad. It's not necessary to apply the same transformation to all attributes; but if youe attributes differ that much, you likely have some major problem anyway.
It depends on what you want to achieve what is appropriate. Try to approach it from having to explain your choices later. Can you argue why you transform the data this way? What ...
1
That's a good question.
Is there such a thing? Can we use the tree to identify which combination will give us higher cluster purity?
Clustering and simple decision tree fitting are used together in many cases such as:
First, like you mentioned quality of clustering can be measured by using decision tree leafs. I heard this calculation first time (I know ...
1
The nth centroid is chosen from a distribution proportional to $D(x)^2$, but pay careful attention to how $D(x)$ is defined. From the paper (top of page 3):
In particular, let $D(x)$ denote the shortest distance from a data point to the closest center we have already chosen.
Notice that $D(x)$ is the distance from $x$ to the nearest centroid. Compare ...
1
You can cluster discrete data using Jaccard index as a similarity metric. States that share more symptoms and diagnoses will have higher Jaccard index values. The Jaccard index values can be thresholded to form clusters.
1
Interesting problem...
If I understand correctly you'd like to obtain clusters of states which have similar patterns/proportions of symptom+diagnosis, right?
If yes, I would suggest you reorganize the data so that one instance represents a state, with its features being the frequency of each pair (symptom, diagnosis). Based on this representation you ...
1
Since you are looking for a degree of similarity regarding $y$ and the values of $x_1,...,x_5$ do not matter you can view this as a clustering problem regarding $y$:
Let $y_1,...y_5$ be the target values with $f(x_i)=y_i$ for $i\in\{1,...5\}$ then you need to define a distance measure $d(y_i,y_j)$ which, since your variables $y$ are continuous, could be the ...
1
The best method would be to calculate the Haversine distance between the new point and the GeoJSON object (Point, LineString, Polygon, MultiPoint, MultiLineString, and MultiPolygon). Haversine distance is the great-circle distance over the surface of a sphere between two points.
1
Depending on how many datapoints your pattern has and how often you have to do it the easiest way might be a brute force approach and just calculate the distance of all points and check if the minimum is in reach. If your input data is to sparse you could also create first interpolate points to have a decent density of points.
1
This blog has the solution for short text similarity. They mainly use the BERT neural network model to find similarities between sentences.
https://medium.com/@vimald8959/sentence-categorisation-short-text-similarity-61bb88fae15e
Only top voted, non community-wiki answers of a minimum length are eligible
