Normalization is not always required, but it rarely hurts.
K-means clustering is "isotropic" in all directions of space and
therefore tends to produce more or less round (rather than elongated)
clusters. In this situation leaving variances unequal is equivalent to
putting more weight on variables with smaller variance.
When using IPython, you very nearly don't have to worry about it (at the expense of some loss of efficiency/greater communication overhead). The parallel IPython plugin in StarCluster will by default start one engine per physical core on each node (I believe this is configurable but not sure where). You just run whatever you want across all engines by ...
A general rule of thumb is to not distribute until you have to. It's usually more efficient to have N servers of a certain capacity than 2N servers of half that capacity. More of the data access will be local, and therefore fast in memory versus slow across the network.
At a certain point, scaling up one machine becomes uneconomical because the cost of ...
I play around quite a bit with location data and have found examples both where k-means works fine and where k-means is a poor representation and DBSCAN is a great fit.
If you've ever gone hiking or mountain climbing on a day with fog or a low cloud cover, there are times where you get to the top of the peak and can only see the surrounding peaks poking up ...
All things considered equal (cost, CPU perf, etc.) you could choose the smallest instance that can hold all of my dataset in memory and scale out. That way
you make sure not to induce unnecessary latencies due to network communications, and
you tend to maximize the overall available memory bandwidth for your processes.
Assuming you are running some sort ...
When I run the code you posted, I get three points on my plot:
The "point" at (0, 4) corresponds to X and the "point" at (0, 5) is actually three points, corresponding to X, X, and X. The point at (5, 5) is the last point in your X array. The data at (0, 4) and (0, 5) belong to one cluster, and the point at (5, 5) is considered noise (plotted in ...
Standardizing data is recommended because otherwise the range of values in each feature will act as a weight when determining how to cluster data, which is typically undesired.
For example consider the standard metric for most clustering algorithms (including DBSCAN in sci-kit learn) -- euclidean, otherwise known as the L2 norm. If one of your features has ...
There is no good evaluation data at all for clustering that would allow such conclusions.
There isn't even good real data where you could say variant 1 of k-means is better than variant 2 of k-means.
There is also no good evaluation measure that would handle the notion of "noise" well either.
So: don't go by some number.
Clustering is about solving a ...
Harmonic mean is less sensitive to outliers.
Say that you have 3 numbers: 1 1 and 30001
Their arithmetic mean (average) is 10001 and their harmonic mean is 2 that is much closer to the majority of the points.
So in a way, using an harmonic mean is more forgiving.
A Bayesian look at the harmonic mean
Assuming that the closer a point is, the more likely ...
why the equality of both partial derivatives correspond to these
hypothesis. I would rather understand when one partial derivative
equals the other partial derivative multiplied by minus one.
Your intuition of "trading off" by "subtracting" a value is correct when you speak in terms of $\Delta R$ and $\Delta P$ (as you yourself noticed in the edit), but ...
The complexity of that algorithm is O(n³), and it needs O(n²) memory.
So if your data grows "exponentially", you better settle for a sampling-based approach!
Seriously: benchmark the run time and memory requirements for 5k, 10k, 20k, 40k, 80k instances. You should be able to observe something between O(n²) (for computing the distance matrix) and O(n³) for ...
You are overfitting your data. You are fitting 5 clusters for ~20 data points. The red and blue clusters only have a single data point. Either get more data or fit fewer clusters.
The Elbow method will help decide how many clusters are appropriate.
If you read the package News section you will find:
pam() now signals an error when there are more than 65536 observational units (whereas it could segfault previously), thanks to a patch from Mikko Korpela, Helsinki.
So it seems there isn't a solution.
You're likely going to have to do a little data wrangling to get the data in a better format.
I'm assuming each file has a varying amount of rows, one for each game the user purchased? So, you could create one big matrix with the rows representing users and the columns representing games, and create an indicator matrix for to map purchases to games. Then ...
Clustering is the wrong tool for this. And so is cosine similarity, bag of words model etc.
Obviously, standard scaling on house numbers and zip codes is a svery bad idea, don't you think?
There are special techniques developed for this kind of data with a reason. For example, it's unlikely that the customer entered house number 100 but lives in 5. That ...
Yes, it causes almost no trouble. The only caution you must have is the possibility of having a regular ordinality where the ordinality is actually irregular. For example, having shirt size 1, 2 and 3 but the size "3" is not three times bigger than size "1".
The dummy variable creation is a very valid approach when having categorical variables in your ...
If your objective is to find clusters of users, then you are interested in finding groups of "similar" reviewers.
Therefore you should:
Retain information which relates to the users in a meaningful way - e.g. votes_for_user.
Discard information which has no meaningful relationship to a user - e.g. user_id (unless perhaps it contains some information such ...
Great question, I will try to answer the aspects related to dimensionality reduction mentioned above.
$Dimensionality\: Reduction:$ The number of dimensions which you want to keep after doing PCA is an experimental value you can experiment with the number of dimensions and check you results. Although you have mentioned all 40 features are independent i would ...
KMeans does correctly do what it is supposed to do.
Just plot your data correctly, with the same scale on both axes...
Y deviations do not matter, they are tiny compared to the X axis. Deviations there are 100x larger, so squared deviations even 10000x. Since KMeans minimized squared errors, only x matters
When plotted correctly, your data more looks like ...
It depends on how the cluster (i.e., compute resources) are being managed. For example, Apache Mesos is an open-source cluster management tool.
One established option is Horovod. It is designed to manage workloads for
TensorFlow and Keras across multiple cores in a single machine or across multiple machines.
You can either use the medoid, you can sometimes compute a centroid (and just ignore that it may be outside of the cluster), or you can do pairwise comparisons and take the average of that rather than comparing centers.
Optimal in which sense?
The crucial thing with clustering is that there is no optimal solution. Different solutions tell you a different part of the story. And to be able to get different views, you will need parameters. It is a exploratory technique.
Various attempts at defining "optimal" solutions have failed for practical use, just think of k-means.
A heap works in clustering the same way it works outside of clustering.
It's purpose is to efficiently find the minimum or maximum of a set, remove it, then find the next.
Efficient implementations of heaps in scripting languages may be impossible. For good performance, you usually need low-level memory access to avoid copying.
I will be using these facts without proof (but the proofs either follow directly from definitions or are straightforward): The vector of the centroid of a set of points is arithmetic mean of all vectors of the points in the set. Also, the centroid of the union of two sets of point is on the straight line connecting centroids of these two sets, and it divides ...
Mahalanobis distance depends on the covariance matrix, which is usually local to each cluster.
If you want a distance of two clusters, the following two approaches stand out:
the weighted average distance of each object to the other cluster, using the other clusters Mahalanobis distance. You could approximate this by using the distance of the centroid only....
---------Please check the edits also for this answer---------------------
According to me, its very application specific, and depends on what you want to do. I will prefer second approach in a generic application because if 2 clusters between whom we are calculating distance are having high standard deviation, should have small distance.
Another approach I ...
These weights should be introduced by a user. With a weight you tell the K-means algorithm, that one feature is more important than the other.
 These might represent a measure of importance, a frequency count, or some other information. The intent is that a point with a weight of 5.0 is twice as "important" as a point with a weight of 2.5, for instance....
It isn't strictly necessary to standardise, whether it is required or not may depend on the distance metric you choose.
For instance if you opt for the Mahalanobis distance then separation will be based upon the number of standard deviations separating points and not the absolute distance between them, as such it is a scale invariant metric.
As with many ...
I don't understand the purpose of imposing a condition that requires any cluster to contain at least some (let's say even 1) point from every dataset but at the same time find a solution that also allows that condition to be broken by outliers (I assume these are specific cluster outliers or data set outliers?).
Have you considered implementing an ...