Normalization is not always required, but it rarely hurts.
K-means clustering is "isotropic" in all directions of space and
therefore tends to produce more or less round (rather than elongated)
clusters. In this situation leaving variances unequal is equivalent to
putting more weight on variables with smaller variance.
When using IPython, you very nearly don't have to worry about it (at the expense of some loss of efficiency/greater communication overhead). The parallel IPython plugin in StarCluster will by default start one engine per physical core on each node (I believe this is configurable but not sure where). You just run whatever you want across all engines by ...
A general rule of thumb is to not distribute until you have to. It's usually more efficient to have N servers of a certain capacity than 2N servers of half that capacity. More of the data access will be local, and therefore fast in memory versus slow across the network.
At a certain point, scaling up one machine becomes uneconomical because the cost of ...
I play around quite a bit with location data and have found examples both where k-means works fine and where k-means is a poor representation and DBSCAN is a great fit.
If you've ever gone hiking or mountain climbing on a day with fog or a low cloud cover, there are times where you get to the top of the peak and can only see the surrounding peaks poking up ...
All things considered equal (cost, CPU perf, etc.) you could choose the smallest instance that can hold all of my dataset in memory and scale out. That way
you make sure not to induce unnecessary latencies due to network communications, and
you tend to maximize the overall available memory bandwidth for your processes.
Assuming you are running some sort ...
Standardizing data is recommended because otherwise the range of values in each feature will act as a weight when determining how to cluster data, which is typically undesired.
For example consider the standard metric for most clustering algorithms (including DBSCAN in sci-kit learn) -- euclidean, otherwise known as the L2 norm. If one of your features has ...
When I run the code you posted, I get three points on my plot:
The "point" at (0, 4) corresponds to X and the "point" at (0, 5) is actually three points, corresponding to X, X, and X. The point at (5, 5) is the last point in your X array. The data at (0, 4) and (0, 5) belong to one cluster, and the point at (5, 5) is considered noise (plotted in ...
There is no good evaluation data at all for clustering that would allow such conclusions.
There isn't even good real data where you could say variant 1 of k-means is better than variant 2 of k-means.
There is also no good evaluation measure that would handle the notion of "noise" well either.
So: don't go by some number.
Clustering is about solving a ...
Harmonic mean is less sensitive to outliers.
Say that you have 3 numbers: 1 1 and 30001
Their arithmetic mean (average) is 10001 and their harmonic mean is 2 that is much closer to the majority of the points.
So in a way, using an harmonic mean is more forgiving.
A Bayesian look at the harmonic mean
Assuming that the closer a point is, the more likely ...
Neither, there is not enough discriminatory information in data (yet)
Dont squeeze the data until it tells you the truth. You can change the metric (malahobian distance for example) and the algo but you cant expect it to show miracles.
Using elbow method, as you increase the number of clusters it will always become more homogenous. You dont have a "kink" ...
why the equality of both partial derivatives correspond to these
hypothesis. I would rather understand when one partial derivative
equals the other partial derivative multiplied by minus one.
Your intuition of "trading off" by "subtracting" a value is correct when you speak in terms of $\Delta R$ and $\Delta P$ (as you yourself noticed in the edit), but ...
The complexity of that algorithm is O(n³), and it needs O(n²) memory.
So if your data grows "exponentially", you better settle for a sampling-based approach!
Seriously: benchmark the run time and memory requirements for 5k, 10k, 20k, 40k, 80k instances. You should be able to observe something between O(n²) (for computing the distance matrix) and O(n³) for ...
You are overfitting your data. You are fitting 5 clusters for ~20 data points. The red and blue clusters only have a single data point. Either get more data or fit fewer clusters.
The Elbow method will help decide how many clusters are appropriate.
If you read the package News section you will find:
pam() now signals an error when there are more than 65536 observational units (whereas it could segfault previously), thanks to a patch from Mikko Korpela, Helsinki.
So it seems there isn't a solution.
You're likely going to have to do a little data wrangling to get the data in a better format.
I'm assuming each file has a varying amount of rows, one for each game the user purchased? So, you could create one big matrix with the rows representing users and the columns representing games, and create an indicator matrix for to map purchases to games. Then ...
A Silhouette score close to 0 says the clustering is not reliable.
And the Elbow method is crap. On random data the curve would drop roughly like 1/(k-1); so it's largely undefined wh em they is an elbow and when not. In your case, what troubles me most is that the values appear to stagnate to a cake much larger than zero. Maybe there is an error in your ...
Yes, it causes almost no trouble. The only caution you must have is the possibility of having a regular ordinality where the ordinality is actually irregular. For example, having shirt size 1, 2 and 3 but the size "3" is not three times bigger than size "1".
The dummy variable creation is a very valid approach when having categorical variables in your ...
If your objective is to find clusters of users, then you are interested in finding groups of "similar" reviewers.
Therefore you should:
Retain information which relates to the users in a meaningful way - e.g. votes_for_user.
Discard information which has no meaningful relationship to a user - e.g. user_id (unless perhaps it contains some information such ...
The closest line of work to your problem is Multi-View Clustering. Each data set DS is considered as a view, and views share a central entity (e.g. an individual).
Multi-View Clustering is a survey on the topic. Also, here are two git projects from two papers (git 1, git 2) on the topic (they are implemented in Matlab and unfortunately are ...
Great question, I will try to answer the aspects related to dimensionality reduction mentioned above.
$Dimensionality\: Reduction:$ The number of dimensions which you want to keep after doing PCA is an experimental value you can experiment with the number of dimensions and check you results. Although you have mentioned all 40 features are independent i would ...
KMeans does correctly do what it is supposed to do.
Just plot your data correctly, with the same scale on both axes...
Y deviations do not matter, they are tiny compared to the X axis. Deviations there are 100x larger, so squared deviations even 10000x. Since KMeans minimized squared errors, only x matters
When plotted correctly, your data more looks like ...
It depends on how the cluster (i.e., compute resources) are being managed. For example, Apache Mesos is an open-source cluster management tool.
One established option is Horovod. It is designed to manage workloads for
TensorFlow and Keras across multiple cores in a single machine or across multiple machines.
You can either use the medoid, you can sometimes compute a centroid (and just ignore that it may be outside of the cluster), or you can do pairwise comparisons and take the average of that rather than comparing centers.
Optimal in which sense?
The crucial thing with clustering is that there is no optimal solution. Different solutions tell you a different part of the story. And to be able to get different views, you will need parameters. It is a exploratory technique.
Various attempts at defining "optimal" solutions have failed for practical use, just think of k-means.
A heap works in clustering the same way it works outside of clustering.
It's purpose is to efficiently find the minimum or maximum of a set, remove it, then find the next.
Efficient implementations of heaps in scripting languages may be impossible. For good performance, you usually need low-level memory access to avoid copying.
I will be using these facts without proof (but the proofs either follow directly from definitions or are straightforward): The vector of the centroid of a set of points is arithmetic mean of all vectors of the points in the set. Also, the centroid of the union of two sets of point is on the straight line connecting centroids of these two sets, and it divides ...
Mahalanobis distance depends on the covariance matrix, which is usually local to each cluster.
If you want a distance of two clusters, the following two approaches stand out:
the weighted average distance of each object to the other cluster, using the other clusters Mahalanobis distance. You could approximate this by using the distance of the centroid only....
---------Please check the edits also for this answer---------------------
According to me, its very application specific, and depends on what you want to do. I will prefer second approach in a generic application because if 2 clusters between whom we are calculating distance are having high standard deviation, should have small distance.
Another approach I ...
These weights should be introduced by a user. With a weight you tell the K-means algorithm, that one feature is more important than the other.
 These might represent a measure of importance, a frequency count, or some other information. The intent is that a point with a weight of 5.0 is twice as "important" as a point with a weight of 2.5, for instance....