10

The cosine distance formula is: And the formula used by the cosine function of the spatial class of scipy is: So, the actual cosine similarity metric is: -0.9998. So, it signifies complete dissimilarity.


9

Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance. Given two probability distributions, $P$ and $Q$, Hellinger distance is defined as: $$h(P,Q) = \frac1{\sqrt2}\cdot \|\sqrt{P}-\sqrt{Q}\|_2$$ It is useful when quantifying the difference between two ...


8

Well, there is a book called Deza, Michel Marie, and Elena Deza. Encyclopedia of distances. Springer Berlin Heidelberg, 2009. ISBN 978-3-642-00233-5 I guess that book answers your question better than I can... Choose the distance function most appropriate for your data. For example, on latitude and longitude, use a distance like Haversine. If you have ...


8

The are some techniques to choose the number of clusters K. The most common ones are The Elbow Method and The Silhouette Method. Elbow Method In this method, you calculate a score function with different values for K. You can use the Hamming distance like you proposed, or other scores, like dispersion. Then, you plot them and where the function creates "...


6

According to this interesting paper, Manhattan distance (L1 norm) may be preferable to Euclidean distance (L2 norm) for the case of high dimensional data. The authors of the paper even go a step further and suggest to use Lk norm distances, with a fractional value of k, for very high dimensional data in order to improve the results of distance-based ...


5

There are several important points to keep in mind in considering your questions: You should always normalize or standardize your data before applying k-means clustering. This is true of most other clustering algorithms also. If you are clustering in more than one dimension, the distance metric is meaningless unless each dimension has the same weight, so ...


5

For example, consider the green line. What is its length? In $L_2$, the answer is $1$, in $L_1$, the answer is $1$ as well. Now, for the same line, let's rotate it $45^\circ$ counterclockwise. What is the length again? In $L_2$, its length remains to be $1$. However, in $L_1$, using Manhattan distance, it's length is now $\frac{1}{\sqrt{2}}+\frac{1}{\sqrt{...


4

This is similar to the fundamental information theory problem that Shannon explored. In that domain, it is framed this way: given two rvs, X and Y, what information does X convey about Y? An example would be that I create sequence of numbers/bit/letters from a known PDF (X) and you receive a distorted version of those values for which you also know the PDF ...


4

The easiest solution for a non-Euclidean cluster center is the medoid, as in the algorithm PAM. This works with arbitrary metrics, but unfortunately for a 2-element cluster the result by definition is random (by metric properties, each point is an equally good medoid). Or you step back and rethink what the center is. It is the point which has the least ...


4

Great question, @gsamaras! The way you've set up this experiment makes a lot of sense to me, from a design point of view, but I think there are a couple aspects you can still examine. First, it's possible that uninformative features are distracting your classifier, leading to poorer results. In text analytics, we often talk about stop word filtering, which ...


4

I see a lot of people post this similar question on StackExchange, and the truth is that there is no methodology to compare if data set A looks like set B. You can compare summary statistics, such as means, deviations, min/max, but there's no magical formula to say that data set A looks like B, especially if they are varying data sets by rows and columns. I ...


4

I would take a look at Canonical correlation Analysis.


4

I can suggest a couple ideas, from wikipedia. If you want to place less emphasis on outliers, manhattan distance will try to reduce all errors equally since the gradient has constant magnitude. If your noise is distributed Laplacian, the MLE is found by minimizing the manhattan estimate.


4

Let $x, y \in \mathbb{R}^n$. The Euclidean distance $d$ is defined as $$ d(x,y) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}. $$ The squared Euclidean distance is therefore $$ d(x,y)^2 = \sum_{i=1}^n (x_i - y_i)^2. $$ We know that Euclidean distance is a metric. Let us check whether squared Euclidean distance is also a metric. I will use the definition from ...


4

No need for algorithms, or recommendation systems. You have: For each user a have a bunch of features. As long as they're numeric, or can be made numeric (e.g. aggregating the values or one-hot-encoding them), you already have distances. What you may not have is the proper variance across the feature space, i.e. features are scaled in different orders ...


4

It is not a case of paired nominal data. Hence, Mc Nemar's test can not be applied to check whether there is a higher fatality rate in Philippines ?. THE fatality rate is given for Philippines and world ( excluding Philippines ). As defined, it is expressed as proportion. Therefore, t-test/z-test shall be appropriate given that you meet other conditions ...


3

I found something which might be intuition about this problem in Hands-On Machine Learning with Scikit-Learn and TensorFlow Both the RMSE and the MAE are ways to measure the distance between two vectors: the vector of predictions and the vector of target values. Various distance measures, or norms, are possible: Computing the root of a sum of ...


3

You can try using an auto-encoder which also is a non-linear dimension reduction technique. It uses a neural network framework to find the most efficient transformation from $p$ dimensions down to whatever you choose. It then finds how well it can reconstruct the origianl $p$ variables, and keeps tuning until it can optimally recreate the original values ...


3

The object with the lowest mean distance (= the object with the lowest distance sum) is known as the medoid and the basis of k-medoids algorithms such as PAM. Because you must not use k-means with arbitrary distances.


3

Since this dataset is already organised in a table, you can leverage standard SQL functions to perform a large part of the cleanup. A record seems to be composed of 4 fields, for example: university name, city, state, country stanford law school - stanford - ca - united states of america You could follow these steps to get a cleaner representation of this ...


3

Like suggested in one answer on this SO question, you could use elastic matching with Levenshtein distance to your task. Levenshtein distance obeys triangle inequality and is therefore a metric distance. Use of elastic matching was suggested for time series data comparison. Levenshtein distance works with characters data. There is an implementation of ...


3

You can think of examples as vectors in $\mathbb{R}^p$, where $p$ is the number of features. Two examples will be very similar if the distance between them is close to $0$ (in the extreme case, if two examples are equal their euclidean distance is $0$). One way to measure the distance is using euclidean distance, but other distances can be used, as cosine ...


3

a(i) : the average distance between 'i' and all other data within the same cluster (source) b(i) : the lowest average distance of 'i' to all points in any other cluster, of which 'i' is not a member (source) So, from the question, a(i) will be 24 as point 'Pi' belongs to cluster A and b(i) will be 48 as it is the least average distance that 'Pi' has from ...


3

You can use sklearn's euclidean_distances function. http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.euclidean_distances.html#sklearn.metrics.pairwise.euclidean_distances In [1]: from sklearn.metrics.pairwise import euclidean_distances In [2]: import numpy as np In [3]: x = np.random.rand(10, 3) In [4]: euclidean_distances(x)


3

There are hundreds of algorithms to choose from. Hierarchical clustering in it's myriad of variants. Cut the dendrogram as desired, e.g., to get k clusters PAM, the closest match to k-means on a distance matrix (minimizes the average distance from the cluster center) Spectral clustering DBSCAN OPTICS HDBSCAN* Affinity Propagation ...


3

Within the documentation for HDBSCAN (Hierarchical DBSCAN), there is a really nice comparison of clustering algorithms. It is a bit biased, highlighting its own strengths (of course), but will still give you the examples and some boilerplate code to get up and running quickly. DBSCAN and HDBSCAN are generally known not to be so good at handling high variance ...


3

Well, one may argue that DBSCAN is based on all pairwise distances, but it uses data indexing to avoid computing all of them using geometric bounds. And there are other examples if you browse through literature. For example, the classic CLARA method is an approximation to PAM that avoids computing all pairwise distances. And there are many more such ...


3

This is counter-intuitive, because one would expect [0,2,0,0] to be more similar to [0,1,0,0] than [0,1,1,0]. No this is expected, since the two points are exactly at the same distance in the Euclidean space. To see it take a simplified 2D version of your points: A (1,0) B (2,0) C (1,1) Both B and C are exactly at distance 1 from A. But for my ...


3

I'd suggest looking at hierarchical clustering: It's simple so you could implement and tune your own version It lets you decide at which level you want to stop grouping elements together, so you could have a maximum distance. Be careful however that this approach can sometimes lead to unexpected/non-intuitive clusters.


3

I would try DBSCAN algorithm first: fairly easy to tune (with, in particular, a notion of distance as you requested), and does not need to know the number of clusters. There are a few other algorithms that can help you decide the number of clusters: Bayesian Gaussian Mixtures (see sklearn implementation) for instance, but it requires a bit more knowledge ...


Only top voted, non community-wiki answers of a minimum length are eligible