10

The cosine distance formula is: And the formula used by the cosine function of the spatial class of scipy is: So, the actual cosine similarity metric is: -0.9998. So, it signifies complete dissimilarity.


8

Well, there is a book called Deza, Michel Marie, and Elena Deza. Encyclopedia of distances. Springer Berlin Heidelberg, 2009. ISBN 978-3-642-00233-5 I guess that book answers your question better than I can... Choose the distance function most appropriate for your data. For example, on latitude and longitude, use a distance like Haversine. If you have ...


7

Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance. Given two probability distributions, $P$ and $Q$, Hellinger distance is defined as: $$h(P,Q) = \frac1{\sqrt2}\cdot \|\sqrt{P}-\sqrt{Q}\|_2$$ It is useful when quantifying the difference between two ...


6

The are some techniques to choose the number of clusters K. The most common ones are The Elbow Method and The Silhouette Method. Elbow Method In this method, you calculate a score function with different values for K. You can use the Hamming distance like you proposed, or other scores, like dispersion. Then, you plot them and where the function creates "...


5

There are several important points to keep in mind in considering your questions: You should always normalize or standardize your data before applying k-means clustering. This is true of most other clustering algorithms also. If you are clustering in more than one dimension, the distance metric is meaningless unless each dimension has the same weight, so ...


4

This is similar to the fundamental information theory problem that Shannon explored. In that domain, it is framed this way: given two rvs, X and Y, what information does X convey about Y? An example would be that I create sequence of numbers/bit/letters from a known PDF (X) and you receive a distorted version of those values for which you also know the PDF ...


4

The easiest solution for a non-Euclidean cluster center is the medoid, as in the algorithm PAM. This works with arbitrary metrics, but unfortunately for a 2-element cluster the result by definition is random (by metric properties, each point is an equally good medoid). Or you step back and rethink what the center is. It is the point which has the least ...


4

Great question, @gsamaras! The way you've set up this experiment makes a lot of sense to me, from a design point of view, but I think there are a couple aspects you can still examine. First, it's possible that uninformative features are distracting your classifier, leading to poorer results. In text analytics, we often talk about stop word filtering, which ...


4

According to this interesting paper, Manhattan distance (L1 norm) may be preferable to Euclidean distance (L2 norm) for the case of high dimensional data: https://bib.dbvis.de/uploadedFiles/155.pdf The authors of the paper even go a step further and suggest to use Lk norm distances, with a fractional value of k, for very high dimensional data in order to ...


4

I see a lot of people post this similar question on StackExchange, and the truth is that there is no methodology to compare if data set A looks like set B. You can compare summary statistics, such as means, deviations, min/max, but there's no magical formula to say that data set A looks like B, especially if they are varying data sets by rows and columns. I ...


4

No need for algorithms, or recommendation systems. You have: For each user a have a bunch of features. As long as they're numeric, or can be made numeric (e.g. aggregating the values or one-hot-encoding them), you already have distances. What you may not have is the proper variance across the feature space, i.e. features are scaled in different orders ...


3

I would take a look at Canonical correlation Analysis.


3

I can suggest a couple ideas, from wikipedia. If you want to place less emphasis on outliers, manhattan distance will try to reduce all errors equally since the gradient has constant magnitude. If your noise is distributed Laplacian, the MLE is found by minimizing the manhattan estimate.


3

I found something which might be intuition about this problem in Hands-On Machine Learning with Scikit-Learn and TensorFlow Both the RMSE and the MAE are ways to measure the distance between two vectors: the vector of predictions and the vector of target values. Various distance measures, or norms, are possible: Computing the root of a sum of ...


3

You can try using an auto-encoder which also is a non-linear dimension reduction technique. It uses a neural network framework to find the most efficient transformation from $p$ dimensions down to whatever you choose. It then finds how well it can reconstruct the origianl $p$ variables, and keeps tuning until it can optimally recreate the original values ...


3

The object with the lowest mean distance (= the object with the lowest distance sum) is known as the medoid and the basis of k-medoids algorithms such as PAM. Because you must not use k-means with arbitrary distances.


3

Since this dataset is already organised in a table, you can leverage standard SQL functions to perform a large part of the cleanup. A record seems to be composed of 4 fields, for example: university name, city, state, country stanford law school - stanford - ca - united states of america You could follow these steps to get a cleaner representation of this ...


3

Let $x, y \in \mathbb{R}^n$. The Euclidean distance $d$ is defined as $$ d(x,y) = \sqrt{\sum_{i=1}^n (x_i - y_i)^2}. $$ The squared Euclidean distance is therefore $$ d(x,y)^2 = \sum_{i=1}^n (x_i - y_i)^2. $$ We know that Euclidean distance is a metric. Let us check whether squared Euclidean distance is also a metric. I will use the definition from ...


3

Like suggested in one answer on this SO question, you could use elastic matching with Levenshtein distance to your task. Levenshtein distance obeys triangle inequality and is therefore a metric distance. Use of elastic matching was suggested for time series data comparison. Levenshtein distance works with characters data. There is an implementation of ...


3

You can think of examples as vectors in $\mathbb{R}^p$, where $p$ is the number of features. Two examples will be very similar if the distance between them is close to $0$ (in the extreme case, if two examples are equal their euclidean distance is $0$). One way to measure the distance is using euclidean distance, but other distances can be used, as cosine ...


3

You can use sklearn's euclidean_distances function. http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.euclidean_distances.html#sklearn.metrics.pairwise.euclidean_distances In [1]: from sklearn.metrics.pairwise import euclidean_distances In [2]: import numpy as np In [3]: x = np.random.rand(10, 3) In [4]: euclidean_distances(x)


3

There are hundreds of algorithms to choose from. Hierarchical clustering in it's myriad of variants. Cut the dendrogram as desired, e.g., to get k clusters PAM, the closest match to k-means on a distance matrix (minimizes the average distance from the cluster center) Spectral clustering DBSCAN OPTICS HDBSCAN* Affinity Propagation ...


3

For example, consider the green line. What is its length? In $L_2$, the answer is $1$, in $L_1$, the answer is $1$ as well. Now, for the same line, let's rotate it $45^\circ$ counterclockwise. What is the length again? In $L_2$, its length remains to be $1$. However, in $L_1$, using Manhattan distance, it's length is now $\frac{1}{\sqrt{2}}+\frac{1}{\sqrt{...


3

This is counter-intuitive, because one would expect [0,2,0,0] to be more similar to [0,1,0,0] than [0,1,1,0]. No this is expected, since the two points are exactly at the same distance in the Euclidean space. To see it take a simplified 2D version of your points: A (1,0) B (2,0) C (1,1) Both B and C are exactly at distance 1 from A. But for my ...


2

You cannot just use arbitrary distance functions with k-means. because the algorithm is not based on metric properties but on variance. https://stats.stackexchange.com/q/81481/7828 Fact is that k-means minimizes the sum of squares. This does not even give you the "smallest distances" but only the smallest squared distances. This is not the same (see: ...


2

I think I have read your question correctly it looks like you need a nearest neighbor implementation. If you are unfamiliar with the concept you can find the wiki article here https://en.wikipedia.org/wiki/Nearest_neighbor_search. I went ahead a wrote an example implementation you can use as a guide. Please note that this is a brute force method and not ...


2

Not unless you have an idea of what makes two people similar to each other. By choosing a distance function you are defining what it means for two people to be similar to one another. The question of whether one metric is quantitatively better than another may not be answerable without a defined objective. For example, compare the Manhattan distance (L1 ...


2

The use of Manhattan distance depends a lot on the kind of co-ordinate system that your dataset is using. While Euclidean distance gives the shortest or minimum distance between two points, Manhattan has specific implementations. For example, if we were to use a Chess dataset, the use of Manhattan distance is more appropriate than Euclidean distance. ...


2

A common method is defining "landmarks", a set of specific points that exist on every face . For example, the outside edge of each eye or the bottom of the chin are landmarks on most faces. Train the deep learning network to learn the relationship between these landmarks which will form the embedding space. Then define a distance metric in this embedding ...


2

a(i) : the average distance between 'i' and all other data within the same cluster (source) b(i) : the lowest average distance of 'i' to all points in any other cluster, of which 'i' is not a member (source) So, from the question, a(i) will be 24 as point 'Pi' belongs to cluster A and b(i) will be 48 as it is the least average distance that 'Pi' has from ...


Only top voted, non community-wiki answers of a minimum length are eligible