Given that a document is an object represented by what is called a term frequency vector. How can we calculate the dissimilarity between term frequency vectors?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
There are several ways to find the relationship between vector representations in NLP, such as the cosine distance (you can check this for instance to apply it as a quick proof of concept) or L2 distance, which aim to find the relationship between such vectors in the vectors space they lay in.
Nevertheless, to associate the geometric distance to semantic similarity, it is interesting to apply word embeddings, with which you get more lower-dimensional vectors directly learned from your data (you can check for it).
$\begingroup$
$\endgroup$
Given two vectors of attributes, A and B, the cosine similarity, cos(θ), is represented using a dot product and magnitude as
${\displaystyle {\text{cosine similarity}}=S_{C}(A,B):=\cos(\theta )={\mathbf {A} \cdot \mathbf {B} \over \|\mathbf {A} \|\|\mathbf {B} \|}={\frac {\sum \limits _{i=1}^{n}{A_{i}B_{i}}}{{\sqrt {\sum \limits _{i=1}^{n}{A_{i}^{2}}}}{\sqrt {\sum \limits _{i=1}^{n}{B_{i}^{2}}}}}},}$
Therefore the dissimilarity can be calculated as 1- ${S_{C}(A,B):=\cos(\theta )}$