1
$\begingroup$

I want to assess how similar, or different two corpora are, and if the similarity is statistically significant. Something close to a Kolmogorov–Smirnov test in statistics, but for text data.

For additional context, two corpora are related to the same event, and one corpus is subsequently larger than the other.

Any leads /suggestions on this would be greatly appreciated. Thanks

$\endgroup$

1 Answer 1

1
$\begingroup$

First, there are different of text similarity: lexical (using mostly same words), semantic (talking mostly about the same topics), stylistic (written mostly in the same style, possibly by the same author), etc.

I'm assuming the most standard case, i.e. that you're looking for some kind of semantic similarity. There are different methods:

  • Probably the most simple would be to represent the two texts as TFIDF vectors and use cosine TFIDF to compare them. Mind that preprocessing options can have a huge impact, for example filtering out low frequency, lemmatizing, etc.
  • More advanced methods would represent the two texts as embeddings and compare these vectors. This requires pretrained word embeddings and would require more computation.

Anyway there is no binary answer, text similarity is typically considered continuous. This makes more sense, because there is often no clear answer: different people may not agree whether two texts are similar or not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.