5
$\begingroup$

I am new to text processing and stuck on a problem to identify the similarity of columns. To detail the problem, consider we have two columns with string values:

Column A      |        Column B
-------------------------------
abcd          |          xyz
foo           |          bar
xyzzy         |          acct
xyz           |          world
onex          |          foo
...           |          ...
...           |          ...

The length of columns can be in order of thousands. Is there an approach to identify how similar the columns are?

Currently, I am creating Minhash signatures for both the columns and computing the Jaccard similarity b/w the signatures. But the problem is, the similarity scores are coming too low even for the columns which have a considerate overlap of values.

Then, I tried creating signatures by taking fractions of values that are most frequently occurring but that does not seem to help either.

Is there any other approach to work on this?

$\endgroup$
2
  • $\begingroup$ Maybe revise your code... MinHash seems completely redundant for an order of thousands of strings. If you can see a "considerate overlap" Jaccard similarity should show it too. $\endgroup$
    – Valentas
    Nov 12, 2021 at 9:27
  • $\begingroup$ Do you want to take into account the string similarity between pairs of strings, or just the number of strings in common? $\endgroup$
    – Erwan
    Nov 12, 2021 at 11:45

1 Answer 1

3
$\begingroup$

You could use similarity metrics for strings. There are a number of "off the shelf" packages to compare string similarity, such as stringdist for R.

The stringsim function - for instance - allows you to compare string similarity (and there are options to use different metrics).

Example (in R):

library(stringdist)

stringsim("cat", "catfish")
> [1] 0.4285714

# Also works with vectors
df = data.frame(a=c("cat","dog","tree"),b=c("catfish","hotdog","forest"))

stringsim(df$a,df$b, method="jaccard")
> [1] 0.4285714 0.6000000 0.5000000

Also see this github-repo for fuzzy-matching etc.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.